The Engineering Codex/Cloud Security Engineering
DAY 5
07 / 09

Software Supply Chain Security

schedule10 minsignal_cellular_altAdvanced2,271 words
From SolarWinds to xz, the recent security headlines have been supply chain. Master SLSA levels, SBOMs (CycloneDX/SPDX), Sigstore (cosign, Fulcio, Rekor), in-toto, build provenance, and the engineering practice that produces verifiable, signed, reproducible artifacts.

What you will learn

01The Supply-Chain Threat Model
02SLSA — Levels and What They Buy
03SBOM — Software Bill of Materials
04Sigstore — Keyless Signing for the Rest of Us
05in-toto and Attestations
06Build Reproducibility

Almost every recent high-impact incident — SolarWinds, Codecov, npm event-stream, ua-parser-js, the xz-utils backdoor, MOVEit — was a supply-chain compromise. The attacker did not breach the victim. They tainted something the victim trusted and built upon. The defenses are no longer optional: producing a signed, traceable, attestable software artifact is becoming the baseline expectation, codified in EO 14028, NIST SSDF, and the EU CRA.

🔑
The supply-chain spine
1) SLSA — the levelled framework for build integrity. 2) SBOM — what is in your artifact (CycloneDX or SPDX). 3) Sigstore — keyless signing of artifacts and attestations (cosign + Fulcio + Rekor). 4) in-toto — the format for attestations of every step in the supply chain. 5) Verification at admission — Kyverno + cosign at the cluster gate. The four together are the modern paved road from source to running container.

The Supply-Chain Threat Model

SLSA's threat model decomposes the chain into source → build → package → consumer, with eight distinct attack vectors at the joints. Memorize this picture; every recent incident maps onto exactly one of these arrows.

Sourcegit, branches, PRs BuildCI runner Packageregistry Consumercluster, end-user A: code submitted B: source repo C: build script D: deps E: build CI F: artifact G: package repo H: dep at use SLSA's eight threats. Every supply-chain incident maps onto one of these letters.
SolarWinds was an E (build CI tampering); event-stream was a D (malicious dependency); xz-utils was a B (slow infiltration of source).

What dominates the data

  • Dependency confusion / typosquatting — uploading my-internal-pkg to npm hoping a CI accidentally fetches it. Endemic in npm, PyPI, RubyGems.
  • Compromised maintainer accounts — phished/MFA-bypassed, then a malicious release. event-stream, ua-parser-js, the entire 2023 PyPI wave.
  • Build-system compromise — SolarWinds (Orion), Codecov (bash uploader), Reuters' MOVEit. The attacker writes themselves into the binary you trust.
  • Source repo infiltration — slow social engineering of a maintainer position (xz-utils 2024).
  • Registry poisoning / cache poisoning — modifying the package between upload and your fetch.

SLSA — Levels and What They Buy

SLSA (Supply-chain Levels for Software Artifacts) is the OpenSSF / Linux Foundation framework. Versions 1.0 (2023) and 1.1 (2024) define a build track with four levels (L0-L3) describing increasing rigor on how build integrity is established. Source and dependency tracks are in development.

LevelBuys youConcretely
L0Nothing.No provenance produced.
L1Provenance exists.The build emits a signed metadata document declaring source, builder, and inputs. Caught the SolarWinds class.
L2Provenance authenticated and from a hosted build platform.Builder signs with a key only it can use; provenance is non-forgeable. Hosted CI (GitHub Actions, GitLab, Buildkite-managed).
L3Hardened build platform.Build runs on isolated, hardened workers; non-falsifiable provenance; build script cannot tamper with provenance signing.

The reusable workflow path

The fastest way to SLSA L3 today is the slsa-github-generator — a set of reusable GitHub Actions workflows that produce SLSA-L3 provenance for Go binaries, Node packages, container images, and Python wheels. The workflow runs on hosted runners with isolation and emits a Sigstore-signed in-toto attestation alongside your artifact.

SBOM — Software Bill of Materials

You cannot defend what you cannot enumerate. An SBOM lists everything that went into your artifact: direct and transitive dependencies, license, version, hash, optionally provenance. When Log4Shell hit, the teams who recovered fastest were those with SBOMs in their CI artifact registry — they could query "which of our 400 services include log4j-core 2.14.x" in seconds. The teams without SBOMs spent days running grep across container images.

The two formats

  • SPDX — Linux Foundation, ISO/IEC 5962. Comprehensive, license-focused.
  • CycloneDX — OWASP. Tighter for security use; first-class support for vulnerability and VEX.

Both are widely supported. Generate both if your tooling allows; consumers will ask for whichever the auditor prefers. syft (Anchore), cdxgen, trivy, and Microsoft's sbom-tool all produce them from images, repos, or filesystems.

bash — generate, sign, and attach an SBOM to an image
# 1. Build SBOM from the image we just pushed
syft ghcr.io/my-org/payments:1.4.0 -o cyclonedx-json > sbom.cdx.json

# 2. Attach as an OCI artifact alongside the image
cosign attach sbom \
  --sbom sbom.cdx.json \
  --type cyclonedx \
  ghcr.io/my-org/payments:1.4.0

# 3. Sign the SBOM (keyless, OIDC identity from CI)
cosign attest \
  --predicate sbom.cdx.json \
  --type cyclonedx \
  ghcr.io/my-org/payments:1.4.0

VEX — telling the truth about vulnerabilities

An SBOM with raw CVE counts becomes wallpaper. VEX (Vulnerability Exploitability eXchange — NTIA / CycloneDX VEX / OpenVEX) lets you publish per-CVE statements: "in our build, CVE-2024-12345 in libfoo is not_affected because the vulnerable code path is not reached." Tools like Grype consume VEX to suppress the noise. Teams that ship VEX with their releases turn the scanner output back into a useful signal.

Sigstore — Keyless Signing for the Rest of Us

The historical reason almost no one signed software is that running PKI is hard. Sigstore (CNCF graduated, 2024) inverts the problem: no long-lived signing keys. Instead, your CI authenticates to Fulcio using its OIDC identity and gets a short-lived signing certificate (~10 minutes) that binds the signature to that identity. Every signature is logged in Rekor, a public transparency log. Verification looks up the identity, the issuer, and the Rekor entry.

CI runnerOIDC token Fulcio (CA)10-min cert Sign artifact & attestationscosign sign / attest Rekor (transparency log)tamper-evident, public No long-lived signing keys. Identity is the OIDC subject; integrity is the public log.
Keyless signing. Verification matches signature → Fulcio cert → OIDC subject → Rekor entry to a known builder identity.

Verification at admission

The full payoff lands at the cluster gate. Kyverno's verifyImages (Day 3) checks Sigstore signatures and rejects any pod referencing an image not signed by the expected OIDC subject — typically your release workflow. The check is a few lines; the payoff is enormous: a stolen registry credential cannot push a working malicious image, because the signing identity will be wrong.

yaml — Kyverno: only signed images, with provenance attestation
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: signed-and-attested }
spec:
  validationFailureAction: Enforce
  rules:
  - name: keyless-signed-by-our-release-workflow
    match: { resources: { kinds: ["Pod"] } }
    verifyImages:
    - imageReferences: ["ghcr.io/my-org/*"]
      attestors:
      - entries:
        - keyless:
            issuer: "https://token.actions.githubusercontent.com"
            subject: "https://github.com/my-org/*/.github/workflows/release.yml@refs/heads/main"
      attestations:
      - type: "https://slsa.dev/provenance/v1"
        attestors:
        - entries:
          - keyless:
              issuer: "https://token.actions.githubusercontent.com"
              subject: "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/*"

Two requirements: image is signed by our release workflow, and a SLSA provenance attestation exists, signed by the SLSA generator workflow. Forging both requires compromising GitHub's OIDC + the SLSA reusable workflow + Sigstore's transparency log. The attack surface is now "break the entire ecosystem", not "steal a registry token."

in-toto and Attestations

in-toto is the data format Sigstore wraps: a structured statement that something attests some predicate about some subject. The subjects are content-addressable (SHA-256 digests). The predicates are extensible — SLSA provenance, SPDX SBOM, vulnerability scan, code review approval. A modern release ships an attestation bundle alongside the artifact:

  • Provenance (SLSA) — built by X workflow at commit Y from inputs Z.
  • SBOM (CycloneDX/SPDX) — these are the components.
  • Vulnerability scan (Trivy/Grype) — known CVEs at build time.
  • VEX — exploitability statements.
  • Code review (review-bot attestation) — approved by N reviewers.
  • Test results — coverage, integration, fuzzing.

Verifiers like cosign verify-attestation + policy-controller let you write declarative policies: "the image must have an SLSA-L3 provenance, an SBOM, and zero CRITICAL CVEs without VEX." The result is a verifiable claim chain you can audit.

Build Reproducibility

The strongest provenance claim is reproducibility: rebuild from the same source and you get the bit-identical artifact. With reproducibility, anyone can independently verify a build did what its provenance says. Without it, you trust a builder.

Achieving reproducibility means controlling timestamps, file ordering, locale, randomness — the Reproducible Builds project documents techniques per language. Bazel and Nix are the build systems that get this right by construction; Buildah / ko for containers; Debian achieves it for the majority of its packages. For most teams, strict reproducibility is aspirational; prioritize SLSA L3 hosted-builder provenance first, work toward reproducibility for sensitive artifacts later.

Dependency Hygiene

The supply-chain is mostly someone else's source code. The hygiene rules are old and unglamorous:

  • Pin dependencies — hash-pinned (npm integrity, pip --require-hashes, Go's checksum DB, cargo Cargo.lock + --frozen).
  • Vendor or mirror — a private package mirror (Artifactory, Nexus, or AWS CodeArtifact / GCP Artifact Registry) is the only audit point you control. Direct upstream is a third-party trust delegation.
  • Restrict scope — each project pulls only the dependencies it declares; no transitive cross-project leaks. workspace-level restrictions matter at scale.
  • Auto-update with review — Renovate / Dependabot create PRs; CI runs the full suite; humans approve. The point is not zero-day patching but visibility.
  • Beware install-time scripts — npm's postinstall, Python's setup.py, Cargo build scripts. npm ci --ignore-scripts, pip install --no-build-isolation with reviewed wheels, and prebuilt-only dependencies eliminate huge classes of supply-chain payloads.
  • Watch the maintainer surface — projects with one maintainer, no recent activity, or recently transferred ownership are higher risk. scorecard (OpenSSF) automates checks.
🚨
Dependency confusion is a one-line attack
If your private package is named @my-org/internal-tool and lives only in your private registry, an attacker can publish internal-tool (no scope) on public npm. A misconfigured CI that searches public first will fetch the malicious one. The fix: always use scoped names, configure registry resolution to private-first or private-only for those scopes, and verify with hash-pinning. Microsoft, Apple, PayPal, Tesla and Yelp all paid security researchers for finding this in 2021.

The Producer-Side Paved Road

Pulling everything together, a modern "green" pipeline looks like:

  1. Source — branch protection on default branch, signed commits (git config commit.gpgsign or SSH signing), required reviews from CODEOWNERS, no force-push.
  2. CI on a hosted runner — GitHub Actions, GitLab, Buildkite-managed, with an OIDC trust to your cloud and to Sigstore. No long-lived secrets in the runner.
  3. Hermetic build — Bazel, Nix, or a containerized build with sealed dependencies; offline once started.
  4. SLSA L3 provenance via slsa-github-generator or equivalent, signed by Sigstore.
  5. SBOM generated by syft, signed and attached as an OCI artifact.
  6. Vulnerability scan (Trivy/Grype) with VEX overlays for known-non-exploitable.
  7. Push to a private registry, by digest only.
  8. Admission policy in target clusters: signed by the right OIDC subject, SLSA provenance present, no critical-without-VEX, image age within policy.
  9. Artifact retention long enough for audit (3-7 years for some regulated industries).
Quick check
An attacker steals your CI runner's GitHub Actions OIDC token by exploiting a vulnerability in a third-party action you used. They try to publish a malicious image signed with that token. Why does Kyverno's verifyImages still reject the image at the cluster?
Show answer
Because the OIDC subject won't match. The Sigstore signing certificate from Fulcio embeds the OIDC subject — for GitHub Actions that includes repo:my-org/payments-service:ref:refs/heads/main and the workflow path .github/workflows/release.yml. Even if the attacker has a valid token, the workflow they ran is theirs, not yours; the resulting subject would be something like repo:attacker-org/evil:ref:refs/heads/main. Kyverno's subject regex on the verifyImages policy rejects it. This is the core promise of keyless signing — the identity is in the cert, and the cert binds to the actual workflow run.

The Consumer-Side Picture

Most engineers are also consumers. Three rules cover most of it.

  • Read SBOMs you receive. If a vendor ships an SBOM, ingest it into your asset inventory. When the next Log4Shell hits, your search is across all vendors.
  • Verify signatures by default. cosign verify in your image-pull script. chainguard images ship signed by default; expand outward.
  • Pin third-party Actions and CI plugins. Use full SHA, not a tag. The 2022 ESLint, 2023 cdxgen, and 2024 tj-actions/changed-files compromises all hit teams using floating tags.
Mnemonic — supply-chain hygiene
"Sign, attest, scan, pin, mirror."
  • Sign every release with Sigstore (keyless).
  • Attest provenance (SLSA), SBOM, scans.
  • Scan + VEX to keep noise low.
  • Pin by hash; never by floating tag.
  • Mirror upstream registries; vendor when stable.
Flashcard
A regulator asks: "prove the binary running in production was built from the source commit you say it was, by your real build pipeline." What artifacts can you show?
Click to flip ↻
Answer
The chain: 1) the artifact's digest from the registry; 2) the Sigstore signature (cosign verify) showing it was signed by your release workflow's OIDC subject; 3) the SLSA provenance attestation listing source repo, commit SHA, builder identity, build inputs; 4) the Rekor transparency-log entry containing the same; 5) the source commit's own signature; 6) the cluster admission audit log proving only that signed/attested image was admitted. None of those alone is sufficient; together, the chain is verifiable end-to-end without trusting any single component. This is the entire point of the modern supply-chain stack.
🔑
Key takeaways
1) SLSA levels build integrity from "none" to "hardened isolated builder with non-falsifiable provenance." 2) SBOMs turn the next Log4Shell from "days of grep" into "seconds of query" — generate and ship them. 3) Sigstore killed the "PKI is too hard" excuse — keyless signing tied to OIDC identity is now table stakes. 4) in-toto attestations bundle provenance, SBOMs, scans, and reviews into a verifiable chain. 5) Verify at admission with Kyverno + cosign — the attack surface flips from "steal a token" to "break the ecosystem." 6) Pin, mirror, scope, scan on the consumer side; supply-chain incidents are still mostly preventable with old hygiene.

Finished reading?