ProvenancePipeline halfway through: the pipeline is green, the project is not done

The release workflow is green.

The image is built, scanned, signed, recorded, and independently verified.

ProvenancePipeline is not finished.

Kubernetes does not care yet.

If I apply a manifest using an unsigned image, KateClusters will still attempt to run it. The CI work has produced evidence, but the runtime boundary has no rule requiring that evidence.

This is the exact point where supply-chain projects tend to stop because the remaining step is less cooperative than the pipeline screenshot.

The container build is intentionally boring

The Go application compiles to one small binary.

The build stage contains the compiler and build tooling. The final stage is distroless and runs with a non-root identity. The base references are pinned to digests.

I also checked the final image rather than relying on the Dockerfile:

1. The compiler is not present.

2. The process does not run as root.

3. The expected binary starts and answers its health endpoint.

4. Rebuilding from the same source and base inputs produces the expected application behavior.

I am not claiming full bit-for-bit reproducibility from this workflow. Build timestamps, metadata, and surrounding tooling still require care. The important artifact identity for the current project is the digest produced and signed by the release.

Two SBOM formats showed me the same image differently

Syft produces both SPDX and CycloneDX output.

They describe the same artifact through different schemas and conventions. Keeping both forced me to verify that the pipeline was attaching the intended files rather than treating "SBOM generated" as one opaque step.

The Go standard-library choice kept the application dependency list small. The runtime base still contributes its own components and metadata. A small application does not create an empty software supply chain.

The SBOMs are saved as workflow artifacts and attached as attestations to the image digest.

That last phrase matters. An SBOM sitting beside a release is useful documentation. An attested SBOM tied to the digest gives the verifier a stronger reason to associate the document with that exact image.

The vulnerability gate failed for the right reason

Grype is configured to fail the release on a CRITICAL vulnerability.

During setup, I had to distinguish three different failures:

1. The scanner found a vulnerability above the threshold.

2. The scanner could not obtain or read its vulnerability data.

3. The workflow passed the wrong artifact or SBOM to the command.

Those cases should not be collapsed into one "scan failed" message during investigation.

I kept continue-on-error out of the gate. I also avoided adding an allowlist just to stabilize the first successful release.

That does not mean allowlists are always wrong. A time-limited, justified exception can be part of a real vulnerability process. This project does not have that process yet, so an allowlist would mostly be a way to hide an inconvenient result.

The current rule is simpler: a CRITICAL finding stops publication.

The first keyless signature made the identity concrete

Before the real release, the expected certificate identity was a line in the design.

After signing, I could inspect what Cosign and the certificate actually recorded for the GitHub workflow. I used that observed value to tighten the verification.

The release does not keep a private key. GitHub's OIDC identity participates in obtaining a short-lived certificate for the signing operation. The signature is associated with the image digest, and the transparency record provides a public, append-only record of the event.

Keyless signing removes key storage from my repository. It does not remove the need to protect the workflow.

If an attacker can modify the trusted release workflow on the trusted branch, the expected identity may sign an attacker-controlled artifact. Branch protection, review, workflow permissions, and the builder model remain part of the security argument.

This is one reason I am keeping the SLSA claim at Level 2.

The provenance is useful because of what it does not claim

The provenance records the source repository, commit, workflow, builder context, and resulting digest.

It answers where the artifact came from under the workflow's recorded process.

It does not inspect the source for a backdoor. It does not make the GitHub runner an isolated trusted builder. It does not prove that a maintainer approved every line.

Those limitations make the provenance more precise, not less valuable.

I can now take the image digest and verify that its provenance refers to the intended repository and release path. That is a much stronger statement than "the tag looks correct."

It is still a statement about origin.

Independent verification caught an overly broad assumption

The clean verification job pulls the published image and checks the signature, issuer, identity, SBOM attestation, provenance attestation, and digest.

The first version of the identity match was broader than I wanted. It would have made maintenance easier because several related workflow identities could qualify. It also weakened the answer to "which workflow is trusted?"

The negative control exposed that discomfort.

I now run one verification with the exact expected identity and one with an intentionally incorrect identity. The job succeeds only when the first passes and the second fails.

This test is simple enough to look redundant. It protects against a command that is syntactically correct while checking a different boundary than the one described in the README.

Why the registry tag is still present

The workflow publishes human-readable tags because people and release tooling use them.

The security decisions use the digest.

A tag is a pointer that can move. A digest identifies the content. The signature and attestations are associated with the digest, and the verification output records it.

The Kubernetes manifest can begin with a tagged reference, but the admission policy must verify the resolved image and prevent the tag from becoming a way around the signature requirement.

I am testing that resolution behavior in Kyverno rather than assuming the registry and admission controller agree about the final reference.

What is missing

Kyverno is not installed in the TrustStack cluster yet.

The draft policy has the expected image scope, issuer, and identity, but it has not denied anything. There are no allow and deny captures. There is no proof that registry access works from the admission path. There is no proof of fail-closed behavior.

That leaves several practical questions:

1. Can the admission controller retrieve the verification material from the registry?

2. Does the policy match only the intended image repository?

3. Does the signed image pass without a broad exception?

4. Does an unsigned image fail before pod creation?

5. What happens when verification is unavailable rather than invalid?

6. Does the audit record clearly explain the denial?

I do not want to discover the answer to number 5 during the final test.

One week left

The next work is entirely in the boundary between ProvenancePipeline and KateClusters.

I will install Kyverno, confirm its own health, apply the policy in audit mode long enough to inspect matching behavior, and then move it to enforcement. The final policy will use a closed failure mode.

After that I will run the two deployments.

If the signed image starts and the unsigned image does not, the project will be complete.

If both start, the green release workflow will remain what it is today: a successful producer of evidence nobody is required to read.