TrustStack finished with a failed deployment

TrustStack finished when Kubernetes printed an error.

The image existed in the registry. The manifest was valid. The cluster was healthy. The deployment still failed because the image did not have the identity and evidence required by the admission policy.

That denial was the last planned test.

It was also a better ending than another green pipeline.

The last piece

ProvenancePipeline uses a small Go service built with the standard library. Keeping the application boring was useful. The project is about the path from source to running artifact, not about hiding supply-chain results inside a large dependency tree.

The Docker build uses separate build and runtime stages. The runtime image is distroless, the process is non-root, and the base images are pinned by digest.

The release workflow does more than publish a tag:

1. It runs the application tests.

2. It builds the image.

3. It generates SPDX and CycloneDX SBOMs with Syft.

4. It scans for vulnerabilities with Grype.

5. It stops the release on a CRITICAL finding.

6. It signs the image digest with Cosign using GitHub's OIDC identity.

7. It records the signature through the Sigstore transparency system.

8. It attaches SBOM and SLSA provenance attestations.

9. A separate job on a clean runner verifies the signature, issuer, identity, attestations, and digest.

10. A deliberately incorrect verification must fail.

There is no private signing key stored in GitHub.

The trust anchor is the workflow identity and its issuer. The policy expects the release workflow in this repository on the main branch, issued through GitHub Actions. Checking only the issuer would trust any GitHub workflow. Checking only a similar-looking identity without the expected issuer would leave another gap.

Both are required.

What the provenance proves

The SBOM answers what is inside the image.

The vulnerability scan compares those components with known findings under the project's release policy.

The signature ties the image digest to the signing identity.

The provenance records the source repository, workflow, commit, builder context, and produced digest.

It does not prove the code is good. It does not prove the workflow cannot be compromised. It does not turn a CRITICAL threshold into a complete risk program.

This pipeline reaches SLSA Build Level 2 for the implemented design. I am not calling it Level 3 because the job producing the image also assembles the provenance. A stronger level would require an appropriately isolated build service rather than a nicer badge.

That limitation belongs beside the result.

The verification job had to disagree once

A verification command that always succeeds is worse than no verification because it looks like evidence.

The clean verification job includes a negative control with the wrong expected identity. The release passes only when the valid verification succeeds and the invalid verification fails.

This caught mistakes while I was wiring the identity flags. A command can exit successfully while verifying a broader identity than the policy designer had in mind. The negative control made the intended boundary observable.

That pattern now appears throughout TrustStack:

1. An allowed case shows that the system can function.

2. A denied case shows that the boundary exists.

3. An ambiguous case shows whether the system guesses or asks for help.

The third one is easy to omit. It is where many security automations become dangerous.

Kyverno closed the loop

The CI evidence was complete before the cluster enforcement was complete.

That distinction bothered me because a valid signature in a registry changes nothing by itself. Kubernetes will still run an unsigned image if admission does not care.

KateClusters now runs a Kyverno verifyImages policy in enforcement mode. Its failure policy is closed. If verification cannot be completed, the request is rejected rather than admitted on trust.

I tested two deployments.

The signed image built by the expected release workflow was admitted. Kyverno verified the image against the required identity and issuer, and the pod ran under the security controls already present in KateClusters.

The unsigned image was rejected at admission. No pod started. Falco had nothing to detect because the untrusted workload never reached runtime.

This is why the final TrustStack evidence is an error. The error is the control doing its job at the earliest useful point.

The four projects now form one system

AwLZ establishes the AWS organization, separates duties, protects Terraform state, applies organization guardrails, centralizes immutable logs, and delegates the security services.

PontoAntiCrack uses that environment to detect risky AWS activity. It plans before changing, preserves the original state, runs through a circuit breaker, supports dry-run, and escalates cases it cannot remediate safely.

KateClusters owns the Kubernetes control plane and runtime. It hardens the API path, encrypts Secrets at rest, restricts pod privileges and workload identities, denies network traffic by default, records audit activity, and uses Falco and Grafana to detect attacks and missing detection signals.

ProvenancePipeline gives the cluster a reason to trust an image before the process starts.

The connections matter more than the project count.

CloudTrail events from the governed AWS organization feed a response system with an audit trail. A signed workload passes admission into a hardened cluster. An unsigned workload stops at the boundary. A permitted workload remains constrained and observable after it starts.

There is prevention, detection, response, and evidence at different points. None of them is pretending to be the whole answer.

The failures I kept

The clean version of this project would contain four architecture diagrams and four final demos.

The real version also contains:

1. SCPs that blocked the Terraform deployment they were supposed to protect.

2. A Calico policy that removed DNS from a namespace.

3. Falco rules that were technically active and practically too noisy.

4. Pod manifests that failed restricted admission because I had relied on defaults.

5. Documentation-derived AWS fixtures that needed comparison with live events.

6. An S3 policy classifier that initially treated conditions too generously.

7. A verification command that needed a negative identity test to prove its boundary.

These are not bloopers added for personality. They show where my mental model was incomplete.

Keeping them also made the final evidence more believable to me. Every control in the last demo has a history of being wrong in a specific way.

Working with minimal AI

I kept the constraint I set at the start.

My first path was documentation, logs, source, local manuals, tests, and experiments. I wrote the initial implementations myself and used automated validators heavily. Terraform validation, policy checks, linters, unit tests, security scanners, Kubernetes dry runs, benchmark tools, and API responses gave me feedback without making the design decisions for me.

When I used assistance for a narrow editorial or organizational task, it did not become implementation evidence. I still had to explain the control, reproduce the result, and decide whether the observed behavior matched the claim.

This took longer.

It also means I can trace the projects through the problems I encountered instead of remembering only the final files.

That was the point.

What TrustStack does not prove

It is still a personal lab.

The AWS organization is small. The Kubernetes cluster is single-node. The attack volume is controlled. There is no on-call rotation, change advisory process, hostile multi-tenant production environment, or year of billing data.

The availability story is limited. Disaster recovery is not the central scope. The SLSA level is deliberately capped by the builder design. Detection tuning has not seen production-scale noise.

Those are real gaps. Another dashboard would not remove them.

What TrustStack proves is narrower and useful:

1. I can design security boundaries across accounts, identities, workloads, and artifacts.

2. I can implement those boundaries with infrastructure and policy as code.

3. I can test allowed, denied, and ambiguous behavior.

4. I can preserve evidence before an automated response.

5. I can document limitations without converting them into marketing language.

6. I can follow a failure far enough to change the design.

Finished does not mean sealed

The four repositories have follow-up lists.

AwLZ can gain more workload permission boundaries and longer cost data. KateClusters can move to a multi-node environment and test backup recovery. PontoAntiCrack can add detections after the framework has more operational time. ProvenancePipeline can adopt a stronger isolated builder.

I am not turning those possibilities into a fifth unfinished mega-project.

TrustStack now has a complete path someone else can inspect: build the foundation, attack the controls, preserve the evidence, verify the artifact, and watch the cluster reject what it cannot trust.

The last command failed.

Good.