KateClusters in the middle: default-deny means DNS too

DNS stopped working.

That was the first clear sign that the NetworkPolicy was being enforced.

It was not the most elegant test, but it removed any doubt that Calico was merely accepting YAML and doing nothing with it. I had applied default-deny egress to a namespace without adding the DNS allowance first. Pods stayed healthy enough to be confusing, names stopped resolving, and the logs pointed me back to the policy.

The fix was small. The lesson was not.

The baseline now exists

KateClusters is no longer a folder of validated manifests.

The cluster was initialized with kubeadm, Calico is running, CoreDNS is healthy again, and the node schedules both control-plane components and lab workloads. Before hardening, I ran the first kube-bench assessment and preserved the raw output.

I am not publishing a comparison score yet. The hardening pass is still changing, and a mid-project number would be easy to quote without useful context.

The baseline did what I needed it to do. It identified which checks applied to this setup, which failures were real configuration gaps, and which recommendations needed interpretation for a single-node lab.

Not every benchmark result should be fixed blindly. A benchmark is a structured review, not a substitute for understanding the environment.

Audit policy before dashboards

The API server is now writing audit events according to a custom policy.

I started with the event types I know I will need for the attack scenarios: authentication failures, authorization denials, changes to RBAC, namespace operations, workload creation, and access to sensitive resource types.

Secrets remain at Metadata. The logs record that an operation happened, who requested it, and which object was involved, but they do not copy the Secret value into the log pipeline.

I tested this with a temporary Secret and inspected both the API behavior and the resulting event. The object reference was present. The sensitive payload was not.

That test was worth doing before Loki. Once logs are shipped and indexed, removing an accidentally recorded secret becomes a much bigger job than correcting the policy.

Encryption at rest was more than adding a file

The encryption provider configuration is active for Kubernetes Secrets.

To verify it, I created a test Secret, then inspected the stored representation through the controlled local procedure for the lab. The data was not present as the ordinary base64 value Kubernetes would otherwise store.

I also tested that the API still returned the correct Secret to an authorized client. Encryption at rest that makes the cluster unable to read its own objects would be secure only in a very literal and useless sense.

The key material is kept off the repository. The configuration references the expected local path, and the repository provides setup instructions rather than a real key.

This is one of the places where a public lab can accidentally teach the wrong lesson. A redacted screenshot is not a secret-management strategy.

RBAC is being tested from inside the pod

The Service Account work is mostly in place.

Workloads do not receive a token by default. The scenario that requires Kubernetes API access gets a projected token with a limited lifetime and a narrow Role.

Testing the Role from my administrator shell would prove very little. The important view is from the workload identity.

From inside the test pod, an allowed read succeeds. An unrelated write receives a 403. The audit event names the Service Account, not my local administrator. That ties the result to the identity the control is meant to restrict.

The sa-token-abuse scenario is not finished yet, but its boundary is visible now. A token alone should not equal control of the cluster.

Pod Security Admission found assumptions in my own manifests

Applying restricted labels to the lab namespaces caused several manifests to fail admission.

Some fixes were expected: explicit non-root execution, seccomp, dropped capabilities, no privilege escalation. Others came from defaults I had allowed Kubernetes or a chart to choose for me.

I am resisting the urge to move difficult components into a permanent privileged namespace. System components sometimes need exceptions, but an exception should describe the actual requirement. "This chart would not install" is not a security requirement.

For attack scenarios that intentionally need a weaker policy, the namespace will exist only for the duration of the test. Its creation and deletion will be visible in the audit log.

This makes the attack slightly more complicated. It also prevents the testing infrastructure from becoming an undocumented back door.

Falco is running, and noise is already a problem

Falco is observing the node through the modern eBPF driver.

The first rule set produced too many events to be useful. Normal administrative activity looked suspicious because, in isolation, it is suspicious. A shell in a container can be an attack, a health check, or me debugging the rule that says shells are attacks.

I am adding context rather than simply lowering severity.

The custom rules care about:

1. Shells in workloads that are not expected to have a shell.

2. Reads from projected Service Account token paths.

3. Attempts involving host namespaces or sensitive host mounts.

4. Miner-like process names and selected network behavior.

5. Changes that could interfere with Falco or its reporting path.

The rules also need exceptions for known system behavior. Each exception is scoped to a workload, namespace, executable, or other observable context. A global exception would make the dashboard quiet and the detector blind.

I applied NetworkPolicy early, then backed up

My original plan placed NetworkPolicy near the end.

I applied part of it early because I wanted confirmation that Calico enforcement was real. The DNS failure provided that confirmation, and I removed the broader policies until the observability flows are completely mapped.

That is not a rollback of the design. It is sequencing.

Loki, Grafana, Falco, DNS, and the applications all have connections that need to be understood before default-deny is enabled everywhere. I am recording those flows now. When the final policies arrive, every allow rule should have a reason.

Current problems

The cluster works, but three parts still need attention.

The Falco rules need another round of tuning against normal workload activity. The Grafana heartbeat alert needs to distinguish a stopped Falco process from a broken log path. The attack namespaces and cleanup scripts need to be repeatable so one test does not contaminate the next.

I also need to rerun the CIS checks after the control-plane changes settle. That final run will be compared with the untouched baseline, not with a convenient intermediate state.

For now, the cluster can deny a pod, restrict a Service Account, encrypt a Secret, and break its own DNS on command.

Progress.