Starting KateClusters from a clean Debian VM

The most valuable file in KateClusters right now is a virtual machine snapshot.

It was taken before kubeadm touched the host.

The VM runs Debian 13 with 4 vCPUs, 8 GB of RAM, and a 60 GB disk. Swap is disabled. The kernel has BTF support for the eBPF work I plan to do with Falco. Networking is bridged so the cluster can behave like a real machine on my local network instead of a disposable container network.

There is no cluster yet. That is intentional.

Why I am not using a smaller Kubernetes distribution

For many application projects, kind, k3s, or a managed cluster would be the sensible choice. I am using kubeadm because the control plane is part of this project.

I want to configure and inspect:

1. The API server.

2. The scheduler.

3. The controller manager.

4. The kubelet.

5. Audit policy and audit log paths.

6. Encryption of Kubernetes Secrets at rest.

7. Authentication and authorization settings.

8. File ownership and permissions on the host.

A managed service would teach me a different and useful set of skills, but it would hide several of the controls KateClusters is meant to test.

The tradeoff is clear. This is a single-node lab. Workloads and the control plane will share the same machine, and I will remove the control-plane taint to schedule them. It is not highly available. It does not demonstrate strong tenant isolation. A VM failure takes the cluster with it.

I would rather state those limits than decorate the architecture with claims it cannot support.

Measure before changing anything

The first real milestone is not hardening. It is the baseline.

After the cluster starts and Calico is functioning, I will run kube-bench before modifying the security configuration. That output will be preserved as the before state.

This sounds obvious, but it is easy to get excited and fix the first warning immediately. Once I do that, I can no longer produce an honest baseline. Reverting a few YAML files later would not reproduce the exact original state.

The order for the early build is:

1. Prepare the host.

2. Initialize the cluster.

3. Join the local node and verify control-plane health.

4. Install Calico.

5. Confirm DNS and basic pod networking.

6. Run the baseline benchmark.

7. Take another snapshot.

Only then does hardening begin.

The five layers I want to test

KateClusters is split into layers because no single control can carry the cluster.

The first layer is the control plane. This includes secure API server settings, audit logging, encryption at rest, restricted access to administrative files, and the host configuration around the Kubernetes components.

The second is workload identity. Service Accounts will be specific to their workloads. Automatic token mounting will be disabled by default. When a token is needed, it should be projected, short-lived, and paired with an RBAC role that can do only its job.

The third is pod security. Namespaces will use Pod Security Admission, usually at the restricted level. Workloads should run as non-root, drop unnecessary Linux capabilities, use seccomp, and keep their root filesystem read-only where possible.

The fourth is network policy. Calico will enforce default-deny ingress and egress. DNS, telemetry, and required application flows will be added back explicitly.

The fifth is runtime detection and evidence. Falco will observe behavior through eBPF. Loki will store the relevant logs. Grafana will show signals and alerts. The Kubernetes audit log will record changes at the API boundary.

None of these replaces the others. A denied privileged pod is good. A shell appearing in an allowed container is a different problem.

Four attacks, chosen before the defenses

I defined the attack scenarios before finishing the controls so I would know what each control was expected to do.

escape-attempt tries actions associated with container escape and host access. I want to see what Pod Security Admission blocks at admission and what Falco observes if a behavior reaches runtime.

cryptominer simulates miner-like process and network behavior without installing an actual miner. The interesting result is whether egress policy restricts the connection and whether the runtime rules identify the behavior.

sa-token-abuse tests a workload that tries to use a Service Account token against the Kubernetes API. This should exercise token mounting, RBAC, audit logs, and repeated HTTP 403 responses.

falco-kill attacks the detection path itself. Preventing tampering is one control. Detecting that Falco's heartbeat disappeared is another. Silence cannot be treated as proof that nothing happened.

The attack manifests will not live in a permanently privileged namespace. If a scenario requires an exception, the namespace will be created for the test and removed afterward. Otherwise, the lab would quietly normalize the exact configuration it is supposed to treat as dangerous.

Audit logs can leak the thing they audit

The audit policy needs care around Secrets.

For a Secret, I plan to log metadata rather than the full request and response body. Recording RequestResponse for every Secret would place sensitive content in the audit log. Loki would then copy and index it. I would have built a central, searchable secret leak and called it observability.

This is a useful example of why "more logging" is not automatically safer.

The same applies to Falco rules. A rule that fires on every shell, every package manager, and every network connection may look active during a demo. In normal use it becomes a stream people learn to ignore. I want enough context to understand the signal and enough restraint for the signal to remain visible.

The work I am deliberately postponing

NetworkPolicy comes late.

Default-deny is easy to write and very good at making a new cluster look broken. If I apply it before DNS, logging, metrics, and Falco are understood, I will be debugging several layers at once.

The plan is to bring up observability first, confirm the flows, then deny by default and add only the paths the workloads need. That makes each allowed connection a decision instead of a leftover.

I am also postponing any final CIS score. A planned score is not a result. The repository can contain the benchmark command and the comparison script today, but the evidence stays empty until the cluster produces it.

Tomorrow I initialize the cluster.

If everything goes well, the next post will contain fewer plans and more logs. If it doesn't, that is probably more useful.