AwLZ halfway through: when the guardrails guard against me

I expected IAM and KMS to consume most of the time in AwLZ.

They have consumed plenty. The more useful problem, though, came from the Service Control Policies.

The guardrails worked. They also blocked the infrastructure that was trying to finish configuring the guarded services.

That sentence is funny after the fact. During the apply, less so.

Where the build stands

The Terraform bootstrap is in place. State is stored remotely in S3, encrypted with KMS, versioned, and protected from non-TLS access. Locking is enabled, and the later stacks use the remote backend instead of carrying local state between machines.

The organization structure also exists. The Security and Workloads OUs are present, and the member accounts have their assigned roles.

At this point the work has moved from creating boxes in an organization chart to making services operate across those boxes:

1. CloudTrail must write events from the organization into Log Archive.

2. The log bucket and KMS key must trust the required service paths without trusting every principal in every account.

3. Security Hub, GuardDuty, Config, and Access Analyzer need administration and enrollment in the correct accounts.

4. Terraform must be able to make intended changes while member accounts remain unable to tear the controls down.

The fourth item exposed the SCP problem.

A deny statement can be correct and still be wrong

My first version focused on the result I wanted: member accounts should not be able to disable security controls.

The difficulty is that an AWS API action name does not always map cleanly to the sentence I wrote in the policy description. Some actions used during configuration also appear in destructive workflows. If I deny the entire action without understanding its parameters and call path, I can prevent both the attack and the maintenance operation.

That is what happened.

The apply reached a service configuration step and received an explicit deny from the organization policy. The principal had IAM permission. The SCP set the ceiling lower, so the call stopped there.

My first instinct was to add the deployment role to a broad exception and move on. That would have made the apply green, but it would also have created an administrative tunnel through the guardrail. Anyone able to assume that role would have inherited the exception.

Instead I went back through the failing calls and divided them into three groups:

1. Actions that are always destructive for this project.

2. Actions needed during setup or an intended update.

3. Actions that require more context before they can be classified safely.

The revised policies are narrower. The deployment path is explicit. I am also treating every exception as something that needs its own test, not as a comment explaining why the risk is probably acceptable.

The result is less elegant than one large deny block. It is also more honest.

The log archive is becoming real

The organization trail is now sending data into the dedicated account.

This was the moment the account split stopped feeling like a diagram. An action in one account appeared as an object in another account, encrypted under a key controlled there. The source account could produce the event but did not own the archive.

Object Lock required an early decision because it must be considered when the bucket is created. The archive uses COMPLIANCE mode and a 30-day retention period.

I tested the behavior with a retained object rather than assuming the setting was enough. The deletion attempt failed. That evidence is much more useful than a screenshot of the bucket properties.

There is also a CloudWatch copy with a shorter retention period for quick searches. S3 is the durable evidence store. CloudWatch is the convenient operational view. Treating one as a substitute for the other would either make investigation painful or make retention unnecessarily expensive.

KMS policy work is mostly boundary work

The KMS key policy took several passes.

It needs to allow CloudTrail to use the key for the intended trail and account structure. It should allow the log archive administrators to manage the key. It should not turn into an account-wide permission slip because a service principal appears in the statement.

The useful habit here has been reading every principal, action, resource, and condition aloud as a sentence:

"This principal can perform these cryptographic actions on this key when these context values are true."

If the sentence ends with "from anywhere in the organization" and I cannot explain why, the condition is not finished.

This is slower than adapting a policy example and waiting for AWS to accept it. It is also the part of the project I came here to learn.

Delegated services do not all behave the same way

I wanted a single Terraform pattern for the detection services. AWS had other ideas.

Each service has its own sequence for trusted access, delegated administration, organization configuration, and member enrollment. Some state becomes visible immediately. Other state takes time to settle. A successful API response does not always mean the final organization view is already consistent.

I am keeping the service modules separate enough to represent those differences. Forcing them behind an identical abstraction would make the root configuration shorter while moving the real behavior somewhere harder to see.

GuardDuty, Security Hub, Config, and Access Analyzer are now wired into the security account at different stages of completion. I am not recording a final organization-wide compliance result yet. The services need to stabilize, and I still need to verify member enrollment from both the administrator and member views.

What I have changed about the evidence

I originally planned an evidence folder with one successful output per control.

That was too clean.

The folder now includes failed deployment outputs, denied calls, and notes about why a policy changed. A guardrail blocking Terraform is evidence that the guardrail has authority. The follow-up evidence has to show that I corrected the policy without removing that authority.

For each control, I am trying to preserve three things:

1. The intended behavior.

2. The action or query used to test it.

3. The observed result, including the account and role from which it was observed.

The account and role matter. "Access denied" means very little if I do not record who was denied.

What is left

The remaining work is concentrated in detection service verification, CI role restrictions, and the final guardrail test matrix.

I also need to let the Security Hub data settle before treating any score as meaningful. GuardDuty member status needs a second check. Cost needs time as well. A partial billing period is not a reliable monthly estimate.

AwLZ is past the stage where everything can be destroyed and recreated without thought. There are retained logs, account relationships, and organization policies now. Changes require more care, which is exactly the condition the project was meant to create.

The foundation is doing its job, including the inconvenient parts.