Starting AwLZ: building the AWS foundation before the security demos

The first repository in TrustStack is AwLZ, short for AWS Landing Zone.

I could have started with a Lambda that reacts to an attack. It would have been more exciting on day one. It also would have hidden a basic question: what AWS environment is that Lambda protecting?

AwLZ comes first because security services need somewhere sensible to live. Logs should not sit beside the workload they describe. Detection findings should not be administered by the account being monitored. A compromised development account should not have the final say over the evidence used to investigate it.

So this week is mostly account structure, state management, policies, and a surprising amount of reading about which AWS service must be enabled from which account.

The shape of the environment

The organization will have a management account and two Organizational Units.

The Security OU will contain a Log Archive account and a Security account. The Workloads OU will contain Dev and Lab accounts.

The split gives each account a narrow job:

1. The management account owns AWS Organizations and the administrative entry point.

2. Log Archive receives organization-wide audit logs and protects them from early deletion.

3. Security administers GuardDuty, Security Hub, AWS Config, and IAM Access Analyzer.

4. Dev is for normal development workloads.

5. Lab is where I can run controlled security tests without mixing them with the organization controls.

This is not a production-sized organization, but it is large enough to make cross-account boundaries real. Terraform has to assume roles. Bucket policies must recognize organization services. KMS policies must allow the right writers and readers. A mistake can occur in one account and become visible in another.

That is what I need for the rest of TrustStack.

Starting with the unglamorous part

Before creating the organization, I need a safe place for Terraform state.

The bootstrap stack will create an S3 bucket with versioning and a KMS key. Access without TLS will be denied. State locking will prevent two applies from racing each other. The state is infrastructure metadata, but treating it as harmless would be a mistake. It can reveal account IDs, resource relationships, policy structure, and sometimes values that should never have been placed there.

I am keeping bootstrap separate from the later stacks. The resources that hold Terraform state should not depend on the state they are supposed to hold.

There is an awkward transition in every remote-state setup: the first apply begins locally, then the backend is migrated. I am documenting that step instead of burying it in setup notes because it is part of the trust chain for everything after it.

Guardrails are not IAM permissions

The part I most want to understand is Service Control Policies.

An SCP does not grant a user permission. It sets the maximum permission available inside an account. A role can have AdministratorAccess and still be unable to perform an action denied by the organization.

That distinction sounds simple when written in one sentence. It becomes less simple when a service performs several API calls behind one configuration change, or when an API action is used both to update a control and to disable part of it.

My initial guardrail goals are:

1. Restrict AWS regions where the lab does not need to operate.

2. Prevent member accounts from disabling or deleting central audit controls.

3. Protect security services and critical log resources from destructive changes.

4. Keep enough administrative access for the infrastructure pipeline to maintain the system.

Number 4 is where I expect trouble. A policy that never blocks anything is decoration. A policy that blocks every legitimate update is also a failed design.

I am writing the expected denied and allowed actions beside each policy before applying it. That gives me a test target and makes broad deny statements harder to justify with vague language.

Logging has to survive the account

CloudTrail will be organization-wide and deliver into Log Archive.

The S3 bucket will use Object Lock in COMPLIANCE mode with a 30-day retention period. Once an object is protected, an administrator should not be able to delete it before that period expires. I will also keep a shorter CloudWatch copy for queries during the lab.

This is one of the controls where the negative test matters more than the configuration screenshot.

Showing that Object Lock is enabled proves a setting exists. Attempting to remove a protected test object and receiving a denial proves what that setting means.

I also need to be careful with cost. Organization trails, CloudWatch ingestion, Config recorders, security findings, KMS requests, and retained objects are individually understandable. Together they can create a bill that is easy to ignore while focusing on the technical work. Cost notes will be part of the project, not a surprise at the end.

CI without permanent AWS keys

GitHub Actions will authenticate to AWS through OIDC.

I will use one role for plans and another for applies. The plan role should be read-only. The apply role will have the permissions required by the infrastructure stacks and will be protected by the production environment in GitHub.

There will be no long-lived AWS access key stored as a repository secret.

OIDC does not make the pipeline automatically safe. The trust policy still needs to restrict the repository, branch, and workflow context. A temporary credential issued to an overly broad identity is still overly broad. The difference is that I can define the identity at the boundary and avoid a permanent secret that survives long after a workflow finishes.

My definition of done for AwLZ

I am not using terraform apply as the finish line.

AwLZ is done when:

1. The accounts and OUs exist in the intended structure.

2. The remote state is encrypted, versioned, locked, and accessible only through the expected path.

3. Organization-wide CloudTrail events arrive in Log Archive.

4. A retained test log cannot be deleted early.

5. The security account administers the delegated services.

6. Guardrails block the actions they were designed to block and allow the maintenance actions they must allow.

7. GitHub Actions can plan without permanent credentials, while apply remains separately controlled.

8. The repository contains evidence for those statements, including failures.

My first task is the bootstrap. No attack simulation yet. No dashboard. Just a bucket, a key, state locking, and the patience to get the base right before I build on top of it.

It feels like the correct place to start.