Multi-tenant Terraform without nightmares: state isolation, lock tables, and disposable runners

Running Terraform on your own infrastructure is routine. Running it on behalf of hundreds of customers, against theirAWS accounts, unattended, is a different sport entirely. The failure modes stop being “oops, re-run the pipeline” and start being “we corrupted a customer’s state file” or “tenant A’s apply just read tenant B’s credentials.”

We’ve been building a platform that does exactly this — it reads a repo, infers the infrastructure, and provisions it in the customer’s own AWS account via Terraform. Here’s the isolation model we landed on, layer by layer, and what still makes us nervous.

The threat model #

Three things keep you up at night when you operate Terraform as a multi-tenant service:

1. State corruption. Two applies racing on the same state file, or a crashed run leaving state half-written.
2. Cross-tenant leakage.Customer A’s credentials, state, or plan output becoming visible to a run for customer B.
3. Runaway applies.A run that does more than it should — wrong account, wrong scope, or a workspace polluted by a previous run.

Each layer below targets at least one of these. None of them is sufficient alone.

Layer 1: state keys that cannot collide #

Every (project, environment) pair gets its own state object in S3:

s3://<state-bucket>/
└── vylara/
    ├── projects/<project-uuid>/production/terraform.tfstate
    ├── projects/<project-uuid>/staging/terraform.tfstate
    └── networks/<managed-network-uuid>/terraform.tfstate

The key is built from the project UUID and the environment name, nothing else — the backend config is generated per-deployment, never hand-written:

# backend.hcl — generated per run
bucket         = "<state-bucket>"
key            = "vylara/projects/<project_id>/<env>/terraform.tfstate"
dynamodb_table = "vylara-tf-locks"
encrypt        = true

UUIDs make collisions structurally impossible, and environments under the same project never clobber each other because production and staging are distinct path segments. Managed networks (shared VPC roots) get their own namespace under vylara/networks/ so a network apply and a workload apply can run concurrently.

One deliberate choice: state keys are not namespaced by user ID. Tenancy is enforced in the API and database (org membership checks), not in the key layout. Encoding ownership into the path felt safer at first, but ownership changes— projects move between orgs — and a state key that has to migrate when ownership changes is a much worse failure mode than a clean lookup-table indirection.

Layer 2: DynamoDB locking, with patience #

Every backend config points at a DynamoDB lock table. This prevents concurrent applies on the same state object while letting different (project, env) pairs run fully in parallel.

The non-obvious part: Terraform’s default behavior on a held lock is to fail instantly. In a queue-driven system, a retry landing while the previous attempt is still finishing is normal, not exceptional — so we pass -lock-timeout on every plan/apply/destroy and let the second run wait its turn instead of erroring into the retry path and confusing everyone.

Layer 3: one disposable runner per run #

State isolation handles data; you also need execution isolation. Each Terraform run executes as its own Kubernetes Job — no shared long-lived worker, no shared workspace directory:

spec:
  ttlSecondsAfterFinished: 3600    # gone in an hour
  template:
    spec:
      serviceAccountName: terraform-runner-sa   # IRSA
      containers:
      - name: terraform
        securityContext:
          readOnlyRootFilesystem: true
          allowPrivilegeEscalation: false
        resources:
          limits: { cpu: "1", memory: 1Gi, ephemeral-storage: 2Gi }
      volumes:
      - name: workspace
        emptyDir: { sizeLimit: 1Gi }
  backoffLimit: 0

The properties that matter:

• Ephemeral workspace. emptyDir with a size limit. When the Job dies, the rendered main.tf, the plan file, the .terraform/provider cache — all of it dies with it. Nothing from run N exists for run N+1 to read.
• Auto-cleanup. ttlSecondsAfterFinished: 3600 means even the Job object and its logs are garbage-collected after an hour. We keep sanitized error events in our own database; the raw pod is not an archive.
• No retries at the Job level. backoffLimit: 0. A failed apply must not blindly re-run — partial applies need a fresh plan, not a replay.
• Egress lockdown. A NetworkPolicy restricts the runner to HTTPS toward public AWS APIs (explicitly excluding RFC1918 ranges) plus the NATS port for status updates. A compromised provider plugin in the runner cannot crawl our cluster network.

Layer 4: credentials that can't cross-contaminate #

This is where we got burned, and where the design got opinionated. The rule: the customer’s identity never enters the Terraform process environment.

The runner process holds only the platformidentity (via IRSA), which it uses for exactly two things: the state bucket and the lock table. Access into the customer’s account flows exclusively through the AWS provider’s assume_roleblock — an STS role hop, scoped per run, with an external ID, into a role the customer created via a CloudFormation onboarding stack. We never accept or store customer access keys.

And because env vars are sneaky, the runner actively strips every AWS identity variable (AWS_ACCESS_KEY_ID, AWS_SESSION_TOKEN, AWS_PROFILE, …) from both the inherited environment and any caller-supplied credential map before launching Terraform. The environment is built from an allowlist, not a blocklist. A leaked customer key sitting in some shell variable cannot silently become the identity that writes state.

Layer 5: the bucket-policy backstop #

Defense in depth means assuming layer 4 fails anyway. The state bucket policy allows S3 actions from the platform account and denies every principal whose aws:PrincipalAccount is not ours. If a customer IAM identity ever does end up driving the S3 backend — the exact failure that motivated this policy — the explicit deny fires before a single state byte is written.

One lesson from iterating on this policy: we originally also denied any PutObjectthat didn’t specify SSE-KMS. That produced baffling explicit deny in a resource-based policyerrors whenever the runner used the bucket’s default SSE-S3 encryption. We dropped it. Default SSE-S3 is sufficient for our threat model; KMS remains available as an opt-in override for customers who provision a key on purpose. The PrincipalAccountdeny is the guard that actually maps to a real failure mode — the KMS deny was mostly mapping to our anxiety.

Human gates, where they actually help #

Approval lives where humans add signal: the customer approves the proposed architecture and its cost estimate, and reviews the generated config as a PR, beforeanything runs. Once approved, plan and apply execute unattended — a human staring at a wall of plan output at 2 a.m. is not a safety mechanism, it’s a rubber stamp with latency.

What still keeps us up at night #

Honesty section. Partial appliesremain the worst case: Terraform crashes mid-apply, state says X, the cloud says Y. Locking prevents the race, but recovery is still “plan again and read the diff very carefully” — we’ve built tooling to surgically patch state via state pull/state push for known cases (deletion-protection flags, self-attached EIPs), and each one was earned through an incident. Provider bugsare the other: we’ve hit AWS provider ordering issues where deletion protection must be flipped off in a separate targeted apply before destroy works at all. No isolation layer saves you from a provider doing the wrong thing with correctly-scoped credentials.

Multi-tenant Terraform isn’t one clever trick. It’s five boring layers, each of which has independently caught something.

Built while working on Vylara — connect your repo, and it figures out and provisions the infrastructure in your own AWS account.