Anatomy of an AWS bill shock: the 5 line items that ambush startups

“My $23/mo side project just billed me $2,657.” If you’ve spent any time in cloud forums, you’ve read a hundred versions of that post. The details vary — sometimes it’s a scraped bucket, sometimes a forgotten NAT gateway — but the structure never does: a small, predictable bill, then one invoice that costs more than the laptop it was built on.

I’ve run AWS stacks from ~$180/mo MVPs to ~$680/mo production setups, and I’ve done the post-mortem on enough surprise invoices to know this: bill shock is almost never caused by the resources you think about. It’s caused by five specific line items that are invisible while they accumulate and only become visible on the invoice. Here they are, with real numbers and the specific fix for each.

1. NAT Gateway data processing — the silent classic #

Why it’s invisible:A NAT gateway costs ~$32.85/mo per AZ just to exist, which people budget for. What they don’t budget for is $0.045 per GB processed, in both directions. Nothing in your architecture diagram shows this. Your containers in private subnets pull images from ECR, download packages, talk to S3 — all of it routes through NAT, all of it is metered.

Realistic example:An EKS cluster in private subnets, pulling 1.5 GB of container images per deploy across 6 services, deploying 10 times a day, plus an app that writes 20 GB/day to S3. That’s roughly 1,050 GB/month through NAT ≈ $47/mo in data processingon top of $33–99/mo in hourly charges — for traffic that never needed to leave the building. I’ve seen this single line hit $400/mo on a “small” stack with a chatty data pipeline.

The fix: Add a gateway VPC endpoint for S3 (free) and interface endpoints for ECR (~$7.30/mo each, almost always worth it). Route S3 and image-pull traffic around the NAT entirely. And count your NAT gateways: one per AZ × 3 AZs = ~$99/mo before a single byte. At MVP scale, one NAT in one AZ is a defensible $66/mo savings.

2. Cross-AZ data transfer — paying twice for your own chatter #

Why it’s invisible: Traffic between availability zones costs $0.01/GB in each direction — so $0.02/GB effectively, billed on both sides. It appears in Cost Explorer as the gloriously unhelpful “EC2-Other.” Multi-AZ is sold as pure resilience; nobody mentions it’s also a meter.

Realistic example:A chatty microservice setup — API pods in AZ-a, Redis in AZ-b, a queue consumer in AZ-c — pushing 30 GB/day across zones is ~900 GB/mo ≈ $18/mo. Sounds small until you add a log shipper, a metrics agent, and a service mesh doing health checks, and suddenly it’s $150/mo of internal whispering.

The fix: Keep chatty pairs co-located (topology-aware routing in Kubernetes, AZ affinity for cache clients), compress internal traffic, and check whether that read replica in another AZ is being hammered by a reporting job that could run nightly.

3. The graveyard: idle ALBs, unattached EIPs, orphaned EBS, ancient snapshots #

Why it’s invisible: These are resources that did something once. The load balancer from a killed experiment: ~$16–20/mo. An Elastic IP not attached to a running instance: ~$3.60/mo. That 100 GB gp3 volume from a terminated instance: $8/mo. Four years of automated snapshots nobody pruned: $0.05/GB-mo, quietly compounding. None of them alert. None of them appear in your deploys. They just bill.

Realistic example: Two forgotten ALBs ($40), three unattached EIPs ($11), 400 GB of orphaned volumes ($32), 1.5 TB of stale snapshots ($75) = $158/mo for literally nothing. I’ve audited accounts where the graveyard was a third of the bill.

The fix: A monthly 15-minute sweep: list load balancers with zero healthy targets, EIPs with no association, volumes with state available, snapshots older than your retention policy. Or codify it — snapshot lifecycle policies via Data Lifecycle Manager, and tag everything so orphans are findable.

4. CloudWatch Logs ingestion — $0.50/GB for your own debug spam #

Why it’s invisible: Storage is what people fear; ingestion is what bills. CloudWatch charges $0.50 per GB ingested— 10× the S3 storage price — and the most common cause is debug logging accidentally left on in production.

Realistic example: Someone ships LOG_LEVEL=debug to prod. Each pod now logs 50 lines/second of request dumps. Six pods × ~86 GB/day combined is small per-pod but adds up to ~2.5 GB/day... or with a genuinely chatty framework, 20+ GB/day = $300+/mo for logs nobody will ever read. The infamous variant: a Lambda in a retry loop logging its own error payload, generating hundreds of GB in a weekend.

The fix: info level in prod, period. Set retention on every log group (the default is never expire). Sample noisy logs. If you need verbose logs, ship them to S3 via Firehose at a fraction of the price. Put a CloudWatch alarm on IncomingBytesper log group — it’s the smoke detector for this exact fire.

5. Data transfer out to the internet — the $2,657-overnight genre #

Why it’s invisible: Egress is ~$0.09/GBafter the free tier, and it’s the only line item on this list that someone elsecan run up for you. A public S3 bucket that gets scraped, a 200 MB video hotlinked from a viral page, an unauthenticated API endpoint returning large payloads — your code doesn’t change, your traffic does, and you pay per byte.

Realistic example: A 250 MB demo file in a public bucket gets posted somewhere popular. 100,000 downloads = 25 TB of egress ≈ $2,250, in a weekend, with zero alerts in default configuration. This is the actual mechanism behind most “$23 became $2,657” posts.

The fix: Nothing user-facing should be served directly from S3 or an instance. Put CloudFrontin front (cheaper per GB, free tier of 1 TB, and S3→CloudFront transfer is free), enable Origin Access Control so the bucket isn’t directly reachable, and require auth or signed URLs for anything large. AWS will often forgive a first-time incident if you ask support — but that’s a lottery ticket, not a strategy.

The 20-minute prevention setup #

Every one of these is observable before the invoice. Wire it up once:

The mindset #

The bill is not a force of nature. Every dollar above is a metric AWS exposes in near-real time — most founders just never wire up the observability. You’d never run production without uptime monitoring; your invoice deserves the same. Twenty minutes of setup converts bill shock from “a horror story that happens to you” into “an alert you acknowledge on Tuesday.”