A client’s NAT Gateway bill hit a record ~$3,000/month. VPC Flow Logs pointed straight at S3 — except the client swore they barely used S3. Here’s what was actually going on.

Hi everyone 👋

A client came to us about a month after going live with a new system built by a different vendor — one they no longer had a maintenance contract with. Their first full invoice after launch showed a NAT Gateway bill at a record high, roughly $3,000/month. Standard architecture on their end: ECS services running in private subnets for security, NAT Gateway for outbound access. Textbook setup on paper — which made a bill that high, right out of the gate, more confusing, not less.

We started tracing the same day.

🔍 First stop: VPC Flow Logs

We turned on VPC Flow Logs and broke down NAT Gateway traffic by destination. The result was clear almost immediately: the overwhelming majority of bytes flowing through the NAT Gateway were headed to Amazon S3 IP ranges.

That’s when the conversation with the client got interesting.

❗ “But we barely use S3”

We told the client what we found, and they pushed back — their application, by their own account, did very little direct reading or writing to S3. No large media uploads, no log shipping to S3 buckets, nothing that should account for that volume.

They weren’t wrong about their own application code. So we went a layer deeper and reviewed the full architecture instead of just the app — which, since the previous vendor was long gone and there was no maintenance contract to fall back on, meant reconstructing the full picture ourselves rather than asking someone who already knew it.

That review turned up the real story: the system ran a surprisingly large number of scheduled batch jobs — far more than the client had front-of-mind when we first asked about their S3 usage. Each one of those jobs, plus the backend ECS services scaling up and redeploying throughout the day, pulled container images from Amazon ECR on every single run — every batch job invocation, every deployment, every autoscaled task launch re-pulling the same multi-hundred-MB images.

At that point I told the team: this is it, this is exactly the spot we’ve been looking for.

One of our team members immediately asked the obvious question: “Wait, why would pulling images from ECR show up as traffic to S3?”

📖 Why ECR traffic is actually S3 traffic

Here’s the part that isn’t obvious unless you’ve dug into ECR’s architecture before: Amazon ECR stores container image layers in Amazon S3. When something pulls an image from ECR, the flow is actually two steps:

  1. Talk to the ECR API to get the image manifest (which layers make up this image).
  2. Download the actual image layers — which are served from an S3 bucket that ECR manages behind the scenes.

Step 2 is almost always the overwhelming majority of the bytes, since image layers are what’s actually large; the manifest call is tiny by comparison. So every batch job and every task launch pulling a large image was, under the hood, generating a large S3 download — routed through the NAT Gateway like any other internet-bound traffic, because nothing in the VPC was telling it to do otherwise.

💸 Why that’s expensive specifically because of NAT Gateway

NAT Gateway doesn’t just charge an hourly rate — it charges a data processing fee per GB that flows through it, in both directions, and that charge applies even when the destination is another AWS service in the same region. S3 traffic routed through a NAT Gateway is billed exactly like any other NAT-routed traffic — there’s nothing about “it’s just AWS-to-AWS” that makes it free by default.

At the client’s actual pull volume — dozens of large images, repeated across many batch runs and deployments per day — that adds up to a NAT Gateway bill dominated by data that never needed to touch a NAT Gateway at all.

🛠️ The fix: an S3 Gateway VPC Endpoint

The fix here was refreshingly simple: enable an S3 Gateway VPC Endpoint on the client’s VPC.

  • It’s free — no hourly charge, no per-GB processing fee.
  • Traffic to S3 now travels over AWS’s internal network path instead of through the NAT Gateway.
  • Setup took minutes, and required no changes to application code, no endpoint URLs to update — S3 traffic from within the VPC is automatically routed through the gateway endpoint once it’s attached to the relevant route tables.
  • As a side effect, it also sped up ECR pulls, since traffic no longer has to funnel through the NAT Gateway’s throughput.

Once it was in place, the S3-destined portion of the NAT Gateway bill — the vast majority of it — simply disappeared the following billing cycle.

💡 What we learned

  • “We barely use S3” can be true for the application code and still be wrong for the account as a whole. ECR, and several other AWS services, use S3 under the hood — the traffic is real even when no one wrote a single s3:// reference in the app.
  • VPC Flow Logs tell you the destination, not the reason — tracing to “traffic to S3” was step one; understanding why required reviewing the full architecture, not just the app’s direct integrations.
  • Gateway VPC Endpoints for S3 are close to a free win — no cost, minimal setup, no code changes, and they fix both the NAT bill and ECR pull latency at the same time. If EC2/ECS/EKS workloads in private subnets pull from ECR at any real frequency, this endpoint should arguably be there by default, not added reactively after a shock invoice.

📚 References