AWS

HTTP 429 When Calling the Gemini API on AWS

Running into HTTP 429 calling the Gemini API on AWS — and how our team fixed it. Hi everyone 👋 While building our capstone project (a RAG chatbot running on EC2 that calls the Gemini API), we hit an error that anyone who’s integrated a third-party API has probably run into at least once: HTTP 429 – Too Many Requests. At first we were pretty confused, because everything ran fine locally — no signs of trouble at all. But the moment we deployed to EC2 and tested with multiple concurrent users, requests piled up and Gemini politely declined with a 429.

Sunday, August 2, 2026 Read

When RDS Proxy Eats 70% of Your Total CloudWatch Ingestion Cost

Pulling up a client’s Cost Explorer, I found RDS Proxy responsible for 70% of their total CloudWatch Logs ingestion cost — while their actual “RDS Proxy” line item looked completely normal. Hi everyone 👋 My first reaction, and the first reaction of nearly every client I’ve shown this number to, is always the same: “I’m already paying for RDS Proxy by the vCPU-hour — why is it also eating into my CloudWatch bill? Aren’t those two completely separate things?”

Sunday, August 2, 2026 Read

AWS Lambda's /tmp Directory Isn't as Clean as You Think

Assuming every Lambda invocation runs in a fresh, isolated environment is a very reasonable guess. It’s also wrong. Hi everyone 👋 A lot of developers write Lambda functions on the assumption that each invocation starts in a completely clean, isolated sandbox — new environment, empty disk, nothing left behind from whatever ran before. That assumption is exactly what “serverless” seems to promise. It isn’t quite what happens. 🔍 What’s actually happening: warm starts To keep invocations fast, Lambda reuses the same underlying execution environment (container) for consecutive invocations whenever it can — this is a warm start, as opposed to spinning up a brand-new one from scratch (a cold start). Reusing the container is exactly what makes warm invocations fast.

Sunday, August 2, 2026 Read

The IMDSv2 Hop Limit Trap When Running Containers on EC2

Switching to IMDSv2 is the right security move, full stop. Do it on an EC2 instance running Docker containers, though, and the app inside loses access to its IAM Role credentials immediately. Hi everyone 👋 Upgrading from IMDSv1 to IMDSv2 is standard, correct security guidance — it closes off a real class of credential-theft attacks against the EC2 instance metadata service. But flipping that switch on an instance that runs its application inside a Docker container can break AWS SDK access instantly, with no obvious explanation in the error message.

Saturday, August 1, 2026 Read

The Invisible Files on S3: Paying for Data You Can't See (Incomplete Multipart Uploads)

A large upload to S3 dies mid-transfer. The parts that already made it up don’t just vanish — they keep sitting there, billed every month, invisible to the console and to aws s3 ls. Hi everyone 👋 When an application uploads a large file to S3 using multipart upload and the connection drops partway through, the obvious assumption is: the upload failed, nothing happened, no harm done. That assumption is wrong in a way that quietly costs money.

Saturday, August 1, 2026 Read

iam:PassRole + Wildcard: The Privilege Escalation Hiding in Plain Sight

A developer account with nothing but lambda:CreateFunction and a wildcard iam:PassRole can walk itself up to full Administrator — no exploit, no misconfigured bucket, just permissions doing exactly what they were asked to do. Hi everyone 👋 This one doesn’t show up as a vulnerability scan finding or a CVE — it’s a permissions grant that looks completely reasonable in a policy review, right up until someone connects the dots. 😱 The convenient habit To make deployments easier, it’s extremely common to grant developer accounts iam:PassRole against * (a wildcard covering every role in the account). The reasoning is simple and, on its own, sounds harmless: developers need to attach an IAM role to whatever they deploy — a Lambda function, an EC2 instance — and wildcarding PassRole means nobody has to update a policy every time a new role gets created.

Saturday, August 1, 2026 Read

Archiving Millions of Small Files to Glacier: When the 'Cheap Storage' Trick Backfires

S3 Glacier Deep Archive is famous for dirt-cheap long-term storage. Point a Lifecycle Rule at millions of tiny log files, though, and “cheap” can flip into “more expensive than doing nothing at all.” Hi everyone 👋 I once confidently set up an S3 Lifecycle Rule: “any object older than 30 days automatically transitions to Glacier Deep Archive.” Simple, standard cost-optimization advice. The catch: the bucket in question held millions of small log files, each only a few KB.

Friday, July 10, 2026 Read

The NAT Gateway 'Trick' That Was Actually Billing You for ECR, Not S3

A client’s NAT Gateway bill hit a record ~$3,000/month. VPC Flow Logs pointed straight at S3 — except the client swore they barely used S3. Here’s what was actually going on. Hi everyone 👋 A client came to us about a month after going live with a new system built by a different vendor — one they no longer had a maintenance contract with. Their first full invoice after launch showed a NAT Gateway bill at a record high, roughly $3,000/month. Standard architecture on their end: ECS services running in private subnets for security, NAT Gateway for outbound access. Textbook setup on paper — which made a bill that high, right out of the gate, more confusing, not less.

Thursday, June 18, 2026 Read

That CloudWatch Logs Insights Query Just Cost You $2.50

One click on CloudWatch Logs Insights, a few seconds waiting for results — and a bill for scanning half a terabyte of logs. Hi everyone 👋 CloudWatch Logs Insights is a genuinely convenient SQL-like log query tool. The way AWS bills for it, though, catches a lot of people off guard. 😱 The common mistake Open Logs Insights, pick a log group that’s accumulated 6 months of logs (~500 GB), type a broad search query with no explicit time range set, hit Run.

Tuesday, May 5, 2026 Read

DynamoDB On-Demand Has a Ceiling — and Flash Sales Find It

On-Demand is marketed as scale-to-anything with zero capacity planning. It isn’t quite that — and a flash sale is exactly the kind of event that finds the gap. Hi everyone 👋 DynamoDB On-Demand is pitched as the effortless auto-scaling option — no need to pre-calculate WCU/RCU. That word “automatic” comes with a fine-print condition that matters a lot the first time you hit it. 😱 The disaster scenario A system runs smoothly at a steady ~1,000 requests/second. At midnight, a flash sale kicks off, and traffic jumps to ~20,000 requests/second within seconds.

Friday, April 10, 2026 Read

Cutting a Client's CloudFront Bill by ~87% with AWS's New Flat-Rate Plans

A client came to us in shock over their CloudFront bill — $1,500/month during their peak sales period. Here’s how we got it down to $200/month. Hi everyone 👋 A client reached out to us after opening their AWS invoice during their busiest sales month of the year and seeing CloudFront alone hit ~$1,500/month, with the vast majority of that coming from data transfer out to their customers. They weren’t looking for a full re-architecture — they just wanted to know: is this normal, and can anything be done about it?

Monday, March 2, 2026 Read

The MTU 9001 Trap: Flaky Connections Over VPC Peering and VPN

Ping works. SSH works. But the moment the app sends a large payload over VPN or cross-region peering, it just… hangs. This is why. Hi everyone 👋 This is a classic debugging trap that can burn a new DevOps engineer several days, precisely because the system never throws a clear error log. 😱 The confusing symptom Two EC2 instances in different VPCs, connected over VPC peering, VPN, or Direct Connect. Ping between them responds fine. SSH works fine too. But the moment the application transfers a large file, or sends a long JSON payload over an API call on that same connection, the request hangs indefinitely until it eventually times out.

Friday, February 20, 2026 Read