AWS

System Design Survival Guide: The Dream of "Boring" vs Ugly Reality 🚀

Mixing Sean Goedecke’s take with the full-contact roast the Hacker News wizards gave it (957 points, 390 comments), you start to see that being a software engineer means trying to please everyone at once. Here’s a savage-but-affectionate recap of the internet’s opinions on system design — full credit to Sean for the original ideas, the roasting is all Hacker News, and the part at the end where I get petty is entirely my own doing.

Monday, August 17, 2026 Read

I Benchmarked Lambda arm64 vs x86_64 on an I/O-Bound Workload So You Don't Have To

Graviton is always pitched as cheaper. I wanted to know if it’s also faster — for a boring, I/O-bound Lambda function, not the CPU-crunching demos AWS likes to show off. AWS says faster. I don’t do “trust me bro” benchmarks. AWS markets Lambda on Graviton (arm64) as up to 20% cheaper and faster than x86_64 for “most workloads.” Most of the public benchmarks backing that up are CPU-bound — image compression, number crunching, that kind of thing. But most Lambda functions running in production aren’t doing math, they’re doing I/O: call DynamoDB, call S3, call some other API, then wait.

Sunday, August 16, 2026 Read

HTTP 429 When Calling the Gemini API on AWS

Running into HTTP 429 calling the Gemini API on AWS — and how our team fixed it. 429 is Google’s polite way of saying “slow down.” While building our capstone project (a RAG chatbot running on EC2 that calls the Gemini API), we hit an error that anyone who’s integrated a third-party API has probably run into at least once: HTTP 429 – Too Many Requests. At first we were pretty confused, because everything ran fine locally — no signs of trouble at all. But the moment we deployed to EC2 and tested with multiple concurrent users, requests piled up and Gemini politely declined with a 429.

Sunday, August 2, 2026 Read

When RDS Proxy Eats 70% of Your Total CloudWatch Ingestion Cost

Pulling up a client’s Cost Explorer, I found RDS Proxy responsible for 70% of their total CloudWatch Logs ingestion cost — while their actual “RDS Proxy” line item looked completely normal. The line item said normal. The bill said otherwise. My first reaction, and the first reaction of nearly every client I’ve shown this number to, is always the same: “I’m already paying for RDS Proxy by the vCPU-hour — why is it also eating into my CloudWatch bill? Aren’t those two completely separate things?”

Sunday, August 2, 2026 Read

AWS Lambda's /tmp Directory Isn't as Clean as You Think

Assuming every Lambda invocation runs in a fresh, isolated environment is a very reasonable guess. It’s also wrong. “Fresh environment” is carrying a lot of weight in that sentence. A lot of developers write Lambda functions on the assumption that each invocation starts in a completely clean, isolated sandbox — new environment, empty disk, nothing left behind from whatever ran before. That assumption is exactly what “serverless” seems to promise. It isn’t quite what happens.

Sunday, August 2, 2026 Read

The IMDSv2 Hop Limit Trap When Running Containers on EC2

Switching to IMDSv2 is the right security move, full stop. Do it on an EC2 instance running Docker containers, though, and the app inside loses access to its IAM Role credentials immediately. Turns out “do the secure thing” has a body count. Upgrading from IMDSv1 to IMDSv2 is standard, correct security guidance — it closes off a real class of credential-theft attacks against the EC2 instance metadata service. But flipping that switch on an instance that runs its application inside a Docker container can break AWS SDK access instantly, with no obvious explanation in the error message.

Saturday, August 1, 2026 Read

The Invisible Files on S3: Paying for Data You Can't See (Incomplete Multipart Uploads)

A large upload to S3 dies mid-transfer. The parts that already made it up don’t just vanish — they keep sitting there, billed every month, invisible to the console and to aws s3 ls. The upload failed. AWS billed it anyway. When an application uploads a large file to S3 using multipart upload and the connection drops partway through, the obvious assumption is: the upload failed, nothing happened, no harm done. That assumption is wrong in a way that quietly costs money.

Saturday, August 1, 2026 Read

iam:PassRole + Wildcard: The Privilege Escalation Hiding in Plain Sight

A developer account with nothing but lambda:CreateFunction and a wildcard iam:PassRole can walk itself up to full Administrator — no exploit, no misconfigured bucket, just permissions doing exactly what they were asked to do. No exploit. No CVE. Just IAM doing exactly what you told it to. This one doesn’t show up as a vulnerability scan finding or a CVE — it’s a permissions grant that looks completely reasonable in a policy review, right up until someone connects the dots.

Saturday, August 1, 2026 Read

Archiving Millions of Small Files to Glacier: When the 'Cheap Storage' Trick Backfires

S3 Glacier Deep Archive is famous for dirt-cheap long-term storage. Point a Lifecycle Rule at millions of tiny log files, though, and “cheap” can flip into “more expensive than doing nothing at all.” “Cheap storage” had an asterisk, and nobody reads asterisks. I once confidently set up an S3 Lifecycle Rule: “any object older than 30 days automatically transitions to Glacier Deep Archive.” Simple, standard cost-optimization advice. The catch: the bucket in question held millions of small log files, each only a few KB.

Friday, July 10, 2026 Read

The NAT Gateway 'Trick' That Was Actually Billing You for ECR, Not S3

A client’s NAT Gateway bill hit a record ~$4,000/month. VPC Flow Logs pointed straight at S3 — except the client swore they barely used S3. Here’s what was actually going on. Another AWS bill, another crime scene. A client came to us about a month after going live with a new system built by a different vendor — one they no longer had a maintenance contract with. Their first full invoice after launch showed a NAT Gateway bill at a record high, roughly $4,000/month. Standard architecture on their end: ECS services running in private subnets for security, NAT Gateway for outbound access. Textbook setup on paper — which made a bill that high, right out of the gate, more confusing, not less.

Thursday, June 18, 2026 Read

That CloudWatch Logs Insights Query Just Cost You $2.50

One click on CloudWatch Logs Insights, a few seconds waiting for results — and a bill for scanning half a terabyte of logs. One query, half a terabyte scanned, and CloudWatch didn’t even pretend to warn you first. CloudWatch Logs Insights is a genuinely convenient SQL-like log query tool. The way AWS bills for it, though, catches a lot of people off guard. 😱 The common mistake Open Logs Insights, pick a log group that’s accumulated 6 months of logs (~500 GB), type a broad search query with no explicit time range set, hit Run.

Tuesday, May 5, 2026 Read

DynamoDB On-Demand Has a Ceiling — and Flash Sales Find It

On-Demand is marketed as scale-to-anything with zero capacity planning. It isn’t quite that — and a flash sale is exactly the kind of event that finds the gap. “Scale-to-anything” is doing a lot of marketing’s heavy lifting. DynamoDB On-Demand is pitched as the effortless auto-scaling option — no need to pre-calculate WCU/RCU. That word “automatic” comes with a fine-print condition that matters a lot the first time you hit it.

Friday, April 10, 2026 Read