Cloud on The David Page

"Policy ArmResourceId has incorrect formatting" Has Nothing to Do With ARM Resource IDs

Sun, 02 Aug 2026 09:00:00 +0700

Policy ArmResourceId has incorrect formatting — an Azure CLI error message that cost me a whole debugging session, when the real reason was just a hyphen in the policy name.

Hi everyone 👋

Setting up an Azure Front Door WAF Policy through the CLI, I ran into an error that — as it turns out — a lot of other people have been hitting for over three years, ever since it was first reported back in December 2022.

HTTP 429 When Calling the Gemini API on AWS

Sun, 02 Aug 2026 09:00:00 +0700

Running into HTTP 429 calling the Gemini API on AWS — and how our team fixed it.

Hi everyone 👋

While building our capstone project (a RAG chatbot running on EC2 that calls the Gemini API), we hit an error that anyone who’s integrated a third-party API has probably run into at least once: HTTP 429 – Too Many Requests.

At first we were pretty confused, because everything ran fine locally — no signs of trouble at all. But the moment we deployed to EC2 and tested with multiple concurrent users, requests piled up and Gemini politely declined with a 429.

When RDS Proxy Eats 70% of Your Total CloudWatch Ingestion Cost

Sun, 02 Aug 2026 09:00:00 +0700

Pulling up a client’s Cost Explorer, I found RDS Proxy responsible for 70% of their total CloudWatch Logs ingestion cost — while their actual “RDS Proxy” line item looked completely normal.

Hi everyone 👋

My first reaction, and the first reaction of nearly every client I’ve shown this number to, is always the same: “I’m already paying for RDS Proxy by the vCPU-hour — why is it also eating into my CloudWatch bill? Aren’t those two completely separate things?”

AWS Lambda's /tmp Directory Isn't as Clean as You Think

Sun, 02 Aug 2026 08:00:00 +0700

Assuming every Lambda invocation runs in a fresh, isolated environment is a very reasonable guess. It’s also wrong.

Hi everyone 👋

A lot of developers write Lambda functions on the assumption that each invocation starts in a completely clean, isolated sandbox — new environment, empty disk, nothing left behind from whatever ran before. That assumption is exactly what “serverless” seems to promise. It isn’t quite what happens.

🔍 What’s actually happening: warm starts

To keep invocations fast, Lambda reuses the same underlying execution environment (container) for consecutive invocations whenever it can — this is a warm start, as opposed to spinning up a brand-new one from scratch (a cold start). Reusing the container is exactly what makes warm invocations fast.

The IMDSv2 Hop Limit Trap When Running Containers on EC2

Sat, 01 Aug 2026 18:00:00 +0700

Switching to IMDSv2 is the right security move, full stop. Do it on an EC2 instance running Docker containers, though, and the app inside loses access to its IAM Role credentials immediately.

Hi everyone 👋

Upgrading from IMDSv1 to IMDSv2 is standard, correct security guidance — it closes off a real class of credential-theft attacks against the EC2 instance metadata service. But flipping that switch on an instance that runs its application inside a Docker container can break AWS SDK access instantly, with no obvious explanation in the error message.

The Invisible Files on S3: Paying for Data You Can't See (Incomplete Multipart Uploads)

Sat, 01 Aug 2026 14:00:00 +0700

A large upload to S3 dies mid-transfer. The parts that already made it up don’t just vanish — they keep sitting there, billed every month, invisible to the console and to aws s3 ls.

Hi everyone 👋

When an application uploads a large file to S3 using multipart upload and the connection drops partway through, the obvious assumption is: the upload failed, nothing happened, no harm done. That assumption is wrong in a way that quietly costs money.

iam:PassRole + Wildcard: The Privilege Escalation Hiding in Plain Sight

Sat, 01 Aug 2026 09:00:00 +0700

A developer account with nothing but lambda:CreateFunction and a wildcard iam:PassRole can walk itself up to full Administrator — no exploit, no misconfigured bucket, just permissions doing exactly what they were asked to do.

Hi everyone 👋

This one doesn’t show up as a vulnerability scan finding or a CVE — it’s a permissions grant that looks completely reasonable in a policy review, right up until someone connects the dots.

😱 The convenient habit

To make deployments easier, it’s extremely common to grant developer accounts iam:PassRole against * (a wildcard covering every role in the account). The reasoning is simple and, on its own, sounds harmless: developers need to attach an IAM role to whatever they deploy — a Lambda function, an EC2 instance — and wildcarding PassRole means nobody has to update a policy every time a new role gets created.

Archiving Millions of Small Files to Glacier: When the 'Cheap Storage' Trick Backfires

Fri, 10 Jul 2026 09:00:00 +0700

S3 Glacier Deep Archive is famous for dirt-cheap long-term storage. Point a Lifecycle Rule at millions of tiny log files, though, and “cheap” can flip into “more expensive than doing nothing at all.”

Hi everyone 👋

I once confidently set up an S3 Lifecycle Rule: “any object older than 30 days automatically transitions to Glacier Deep Archive.” Simple, standard cost-optimization advice. The catch: the bucket in question held millions of small log files, each only a few KB.

The NAT Gateway 'Trick' That Was Actually Billing You for ECR, Not S3

Thu, 18 Jun 2026 09:00:00 +0700

A client’s NAT Gateway bill hit a record ~$3,000/month. VPC Flow Logs pointed straight at S3 — except the client swore they barely used S3. Here’s what was actually going on.

Hi everyone 👋

A client came to us about a month after going live with a new system built by a different vendor — one they no longer had a maintenance contract with. Their first full invoice after launch showed a NAT Gateway bill at a record high, roughly $3,000/month. Standard architecture on their end: ECS services running in private subnets for security, NAT Gateway for outbound access. Textbook setup on paper — which made a bill that high, right out of the gate, more confusing, not less.

That CloudWatch Logs Insights Query Just Cost You $2.50

Tue, 05 May 2026 09:00:00 +0700

One click on CloudWatch Logs Insights, a few seconds waiting for results — and a bill for scanning half a terabyte of logs.

Hi everyone 👋

CloudWatch Logs Insights is a genuinely convenient SQL-like log query tool. The way AWS bills for it, though, catches a lot of people off guard.

😱 The common mistake

Open Logs Insights, pick a log group that’s accumulated 6 months of logs (~500 GB), type a broad search query with no explicit time range set, hit Run.

DynamoDB On-Demand Has a Ceiling — and Flash Sales Find It

Fri, 10 Apr 2026 09:00:00 +0700

On-Demand is marketed as scale-to-anything with zero capacity planning. It isn’t quite that — and a flash sale is exactly the kind of event that finds the gap.

Hi everyone 👋

DynamoDB On-Demand is pitched as the effortless auto-scaling option — no need to pre-calculate WCU/RCU. That word “automatic” comes with a fine-print condition that matters a lot the first time you hit it.

😱 The disaster scenario

A system runs smoothly at a steady ~1,000 requests/second. At midnight, a flash sale kicks off, and traffic jumps to ~20,000 requests/second within seconds.

Cutting a Client's CloudFront Bill by ~87% with AWS's New Flat-Rate Plans

Mon, 02 Mar 2026 10:00:00 +0700

A client came to us in shock over their CloudFront bill — $1,500/month during their peak sales period. Here’s how we got it down to $200/month.

Hi everyone 👋

A client reached out to us after opening their AWS invoice during their busiest sales month of the year and seeing CloudFront alone hit ~$1,500/month, with the vast majority of that coming from data transfer out to their customers. They weren’t looking for a full re-architecture — they just wanted to know: is this normal, and can anything be done about it?

The MTU 9001 Trap: Flaky Connections Over VPC Peering and VPN

Fri, 20 Feb 2026 09:00:00 +0700

Ping works. SSH works. But the moment the app sends a large payload over VPN or cross-region peering, it just… hangs. This is why.

Hi everyone 👋

This is a classic debugging trap that can burn a new DevOps engineer several days, precisely because the system never throws a clear error log.

😱 The confusing symptom

Two EC2 instances in different VPCs, connected over VPC peering, VPN, or Direct Connect. Ping between them responds fine. SSH works fine too. But the moment the application transfers a large file, or sends a long JSON payload over an API call on that same connection, the request hangs indefinitely until it eventually times out.

The Hidden Cost of Cross-AZ Data Transfer on AWS

Wed, 14 Jan 2026 09:00:00 +0700

Multi-AZ for high availability quietly turned into a network bill bigger than the compute bill — here’s why.

Hi everyone 👋

This is one of those “hidden landmines” on AWS: things that never throw a clear error, never show up in a StackOverflow top answer, but can quietly make a system flaky or burn thousands of dollars a month. First up: cross-AZ data transfer.

😱 The assumption that gets everyone

Everyone knows data transfer out to the internet costs money. Far fewer people realize that traffic between Availability Zones inside the same Region isn’t free either — a lot of us assume “it’s all internal, so it’s free.”