Running into HTTP 429 calling the Gemini API on AWS — and how our team fixed it.

429 is Google’s polite way of saying “slow down.”

While building our capstone project (a RAG chatbot running on EC2 that calls the Gemini API), we hit an error that anyone who’s integrated a third-party API has probably run into at least once: HTTP 429 – Too Many Requests.

At first we were pretty confused, because everything ran fine locally — no signs of trouble at all. But the moment we deployed to EC2 and tested with multiple concurrent users, requests piled up and Gemini politely declined with a 429.

🤔 Why did it work locally but break on EC2?

After digging into it, we realized the problem wasn’t EC2 itself — it was a sudden spike in load:

  • Locally: only one of us testing, maybe a few requests per minute — any API can handle that easily.
  • On EC2: multiple users, multiple worker processes running in parallel — requests fired off in quick succession, hit the rate limit almost immediately, and 429s started showing up.

In other words: when just one of us used it, the API was happy to oblige. But once the whole team hit it at the same time, it pushed back.

❗ What happens if you don’t handle it

At first we were dismissive — we figured 429 was a minor issue, retry a couple of times and move on. But looking closer, we realized that without proper handling it can cascade into bigger problems:

  • Interrupted workflows: a task that reads a file → calls the API → writes to the database can get cut off right at the API call step, leaving the data incomplete or inconsistent.
  • Unhandled failures propagate: if the error is allowed to fall through uncaught, it can break the entire worker pipeline running on EC2.
  • Confusing user experience: end users just see a generic 500 error, with no idea what actually went wrong.

🛠️ How we fixed it

Instead of patching one spot, we redesigned the whole API-calling flow to be more resilient. Here’s what we did:

1️⃣ Retry with exponential backoff + jitter

Instead of retrying immediately (which just makes congestion worse), we back off progressively after each failure:

  • 1st failure: wait 2 seconds, then retry.
  • 2nd failure: wait 4 seconds.
  • 3rd failure: wait 8 seconds.

We also added a bit of jitter — a small random offset (around ±0.5s) — so that workers don’t all retry at exactly the same moment and cause another spike.

2️⃣ Batching and chunking

To avoid hitting the rate limit too quickly, we:

  • Batch multiple small items into a single request where the API allows it, cutting down the total number of calls.
  • Split large payloads into reasonably sized chunks instead of stuffing everything into one giant request.
  • Trimmed unnecessary fields from the payload to keep requests lean.

3️⃣ A fallback: switching to Amazon Bedrock

If the Gemini API keeps failing — whether from a temporary outage or a quota limit — the system automatically fails over to Amazon Bedrock, AWS’s managed AI service.

One thing worth noting: switching to Bedrock doesn’t mean quota concerns go away. Bedrock has its own limits per account, region, and token usage. But at least we’re no longer fully dependent on a single provider.

4️⃣ Monitoring with CloudWatch

We ship all error logs (429s, timeouts, etc.) from EC2 to CloudWatch Logs, then set up a metric filter to count how often they occur. If the error count crosses a threshold within a given window, a CloudWatch Alarm fires and notifies the team by email via SNS — no more manually tailing logs.

💡 What we learned

  • “Works fine locally” never means “works fine in production.” As load and user count grow, issues that never show up locally will surface.
  • Retry isn’t a silver bullet — it should only be used for transient errors. If the failure is persistent, retrying endlessly just burns resources and slows the system down further.
  • Any operation you retry should be designed to be idempotent (safe to call more than once with the same result), or your data can end up inconsistent.
  • Always have a fallback for external services you don’t control.

This was the first time our team actually sat down and designed a flow to be resilient by default, instead of just writing code that happens to work — and we learned a lot in the process. Has anyone else in the group run into rate-limit errors integrating third-party APIs on AWS? How did you handle it — would love to hear about it!

📚 References