Running into HTTP 429 calling the Gemini API on AWS — and how our team fixed it.

Hi everyone 👋

While building our capstone project (a RAG chatbot running on EC2 that calls the Gemini API), we hit an error that anyone who’s integrated a third-party API has probably run into at least once: HTTP 429 – Too Many Requests.

At first we were pretty confused, because everything ran fine locally — no signs of trouble at all. But the moment we deployed to EC2 and tested with multiple concurrent users, requests piled up and Gemini politely declined with a 429.

🤔 Why did it work locally but break on EC2?

After digging into it, we realized the problem wasn’t EC2 itself — it was a sudden spike in load:

  • Locally: only one of us testing, maybe a few requests per minute — any API can handle that easily.
  • On EC2: multiple users, multiple worker processes running in parallel — requests fired off in quick succession, hit the rate limit almost immediately, and 429s started showing up.

In other words: when just one of us used it, the API was happy to oblige. But once the whole team hit it at the same time, it pushed back.

❗ What happens if you don’t handle it

At first we were dismissive — we figured 429 was a minor issue, retry a couple of times and move on. But looking closer, we realized that without proper handling it can cascade into bigger problems:

  • Interrupted workflows: a task that reads a file → calls the API → writes to the database can get cut off right at the API call step, leaving the data incomplete or inconsistent.
  • Unhandled failures propagate: if the error is allowed to fall through uncaught, it can break the entire worker pipeline running on EC2.
  • Confusing user experience: end users just see a generic 500 error, with no idea what actually went wrong.

🛠️ How we fixed it

Instead of patching one spot, we redesigned the whole API-calling flow to be more resilient. Here’s what we did:

1️⃣ Retry with exponential backoff + jitter

Instead of retrying immediately (which just makes congestion worse), we back off progressively after each failure:

  • 1st failure: wait 2 seconds, then retry.
  • 2nd failure: wait 4 seconds.
  • 3rd failure: wait 8 seconds.

We also added a bit of jitter — a small random offset (around ±0.5s) — so that workers don’t all retry at exactly the same moment and cause another spike.

2️⃣ Batching and chunking

To avoid hitting the rate limit too quickly, we:

  • Batch multiple small items into a single request where the API allows it, cutting down the total number of calls.
  • Split large payloads into reasonably sized chunks instead of stuffing everything into one giant request.
  • Trimmed unnecessary fields from the payload to keep requests lean.

3️⃣ A fallback: switching to Amazon Bedrock

If the Gemini API keeps failing — whether from a temporary outage or a quota limit — the system automatically fails over to Amazon Bedrock, AWS’s managed AI service.

One thing worth noting: switching to Bedrock doesn’t mean quota concerns go away. Bedrock has its own limits per account, region, and token usage. But at least we’re no longer fully dependent on a single provider.

4️⃣ Monitoring with CloudWatch

We ship all error logs (429s, timeouts, etc.) from EC2 to CloudWatch Logs, then set up a metric filter to count how often they occur. If the error count crosses a threshold within a given window, a CloudWatch Alarm fires and notifies the team by email via SNS — no more manually tailing logs.

💡 What we learned

  • “Works fine locally” never means “works fine in production.” As load and user count grow, issues that never show up locally will surface.
  • Retry isn’t a silver bullet — it should only be used for transient errors. If the failure is persistent, retrying endlessly just burns resources and slows the system down further.
  • Any operation you retry should be designed to be idempotent (safe to call more than once with the same result), or your data can end up inconsistent.
  • Always have a fallback for external services you don’t control.

This was the first time our team actually sat down and designed a flow to be resilient by default, instead of just writing code that happens to work — and we learned a lot in the process. Has anyone else in the group run into rate-limit errors integrating third-party APIs on AWS? How did you handle it — would love to hear about it!

📚 References