On-Demand is marketed as scale-to-anything with zero capacity planning. It isn’t quite that — and a flash sale is exactly the kind of event that finds the gap.

Hi everyone 👋

DynamoDB On-Demand is pitched as the effortless auto-scaling option — no need to pre-calculate WCU/RCU. That word “automatic” comes with a fine-print condition that matters a lot the first time you hit it.

😱 The disaster scenario

A system runs smoothly at a steady ~1,000 requests/second. At midnight, a flash sale kicks off, and traffic jumps to ~20,000 requests/second within seconds.

🔍 What’s actually happening

On-Demand capacity mode doesn’t scale to infinity instantly. It instantly accommodates up to double the table’s previous peak traffic — and throttling can occur if you exceed double your previous peak within a 30-minute window. Growth spread out gradually over 30+ minutes is handled far better than an instant step spike.

Twenty thousand requests/second against a previous peak of a thousand is a 20x jump, nowhere near covered by that “up to double” ceiling. DynamoDB immediately starts returning ProvisionedThroughputExceededException and drops requests. Its internal partition-splitting mechanism does catch up — but that takes minutes, and by then, customers have already bounced off the flash sale page.

🔬 Why the ceiling exists at all

This isn’t an arbitrary throttle — it’s a physical consequence of how DynamoDB shards data. Each partition can serve at most 3,000 read capacity units or 1,000 write capacity units, and holds up to 10 GB of data. Scaling a table’s throughput up doesn’t magically add capacity to existing partitions — DynamoDB has to split partitions to spread the new load across more of them, and that splitting takes real wall-clock time. “Up to double, instantly” is roughly the range DynamoDB can absorb using existing partitions before it needs to go through that splitting process.

🛠️ The fix

  • If a short, predictable traffic explosion is coming (flash sale, product launch, marketing push), pre-warm the table using DynamoDB’s warm throughput feature ahead of time — it lets you raise the instant-capacity value a table or index is ready to handle, asynchronously, without switching capacity modes at all. This is the more modern option and doesn’t require leaving On-Demand.
  • If pre-warming isn’t available or practical, the older workaround still applies: switch the table to Provisioned Capacity a few hours ahead of time, sized generously for the expected peak, then switch back to On-Demand afterward if you want.
  • If you can’t pre-provision at all, ramp synthetic or real traffic up gradually over at least 30 minutes rather than a hard step, so DynamoDB’s automatic scaling has a chance to keep pace instead of getting blindsided.
  • For read-heavy spikes specifically, putting DynamoDB Accelerator (DAX) in front of the table absorbs a lot of the burst before it ever reaches the table’s own throughput ceiling.
  • Don’t treat “On-Demand” as a substitute for capacity planning — it removes the need to guess your baseline, not the need to plan for known spikes.

💡 What we learned

  • “Auto-scaling” almost always means “scales automatically within some envelope,” not “scales to infinity instantly” — the envelope is the part that matters operationally.
  • A capacity mode chosen for its simplicity (On-Demand) can still need manual intervention for events you can see coming on a calendar.
  • The failure mode here is silent right up until it isn’t — a smoothly running system for months, then a flash-sale-shaped cliff.

📚 References