S3 Glacier Deep Archive is famous for dirt-cheap long-term storage. Point a Lifecycle Rule at millions of tiny log files, though, and “cheap” can flip into “more expensive than doing nothing at all.”

Hi everyone 👋

I once confidently set up an S3 Lifecycle Rule: “any object older than 30 days automatically transitions to Glacier Deep Archive.” Simple, standard cost-optimization advice. The catch: the bucket in question held millions of small log files, each only a few KB.

😱 The confident setup

Deep Archive storage runs at roughly $0.00099/GB/month — about 23x cheaper than S3 Standard’s ~$0.023/GB/month. On paper, auto-archiving anything older than 30 days looks like free savings. Nobody thought to check what happens when “anything” means ten million objects that are each smaller than a phone photo’s thumbnail.

🔍 Hidden cost #1: the transition request fee

AWS charges for each Lifecycle transition request — the S3 API call that actually moves an object into Glacier Deep Archive — at roughly $0.05 per 1,000 requests. That fee is charged per object, regardless of how small the object is.

Ten million small files means ten million transition requests: 10,000,000 ÷ 1,000 × $0.05 = $500, as a one-time cost, just to move the data. If the actual storage savings from archiving all that data only amount to a few dollars a year (which, for a few tens of GB of tiny logs, it will), that one-time transition bill alone can dwarf an entire year — or several years — of the storage savings it was supposed to unlock.

🔍 Hidden cost #2: the per-object overhead

Here’s the part that’s easy to underestimate even after you know about the transition fee: every object moved into S3 Glacier Flexible Retrieval or Glacier Deep Archive carries a 40 KB metadata/indexing overhead, billed as:

  • 8 KB billed at S3 Standard rates
  • 32 KB billed at the Glacier/Deep Archive rate

That overhead is per object, on top of the object’s actual size. A 4 KB log file archived to Deep Archive doesn’t get billed as 4 KB — it gets billed as roughly 44 KB, and a meaningful chunk of that (the 8 KB piece) is charged at the expensive Standard rate, not the cheap archive rate.

🔬 A back-of-envelope example

Take 10 million log files averaging 4 KB each (~38 GB of real data):

Before (S3 Standard):
  38 GB × $0.023/GB           ≈ $0.87/month  (~$10.5/year)

After "optimizing" into Deep Archive:
  real data (38 GB)   × $0.00099/GB           ≈ $0.04/month
  32 KB overhead × 10M objects (~305 GB) × $0.00099/GB ≈ $0.30/month
  8 KB overhead  × 10M objects (~76 GB)  × $0.023/GB   ≈ $1.76/month
  ---------------------------------------------------------------
  Total                                        ≈ $2.10/month (~$25/year)

Plus a one-time transition fee: 10,000,000 ÷ 1,000 × $0.05 = $500

This is illustrative back-of-envelope math using published per-GB rates, not a guaranteed number for every account — but the shape of it holds generally: the ongoing storage cost after “optimizing” ends up roughly 2.4x higher than just leaving the data on S3 Standard, driven almost entirely by that 8 KB-per-object overhead billed at Standard rates. Add the one-time transition fee, and the first year alone costs about 50x more than doing nothing.

🛠️ The fix

  • Pack small files before archiving. Combine many small log files into fewer, larger archives (tar/zip) before they hit the Lifecycle transition age. One 40 MB tarball pays the 40 KB overhead and one transition-request fee exactly once; ten thousand 4 KB files pay it ten thousand times.
  • Add a minimum object size filter to the Lifecycle Rule. S3 Lifecycle Rules support filtering by object size — set a floor (commonly recommended around 128 KB+) so only objects actually large enough to benefit stay in the rule, and small files are excluded or routed to a cheaper-but-not-archive tier instead (S3 Standard-IA, for instance).
  • Consider S3 Intelligent-Tiering for irregular access patterns instead of a hand-written age-based rule — it moves objects between tiers automatically based on actual access patterns, and it’s designed with per-object monitoring costs in mind rather than blindly archiving everything past a fixed age.

💡 What we learned

  • “Cheaper per GB” doesn’t mean “cheaper overall” once request fees and per-object overhead enter the picture — those scale with the number of objects, not the amount of data, and millions of tiny objects is exactly the shape of workload where that bites hardest.
  • A Lifecycle Rule with no size filter treats a 2 TB backup and a 2 KB log line identically — and archive storage classes were designed with the former in mind, not the latter.
  • Do the per-object math before enabling a rule, not after the first invoice. The one-time transition fee and the recurring overhead cost are both trivial to estimate ahead of time from the object count alone — no need to wait for the bill to find out.

📚 References