Switching to IMDSv2 is the right security move, full stop. Do it on an EC2 instance running Docker containers, though, and the app inside loses access to its IAM Role credentials immediately.

Hi everyone 👋

Upgrading from IMDSv1 to IMDSv2 is standard, correct security guidance — it closes off a real class of credential-theft attacks against the EC2 instance metadata service. But flipping that switch on an instance that runs its application inside a Docker container can break AWS SDK access instantly, with no obvious explanation in the error message.

🔍 What’s actually happening

IMDSv2 requires a session token, fetched via a PUT request, before any metadata GET request will succeed. That token request uses the IP packet’s TTL (Time to Live) to guard against session-hijacking from outside the instance — and AWS enforces this with a setting called HttpPutResponseHopLimit, which defaults to 1.

A hop limit of 1 means: the token response is allowed to travel exactly one network hop before AWS considers it invalid. A request made directly from the EC2 instance’s own OS to 169.254.169.254 is one hop — fine. A request made from inside a Docker container has to cross the container’s virtual network interface (the veth bridge) to reach the instance’s network stack, and then from there to the metadata endpoint — two hops. That second hop is exactly what a hop limit of 1 is designed to reject.

The result: the token request from inside the container silently fails, the AWS SDK inside the container can’t retrieve IAM Role credentials, and every AWS API call from that container starts failing — with nothing about it looking like a “networking hop count” problem from the application’s point of view.

🛠️ The fix

Raise the hop limit from 1 to (at least) 2, at the instance metadata options level:

aws ec2 modify-instance-metadata-options \
  --instance-id i-xxxxxxxxxxxx \
  --http-put-response-hop-limit 2 \
  --http-endpoint enabled
  • A hop limit of 2 accommodates exactly the container-to-host-to-metadata path, without opening the door wide — it’s still a small, deliberate number, not disabling the protection.
  • If containers are nested further (a pod network on top of a container network, for instance), the hop limit needs to be increased accordingly to cover each additional layer.
  • Worth knowing: Amazon Linux 2023 AMIs now default to HttpPutResponseHopLimit = 2 out of the box specifically to support containerized workloads — so this trap is mostly a concern on older AMIs or instances configured before that default changed.

💡 What we learned

  • A security upgrade with universally correct guidance (“move to IMDSv2”) can still have an environment-specific gotcha that generic advice doesn’t mention — the fix isn’t “don’t upgrade,” it’s “tune the one parameter that assumes bare-metal-style networking.”
  • The failure mode gives almost no hint about the real cause. “AWS SDK suddenly can’t get credentials” points everywhere except network hop count, unless you already know IMDSv2’s TTL mechanism exists.
  • Defaults tuned for a simpler topology (no container layer) don’t automatically adapt when you add one — anything that introduces a new virtual network hop between an app and the instance metadata endpoint is worth re-checking against this setting specifically.

📚 References