Ping works. SSH works. But the moment the app sends a large payload over VPN or cross-region peering, it just… hangs. This is why.

Hi everyone 👋

This is a classic debugging trap that can burn a new DevOps engineer several days, precisely because the system never throws a clear error log.

😱 The confusing symptom

Two EC2 instances in different VPCs, connected over VPC peering, VPN, or Direct Connect. Ping between them responds fine. SSH works fine too. But the moment the application transfers a large file, or sends a long JSON payload over an API call on that same connection, the request hangs indefinitely until it eventually times out.

🔍 What’s actually happening

By default, EC2 instances inside the same VPC talk to each other using jumbo frames — an MTU of 9001 bytes. But once traffic crosses a VPC peering connection to another region, a VPN connection, or Direct Connect, the effective path MTU typically drops back down to the standard 1500 bytes.

Small packets (ping, SSH) fit under 1500 bytes either way, so they sail through without a hitch. But a packet larger than the real path MTU needs to be fragmented. If a router along the path can’t fragment it — and the sender set the “don’t fragment” flag, which is common — that router is supposed to send back an ICMP message saying “this is too big, please resend smaller” (ICMP Type 3, Code 4 — Destination Unreachable: Fragmentation Needed).

If a Security Group anywhere along the path blocks all ICMP traffic, that message never makes it back to the sender. The sender just keeps waiting for a response that will never come. This is the classic Path MTU Discovery (PMTUD) black hole.

🔬 How to actually prove it before it bites you

Don’t wait for a hung request to find out. On Linux, send a probe packet at exactly the size that would overflow a 1500-byte path with the “don’t fragment” flag set:

ping -M do -s 1472 <target-ip>

(1472 + the 28 bytes of IP/ICMP headers = 1500.) If this fails while a normal ping <target-ip> succeeds, that gap is your proof: the path’s real MTU is smaller than what your instance thinks it is, and something along the way is silently swallowing the “too big” ICMP response.

🛠️ The fix

  • Always allow ICMP Type 3, Code 4 (Destination Unreachable / Fragmentation Needed) through Security Groups on any path that crosses VPN, peering, or Direct Connect — don’t blanket-block all ICMP for “security.”
  • Alternatively, if a workload depends heavily on inter-VPC/VPN links, lower the EC2 network interface’s MTU to 1500 to match the real path MTU and avoid the fragmentation problem altogether.
  • For VPN/IPsec tunnels specifically, the more robust fix used in production is TCP MSS clamping on the gateway — it forces TCP to negotiate a smaller maximum segment size that already fits inside the tunnel’s real MTU, so oversized packets never get sent in the first place. This works even if you can’t guarantee ICMP makes it through every hop, which is why it’s generally preferred over “just allow ICMP” for VPN links.
  • Direct Connect is a bit of a special case: private and transit virtual interfaces can be configured to support jumbo frames up to 9001 bytes end-to-end — the 1500-byte drop mainly bites you on standard site-to-site VPN and cross-region peering, not necessarily on a properly configured Direct Connect link.

💡 What we learned

  • A working ping proves almost nothing about a path’s real MTU — small packets hide MTU mismatches completely.
  • “Block all ICMP” is a security instinct that quietly breaks a core TCP/IP mechanism (PMTUD) that large-payload connections depend on.
  • The fact that this has no application-level error message is exactly what makes it expensive to debug — it looks like the app is broken, when the actual issue is three network hops away.

📚 References