OOMKilled at 2am: requests, limits, and the JVM that didnt know about either

Quick write up because I have now explained this in three separate incident channels this month and I would like to just paste a link next time.

Symptom: a Java service in kubernetes restarts every few hours. No exception in the logs, no stack trace, nothing. The app just... stops. kubectl describe pod shows this:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

137 is 128 + 9, meaning the process got SIGKILL. The kernel OOM killer did it because the container went over its memory limit. The JVM never got a chance to throw OutOfMemoryError, which is why your logs are clean. People lose hours looking for an exception that was never going to be there.

Why the JVM blew past the limit

Modern JVMs (JDK 10+, and 8u191+) are container aware, they read the cgroup limit. But by default max heap is only 25% of that limit (MaxRAMPercentage=25). So teams "fix" it by setting -Xmx equal to the container limit. Thats the bug. Heap is not the only memory a JVM uses. You also have metaspace, thread stacks (around 1MB each by default), code cache, GC overhead, and direct buffers if you use netty or anything NIO heavy. Heap at 100% of the limit means everything else pushes you over.

What we do now:

• Set -XX:MaxRAMPercentage=70 (sometimes 75) instead of a hard Xmx, so it follows the limit if someone changes it later.
• Memory request = memory limit. Memory is not compressible, and when a node is under pressure the pods using more than they requested are the first to get evicted. Overcommitting memory on prod services is gambling.
• Turn on Native Memory Tracking in staging (-XX:NativeMemoryTracking=summary) and actually look at where the non-heap memory goes, at least once.
• Alert on the container memory working set approaching the limit, not on heap usage. Heap can look totally fine right up until the kill.

CPU limits are a different story

While you're in there: CPU is compressible. Going over a CPU limit doesnt kill you, it throttles you, and CFS throttling shows up as weird latency spikes that look exactly like a slow database. A lot of teams (us included) now set CPU requests but no CPU limits on latency sensitive services. Check container_cpu_cfs_throttled_periods_total before you blame postgres. Sorry @arjunmehta, postgres was innocent that time.

Also the JVM sizes GC threads and thread pools from the CPU count it sees, and with a tiny CPU limit you get a JVM that thinks its running on one core. -XX:ActiveProcessorCount exists if you need to override it.

None of this is new, but every new service seems to rediscover it at 2am so here we are.