Linux hurt itself in its confusion! Exploiting Out-of-Memory Killer for Confusion Attacks via Heuristic Manipulation

The Linux kernel’s Out-of-Memory (OOM) killer ensures system stability by terminating processes when memory is exhausted, but its heuristic-based design was not built for adversarial contexts. This paper introduces OOM Confusion Attacks, a novel class of Denial of Service (DoS) attacks that exploit the OOM killer to execute privileged process termination, targeting critical services rather than attacker processes. By orchestrating memory exhaustion through numerous unprivileged processes, these attacks may kill target applications, block service recovery, and destabilize systems. We demonstrate the feasibility of OOM Confusion Attacks on default Linux configurations commonly used by cloud providers, formulate and quantify the resource constraints for success, and evaluate application exposure to OOM Confusion Attacks. Additionally, we identify race conditions that can be exploited to block the recovery of privileged services. To mitigate these threats, we propose strategies to increase the resilience of critical applications.