Resource limits: stopping a machine from being talked into dying

The fork bomb is the whole argument in one line:

:(){ :|:& };:

A function that calls itself twice and backgrounds both copies, run once and the process count doubles every fraction of a second until the process table is full, no new process can start and the machine is a brick needing a hard reboot. It is not an exploit, it needs no privilege and any local user can type it by accident. The same shape drives the rest of the family: a loop that allocates memory until the OOM killer fires, one that opens file descriptors until fork fails with “too many open files”, one that fills /tmp or /var/log until the root filesystem wedges. None of them are clever. All of them are stopped by the same thing, a ceiling the user cannot exceed and the only real work is putting the ceiling in the right place.

The login-session ceiling: pam_limits

pam_limits caps resources per login session and for the fork bomb specifically the relevant knob is nproc. A sane baseline denies any single account enough processes to exhaust the table, raises the ceiling for the accounts that genuinely need it and disables core dumps so a crashing loop cannot also fill the disk:

# /etc/security/limits.conf
*        soft  nproc    4096
*        hard  nproc    8192
*        soft  nofile   1024
*        hard  nofile   65536
*        hard  core     0
@admins  hard  nproc    16384

The trap with pam_limits is the word “login”. It applies to things that come through PAM, which is interactive logins, ssh and su and it does not apply to processes started by cron, by a daemon, or by a service manager. A limit you set here and assume covers everything is a limit with a hole exactly where the long-running services live. For those, the ceiling has to come from somewhere PAM never touches.

The service ceiling: systemd

That somewhere is systemd, which is the modern primary control because every service is already a cgroup and the limits are declarative properties rather than a file the service may or may not read:

# /etc/systemd/system/app.service  (drop-in)
[Service]
TasksMax=200          # the fork-bomb ceiling, per service
MemoryMax=1G          # hard memory cap; the service's own OOM, not the machine's
MemoryAccounting=yes
CPUQuota=50%
LimitNOFILE=16384

TasksMax is the fork-bomb defense for a daemon the way nproc is for a login: a runaway inside the service hits the service’s own ceiling and the rest of the machine never notices. MemoryMax turns a memory leak into a problem confined to one unit, where the cgroup kills the offending service instead of the kernel’s global OOM killer picking a victim at random. The same drop-in mechanism, applied to user@.service, puts a ceiling under a whole interactive user slice:

# /etc/systemd/system/user-.slice.d/limits.conf
[Slice]
TasksMax=4096
MemoryMax=8G

systemd sits on cgroup v2 underneath and dropping to the raw interface is rarely necessary now, but it is the same mechanism: a pids.max, a memory.max, a cpu.max on a cgroup, which is exactly what the unit properties above write for you.

A few system-wide floors and ceilings

Some limits belong to the whole machine, set once in sysctl:

# /etc/sysctl.d/limits.conf
kernel.pid_max = 65536
fs.file-max = 2097152
fs.suid_dumpable = 0          # no core dumps from setuid binaries
kernel.yama.ptrace_scope = 1  # a process may only ptrace its own children

Two of these are about more than exhaustion. suid_dumpable = 0 stops a setuid program’s crash from leaving a core dump full of secrets and ptrace_scope = 1 stops one unprivileged process from attaching to another, which closes a common credential-theft path. They sit in this note because they travel with the limits, not because they stop a fork bomb.

The memory overcommit knobs (vm.overcommit_memory, vm.overcommit_ratio) tempt people into strict accounting and they can help, but they change allocation behaviour for every workload on the box and routinely break applications that allocate lazily. I would not reach for them as a general hardening default; the per-service MemoryMax gets you the containment without the global surprise.

Monitoring and what it is for

The honest view of resource usage is per-cgroup, which means per-service and systemd already aggregates it:

systemd-cgtop                               # live, by slice and service
systemctl show app.service -p MemoryCurrent -p TasksCurrent

What I would not build is the reflex this kind of document usually recommends, a script that greps for the fork-bomb pattern and kills matching processes, or worse, “ML-based anomaly detection”. By the time a fork bomb is visible the table is already full and the watcher cannot start a new process to do anything about it. That is precisely why the defense is a static ceiling set in advance, not a detector reacting after the fact: the limit refuses the thousandth process whether or not anyone is watching and the watcher exists for the post-mortem, not the save.

Where to land

Set nproc, nofile and core in limits.conf for the interactive and ssh sessions, remembering it stops at the PAM boundary. Put the real ceilings on services with systemd, TasksMax and MemoryMax per unit, which is the half that covers the daemons PAM never sees. Add the handful of sysctl floors, keep suid_dumpable and ptrace_scope locked down while you are there and leave the global overcommit knobs alone unless a specific workload asks for them. The fork bomb is trivial, which is the point: the defense is equally trivial and the only failure is not having set it before the user types it.