blog.6admin.io
guest@6admin.io:~$
devops, kernel, linux, ssh

Magic SysRq: the sysadmin's crutch

A server is frozen. No SSH, no console, no ping response. You are in a datacenter or on a serial console and the only option before pulling the plug is the SysRq key.

What it is

The SysRq (System Request) key is intercepted directly by the Linux kernel, regardless of system state. The server may be in kernel panic, the scheduler may be blocked: the kernel answers SysRq as long as the hardware interrupts work. It is the emergency crutch.

The physical combination: Alt + SysRq + <key> (SysRq is the Print Screen key on most keyboards). On a serial console (the real datacenter reflex), it is a hardware-dependent escape sequence.

The REISUB sequence

The mnemonic that survives everything: REISUB, read backwards in Latin languages (Busier), gives the clean shutdown order:

  • R - Raw (take back the keyboard from X or whatever grabbed it)
  • E - tErminate (SIGTERM to all processes except init)
  • I - kIll (SIGKILL to all remaining processes)
  • S - Sync (flush disk buffers)
  • U - UnmounT (remount filesystems read-only)
  • B - reBoot (immediate reboot)

Without SysRq, the sequence is: pull the plug and hope the filesystem survives the next boot’s fsck. With SysRq, the S and U guarantee the disk is clean before the B. It is the difference between a 4-second fsck and a 4-hour fsck on a 2 TB volume.

Enabling it

On recent distributions, SysRq is limited or disabled by default. To enable it fully:

echo 1 > /proc/sys/kernel/sysrq

The value is a binary mask: 1 = all allowed, 4 = klog safe, 16 = sync, 32 = remount, 64 = signalling, 128 = reboot/poweroff, 256 = nicing. The 1 enables everything.

What it is for in practice

Two situations I encountered this year:

  1. The server answers ping but not SSH. The scheduler is saturated by a process that ate all the file descriptors. Alt+SysRq+E (SIGTERM) and the processes die, the server takes SSH again. No reboot needed.
  2. The server is completely frozen after an OOM. Alt+SysRq+S,U,B: sync, remount, clean reboot. The next boot’s fsck takes 4 seconds instead of 4 hours.

Before you pull the plug

SysRq is not a management tool, it is a last-resort crutch. The kernel documentation says “crash recovery”. That is exactly what it is: you are recovering a crash and the REISUB sequence is the only thing between you and the plug.