Surviving Windows Update in the middle of a build

Some context first, because it is missing. Back then I was running build agents at an employer that developed Windows applications, on Windows, whose update policy was endured rather than chosen: updates landed whenever they decided to, pushed across the fleet, with no regard for whatever was compiling at that moment. Hence the problem that follows.

Anyone who has operated Windows build agents knows that special moment when Windows Update decides, in the middle of a sensitive stage, that it is time to install three cumulative updates and schedule a reboot, wiping out twenty minutes of compilation. The naive reaction is to stop the update service at the start of the build and restore it at the end and that naive reaction holds exactly until the day two builds run in parallel on the same agent, or until the day a build crashes before reaching its cleanup stage, leaving the agent with its updates disabled for eternity. The good news: this problem has a clean solution and its vocabulary comes from shared resource management far more than from Windows administration.

Why stop/start is not enough

The naive approach fails on two fronts that must be handled together. First concurrency: if build A disables, then build B disables and A finishes and re-enables while B is still running, you just handed B back the updates it did not want. Then robustness: any restoration logic placed at the end of the build does not run when the build dies before getting there. You therefore need a reference count that knows how many builds simultaneously demand the disabling, a restoration that happens only when the last one leaves and a guaranteed execution of that restoration even on failure.

The pattern

The idea is a shared library exposing two primitives, acquire and release, resting on three elements. A JSON state file persisting on the agent’s disk the reference count and, crucially, the exact initial state captured at the very first acquire. A global named mutex, of the Global\... kind, serializing access to that file, because the counter’s read-modify-write sequence must be atomic across processes, otherwise two builds incrementing at once will stomp each other. And a best-effort restoration block placed in the pipeline’s post or finally, so a failing build still decrements.

// English comments per house rule.
// Acquire: increment the ref count under the named mutex,
// and on the very first acquire, snapshot the exact prior state.
withGlobalMutex('Global\\6admin_wu_guard') {
    def state = readState()            // {refCount, initial: {...}} or null
    if (state == null) {
        state = [refCount: 0, initial: snapshotWindowsUpdateState()]
    }
    if (state.refCount == 0) {
        disableWindowsUpdate()         // wuauserv + policy keys
    }
    state.refCount++
    writeState(state)
}

// Release (in post/always): decrement, and only the last one out restores.
withGlobalMutex('Global\\6admin_wu_guard') {
    def state = readState()
    state.refCount = Math.max(0, state.refCount - 1)
    if (state.refCount == 0) {
        restoreWindowsUpdateState(state.initial)  // put back EXACTLY what was there
        clearState()
    } else {
        writeState(state)
    }
}

The point I want to underline, because it is the one people forget: you do not restore “enabled”, you restore what was there. The agent may have had its updates driven by a WSUS policy, or already disabled for a reason beyond you and assuming a baseline state corrupts the machine’s configuration while believing you repair it. You capture the service start type and the relevant policy keys, typically NoAutoUpdate under HKLM\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU and you put back precisely that snapshot, no more no less.

The limit we own

One case this pattern does not cover, better written in black and white than pretended away: if the agent reboots during a build, the reference count survives the reboot while the build, itself, is dead and you are left with a ghost refCount that will forever block restoration. The countermeasure is a startup scheduled task which, finding no active build, resets the counter to zero and restores the captured state. It is not elegant, but it is honest and a documented limit with its mitigation will always beat a pattern claimed infallible until the first production incident.

[edit: 2026] The Ansible implementation of all this: the jenkins-agent role .