blog.6admin.io
guest@6admin.io:~$
ci-cd, docker, foreman, jenkins, open-source, security, supply-chain

One Blocking Pin, Four CVEs: The Band-Aid That Wins Is the One That Disappears

The setup: a Docker image that behaves a little too well

I maintain a Dockerized Foreman stack: four reproducible images, built from official tarballs pinned by sha256, with patches, Trivy scans, cosign signatures and a GitLab CI that checks everything. In short, the classic paranoid setup: nothing moves without my say-so.

The problem with pinned versions is that the world keeps moving. So I ran a security audit on the image, the kind that goes “show me your CVEs, big guy”. Trivy spits out its list, I sort it and two things jump out:

  • fixes exist in Foreman’s develop branch, but not in the 5.0.0 release I was pinning;
  • CVE fixes in concurrent-ruby are blocked by a transitive dependency pin (dynflow demands ~> 1.1).

I could keep patching each CVE locally, again and again, with increasingly creative band-aids. Or I climb the chain and contribute directly upstream. Faced with the monumental choice between “repeat the same effort forever” and “do it once so everyone benefits”, even a lazy dev caves in.

Treasure hunt, bug-tracker edition

First instinct: what’s the real state of security bugs in Foreman? Answer: ~4800 open issues, 298 of them security candidates after keyword triage (auth, token, session, csrf, injection…). Cross-referenced with GitHub PRs, the punchline is brutal: 19 have a PR, 279 have none.

After filtering the noise (features dressed as bugs, 2013 issues nobody will resurrect), what remains is a shortlist of ~132 real, recent, impactful bugs with zero PRs. In other words: an endless sandbox for anyone who likes real security problems.

The PRs and the golden rule

The process is strict, enforced as much by their CI as by common sense: one commit per PR (their “Assert Single Commit” check rejects anything else), tests written red before they go green, local reproduction in a postgres:13 + ruby:3.3 Docker setup before touching any code.

The campaign’s haul:

  • foreman #11256: sensitive fields (passwords, tokens) are audited as “not at all” instead of “in plain text in the logs”. The kind of fix where the regression test alone is worth gold.
  • foreman #11279: _ForemanSelected* cookies outlive the logout. Basic session hygiene, but nobody had gotten around to it.
  • Dynflow #473: the infamous blocking pin, more on that below.
  • foreman #11228: the puma pin, also below.
  • jenkins #27432: dead code in CrumbFilter claiming to defend against a 2014 CSRF vector that never actually worked.

The lesson is carved in stone: before opening a PR, search for open AND closed PRs, by issue number and by files touched. Issues open since 2014 attract contributors the way lamps attract moths.

The story worth the detour: the pin that blocked the CVEs

Here’s the textbook supply-chain nightmare:

concurrent-ruby CVE fix (1.3.x)  ->  blocked by
dynflow ~> 1.1                   ->  waiting for
a pin-relaxation PR to merge

So I open Dynflow #473: relax the pin (concurrent-ruby >= 1.1.3, < 2.0), with the bare minimum that follows. Except CI explodes on their side: the “pg goes away for good” test blows past its 60-second window. Tracing the logs, I discover the world does terminate, but the sequence stalls inside start_termination: the timeout watchdog and its completion callback are both scheduled on the same global IO executor, which we just saturated. The death detector died.

I document this in the PR, keep my scope (pin relaxation, nothing more) and jakduch steps in with the real porting (#475): the watchdog moves to a dedicated short-lived thread. Their reproduction measures 80 seconds of termination - hence the 180s window in my PR, CI headroom included. Two PRs, two responsibilities, zero competition.

Merged on Sept 25, approved by a maintainer, polite thank-you. All that remains is waiting for the dynflow gem release on RubyGems so all of Foreman (Redmine #39353 first in line) can unlock its lockfile. The code is merged; the publication clock is what’s ticking.

The puma pin, or how to upstream your own band-aid

Another flavor of the same problem: Foreman’s puma constraint (>= 5.1, '< 7') kept every vulnerable version inside the allowed range (CVE-2026-47736/47737, request injection via PROXY protocol) and blocked the fixed releases - the first patched version above 6.6.1 being 7.2.1.

My Docker image was already carrying the contortion: a sed that rewrites the constraint at build time. Rather than keep it until the heat death of the universe, I filed PR #11228: raise the ceiling from '< 7' to '< 9', so packaging can advance at its own pace. Merged on Sept 11 by ehelms. Once 5.1 lands, my sed goes in the trash and I couldn’t be happier about it.

Closing the loop

The scoreboard:

  • merged: foreman #11228 (puma), Dynflow #473 (concurrent-ruby), jenkins #27432;
  • in review: foreman #11256 (audits), #11279 (cookies), CI 100% green;
  • waiting on publication: the dynflow 2.0.2+ gem.

And when Foreman 5.1 ships, my two Docker band-aids (the puma sed and the concurrent-ruby pin) disappear from my repo. That’s precisely the point: upstream the real fix, then delete your own workaround. A local patch that vanishes when upstream absorbs it is a patch that won.

Three patches later

  1. Search for existing PRs before writing code. Search by issue AND by files touched, open AND closed.
  2. Minimal scope, documented diagnosis. On Dynflow, my PR does exactly one thing (the pin) and the stall diagnosis let the real porting land elsewhere. Had I insisted on doing everything, I’d have been slower, worse and competing with the right fix.
  3. Flaky CI isn’t bypassed, it’s restarted cleanly. Empty amend + force-push to retrigger, one commit preserved and we move on.

The moral of the story: a security audit is never just a report. Every CVE in a list is either a band-aid to slap on, or an upstream to convince. That day, I picked the second box and it merged.

Epilogue, three weeks later

Published on Sept 26 with the dynflow gem still in publication limbo. The clock eventually stopped: dynflow 2.0.2 shipped on RubyGems and on Oct 11 I consumed it end to end. Both tiers unlocked (foreman core and the proxy through smart_proxy_dynflow which already allowed >= 1.6, < 3.0): concurrent-ruby 1.3.8 and edge 0.7.2. The two matching .trivyignore entries went to the trash, documented findings dropped from eight to six and the images went out in the same breath, cosign-signed. The concurrent-ruby loop is closed; the puma sed still waits for 5.1.

A scoreboard correction too: #11279 (cookies) merged on Sept 23. The table above is dated from publication day; the review landed faster than my pen. Still in the race: #11256 (audits), still in review and #11252 / #945, the two PRs that will absorb my last two band-aids when they merge. Redmine #39353, the upstream lockfile, moved to “Ready For Testing”. The rest is on the maintainers’ clock.

The moral still holds: the band-aid that won is the one that disappears. The concurrent-ruby pin is already gone. The next one is the sed.