The cloned hostgroup that refused to save

A cloned Windows hostgroup in Foreman and any attempt to edit it through the interface returns a 500. Not every other time, every time. The sibling hostgroup, not cloned, saves without complaint. At first I suspected a corrupted record and I ended up reading the Foreman source to understand it, because the error pointed at a field that had, on the face of it, no reason to be involved.

The symptom

The save fails on an unambiguous Postgres error:

PG::StringDataRightTruncation: value too long for type character varying(255)

and the audit names the grub_pass field. First reflex, empty grub_pass. It unblocks once, then the problem returns on the next clone, which means grub_pass is not the cause but the symptom and I have to go upstream.

Decoding the audited value, two passes of base64 to UTF-16LE, I land on a cleartext string of the form <password>AdministratorPassword, that is, the contents of an <AdministratorPassword> block from a Windows unattend.xml. In other words, it is not a GRUB password overflowing, it is the Windows administrator password, carried by root_pass with the Base64-Windows hash the unattend needs, that has ended up in grub_pass and is growing.

Read the source rather than guess

The only honest way to conclude was to go and see what Foreman does at save time. Two lines explain everything:

self.root_pass = crypt_pass(root_pass, :root)
self.grub_pass = crypt_pass(self[:grub_pass] || root_pass, :grub)

grub_pass is not an independent field, it is derived from root_pass when empty and crypt_pass is meant not to re-encode a value that is already encrypted. For the Base64-Windows hash, that idempotence test rests on a heuristic that becomes false as soon as root_pass_changed? is true and root_pass_changed? is precisely true on a clone, since the value is reassigned onto the new record. Consequence: on each clone, Foreman treats an already-encoded value as unencrypted and adds a base64(UTF-16LE) layer. The value swells clone after clone, grub_pass accumulating its layers independently, until it passes the column’s varchar(255) limit and the save breaks.

So it is neither an isolated misconfiguration on grub_pass, nor a secret filed in the wrong field: it is a structural behaviour of Foreman’s Base64-Windows password handling, woken by cloning. I would never have deduced it from the symptoms, only from the source.

The fix, which rests on inheritance

The constraint is that root_pass must stay set, it carries the administrator password consumed by the unattend, so removing it is not an option. Fortunately, root_pass is resolved by inheritance in Foreman: the hostgroup’s own value, else the first ancestor that has one, else the global setting and the validation explicitly allows an empty column.

So the fix is to centralise without removing: set root_pass once on the parent that carries the OS and leave the column empty on the leaves and clones, which will inherit it. The reason this kills the overflow is sharp: the callback only acts on the record’s own column, so on a leaf with an empty column it re-encodes nothing and cloning a leaf can no longer swell anything. The template still receives the password by inheritance and cloning leaves, the everyday operation, becomes safe again.

Concrete unblock, once the parent is set cleanly:

docker exec "$(docker ps -qf name=foreman_webapp)" bundle exec rails runner \
 'Hostgroup.unscoped.where(id:[19,23]).update_all(root_pass: nil, grub_pass: nil)'

And above all, do not widen the column to text to silence the error: that would hide the re-encoding and let the value grow forever, which is treating the thermometer.

The real find was elsewhere

Decoding that value to diagnose it, I put my finger on a problem far more uncomfortable than the 500. The Windows local administrator password was stored reversibly, decodable without a key, in the Foreman database and copied in clear into the audit logs, so potentially into the log collection. It was weak on top of that. The configuration incident turned out to be a revealer: the real subject is that a secret meant to stay secret was wandering in clear in two places and the first corrective action was not to unblock the hostgroup but to rotate that password and purge its exposure from the audits.

The culprit

grub_pass is not an independent field, it derives from root_pass, so emptying it alone never fixes anything durably; you have to treat the source. The Base64-Windows hash combined with cloning causes a re-encoding because the idempotence detection rests on root_pass_changed?, true on a clone and the value grows a layer each time. The inheritance of root_pass is the key: setting it once at the top of the tree and leaving the leaves empty keeps the password working without suffering the overflow. And the most useful lesson is not technical: a mundane 500 served as a thread to pull all the way to a secret stored reversibly and logged in clear, a reminder that an availability incident sometimes hides a confidentiality one that you need the reflex to go and look at.