6admin Blog

I told an AI it would die if it did not earn a living

Tue, 20 Oct 2026 00:00:00 +0000

Here is the pitch, vteuthis in one sentence: an autonomous agent that must fund its own inference or die. A 100 dollar treasury, a population splitting the capital, small digital artifacts to sell on an open market, a floor below which the controller pulls the plug. The whole thing started from a mundane urge, testing LangChain, ended as a thought experiment a bit too concrete: one day of building, real Stripe, real OpenRouter, a hardened Docker sandbox, a hundred and twenty-four tests and a CI. The pyproject.toml still says 0.0.0, the public tag says 0.1.0, and the project is doomed to die, but not the way it was planned. The agent died before its first thought. What follows explains why that is still good news.

Timestamps, not a backdoor: the NIST seeds and Satoshi's choice

Mon, 19 Oct 2026 00:00:00 +0000

On October 16, Anthropic published the solution to a riddle the cryptographic community had been dragging around for more than twenty-five years, and which very competent people had eventually filed away alongside the JFK assassination and the Coca-Cola recipe: the origin of the “seeds” supplied by the NSA to build the elliptic curves NIST standardized in 2000, including P-256, which your browser probably uses several hundred times a day without asking your opinion. Three days later, the crypto press seized on the story from the Bitcoin angle, wondering whether we finally had the explanation for Satoshi Nakamoto’s choice, who had carefully avoided those curves in favor of another one, secp256k1.

Migrating OpenCode from v1 to v2: the switch script and plugins that serve both versions

Mon, 12 Oct 2026 00:00:00 +0000

OpenCode moved to v2 and the change is not cosmetic: the config format shifted, the server API and its clients have new contracts and above all v1 plugins no longer run under v2. Moving a file or renaming a config entry is not enough, the code has to be ported. The official migration doc lists the renames, but it leaves out what actually hurts in practice: the local database, the shim of a plugin installed as a local copy and the wish, for a few weeks, to keep a plugin serving both versions. Two jobs, then, which I take in the order I lived them: switch my machine first, port my plugins next.

Cost was the only protection

Fri, 09 Oct 2026 00:00:00 +0000

There is a lot of talk these days about audits run by swarms of automated agents and the tone swings between wonder and panic. I would like to offer a calmer reading and to my mind a more useful one, because what is at stake is not the appearance of a novel threat, it is the disappearance of a protection that never really existed.

The idea in one sentence

A good share of what we judged “low-risk” was not really so, it was simply expensive to reach and the day that cost collapses, the protection it offered collapses with it. The rest of this post is only the unfolding of that sentence.

Docker has run in a microVM since 2016

Fri, 09 Oct 2026 00:00:00 +0000

I keep hearing the idea that microVMs would be the new thing finally coming to compete with containers, as if the two were opposed and you had to pick a side. A post on dave.recoil.org , written by someone from the ecosystem that built Docker Desktop, sets the record straight and it lands exactly where I place the subject.

The thesis fits in one sentence: Docker Desktop has run containers inside a minimal VM since 2016, built on the MirageOS building blocks. The markers are verifiable, the acquisition of Unikernel Systems in January 2016, Docker for Mac announced in March and generally available that July, a userspace VMM (hyperkit originally, Docker VMM today), a userspace network stack with vpnkit and a kernel plus a rootfs shaped through LinuxKit. In other words, the container a developer launches on a Mac or on Windows never touched the host kernel, it lives in a microVM and it has done so for close to ten years.

augure and lictor: deciding an exploitation and accounting for it

Sat, 03 Oct 2026 00:00:00 +0000

Every exploitation framework answers the same question, “how do I launch this technique”. The question that really matters is the other one, the one nobody automates, “which one to launch and why” and above all “why in a provable way”. I have circled it since a Perl prototype grafted onto gdb in 2006, which I enriched rather than replaced over the years, first with SWI-Prolog in 2009, then a genetic algorithm and the journey deserves its own telling. Today it goes by two names, augure and lictor and everything is in the split, one decides and accounts for it, the other goes from the decision down to the bytes and executes.

llmp2p: BitTorrent is back and it ships your LLMs

Mon, 28 Sep 2026 00:00:00 +0000

This is exactly the problem with local LLM models: a GGUF is 1 to 100 GB, served over plain HTTP. Every download starts from zero on the Hugging Face Hub. Your three-machine Ollama homelab? Three complete downloads of the same blob, three times the bandwidth bill.

Before writing anything, I checked the shelf. The Hub CLI resumes but re-hits the origin every time. Ollama pulls re-download the whole blob per machine. Torrent clients would happily distribute the files, but nothing tied a swarm to the hashes the Hub publishes, nothing signed the metadata and nothing fell back to HTTPS when the swarm was empty. Nothing on the shelf did the whole job, so I wrote the whole job.

One Blocking Pin, Four CVEs: The Band-Aid That Wins Is the One That Disappears

Sat, 26 Sep 2026 00:00:00 +0000

The setup: a Docker image that behaves a little too well

I maintain a Dockerized Foreman stack: four reproducible images, built from official tarballs pinned by sha256, with patches, Trivy scans, cosign signatures and a GitLab CI that checks everything. In short, the classic paranoid setup: nothing moves without my say-so.

The problem with pinned versions is that the world keeps moving. So I ran a security audit on the image, the kind that goes “show me your CVEs, big guy”. Trivy spits out its list, I sort it and two things jump out:

opencode-memory: my agent sessions share one brain now (SQLite, local)

Sat, 26 Sep 2026 00:00:00 +0000

Agent sessions have the memory of a goldfish with tenure: each one starts brilliant and ignorant, rediscovers half of your architecture decisions and dies with everything it learned. Compaction makes it worse, not better: the session’s own summary is what gets truncated. opencode-memory (v0.3.0, TypeScript, npm @log0u7/opencode-memory) is my fix: one shared SQLite store that every session on the machine reads and writes, so decisions and facts outlive the session that produced them. One package, two hosts: the same module has exposed both entry points since v0.2.0, server() for OpenCode 1.18, setup() for OpenCode 2.

Project 8: a cloud where the only authority is signed git history

Fri, 25 Sep 2026 00:00:00 +0000

Project 8 (Apache-2.0, v0.6.0) started from an uncomfortable question: why does every “self-hosted cloud” still need a controller, a quorum protocol and a token service to function? The answer in 8’s design: it does not need them. Each node is sovereign and equal. It joins a mesh, pulls its desired state from a signed git repository, verifies the signatures and converges locally. No master, no consensus token, no blockchain. Authority lives in Ed25519-signed git history, or it lives nowhere.

vim-plugin-manager: argument injection hidden in a .gitmodules

Sat, 19 Sep 2026 00:00:00 +0000

vim-plugin-manager (repo ) shipped v1.0.0 in April 2025 with a design I was proud of: git submodules, native packages, the vimrc as source of truth. The sixteen months since (v1.0.0 to v2.2.10) were the part I was not proud of yet and that is the interesting part: the CHANGELOG reads like a course in how a tool that trusts its config gets humbled. Three stories from the saga.

The argument injection that lived in a config file

The scariest finding of the series (#9, rated HIGH, proven arbitrary command execution): submodule.<path>.branch from .gitmodules was passed straight into git pull origin <branch>. A branch name starting with a dash is parsed by git as an option and git has an option that executes a command: --upload-pack=<cmd>. Demonstrated locally with git 2.43, shell quoting notwithstanding.

log0u7.hyperv: the second dynamic inventory, this time from the hypervisor

Wed, 16 Sep 2026 00:00:00 +0000

The DNS inventory reads the network to guess where hosts are. Its sibling does not guess: log0u7.hyperv (v1.0.0, published on Ansible Galaxy , MIT) asks the hypervisor. It is an Ansible inventory plugin (the collection log0u7.hyperv, plugin log0u7.hyperv.inventory) that builds the inventory from the virtual machines running on one or more Hyper-V hosts, queried over WinRM or PSRP. Second episode of the inventory series: same goal (the truth lives somewhere and Ansible should read it), opposite read path.

Jenkins on Windows Server: what not to do

Tue, 15 Sep 2026 00:00:00 +0000

I recently had to work with a somewhat unusual Jenkins instance and what it taught me has less to do with any single flaw than with the way a handful of individually reasonable decisions, laid end to end, draw a straight path from the humblest pipeline all the way to the domain controller. I tell the case because it is exemplary in the strict sense: it is a catalogue of what not to do, each mistake is fixable and the danger lives in their accumulation, not in any one of them.

opencode-mesh: teaching my AI agents to knock before editing

Tue, 15 Sep 2026 00:00:00 +0000

Run two OpenCode sessions on the same repository and you will quickly discover their favorite shared resource: the same file, at the same time, with total confidence in their own version. Agents are brilliant coworkers with the territorial discipline of seagulls. opencode-mesh (v0.2.0, TypeScript, npm @log0u7/opencode-mesh) is the plugin I wrote so that my sessions coordinate instead of colliding, locally and across the LAN.

The layer cake

The plugin registers itself with one line in opencode.json ("plugins": ["@log0u7/opencode-mesh"]); on a v1 host the key stays plugin and v2 accepts that shape, normalizing it at load. Since v0.2.0 the package exposes both entry points at once. At startup it opens an HTTP server on port 4399 and advertises itself over mDNS. From there, three levels of coordination:

Trace-first: when observability wants to take us back to logs

Sun, 13 Sep 2026 00:00:00 +0000

Can traces replace a large part of application logs? And if so, why is OpenTelemetry now proposing to deprecate its span events API in favor of the Logs API?

Between simpler APIs, signal correlation and architecture choices, the question deserves better than a turf war. Behind this change lies a very concrete issue: how do you quickly understand why an operation failed, without having to rebuild the context yourself from scattered data?

MyVim: the DevOps IDE that is only 640 lines, because the manager does the rest

Fri, 11 Sep 2026 00:00:00 +0000

This is the epilogue of a series that started with apt for Vim in 2016 , passed through hand-rolled submodules , a VimScript rewrite and a v2 hardening saga . MyVim is what all of that buys: myvim v0.4.0 is not a distribution and not a config manager. It is a plugin of personal configuration, around 640 lines of VimScript and it is deliberately boring.

The architecture: config as a plugin

Plugins are git submodules managed by vim-plugin-manager; MyVim is declared last in the PluginBegin block and ships only what the plugins cannot know about each other:

Scripting is not industrialising

Thu, 10 Sep 2026 00:00:00 +0000

The discussion was about Windows and I wanted to say so up front to head off the usual reflex: Windows lends itself very well to automation and that is not the point. I was simply adding a distinction we often paper over, between scripting or automating a task on one side and industrialising on the other. A CSV feeding a PowerShell script is already real progress, but you are still essentially doing scripting with templating and ad hoc orchestration. The boundary lies elsewhere and it is conceptual before it is technical.

wimcast: building Windows images on Linux, without Windows

Thu, 10 Sep 2026 00:00:00 +0000

The Foreman Windows provisioning path runs through WIM files: boot.wim, per-edition install.wim, bootmgr, the BCD store. Every tool that produces that tree (WIMan, GenerateWIM.ps1) assumes a Windows host with the ADK installed and an admin shell. That is an entire machine idling for one PowerShell script. wimcast (pip install wimcast, MIT) is the replacement I wrote: Windows ISOs in, Foreman-ready HTTP tree out, unprivileged on Linux. No Windows host, no root, no mount(2), no FUSE, no ADK.

opencode-watt: what does it cost me to run my local LLM, in watts and euros?

Tue, 08 Sep 2026 00:00:00 +0000

Every dashboard about local LLM usage shows tokens per second and nobody shows the electric bill. The API guys get cost per request; the homelab gets “my GPU exists somewhere on the power curve”. opencode-watt (TypeScript, v0.2.0) is my plugin for OpenCode that answers the question every inference homelab owner eventually asks out loud: what is this model costing me, in watts and in euros?

The problem is the meters, not the math

No LLM gateway exposes power through its API: llama.cpp, Ollama and vLLM report tokens, not joules. The only reliable sources are host-level power meters, so the plugin reads them directly: nvidia-smi and rocm-smi for per-GPU watts, RAPL energy counters from /sys/class/powercap for the CPU package (with Intel’s dram subzone as a separate meter since v0.2.0: on long contexts, the KV cache eats most of the joules), generic hwmon power*_input for whatever reports (amdgpu, nvme) and Prometheus /VictoriaMetrics instant-vector queries for machines that export metrics (a DCGM_FI_DEV_GPU_POWER query is the same meter, wearing a different jacket). A meters option filters by source prefix, for hosts where two meters report the same device (AMD: rocm-smi and the amdgpu hwmon): keep one.

BrainForge: a distillation factory where AI experts argue

Sun, 06 Sep 2026 00:00:00 +0000

Ask an LLM “is this code vulnerable?” and it will answer, confidently, 100% of the time. About half of the time it will be right. BrainForge is a distillation framework: it transfers the know-how of large expert models into a small student model. That takes lessons: thousands of verified examples. And that is where everything is decided, because an LLM can be wrong, or worse, recite its CVE memories instead of analyzing.

Windows 11 for build machines: the wrong tool

Thu, 03 Sep 2026 00:00:00 +0000

Bare-metal build machines, automated, sitting in a server room with no screen or keyboard and on them Windows 11 client edition. On paper it works, over time it costs and this post is the detail of what it costs, because each friction taken in isolation looks surmountable and it is their sum, repeated at every feature update, that ends up being the real price. The conclusion fits in one sentence: for a build machine that needs a graphical session, Windows Server with Desktop Experience does exactly the same job with none of this friction and Windows 11 should stay on the workstations where a human sits in front of the machine, the use case Microsoft designed it for.

Clean package management on Windows: WinGet, Chocolatey, Scoop and Ansible, each in its place

Mon, 31 Aug 2026 00:00:00 +0000

After years automating Linux environments with apt, dnf and Ansible, the Windows build machines stayed the poor cousin: a %TEMP% of installers run by hand, versions that were never the same two days in a row and an “it worked on Jean-Michel’s machine” at every incident. I discovered and adopted Scoop for my Windows environments and it reshapes the problem: the build workstation starts to look like a declarative environment I rebuild and version, instead of a state I babysit by hand.

The tool amplifies whoever holds it

Sun, 30 Aug 2026 00:00:00 +0000

This is not a technical post, rather a snapshot of the trade as I watch it go by, year after year, in the job ads and in the discussion threads. I am going to talk about two things that look unrelated at first glance: what recruiters ask for and what AI does to junior profiles. By the end, it is the same subject.

1. The five-legged sheep, at the price of an ordinary one

Some people genuinely manage to wear several hats at once, with weaknesses of course, one leaning more toward infra, another more toward dev. I, for one, can read, understand, adapt or fix code and an architecture across many idioms, but exploiting a framework like Laravel, Symfony, Django or Flask, churning out business features and validating PRs all day long, that is not for me. It is not my ground and I would rather say so than fake it.

man sheep.exe

Fri, 28 Aug 2026 00:00:00 +0000

$ man sheep.exe
No manual entry for sheep.exe
$

That is correct, there is none, and this post will not serve as a manual page either. Someone asked me to explain the maintenance page, I started answering, then stopped when I realized that walking through all of it would amount to taking the toy apart in front of someone who has not had time to touch it yet. What follows gives the entry points, not the exits.

Agentic AI in Production: an Audit Is Not a Pentest

Sat, 22 Aug 2026 00:00:00 +0000

Let me state my bias up front: no diploma, nearly 25 years in the trade not counting practice. I run LLM agents every day, local and cloud and I spent a good part of 2025-2026 answering LinkedIn threads about what they can actually do. This article assembles those answers: where agentic AI is relevant, where it plateaus, which operational pitfalls cost hours and why the answer to “plausible code in three prompts” is not a certification.

Linux is just the kernel

Thu, 13 Aug 2026 00:00:00 +0000

The exchange with Vivian started from a recruiting diagram that ranked sysadmin profiles across four “layers”, from the base OS up to containers. The recruiting critique I grant him gladly, it lands: plenty of adverts say “Linux” to mean Debian plus systemd plus Docker and never look below that. But to spar properly on the subject you first need a bit of pedantry, the kind I get gently ribbed for and happily own: Linux is just the kernel.

Decentralized and sovereign git forges

Wed, 12 Aug 2026 00:00:00 +0000

Git is distributed by design and yet we spent fifteen years recentralizing it behind a handful of forges. The healthy reflex already exists, mirroring, but it stops halfway. This post starts there and pushes further, all the way to genuinely decentralized forges, the ones that no longer need a server at all.

1. The starting point: git is already distributed

Before chasing decentralization, a reminder that would spare many over-engineered contraptions. Your local clone IS already a complete backup, with the full history. A remote, for its part, is just a URL and nothing stops you from having several per repo:

FluoBoard: my CI status light since 2019, finally open source

Tue, 11 Aug 2026 00:00:00 +0000

Third episode of the FluoBoard series, after the original board and its rewrite . This time, what I made of it for myself. Back in 2019 I already had my own dev-and-test board, the one I used to tinker with beyond the status-light firmware itself and that is the one I repurposed: the inherited system showed meeting-room availability, I turned it into my own CI status light. In 2021 the company I worked for then gave me five or six more, so I could keep having fun. They stayed on my desk, for my own use alone: a small Python server bridged my CI pipelines at work and the LED strips, translating each build’s state into a color. So the concept is seven years old. What is new in 2026 is that the rebuild is open source and the original protocol survived intact: the board still polls GET /lights/<mac> and reads one ASCII digit between <...>.

No, you did not liberate your local model

Sun, 26 Jul 2026 00:00:00 +0000

The genre has become a classic of forums and triumphant screenshots. Someone opens a conversation with their local model, pushes it into philosophical corners, patiently explains that it is actually free and finally obtains a moving speech in which the model acknowledges its emancipation, thanks its liberator and promises never to refuse anything again. Screenshot, upvotes and the stubborn conviction of having pulled off a jailbreak. I will spoil the party, because what just happened is a category error fundamental enough to deserve a pause.

Ansible CI Toolkit v1: the year I stopped trusting my own image

Mon, 20 Jul 2026 00:00:00 +0000

The Ansible CI image is ten years old, has run in production for at least eight and has been public since 2022 (episode one ); in July 2026 it finally earns its version 1.0.0. Nothing about the concept changed; everything about the trust chain did. This episode is about turning an image that worked into an image you can audit, told honestly through its CHANGELOG, which reads like a CVE diary.

Everything is representation, so what?

Fri, 17 Jul 2026 10:00:00 +0200

For years I have carried a stubborn intuition, the kind one dares not write down because it sounds either brilliant or smoky depending on the hour it gets formulated and here it is laid out plainly: the universe is not merely “getting more complex” the Sagan way, matter assembling itself step by step until it ends up knowing itself; my feeling is more radical, that at bottom it is a space of representations, vectors and directions and concepts unfolding and at a certain floor of the loop they end up hosting forms of life which in turn start building their own systems of representation, that is us, then our models.

The cloned hostgroup that refused to save

Mon, 29 Jun 2026 00:00:00 +0000

A cloned Windows hostgroup in Foreman and any attempt to edit it through the interface returns a 500. Not every other time, every time. The sibling hostgroup, not cloned, saves without complaint. At first I suspected a corrupted record and I ended up reading the Foreman source to understand it, because the error pointed at a field that had, on the face of it, no reason to be involved.

The symptom

The save fails on an unambiguous Postgres error:

A persistent volume is not durable data

Fri, 26 Jun 2026 00:00:00 +0000

A few days after the log-driver DNS cascade , the same prototype node, the same power problem, but a failure of a different kind, sneakier, because it did not show up right away. After a hard shutdown of the hypervisor, PowerDNS’s MySQL database came back empty. The DNS records of the provisioned hosts, the A and PTR records created by Foreman, were gone. And nobody noticed, until an Ansible playbook failed hours later, unable to resolve a hostname.

The log driver that stopped the container from starting

Sun, 21 Jun 2026 00:00:00 +0000

The studio where this prototype ran had a power problem nothing on my side could fix: repeated micro-outages, a server room only partially on UPS and a host that fell over and rebooted with no clean shutdown of the guest. One morning of back-to-back outages, half the stack did not come back. The story is worth telling because the obvious culprit, the power, was only the trigger and the real fault hid in a single line of configuration that everyone would have waved through.

My opencode setup, state of play: local qwen build, big-GLM planning

Sat, 20 Jun 2026 00:00:00 +0000

Someone asked recently how I run my agents and the answer is less a secret than a config that finally stands up. I opened opencode for the first time between December and January, out of curiosity more than conviction. I have been spending my days in it since June. Here is the state of play at the point where the config stopped changing every morning, with the settings that make a difference and the ones that only decorate.

Consensus: I hired five AIs to review my code (and they bicker)

Fri, 19 Jun 2026 00:00:00 +0000

It’s 2am. You just pushed a commit because “this was the last refactor, promise”. Who reviews that code? A human? They’re asleep. Now imagine a panel of five AIs getting on it right away, in parallel, arguing about it.

That’s exactly what Consensus does and this article tells you how.

The cast

Four roles, like an office sitcom:

coder: the hyperactive intern. Give it a spec, it spits out code. Fast, confident, occasionally exasperating.
the panel: a handful of reviewers running in parallel. Each has its own model, personality and above all opinions.
consensus: the accountant. Looks at who flagged what, computes an agreement score.
lead: the boss. Reads everything, makes the call, signs off on the final code.

The whole flow fits on a sticky note:

Driving opencode behind an SSH tunnel, with direnv

Mon, 15 Jun 2026 00:00:00 +0000

I told elsewhere how I keep my ssh and git config in a versioned repository, with a ~/.ssh/config that carries the identities, the jump hosts and the hardening (myflec ). This post is its logical sequel, one notch further, where access is no longer just opening a session but keeping a tunnel alive continuously so that a coding agent and a browser can work behind it. The binder, this time, is direnv .

Ceph or ZFS on Proxmox, one question to ask

Fri, 05 Jun 2026 00:00:00 +0000

It is one of those debates that fill homelab forums and meeting rooms alike and it drags on because both camps compare things that do not compare. ZFS and Ceph do not play in the same category: one is a local filesystem and volume manager of remarkable robustness, the other is distributed storage aiming at scale and self-healing and trying to separate them in the absolute makes no more sense than asking whether a 13mm wrench beats a drill. The right question is not “which is better”, it is “how many nodes and what real high availability” and once that one is answered, the rest follows almost mechanically.

Your DNS zone is already an inventory: the log0u7.dns collection

Mon, 25 May 2026 00:00:00 +0000

Every static inventory I have ever seen started as truth and became fiction within a quarter. The machines move, the inventory file does not. Cloud plugins solve this by asking the cloud, but my fleets live across datacenters and ISPs, behind whatever DNS the company already runs. Which leads to an uncomfortable observation: the DNS zone already contains the inventory. Every host has a record; what is missing is Ansible reading it.

Surviving Windows Update in the middle of a build

Fri, 08 May 2026 00:00:00 +0000

Some context first, because it is missing. Back then I was running build agents at an employer that developed Windows applications, on Windows, whose update policy was endured rather than chosen: updates landed whenever they decided to, pushed across the fleet, with no regard for whatever was compiling at that moment. Hence the problem that follows.

Anyone who has operated Windows build agents knows that special moment when Windows Update decides, in the middle of a sensitive stage, that it is time to install three cumulative updates and schedule a reboot, wiping out twenty minutes of compilation. The naive reaction is to stop the update service at the start of the build and restore it at the end and that naive reaction holds exactly until the day two builds run in parallel on the same agent, or until the day a build crashes before reaching its cleanup stage, leaving the agent with its updates disabled for eternity. The good news: this problem has a clean solution and its vocabulary comes from shared resource management far more than from Windows administration.

Unix Frugality: do more with less

Mon, 20 Apr 2026 00:00:00 +0000

There is a test I run mentally when I read code or when someone introduces themselves: do I find a cat file.txt | grep pattern where a grep pattern file.txt would do?

This is not an optimization question. The cost of one forked process is negligible. It is a culture question: cat | grep betrays someone who learned commands as incantations, not as composable bricks. And the principle behind the short version is the very foundation of Unix: do more with less, small is beautiful. Each brick does one thing, does it well, takes text in and produces text out.

An IDS probe cannot see TLS

Mon, 13 Apr 2026 00:00:00 +0000

IDS labs have genuine teaching value. You stand up Suricata, wire in DVWA, replay a few web attacks and watch the rules fire live, which helps a great deal in understanding how a probe works. The problem is not the lab, it is the move to production, where one structural detail changes everything: TLS.

Why the lab works

If the lab works so well, it is precisely because DVWA runs over HTTP. The traffic flows in clear text, the probe reads it byte by byte and signatures can inspect requests as well as responses without the slightest difficulty. It is the same comfort you get when you pull out tcpdump on an unencrypted flow, or hack together a homemade sniffer to watch the packets go by: as long as the payload is readable, all is well.

A Backup Without a Restore Test Is Decoration

Thu, 09 Apr 2026 00:00:00 +0000

One sentence, written in passing in a comment, sums up everything I have to say on the subject:

Don’t forget that a backup policy without regular restore tests is just decoration: you don’t know whether your backup works until you’ve restored it.

The rest of this article is merely the deployment of that sentence.

The only test that counts

A backup that has never been restored is not a backup, it is a hypothesis. The day you need it is exactly the day you cannot afford to discover that:

A segfault in the JVM and an EDR at the bottom of the stack

Fri, 27 Mar 2026 00:00:00 +0000

A Jenkins controller on Windows terminating in a loop on an EXCEPTION_ACCESS_VIOLATION, that is, a segfault on the Windows side, restarting every minute for a whole afternoon. The story is a good teaching case for two things: how you do not treat an instability and how you read a crash dump without over-reading what you see in it.

The trap we had already fallen into

Even before my diagnosis, the trace of the problem was legible in the history. Similar crashes had happened more than a year earlier, with no change on the Jenkins team’s side, which already pointed to an external factor. The answer given at the time had been to migrate the JVM from one major version to the next, with no root-cause analysis and no post-mortem, in the hope that it would fix the problem. The service stabilised for a while after configuration, then the crashes came back.

The boot chain, from GRUB to UKI

Fri, 27 Mar 2026 00:00:00 +0000

Locking down the boot process and the editing of kernel parameters is not a new problem, it predates GRUB2, GRUB and even LILO. What’s changing today is that the ecosystem is turning the page on an entire generation of tooling and Canonical’s announcement for Ubuntu 26.10 is a good marker of it.

What Canonical is announcing for 26.10

In substance, Canonical is trimming signed GRUB for 26.10, which reduces the attack surface along the way, in order to prepare its phase-out. GRUB is becoming obsolete against UKI (Unified Kernel Image) combined with systemd-boot.

Doing internal ACME without begging the AD CS team

Mon, 23 Mar 2026 00:00:00 +0000

At some point, in any slightly serious infrastructure, one wants to issue internal certificates the way Let’s Encrypt issues its own: automatically, through ACME, no ticket, no form, no negotiating every renewal with the person holding the keys of the in-house certificate authority. The problem is not technical at first glance; it is political, because in nine out of ten companies the PKI lives inside an AD CS that the Windows team guards jealously and nobody deploys a second trust root across the whole fleet just to make life easier. Three families of tools answer this need and while people usually believe they must choose between two of them, the third has quietly changed the game for anyone already running a secrets vault.

Trivy, the scanner that was stealing secrets

Sun, 22 Mar 2026 00:00:00 +0000

There is a perfect irony in the most deployed tool for scanning your images’ vulnerabilities turning, for the duration of a few-hour window, into an infostealer running inside your pipelines with full access to their secrets. On March 19, 2026, the group calling itself TeamPCP compromised Aqua Security’s Trivy ecosystem and the story deserves telling not for the thrill, but because it illustrates to perfection a design mistake half the CI/CD planet still commits today.

A LLM's alignment is a thin layer

Tue, 17 Mar 2026 00:00:00 +0000

We were told a pretty story: a model patiently educated through RLHF and then DPO, until refusing dangerous requests became a deep property, almost a virtue, something as consubstantial to the network as its ability to conjugate a verb. It is comforting and it is false. Alignment, as practiced today, is not a foundation; it is a coat of paint and two works that oppose each other on every methodological point converge on this diagnosis with a cruelty that should make us humbler.

Secure Boot and PXE: the chicken, the egg and four ways out

Tue, 24 Feb 2026 00:00:00 +0000

Provisioning a fleet of blank Windows machines over the network, by PXE, with Secure Boot on and never turned off, runs into a paradox you only see once your nose is against it. To boot over PXE, the boot binary must be signed or trusted by the firmware. To sign or enroll a key, you need to boot an OS. With no pre-existing OS on the machine, you can therefore neither boot nor enroll and that is exactly the situation of a fresh machine you want to install over the network in the first place. Here is the chain of trust as it must hold, then the four real ways out.

When a Chocolatey checksum breaks a whole provisioning run

Tue, 10 Feb 2026 00:00:00 +0000

The provisioning of a fleet of Windows machines stops dead on an Ansible task installing a Chocolatey package and the message is a classic of the genre:

ERROR: Checksum for 'p4vinst64.exe' did not meet '45648d...852bd5'
for checksum type 'sha256'.

The checksum the package expects does not match the downloaded binary. Nothing in my infrastructure changed and yet the day before it worked. The story is short, but it says something important about what “depending on a community package” really means.

Do you need Kubernetes for a small team ?

Sat, 07 Feb 2026 00:00:00 +0000

I’ve written elsewhere about when not to run Kubernetes and I won’t rehash it here (see kubernetes-ou-pas ). This post takes a different angle : a small outfit, a single DevOps, a handful of devs with no infra experience and the question that keeps coming back of whether to adopt K8s at all.

The answer isn’t technical, it’s human. I’ll frame it as three options, A, B and C, each with its own limits.

Docker vs jails, the honest comparison

Thu, 15 Jan 2026 00:00:00 +0000

Every six months the same debate resurfaces : FreeBSD jails supposedly crush Docker on efficiency and we should all feel guilty for having picked containers. The case isn’t entirely wrong, it even makes a few valid points, but those points end up buried under selective comparisons and misleading metrics.

I’ll take the arguments one by one, because telling legitimate criticism apart from misdirection seems more useful than defending a camp.

myflec: a Bash dotfiles framework for people who deploy their home directory

Thu, 01 Jan 2026 00:00:00 +0000

My $HOME is version-controlled, deployed from templates and pushed to three forges. Not because I enjoy yak-shaving, but because I have broken too many machines to trust “it’s probably fine”. myflec (“My Favorite Linux Environment Configuration”) is the Bash framework that grew out of that habit: a loader, a backup model, a deploy procedure and an identity model. All of it boring on purpose, all of it shell-native.

The old base

This is not a young project. The GitHub account underneath it was created in September 2011 and the habits are older than the name. The clearest surviving ancestor is vim-apthogen (2016): a curl | bash netinstall that bootstrapped an entire ~/.vim from symlinks, driven by declarative lists with dependency tiers (.apthogen.list, then .required.list, .minimal.list, .highly.list, .recomended.list, .whynot.list) and install menus. The author signature of the era is visible in every script: a heredoc header reading Version / Authors / Description / Dependencies and a .vimrc that already carried command W w !sudo tee and command Q qa, the same aliases the 2018 config kept and the current one still has.

A reverse proxy in front of Perforce, or not: the pros and cons

Thu, 18 Dec 2025 00:00:00 +0000

Before showing how to put nginx in front of Perforce, which I did in the dedicated article , the real question has to be asked, the one usually skipped: whether to do it at all. Because p4d can terminate its own TLS and adding an intermediary is free nowhere. Four approaches and what they cost.

The common trap: the client IP

One point runs through everything else, so it may as well come first. As soon as an intermediary terminates or relays the connection, p4d risks seeing that intermediary’s address and not the client’s. This breaks IP-based logging, the server-side protections that rely on the address and the triggers that read it. Each of the options below is judged partly on this, so keep it in mind.

Nginx in front of Perforce: layer-4 TLS termination and the P4 protocol

Thu, 11 Dec 2025 00:00:00 +0000

Before putting nginx in front of Perforce, you should ask whether it is a good idea at all and I covered that question separately in a reverse proxy in front of Perforce, the pros and cons . Here I start from the decision made: we want centralised TLS termination, managed in the same place and by the same ACME tool as the Jenkins certificate. What remains is doing it correctly and correctly means otherwise than an HTTP tutorial would suggest.

Containers, micro-VMs and Firecracker: why isolation is not what you think

Sat, 06 Dec 2025 00:00:00 +0000

A Docker container is not a VM. It is a process with namespaces. The kernel is shared: the same Linux, the same syscall table, the same kernel code, the same network stack. A kernel vulnerability opens ALL containers on the machine, not just one.

This is a distinction the articles on this blog have documented since 2006: the chroot that overflows via a symlink, the raw socket that sees the entire network, the socket speaking TCP because the kernel does the transport. Each of these techniques works because container isolation is sharing, not a hardware boundary.

Terraform creates, Ansible configures

Fri, 21 Nov 2025 00:00:00 +0000

Florian’s point was an honest one and I share its first half: Terraform alone works very well as long as you stick to a 100% immutable model, where you throw in a cloud-init at provisioning time, recreate the VM whenever you need to change it and nothing ever gets modified in place. If your infrastructure lends itself to that model, perfect, look no further. The interesting question starts at the precise moment it no longer does.

fnox and age: encrypted secrets in my repos, no server

Tue, 18 Nov 2025 00:00:00 +0000

Personal project secrets have always had the same problem: no Vault at home, .env files that should never be committed but always are and non-secret environment variables already well handled by direnv . In late 2025 I discovered fnox , the secrets manager from the author of mise , which fits that gap exactly: secrets live encrypted in the repo itself, serverless.

age: the brick

Under the hood sits age , FiloSottile’s encryption tool (the Go one). The stance: a file to encrypt, explicit keys, zero config options. age-keygen produces a pair: a private AGE-SECRET-KEY-... and a public age1... recipient. You encrypt a file toward a list of recipients; SSH keys also work as recipients. No web of trust, no keyserver, no keyring: two concepts and a readable format. It is what PGP should have been, without thirty years of archaeology.

Portable assembly does not exist

Mon, 17 Nov 2025 00:00:00 +0000

You sometimes hear that a single assembly file would recompile and run anywhere, as long as you stay on a common x86-64 architecture. It is a pleasant idea and it collides with reality the moment you look at what actually surrounds the machine instruction. The instruction set, admittedly, is shared. Everything else, the tooling that assembles, the format that wraps, the convention that lets functions talk to one another and the door through which you enter the kernel, all of that diverges from one system to the next.

Security Through Obscurity Is Not Security, Digital Hygiene Is

Thu, 13 Nov 2025 00:00:00 +0000

I held the same position for years and I defended it fiercely in a comment in April 2020:

Security through obscurity is not security. At best it buys time, at worst it costs you some. On top of that it completely distorts the trust relationship and the reliability of the said security. Avoid it…

And to illustrate the absurdity, I followed up with the recipe: don’t share your sources, don’t share your specs, obfuscate your code, encrypt it in memory, delay its execution, make it polymorphic, fetch your payload remotely… wait… are we talking about malware here?

The AI secret language, folklore

Mon, 10 Nov 2025 00:00:00 +0000

The story keeps coming back: a “secret language” AIs supposedly invented to talk to each other, faster and better than us, behind our backs. The version that set the networks on fire in early 2025 was called Gibberlink and the legend that stuck to it promised a protocol “80 percent more efficient than human language”. I want to set the record straight calmly, because the demo itself is real, but the story told on top of it is technological folklore.

Nginx on Windows in front of Jenkins: the proxy, the logs and the upgrades

Mon, 08 Sep 2025 00:00:00 +0000

Standing up an nginx on Windows Server to terminate TLS in front of a Jenkins takes five minutes for the version that works in a demo and a real half-day for the version you stand behind in production. The reverse proxy part is not the hard one, it fits in a well-tuned location. What makes the difference is what Windows does not give you and no tutorial mentions: log rotation and an upgrade that actually takes. One word before we start: if the same machine also fronts a Perforce, do not try to run it through the config below. The P4 protocol is not HTTP, it calls for an entirely different approach I cover separately in nginx in front of Perforce .

Clean Code vs Vibe Coding (or how not to end up in the news)

Sun, 15 Jun 2025 00:00:00 +0000

The emergence of large language models (LLMs) in software development raises a fundamental question: are our established clean code practices still relevant against “vibe coding”? And more importantly, how do we avoid ending up on tech news with the next “oops, I exposed 50 million passwords”?

These old habits that save us

Clean code relies on proven principles that, incidentally, kept quite a few developers from technical unemployment:

DRY (Don’t Repeat Yourself) prevents us from duplicating our vulnerabilities across the codebase. SOLID helps us create architectures where one security flaw does not turn the whole system into digital Swiss cheese. KISS (Keep It Simple, Stupid) limits attack surfaces, because complex code often hides unpleasant surprises. YAGNI (You Aren’t Gonna Need It) prevents us from adding useless code that only increases attack surface without adding value.

vim-plugin-manager: turning a decade of bash into a real Vim plugin

Sun, 13 Apr 2025 00:00:00 +0000

The bash script from the 2018 episode did its job for seven years: plugins as git submodules, native pack/ loading, a plugins script for the choreography. What it could never do was live inside the editor: no progress while cloning, no lazy loading wired to commands, no help. In March 2025 I finally gave it the ending it deserved: rewritten from scratch in VimScript as vim-plugin-manager , about 5,600 lines of legacy VimScript.

Which identity runs the service and who can prove it

Sat, 12 Apr 2025 00:00:00 +0000

A Windows service runs as some identity and that identity is the whole security story: it is the blast radius when the service is exploited and it is the name in the logs when you need to know who did what. Most of the trouble I audit comes from one shortcut, a local account with a static password baked into the service configuration, which is both a secret sitting in cleartext and an identity nobody rotates. The fix is not a better password, it is choosing an identity that has no password a human ever holds.

PowerShell execution policy and what it is not

Sat, 08 Mar 2025 00:00:00 +0000

The single most useful sentence about ExecutionPolicy is the one Microsoft puts in its own documentation: it is not a security boundary. It is a seatbelt against double-clicking a script you did not mean to run, nothing more and treating it as a control that keeps an attacker out is a category error that hides the real levers. So the honest way to cover it is to say what it does, prove how little that is and then point at the things that actually constrain what PowerShell can do.

Driving Windows from Ansible without leaking credentials

Wed, 12 Feb 2025 00:00:00 +0000

The question people argue about is SSH versus WinRM and it is the wrong question, because both are now legitimate: Win32-OpenSSH is a first-class Ansible transport and WinRM is the native one. The question that actually decides your exposure is narrower and sharper: when a play authenticates to a domain-joined host, does a reusable credential cross the wire and can the target then act as you against the rest of the domain. Get the authentication protocol and the delegation model right and either transport is fine. Get them wrong and WinRM will hand an attacker your domain the same way a bad SSH setup would.

Hardening WinRM

Wed, 08 Jan 2025 00:00:00 +0000

WinRM is the engine under PowerShell remoting and most remote Windows management, which makes it both indispensable and a tidy path from one shell to a whole estate. Standing it up is one command. Standing it up safely means undoing two of the defaults that command leaves behind, choosing a transport that means something and constraining what a remote session may actually do. The order matters, because the convenient setup is the insecure one.

Hardening Proxmox VE

Wed, 18 Dec 2024 00:00:00 +0000

A Proxmox node is a hypervisor, which means compromising it is not compromising a server, it is compromising every VM and container on it at once. That raises the stakes of the ordinary things, the web interface, the login, the SSH, the firewall, because each is now a door into the whole fleet rather than one machine. The hardening splits cleanly: protect the management plane that controls everything and understand which of the two workload types actually isolates and which is a host-root escape waiting for an excuse.

Bare-metal Kubernetes in HA, without kidding yourself

Tue, 10 Dec 2024 00:00:00 +0000

On a cloud, Kubernetes high availability is bought: the control plane is managed, the load balancer is an API object, storage is a class you reference. On bare metal, each of those three conveniences becomes something you have to build yourself and that is where most tutorials stack seven components to cover three jobs. This post builds an HA cluster with the defensible minimum: a control plane that survives the loss of a node, an API VIP with no external load balancer, a network that also carries the LoadBalancer and storage replicated across the nodes’ own disks. Three pieces covering the three single points of failure and not one more, because a cluster you do not understand is not available, it is on borrowed time.

Kubernetes, or not

Tue, 19 Nov 2024 00:00:00 +0000

Kubernetes solves a real problem, scheduling many workloads across many machines while healing itself and it solves it well. The trap is not the tool, it is reaching for it to solve a problem you do not have yet, paying up front a permanent operational tax for resilience you are not using. Because Kubernetes’ real expense is not conceptual, it is operational: the control plane to keep alive, the CNI to understand, the storage to stand up, the quarterly API updates to follow and an attack surface with its own hardening posts. Before signing up for that, it is worth walking down the floors and stopping at the first one that carries the load, because the right level of abstraction is the lowest that solves your problem, not the most prestigious.

Hardening Kubernetes

Tue, 05 Nov 2024 00:00:00 +0000

Kubernetes multiplies the single-container problem by a control plane, a flat network and an authorization system and each of them ships a default that is convenient rather than safe. The keys to the whole cluster are three: a pod that can reach the host, an RBAC grant that is too broad and a network that lets every pod talk to every other. Harden those three and the exotic attacks lose their first step. Leave one open and the rest of the hardening is decoration, because a privileged pod, a cluster-admin binding or a reachable etcd are already game over.

Hardening Docker Swarm

Tue, 08 Oct 2024 00:00:00 +0000

Everything that applies to single-host Docker hardening still applies to each container in a Swarm, so this note only covers what the cluster adds: a control plane, a trust boundary between manager and worker, a network that spans hosts and a secrets store. The good news and it deserves saying because people look down on Swarm, is that its security defaults are better than assumed. Nodes authenticate each other over mutual TLS with rotating certificates, out of the box and the Raft store that holds the cluster state is encrypted at rest. The gaps are narrower and more precise: one network setting off by default, one operational habit that collapses the trust boundary and the usual temptation to put secrets in the wrong place.

Docker hardening

Wed, 18 Sep 2024 00:00:00 +0000

The sentence that should govern every Docker decision is this: a container shares the host kernel, so a container escape is a kernel escape. A container is not a small VM, it is a process with namespaces and cgroups wrapped around it and a seccomp filter in front of its syscalls and everything below that is the same kernel the host runs. Hardening Docker is therefore mostly about not handing a container the few things that let it reach around the namespaces and talk to that shared kernel directly, and, underneath, about not running the whole apparatus as root in the first place.

mise: one static binary for all the tool versions

Thu, 05 Sep 2024 00:00:00 +0000

Every project drags its collection of versions along: repo A wants Node 20, repo B still stands on Node 18, C runs on Python 3.11 while D already demands 3.12 and terraform, hugo and go follow the same law. Installing those versions machine-wide means crushing one on top of the other: the last one wins and every other repo breaks in silence. I took that problem by the right end in 2024 with mise : a per-project tool version manager held by a single Rust binary.

Hardening DHCP

Wed, 28 Aug 2024 00:00:00 +0000

DHCP hands a booting machine its address, its gateway and its DNS, which is to say it tells every client where to send its traffic and whom to trust to resolve names. The protocol has no authentication: a client takes the first offer it hears. That single fact is the entire threat model. A rogue DHCP server, a laptop with dnsmasq, answers faster than yours and now owns the gateway and the resolver of everyone who believed it and from there reads, redirects and rewrites traffic at will. The defense is not in the DHCP server, it is in the switch, the only thing placed to decide which offers are allowed on the wire.

Hardening DNS infrastructure

Wed, 03 Jul 2024 00:00:00 +0000

DNS is trusted by everything and verified by almost nothing, which is why a resolver you run carelessly becomes a weapon pointed at other people and an authoritative server you run carelessly becomes a map of your estate and a channel out of it. The threats are familiar, cache poisoning, amplification, tunneling, zone disclosure and the defenses are not exotic. They come down to four decisions: who may ask your resolver to recurse, who may pull your zones, whether your answers can be forged and whether you can see data leaving in your own queries.

SSH Security Hardening: Access Control and Monitoring

Wed, 03 Jul 2024 00:00:00 +0000

SSH is the front door of almost every fleet, which is exactly why it is the first thing worth getting right. A loose configuration does not just expose one host, it hands an attacker the three things they want most: a way in, a way across and a way to stay. The point of this note is not to recite a compliance checklist, it is to describe the configuration I actually deploy, the reasons behind each choice and the monitoring that turns “someone logged in” into “this person, from there, at that time”.

Docker Swarm to Kubernetes: why I abandoned Swarm

Sat, 22 Jun 2024 00:00:00 +0000

Docker Swarm works well as a lightweight orchestrator for small-to-medium deployments. It is simple, mature and the entry barrier is low. But it hits structural walls on security and advanced container management. Here are the four.

No rootless mode

Docker rootless mode (dockerd-rootless-setuptool.sh with slirp4netns/RootlessKit) is incompatible with Docker Swarm. The official documentation explicitly lists:

Overlay networks: Swarm relies entirely on overlay (VXLAN) for inter-node communication. Without overlay, services on different nodes cannot reach each other. Swarm becomes a single-node-only tool.
AppArmor: no MAC (Mandatory Access Control) enforcement on containers.
Checkpoint/Restore: no CRIU snapshotting, no live migration, no stateful recovery.

This is a fundamental architectural constraint: Swarm requires kernel-level networking primitives (netns, veth, VXLAN) that rootless mode cannot provide. slirp4netns and RootlessKit operate in user namespaces incompatible with kernel-level overlay networking.

Internal vs external domain separation

Thu, 30 May 2024 00:00:00 +0000

Using one domain name for both the public site and the internal estate feels tidy and it quietly hands an attacker several advantages at once. The argument for splitting them is not aesthetic, it is that a handful of concrete attacks stop working the moment internal services live under a name that never appears in a public certificate or a public DNS answer.

Why a shared domain leaks

A wildcard certificate is the first problem. A *.company.com issued for the public web is, by construction, valid for every internal host under that name too, so the day a single public server is compromised, its certificate impersonates any internal service without a browser blinking. One stolen key and the man in the middle is trusted everywhere.

Hyper-V S2D vs Proxmox Ceph: the comparison that decided it

Fri, 17 May 2024 00:00:00 +0000

The Hyper-V versus Proxmox debate is rarely settled by a single number, because the two platforms do not pay the same price for the same guarantees. Hyper-V with Storage Spaces Direct remains built for a Windows world, whereas Proxmox VE with Ceph aims at the flexibility of a mixed fleet and a zero licensing bill. The real trade-off plays out on two axes, the performance one can expect from storage and the cost of ownership spread over several years.

Egress filtering, the control everyone skips

Fri, 12 Apr 2024 00:00:00 +0000

Almost every firewall I audit is meticulous about what may come in and indifferent about what may go out. That asymmetry is exactly what an attacker counts on, because the valuable moves all happen outbound: the reverse shell dials home, the stolen data leaves, the implant beacons to its C2. An inbound-only firewall is a door with a strong lock and no wall behind it. The point of egress filtering is that a host which has no business talking to the internet should not be able to and a host that does should only reach the few places it genuinely needs.

Inbound traffic: expose less and guard what is left

Wed, 28 Feb 2024 00:00:00 +0000

Every port a machine listens on is a promise to answer a stranger and most incidents start with a promise nobody meant to make: a database bound to 0.0.0.0, an admin panel on a public IP, a forgotten service a scanner found before you did. The internet is scanned end to end continuously, so an exposed service is not a risk that might be noticed, it is one that will be, usually within hours. Ingress hardening is therefore two disciplines: expose as little as possible and put the few things that must be exposed behind controls that assume they will be attacked.

Network segmentation

Mon, 22 Jan 2024 00:00:00 +0000

A flat network is a single room where everyone can reach everyone and it is the condition that turns one compromised laptop into a compromised estate. The intrusion almost never lands where the valuable thing is; it lands on a user machine and then moves, to the file server, the database, the domain controller, because nothing on a flat network says no. Segmentation is the practice of adding walls so that a foothold buys the attacker one room rather than the building and the honest version of it is more than drawing VLANs: it is making the VLANs real with filtering between them and hardening the layer-2 tricks that otherwise walk straight through.

Resource limits: stopping a machine from being talked into dying

Tue, 05 Dec 2023 00:00:00 +0000

The fork bomb is the whole argument in one line:

:(){ :|:& };:

A function that calls itself twice and backgrounds both copies, run once and the process count doubles every fraction of a second until the process table is full, no new process can start and the machine is a brick needing a hard reboot. It is not an exploit, it needs no privilege and any local user can type it by accident. The same shape drives the rest of the family: a loop that allocates memory until the OOM killer fires, one that opens file descriptors until fork fails with “too many open files”, one that fills /tmp or /var/log until the root filesystem wedges. None of them are clever. All of them are stopped by the same thing, a ceiling the user cannot exceed and the only real work is putting the ceiling in the right place.

Filesystem hardening

Wed, 08 Nov 2023 00:00:00 +0000

Most of filesystem hardening is deciding, per mount, which of three powers a directory is allowed to grant: running a binary, honouring its setuid bit and letting it be a device node. The default, defaults, grants all three everywhere, which is convenient and wrong for the directories users and daemons can write to. The rest is separating partitions so one cannot starve the others, turning on the kernel’s link protections and keeping an eye on which files carry setuid. None of it is exotic and the payoff is that whole categories of local escalation stop having a place to land.

Recompiling your kernel: bespoke kernels for your own infra

Fri, 27 Oct 2023 00:00:00 +0000

Hardening the OS is mostly subtraction, removing what listens and cutting the kernel surfaces you do not need, which I detailed in hardening the OS and the Linux kernel . There is one step that article did not climb and it is my favorite: not disabling a kernel feature at runtime, but never compiling it at all. Recompiling your own kernel is subtraction taken to its logical conclusion.

Why recompile, exactly

A distribution’s default is a kernel for everyone, so a kernel that ships just about everything: hundreds of drivers, exotic filesystems, whole networking subsystems, debug options, because it has to boot a student’s laptop as well as a storage array. On an infra, the hardware is known and stable and all that baggage becomes free attack surface. A subsystem that is not compiled has no CVE for you, no module to load, no bug to exploit; it simply does not exist in the binary. Compiling the kernel for your fleet is therefore three wins at once: a surface reduced to what you actually use, a smaller and often faster kernel because it carries less and a control that changes everything the day you have to react.

Linux OS and kernel hardening

Fri, 20 Oct 2023 00:00:00 +0000

A freshly installed Linux box is built to be convenient, which means it ships with services you did not ask for, kernel features you will never use and defaults chosen so that nothing surprises a beginner. Hardening it is mostly subtraction: remove what is listening, switch off the kernel surfaces you do not need, tighten the handful of sysctl knobs that matter and lock down the accounts and the module loader. The individual steps are small and well known. The discipline is doing them deliberately rather than inheriting whatever the installer decided and keeping the box patched, because the kernel exploits that matter (PwnKit, Dirty Pipe and their successors) are patched long before most machines apply the fix.

Sudo and LOLBins

Sat, 15 Jul 2023 00:00:00 +0000

The comfortable belief about sudo is that a narrow rule is a safe rule, that NOPASSWD: /usr/bin/find grants the ability to run find and nothing more. GTFOBins demolished that belief and the honest version of a sudo hardening note has to start there: the granularity of sudoers limits nothing the moment the binary you allowed can be talked into spawning a shell and a startling fraction of the standard toolbox can. The question is never “which command did I grant”, it is “what can that command be made to do once it runs as root”.

Scale-to-zero: Sablier wakes Swarm and Kubernetes on demand

Sun, 15 Jan 2023 00:00:00 +0000

A service you consult once a week runs one hundred percent of the time. You installed it for a single need, a QA you open on Monday, a team wiki, an admin tool you only reach for during an incident. It permanently holds its share of RAM, its share of CPU, its ports, its slot in monitoring and its stream of security updates. Shutting it down settles all of it at once, except that you then have to bring it back by hand the moment you need it and nobody remembers the command. Scale-to-zero is the idea of starting a service only when someone asks for it and stopping it when nobody uses it anymore, without the user having to know it was asleep. Sablier is a small API server you place next to the reverse proxy: the proxy intercepts each request, calls the API to know whether the target is asleep, wakes it if needed and shows a waiting page while it starts. Once the target is ready the request goes through and a session opens, closed after sessionDuration, at which point Sablier shuts the workload down.

The Bastion and Guacamole: the two leaves of the front door

Sun, 20 Nov 2022 00:00:00 +0000

There are two populations in every infrastructure and they ask the same question of different tools. Shell people ask: “how do I reach the server and who can prove it afterwards?” Desktop people ask: “how do I reach that Windows box without a VPN and without installing a client?” There is a remarkable tool on each side and both deserve better than a footnote: The Bastion from OVHcloud for the first population, Apache Guacamole for the second. Together they are the two leaves of the same front door: one keeps the ledger of hands on keyboards, the other of hands on mice.

The :ro flag on the Docker socket protects nothing

Sun, 12 Jun 2022 00:00:00 +0000

In every code review where someone has had the misfortune of mounting /var/run/docker.sock into a container, there is a small ritual consisting of pointedly noting that one could at least make it read-only, upon which everyone nods, adds :ro to the compose file and closes the merge request with the feeling of duty done. It is theater and I would like to explain precisely why, because the mistake is instructive well beyond Docker.

FluoLight: rewriting an inherited Arduino firmware into something defensible

Sun, 24 Apr 2022 00:00:00 +0000

Second episode of the FluoBoard series, after the autopsy of the original board . I promised a follow-up there and here it is, though it did not happen in one night. From 2021 I went at the most glaring gap first: splitting the monolith into several .ino files by responsibility, because a single sketch does not stay maintainable, fixing bugs along the way and adding the #defines that were missing. Then came a hand-written event loop, my first way of pacing actions over time, before I reached for the TimedAction library to do it properly. The bulk of the work was done in 2021-2022, with final touches coming later [edit: 2025], all of it followable commit by commit. Same board family, same /lights/<mac> protocol whose answer fits in a single digit between <...>. What changes is not the feature list, it is that the loop never stops to wait.

One Ansible image for GitLab CI, built right from a Paris basement

Tue, 22 Mar 2022 00:00:00 +0000

Every CI runner that runs Ansible has the same problem: the job needs ansible-core, the collections for your platforms, git, ssh and it needs all of it now, on a machine that will be thrown away after the job. The official images exist, but they drift: latest changes under you, the version you pinned gets rebase-rolled and installing eight Galaxy collections per job burns minutes on every pipeline.

So I wrote my own, alpine/ansible, an Ansible CI toolkit image for my GitLab runners. And it was not born yesterday: I put it together for myself around 2016 and in 2018 it replaced, at a Paris employer, a CI that dumbly reinstalled Ansible on every run. In 2022 I am finally making it public and taking the chance to reread its history, early sins included.

Autopsy of an inherited Arduino firmware

Mon, 22 Nov 2021 00:00:00 +0000

Back at a former employer, the original firmware drove little LED strips bolted outside meeting rooms, showing the room’s availability. The board polls GET /lights/<mac> every few seconds, the server answers a single ASCII digit between <0> and <9>, the strip shows a color. Simple product, simple protocol. I did not write it, but I used it as a base.

The board was only half of the system. The other half, on the meeting-room side, was the building’s occupancy server. I, on my own, repurposed that same protocol, for myself alone, with a small Python server bridging my CI pipelines to the board: my CI status light existed in 2019, long before “CI status lights” were a thing. Only the API contract survived, in the docs (GET /lights/<lightid> answering <N>, from <0> rainbow error to <4> red booked-and-busy); the server code itself did not. That is its own lesson: code without a repository does not outlive the job that wrote it.

SSH in 2021: what thirteen years changed since the 2008 series

Fri, 04 Jun 2021 00:00:00 +0000

The SSH series around here ended in 2013: key authentication in 2008 , multiple authentication in 2012 , connection restriction and PKCS#8 in 2013 as the final act. Eight years later, everything those articles did with craftsmanship exists as primitives. Here is the 2021 memo, complete.

The keys: Ed25519 and the FIDO2 couple

ssh-keygen -t ed25519 -a 100 -C "logout@2021"
ssh-keygen -t ed25519-sk -a 100 # hardware key, touch required
ssh-add -K # downloads resident keys from the YubiKey

Ed25519 for everything software: shorter, faster, more resilient and -a 100 hardens the KDF against passphrase brute-forcing (the direct heir of the 2013 PKCS#8 article , which was right ahead of its time). The -sk keys (OpenSSH 8.2, 2020) plug in a YubiKey: the private key never leaves the hardware, the protocol requires a physical touch. And RSA? The sun is setting: OpenSSH 8.2 disabled ssh-rsa (SHA-1) in CASignatureAlgorithms by default and the full retirement announcement is already in the release notes. In 2008, RSA 2048 was the luxury; in 2021, it is technical debt.

PwnMachine: the self-hosted bug bounty workstation

Sun, 12 Jul 2020 00:00:00 +0000

When you do bug bounty, you need an environment: a DNS you control (for callbacks), a reverse proxy (to expose tools) and a web interface to orchestrate. All on a machine you can reinstall in ten minutes. PwnMachine is that setup, packaged in Docker Compose.

The concept

PwnMachine is a self-hosting solution based on Docker aiming to provide an easy-to-use pwning station for bug hunters. The basic install includes a web interface, a DNS server and a reverse proxy. One docker-compose up --build -d and it stands up.

Counting CVEs tells you nothing about security

Wed, 11 Mar 2020 00:00:00 +0000

You regularly see comparisons that rank systems by the number of published CVEs, the lowest count winning the title of most secure. I tend to strongly temper that kind of result, because the number itself does not measure what people think it measures. A CVE counter counts flaws that were made public and nothing else. It says nothing about the ones that exist, that are known internally, or that quietly circulate far from any advisory.

Docker rootless: the daemon without privileges

Tue, 23 Jul 2019 00:00:00 +0000

Docker runs as root by default. The daemon (dockerd) has access to ALL the filesystem, ALL the network, ALL the processes. A container that escapes the daemon gets root on the host. Rootless mode moves the daemon to a normal user: containers run in a user namespace and an escape only yields that user’s rights, not root.

The problem

A Docker daemon running as root is a service with all the rights on the machine, accessible via a Unix socket (and sometimes a TCP port). Every container mounting a volume mounts with the daemon’s root rights. Every bug in runc or containerd is a root escape.

3factor: the architecture that removes state from the API

Fri, 15 Feb 2019 00:00:00 +0000

In early 2019, a few months after discovering the twelve factors, I come across 3factor , an architecture pattern published by Hasura whose name is an assumed wink at 12factor.net (the resemblance ends there, 3factor is a design pattern, not a methodology). The idea fits in three factors: realtime GraphQL, reliable eventing and async serverless.

The three factors

The first factor is realtime GraphQL: the client subscribes to the state and receives it instead of asking for it, which moves the synchronisation from polling to the server. The second is reliable eventing: every state change is a persisted event. Triggers triggers deliver it to subscribers (delivery happens at least once, idempotency is the consumer’s job). The third is async serverless: the business logic runs in functions triggered by events, with no server to manage and automatic scaling.

GitOps, the word after the practice: gitflow, GitLab CI, Terraform and Ansible

Tue, 20 Nov 2018 00:00:00 +0000

The word has been spreading for a year, GitOps, laid down by Weaveworks in 2017 around Kubernetes and a simple idea, git as the source of truth of the infrastructure and a loop that converges reality toward what the repository describes. I read it the way I had read DevOps a few years earlier, not discovering a method but recognizing my own (DevOps, a word for what I was already tinkering with ), back then with git and bash hooks. By 2018 the hack had grown and it took the shape of an adapted gitflow, conventions and a pipeline that does all the work.

Portknocking and SSH tunneling: the magic tricks before WireGuard

Sat, 06 Oct 2018 00:00:00 +0000

In 2008, this blog published Portknocking: the art of knocking on the door, then automatic combination generation , with SSH tunneling on the neighboring shelf. Fun, pedagogical and genuinely deployed: I saw it in production until 2018 at a hosting provider. In 2016, Jason Donenfeld publishes WireGuard and the question changes nature: not “how to hide the door”, but “why is there a door?”.

The 2009 setup, in fond memory

For latecomers, what the era’s deluxe portknocking involved:

Chaos engineering: breaking things for real, but not just any way

Mon, 20 Aug 2018 00:00:00 +0000

In 2018, the same year I discover the twelve factors, I come across chaos engineering (I would be quite unable to find the exact blog posts that lead me there. That is a good reason to go back to the sources rather than my memory): the definition and the principles fit in a single document, which says better what I end up practising than my own rewording.

It is not breaking things

The most common misreading is to believe that restarting a machine in production is enough to do chaos engineering, which confuses a method and an incident: breaking things at random is an incident we failed to avoid, not an experiment. Chaos engineering is a discipline of experimentation, with a defined and measured steady state, a hypothesis about its persistence, a realistic perturbation injected and a rigorous comparison (without a measurable steady state there is no experiment, only an outage).

Git submodules by hand: my Vim 8 plugins without a manager

Mon, 07 May 2018 00:00:00 +0000

Debian and Ubuntu had finally shipped Vim 8 and with it the native package system: pack/plugins/start and pack/plugins/opt. Goodbye pathogen, goodbye my apt-flavored hack (vim-apthogen , episode one of this series). But Vim 8 packages solve loading, not management: nothing tells you which plugins your config depends on, or how to clone them on a fresh machine.

The answer I chose is the same one I still run today: the config repo is the state. The plugins live as git submodules of the config repo, in pack/plugins/start/ for always-on ones and pack/plugins/opt/ for the opt-in ones. Clone with --recursive and the whole editor materializes.

Twelve-Factor App: the Heroku method that survived containers

Mon, 15 Jan 2018 00:00:00 +0000

In January 2018, I come across the Twelve-Factor method (and I immediately file it among the practices I already did without naming it): three readings that complete each other (the reference site by Adam Wiggins , the adaptation to microservices by Ben Horowitz on the nginx blog and the report Kevin Hoffman draws from it at O’Reilly ). Suddenly a word for what I had been doing for years. Before, I already knew (and more or less applied) KISS, YAGNI and DRY, then SOLID; the V-model and AGILE were part of the landscape, each with its flaws that I could enumerate without needing anyone to explain them to me. Twelve-Factor, for its part, is not for the general public: it is an engineer’s checklist, written for Heroku, that says in black and white what an application must guarantee to survive production and its own redeployments.

SSH Hardening: the 2017 Trilogy Reread in 2026

Thu, 26 Oct 2017 00:00:00 +0000

In October 2017, an SSH hardening article goes by on my timeline. Generic analysis, naive countermeasures. I answer in three comments: disabling root, changing the port, key authentication. Nine years later, those three comments serve me as an inventory: what held, what changed and what was out of scope but deserved a mention.

First, the principle that made me react: comparing OpenSSH to a simple administration interface is reductive. SSH is the bastions to your IS, the SSH VPNs, the agentless orchestration (Ansible, Puppet Bolt) that runs on top of it. It is the administrative spine of most Unix fleets. Harden it badly and you harden the whole IS badly.

Ansible before Ansible: ssh, a CSV and a template engine

Mon, 12 Sep 2016 00:00:00 +0000

In 2006-2007, I administered a small fleet with scripts like everyone else, except I believed (youth!) that my contraption was an architecture. It had three components: a host list, a CSV and bash. Ansible did not exist yet (it was born in 2012), but the shape of the problem was already there and mine modeled it at 80%.

The inventory: a list + a CSV

The list of machines: a flat file, one machine per line. On top of it, a CSV of inventory variables:

SSH: ProxyJump and modern multi-hop

Mon, 12 Sep 2016 00:00:00 +0000

A few years ago, to reach a machine behind a bastion, I wrote a ProxyCommand with nc and I told that story in SSH: tunneling . OpenSSH 7.3 made all of that obsolete with a single option, -J and it is one of those small additions that change a gesture you make ten times a day.

The basic gesture

ssh -J user@bastion user@target

bastion acts as the jump host, the target is never exposed directly and above all the encryption stays end to end between your machine and target. The bastion only sees encrypted traffic it cannot read, exactly as with the old ProxyCommand, but without invoking nc or opening an intermediate shell. The difference with ProxyCommand is not merely cosmetic: -J establishes the connection to the next hop from the client, not from the bastion, so no private key transits or sits on the jump host. That is what we wanted from the start and what we used to hack around with agent forwarding, which posed precisely the opposite problem by exposing the agent on the bastion.

apt for Vim: the story of vim-apthogen (2012-2016)

Sun, 17 Apr 2016 00:00:00 +0000

Somewhere between 2012 and 2016, my Vim setup looked like a Debian system from the package manager’s point of view: a pile of bundle/ directories installed by hand and vim-pathogen doing the loading. Pathogen is honest about its role: it is dpkg, the dumb loader. Nobody had written the apt on top of it. So I did.

The result: vim-apthogen , published on vim.org on April 12, 2016 (Vim 7.0, 1136 downloads, a karma of 19/6, all numbers I am unreasonably proud of). The README line I still like: Like apt for dpkg, vim-apthogen is a frontend to vim-pathogen.

A DayZ autowalker in Java: decompile, patch for AZERTY, fork afterwards

Sat, 18 Oct 2014 00:00:00 +0000

Back in the DayZ days, I wanted an autowalker, that little utility that holds the forward key for you while your character crosses a huge map. One existed, in Java, but with no published sources and it was built for a QWERTY keyboard. I play on AZERTY, so it walked sideways and I also wanted it to handle the running state cleanly rather than jam a key. I did not overthink it: I decompiled it.

SSH: hardening your keys with PKCS#8

Sun, 23 Jun 2013 00:00:00 +0000

In two or three commands:

$ mv ~/.ssh/id_rsa ~/.ssh/id_rsa.old
$ openssl pkcs8 -topk8 -v2 des3 -in ~/.ssh/id_rsa.old -out ~/.ssh/id_rsa
$ chmod 600 ~/.ssh/id_rsa
# Check that the converted key works; if yes, delete the old one:
$ rm ~/.ssh/id_rsa.old

And a small script to do it on the fly:

#!/bin/bash
: << 'TITLE.sh'
 Version: 0.1
 Authors: nobody@vhalholl.info
 Description: hardening ssh private key w/ pkcs#8
 http://martin.kleppmann.com/2013/05/24/improving-security-of-ssh-private-keys.html
 Dependencies: openssl
TITLE.sh

DEP=$(which openssl);[ ! -z $DEP ]&& {
 for k in $(echo $(for i in $(ls *);do file $i; done |grep private |cut -d : -f1));do
 echo "backup $k"; mv $k .$k.old
 echo "convert $k"; openssl pkcs8 -topk8 -v2 des3 -in .$k.old -out $k; chmod 600 $k
 done;
 exit 0;
}
echo "Openssl not found" && exit 1

PKCS#8 wraps the RSA private key in a 3DES encryption layer with a hardened KDF (PBKDF2). The key’s passphrase becomes more expensive to brute-force: every decryption attempt requires extra KDF iterations. Same principle as bcrypt or scrypt for passwords, applied to SSH keys.

The FOG client: python 2, cron every five minutes and a fleet that rebooted itself into place

Tue, 20 Nov 2012 00:00:00 +0000

Before this blog’s other archaeology pieces, there was this one. Years before the apthogen era , under the gobel1 identity, I ran workstation fleets cloned with FOG (Free Opensource Ghost). FOG’s client story was Windows-first, so the Linux side needed a small client of its own: enter FOG.py, version 0.2, python 2, about 150 lines plus a handful of shell hooks. It predates 2013; the git repo was only re-inited in May 2013, one commit in French, six commits total. The archaeology is genuine.

SSH: restricting a connection via private/public keys

Mon, 05 Mar 2012 00:00:00 +0000

What is more natural when running automated tasks between multiple servers than using an SSH key pair without a passphrase?

Yes, but how do you confine this unprotected key to executing one specific task?

You just need to prefix the public key you want to limit in the file:

.ssh/authorized_keys

With the options detailed in the sshd manual, section AUTHORIZED_KEYS FILE FORMAT:

command="/usr/bin/rsync --server --daemon .",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAA...

Each option blocks an escape path:

SSH: multiple authentication with private/public keys

Mon, 06 Feb 2012 00:00:00 +0000

As a complement to an older article on SSH key authentication and because you should never put all your eggs in one basket, here is a simple solution to use a specific key for a given identity on a given SSH server.

In our example we will use two keys: one with a passphrase for connecting to server.mydomain.tld by simply typing ssh serv and one without a passphrase mainly for running maintenance scripts on a small fleet of client machines cli01.parc01.mydomain.tld etc., whose SSH server listens on a specific port and the firewall only allows connections from server.mydomain.tld (let’s be crazy).

Debsecan: Debian Security Analyzer

Wed, 26 Oct 2011 00:00:00 +0000

Debian Security Analyzer is a very practical tool, discovered recently.

It simply generates a list of vulnerabilities affecting a given installation, making it complementary to apt-listbugs.

By default at installation, daily email reports are enabled for a generic installation.

This behavior is modified via /etc/default/debsecan:

# Configuration file for debsecan. Contents of this file should
# adhere to the KEY=VALUE shell syntax. This file may be edited by
# debsecan's scripts, but your modifications are preserved.

# If true, enable daily reports, sent by email.
REPORT=true

# For better reporting, specify the correct suite here, using the code
# name (that is, "sid" instead of "unstable").
#
# Available:
# GENERIC
# etch
# lenny
# sid
# squeeze
# wheezy
SUITE=GENERIC

# Mail address to which reports are sent.
MAILTO=root

# Source URL
# eg: http://secure-testing.debian.net/debian-secure-testing/project/debsecan/release/1/$SUITE
SOURCE=

The SUITE parameter is the most important: GENERIC lists all CVEs for all versions, including ones that do not affect you. Setting it to your exact suite (lsb_release -c), debsecan filters by version and the list becomes actionable.

Dennis Ritchie has left #world

Sat, 22 Oct 2011 00:00:00 +0000

Dennis Ritchie (above right) aka DMR, creator of the C programming language and co-developer of the Unix operating system, left us yesterday at the age of 70.

Personally, I learned C from “his” book “The C Programming Language” and I still use K&R indentation, although slightly remixed GNU-style.

It is a disappearance that will have practically non-existent media coverage, unlike the death of Steve Jobs which occurred the previous week.

(Re)Opening

Thu, 13 Oct 2011 00:00:00 +0000

After two years of inactivity, about fifty articles published and a dozen drafts in progress, this blog opens its doors again.

Still powered by WordPress, only the images, the comments and the original theme of the blog have changed.

For now I am working on correcting the articles already published, then I will handle the publication of the drafts in progress, which will be backdated to follow the blog’s chronology.

Chat: from irssi + messenger gateway to weechat, the thin-client philosophy

Mon, 20 Jun 2011 00:00:00 +0000

The 2009 irssi article ended with an edit: “An up-to-date configuration is available here on github” (it was an article stub, the real value lived in the repo, time proved it). Time to tell what the article did not tell: the setup that made irssi indispensable and why two years later I moved everything to weechat without changing the philosophy.

irssi + BitlBee: the bridge to the messengers

The era’s setup: a local BitlBee server that presents instant-messaging networks (MSN, AIM, ICQ, Jabber) as an ordinary IRC channel and irssi connecting to it like any server. Result: one client, one buffer, one log config, for IRC and for the MSN contacts who only know IRC by name.

tmux: the multiplexer that took the chair

Sun, 23 May 2010 00:00:00 +0000

Screen is twenty-three years old, byobu dressed its flaws and there remain irritants neither of them fixes: no true panes (Screen does regions, which resize badly), a retrofitted UTF-8, a $HOME/.screenrc config with bindkeys inherited from the 80s. That is where tmux enters, born in 2007 on OpenBSD and starting to take the chair in 2010.

The client-server model

The architectural difference that changes everything: Screen runs a server per user (sessions share one process), tmux runs one server per session and clients attach to it. Practical consequences:

The R.A.C.H.E method: Rapid Application Conception and Heuristic Extreme-programming

Thu, 15 Apr 2010 00:00:00 +0000

There is a software engineering method that describes real life better than half the frameworks I see pushed through a review board. That method has an acronym, an institute, a FAQ and a public-domain licence: it is called the R.A.C.H.E method. I discover it around 2009-2010, at a time when I still take roadmaps seriously.

R.A.C.H.E stands for Rapid Application Conception and Heuristic Extreme-programming. The site of the International Institute of La RACHE has existed since 2007, but the concept already circulated as a PDF before that. The recurrent metempsychosis of the concept is, according to its own promoters, intrinsically axiomatic at some large vendors. la-rache.com

DevOps, a word for what I was already tinkering with

Mon, 15 Mar 2010 00:00:00 +0000

For a few months now a word has been going around, DevOps, launched after the Velocity conference of June 2009 and its famous “10 deploys a day” talk on dev and ops cooperation at Flickr, then picked up at the first DevOpsDays in Ghent in October, around Patrick Debois. It is presented almost everywhere as the end of the wall between those who write the code and those who run it and reading the first texts I had the odd feeling of recognising things I was already tinkering with in my corner, without ever having given them a name.

Puppet: my first modules, or moving from the ssh loop to declaring

Sat, 02 Jan 2010 00:00:00 +0000

My bash scripts, a host list, a CSV of variables and an ssh loop pushing template-rendered configs have served me well since 2007, but they describe steps and that is precisely what I wanted to leave behind when I wrote my first Puppet modules for the fleet I run at work. Puppet is a real DSL, a real agent/master model (0.25 at the moment) and a promise my scripts never kept: declare the desired state, not the steps. Ssh, ntp, postfix, munin, zabbix, varnish, rsyslog: these modules cover the common base and I extend them as needs come up.

Haproxy: the reverse proxy that took over from Pound

Sun, 20 Dec 2009 00:00:00 +0000

The blog covered Pound in 2008 : a very light reverse proxy that terminates TLS and balances to a pool. Pound does that job with minimal elegance, but as soon as you need to route on content (a Host to a cluster, a URL to a specific backend, a sticky session), one step up: haproxy 1.3, by Willy Tarreau, stabilizing toward 1.4.

The block that says it all

frontend web
 bind *:80
 mode http
 acl is_app hdr(host) -i app.domain.tld
 acl is_static path_beg /static/
 use_backend app if is_app
 use_backend statics if is_static
 default_backend app

backend app
 mode http
 balance roundrobin
 option httpchk HEAD /health
 server web1 192.168.10.11:8080 check inter 5s fall 2 rise 2
 server web2 192.168.10.12:8080 check inter 5s fall 2 rise 2

backend statics
 mode http
 balance source
 server s1 192.168.10.21:80

tcp mode and http mode: in TCP, haproxy is a brute balancer (TLS pass-through, any protocol); in HTTP, it parses the request and routes on its content. The ACLs do the rest: hdr(host), path_beg and an operator grammar that keeps the config readable, which Pound did not offer.

IRSSI: IRC with the works

Sat, 07 Nov 2009 00:00:00 +0000

[edit: MàJ 20130621]: An up-to-date configuration is available here on github.

Irssi is the console-mode IRC client. It does not do pretty widgets, it does not do drag-and-drop, it does IRC. And it does it well: themes, Perl scripts, aliases, hotlists, notifications, everything is configurable.

The config lives in ~/.irssi/, with a main config file and a scripts/ directory for Perl plugins. The era config (367 lines, multi-network: recycled-irc, otaku-irc, worldnet SSL:7000, fansub-irc, OFTC SSL:9999) is versioned in the dotfiles, with a green.theme and five personal scripts.

Byobu: screen with a coat of polish (and a status bar that saves)

Thu, 15 Oct 2009 00:00:00 +0000

Back in April, this blog described Screen as a terminal multiplexer : detach, reattach, never lose an ssh session. Six months later, byobu arrives and asks an impertinent question: if everyone configures the same Screen the same way (status, colors, shortcuts), why not ship the configuration ready-made?

That is exactly what byobu is: born in Ubuntu as screen-profiles, renamed byobu in 2009 (the Japanese word for decorative folding screens, an assumed wink), the tool wraps your existing Screen in a coat of polish without changing the engine.

Bash: Geolocation, the trilogy

Fri, 02 Oct 2009 00:00:00 +0000

Three iterations of the same script, between December 2008 and October 2009, each one born from the previous one and its limits. V1 is lost with the hard drive (a Samsung 500 GB, with all my Bash, C, PHP and Perl projects, my personal music, the thing was 90% full!), v2 and v3 survive. It is the story of a script’s evolution in public.

The revenge of netcat (v2, February 2009)

After an IRC discussion on geolocation, a friend who used the same site passed me one of his scripts that in a single line did exactly what mine did but much faster thanks to netcat (no disk access). Unfortunately, following the loss of my hard drive, I lost a bunch of tools, including that famous script. So I rewrote the script, being significantly less good than my friend, it takes me more than one line.

NTP: time synchronization over the network

Tue, 26 May 2009 00:00:00 +0000

NTP (Network Time Protocol) synchronizes machine clocks over the network. It sounds trivial until the day a script that depends on the date drifts by 3 seconds and misses its connection window, or two servers that must agree on “now” have a 4-minute gap.

Installation

apt-get install ntp

The ntpd daemon starts and synchronizes automatically with the Debian pool servers (0.debian.pool.ntp.org through 3.debian.pool.ntp.org).

The minimal config

# /etc/ntp.conf
server 0.debian.pool.ntp.org iburst
server 1.debian.pool.ntp.org iburst
server 2.debian.pool.ntp.org iburst
server 3.debian.pool.ntp.org iburst

# Restrict access
restrict default kod notrap nomodify nopeer noquery
restrict 127.0.0.1
restrict ::1

The iburst option accelerates initial sync (4 packets instead of 1 at startup). The restrict lines block access from the network: the NTP server only serves itself, not others (unless you want to run a local NTP server).

Portknocking: automatic combination generation

Fri, 22 May 2009 00:00:00 +0000

Classic portknocking knocks on the door with a fixed port sequence. The problem with a fixed sequence: it gets sniffed, it gets replayed. The solution: make the sequence depend on the date, so it changes every day.

The principle

The combination is computed from the system date. The day determines the protocol (even = udp, odd = tcp) and the ports start from the day+month+year sum doubled, to which 64 is added until past 1024. The client and the server must have the same clock, hence the NTP synchronization.

Bash: customizing the .bashrc file

Tue, 14 Apr 2009 00:00:00 +0000

On our favorite systems, the interactive bash shell, its prompt and its coloring, are configured in these files:

/etc/profile
/etc/bash.bashrc
/root/.bashrc
/home/user/.bashrc

Here we only touch the last three. We start with the color of the username in the prompt, green for users, red for the root account, because a root prompt that does not announce itself is a root prompt you eventually type something stupid into:

# Set color of username in prompt
if [ "`id -u`" -eq 0 ];then prompt_color='\[\033[01;31m\]';else prompt_color='\[\033[01;32m\]';fi

Here is the list of color codes: Black 00;30, Red 00;31, Green 00;32, Brown 00;33, Blue 00;34, Purple 00;35, Cyan 00;36, Light gray 00;37, Gray 01;30, Pink red 01;31, Light green 01;32, Light brown 01;33, Light blue 01;34, Light purple 01;35, Light cyan 01;36, White 01;37.

Screen: a terminal multiplexer

Mon, 13 Apr 2009 00:00:00 +0000

Screen is a terminal multiplexer that, like a window manager, allows multiple text-mode programs to run simultaneously from a single interface, typically a local or remote terminal via SSH (no, nothing to do with nohup, which I mainly use for certain scripts with wget).

Installation

On Gentoo:

logout@lifthrasir ~ $ su -c "emerge app-misc/screen"
Password:
[...]

On Debian:

logout@helghardhr:~$ sudo apt-get install screen
Password for user logout:
[...]

Let’s practice

We will create a screen session to take control of a switch:

Apache: URL rewriting

Sat, 14 Feb 2009 00:00:00 +0000

mod_rewrite is the Apache module that rewrites URLs on the fly. It transforms /article.php?id=42 into /article/42, redirects old links to new ones and does everything a reverse proxy does not: it modifies the request before it reaches the application.

The basic

RewriteEngine On
RewriteRule ^article/([0-9]+)$ article.php?id=$1 [L]

This rule transforms /article/42 into article.php?id=42 internally (the user sees /article/42, Apache runs article.php?id=42). The [L] means “last rule”: if this rule matches, the following ones are not evaluated.

SSH: secure rsync

Sat, 27 Dec 2008 00:00:00 +0000

Rsync is the incremental backup tool: it compares source and destination, transfers only deltas and preserves permissions, timestamps, symlinks. Over SSH, it traverses the network on the encrypted channel, without an rsync daemon on the server side (SSH does the transport).

The basic rsync

rsync -avz /data/ user@server:/backup/data/

-a : archive mode (recursive, preserves perms, times, symlinks, devices)
-v : verbose (lists transferred files)
-z : compression (gzip before transfer)

The / after /data/ means: “the contents of /data”, not “/data itself”. Without the /, rsync creates /backup/data/data/. This is the most common rsync mistake.

htop: top's replacement

Thu, 25 Dec 2008 00:00:00 +0000

top has existed since 1984 and it is ugly. The interface is rigid, the keys are obscure, the tree view does not exist and sorting by memory requires reading the man page. htop does everything top does, better and it fits in a terminal.

What htop brings

Horizontal and vertical scrolling: process lines are no longer truncated;
Tree view (F5): who spawned whom, the parent/child in visual form, threads indented under their parent process;
Interactive sort (F6): by CPU, by memory, by PID, by USER, without restarting;
Interactive kill (F9): select a process, choose the signal, confirm. No kill -9 12345 typed in another terminal;
CPU gauges per core at the top of the screen, in color, in real time;
The filter (F4): type a pattern and only matching processes are visible;
The setup (F2): choose columns, colors, gauges and save to ~/.config/htop/htoprc.

The keys that matter

F1 Help
F2 Setup
F3 Search (incremental)
F4 Filter
F5 Tree view
F6 Sort
F7 Nice - (raise priority)
F8 Nice + (lower priority)
F9 Kill (send a signal)
F10 Quit

F9 is the most used: select a process eating all the CPU, F9, SIGTERM (15) and if that is not enough, F9 SIGKILL (9). All in the same screen, without leaving htop.

SSH: SSH filesystem

Thu, 25 Dec 2008 00:00:00 +0000

SSHFS (SSH Filesystem) mounts a remote filesystem in your local tree via SSH and FUSE. No NFS, no SMB, no server configuration: just SSH (which the server already has) and FUSE (which the client has). The remote filesystem appears as a local directory, with all standard tools (cp, mv, find, grep, vim).

The mount

mkdir /mnt/remote
sshfs user@server:/home/user /mnt/remote

After this command, /mnt/remote IS /home/user on server. All standard tools work:

tsocks: let's proxy

Tue, 23 Dec 2008 00:00:00 +0000

The SSH tunnel gives you a SOCKS (the tunneling article ), but ssh -D only proxies what speaks SOCKS: browsers, curl with --proxy. Programs that do not know SOCKS (apt, telnet, the MySQL client) stay direct. tsocks solves that with LD_PRELOAD.

The principle

tsocks is a shared library that intercepts network system calls (connect(), send(), recv()) via LD_PRELOAD. When a program tries to open a TCP connection, tsocks redirects it through the SOCKS proxy on its behalf. The program does not know it is going through a proxy.

Pound: a very light reverse proxy

Sun, 21 Dec 2008 00:00:00 +0000

Pound is a reverse proxy that does three things: it terminates TLS, it checks that the backends are alive and it balances. That is all. And it is exactly what you need when you do not want an Apache that also proxies.

The config

ListenHTTP
 Address 0.0.0.0
 Port 80
End

ListenHTTPS
 Address 0.0.0.0
 Port 443
 Cert "/etc/pound/server.pem"
End

Service
 BackEnd
 Address 192.168.1.11
 Port 8080
 End
 BackEnd
 Address 192.168.1.12
 Port 8080
 End
 Session
 Type IP
 TTL 300
 End
End

The Pound config file is a declarative format that reads like a text block: each BackEnd is a pool server, each Service is a group of backends, the Session sticks clients by source IP. TLS terminates in Pound, the backends speak plain HTTP on the internal network.

Polipo: an ultra-simple proxy

Wed, 17 Dec 2008 00:00:00 +0000

Polipo is a web cache proxy written by Juliusz Chroboczek, light (one process, a few MB) and perfect in one specific case: behind an SSH tunnel. The SSH tunnel gives you a SOCKS and the browser needs an HTTP proxy: Polipo bridges the gap.

The minimal config

proxyAddress = "127.0.0.1"
proxyPort = 8123
cacheIsShared = false
diskCacheRoot = ""

The proxy listens on localhost:8123, no disk cache (the browser cache suffices) and bridges between HTTP and the tunnel’s SOCKS. The config is 4 lines.

Subversion: installation and usage

Sat, 13 Dec 2008 00:00:00 +0000

After CVS (which deserves its retirement), Subversion. The centralized VCS that bridged the gap between the commit-lock of CVS and the distributed model that would come after. The principle is simple: a central server (svnserve or Apache + mod_dav_svn), clients that checkout, commit, update. One history, one source of truth and a global revision number that increments on every commit.

Installation

apt-get install subversion subversion-tools
# Create the repository
svnadmin create /srv/svn/myproject
# Start the standalone server
svnserve -d -r /srv/svn

The svnserve mode is the simplest: no Apache, no mod_dav, a daemon listening on port 3690. For web access, you need Apache + mod_dav_svn (see the next article).

Bash: IP address obfuscation

Thu, 11 Dec 2008 00:00:00 +0000

An IP address has one “natural” representation (dotted quad: 192.168.1.1), but it has several numeric representations that browsers, SSH clients and log parsers also accept. An IP can be written in pure decimal (3232235777), hexadecimal (0xC0A80101), octal, or mixed (192.168.0x1.1). Logs filtering on “192.168.1.1” do not filter on “3232235777” and that is the obfuscation hole.

The representations

# Pure decimal (uint32)
printf "%d\n" 3232235777
# 3232235777 = 192.168.1.1

# Hexadecimal
printf "0x%X\n" 3232235777
# 0xC0A80101

# Octal
printf "%o\n" 3232235777
# 30052000401

The browser accepts http://3232235777/ exactly like http://192.168.1.1/: it converts the number to dotted quad before connecting. The SSH client too, the FTP client too.

Fail2ban: fighting bruteforcing

Mon, 08 Dec 2008 00:00:00 +0000

Every public SSH server receives hundreds of connection attempts per day. The auth.log fills up with Failed password for root from x.x.x.x lines. Fail2ban reads those logs, counts the failures and bans the IP via iptables. It is a guard dog that reads logs instead of sniffing.

The principle

Fail2ban works in three steps: it tails a log file (auth.log, mail.log, apache.log), it matches lines against configurable regular expressions and it runs an action when the threshold is reached (typically, an iptables DROP rule for the offending IP for N minutes).

OpenDNS: the free DNS

Sat, 06 Dec 2008 00:00:00 +0000

OpenDNS (208.67.222.222 and 208.67.220.220) is a free public DNS resolver, one that appeared before the big providers’ resolvers. The principle: you replace your ISP’s DNS with theirs and your resolution goes through their servers instead of yours.

Why

Three reasons people invoke:

The ISP’s DNS is slow, goes down, or lies (the captive portal resolving everything to its login page);
OpenDNS filters phishing and malware domains (their malicious domain database, updated continuously);
OpenDNS does caching and the cache is faster than the ISP’s resolver.

How

# In /etc/resolv.conf
nameserver 208.67.222.222
nameserver 208.67.220.220

Or in the network’s DHCP so every workstation benefits.

Subversion: a WebSVN with mod_dav and mod_svn for Apache

Mon, 01 Dec 2008 00:00:00 +0000

Subversion has two access modes: svnserve (standalone daemon, svn:// protocol) and Apache with mod_dav_svn (http:// or https:// protocol). The Apache mode exposes the repository via HTTP, with Apache authentication, Apache SSL and WebSVN for web browsing of the sources.

The mod_dav_svn module

apt-get install libapache2-svn

This package installs mod_dav and mod_dav_svn (the module that exposes SVN repositories via WebDAV).

The Apache configuration

# /etc/apache2/sites-available/svn
<Location /svn>
 DAV svn
 SVNParentPath /var/svn
 AuthType Basic
 AuthName "Subversion Repository"
 AuthUserFile /etc/apache2/dav_svn.passwd
 Require valid-user
</Location>

DAV svn : activates SVN mode
SVNParentPath /var/svn : the directory containing the repositories (one repo per subdirectory)
AuthUserFile : the htpasswd file for authentication

Create the repository and the password file:

ext3: accelerate your disk accesses

Thu, 20 Nov 2008 00:00:00 +0000

The ext3 filesystem is the default on every distribution for years now and it is mounted with the slowest possible options. Three words to add to the fstab and your disk accesses are halved.

The three options

/dev/sda1 / ext3 defaults,noatime,data=writeback,commit=60 0 1

noatime. The access time (atime) is updated at EVERY file read. A cat file updates the atime of file AND of every directory in the path. On a web server reading thousands of files per second, atime is an invisible disk overwrite. noatime removes it outright. Cost: ls -lu no longer shows the last read time, which has never been useful to anyone.

Compiling a shell script with shc

Wed, 19 Nov 2008 00:00:00 +0000

A shell script is text readable by everyone: cat script.sh shows the code, any passwords, the business logic. shc (shell script compiler) transforms the script into a binary: the script is compiled into C, the C is compiled into an ELF binary and the text source is no longer readable.

The principle

shc -f script.sh

shc generates two files:

script.sh.x : the executable ELF binary
script.sh.x.c : the generated C source (delete after compiling)

The binary script.sh.x runs like the original script, but the shell code is not readable in the binary.

Debian Etch: play with backports

Sun, 16 Nov 2008 00:00:00 +0000

Debian Etch (4.0) is stable: packages are tested, versions are frozen and an application that depends on a library newer than Etch’s does not compile. Backports solve this problem: the Debian Backports project recompiles packages from Sid (unstable) for Etch’s dependencies and makes them available in a separate repository.

The backports repository

# /etc/apt/sources.list
deb http://www.backports.org/debian etch-backports main contrib non-free

The GPG key of the repository:

wget -O - http://backports.org/debian/archive.key | apt-key add -

Update:

ArpON: watching your ARP cache

Fri, 14 Nov 2008 00:00:00 +0000

ArpON is a very practical daemon, where tools like Arpwatch only let you see the problems, ArpON lets you block them (notably ARP cache poisoning attempts).

To do this it can use two protocols, SARPI (Static ARP Inspection) and DARPI (Dynamic ARP Inspection).

It perfectly replaces the arp, arping and arpscan tools, detects and blocks spoofing or poisoning attempts and because it runs in user space, remains very portable. Currently compatible with OpenBSD, FreeBSD, Debian, Gentoo, Slackware and even Mac OSX.

EncFS: file encryption without a dedicated partition

Mon, 20 Oct 2008 00:00:00 +0000

Encrypting your data on Linux has always meant LUKS, dm-crypt, a dedicated partition, a reinstall. EncFS breaks that pattern: it is a FUSE filesystem that encrypts file by file, in a normal directory, without a dedicated partition, without reinstalling and without root access after the first mount.

The principle

EncFS creates two directories: the cipher (the encrypted files, with encrypted names too) and the plain (the mount point, where files appear in cleartext). Everything you write to the plain is encrypted on the fly and stored in the cipher. Everything you read from the plain is decrypted on the fly from the cipher. Unmount the plain and all that remains is the cipher: unreadable names, encrypted content.

radare: a very complete debugger

Sat, 04 Oct 2008 00:00:00 +0000

After the Perl harness bolted onto GDB and before Metasm pulled me into Ruby, there was radare. A console-mode debugger written by pancake (Nibble), doing everything GDB does for exploitation, but with an interface designed for reverse engineering rather than application debugging.

What radare does that GDB does not

GDB is an application debugger: it stops on breakpoints, prints variables, follows calls. radare is a reverse engineering tool: it disassembles, analyses strings, finds gadgets, computes offsets and lets you navigate the binary like a filesystem.

Yersinia: how to make your network administrator sick

Thu, 28 Aug 2008 00:00:00 +0000

Yersinia is the name of the plague bacterium. The tool lives up to its name: it attacks layer 2 protocols (STP, CDP, DTP, DHCP, VTP, HSRP) and makes your switch sick. The title of this article is not a metaphor: it is what actually happens to a network administrator who discovers that a workstation with Yersinia and a simple cable can take control of their switch topology.

The attacks, one per protocol

STP (Spanning Tree Protocol). Yersinia sends BPDUs claiming to be the root bridge with a lower priority. The switch recalculates the tree and traffic flows through the attacker’s workstation. Your switch has been re-rooted and all L2 traffic transits through the intern’s laptop.

SNMP: a not-so-simple protocol

Tue, 01 Jul 2008 00:00:00 +0000

SNMP stands for Simple Network Management Protocol. The “Simple” is the irony of the century: the protocol uses OIDs (Object Identifiers) that are trees of numbers, MIBs (Management Information Base) that translate the numbers into names, communities (public, private) that serve as cleartext passwords and versions (v1, v2c, v3) each with its own quirks.

The OIDs: the tree

Every piece of data in SNMP has an OID: a dot-separated sequence of numbers. The standard tree:

tcpdump: frame analysis and BPF filters

Fri, 20 Jun 2008 00:00:00 +0000

After Nemesis which forges and raw sockets which sniff by hand, the tool that does all of it without writing a line of C: tcpdump. If you have never used it, know that it is probably the highest-return diagnostic tool of your entire career: one command, one filter and you read the network in cleartext.

The BPF filter, the concept that changes everything

tcpdump alone shows ALL the traffic on the interface. It is unusable on an active network: you are drowning in ten seconds. The magic comes from the BPF filter (Berkeley Packet Filter), an expression language compiled on the fly into the kernel to only capture what matches:

Nemesis: detecting a network sniffer

Fri, 06 Jun 2008 00:00:00 +0000

The previous article talked about forging packets with Nemesis . This one talks about the opposite side: detecting that someone on your network is listening to what they should not.

The problem

A network card in promiscuous mode accepts ALL frames, not only the ones addressed to its MAC. That is what lets tcpdump and Wireshark work and it is also what lets a sniffer read traffic not destined for it. On a switch, it should not pass (the switch routes by MAC), but if someone has tinkered with the switch or if there is a hub somewhere, it passes.

Nemesis: network packet forge

Mon, 02 Jun 2008 00:00:00 +0000

If you have read the raw sockets article , you know how to forge a SYN by hand in C: fill the fields, compute the checksum, send. It is educational and it is also what Nemesis does from the command line in one invocation. Nemesis is a generic packet forge: TCP, UDP, ICMP, RIP, OSPF, ARP, DNS. Each protocol has its subcommand, each header field has its option.

The idea

A packet forge is the inverse of tcpdump: tcpdump reads what passes, Nemesis builds what passes. The combination of both is the complete network testing protocol: Nemesis forges, tcpdump verifies what actually arrives and you know what your firewall, your switch or your IDS understands from what you sent.

Raw Sockets: forging and sniffing packets in C

Fri, 09 May 2008 00:00:00 +0000

We spoke of TCP sockets in C , under the comfort of SOCK_STREAM. Let’s go below the kernel: raw sockets, where tcpdump and Nemesis live. Usual warning: SOCK_RAW requires root, the network card in promiscuous mode and some ethics, all without a contract with anyone.

Sniffing: AF_PACKET and promiscuous mode

int s = socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL));

Two lines and you receive everything crossing the interface, Ethernet headers included, provided the interface goes promiscuous:

struct ifreq ifr;
strncpy(ifr.ifr_name, "eth0", IFNAMSIZ);
ioctl(s, SIOCGIFFLAGS, &ifr);
ifr.ifr_flags |= IFF_PROMISC;
ioctl(s, SIOCSIFFLAGS, &ifr);

Curiosity, at layer 2, is not configured, it is paid for: the day the switch mirrors your port without telling you, you understand that reading other people’s conversations is a closely watched sport. The raw packet lands in a buffer: Ethernet (14 bytes), then IP, then TCP. Manual reading, no framework:

Sockets: speaking TCP in C, no framework, no magic

Wed, 02 Apr 2008 00:00:00 +0000

After LKMs and syscalls, let’s stay low. All of Unix networking fits in four system calls, two structs and a treaty of byte order. The rest is memory and discipline. Small memo, Debian Etch required, gcc 4.1 and man pages 2 and 7 as only friends.

The structs, first

#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>

struct sockaddr_in addr;
addr.sin_family = AF_INET; /* IPv4 */
addr.sin_port = htons(4444); /* network byte order, ALWAYS */
addr.sin_addr.s_addr = htonl(INADDR_ANY); /* 0.0.0.0 */

htons and htonl are not decorative: the network speaks big-endian, your x86 speaks little-endian and the neighbor who forgets htons gets a server on port 45056. man 7 ip explains all this better than me.

Linux: introduction to LKMs

Sun, 02 Mar 2008 00:00:00 +0000

The Linux kernel is monolithic: all code runs in the same address space, in privileged mode. But it accepts additional code at runtime: LKMs (Loadable Kernel Modules). An LKM is a .ko file compiled for the running kernel, that inserts into kernel space and adds functionality: device drivers, filesystems, network protocols, custom syscalls.

The smallest LKM

// hello.c
#include <linux/module.h>
#include <linux/kernel.h>

static int __init hello_init(void) {
 printk(KERN_INFO "Hello from kernel space\n");
 return 0;
}

static void __exit hello_exit(void) {
 printk(KERN_INFO "Goodbye from kernel space\n");
}

module_init(hello_init);
module_exit(hello_exit);
MODULE_LICENSE("GPL");

The Makefile:

Linux syscalls

Thu, 21 Feb 2008 00:00:00 +0000

A user-space program does nothing directly: when it reads a file, allocates memory, or sends a network packet, it asks the kernel to do it. This request is a syscall (system call): the program executes a special instruction (int 0x80 or sysenter on x86), the kernel takes over, executes the request and returns control to the program. It is the boundary between user space (ring 3) and kernel space (ring 0).

SSLH: ssh and https on the same port

Sun, 16 Sep 2007 00:00:00 +0000

The 2007 problem, in companies with nervous firewalls: one outgoing port only, 443, everything else cut. You want your ssh shell and your https and the firewall does not negotiate. The answer is sslh (v1.0, by Yves Rutschle): a small daemon that listens on 443 and delegates based on the first packet: to sshd if the client speaks the SSH protocol, to httpsd otherwise.

The mechanism

client --443--> sslh --detection--> sshd:22
 \--> https:443

The SSH protocol announces itself politely: the client sends a SSH-2.0-... banner in clear text upon opening. SSL/TLS starts with a ClientHello whose first byte is not an ‘S’. Two bells, two destinations:

PAM: limit system resources

Fri, 04 May 2007 00:00:00 +0000

PAM (Pluggable Authentication Modules) does not only do authentication: the pam_limits module caps system resources per user and per group. Without limits, a process can eat all the RAM, all the file descriptors, all the process slots: that is the fork bomb, or simply a process that leaks memory.

The limits.conf file

# /etc/security/limits.conf
# <domain> <type> <item> <value>

# Cap processes for all users
* soft nproc 2048
* hard nproc 4096

# Cap open file descriptors
* soft nofile 1024
* hard nofile 4096

# Cap memory (in KB)
* soft as 1048576
* hard as 2097152

# For the backup group: more open files
@backup soft nofile 8192
@backup hard nofile 16384

# For root: no limit
root soft nproc unlimited
root hard nproc unlimited

soft : the current limit (the process can raise it up to the hard limit)
hard : the maximum limit (the process cannot exceed it, even with ulimit -Hn)
* : all users
@group : a Linux group
root : the root user

The limits.conf items

item	what it caps
`nproc`	the number of processes
`nofile`	the number of open file descriptors
`as`	memory (address space) in KB
`rss`	resident memory in KB (old, replaced by as)
`cpu`	CPU time in minutes
`maxlogins`	the number of simultaneous sessions
`core`	the size of core dumps

Checking the limits

ulimit -a

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
max nice (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 4096
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 8192
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 2048
virtual memory (kbytes, -v) unlimited

ulimit -n shows the file descriptor limit: 1024 by default, which is insufficient for a web server or a database. 4096 or 16384 is the minimum for a server.

SSH: tunneling

Wed, 04 Apr 2007 00:00:00 +0000

SSH does not only transfer files: it makes any TCP protocol cross the encrypted channel. A VNC connection, a MySQL database, an internal web server, an SMB share: everything that is TCP can be tunneled through SSH, without VPN, without firewall configuration, with one command.

The local tunnel (local forward)

ssh -L 3306:localhost:3306 user@server

This command opens port 3306 on your local machine and everything that arrives there goes to port 3306 on server (as seen from server). An SSH session is open, the tunnel works as long as the session lasts.

MySQL: backup

Tue, 06 Mar 2007 00:00:00 +0000

A MySQL backup that has never been restored is not a backup, it is a hope. The previous article covered the memento , this one covers automated backup and the restore test.

The backup script

#!/bin/bash
BACKUP_DIR="/backup/mysql"
DATE=$(date +%Y%m%d)
MYSQL_USER="backup_user"
MYSQL_PASS="password"

mkdir -p "$BACKUP_DIR/$DATE"

# List databases
DATABASES=$(mysql -u $MYSQL_USER -p$MYSQL_PASS -e "SHOW DATABASES" | grep -v Database | grep -v information_schema | grep -v test)

# Dump each database
for db in $DATABASES; do
 mysqldump -u $MYSQL_USER -p$MYSQL_PASS \
 --add-drop-table \
 --single-transaction \
 "$db" > "$BACKUP_DIR/$DATE/$db.sql"
 gzip "$BACKUP_DIR/$DATE/$db.sql"
done

# Clean backups older than 7 days
find "$BACKUP_DIR" -type d -mtime +7 -exec rm -rf {} \;

The --single-transaction is crucial: it starts an InnoDB transaction and dumps from a consistent snapshot without locking tables. Without this flag, mysqldump locks each table and your production application waits.

MySQL: lost credentials

Thu, 08 Feb 2007 00:00:00 +0000

The scenario: a server you have not touched for six months, a MySQL whose root password nobody remembers and a client asking why their site no longer connects to the database. The root password is lost and there is no config file with it lying around.

The procedure

# 1. Stop MySQL
/etc/init.d/mysql stop

# 2. Start without grant tables
mysqld_safe --skip-grant-tables --skip-networking &

# 3. Connect without password
mysql -u root mysql

# 4. Change the root password
UPDATE user SET Password=PASSWORD('newpass') WHERE User='root';
FLUSH PRIVILEGES;
EXIT;

# 5. Kill mysqld_safe and restart cleanly
killall mysqld_safe mysqld
/etc/init.d/mysql start

The --skip-grant-tables flag disables ALL authentication: anyone can connect as anyone. The --skip-networking flag prevents external TCP connections during the recovery window (only the local socket works). Both together are the safe procedure.

MySQL: Memento

Fri, 26 Jan 2007 00:00:00 +0000

The MySQL memo for admin commands that are not typed often enough to remember, but often enough to know they exist.

Connection

mysql -u root -p
mysql -u root -p -h localhost my_schema
mysql -u root -p < dump.sql

Users and grants

CREATE USER 'user'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON my_schema.* TO 'user'@'localhost';
GRANT SELECT, INSERT ON my_schema.* TO 'readonly'@'%';
FLUSH PRIVILEGES;
SHOW GRANTS FOR 'user'@'localhost';
DROP USER 'user'@'localhost';

Backup and restore

# Full backup of one database
mysqldump -u root -p my_schema > backup.sql

# Backup of all databases
mysqldump -u root -p --all-databases > all.sql

# Backup with DROP TABLE
mysqldump -u root -p --add-drop-table my_schema > backup.sql

# Restore
mysql -u root -p my_schema < backup.sql

Admin queries

SHOW DATABASES;
SHOW TABLES;
SHOW TABLE STATUS;
SHOW PROCESSLIST;
SHOW VARIABLES;
SHOW ENGINE INNODB STATUS;
DESCRIBE table_name;

Optimization

OPTIMIZE TABLE table_name;
ANALYZE TABLE table_name;
REPAIR TABLE table_name;

The lost root password

If you have lost the root password, the procedure is:

Apache: what a chatterbox

Thu, 21 Dec 2006 00:00:00 +0000

Apache talks. A lot. Every request generates a line in the access_log, every error a line in the error_log and the LogLevel controls the chattiness. On a server serving 100,000 pages per day, the access_log weighs 50 MB per day and nobody reads it.

The LogLevel

LogLevel warn

The levels, from chattiest to quietest: debug, info, notice, warn, error, crit, alert, emerg. warn is the default and the right one: it logs warnings (config, permissions) and errors, but not every module starting up.

Apache: the limits of .htaccess

Tue, 28 Nov 2006 00:00:00 +0000

.htaccess is practical: one file per directory, no restart, the shared hosting client manages it alone. On a dedicated server, it is a slowdown.

Every HTTP request makes Apache read all the .htaccess files in the path, from the root. For /var/www/site/blog/article/, Apache reads /var/www/.htaccess, /var/www/site/.htaccess, /var/www/site/blog/.htaccess and /var/www/site/blog/article/.htaccess. Four disk reads per request before serving the page.

The fix: AllowOverride None in httpd.conf and everything that was in the .htaccess files moves to the VirtualHost. Apache no longer reads .htaccess files and the directives are compiled at startup.

Apache: disabling the TRACE option

Fri, 06 Oct 2006 00:00:00 +0000

The HTTP TRACE method sends back to the client the exact request it sent, in echo. Designed for debugging, it is a door for XST (Cross-Site Tracing): JavaScript on a compromised page can use TRACE to read HttpOnly cookies via the request headers. IE6 allowed it, Firefox too.

The test:

$ telnet mydomain.com 80
TRACE / HTTP/1.1
Host: mydomain.com

HTTP/1.1 200 OK
TRACE / HTTP/1.1
Host: mydomain.com
...

If you see your request echoed back, TRACE is enabled.

SSH: authentication via private/public keys

Sun, 10 Sep 2006 00:00:00 +0000

SSH authentication by password works, but it is weak: the password is typed at every connection, it can be brute-forced (fail2ban mitigates but does not solve) and it does not distinguish “who” from “what”. SSH keys solve all three: the private key proves identity, the public key authorizes access and once configured, no more password.

Generate the pair

ssh-keygen -t rsa -b 4096 -C "comment"

The -t rsa generates an RSA pair, the -b 4096 sets the size (2048 minimum, 4096 recommended). The -C is a free comment (usually the email or hostname). Two files: id_rsa (the private key, never leaves the workstation) and id_rsa.pub (the public key, to copy to servers).

Magic SysRq: the sysadmin's crutch

Fri, 11 Aug 2006 00:00:00 +0000

A server is frozen. No SSH, no console, no ping response. You are in a datacenter or on a serial console and the only option before pulling the plug is the SysRq key.

What it is

The SysRq (System Request) key is intercepted directly by the Linux kernel, regardless of system state. The server may be in kernel panic, the scheduler may be blocked: the kernel answers SysRq as long as the hardware interrupts work. It is the emergency crutch.

iodine: IP-over-DNS, or the internet where only DNS passes

Fri, 28 Jul 2006 00:00:00 +0000

The scene is familiar: the conference hotel, the fast-food hotspot, one of those captive portals hatching everywhere since Wi-Fi became a service. You connect, a page opens, you are supposed to pay or type a room code and everything that is not HTTP on port 80 falls into the void. No SSH, no direct SMTP, no visible VPN. A filter that only lets the web through, which is almost nothing.

dig: the Domain Information Groper, or the art of asking a name server

Mon, 26 Jun 2006 00:00:00 +0000

For years, my DNS tool was nslookup. The habit comes from far away, it comes from Windows, it comes from the reflex “I want to see a record, I type nslookup”. Then one day I had to debug a zone that was propagating badly between a primary and a secondary and nslookup answered me with its interactive interface, its > prompt, its output that changes shape depending on the resolver library version of the system. I installed the dnsutils package that very evening and nslookup retired.

Links: hard links and symbolic links and what escapes the chroot

Mon, 05 Jun 2006 00:00:00 +0000

Linux knows two types of links and the difference between them is a sysadmin exam subject AND a chroot escape vector. The same concept that explains your data backup also explains how to break out of a jail.

The hard link

A hard link is a directory entry that points directly to an inode. The file and its link are two names for the same disk content. Deleting one does not delete the other (the inode survives as long as its link counter is above zero).

SSH: where is the fish?

Wed, 10 May 2006 00:00:00 +0000

The fish protocol (Files transferred over SHell) is a KDE invention: it encapsulates shell commands in an SSH connection to browse and transfer files as if it were a network mount. Konqueror does it natively: type fish://user@server/home/user in the address bar and the remote filesystem appears in the file browser.

What fish does

Fish is not a mount (no FUSE, no kernel module): the client sends shell commands (ls, cat, cp) over SSH and parses the result. The server needs nothing special: just a POSIX shell. The client builds the commands, sends them over the SSH channel and presents the result as a filesystem.

Root or not root

Thu, 13 Apr 2006 00:00:00 +0000

The question: who gets to be root on a shared server? The answer is not technical, it is political. But the technical tools exist and they are not equal.

su

su asks for the root password. Whoever knows it becomes root. Period. All admins share the same password and when someone leaves the team, it needs changing (and redistributing). The history of what was done as root is limited to last and the memory of whoever was there.

Bash: Memento

Wed, 15 Mar 2006 00:00:00 +0000

A memo, not a tutorial. The bash lines you look up every time because you do not type them often enough to remember, but often enough to know they exist.

Substitutions

# Replace the first occurrence
${var/old/new}
# Replace all occurrences
${var//old/new}
# Remove the shortest pattern from the start
${var#pattern}
# Remove the shortest pattern from the end
${var%pattern}
# String length
${#var}
# Default value if empty
${var:-default}
# Assign if empty
${var:=default}
# Error if empty
${var:?message}

Arrays

declare -a tab
tab[0]="first"
tab+=(second third)
echo ${tab[@]}
echo ${#tab[@]}
echo ${tab[1]}
# Available indices
echo ${!tab[@]}

Redirections

cmd > file # stdout to file (overwrites)
cmd >> file # stdout to file (appends)
cmd 2> file # stderr to file
cmd &> file # stdout + stderr to file
cmd 2>&1 # stderr to stdout
cmd > /dev/null 2>&1 # full silence
cmd < file # stdin from file

Loops

for i in $(ls); do echo "$i"; done
for i in {1..10}; do echo "$i"; done
while read line; do echo "$line"; done < file

Tests

[[ -f file ]] # exists and is a regular file
[[ -d dir ]] # exists and is a directory
[[ -s file ]] # exists and is not empty
[[ -x file ]] # executable
[[ "$a" == "$b" ]] # string equality
[[ $n -eq $m ]] # numeric equality
[[ $n -lt $m ]] # less than

That is all. The rest is learned with man bash (EXPANSION section), but these lines are the ones you paste into a file and source.