BrainForge: a distillation factory where AI experts argue

Ask an LLM “is this code vulnerable?” and it will answer, confidently, 100% of the time. About half of the time it will be right. BrainForge is a distillation framework: it transfers the know-how of large expert models into a small student model. That takes lessons: thousands of verified examples. And that is where everything is decided, because an LLM can be wrong, or worse, recite its CVE memories instead of analyzing.

So the lesson factory is organized like a trial. Experts testify, a devil’s advocate challenges them, an independent judge rules: every lesson in the dataset is vetted before anything is taught. Out comes a clean, labeled dataset, ready to fine-tune a student model on a gamer-grade GPU.

The actual problem: learning without cheating

Fine-tuning a model on security tasks requires verified examples. But LLMs have memorized everything: thousands of CVEs, articles and write-ups dated before their training cutoff. If I feed my dataset with vulnerabilities published before the teachers’ knowledge cutoff, the student does not learn to analyze code: it learns to recite memories. Worse, my benchmark would then measure recall, not competence.

BrainForge refuses that shortcut. Every record carries two dates: the date of the case (the git commit date of the source file, captured automatically) and the knowledge cutoff of each teacher model. When the source predates the cutoff, the record is tagged recitation_risk and recent cases go to a separate split: the final exam, reserved for cases the teachers have never seen.

The courtroom

Every lesson in the dataset is manufactured in a workshop that looks suspiciously like a trial. The flow of a case:

The evidence. It starts from code files (or JSON cases) and everything is recorded: repository, path, language and above all the git commit date, which becomes the source date. Alongside, documents (reports, standards, notes) are indexed by a small homegrown vector search engine (sqlite + numpy): that is the RAG, reserved for the adults in the room (teachers and judge), never handed to the student.

The experts. Depending on the chosen mode (cheap, standard, maximum), one to three teachers analyze the case with its RAG context: a security expert, a coding expert, a generalist. Each hands in an answer as JSON, validated against a pydantic schema. The schema is injected into the system prompt and, when the model hands in an unreadable answer, a repair loop re-runs the exam with its own faulty answer attached as evidence (two attempts). The mock provider can generate a valid output for any schema, so the whole pipeline runs without API keys, at zero cost.

The devil’s advocate. The adversarial critic receives the same file, with instructions to challenge it. It forces nuance where a lone expert would have ruled too fast.

The judge. It sees the case, the RAG context and all testimony. Its verdict is constrained: confirmed, rejected or insufficient_information, with a confidence score, a CWE if a vulnerability is confirmed and evidence it must re-derive itself (not just rubber-stamp the experts).

The clerk. The quality gate filters the output: malformed CWE, claimed vulnerability without evidence, confirmed verdict without CWE, confidence too low: rejected. And rejections are not thrown away: they are quarantined with their reasons, ready to be studied for a better iteration. Every final record is a question/answer pair in JSONL, with full provenance: which teachers, which models, which providers, agreement rates, RAG chunks used, dates, recitation risk.

The golden rule: a judge who does not know the witnesses

In a real courtroom, the judge must not be the witnesses’ cousin. Here it is baked into the config: loading refuses to start if the judge shares a provider or a model family with any teacher. In my setup, the judge is a GLM served by Zen, the experts are DeepSeek and Qwen served by OpenRouter and MLGW. The rule has three positions (enforced, warn, off) and the default is enforced.

Another safeguard: the teacher/judge agreement rate is computed and stored per record. Above 95% systematic agreement, a correlated-bias warning shows up. In the demo below the mock agrees 100% of the time: the warning appears, on purpose.

The final exam: post-cutoff cases

The train/test split is not a random per-record shuffle: it is grouped by source (repository, path or URL), by deterministic hash. Two cases from the same repository never end up one in training and one in testing. Records at recitation risk are excluded from the training split and post-cutoff cases form a dedicated split, test_postcutoff: the final exam. If fewer than 20 cases survive it, a warning says so bluntly.

The task-level evaluation happens there: verdict accuracy, false positive and false negative rates, CWE accuracy, all compared against the judge’s ground truth. It is the only score I trust.

The actual distillation

The distillation happens here: the teachers’ knowledge, locked into the vetted dataset, is transferred to the student. Hardware-wise, QLoRA works wonders: an 8-billion-parameter base (Qwen3) frozen in 4-bit (NF4), LoRA adapters of rank 16, gradient accumulation over 16 steps and it fits on an RTX 3080. The student learns in chat format: question (the case), answer (the judge’s verdict). An assistant_only_loss setting exists to count only the assistant’s answer (off by default).

Evaluation is twofold: corpus perplexity (token-weighted, not the mean of means) to monitor training and the task-level evaluation above to judge real competence. At the end, the adapter is merged into the model and exported in bf16.

Try it at home

Three commands, no API key:

uv run brainforge pipeline run security_dataset --input examples/cases --provider mock
uv run brainforge dataset validate datasets/security_dataset.jsonl
uv run brainforge dataset inspect datasets/security_dataset.jsonl

The first runs the full pipeline on mock (3 cases accepted, 0 rejected), the second validates the resulting dataset, the third prints the contamination table and the teacher/judge agreement rates, correlated-bias warning included.

An honest summary

About 3,100 lines of Python, more than 100 unit tests, a blocking 85% coverage gate, 10 ADRs (one per design decision, including the why) and a CI with the same six gates (lint, secrets, test, config, build, audit) on three forges, with gitleaks scanning the whole git history. It is not magic: it is a well-organized trial, with a manic clerk.

Next time an LLM tells you “this code is vulnerable”, ask for its evidence. Or build your own courtroom.