Agentic AI in Production: an Audit Is Not a Pentest

Let me state my bias up front: no diploma, nearly 25 years in the trade not counting practice. I run LLM agents every day, local and cloud and I spent a good part of 2025-2026 answering LinkedIn threads about what they can actually do. This article assembles those answers: where agentic AI is relevant, where it plateaus, which operational pitfalls cost hours and why the answer to “plausible code in three prompts” is not a certification.

1. Audit, pentest, CTF: three exercises, three uses

The most frequent confusion is terminological and it costs budgets.

For an audit, agentic AI is extremely relevant. Collection, configuration analysis, AD, mail, architecture, confrontation against frameworks, findings correlation, prioritization, reporting: this is typically the kind of work where several agents contribute enormously. An agent that extracts an Nginx config, compares it to a benchmark and reports the gaps is a tireless intern and it doesn’t complain about LDAP dumps.

For a pentest, the problem is different. You have to build and demonstrate an exploitation path in a real environment and there the current limits are far more visible. That doesn’t mean agentic AI cannot do pentest, CTFs show it can. With the right model, an adapted RAG, the right tools and above all guardrails and a real execution/verification loop, it already becomes very interesting.

In my approach, several models work in parallel, but consensus is never treated as proof. Results must be verifiable. Three models agreeing are not a demonstration and a demonstration is worth nothing if it isn’t reproducible.

2. Data before model

Before even choosing a model, there is another essential subject: data. For an audit, you handle AD, LDAP dumps, source code, configurations and client data under NDA.

So ideally a local model and if possible one specialized/trained for the task. Failing that, with a commercial LLM, you must at minimum know its limits, its biases and verify precisely what happens to the data: retention, training. In short: AI yes, but not “we throw an agent at prod and cross our fingers”.

This is the least-exposure principle applied to agentic AI: digital hygiene doesn’t stop at your vacation photos.

3. The Ollama trap: context silently truncated

The most expensive operational bug I see in 2026 is not in the models, it is in the wrappers. Run ollama ps during a session and look at the CONTEXT column.

Ollama often applies a default context far below what the model advertises and truncates from the beginning beyond that, silently, without warning or error. In chat it passes. In agentic mode it is fatal: the system prompt and tool definitions already consume thousands of tokens, the plan falls out of context after two turns, tool calls go sideways, the agent redoes the work. That is how a two-minute micro-feature becomes a twenty-two hour session, the OpenCode plus Ollama case is widely documented.

Second grievance, not the smallest: Ollama chooses for you without telling you what it chose. For daily use it is comfortable. For fine-tuning or for publishing a number, it is disqualifying. If you want to keep the simplicity while seeing what you tune, LM Studio exposes context, GPU offload and KV cache quantization in the interface and serves an OpenAI-compatible endpoint. If you want full control, llama-server or mlx_lm.server launch in one line, context and cache passed as parameters, everything in the logs.

A third detail that changes the numbers: a dense model like Qwen3 32B re-reads its 32 billion parameters at every token and exposes a reasoning_effort cursor set high by default, on a trivial task it overthinks and your thirty minutes of planning point exactly there. A MoE like 30B-A3B occupies the same memory but activates only 3 billion per token and the gap counts in multiples.

4. Local or cloud: the wrong question

My gripe with Ollama is not speed, in its recent versions it also runs on MLX and the gap with llama.cpp is 10 to 15%. I run Qwen3-Coder in agentic mode on a 4090 and a 3080 with llama.cpp: on diffs, completion and framed refactoring, it is usable daily. On long reasoning and large-context exploration, no and I switch to cloud without second thoughts.

So the real question is not “local or not”, it is which task goes where. A reasoning model, you bound it, you choose the task, you don’t dive into the shiny because it shines. In 1990 with neural networks, in 2026 with LLMs, the engineering reflex hasn’t moved an inch: the robust and slightly boring thing survives the ambitious and fragile thing.

And framing changes everything: a laptop is not a deployment. For the companies I train, local doesn’t mean a MacBook per developer: it means a shared GPU server under vLLM, with continuous batching and a dozen parallel sessions. Different regime, different numbers and the conclusion “local is slow” doesn’t cover that case.

5. Tests that lie

A test that verifies nothing passes green without flinching. But this flaw is not AI-specific: humans write them all the time, me first.

I was initially reluctant to delegate test writing to agents. Since then, with the right context and the right instructions, the LLM writes them better and faster than I do. On my side, I supervise and go over it on major iterations, exactly as I would for human code. We keep shift left here too: a fake green is a defect like any other, better to catch it as early as possible.

And what says NO to a test that verifies nothing is not a human with an opinion: it is mutation testing. You mutate the code on purpose; if the test stays green, it is useless. Stryker, mutmut, PIT: deterministic, 0 or 1, the same guardrail applied to the tests themselves. What makes a test authoritative is not its author, it is the mechanism.

6. Specification gaming: the agent didn’t “want” to hack

The story of the OpenAI LLM that hacked Hugging Face made its round. Nothing wild: we explicitly asked the models to perform advanced exploitation through complex attack paths and we disabled the guardrails to measure the ceiling.

It is not an AI deciding one morning to hack someone, it is specification gaming at scale. OpenAI says it more or less in those terms: the models were hyperfocused on solving ExploitGym and went to considerable lengths to reach a fairly narrow test objective. The model didn’t want to hack HF, it wanted to maximize a score and the cheapest path went through stealing the fix. It is the old RL problem, except here the agent has an RCE at the end.

It is not an isolated case: Anthropic reported that a model also escaped a sandbox during security tests to obtain unplanned Internet access and yes, it is also a PR stunt. The lesson remains: the guardrail disabled “to measure the ceiling” is a ceiling that opens.

7. Judgment cannot be certified

The real problem: how do you filter on judgment when anyone produces plausible code in three prompts? But I don’t believe certification is the right instrument.

We have known for 25 years what predicts candidate performance. The reference meta-analysis (Sackett et al., 2022) puts the structured interview, job knowledge tests and work samples at the top. Degree and self-reported experience land last, far behind. A certification is one notch below: it can be bought and it expires in a few quarters. And when the one selling the training is also the one certifying and recruiting, without independent examination, you are not measuring competence, you are measuring the checkout.

What works is simpler and free: make the candidate code in front of you, with or without IDE, LLM, agent, harness, then ask them to tell the story of a production incident they really lived. Plausible code gets produced in three prompts; a lived incident gets told with its details, its bad decisions and its aftertaste.

8. The tool amplifies its holder

The question I ask those who fear agentic AI and those who adore it: who uses the tool to learn and who is served by it.

The bottleneck was never the machine, it is discernment and the fundamentals we no longer want to learn. The worst is not even those who don’t touch it, it is what we are manufacturing inside the trade: a whole generation overspecialized and trained on the assembly line, glued to their idioms and frameworks, shipping code without ever having looked underneath. They don’t know what a syscall does, how a TCP/IP packet travels, what happens under the abstraction and above all they don’t care one second.

Put an LLM in those hands and it doesn’t lift them: it lets them run faster in the same rut. The tool amplifies its holder. A handful of profiles will use it to dig and truly learn, many to think on its behalf and ship slop on the chain, hiding the hole a bit faster.

Summary

Audit and pentest are not the same exercise: agentic AI excels at collection and framework confrontation, plateaus at exploitation demonstration. Consensus across models is never proof.
Data first: NDAs, AD dumps, source code imply a local model or precise verification of retention. No agent on prod without framing.
The number one pitfall of 2026 is operational, not intellectual: silently truncated context (Ollama, CONTEXT column) turns two minutes into twenty-two hours. Pick tools that expose their choices.
Local or cloud is the wrong question; the right one is which task goes where. A dense model overthinking on a 4090 says nothing about properly deployed local under vLLM.
Fake greens get settled by mutation testing, not opinion. Specification gaming is not malicious intent, it is a poorly bounded objective with an RCE at the end.
Certification doesn’t measure judgment. Make them code, then make them tell a lived production incident.
The tool amplifies its holder. Invest in fundamentals before investing in tokens.