Local LLMs and agentic security

Generating one token means streaming every active weight through the chip once. So the ceiling for a single conversation is arithmetic, not benchmarking: tokens per second ≈ memory bandwidth ÷ bytes read per token, where the bytes are the active weights at your quantisation plus the KV cache you drag along for the current context. TFLOPS, TOPS and core counts start to matter when you serve many requests at once, and for prompt processing, which is a separate workload: NVIDIA's serving docs describe prefill as compute-bound and decode as memory-bound.

My box: a 64-core EPYC at 2.0 GHz with 512 GB of DDR4 on eight channels, and an RTX PRO 6000 Blackwell with 96 GB of GDDR7 at 1.8 TB/s; before that, an RTX 3090. Sixteen sticks means two per channel, so the memory runs at 2933 MT/s rather than 3200: 187.7 GB/s on paper, 140.8 GB/s measured with sysbench. Take my daily driver, Qwen3.8 27B: at Q4 it reads about 15 GB per token, so the arithmetic ceiling is about 118 tok/s on the PRO 6000 and 9 on the EPYC at measured bandwidth. For a measured calibration point I use an older, public run: Qwen3-235B-A22B at Q4 reads about 12.4 GB of active weights per token, so the CPU ceiling at short context is about 15 tok/s on paper and 11 at the measured bandwidth. llama.cpp gave me 7.2–8.0 tok/s, a normal ratio for CPU decode. On a GPU with mature kernels, 50–70% of the ceiling is what to expect. When the number is far below that, look at software first: my notes record Qwen3.5-122B at 14 tok/s on vLLM where 200+ was expected, and the fix was engine configuration, not the silicon.

Capacity decides what you can run at all; bandwidth decides how fast. For a mixture-of-experts model those two split cleanly: total parameters set the memory you need, active parameters set the speed. That is why 512 GB of slow DDR4 was a better buy for me than 24 GB of fast GDDR6X. It holds 750B-class mixture-of-experts models such as GLM-5.3 at 4-bit, 386 GB according to CanIRun, which no single GPU can. The CPU side of the build cost about €3,340. If it generated 7.5 tok/s around the clock and I would otherwise pay median API prices, it would pay for itself in about two years, electricity at 400 W included. That is a sanity check, not the reason. I built it for privacy and to understand and control the full stack.

The three are not competitors; they sit at different points on the same bandwidth-versus-capacity chart. A GPU gives you 1–3.4 TB/s but 24–96 GB. A server CPU gives you roughly 190–460 GB/s and half a terabyte or more. Unified-memory machines — Apple M-series, Strix Halo, DGX Spark — sit between: 128–512 GB at 256–819 GB/s, with the convenience of one address space and the ceiling of soldered memory you can never upgrade.

NPUs: the 50 TOPS in a Strix Halo are real, but decode is bandwidth-bound and the NPU shares the same 256 GB/s with the CPU and the integrated GPU. It helps power efficiency for small models and prompt processing; it does not change which models you can run or how fast a 27B dense model decodes, and engine support trails CUDA by years. Most local engines target CPU and GPU first, so for now I do not count NPU TOPS when choosing hardware.

Engine choice follows the hardware. On a tight card, llama.cpp: runtime overhead in the hundreds of megabytes leaves room for context, and GGUF kernels are excellent at batch size one. With headroom, vLLM: continuous batching, paged KV cache, an OpenAI-compatible API, tool and reasoning parsers — but budget 3–5 GiB for the CUDA context and torch.compile; I measured 14.5 GiB of overhead with CUDA graphs on a 122B AWQ model. Three lessons from my notes: measure throughput with a direct curl before blaming the model, because streaming UIs understate the backend; when CUDA graphs do not fit, keep the model and use multi-token prediction in eager mode instead of downgrading; and never load community checkpoints with --trust-remote-code, which executes arbitrary Python from the repository.

Open weights moved fast this year, and the shape changed more than the size. The table compiles public figures from CanIRun (sizes at Q4_K_M, context, the Artificial Analysis score it displays, with the index version unverified) and adds my own column: the decode ceiling from the arithmetic above, on my two machines. Ceilings are upper bounds at short context, not measurements.

What I read from it. Small active counts are the story of 2026: three of these models fit a 24 GB card at Q4 with room for a modest context. Dense models are back at the small end: Qwen3.8 27B has about a quarter of gpt-oss-120b's Q4 footprint. The AA scores shown by CanIRun lack a verified index version, so I would not rank the two on those numbers alone. It also uses hybrid attention: its config.json lists 48 linear-attention layers and 16 full-attention layers, and only the full ones keep a KV cache. That is 64 KB of growing KV cache per token at fp16, so 256K tokens cost about 17 GB of KV instead of the 69 GB a cache in every layer would take. Fixed recurrent state, scratch buffers and other runtime allocations are extra. Ceilings above a few hundred tokens per second mean "fast": past that point kernels, attention and sampling set the pace, not the memory. At the top end, frontier open weights have outgrown home hardware: my 512 GB box holds GLM-5.3 at Q4 with a ceiling around 6 tok/s, and Kimi K3 does not fit anything I own.

What I plan for, with a public example of each:

1. Exposed inference servers. Local inference servers are easy to expose by accident. Ollama and LM Studio bind to localhost by default, but one changed bind address or a careless proxy publishes an API with no authentication. Over 293 days, SentinelOne and Censys observed 175,108 unique exposed Ollama hosts in 130 countries, about 23,000 of them persistent. A Kaspersky decoy pretending to be such a server was indexed by Shodan within three hours and took 113,000 requests in a month, including probes for .env files and /.well-known/mcp.json. Unglamorous, and the easiest one to prevent.
2. Prompt injection through content. The agent reads a page, an issue, a README, a transcript; the text contains instructions; the model follows them. LLMs cannot reliably separate the control plane from the data plane, and no system prompt fixes that. Simon Willison's lethal trifecta names the fatal combination: private data, untrusted content, and a way to communicate out.
3. Hallucinated and malicious dependencies. A USENIX Security 2025 study of 16 models found that 19.7% of 2.23 million suggested packages did not exist. When the authors re-ran prompts that had produced a hallucination, 43% of the invented names came back in all ten runs, which makes them predictable enough to pre-register. I started a beam-search project precisely to enumerate the names a model is most likely to invent.
4. Model files and loaders. Pickle checkpoints execute code at load; in February 2025 two Hugging Face models hid a reverse shell in deliberately broken pickles that slipped past the Hub's scanner. --trust-remote-code is the same hole with a flag.
5. Self-approval. An agent that can edit its own tests, CI configuration or permission policy has a short path to a green check that does not involve solving the task. As my design notes put it, otherwise the AI eventually discovers that allowEverything: true is a convenient way to solve its task. The fix is structural: what defines "passing" lives where the agent cannot write.
6. Persistence and lateral movement. Shell rc files, git hooks, cron, cached credentials, package caches: anything writable that outlives the task becomes the next task's attacker.

None of these need a bad model. A well-behaved model given untrusted input, a credential and a network can still be steered by whoever places a paragraph in front of it. So the useful question is not whether a model is trustworthy, but what its bytes can touch.

"Local" is a claim about the whole machine, not about where the weights sit. The weights are the top of a stack that starts in a management controller with its own network port and firmware you will never see the source of. Every layer below has a published advisory or paper showing how it can go wrong, and most were fixed soon after disclosure. That is the system working. In a home lab your part is to be the one who receives the patch: nobody else is going to update your BMC.

I don't harden every layer equally. I decide per layer whether I can rebuild it from source (OS, runtime, engine: yes, with Nix), prepare it for verification (boot chain: unified kernel images built; Secure Boot enforcement and TPM measurement still planned), or merely contain it (firmware, GPU driver: yes, by limiting what it can reach). The stack below says what I do at each layer, links to the file where I do it, and marks what is still planned.

Three things can be hostile at once: the code the agent writes, the agent itself after reading something poisonous, and the toolchain that builds and runs the result. So code, agent and build/runtime have to be mutually distrustful. The path is agent → source tree → builder → artifact → deployment → runtime, and every arrow is a trust boundary where the next stage should receive explicit capabilities, not ambient authority. A shell is approximately doAnything(string): it grants broad authority within whatever boundary contains it, so the boundary is where the real decision is made.

My design has three domains, each with its own Unix user, directories at mode 700, and an encrypted ZFS dataset mounted nodev,nosuid,noexec. Public can reach the internet freely and holds nothing I would miss. Private holds my data and code and is meant to reach the internet only through a content gateway (next section). Secret has no internet at all, gets encrypted caches and, as planned, SEV-ES memory encryption, and shares files with nobody. Isolation is also something to test, not to assume; the invariants I have written down to test are plain assertions: the VM cannot send or receive outside its WireGuard tunnel, cannot execute code on the host, cannot read host memory through DMA on the passed-through GPU, and cannot log in to the gateway.

A prompt-injected agent with code execution and unrestricted outbound networking can misuse both: send spam, abuse someone else's API, open a reverse shell, or send data out. Filtering URLs helps little, because the agent needs to read arbitrary pages to be useful. What it does not need is direct network access or anything that writes. So the design I am working on is a content-only gateway. I call it a diode, but only as a nickname: a real data diode lets data travel in one direction only, and here queries go out and content comes back. The agent asks for content by type and identifier — /search?q=, /article?url=, /youtube?id=, /github?repo=, /wikipedia?title= — and a narrow adapter per source fetches it and returns extracted text or structured JSON. Never raw HTML, never binaries, never cookies or sessions, no write operations, and no general fetch endpoint. The gateway sits on an internal network; only it reaches outside. Still to specify: which destinations, redirects and parameters /article may follow.

Packages need the same care. A read-only mirror is not enough on its own: a typical proxy repository fetches an uncached package from upstream and serves it. The plan is a curated repository instead: approved versions with pinned hashes and no upstream fallback, so unapproved packages fail closed. Even then, an approved name can ship a compromised new version, which is why versions are pinned too.

What the gateway does not do: retrieved text remains untrusted and can carry an injection, and the queries themselves can carry data out. Keeping credentials in the offline Secret domain protects those credentials; the private code and documents in Private need their own disclosure controls.

The same idea runs through everything else. Capabilities, not ACLs: a stage receives a signed upload URL, not storage credentials; a task-scoped token, not my GitHub account. Control plane separate from execution plane: the agent requests, a broker outside its reach decides, and the policy file is something the agent can read but never write. In my design, credentials live in the Secret domain, which has no internet, so for them the trifecta cannot complete. And the reviewer should not inherit the generator's context: the session that wrote the code should not be the one that approves it.

Secrets also taught me that a filter is not a boundary. Hermes Agent, my daily agent, redacts anything that looks like an API key from tool output by default, and its pattern list matches the sk- prefix that OpenAI and other providers use for API keys. The public key of my hardware-backed YubiKey SSH key starts with sk-ssh-ed25519@openssh.com, as you can see in my flake. So the agent kept breaking my NixOS config while telling me it was fine: in what it showed me and in the edits it wrote, the key came out masked. The redactor did its job; erring towards hiding is the safe direction. But a lossy filter between an agent and the truth fails silently. Keep secrets out of reach structurally, and check results against the source of truth, not against the agent's view of it.

To check it is not just a memory, I re-ran the line from my flake through Hermes' own redact_sensitive_text at commit 74195d3 (28 October 2026):

in:  yubiSshKey = "sk-ssh-ed25519@openssh.com AAAAGnNrLXNzaC1l…";
out: yubiSshKey = "***@openssh.com AAAAGnNrLXNzaC1l…";

The gateway is a crawler turned inside out: the same fetch–extract–normalise pipeline a search engine runs at scale, pointed at one consumer, with the write path removed and, in the plan, every request logged and rate-limited. Most of the engineering is in the extractors, which is the unglamorous half of crawling that decides whether the model sees an article or a cookie banner.

My lab is two monorepos. tigor-ai is autonomously edited by an agent under my direction; its README says plainly that it is not a security boundary or a source of truth, and that control is retroactive, with traceability enforced by a linear git history; the agent commits under its own identity, enovikov11-ai-agent. tigor-no-ai requires human review for every commit and holds the authoritative specifications and the security-critical code: compartmentalisation, virtualisation, ACLs. The split is not a comment on quality. It is a statement about where a hostile byte can originate and how far it can travel before a human looks at it.

Everything deployable is Nix. The host and each VM are built with nix build into unified kernel images — kernel, initrd and command line in one EFI binary — so a running machine is a function of a flake and a commit hash, not of whatever was typed into a shell last year. That buys rollbacks, and it buys something more important for agents: a diff of the machine is a diff in git, reviewable with the same tools and the same budget as the code. The remaining gaps are on my list in the open: Secure Boot and TPM measurement, an independent builder to cross-check digests, rejecting unsigned commits, secrets scanning, and automated checks of the isolation invariants.

Content-address everything and bind approvals to digests, not to names. "Approve main" means approving whatever main points to by the time the deploy runs — a classic time-of-check/time-of-use hole. "Approve sha256:…" means exactly one artifact, and an independent rebuild that produces the same digest is the cheapest provenance check there is.

Twice this summer I ran a coding agent against my own monorepo with no internet, a local model, ten parallel task slots and a quality rulebook, for about seven hours each time. Across the two runs I reviewed 22 branches. I accepted one. Five were "maybe". Sixteen I rejected: useless guard clauses, defensive checks on paths that cannot fail, a bash "fix" that fixed nothing, one change that broke the code. These runs exposed a process problem: I generated more low-value work than I had a plan to review. Autonomy without a review budget produces noise, and noise with a commit bit is a liability.

So my instructions optimise for the reader, not the writer. The rules in my AGENTS.md are short and mostly about attention: a reviewer should be able to understand a diff's purpose in ten seconds, or it gets broken up; validating that it is correct can take longer. A 200-line script that runs unattended beats a 20-line script that needs seven approval prompts. Never destroy data. Trust escalation is non-negotiable; if the task needs more access than it has, stop and say so. "Never hallucinate NixOS option or package paths." And for the bloat that models produce by default — the AbstractJsonReaderFactory for a ten-line task — the single most effective line I have found is: "Implement only the requirements explicitly stated. Treat unspecified future requirements as nonexistent." Ask for the simplest solution first, then separately for the list of things that would force a refactor. The second list is usually short, and usually never happens.

Attention is also the honest limit on autonomy. Each rung up the isolation ladder in section 05 buys permission to review less often, because the blast radius of an unreviewed mistake shrinks. That is the whole argument of these notes in one sentence: isolation is what converts cheap generation into something a single person can afford to supervise.