Latent
Start free

vLLM

Hallucination detection inside vLLM, at no measurable cost.

Latent's plugin loads into the vLLM that already serves your model, reads one layer as it writes and scores every answer, close to word by word. Throughput and first-token time stay within run-to-run noise, and your outputs stay as they are until you turn on early stop.

Book a call Deployment docs
Requests per second, with and without Latent14.56 vs 14.66
Extra GPU memory at peak58 MiB
Requests stored in an 8-hour soak28,800 of 28,800
Changes to vLLM's codeNone

The plugin reads your model as it writes

Load
The plugin loads through vLLM's general_plugins entry point into the stock vllm/vllm-openai image. There is no fork, and vLLM's own code is unchanged.
Read
It reads one layer of the activations your model already computes. Weights, logits, sampling and outputs are untouched.
Score
Every answer carries a live risk while it is written and a verdict when it finishes, written to your audit log.
Act
Turn on early stop to end a risky answer mid-stream, or put Latent's gateway in front to hold, reroute or regenerate a flagged answer. LLM guardrails covers each action.

Latent adds no measurable overhead to vLLM

We benchmark the plugin every night against plain vLLM on the same card. The benchmarks page lists each number with its setup.

None

measurable serving overhead inside vLLM: 14.56 requests per second with Latent against 14.66 without it, and first tokens in 298 ms against 316 ms. The plugin adds 16 MiB of GPU memory at idle and at most 58 MiB at peak.

Nightly benchmark: A100 40 GB, Llama 3.1 8B Instruct, vLLM 0.19.1, CUDA graphs on, 32 requests at a time, three runs of 256 requests each way. September 2026.
5.68

requests per second with 512-token answers, with Latent and without it. Time per output token was 21.77 ms against 21.63 ms, and first tokens on 8,000-token prompts took 1,453 ms against 1,451 ms. Eight probes on the same activations served 5.70 requests per second against 5.67 with none.

The same nightly benchmark: A100 40 GB, Llama 3.1 8B Instruct, vLLM 0.19.1, CUDA graphs on, 32 requests at a time. The full table
Zero

requests dropped over an 8-hour soak of 28,800 requests: every one was stored at every layer of the stack.

A100 40 GB, Llama 3.1 8B, vLLM with the plugin, gateway and review service, one request per second. September 2026.
10,654

scored events, every one accounted for, through service, gateway and engine restarts and a live swap of the read under traffic. A newly calibrated read goes live within 30 seconds without restarting vLLM.

A three-hour chaos run of 10,800 requests on an A100 40 GB. Two events lost in flight at a gateway restart were recovered from the local audit file. September 2026.

Early stop ends a risky answer mid-generation

With early stop on, the plugin ends an answer once its running risk has stayed above your cut, and your client sees vLLM's own length finish with Latent's reason beside it. It is off by default.

{"index": 0, "delta": {}, "finish_reason": "length", "stop_reason": "latent_early_stop"}
80%

of the answers early stop ended were wrong by a strict judge's reading of the full answer. The rule stopped 17% of answers, ended 5.8% of good answers early and saved 15.2% of decode tokens.

Served on an A100 with Llama 3.1 8B, 300 held-out prompts, the default running-mean rule. On a separate run of 1,000 answers, a sentence-level rule that decides later reached 93% at an 8.3% stop rate.

Other engines use Latent's sidecar

For SGLang, models behind an API, or a serving setup that must stay exactly as it is, Latent's sidecar re-reads each finished answer with its own GPU. Beside SGLang it ran with no changes to SGLang and gave the same verdicts as the vLLM plugin.

We install it with your team

The images are built on your host from digest-pinned public bases, and nothing is pulled from Latent. An engineer works through the install with you, and latent doctor checks a new read or a new vLLM version before you serve with it. In your VPC, prompts, answers and activations stay on your infrastructure. Deployment options and the security page cover the details.

Questions

Which vLLM version does it support?
The plugin is verified on vLLM 0.19.1, and latent doctor checks a new version before you serve with it.
Does it change my model's outputs?
It reads one layer and never touches weights, logits or sampling. The one change it can make is ending an answer early, and only when you turn early stop on.
Which models has it run on?
We have measured it serving Llama 3.1 8B Instruct, Hermes 3 and gpt-oss-120b.
Can I use Latent without running my own model?
Yes. For models behind an API, Latent's reader model scores each finished answer, hosted by Latent. How Latent detects hallucinations

Research