Latent reads your model's internal state as it writes and holds the answers it is making up before a customer sees them.
Backed by Y Combinator
As your model writes each word, Latent reads what is happening inside it. When the answer is finished, it sends it where your policy says. Pick an answer and a policy, and watch it run.
You can. It is slow, it is expensive, and every answer has to leave your environment to be checked. Latent reads the model that wrote the answer, so it checks output validity in real time.
One trip is one answer checked: 3.4 seconds for a judge, 33 ms for Latent.
Let Latent choose which answers the judge reads. The judge bill falls tenfold, and more of what gets flagged is really wrong.
Judges, guardrails and dashboards look at what your model said. Latent reads the model while it writes, and it works alongside all of them if your team wants to keep them.
Latent sits at runtime, between the input and the answer, where the rest of your stack can't see. Keep your judge, guardrails and traces on the output. Latent feeds them through a Prometheus endpoint, OpenTelemetry trace ids and labelled answers for fine-tuning.
Each flag points to the sentence and the words that drove it, the sentence in your source the answer leaned on most, and the kind of failure, such as an unsupported number, a wrong date or an invented name. Your reviewers know where to look.
Slack, PagerDuty, email or a webhook, when flags spike or stop, confirmed accuracy drops, scoring slows, the review queue backs up or your traffic drifts.
Latent joins your serving stack. A preflight checks the box, and Latent reads every answer from then on. Your prompts and answers stay on your hardware.
Latent keeps a sample of what your model actually served. For calibration, an opt-in judge labels it against your sources.
Latent studies that sample and learns what a wrong answer looks like in your domain. The run takes minutes, and the update goes live without a restart.
Held answers wait in a queue. Your reviewers release them or confirm the failure, and every decision lands in the audit log.
docker compose -f deploy/docker-compose.yml up -d ✔ Container deploy-service-1 Started ✔ Container deploy-vllm-1 Startedlatent doctor ✓ plugin_entry_point latent -> latent_vllm.plugin:register ✓ gpu 1 GPU (NVIDIA A100-SXM4-40GB) ✓ LATENT_CAPTURE tap (in-graph capture; CUDA graphs stay on) ✓ sink_reachable http://service:8080/events ✓ artifact Llama-3.1-8B-Instruct_rag_mean.joblib OK to servedocker compose logs vllm | grep 'latent_vllm ACTIVE'latent_vllm ACTIVE: model=meta-llama/Llama-3.1-8B-Instruct scoring=on capture=tap stream=on
The sample stays on your box: at most 5,000 scored requests per calibration, kept by the plugin itself.
latent calibrate --from-service http://localhost:8080 --work calib/
Each decision your reviewers make is kept as a labelled example. Latent learns from them, and the more answers your team validates, the better the data you post-train your model on.
Latent recalibrates on your reviewers' decisions while it runs, with no restart. Every validated answer can also be exported as fine-tuning or preference data for your model.
Nothing is sent to Latent. No prompts, answers, activations, weights or telemetry.
Sees what it sees today.
Receives nothing.
The only outbound calls are ones you configure yourself: the model download, an alert webhook, a stronger model you name.
Latent never receives or copies model weights, and calibration does not need them.
Each scored request writes a 1.7 KB append-only record with hashes of the prompt and answer, never the text.
Anything else, ask us. Get in touch
One GPU in your environment for Latent's reader model, the smallest of which fits in about 10 GB, and Latent's gateway in front of your API calls. Latent can hold, replace or reroute complete answers. Answers you stream to the customer as they are written are scored and flagged, but they cannot be held.
Any model, including closed-weight models behind an API, which Latent checks with its own reader model. With open-weight models served on vLLM, Latent reads your model directly, which adds per-token risk and the ability to act while the answer is being written. It is measured in serving on Llama-3.1-8B-Instruct and Hermes-3, and it runs beside SGLang too.
The GPU already serving your model. The plugin adds 16 MiB of GPU memory at idle and at most 58 MiB at peak, whatever the prompt length. Every number on this site was measured on NVIDIA A100 40 GB or H100 GPUs.
Only when you tell it to. Out of the box Latent scores and records every answer. Holding, replacing or rerouting flagged answers is a policy you switch on in the review console.
Reviewers working from Latent's flags find 2.7 to 4.2 times more wrong answers than checking at random, and when it checks other models' answers, 96 in 100 of the answers it flags are really wrong. Calibrating on your own traffic is what moves you to the top of that range.
Five to eight minutes for one run over about 1,000 answers, for $6 to $8 of judge calls on your own key. The new calibration goes live within six seconds, with no restart.
Yes. After the model download on first start you can block all egress from the host, and the stack keeps running.
A drift alert watches the scores. It stays silent on steady traffic and fires within 120 to 181 answers when a shift matters, so you recalibrate when you need to.