Reference
Python SDK
Every option of the runlatent package: setup, per-client wrappers, environment, what is captured, streaming, redaction, counters and limits.
The runlatent package captures every finished call your app makes through the OpenAI, Anthropic, Google GenAI, or Ollama Python SDK and posts the prompt and answer to Latent's scoring API. Until you change When Latent flags an answer from Deliver it and log it on the console's Policy page, posting happens after the provider's SDK returns, on a background thread, so your app's call is never slower by a network round trip and never sees a different return value. It never gets an exception from Latent.
Install
pip install runlatent
The package has no runtime dependencies. The provider SDKs are yours: whichever of openai, anthropic, google-genai and ollama is importable gets patched, and the rest are skipped.
auto_instrument
import runlatent
runlatent.auto_instrument() # reads LATENT_API_URL and LATENT_API_KEY
Call it once at startup. The patch applies to the SDK classes, so every client in the process is covered, including clients a framework builds for you.
| Argument | Default | Meaning |
|---|---|---|
openai, anthropic, google_, ollama |
True |
Which SDKs to patch |
api_url |
$LATENT_ |
The API base. The SDK posts to <api_ |
api_key |
$LATENT_ |
Sent as Authorization: Bearer <key> |
redact |
None |
A function run on every pair before it is queued. See Redaction |
timeout |
5.0 |
Seconds per post |
max_queue |
10000 |
Pairs waiting to be posted. A full queue drops new pairs and counts them |
enforce |
None |
Apply the console policy (below). False never changes a response |
stream_ |
None |
Follows the console's Scoring setting (runlatent 0.3.5 or later). True or False pins the streaming read on or off |
Calling it twice is safe: nothing is patched twice. A second call with a different URL, key, timeout, or queue size switches to the new settings and drains the old queue in the background. With no URL configured anywhere, auto_ logs one warning, patches nothing, and returns stats() with enabled: False.
runlatent. restores every original method and wrapped client.
Hold an answer before your user sees it
runlatent. scores one answer and returns its verdict before you return the answer, so your code decides what your user gets. It needs runlatent 0.2.0 or later.
import runlatent
r = runlatent.check(prompt, answer, context=docs)
if r.flagged:
answer = SAFE_FALLBACK
elif not r.scored:
... # Latent could not score it in time: deliver the answer, or hold it
contexttakes your source documents as one string. On hosted plans, an answer checked with no source (nocontext, nosystem, and no system turn inmessages) gets no grounding verdict. The no-source head, on by default, gives it a verdict of its own. With the head switched off,r.scoredisFalse, andr.detailsays so. See Give Latent your sources.system,messages,model,timeout(2 seconds by default) andrequest_idare optional.- The result carries
risk,verdictandrequest_id.r.flaggedis true for anescalateoroodverdict, andr.passedfor a scoredpass. check()never raises when Latent refuses or times out. A 401, 403, 429, a 5xx or a timeout returnsscored=Falsewithstatusanddetail, so you choose to fail open or fail closed. Passraise_to get an exception instead.on_ error=True - With no URL or key configured,
check()raisesConfigurationError. acheck()is the async version.- With
auto_on, a checked answer appears once in the console when you pass the provider's response id asinstrument() request_id(for examplerequest_) or the conversation asid=resp. id messages=. Without either, the check is scored on its own, so the answer appears twice and counts twice toward your plan.
Apply your console policy
auto_ also applies what you choose under When Latent flags an answer on the console's Policy page. While it reads Deliver it and log it, the default, nothing changes: every call returns as the provider sent it, and the SDK only observes. With a Hold, or the judge or regenerate under Other actions, each complete, single-choice answer is scored before the call returns, within the policy's wait (500 ms by default), and a flagged answer gets the action below. Hold it and say is abstain and Hold it and ask a stronger model is route. The SDK reads the policy every 60 seconds, so a change in the console reaches your app within a minute, and calls made before the first read are only observed.
| Action | What your app receives |
|---|---|
abstain |
The provider's own response object, with the policy's fallback text in place of the answer. Unless the policy sets one, the text is "I can't answer that reliably right now; a human will follow up." |
route |
The answer from the model set as the policy's route_, on the same provider, through your own client and key. Without one, the answer is delivered unchanged with its verdict. A route_ is a bare model name, or provider:model when the prefix is a provider the SDK patches, so an Ollama tag such as llama3. works as written; write ollama:cohere:7b for a tag whose name is also a provider's |
regenerate |
A second answer from the same model. It is delivered without a second check and scored afterwards as its own row |
judge |
The answer unchanged, queued for the judge's written analysis |
If the verdict does not arrive in time, abstain delivers the fallback text, and the other actions deliver the answer unchanged with the verdict unknown. If a regenerate or route call fails, the original answer is delivered. The policy's fail_ setting can change either rule. Requests for several answers are delivered unchanged, with the verdict recorded. A streamed answer is never held: under a Hold the SDK cuts it at the first unsupported sentence, and without the streaming read it is delivered unchanged with the verdict recorded. The SDK never raises into your app, and every call returns the provider's own type. It covers OpenAI, Anthropic, Gemini, and Ollama clients. The decision rides on OpenAI and Anthropic responses as response.; Gemini's and Ollama's response objects take no extra fields, so read it there with runlatent.. To keep the SDK observing only, pass auto_ or set LATENT_. Needs runlatent 0.3.0 or later, 0.3.2 for Ollama and 0.3.6 for per-model overrides (pip install -U runlatent).
Per model (runlatent 0.3.6). The SDK acts on the policy of each call's model: the default with that model's override on top. An override is in force from the first call that names its model. If a model's override cannot be read, the call waits up to the policy's wait once, then the default decides and the answer is marked pending_. stats()["enforce"]["last_ reads default or model:<id>. runlatent 0.3.5 and earlier read the default alone, so a Hold saved only on a model's override is not applied. When a provider's response object cannot carry the replacement, the SDK delivers the answer and reports it as released: true, replaced: false, and the console shows Hold failed (runlatent 0.3.7).
Group the turns of a call
runlatent. tags every captured call and every check() inside the block, so the console groups them:
with runlatent.session(call_id, user_id=caller_hash, label="support line"):
... # every model call in the conversation
It works as a decorator too, with an id taken from the call's arguments. check() and acheck() also take session_ and user_id=, which win over the block. A call keeps the session it was made in, so a call created in one block and awaited in another is filed under the first. The ids are stored as given, so hash personal data first. In the console, the queue offers a Session column, a Session filter and Group by session, and the request page header reads "Session <label> · caller <user> · turn <n>".
Per-client wrappers
To capture one client and leave the rest of the process untouched:
client = runlatent.wrap_openai(OpenAI()) # or AsyncOpenAI()
client = runlatent.wrap_anthropic(Anthropic()) # or AsyncAnthropic()
client = runlatent.wrap_genai(genai.Client()) # sync and .aio
client = runlatent.wrap_ollama(ollama.Client()) # or ollama.AsyncClient()
Each returns the client. Wrapping twice does nothing more, and a wrapped client under a patched class is captured once.
Environment
| Variable | Meaning |
|---|---|
LATENT_ |
The API base: https://, or your sidecar's address when Latent runs in your environment |
LATENT_ |
Your API key (lk_...), or the sidecar's token when Latent runs in your environment |
LATENT_, LATENT_ |
Used when the two variables above are unset |
LATENT_ |
Seconds to wait at a normal exit for queued pairs. Default 2.0; 0 turns the wait off |
LATENT_ |
off keeps the SDK observing only, like auto_. An explicit enforce= wins |
LATENT_ |
0 stops the inferred session_ (What Latent receives) |
What is captured
| Provider | Calls | Modes |
|---|---|---|
| OpenAI | Chat Completions and the Responses API, including chat. and responses. |
Sync, async, and streaming |
| Anthropic | messages. and messages., including the .stream() manager |
Sync, async, and streaming |
| Google GenAI | generate_, generate_, and chats through chats. |
Sync, async, and streaming |
| Ollama | chat and generate, on the ollama module and on Client and AsyncClient (runlatent 0.3.1 or later) |
Sync, async, and streaming |
It also captures the Groq, Together, Fireworks, Mistral, Cohere, Azure AI Inference, Hugging Face InferenceClient, and boto3 Bedrock SDKs.
OpenAI-compatible servers. vLLM, LM Studio, and other servers that speak the OpenAI API, Ollama's /v1 included, are captured through the openai client pointed at them with base_url, for example OpenAI(base_. A client the SDK does not patch, such as in-process transformers generation, is scored one answer at a time with runlatent..
Each finished call becomes one pair: the request's messages, system prompt included, and the whole answer text. The What Latent receives page lists every field. An answer with no text, such as a tool call only, is counted and not posted. A failed call posts nothing, and its exception reaches your app unchanged.
Streaming
A stream comes back through a thin wrapper that yields the same items and passes every other attribute through. The pair is posted once, at the end:
- Read to the end. The chunks are joined into the full answer, with the final finish reason and token usage.
- Closed early by your app (the stream wrapper's own
close(), or the end of awithblock on it). The partial text is posted withfinish_andreason: "abort" client.. If no text had arrived, nothing is posted (stream_ complete: false stream_).abandoned_ empty - Left unread without closing. Nothing is posted; the stream is counted as
stream_. Leaving OpenAI'sunfinished chat.orcompletions. stream() responses.helper early counts here too: itsstream() withexit closes the HTTP response and leaves the stream itself unread. - Failed mid-way, for example an overloaded error, or an exception your app raised inside the
withblock. Nothing is posted; the stream is counted asstream_.errored
With n greater than 1, only the first choice or candidate is posted.
The streaming read (hosted). The SDK feeds each stream to Latent's reader as your app reads it while the project's policy turns the read on: either Hold, or Advanced > Scoring: sentence by sentence while streaming. No code flag is needed in runlatent 0.3.5 or later, and auto_ or False pins it on or off. runlatent. returns the live read of the stream your code is iterating. The SDK passes every chunk through as it arrives, and ends a stream early only when the project's stop is on, which either Hold turns on. See Streaming and complete answers.
Finish reasons
Each provider's finish reason is mapped to one vocabulary, so the console's review rules read the same way for every model. The provider's own word is kept alongside it.
| Posted | OpenAI Chat | OpenAI Responses | Anthropic | Gemini |
|---|---|---|---|---|
stop |
stop |
completed |
end_turn, stop_ |
STOP |
length |
length |
incomplete:max_ |
max_tokens, model_ |
MAX_TOKENS |
tool_calls |
tool_calls, function_ |
tool_use |
MALFORMED_ |
|
content_ |
content_ |
incomplete:content_ |
refusal |
SAFETY, RECITATION, BLOCKLIST, PROHIBITED_, SPII, IMAGE_, LANGUAGE |
abort |
your app closed the stream early | cancelled, in_, queued |
||
| passed through | pause_turn |
OTHER as other, FINISH_ as unknown |
A Responses answer with status failed and a Gemini prompt blocked before any answer are not posted: there is no answer to score (skipped_, skipped_). Any other word passes through lower-cased, cut to 32 characters. A missing finish reason is left unset, and the service marks it finish_.
Under a new project's default review rule (finish_), an answer cut off at the length limit goes to the review queue whatever its score. A deployment whose stored policy predates the rule reads [] until an admin sets it.
Redaction
By default the SDK posts the text exactly as your app sent and received it. Masking changes what Latent reads, so mask only if you must, and mask the prompt and the answer with the same table of names:
def redact(body):
table = my_entities(body["messages"]) # names from the source, one table
body["messages"] = [dict(m, content=mask(m["content"], table)) for m in body["messages"]]
body["output"] = mask(body["output"], table) # the same table on the answer
return body # or None to skip this pair
runlatent.auto_instrument(redact=redact)
The hook sees the full body and can rewrite it or return None to drop it. If the hook raises, or returns anything other than a dict or None, the pair is dropped and counted (redact_), and the unmasked pair never leaves the process.
With Hold it and say chosen (abstain), an answer whose pair your hook drops or fails on is held, and your app gets the fallback text.
Counters and shutdown
runlatent.stats()
# {"enabled": True, "captured": 120, "posted": 118, "failed": 0, "dropped": 2, "queued": 0,
# "skipped_empty_output": 3, "multimodal_parts_dropped": 7, "redact_skipped": 1,
# "errors": {...}, ...}
runlatent.flush(timeout=5.0) # in your shutdown path
capturedcounts pairs assembled.posted,failed,droppedandqueueddescribe delivery.droppedincludes pairs captured while no destination was set.skipped_*andstream_*count calls that were not posted, and why.errorscounts failures by stage and exception class. None of them ever reached your app. Only the class is logged, never the message, so answer text cannot leak into your logs through an error.
A retriable failure (a 5xx, a 429, or a network error) is tried up to three times, 0.5 seconds and then 2 seconds apart. A 4xx other than 429, such as a 401 for a bad key, or a redirect is final after one attempt. After a pair exhausts its attempts, the destination is held down for 5 seconds, and pairs in that window are lost without an attempt and counted. There is no on-disk spill: a pair that cannot be delivered is lost and counted.
Limits
chat.,completions. parse() responses.,parse() beta.*, and the Assistants and Realtime APIs are not captured.- A pair over 1 MiB is refused (413) and counted as failed. A Free key scores up to 10 answers a second; above that the API answers 429
rate_.limited - Pass arguments by keyword. A positional
messagesis not read, except in Ollama'schat(model, messages)andgenerate(model, prompt), which are read either way. with_andraw_ response. create() with_are not captured; they are counted asstreaming_ response. create() skipped_.raw_ response - The stream wrapper is not an instance of the SDK's
Streamclass, soisinstancechecks on it fail. Attribute access works. max_queuecounts pairs, behind one posting thread. 10,000 queued pairs with large retrieval prompts can hold about a gigabyte in memory, so lower it on memory-tight hosts.- A
kill -9oros._exit()loses queued pairs. Callrunlatent.in your shutdown path when that matters.flush()
Note Gemini thinking models count their thought tokens against
max_. A small budget produces empty answers, which are not posted, and truncated answers, which are posted asoutput_ tokens lengthand, under a new project's default review rule, routed to review. Give thinking models a budget that covers the thinking, or turn thinking off.
Note OpenAI's
chat.can raisecompletions. stream() LengthFinishReasonErrorafter the stream has finished. The pair is still posted, aslength, and your app still gets the exception.
Was this page helpful?
Updated 3 October 2026