LLM Observability: What to Monitor in Production AI Systems

Introduction

A RAG pipeline that returns great answers in testing can start hallucinating in production within days, and nobody notices until a user complains. An AI agent that worked fine on ten sample tasks can start looping, calling the wrong tool, or burning through your API budget on a single bad prompt. Traditional application monitoring - latency, error rates, uptime - won't catch any of this, because the failure isn't a crash. It's a wrong answer that looks confident.

LLM observability is the practice of watching what your AI system actually produces, not just whether it runs. This article covers what to monitor, why standard APM tools fall short, and how to build a monitoring approach that actually catches the failures LLM applications tend to have.

What is LLM observability?

LLM observability extends traditional application observability (logs, metrics, traces) with signals specific to language model behavior: the prompts sent, the completions returned, token usage, latency per call, and - critically - some measure of output quality.

The distinction matters because LLM failures are usually silent. A traditional service either responds correctly or throws an error. An LLM almost always responds - it just might respond with a fabricated fact, an irrelevant answer, a tool call with malformed arguments, or a response that ignores half the user's question. None of that shows up as a 500 error. If you're only watching uptime and response time, a production LLM app can be "healthy" by every metric while quietly failing its users.

Observability for LLM systems generally covers four layers:

1. Request/response logging - what went in, what came out, and the full context (retrieved documents, tool results, system prompt) that shaped the answer.
2. Performance metrics - latency per call, token counts, cost per request, and throughput under load.
3. Quality signals - automated evaluation scores, user feedback (thumbs up/down, corrections), and drift in output patterns over time.
4. Traces - for multi-step systems like AI agents or RAG pipelines, a trace shows the full chain: retrieval → reranking → generation → tool calls → final output, so you can see exactly where a bad result originated.

What to monitor in production

Input and output logging. Log every prompt and completion, along with the metadata needed to reconstruct why the model answered the way it did: retrieved context, conversation history, model version, and any system-level instructions. Without this, debugging a bad response after the fact means guessing.

Token usage and cost. Track tokens per request, broken down by prompt vs. completion, and aggregate by endpoint or feature. Cost tends to creep upward as prompts accumulate few-shot examples, longer context windows, or verbose system instructions that nobody trims. A cost spike is often the first visible sign that something changed upstream - a longer retrieval context, a runaway agent loop, or a prompt template that grew unchecked.

Latency, broken down by stage. For a single LLM call, track time-to-first-token and total generation time separately - users perceive streaming latency very differently from batch latency. For multi-step systems, measure latency per stage (retrieval, reranking, generation, tool execution) so a slowdown in one component doesn't get blamed on the whole pipeline.

Output quality signals. This is the hardest and most important category. Options include:

• Automated evaluation: running a subset of production outputs through a scoring model or rule-based checks (does the answer cite a source, does it stay within the requested format, does it avoid banned topics).
• User feedback: thumbs up/down, explicit corrections, or session abandonment as an implicit negative signal.
• Groundedness checks: for RAG systems, verifying that claims in the output are actually supported by the retrieved documents, which catches hallucination even when the answer reads fluently.
• Consistency checks: sampling the same input multiple times and flagging high variance, which often indicates the model is guessing rather than retrieving a stable answer.

Tool call and agent behavior. For agentic systems, log every tool call: which tool, what arguments, what result came back, and whether the agent retried or gave up. Watch for loop patterns (the same tool called repeatedly with similar arguments), malformed arguments that fail validation, and tasks that terminate without producing a final answer.

Drift over time. Model providers update models behind the same API endpoint, embedding models get swapped, and user query patterns shift. Track output length, refusal rate, and evaluation scores as rolling averages so a gradual change - not just a sudden break - becomes visible.

Common mistakes

Treating "no errors" as "working correctly." An LLM call that returns 200 OK with a hallucinated or off-topic answer looks identical to a successful call in standard monitoring dashboards. Quality has to be measured separately from availability.

Logging outputs but not inputs. A completion is meaningless without the exact prompt, retrieved context, and model parameters that produced it. If you can't reconstruct the full input, you can't debug the output.

Relying entirely on manual review. Spot-checking a handful of conversations a week doesn't scale and misses rare but severe failures. Automated evaluation should run on every request or a statistically meaningful sample, with manual review reserved for flagged cases.

Ignoring cost as a quality signal. A sudden jump in average tokens per request is often a bug - a retrieval step returning too many documents, a conversation history that isn't being truncated, or a prompt template with an unintended loop - before it's a cost problem.

No baseline before launch. Without a recorded baseline for latency, cost, and quality scores from testing, there's nothing to compare production numbers against, and drift becomes invisible until it's severe.

Practical checklist

• Log every prompt, completion, and the context that shaped it (retrieved documents, tool results, conversation history)
• Track latency separately for time-to-first-token and total completion time
• Track token usage and cost per request, aggregated by feature or endpoint
• Run automated quality evaluation on every request or a representative sample, not just spot checks
• For RAG systems, add groundedness checks that verify claims against retrieved sources
• For agents, trace and log every tool call, including arguments and results
• Capture user feedback (explicit ratings or implicit signals like abandonment) as a quality metric
• Set rolling baselines for cost, latency, and quality scores so drift is visible before it becomes a problem
• Alert on quality and cost anomalies, not just error rates and uptime

Conclusion

LLM observability isn't an extension of your existing monitoring stack - it's a different discipline, because the failure mode is different. A traditional service breaks loudly; an LLM breaks quietly, by giving a wrong answer with the same confidence as a right one. Logging inputs and outputs, tracking cost and latency by stage, and layering in automated quality checks turns that silent failure mode into something you can actually see, measure, and fix before users notice it first.

The next step is usually implementation: wiring this into an existing RAG or agent pipeline without slowing it down or adding excessive overhead. The step-by-step guide to setting up LLM observability covers exactly that, with working code for instrumentation, tracing, and evaluation.