How to Set Up LLM Observability: A Step-by-Step Guide

Introduction

Knowing what to monitor in an LLM application is one thing. Actually wiring it into a working pipeline - without doubling your latency or drowning in logs you never look at - is a different problem. This guide walks through building an LLM observability setup from scratch: instrumenting calls, structuring logs, running automated evaluation, and setting alerts that catch real problems instead of generating noise.

If you haven't yet decided what to track, the companion article on LLM observability covers the signals worth capturing and why. This guide assumes you know the "what" and want the "how."

What is an observability pipeline for LLMs?

An observability pipeline is the plumbing that captures data at each stage of an LLM call - request, retrieval, generation, tool execution, response - and routes it somewhere you can query, visualize, and alert on. For a simple single-call application, this can be a lightweight wrapper around your API client. For a multi-step RAG or agent pipeline, it's closer to distributed tracing: each stage gets its own span, and spans are linked into a single trace per user request.

The goal isn't to log everything indiscriminately. It's to capture enough structured data that you can answer three questions quickly when something goes wrong: what did the user ask, what did the system do at each step, and why did it produce this particular output.

Building the pipeline

Step 1: Instrument the LLM call itself. Wrap every call to the model provider in a function that captures the request (prompt, parameters, model version) and response (completion, token counts, latency) before returning the result to the caller. Keep this wrapper thin - its only job is capturing data, not transforming it.

import time

def call_llm(client, messages, **kwargs):
    start = time.monotonic()
    response = client.messages.create(messages=messages, **kwargs)
    latency = time.monotonic() - start

    log_event({
        "type": "llm_call",
        "model": kwargs.get("model"),
        "messages": messages,
        "completion": response.content,
        "input_tokens": response.usage.input_tokens,
        "output_tokens": response.usage.output_tokens,
        "latency_seconds": latency,
    })

    return response

Step 2: Add trace IDs for multi-step flows. Generate a unique trace ID at the start of each user request and pass it through every function call - retrieval, reranking, generation, tool execution. Attach it to every logged event so you can reconstruct the full chain later with a single query, rather than piecing together timestamps across separate logs.

Step 3: Log retrieval and tool context, not just the final answer. For RAG systems, log which documents were retrieved, their relevance scores, and whether they made it into the final prompt after reranking. For agents, log every tool call with its arguments and result. This context is what makes a bad output debuggable instead of mysterious - without it, you're left staring at a final answer with no way to tell whether retrieval, reasoning, or the tool itself went wrong.

Step 4: Route structured logs to a queryable store. Plain text logs work for debugging one request; they don't work for asking "what's our average groundedness score this week." Send structured events (JSON) to a store you can query - a hosted LLM observability platform, or a simpler setup with structured logs shipped to a data warehouse. Either works; what matters is being able to filter and aggregate, not just grep.

Step 5: Add automated evaluation as a post-processing step. Rather than blocking the user-facing response on evaluation, run it asynchronously after the response is returned. A lightweight approach: sample a percentage of production traffic, or evaluate every request for the more error-prone flows, and score it against criteria specific to your use case (groundedness for RAG, task completion for agents, format compliance for structured outputs). Store the score alongside the original trace.

def evaluate_response(trace_id, question, context, answer):
    eval_prompt = f"""Given this context: {context}
Question: {question}
Answer: {answer}

Is the answer fully supported by the context? Respond with a score 0-1 and one sentence of reasoning."""

    result = call_llm(client, [{"role": "user", "content": eval_prompt}])
    log_event({
        "type": "evaluation",
        "trace_id": trace_id,
        "score": parse_score(result),
    })

Step 6: Capture user feedback where it exists. If your interface has a thumbs up/down or correction mechanism, log it against the same trace ID. Even sparse feedback is useful for spot-checking whether your automated evaluation scores actually correlate with what users think.

Step 7: Set alerts on aggregates, not individual events. Alerting on every low-scoring response creates noise that gets ignored. Instead, alert on trends: average evaluation score dropping below a threshold over a rolling window, token usage per request climbing beyond baseline, or tool call failure rate spiking. Reserve per-request alerts for clear failure signals like unhandled exceptions or timeout errors.

Common mistakes

Blocking user responses on evaluation. Running a full evaluation pass synchronously before returning an answer adds latency the user feels directly. Evaluate asynchronously and review results separately.

Logging raw text without structure. Free-text logs are hard to query at scale. Log structured events with consistent fields from day one - retrofitting structure onto years of unstructured logs is a much bigger project than doing it upfront.

No trace linking across steps. Without a shared trace ID, a bad final answer in a RAG or agent pipeline can't be traced back to whether retrieval, reranking, or generation caused it. This is the single most common gap in early observability setups.

Evaluating with the same model and prompt that generated the answer. A model is more likely to rate its own kind of output favorably. Where possible, use a different model or a stricter, more literal evaluation prompt than the one used for generation.

Alerting on noise. Setting alert thresholds too sensitively (e.g., any single low score) trains the team to ignore alerts. Tune thresholds against a baseline collected during a quiet period before going live with alerting.

Practical checklist

• Wrap every LLM call in a thin instrumentation layer that captures request, response, tokens, and latency
• Generate a trace ID per user request and propagate it through every step
• Log retrieval results, rerank scores, and tool call arguments/results, not just final answers
• Ship structured (JSON) logs to a queryable store, not plain text files
• Run automated evaluation asynchronously, sampled or on error-prone flows
• Use a different model or stricter prompt for evaluation than for generation
• Capture user feedback against the same trace ID when available
• Alert on rolling-window trends (score drops, cost spikes, failure rate increases), not individual low-scoring events
• Collect a baseline during a quiet period before enabling alerts

Conclusion

An LLM observability pipeline doesn't need to be complex to be useful. The core pattern - instrument the call, propagate a trace ID, log context alongside outputs, evaluate asynchronously, alert on trends - covers most of what a single-call app or a multi-step RAG or agent system needs. The payoff is that when something breaks in production, you're debugging with a full trace instead of a single confusing output and no way to explain it.

Once this is running, the observability fundamentals guide is worth revisiting periodically - as your application grows, the signals worth tracking tend to expand.