Agent Observability

The Problem:

AI agents are black boxes by default. When an agent gives a wrong answer or takes too long, you have no idea why. Was it the prompt? The tool? The model? A hallucination? Without observability, debugging AI agents is like fixing a car engine with the hood welded shut.

Observability gives you X-ray vision into your agent - every LLM call, every tool invocation, every token spent, every millisecond of latency.

Real-World Analogy - CCTV for Your AI:

Think of running a Swiggy dark kitchen without cameras. Orders go wrong but you do not know if the cook made the wrong dish, the packer mixed up orders, or the delivery partner dropped the food. You need cameras at every station.

AI observability puts "cameras" at every step of your agent - what the LLM was thinking, what tools it called, what data it received, and what it decided to do with it.

What Observability Gives You:

• Debugging: See exactly where an agent went wrong in a multi-step workflow
• Cost Monitoring: Track token usage and API costs per user, per task, per day
• Latency Analysis: Identify which steps are slow (LLM calls? Tool calls? Data retrieval?)
• Quality Monitoring: Track response quality over time, catch regressions early
• Usage Analytics: What are users asking? Which tools are used most? Where do agents fail?

What is a Trace?

A trace is a complete record of everything that happened during a single agent execution - from the user's input to the final response. It captures every LLM call, tool invocation, and decision point in a hierarchical structure.

Trace: "Book flight from Delhi to Goa" (total: 4.2s, Rs 6.50)
|
+-- Span: LLM Call #1 (1.2s, 850 tokens, Rs 1.60)
|   Input: System prompt + user message
|   Output: "I need to search for flights..."
|
+-- Span: Tool Call - search_flights (0.8s)
|   Input: {from: "DEL", to: "GOI", date: "2026-03-15"}
|   Output: [{flight: "AI-301", price: 4500}, ...]
|
+-- Span: LLM Call #2 (1.5s, 1200 tokens, Rs 3.20)
|   Input: Previous context + flight results
|   Output: "I found 3 flights. Cheapest is AI-301..."
|
+-- Span: Tool Call - book_flight (0.5s)
|   Input: {flight: "AI-301", passenger: "Rahul"}
|   Output: {booking_id: "BK12345", status: "confirmed"}
|
+-- Span: LLM Call #3 (0.2s, 300 tokens, Rs 1.70)
    Output: Final response to user

Key Concepts:

• Trace: The top-level container for an entire agent execution. One user request = one trace.
• Span: An individual operation within a trace (LLM call, tool call, retrieval). Spans can be nested.
• Metadata: Additional data attached to spans - model name, token count, cost, latency, user ID.
• Events: Point-in-time markers within spans - "user approved action", "error occurred".

OpenTelemetry for AI:

Just like microservices use OpenTelemetry for distributed tracing (Jaeger, Zipkin), AI agents are adopting similar standards. Many observability tools now support OpenTelemetry-compatible instrumentation, making it easier to integrate AI tracing with your existing monitoring stack.

What is Langfuse?

Langfuse is an open-source observability platform specifically designed for LLM applications. It provides tracing, analytics, prompt management, and evaluation - all in one tool. Think of it as Datadog but specifically for AI agents.

Langfuse Key Features:

• Traces and Spans: Hierarchical view of every LLM call, tool call, and retrieval. Click on any span to see full input/output.
• Cost Dashboard: Track token usage and costs per model, per user, per time period. Set budget alerts.
• Prompt Management: Version your prompts, A/B test them, and see which version performs better. Prompts become deployable assets, not strings in code.
• Evaluation: Built-in LLM-as-Judge scoring, custom evaluation functions, and human annotation workflows.
• Datasets: Create evaluation datasets from production traces. Turn real user conversations into test cases.

Langfuse Integration:

Langfuse integrates with all major frameworks:

• LangChain, LlamaIndex - automatic instrumentation
• OpenAI SDK - drop-in wrapper
• Vercel AI SDK, LiteLLM, Flowise, Langflow
• Custom agents - manual tracing SDK (Python, JS)

Self-host with Docker or use the cloud-hosted version with a generous free tier.

What is Arize Phoenix?

Arize Phoenix is an open-source AI observability tool that focuses on traces, evaluations, and experiments. It excels at helping you understand WHY your agent behaves the way it does and how to improve it systematically.

Phoenix Key Features:

• Trace Visualization: Beautiful waterfall view of agent traces. See timing, tokens, and cost at every level.
• Span Analysis: Filter and analyze spans by type (LLM, retriever, tool), model, latency, or error status.
• Evaluations: Run LLM-as-Judge evals on production traces. Score for relevance, toxicity, hallucination.
• Experiments: Compare different prompts, models, or configurations side-by-side with statistical analysis.
• Embeddings Visualization: See how your RAG embeddings cluster. Identify topics, drift, and data quality issues.

Langfuse vs Phoenix:

Complete Observability Stack:

[Production AI Agent]
        |
        | (OpenTelemetry / SDK instrumentation)
        v
[Observability Platform (Langfuse / Phoenix)]
  |
  +-- Traces Dashboard
  |     - All agent executions with full detail
  |     - Filter by user, status, model, time
  |
  +-- Cost Dashboard
  |     - Daily/weekly/monthly token usage
  |     - Cost per user, per feature
  |     - Budget alerts when thresholds exceeded
  |
  +-- Quality Dashboard
  |     - Automated eval scores over time
  |     - Hallucination rate trending
  |     - User feedback scores
  |
  +-- Alert System
        - Error rate spike -> PagerDuty/Slack
        - Cost anomaly -> Slack
        - Quality drop -> Email to team

Key Metrics to Monitor:

• Error Rate: % of agent runs that fail or produce errors
• Latency P50/P95/P99: How long does a typical/slow agent run take?
• Token Usage: Average tokens per run, daily total, cost trends in INR
• Tool Call Success Rate: Which tools fail most often? API timeouts?
• Hallucination Rate: % of responses flagged by automated eval as hallucinated
• User Satisfaction: Thumbs up/down ratios, NPS from in-app feedback

Debugging Workflow:

1. 1. Alert fires: "Error rate spiked to 15% in last hour"
2. 2. Filter traces: Show only failed traces from the last hour
3. 3. Inspect trace: Click on a failed trace, examine each span
4. 4. Find root cause: "Tool X returned timeout error at step 3, agent could not recover"
5. 5. Fix: Add retry logic to Tool X, update agent prompt to handle tool failures
6. 6. Verify: Run eval suite to confirm fix, monitor error rate

Mistake 1: Logging Everything

Do not log full prompt contents and responses for every request in production. This is expensive (storage), slow (I/O), and a privacy risk (PII in prompts). Sample traces (e.g., log 10% of requests in detail, metadata for 100%).

Mistake 2: No Cost Alerts

Without cost monitoring, a prompt change or a bug that triggers infinite loops can burn through your API budget overnight. Always set daily and weekly cost limits with alerts.

Mistake 3: Ignoring PII

User prompts often contain personal information - names, addresses, phone numbers. Your observability platform stores these traces. Ensure compliance with data privacy regulations. Consider PII masking before logging.

Best Practices:

• Sample traces in production (not 100%) to manage costs
• Set up cost alerts from day one
• Mask PII before sending to observability platform
• Create dashboards for the metrics that matter most to your team
• Review traces weekly to spot patterns and improvement opportunities

Q: Why is observability critical for AI agents in production?

Q: What is a trace in AI agent observability?

Q: Compare Langfuse and Arize Phoenix. When would you use each?