LangSmith (Tracing, Debugging & Evaluation)

Why LangSmith Exists:

Building LLM applications is hard to debug. When your chatbot gives a wrong answer, how do you figure out why? Was it bad retrieval? Wrong prompt? LLM hallucination? Token limit exceeded? Without proper observability, you are debugging blind.

LangSmith captures every step of your LLM application execution - every LLM call with its prompt and response, every retrieval with its results, every tool call with its inputs and outputs. It is like Chrome DevTools but for AI applications.

Real-World Analogy - CCTV in a Restaurant:

Running an LLM app without LangSmith is like running a restaurant without security cameras:

• Without LangSmith: Customer complains about wrong order. You ask the kitchen, waiter, and manager - everyone says everything was fine. No evidence.
• With LangSmith: Check the recording. See that the waiter heard "paneer" but typed "palak". The kitchen made exactly what was typed. Bug found in 2 minutes.

LangSmith records every "conversation" between your components, so when something goes wrong, you can replay exactly what happened.

LangSmith Core Capabilities:

• Tracing: Full execution trace of every run - see every step, input, output, latency, token usage
• Debugging: Click into any failing run, see exactly where it went wrong
• Evaluation: Run test datasets against your pipeline, measure quality with metrics
• Monitoring: Track production metrics - latency, cost, error rate, quality over time
• Datasets: Curate test datasets from production traffic for regression testing
• Annotation: Human annotators label outputs as good/bad for feedback loops

Important Note:

LangSmith is built by the LangChain team but works with ANY LLM application - not just LangChain. You can use it with raw OpenAI SDK, LlamaIndex, Haystack, or custom code. It is a SaaS platform with a free tier.

What is a Trace?

A trace is a complete record of one execution of your LLM application - from the user input to the final output, including every intermediate step. Each trace contains a tree of runs (spans), where each run represents one operation (LLM call, retrieval, tool use, etc.).

What Gets Captured in a Trace:

• LLM Calls: Full prompt (system + user messages), model response, token count, latency, model name, temperature
• Retrieval: Query used, documents retrieved, similarity scores, time taken
• Tool Calls: Tool name, input arguments, output, execution time, errors
• Chain Steps: Each step in a chain with its input/output transformation
• Metadata: User ID, session ID, custom tags, feedback scores

Trace Hierarchy Example:

User asks: "What is the GST rate for electronics?"

• Root Run: RAG Chain (total: 3.2s)
•   Child Run 1: Query Embedding (0.1s)
•   Child Run 2: Vector Retrieval (0.3s) - 5 docs found
•   Child Run 3: Re-ranking (0.5s) - top 3 selected
•   Child Run 4: Prompt Building (0.01s)
•   Child Run 5: LLM Call - GPT-4 (2.1s) - 450 tokens in, 120 out
•   Child Run 6: Output Parsing (0.01s)

Integration Methods:

• Auto-tracing: Set LANGCHAIN_TRACING_V2=true environment variable. All LangChain operations auto-traced.
• @traceable decorator: Add tracing to any Python function. Works outside LangChain.
• RunTree API: Manual trace creation for maximum control. Create parent/child runs explicitly.
• OpenAI Wrapper: Wrap the OpenAI client to automatically trace all API calls.

The Evaluation Problem:

Most teams evaluate LLM apps by running a few test questions and eyeballing the results. This does not scale. LangSmith provides structured evaluation: run a dataset through your pipeline, score outputs with evaluators, track quality over time.

LangSmith Evaluation Workflow:

1. Create Dataset: A set of input-output pairs (questions + expected answers). Can be manually created or curated from production traces.
2. Define Evaluators: Functions that score outputs. Can be LLM-based (judge quality with GPT-4), heuristic (exact match, contains keyword), or custom (domain logic).
3. Run Evaluation: Execute your pipeline on the entire dataset, apply evaluators to each result.
4. Analyze Results: View scores, compare runs, identify regressions, track improvements over time.

Built-in Evaluators:

• Correctness: LLM judges if the output matches the expected answer
• Helpfulness: LLM rates how helpful the response is
• Relevance: Is the output relevant to the input question?
• Hallucination: Does the output contain claims not supported by the context?
• Custom: Write your own scoring function for domain-specific criteria

A/B Testing with Experiments:

LangSmith lets you run the same dataset against different configurations (different models, prompts, chunk sizes) and compare results side by side. This makes data-driven decisions possible:

• GPT-4 vs Claude 3.5 for your use case
• Chunk size 256 vs 512 tokens
• With re-ranking vs without re-ranking
• System prompt v1 vs v2

Why Production Monitoring?

LLM applications degrade silently. Your knowledge base gets stale, user queries drift to topics you did not anticipate, model updates change behavior. Without monitoring, you only find out when users complain. LangSmith provides continuous production observability.

Key Monitoring Metrics:

• Latency: P50, P95, P99 response times. Catch slowdowns early.
• Token Usage: Track input/output tokens per request. Spot cost anomalies.
• Error Rate: LLM failures, timeout errors, parsing failures.
• User Feedback: Thumbs up/down on responses. Direct quality signal.
• Quality Scores: Run online evaluators on production traces (sample-based).

Feedback and Annotation:

LangSmith supports attaching feedback to any trace:

• User Feedback: Thumbs up/down buttons in your UI that send feedback to LangSmith via API
• Automated Feedback: Run evaluators on production traces to auto-score quality
• Human Annotation: Internal team reviews and labels production traces for quality assurance
• Feedback -> Dataset: Turn annotated traces into evaluation datasets for regression testing

The Flywheel:

1. Deploy app with tracing enabled
2. Collect user feedback on responses
3. Identify bad responses from traces
4. Add them to evaluation dataset
5. Fix the pipeline (better prompt, chunking, etc.)
6. Run evaluation to verify improvement
7. Deploy updated version
8. Repeat - continuous improvement loop

LangSmith Pricing (2025):

• Free Tier: 5,000 traces/month. Good for development and small apps.
• Plus: Starts around 39 USD/month for more traces and features.
• Enterprise: Custom pricing with SSO, data retention, etc.

For startups and small projects, the free tier is generous. For high-volume production apps, costs can add up quickly.

Alternatives to LangSmith:

Considerations:

• Data Privacy: LangSmith is cloud-hosted. If you send sensitive data (PII, medical records), consider self-hosted alternatives like LangFuse.
• Vendor Lock-in: Deep LangSmith integration makes switching harder. Use the @traceable decorator pattern for portability.
• Cost at Scale: High-volume apps (100K+ traces/month) should budget carefully. Consider sampling strategies.

Q: What is LangSmith and why is observability important for LLM applications?

Q: How would you set up a continuous evaluation pipeline using LangSmith?

Q: What are the alternatives to LangSmith and when would you choose them?