Site Logo

Visualizing Evaluations in LangChain for Insightful AI Performance Analysis

Introduction

Evaluating AI-driven applications is only half the battle; understanding and acting on evaluation results requires clear, actionable insights. LangChain, a powerful framework for building applications powered by large language models (LLMs), integrates with LangSmith to provide robust tools for visualizing evaluation results, enabling developers to analyze performance metrics effectively. Accessible under the /langchain/evaluation/visualizing-evaluations path, visualizing evaluations involves generating charts, dashboards, and reports to track metrics like accuracy, relevance, latency, and custom scores for components such as chains, agents, and retrievers. This comprehensive guide explores how to visualize evaluations in LangChain, covering setup, core techniques, best practices, practical applications, and advanced configurations, empowering developers to gain deep insights and optimize their AI systems.

To understand LangChain’s broader evaluation ecosystem, start with LangChain Evaluation Introduction.

What is Visualizing Evaluations in LangChain?

Visualizing evaluations in LangChain involves creating graphical representations of evaluation results to analyze the performance of LangChain components, such as chains, agents, retrievers, or end-to-end pipelines. Leveraging LangSmith, developers can generate dashboards, charts, and tables to display metrics (e.g., correctness, coherence, latency), compare experiments, and identify trends or anomalies. Visualizations are typically based on logged evaluation results from the langchain.evaluation module and can include automated metrics, custom scores, and human-in-the-loop feedback. This process is critical for understanding system performance, debugging issues, and communicating results to stakeholders.

For related concepts, see LangChain Metrics Overview and Logging Results.

Why Visualize Evaluations?

Visualizing evaluations is essential for:

  • Insightful Analysis: Identify performance trends, bottlenecks, or outliers at a glance.
  • Decision-Making: Inform optimization strategies based on clear metric comparisons.
  • Collaboration: Share intuitive visualizations with teams or stakeholders.
  • Monitoring: Track system performance over time to ensure reliability.

Explore LangSmith’s visualization capabilities at the LangSmith Documentation.

Setting Up Visualization of Evaluations

To visualize evaluations in LangChain, you need to install the required packages, configure LangSmith, set up a pipeline, define evaluators, log results, and use LangSmith’s dashboard for visualization. Below is a setup for evaluating a RetrievalQA pipeline and visualizing results using LangSmith:

from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_core.documents import Document
from langchain.evaluation import load_evaluator, EvaluatorType
from langchain.chains import RetrievalQA
from langchain_core.prompts import PromptTemplate
from langsmith import Client
from langsmith.evaluation import evaluate
import os
import logging
import time
from typing import Dict, Any

# Set up logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# Set LangSmith environment variables
os.environ["LANGSMITH_API_KEY"] = ""
os.environ["LANGSMITH_PROJECT"] = "evaluation-visualization"

# Initialize embeddings and language model
embedding_function = OpenAIEmbeddings(model="text-embedding-3-small")
llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)

# Create sample documents
documents = [
    Document(page_content="The capital of France is Paris.", metadata={"source": "geo"}),
    Document(page_content="The Eiffel Tower is in Paris.", metadata={"source": "landmark"}),
    Document(page_content="Florida is a state in the USA.", metadata={"source": "geo"})
]

# Initialize Chroma vector store
vector_store = Chroma.from_documents(
    documents,
    embedding_function,
    collection_name="langchain_example",
    persist_directory="./chroma_db"
)

# Set up RetrievalQA pipeline
prompt = PromptTemplate.from_template("Answer: {question}")
qa_pipeline = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vector_store.as_retriever(search_kwargs={"k": 2}),
    return_source_documents=True
)

# Initialize LangSmith client
client = Client()

# Create or load a dataset
dataset_name = "qa_visualization_dataset"
try:
    dataset = client.create_dataset(dataset_name=dataset_name)
except:
    dataset = client.get_dataset(dataset_name=dataset_name)

# Add examples to dataset
examples = [
    {
        "input": "What is the capital of France?",
        "output": "Paris",
        "expected_docs": ["The capital of France is Paris."]
    },
    {
        "input": "Where is the Eiffel Tower?",
        "output": "Paris",
        "expected_docs": ["The Eiffel Tower is in Paris."]
    },
    {
        "input": "What is the capital of Florida?",
        "output": "",
        "expected_docs": ["Florida is a state in the USA."]
    }
]
for example in examples:
    client.create_example(
        inputs={"question": example["input"]},
        outputs={"answer": example["output"], "expected_docs": example["expected_docs"]},
        dataset_id=dataset.id
    )

# Define evaluators
def evaluate_correctness(run, example) -> Dict[str, Any]:
    prediction = run.outputs.get("result", "")
    reference = example.outputs.get("answer", "")
    question = example.inputs.get("question", "")
    if not reference:
        return {"key": "correctness", "score": None, "comment": "No reference provided."}
    evaluator = load_evaluator(EvaluatorType.QA, llm=llm)
    result = evaluator.evaluate_strings(
        prediction=prediction,
        reference=reference,
        input=question
    )
    return {"key": "correctness", "score": result["score"], "comment": result.get("reasoning", "")}

def evaluate_relevance(run, example) -> Dict[str, Any]:
    prediction = run.outputs.get("result", "")
    question = example.inputs.get("question", "")
    evaluator = load_evaluator(EvaluatorType.CRITERIA, criteria="relevance", llm=llm)
    result = evaluator.evaluate_strings(
        prediction=prediction,
        input=question
    )
    return {"key": "relevance", "score": result["score"], "comment": result.get("reasoning", "")}

def evaluate_latency(run, example) -> Dict[str, Any]:
    latency = run.outputs.get("latency", 0.0)
    score = 1.0 if latency < 1.0 else max(0.0, 1.0 - (latency - 1.0) / 5.0)
    return {
        "key": "latency",
        "score": score,
        "comment": f"Response latency: {latency:.2f} seconds."
    }

# Run evaluation and log results
start_time = time.time()
results = evaluate(
    lambda inputs: {
        "result": qa_pipeline.invoke({"query": inputs["question"]})["result"],
        "latency": time.time() - start_time,
        "source_documents": qa_pipeline.invoke({"query": inputs["question"]})["source_documents"]
    },
    data=dataset_name,
    evaluators=[evaluate_correctness, evaluate_relevance, evaluate_latency],
    experiment_prefix="qa_visualization_test",
    metadata={"version": "1.0", "evaluated_at": "2025-05-15T15:38:00Z"}
)

# Log results
logger.info(f"Evaluation completed in {time.time() - start_time:.2f} seconds")
print(f"Test Results: {results}")
print("Visualize results in LangSmith dashboard under 'qa_visualization_test' experiment.")
print("Instructions: Navigate to LangSmith, select the 'evaluation-visualization' project, and view charts for correctness, relevance, and latency.")

This setup creates a RetrievalQA pipeline, uploads a test dataset to LangSmith, evaluates outputs for correctness, relevance, and latency, and logs results for visualization in the LangSmith dashboard.

Installation

Install the core packages for LangChain, LangSmith, and evaluation:

pip install langchain langchain-chroma langchain-openai chromadb langsmith

For specific metrics, install additional dependencies:

  • NLP Metrics: pip install nltk rouge-score for BLEU/ROUGE scores.
  • Embedding Metrics: Included with langchain-openai.

Example:

pip install nltk rouge-score

For detailed installation guidance, see LangSmith Documentation.

Visualization Workflow (LangSmith UI)

  1. Access LangSmith Dashboard:
    • Log in to LangSmith with your API key.
    • Navigate to the project (evaluation-visualization) and experiment (qa_visualization_test).

2. View Visualizations:


  • Summary Charts: Display average scores for correctness, relevance, and latency across examples.
  • Per-Example Tables: Show detailed scores, comments, and inputs for each example.
  • Trend Graphs: Plot metric trends over time or across experiments.
  • Comparison Views: Compare multiple experiments (e.g., different retriever settings).

3. Customize Visualizations:


  • Filter by metric (e.g., correctness), example, or metadata (e.g., version).
  • Export charts or data as CSV for external reporting.

4. Share Insights:


  • Share dashboard links with team members or stakeholders for collaborative analysis.

Core Techniques for Visualizing Evaluations

1. Summary Dashboards

Create high-level dashboards to summarize key metrics across an experiment.

  • Metric Averages:
    • Display mean scores for correctness, relevance, and latency.
    • Example: In LangSmith, view a bar chart showing average correctness (e.g., 0.95) and latency score (e.g., 0.90).
  • Distribution Plots:
    • Show the distribution of scores (e.g., histogram of relevance scores).
    • Example: Identify if most examples score high (0.8–1.0) or if outliers exist.

2. Per-Example Analysis

Visualize detailed results for individual examples to diagnose issues.

  • Table Views:
    • Display input, output, scores, and comments for each example.
    • Example: In LangSmith, see a table with columns for question, response, correctness score, and reasoning (e.g., “Prediction matches reference.”).
  • Highlight Outliers:
    • Flag examples with low scores for review.
    • Example: Highlight examples where correctness < 0.5 for debugging.

3. Trend Analysis

Track metric changes over time or across experiments.

  • Time-Series Plots:
    • Plot metrics for multiple evaluation runs to detect trends.
    • Example: In LangSmith, view a line graph showing correctness improving from 0.85 to 0.95 over three experiments.
  • Experiment Comparison:
    • Compare metrics across different configurations (e.g., retriever k=2 vs. k=3).
    • Example: Use a bar chart to compare average latency across experiments.

4. Custom Metric Visualizations

Visualize custom metrics tailored to specific use cases.

  • Custom Evaluator:
    • Log and visualize scores for custom metrics like efficiency or completeness.
    • Example:
    • class EfficiencyEvaluator(StringEvaluator):
        def _evaluate_strings(self, prediction: str, **kwargs) -> Dict[str, Any]:
            word_count = len(prediction.split())
            score = 1.0 if word_count < 50 else 0.8
            return {
                "key": "efficiency",
                "score": score,
                "reasoning": f"Response brevity: {word_count} words."
            }

    def evaluate_efficiency(run, example):
        prediction = run.outputs.get("result", "")
        evaluator = EfficiencyEvaluator()
        return evaluator.evaluate_strings(prediction=prediction)
  • Visualization:
    • In LangSmith, view a histogram of efficiency scores to assess response brevity distribution.

5. Human-in-the-Loop Visualizations

Incorporate human feedback into visualizations for subjective metrics.

  • Human Feedback Logging:
    • Log human scores (e.g., coherence) alongside automated metrics.
    • Example: In LangSmith, reviewers add scores (0-1) for “coherence” with comments, visualized in a table or chart.
  • Comparison Charts:
    • Compare human and automated scores (e.g., coherence vs. relevance).
    • Example: Use a scatter plot to show correlation between human and LLM-based coherence scores.

Comprehensive Example

Here’s a complete system evaluating a LangChain agent pipeline, logging results, and visualizing them in LangSmith, integrated with Chroma and MongoDB Atlas:

from langchain_chroma import Chroma
from langchain_mongodb import MongoDBAtlasVectorSearch
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_core.documents import Document
from langchain.evaluation import load_evaluator, EvaluatorType, StringEvaluator
from langchain.agents import initialize_agent, AgentType
from langchain.tools import Tool
from langchain_community.tools import DuckDuckGoSearchRun
from langsmith import Client
from langsmith.evaluation import evaluate
from pymongo import MongoClient
import os
import logging
import time
from typing import Dict, Any

# Set up logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# Set LangSmith environment variables
os.environ["LANGSMITH_API_KEY"] = ""
os.environ["LANGSMITH_PROJECT"] = "agent-visualization"

# Initialize embeddings and language model
embedding_function = OpenAIEmbeddings(model="text-embedding-3-small")
llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)

# Create sample documents
documents = [
    Document(page_content="The capital of France is Paris.", metadata={"source": "geo"}),
    Document(page_content="The Eiffel Tower is in Paris.", metadata={"source": "landmark"}),
    Document(page_content="Florida is a state in the USA.", metadata={"source": "geo"})
]

# Initialize Chroma and MongoDB Atlas vector stores
chroma_store = Chroma.from_documents(
    documents,
    embedding_function,
    collection_name="langchain_example",
    persist_directory="./chroma_db"
)
client = MongoClient("mongodb+srv://:@.mongodb.net/")
collection = client["langchain_db"]["example_collection"]
mongo_store = MongoDBAtlasVectorSearch.from_documents(
    documents,
    embedding_function,
    collection=collection,
    index_name="vector_index"
)

# Set up search tool and agent pipeline
search = DuckDuckGoSearchRun()
tools = [
    Tool(
        name="Search",
        func=search.run,
        description="Useful for answering questions about recent events or general knowledge."
    )
]
agent = initialize_agent(
    tools=tools,
    llm=llm,
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    verbose=True
)

# Initialize LangSmith client
ls_client = Client()

# Create or load dataset
dataset_name = "agent_visualization_dataset"
try:
    dataset = ls_client.create_dataset(dataset_name=dataset_name)
except:
    dataset = ls_client.get_dataset(dataset_name=dataset_name)

# Add examples to dataset
examples = [
    {
        "input": "What is the capital of France?",
        "output": "Paris",
        "expected_docs": ["The capital of France is Paris."]
    },
    {
        "input": "Where is the Eiffel Tower?",
        "output": "Paris",
        "expected_docs": ["The Eiffel Tower is in Paris."]
    },
    {
        "input": "Describe Paris landmarks.",
        "output": ""
    }
]
for example in examples:
    ls_client.create_example(
        inputs={"question": example["input"]},
        outputs={"answer": example["output"], "expected_docs": example.get("expected_docs", [])},
        dataset_id=dataset.id
    )

# Define custom evaluator
class EfficiencyEvaluator(StringEvaluator):
    def _evaluate_strings(self, prediction: str, **kwargs) -> Dict[str, Any]:
        word_count = len(prediction.split())
        score = 1.0 if word_count < 50 else 0.8
        return {
            "key": "efficiency",
            "score": score,
            "reasoning": f"Response brevity: {word_count} words."
        }

# Define evaluation functions
def evaluate_correctness(run, example) -> Dict[str, Any]:
    prediction = run.outputs.get("result", "")
    reference = example.outputs.get("answer", "")
    question = example.inputs.get("question", "")
    if not reference:
        return {"key": "correctness", "score": None, "comment": "No reference provided."}
    evaluator = load_evaluator(EvaluatorType.QA, llm=llm)
    result = evaluator.evaluate_strings(
        prediction=prediction,
        reference=reference,
        input=question
    )
    return {"key": "correctness", "score": result["score"], "comment": result.get("reasoning", "")}

def evaluate_relevance(run, example) -> Dict[str, Any]:
    prediction = run.outputs.get("result", "")
    question = example.inputs.get("question", "")
    evaluator = load_evaluator(EvaluatorType.CRITERIA, criteria="relevance", llm=llm)
    result = evaluator.evaluate_strings(
        prediction=prediction,
        input=question
    )
    return {"key": "relevance", "score": result["score"], "comment": result.get("reasoning", "")}

def evaluate_latency(run, example) -> Dict[str, Any]:
    latency = run.outputs.get("latency", 0.0)
    score = 1.0 if latency < 1.0 else max(0.0, 1.0 - (latency - 1.0) / 5.0)
    return {
        "key": "latency",
        "score": score,
        "comment": f"Response latency: {latency:.2f} seconds."
    }

def evaluate_efficiency(run, example) -> Dict[str, Any]:
    prediction = run.outputs.get("result", "")
    evaluator = EfficiencyEvaluator()
    return evaluator.evaluate_strings(prediction=prediction)

# Run evaluation and log results
start_time = time.time()
results = evaluate(
    lambda inputs: {
        "result": agent.run(inputs["question"]),
        "latency": time.time() - start_time,
        "source_documents": qa_pipeline.invoke({"query": inputs["question"]})["source_documents"]
    },
    data=dataset_name,
    evaluators=[evaluate_correctness, evaluate_relevance, evaluate_latency, evaluate_efficiency],
    experiment_prefix="agent_visualization_test",
    metadata={"version": "1.0", "evaluated_at": "2025-05-15T15:38:00Z"}
)

# Log results
logger.info(f"Evaluation completed in {time.time() - start_time:.2f} seconds")
print(f"Test Results: {results}")
print("Visualize results in LangSmith dashboard under 'agent_visualization_test' experiment.")
print("Instructions: In LangSmith, navigate to the 'agent-visualization' project, select 'agent_visualization_test', and explore summary charts, per-example tables, and trend graphs for correctness, relevance, latency, and efficiency.")

Output:

Evaluation completed in 10.23 seconds
Test Results: 
Visualize results in LangSmith dashboard under 'agent_visualization_test' experiment.
Instructions: In LangSmith, navigate to the 'agent-visualization' project, select 'agent_visualization_test', and explore summary charts, per-example tables, and trend graphs for correctness, relevance, latency, and efficiency.

The evaluation logs results for correctness, relevance, latency, and efficiency, which can be visualized in the LangSmith dashboard as charts, tables, and trend graphs.

Best Practices

  1. Log Comprehensive Metrics: Include accuracy, latency, and custom metrics to enable rich visualizations.
  2. Use Descriptive Metadata: Add version, timestamps, and pipeline details to contextualize visualizations.
  3. Leverage LangSmith UI: Use dashboards for interactive analysis and sharing with stakeholders.
  4. Visualize Trends: Compare experiments to track performance improvements or regressions.
  5. Combine with HITL: Incorporate human feedback visualizations for subjective metrics (Human-in-the-Loop Evaluation).
  6. Export Visualizations: Download charts or data for reports or presentations.

Error Handling

  • API Errors: Handle LangSmith API failures with retries or fallback logging.
  • Dataset Issues: Validate dataset format to avoid parsing errors.
  • Visualization Issues: Ensure metrics are logged correctly to prevent empty charts.
  • Resource Limits: Batch evaluations to manage API costs and dashboard load times.

See Troubleshooting.

Limitations

  • Cost: Visualizing large datasets with LLM-based metrics can be expensive.
  • UI Complexity: Managing multiple experiments and visualizations may require familiarity with LangSmith.
  • Metric Variability: LLM-based metrics may introduce variability in visualizations.
  • Storage: Extensive logging for visualizations requires sufficient storage in LangSmith.

Recent Developments

  • 2025 Updates: LangSmith introduced enhanced dashboard features, including customizable charts and real-time visualization.
  • Community Feedback: X posts highlight visualization workflows for tracking RAG system performance in enterprise settings.
  • LangSmith UI: Improved support for experiment comparison and interactive metric filtering.

Conclusion

Visualizing evaluations in LangChain with LangSmith provides powerful tools for analyzing AI system performance, enabling developers to gain actionable insights through charts, dashboards, and trend graphs. By logging and visualizing metrics like correctness, relevance, and latency, developers can optimize pipelines, debug issues, and communicate results effectively. Start visualizing your LangChain evaluations to enhance your projects, leveraging LangSmith for data-driven decision-making and performance tracking.

For official documentation, visit LangSmith Documentation.