Evaluation how-to guides

These guides answer “How do I…?” format questions. They are goal-oriented and concrete, and are meant to help you complete a specific task. For conceptual explanations see the Conceptual guide. For end-to-end walkthroughs see Tutorials. For comprehensive descriptions of every class and function see the API reference.

Key features

• Create a dataset with the SDK or from the UI
• Run offline evaluations with the SDK or from the UI
• Run online evaluations with LLM-as-judge and custom code evaluators
• Analyze evaluation results in the UI
• Log user feedback from your app
• Log expert feedback with annotation queues

Offline evaluation

Evaluate and improve your application before deploying it.

Run an evaluation

• Define a target function to evaluate
• Run an evaluation with the SDK
• Run an evaluation asynchronously
• Run an evaluation comparing two experiments
• Evaluate a langchain runnable
• Evaluate a langgraph graph
• Evaluate an existing experiment (Python only)
• Run an evaluation from the UI
• Run an evaluation via the REST API
• Run an evaluation with large file inputs
• Set up a multi-turn evaluation

Define an evaluator

• Define a custom evaluator
• Define an LLM-as-a-judge evaluator
• Define a pairwise evaluator
• Define a summary evaluator
• Use prebuilt evaluators
• Evaluate an application's intermediate steps
• Return multiple metrics in one evaluator
• Return categorical vs numerical metrics

Configure the evaluation data

• Evaluate on a split / filtered view of a dataset
• Evaluate on a specific dataset version

Configure an evaluation job

• Evaluate with repetitions
• Handle model rate limits
• Print detailed logs (Python only)
• Run an evaluation locally (beta, Python only)

Add default evaluators to a dataset

Set up evaluators that automatically run for all experiments against a dataset.

• Set up an auto-evaluator
• Create a few-shot evaluator

Testing integrations

Run evals using your favorite testing tools.

• Run evals with pytest (beta)
• Run evals with Vitest/Jest (beta)

Online evaluation

Evaluate and monitor your system's live performance on production data.

• Set up an online evaluator
• Create a few-shot evaluator

Analyzing experiment results

Use the UI & API to understand your experiment results.

• Analyze a single experiment
• Compare experiments with the comparison view
• Filter experiments
• View pairwise experiments
• Fetch experiment results in the SDK
• Upload experiments run outside of LangSmith with the REST API
• Download experiment results as a CSV
• Audit and correct evaluator scores
• Renaming an experiment

Dataset management

Manage datasets in LangSmith used by your evaluations.

• Create a dataset from the UI
• Export a dataset from the UI
• Create a dataset split from the UI
• Filter examples from the UI
• Create a dataset with the SDK
• Fetch a dataset with the SDK
• Update a dataset with the SDK
• Version a dataset
• Dataset sharing
• Export filtered traces from an experiment to a dataset

Annotation queues and human feedback

Collect feedback from subject matter experts and users to improve your applications.

• Use annotation queues
• Log user feedback
• Set up a new feedback criteria
• Annotate traces inline in the UI
• Audit and correct evaluator scores