Skip to main content

Querying Saved Runs

Running local analysis of your run data can be incredibly useful at each stage of deployment. This guide reviews some ways you can use the client to fetch runs from the server.

Using the list_runs method, you can filter runs to analyze and export. Most simple requests can be satisfied using simple top level arguments:

  • project_id / project_name: The project to fetch runs from.
  • run_type: The type of run to get, such as llm, chain, tool, retriever, etc.
  • dataset_name / dataset_id: Fetch runs that are associated with an example row in the specified dataset. This is useful for comparing prompts or models over a given dataset.
  • reference_example_id: Fetch runs that are associated with a specific example row. This is useful for comparing prompts or models on a given input.
  • parent_run_id - Fetch runs that are children of a given run. This is useful for fetching runs grouped together using the context manager or for fetching an agent trajectory.
  • error: Fetch runs that errored or did not error.
  • run_ids: Fetch runs with a given list of run ids. Note: This will ignore all other filtering arguments.
  • filter: Fetch runs that match a given structured filter statement. See the run filtering guide below for more information.
  • query (experimental): Query the experimental natural language API, which translates your query into a filter statement.
Required arguments

LangSmith expects at least one of the following to be provided: 'project_id'/'project_name', 'run_ids', 'parent_run_id', or 'reference_example_id'. If none of these are provided, the server will raise an error.

Using keyword arguments

For simple queries, such as filtering by project, run time, name, or run ID's, you can directly use keyword arguments in the list_runs method. These correspond dirctly to query params in the REST API. All the examples below assume you have created a LangSmith client and configured it with your API key to connect to the LangSmith server.

from langsmith import Client

client = Client()

Below are some examples of ways to list runs using keyword arguments:

List all runs in a project

project_runs = client.list_runs(project_name="<your_project>")

List LLM and Chat runs in the last 24 hours

todays_llm_runs = client.list_runs(
project_name="<your_project>",
start_time=datetime.now() - timedelta(days=1),
run_type="llm",
)

List traces in a project

Root runs (or run traces), are runs that have no parents. These are assigned an 'execution_order' of 1. You can use this to filter for root runs.

root_runs = client.list_runs(
project_name="<your_project>",
execution_order=1
)

List runs without errors

correct_runs = client.list_runs(project_name="<your_project>", error=False)

List runs by run ID

If you have a list of run IDs, you can list them directly:

run_ids = ['a36092d2-4ad5-4fb4-9c0d-0dba9a2ed836','9398e6be-964f-4aa4-8ae9-ad78cd4b7074']
selected_runs = client.list_runs(id=run_ids)
Ignores Other Arguments

If you provide a list of run IDs in the way described above, it will ignore all other filtering arguments like project_name, run_type, etc. and directly return the runs matching the given IDs.

Run Filtering

Listing runs with query params is useful for simple queries, but doesn't support many common needs, such as filtering by metadata, tags, or other fields.

LangSmith supports a filter query language to permit more complex filtering operations when fetching runs. This guide will provide a high level overview of the grammar as well as a few examples of when it can be useful.

If you'd prefer a more visual guide, you can get a taste of the language by viewing the table of runs on any of your projects' pages. We provide some recommended filters to get you started that you can copy and use the SDK.

Filters Sidebar

Grammar

The filtering grammar is based on common comparators on fields in the run object. Supported comparators include:

  • gte (greater than or equal to)
  • gt (greater than)
  • lte (less than or equal to)
  • lt (less than)
  • eq (equal to)
  • neq (not equal to)
  • has (check if run contains a tag or metadata json blob)
  • search (search for a substring in a string field)

Additionally, you can combine multiple comparisons through and and or operators.

These can be applied on fields of the run object, such as its id, name, run_type, start_time / end_time, latency, total_tokens, error, execution_order, tags, and any associated feedback through feedback_key and feedback_score.

Examples

The following examples assume you have configured your environment appropriately and have runs stored in LangSmith.

List runs with "star_rating" key whose score is greater than 4

client.list_runs(project_name="<your_project>", filter='and(eq(feedback_key, "star_rating"), gt(feedback_score, 4))')

List runs that took longer than 5 seconds to complete

client.list_runs(project_name="<your_project>", filter='gt(latency, "5s")')

List all runs where total_tokens is greater than 5000

client.list_runs(project_name="<your_project>", filter='gt(total_tokens, 5000)')

List all runs that have "error" not equal to null

client.list_runs(project_name="<your_project>", filter='neq(error, null)')

List all runs with execution_order less than 10

client.list_runs(project_name="<your_project>", filter='lt(execution_order, 10)')

List all runs where start_time is greater than a specific timestamp

client.list_runs(project_name="<your_project>", filter='gt(start_time, "2023-07-15T12:34:56Z")')

List all runs that contain the string "substring"

client.list_runs(project_name="<your_project>", filter='search("substring")')

List all runs that are tagged with the git hash "2aa1cf4"

client.list_runs(project_name="<your_project>", filter='has(tags, "2aa1cf4")')

Complex query: List all "chain" type runs that took more than 10 seconds and had total_tokens greater than 5000

client.list_runs(
project_name="<your_project>",
filter='and(eq(run_type, "chain"), gt(latency, 10), gt(total_tokens, 5000))'
)

Complex query: List all runs that started after a specific timestamp and either have "error" not equal to null or a "Correctness" feedback score equal to 0

client.list_runs(
project_name="<your_project>",
filter='and(gt(start_time, "2023-07-15T12:34:56Z"), or(neq(error, null), and(eq(feedback_key, "Correctness"), eq(feedback_score, 0.0))))'
)

Complex query: List all runs where tags include "experimental" or "beta" and latency is greater than 2 seconds

client.list_runs(
project_name="<your_project>",
filter='and(or(has(tags, "experimental"), has(tags, "beta")), gt(latency, 2))'
)

Check for presence of metadata

If you want to check for the presence of metadata, you can use the has operator. This is useful if you want to log more structured information about your runs.

import json

to_search = {
"user_id": "4070f233-f61e-44eb-bff1-da3c163895a3"
}

client.list_runs(
project_name="default",
filter=f"has(metadata, '{json.dumps(to_search)}')"
)

Check for environment details in metadata

A common pattern is to add environment information to your traces via metadata. If you want to filter for runs containing environment metadata, you can use the same pattern as above:

import json

to_search = {
"environment": "production"
}

client.list_runs(
project_name="default",
filter=f"has(metadata, '{json.dumps(to_search)}')"
)

Check for conversation ID in metadata

Another common way to associate traces in the same conversation is by using a shared conversation ID. If you want to filter runs based on a conversation ID in this way, you can search for that ID in the metadata.

import json

to_search = {
"conversation_id": "a1b2c3d4-e5f6-7890"
}

client.list_runs(
project_name="default",
filter=f"has(metadata, '{json.dumps(to_search)}')"
)

If you want to combine multiple conditions to refine your search, you can use the and operator along with other filtering functions. Here's how you can search for runs named "ChatOpenAI" that also have a specific conversation_id in their metadata:

import json

to_search = {
"conversation_id": "69b12c91-b1e2-46ce-91de-794c077e8151"
}

client.list_runs(
project_name="default",
filter=f"and(has(metadata, '{json.dumps(to_search)}'), eq(name, 'ChatOpenAI'))"
)