Skip to main content

Tracing FAQs

The following are some frequently asked questions about logging runs to LangSmith:

What is a Run?​

A 'Run' represents a single unit of work or operation within your LLM application. This could be anything from single call to an LLM or chain, to a prompt formatting call, to a runnable lambda invocation. If you are familiar with Open Telemetry, this corresponds loosely to a span.

A 'Trace' is a collection of runs organized in a tree or graph. The 'Root Run' is the top level run in a trace. It is the run that is directly invoked by the user or application. The root run is assigned an execution_order=1. The execution order of a run is the order in which it was called in the trace, if viewed as a sequence.

Each run contains a unique ID, run_type, name, run_type, inputs, start times, and optionally outputs, error information, end times, events, tags, and other metadata.

The run_type indicates the kind of operation that was performed. Some common ones include "llm", "chain", "retriever", and "tool".

How do I group traces into a project?​

All runs are logged to a project. If left unspecified, the tracer project is set to default. You can set the LANGCHAIN_PROJECT environment variable to configure a custom project name for an entire application run. This is best done before executing your program.

export LANGCHAIN_PROJECT="My Project"
from langchain.chat_models import ChatAnthropic
llm = ChatAnthropic()

llm.invoke("Hello, World!")

How do I change the tracer project?​

When global environment variables are too broad, you can also set the project name for a specific tracer instance:

from langchain.callbacks.tracers import LangChainTracer

tracer = LangChainTracer(project_name="My Project")
chain.invoke({"query": "How many people live in canada as of 2023?"}, config={"callbacks": [tracer]})

The Python tracer also has a context manager you can use to set the project for anything within the context:

from langchain.callbacks import tracing_v2_enabled

with tracing_v2_enabled(project_name="My Project"):
    chain.invoke({"query": "How many people live in canada as of 2023?"})

How do I add tags to runs?​

The LangChainTracer supports adding tags to any run trace. These tags are visible in the UI and through the API and can be used to filter and analyze runs. You can tag a single object's runs by adding them when the object is initialized, or you can tag all nested runs by passing them to the object's function call. The following code snippet shows how to do both, assuming you've set the appropriate environment variables to log traces.

from langchain.chat_models import ChatOpenAI
from langchain.prompts import PromptTemplate

llm = ChatOpenAI(temperature=0, tags=["my-llm-tag"])
prompt = PromptTemplate.from_template("Say {input}")
chain = (prompt | llm).with_config({"tags": ["my-bash-tag", "another-tag"]})

chain.invoke({"input": "Hello, World!"}, {"tags": ["shared-tags"]})

How do I add metadata to runs?​

Similar to tags, LangSmith permits associating arbitrary key-value pairs as metadata within runs. Below is an example:

from langchain.chat_models import ChatOpenAI
from langchain.prompts import ChatPromptTemplate

chat_model = ChatOpenAI()
prompt = ChatPromptTemplate.from_messages([
  ("system", "You are a helpful AI."),
  ("human", "{input}")
])
chain = prompt | chat_model

chain.invoke({"input": "What is the meaning of life?"}, {"metadata": {"my_key": "My Value"}})

How do I customize the name of a run?​

When tracing within LangChain, run names default to the class name of the traced object (e.g., 'ChatOpenAI'). You can customize chain and runnable trace names by calling new_chain = chain.with_config({"run_name": "MyRunName"}). (Note: this is not currently supported directly on LLM objects.)

Example:

configured_chain = chain.with_config({"run_name": "MyCustomChain"})
configured_chain.invoke({"query": "What is the meaning of life?"})

For more examples of this, check out the recipe on customizing run names.

If you are logging using the LangSmith SDK or REST API directly, you can set the name of the run manually to be whatever you like.

How do I use LangSmith in my AB Testing framework?​

AB tests are a great way to compare the performance of different models or chains. Currently, this is best facilitated using metadata (though tags also suffice). Inject the experiment ID or testing variant(s) in as metadata values, then use the SDK or web app to compare metrics over subsets.

from langchain.chat_models import ChatOpenAI
from langchain.prompts import ChatPromptTemplate
chat_model = ChatOpenAI()
prompt = ChatPromptTemplate.from_messages([
  ("system", "You are a helpful AI."),
  ("human", "{input}")
])
chain = prompt | chat_model

chain.invoke({"input": "What is the meaning of life?"}, {"metadata": {"variant": "abc123"}})

Then in the runs page of the web app, you can filter runs using the query has(metadata, '{"variant": "abc123"}'). Similarly using the LangSmith SDK:

from langsmith import Client

client = Client()
runs = list(client.list_runs(
    project_name="<your_project>",
    filter='has(metadata, \'{"variant": "abc123"}\')',
))

How do I group runs from multi-turn interactions?​

Many application experiences involve multiple interactions outside a single call to an agent or chain. By default, these are saved as separate runs (though the memory and chat history is logged where appropriate).

To group these runs together, we offer convenience methods in both Python and JS that save all runs within a defined context beneath a virtual parent run.

New in 0.0.293

The ability to add inputs and call on_chain_end() in python is only available in langchain>=0.0.293.

from langchain.callbacks.manager import (
    trace_as_chain_group,
    atrace_as_chain_group,
)

with trace_as_chain_group("my_group_name", inputs={"input": "My Input"}) as group_manager:
    """Pass the group_manager as a callback to group all runs
    within this context"""

    # ...
    group_manager.on_chain_end({"output": "Final output"})
# Or for async code
async with atrace_as_chain_group("my_group_name", inputs={"input": "My Input"}) as async_group_manager:
    """Async applications are better suited with the async callback manager"""
    # ...
    await async_group_manager.on_chain_end({"output": "Final output"})

# Example usage:
from langchain.chat_models.anthropic import ChatAnthropic
from langchain.prompts import ChatPromptTemplate

llm = ChatAnthropic()
prompt = ChatPromptTemplate.from_messages(
  [
    ("system", "You are a helpful AI."),
    ("human", "{question}")
  ]
)
chain = prompt | llm

question = "What is your name?"
with trace_as_chain_group("my_group", inputs={"question": question}) as group_manager:
    res1 = chain.invoke({"question": question}, config={"callbacks": group_manager})
    res2 = chain.invoke({"question": "What is your quest?"}, config={"callbacks": group_manager})
    group_manager.on_chain_end({"output": res2})

How do I get the run ID from a call?​

In Typescript, the run ID is returned in the call response under the __run key. In python, we recommend using the run collector callback. Below is an example:

from langchain import chat_models, prompts, callbacks
chain = (
    prompts.ChatPromptTemplate.from_template("Say hi to {name}")
    | chat_models.ChatAnthropic()
)
with callbacks.collect_runs() as cb:
  result = chain.invoke({"name": "Clara"})
  run_id = cb.traced_runs[0].id
print(run_id)

For older versions of LangChain (<0.0.276), you can instruct the chain to return the run ID by specifying the `include_run_info=True` parameter to the call function:

from langchain.chat_models import ChatAnthropic
from langchain.chains import LLMChain

chain = LLMChain.from_string(ChatAnthropic(), "Say hi to {name}")
response = chain("Clara", include_run_info=True)
run_id = response["__run"].run_id
print(run_id)

For python LLMs/chat models, the run information is returned automatically when calling the `generate()` method. Example:

from langchain.chat_models import ChatAnthropic
from langchain.prompts import ChatPromptTemplate
 
 chat_model = ChatAnthropic()
 
 prompt = ChatPromptTemplate.from_messages(
     [
         ("system", "You are a cat"),
         ("human", "Hi"),
     ]
 )
 res = chat_model.generate(messages=[prompt.format_messages()])
 res.run[0].run_id

or for LLMs

from langchain.llms import OpenAI

openai = OpenAI()
res = openai.generate(["You are a good bot"])
print(res.run[0].run_id)

How do I get the URL of the run?​

Runs are streamed to whichever project you have configured ("default" if none is set), and you can view them by opening the project page. To programmatically access the run's URL, you can use the LangSmith client. This can be conveniently used in conjunction with the above method to get the run ID from a call. Below is an example.

Note: This requires langsmith>=0.0.11
from langsmith import Client

client = Client()
run = client.read_run("<run_id>")
print(run.url)

How do I log traces without environment variables?​

Some situations don't permit the use of environment variables or don't expose process.env. This is mostly pertinent when running LangChain apps in certain JavaScript runtime environments. To add tracing in these situations, you can manually create the LangChainTracer callback and pass it to the chain, LLM, or other LangChain component, either when initializing or in the call itself. This is the same tactic used for changing the tracer project within a program.

Example:

from langchain.callbacks import LangChainTracer
from langchain.chat_models import ChatOpenAI
from langsmith import Client

callbacks = [
  LangChainTracer(
    project_name="YOUR_PROJECT_NAME_HERE",
    client=Client(
      api_url="https://api.smith.langchain.com",
      api_key="YOUR_API_KEY_HERE"
    )
  )
]

llm = ChatOpenAI()
llm.invoke("Hello, world!", config={"callbacks": callbacks})

This tactic is also useful for when you have multiple chains running in a shared environment but want to log their run traces to different projects.

How do I turn tracing off?​

If you've decided you no longer want to trace your runs, you can remove the environment variables configured to start tracing in the first place. By unsetting the LANGCHAIN_TRACING_V2 environment variable, traces will no longer be logged to LangSmith.

unset LANGCHAIN_TRACING_V2

Unsetting an environment variable while a program is in the middle of execution is not guaranteed to terminate logging. If you want to selectively log runs or stop tracing mid-execution, you must not start tracing using the LANGCHAIN_TRACING_V2 environment variable and instead selectively log runs by passing the LangChainTracer callback to the LangChain object (or by using the context manager in python).

How do I export runs?​

We are working to add better data connectors and export options. For now, you can use the LangSmith API directly to export runs. Below are a couple examples:

from datetime import datetime, timedelta
from langsmith import Client

client = Client()
# Download all runs in a project
project_runs = client.list_runs(project_name="<your_project>")

# List only LLM runs in the last 24 hours
todays_runs = client.list_runs(
    start_time=datetime.now() - timedelta(days=1),
    run_type="llm",
)

# You can build complex filters and queries if needed.
# Filter by runs that have a feedback key of "Correctness==1.0"
# living in the specified project.
correct_runs = client.list_runs(
    project_name="<your_project>",
    # More complex filters can be specified
    filter='and(eq(feedback_key, "Correctness"), eq(feedback_score, 1.0))',
)

Check out the exporting runs directory for more examples of how to export runs, or the local run filtering documentation on how to construct more complex filters.

How do I ensure logging is completed before exiting my application?​

In LangChain.py, LangSmith's tracing is done in a background thread to avoid obstructing your production application. This means that if you're process may end before all traces are successfully posted to LangSmith. This is especially prevelant in a serverless environment, where your VM may be terminated immediately once your chain or agent completes.

In LangChain.js, the default is to block for a short period of time for the trace to finish due to the greater popularity of serverless environments. You can make callbacks asynchronous by setting the LANGCHAIN_CALLBACKS_BACKGROUND environment variable to "true".

For both languages, LangChain exposes methods to wait for traces to be submitted before exiting your application.

Below is an example:

from langchain.chat_models import ChatOpenAI
from langchain.callbacks.tracers.langchain import wait_for_all_tracers

llm = ChatOpenAI()
try:
    llm.invoke("Hello, World!")
finally:
    wait_for_all_tracers()

How do I log to LangSmith if I am not using LangChain?​

The most reliable way to log run trees to LangSmith is by using the RunTree object. Below is an example:

import datetime
from typing import Any

import openai
from openai.openai_object import OpenAIObject
from langsmith.run_helpers import traceable


@traceable(run_type="llm", name="openai.ChatCompletion.create")
def my_chat_model(*args: Any, **kwargs: Any) -> OpenAIObject:
    return openai.ChatCompletion.create(*args, **kwargs)


@traceable(run_type="tool")
def my_tool(tool_input: str) -> str:
    return tool_input.upper()


@traceable(run_type="chain")
def my_chain(prompt: str) -> str:
    messages = [
        {
            "role": "system",
            "content": "You are an AI Assistant. The time is "
            + str(datetime.datetime.now()),
        },
        {"role": "user", "content": prompt},
    ]
    return my_chat_model(model="gpt-3.5-turbo", messages=messages)


@traceable(run_type="chain")
def my_chat_bot(text: str) -> str:
    generated = my_chain(text)

    if "meeting" in generated:
        return my_tool(generated)
    else:
        return generated


my_chat_bot("Summarize this morning's meetings.")
# See an example run at: https://smith.langchain.com/public/3e853ad8-77ce-404d-ad4c-05726851ad0f/r

traceable functions work out of the box for synchronous and async calls. If you want to call sub-chains/functions on separate threads, you will have to manually pass the run tree to the child functions. To do so, you should update the parent function to accept a run_tree argument (which is injected by the decorator) and pass it to the child functions through the langsmith_extra keyword argument. Below is an example:

import asyncio
import datetime
from concurrent.futures import ThreadPoolExecutor
from typing import Any, Dict, List

import openai
from openai.openai_object import OpenAIObject
from langsmith.run_helpers import traceable
from langsmith.run_trees import RunTree


@traceable(run_type="llm")
def my_llm(prompt: str, temperature: float = 0.0, **kwargs: Any) -> OpenAIObject:
    """Call a completion model."""
    return openai.Completion.create(
        model="gpt-3.5-turbo-instruct", prompt=prompt, temperature=temperature, **kwargs
    )


@traceable(run_type="chain")
def llm_chain(user_input: str, **kwargs: Any) -> str:
    """Select the text from the openai call."""
    return my_llm(prompt=user_input, **kwargs).choices[0].text


@traceable(run_type="llm")
def my_chat_model(messages: List[Dict], temperature: float = 0.0, **kwargs: Any) -> OpenAIObject:
    """Call a chat model."""
    return openai.ChatCompletion.create(
        model="gpt-3.5-turbo", messages=messages, temperature=temperature, **kwargs
    )


@traceable(run_type="chain")
def llm_chat_chain(user_input: str, **kwargs: Any) -> str:
    """Prepare prompt & select first choice response."""
    messages = [
        {
            "role": "system",
            "content": "You are an AI Assistant. The time is "
            + str(datetime.datetime.now()),
        },
        {"role": "user", "content": user_input},
    ]
    return my_chat_model(messages=messages, **kwargs).choices[0].message.content


@traceable(run_type="chain")
async def nested_chain(text: str, run_tree: RunTree, **kwargs: Any) -> str:
    """Example with nesting and thread pools."""
    futures = []
    with ThreadPoolExecutor() as thread_pool:
        for i in range(2):
            futures.append(
                thread_pool.submit(
                    llm_chain,
                    f"Completion gather {i}: {text}",
                    langsmith_extra={"run_tree": run_tree},
                    **kwargs,
                )
            )
        for i in range(2):
            futures.append(
                thread_pool.submit(
                    llm_chat_chain,
                    f"Chat gather {i}: {text}",
                    langsmith_extra={"run_tree": run_tree},
                    **kwargs,
                )
            )
    return "\n".join([future.result() for future in futures])


asyncio.run(nested_chain("Summarize meeting"))

See the trace for a visualization of the run tree generated by this code.

For more information, check out the traceable notebook in the LangSmith cookbook.

When logging with the SDK, which fields can I update when I patch?​

The following fields can be updated when patching a run:

  • end_time: datetime.datetime
  • error: str | None
  • outputs: Dict | None
  • events: list[dict] | None

Once an end_time is set on a run, it is marked as "closed" and can no longer be updated. This is the case if you include an end time in the initial run creation post request or if you do so in a later patch request.

How do I search and filter runs?​

You can flexibly search and filter for runs directly in the web app or via the SDK. See exporting runs locally guide for more examples.

In the web app, you can search through 'traces' (the top level runs) or through all runs using the search bar in the runs table. This uses the run filtering query syntax.

How do I use the playground for runs logged using the SDK?​

The LangSmith playground doesn't yet support re-running arbitrary runs logged using the SDK. We are working to extend logging and playground support for users not using LangChain components in their app.

How do I mask sensitive information in my runs?​

If you need to completely hide the inputs and outputs of your traces, you can set the following environment variables when running your application:

LANGCHAIN_HIDE_INPUTS=true
LANGCHAIN_HIDE_OUTPUTS=true

This will block input or output data from being transmitted to the LangSmith API. The latency and sequence of operation types will still be visible, but all the inputs and outputs will be unavailable. Token count tracking is not available in this setting.