Skip to main content

Alerts in LangSmith

Private Beta

This is feature is currently in private beta. If interested, please express interest in the #feature-requests channel in the LangChain Community Slack.

Overview​

Effective observability in LLM applications requires proactive detection of failures, performance degradations, and regressions. LangSmith's alerts feature helps identify critical issues such as:

  • API rate limit violations from model providers
  • Latency increases for your application
  • Application changes that affect feedback scores reflecting end-user experience

Alerts in LangSmith are project-scoped, requiring separate configuration for each monitored project.

Configuring an alert​

Step 1: Navigate To Create Alert​

First navigate to the Tracing project that you would like to configure alerts for. Click + New Alert in the top right hand corner of the page to set up an alert.

Step 2: Select Metric Type​


Alert Metrics

LangSmith offers threshold-based alerting on three core metrics:

Metric TypeDescriptionUse Case
Errored RunsTracks root runs with error statusMonitors for failures in application traces.
Feedback ScoreMeasures average feedback score metricsTracks feedback from application users or automated evaluation rules to track regressions in prompts or switching models / providers.
Trace LatencyMeasures average end-to-end trace execution timeIdentifies performance bottlenecks from switching models / providers or code regressions.

Step 2: Define Alert Conditions​

Alert conditions consist of several components:

  • Aggregation Method: Average, Percentage, or Count
  • Comparison Operator: >=, <=, or exceeds threshold
  • Threshold Value: Numerical value triggering the alert
  • Aggregation Window: Time period for metric calculation (currently choose between 5 or 15 minutes)
  • Feedback Key (Feedback Score alerts only): Specific feedback metric to monitor

Alert Condition Configuration

Example: The configuration shown above would generate an alert when more than 5% of runs within the past 5 minutes result in errors.

Step 3: Configure Notification Channel​

LangSmith supports the following notification channels:

  1. PagerDuty Integration
  2. Webhook Notifications

Select the appropriate channel to ensure notifications reach the responsible team members.

Best Practices​

  • Adjust sensitivity based on application criticality
  • Start with broader thresholds and refine based on observed patterns
  • Ensure alert routing reaches appropriate on-call personnel

Was this page helpful?


You can leave detailed feedback on GitHub.