_Expect#

class langsmith._expect._Expect(*, client: Client | None = None)[source]#

A class for setting expectations on test results.

Methods

__init__(*[, client])

edit_distance(prediction, reference, *[, config])

Compute the string distance between the prediction and reference.

embedding_distance(prediction, reference, *)

Compute the embedding distance between the prediction and reference.

score(score, *[, key, source_run_id, comment])

Log a numeric score to LangSmith.

value(value)

Create a _Matcher instance for making assertions on the given value.
Parameters:

client (Optional[ls_client.Client])

__init__(*, client: Client | None = None)[source]#
Parameters:

client (Client | None)

edit_distance(prediction: str, reference: str, *, config: EditDistanceConfig | None = None) → _Matcher[source]#

Compute the string distance between the prediction and reference.

This logs the string distance (Damerau-Levenshtein) to LangSmith and returns a _Matcher instance for making assertions on the distance value.

This depends on the rapidfuzz package for string distance computation.

Parameters:

  • prediction (str) – The predicted string to compare.

  • reference (str) – The reference string to compare against.

  • config (Optional[EditDistanceConfig]) –

    Optional configuration for the string distance evaluator. Supported options: - metric: The distance metric to use for comparison.

    Supported values: “damerau_levenshtein”, “levenshtein”, “jaro”, “jaro_winkler”, “hamming”, “indel”.

    • normalize_score: Whether to normalize the score between 0 and 1.

Returns:

A _Matcher instance for the string distance value.

Return type:

_Matcher

Examples

>>> expect.edit_distance("hello", "helo").to_be_less_than(1)
embedding_distance(prediction: str, reference: str, *, config: EmbeddingConfig | None = None) → _Matcher[source]#

Compute the embedding distance between the prediction and reference.

This logs the embedding distance to LangSmith and returns a _Matcher instance for making assertions on the distance value.

By default, this uses the OpenAI API for computing embeddings.

Parameters:

  • prediction (str) – The predicted string to compare.

  • reference (str) – The reference string to compare against.

  • config (Optional[EmbeddingConfig]) –

    Optional configuration for the embedding distance evaluator. Supported options: - encoder: A custom encoder function to encode the list of input

    strings to embeddings. Defaults to the OpenAI API.

    • metric: The distance metric to use for comparison.

      Supported values: “cosine”, “euclidean”, “manhattan”, “chebyshev”, “hamming”.

Returns:

A _Matcher instance for the embedding distance value.

Return type:

_Matcher

Examples

>>> expect.embedding_distance(
...     prediction="hello",
...     reference="hi",
... ).to_be_less_than(1.0)
score(score: float | int | bool, *, key: str = 'score', source_run_id: UUID | str | None = None, comment: str | None = None) → _Matcher[source]#

Log a numeric score to LangSmith.

Parameters:

  • score (float | int | bool) – The score value to log.

  • key (str) – The key to use for logging the score. Defaults to “score”.

  • source_run_id (UUID | str | None)

  • comment (str | None)

Return type:

_Matcher

Examples

>>> expect.score(0.8)  
<langsmith._expect._Matcher object at ...>

>>> expect.score(0.8, key="similarity").to_be_greater_than(0.7)
value(value: Any) → _Matcher[source]#

Create a _Matcher instance for making assertions on the given value.

Parameters:

value (Any) – The value to make assertions on.

Returns:

A _Matcher instance for the given value.

Return type:

_Matcher

Examples

>>> expect.value(10).to_be_less_than(20)