Troubleshooting

This guide will walk you through common issues you may encounter when running a self-hosted instance of LangSmith.

While running LangSmith, you may encounter unexpected 500 errors, slow performance, or other issues. This guide will help you diagnose and resolve these issues.

The first step in troubleshooting is to check the logs of the various services that make up LangSmith.

If running on Kubernetes, you can check the logs of your deployment by running the following command (replace langsmith with the name of your deployment if you deployed with a different name):

kubectl logs -l app.kubernetes.io/instance=langsmith --prefix -n <namespace> >> logs.txt

If running on Docker, you can check the logs your deployment by running the following command:

docker compose logs >> logs.txt

Generally, the main services you will want to analyze are:

• langsmith-backend: The main backend service.
• langsmith-queue: The queue service.

Common issues

DB::Exception: Cannot reserve 1.00 MiB, not enough space: While executing WaitForAsyncInsert. (NOT_ENOUGH_SPACE)

This error occurs when ClickHouse runs out of disk space. You will need to increase the disk space available to ClickHouse.

Kubernetes

In Kubernetes, you will need to increase the size of the ClickHouse PVC. To achieve this, you can perform the following steps:

1. Get the storage class of the PVC: kubectl get pvc data-langsmith-clickhouse-0 -n <namespace> -o jsonpath='{.spec.storageClassName}'
2. Ensure the storage class has AllowVolumeExpansion: true: kubectl get sc <storage-class-name> -o jsonpath='{.allowVolumeExpansion}'
   • If it is false, some storage classes can be updated to allow volume expansion.
   • To update the storage class, you can run kubectl patch sc <storage-class-name> -p '{"allowVolumeExpansion": true}'
   • If this fails, you may need to create a new storage class with the correct settings.
3. Edit your pvc to have the new size: kubectl edit pvc data-langsmith-clickhouse-0 -n <namespace> or kubectl patch pvc data-langsmith-clickhouse-0 '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}' -n <namespace>
4. Update your helm chart langsmith_config.yaml to new size(e.g 100 Gi)
5. Delete the clickhouse statefulset kubectl delete statefulset langsmith-clickhouse --cascade=orphan -n <namespace>
6. Apply helm chart with updated size (You can follow the upgrade guide here)
7. Your pvc should now have the new size. Verify by running kubectl get pvc and kubectl exec langsmith-clickhouse-0 -- bash -c "df"

Docker

In Docker, you will need to increase the size of the ClickHouse volume. To achieve this, you can perform the following steps:

1. Stop your instance of LangSmith. docker compose down
2. If using bind mount, you will need to increase the size of the mount point.
3. If using a docker volume, you will need to allocate more space to the volume/docker.

error: Dirty database version 'version'. Fix and force version

This error occurs when the ClickHouse database is in an inconsistent state with our migrations. You will need to reset to an earlier database version and then rerun your upgrade/migrations.

Kubernetes

1. Force migration to an earlier version, where version = dirty version - 1.

kubectl exec -it deployments/langsmith-backend -- bash -c 'migrate -source "file://clickhouse/migrations" -database "clickhouse://$CLICKHOUSE_HOST:$CLICKHOUSE_NATIVE_PORT?username=$CLICKHOUSE_USER&password=$CLICKHOUSE_PASSWORD&database=$CLICKHOUSE_DB&x-multi-statement=true&x-migrations-table-engine=MergeTree&secure=$CLICKHOUSE_TLS" force <version>'

1. Rerun your upgrade/migrations.

Docker

1. Force migration to an earlier version, where version = dirty version - 1.

docker compose exec langchain-backend migrate -source "file://clickhouse/migrations" -database "clickhouse://$CLICKHOUSE_HOST:$CLICKHOUSE_NATIVE_PORT?username=$CLICKHOUSE_USER&password=$CLICKHOUSE_PASSWORD&database=$CLICKHOUSE_DB&x-multi-statement=true&x-migrations-table-engine=MergeTree&secure=$CLICKHOUSE_TLS" force <version>

1. Rerun your upgrade/migrations.

413 - Request Entity Too Large

This error occurs when the request size exceeds the maximum allowed size. You will need to increase the maximum request size in your Nginx configuration.

Kubernetes

1. Edit your langsmith_config.yaml and increase the frontend.maxBodySize value. This might look something like this:

frontend:
  maxBodySize: "100M"

2. Apply your changes to the cluster.

Details: code: 497, message: default: Not enough privileges. To execute this query, it's necessary to have the grant CREATE ROW POLICY ON default.feedbacks_rmt

This error occurs when your user does not have the necessary permissions to create row policies in Clickhouse. When deploying the Docker deployment, you need to copy the users.xml file from the github repo as well. This adds the <access_management> tag to the users.xml file, which allows the user to create row policies. Below is the default users.xml file that we expect to be used.

<clickhouse>
    <users>
        <default>
            <access_management>1</access_management>
            <named_collection_control>1</named_collection_control>
            <show_named_collections>1</show_named_collections>
            <show_named_collections_secrets>1</show_named_collections_secrets>
            <profile>default</profile>
        </default>
    </users>
    <profiles>
        <default>
            <async_insert>1</async_insert>
            <async_insert_max_data_size>2000000</async_insert_max_data_size>
            <wait_for_async_insert>0</wait_for_async_insert>
            <parallel_view_processing>1</parallel_view_processing>
            <allow_simdjson>0</allow_simdjson>
            <lightweight_deletes_sync>0</lightweight_deletes_sync>
        </default>
    </profiles>
</clickhouse>

In some environments, your mount point may not be writable by the container. In these cases we suggest building a custom image with the users.xml file included.

Example Dockerfile:

FROM clickhouse/clickhouse-server:24.8

COPY ./users.xml /etc/clickhouse-server/users.d/users.xml

Then take the following steps:

1. Build your custom image.

docker build -t <image-name> .

2. Update your docker-compose.yaml to use the custom image. Make sure to remove the users.xml mount point.

langchain-clickhouse:
  image: <image-name>

3. Restart your instance of LangSmith.

docker compose down --volumes
docker compose up