Langsmith is a product offering by Langchain, one of the fastest growing projects for LLM orchestration in the early days. It was launched in July 2023 and so far has over 100K users, one of the largest communities for a tool in the LLM space. They have also launched an LLM Observability platform, Langsmith.

Pricing

While Langchain is open source, we could not find any Github repo for Langsmith apart from it’s SDKs. It offers only a cloud Saas solution with a free-tier offering of 5K traces month per month. Self-hosting is available only as add-on in Enterprise plan.

Things To Consider

There is no self-hosting option within their self-serve module. They provide cost analysis and insights, but only for OpenAI usage currently.

Benefits

In-built with Langchain: Its a tracing tool, comes in-built with Langchain. So, if you are a langchain user, no changes are needed it publishes traces for your LLM calls to its cloud.
Feedback loop: It allows you to manually rate your responses or use an LLM to do so. It works with non-langchain agents also.