Observability refers to the ability to measure the internal state of a system based on the data it generates. This includes capturing metrics, logs and traces. While it is quite popular for engineering teams in terms of measuring their infrastructure & services, a new category has recently been created within it it due to the popularisation of LLMs. This specifically focuses on tools and packages that help measuring the cost, quality & security of LLM usage within organisations.

‍

Fun Fact: As per the State of CI / CD report 2024 by CD Foundation, AI-assisted coding tools is the 5th most popular category of tools adopted by engineers, at par with IDEs!.

User experiences are evolving into more intuitive & personalised as the usage of LLMs is picking up pace across companies. Large corporations like Google, Meta were already offering these experiences in several of their customer & business products but with ChatGPT’s launch in 2022, more companies now have the tools to build these. Since then, the number of available LLMs, both closed and open source, have increased exponentially.

While the architecture around LLM usage still needs the conventional observability setup, since the LLM is mostly a separately deployed entity outside your code and accessed in a prompt + response model, it creates a need of qualitative observability. Every response, before being used, needs to be checked for hygiene and relevance. That means you need to set up logging of your LLM prompts & responses and then contextual analysis on top of them that fulfills your monitoring purpose.

1. Help track & improve response quality
   • Relevant to query and are fact based
   • Don’t have biases for any user groups or type of data
   • Improving the experience to justify being used
   • Use the additional context (passed as a vector dataset) optimally
2. Help track & improve response reliability
   • Latency is acceptable
   • Quality and format remains consistent
   • Maintain user level data silos
3. Build visibility of the cost of using the LLM, especially when using a third party managed model

Architecture of LLM Observability tools

Typically, the instrumentation for LLM Observability is done in 2 ways:

1. Tracing - Trace the calls you make (with or without an agent) and create spans for request, response data to create metrics and do quality analysis. This is useful for cases where you would want to only look at what happened and don’t want any interference in your usage

2. Proxy - Replace the LLM calls to a proxy layer which intercepts your calls and make those calls on your behalf. These help in controlling some trivial parameters without you worrying about them but could lead to application downtime in case the proxy is down.

‍

Looking at the data collected from your LLM usage, you can do multiple optimisations to improve either the cost, quality, user behaviour or security of your product:

1. Cost optimisation for usage:

• The tool would provide you input/output tokens volume for every request being sent to an LLM. This helps you look at the cost overhead of your request, both at an individual level or aggregate.‍
• Custom Tagging: In case you are looking for attributing cost to different entities, by adding tags for use-cases / accounts.

2. A/B Experimentation on Prompts / Prompt Engineering:

• Try out multiple prompt templates, keep them side by side and explore which prompt give best results and iterate on them. What would otherwise require extensive manual interpretation on a UI layer like chatGPT, will be done programmatically and tracked in the LLM Observability tool.

• Fun Fact: “Prompt engineering” as a keyword is less than 3 years old, yet has more than 2000 openings for “Prompt Engineer” just on Linkedin Job portal.

3. Evaluate the right LLM for your use-case: The below screenshot shows an output from empirical.run — an open source project shows the comparison between different models and their outputs for given prompts. Such comparisons are generally available within LLM Observability tools and help you accelerate your journey to identifying the best model for your use-case.

4. Improve RAGs: RAGs is Retrieval-Augmented Generation strategy that help leverage existing / private context to help the LLM answer questions better. An LLM Observability tool will also help you efficiently iterate over your context retrieval based queries, w.r.t prompts, chunking strategy and vector store format.

5. Fine-tuned model performances: Analyse how training the LLM with your data is affecting outputs and if there’s any significant different on the output quality between different versions of your model.

1. Langsmith
2. Langfuse
3. Helicone
4. Lunary
5. Phoenix (by Arize)
6. Portkey
7. OpenLLMetry (by traceloop)
8. Trulens
9. Datadog
10. Other tools evaluated