LLM Observability

Datadog is a popular infrastructure & application monitoring platform which has extended some of it’s integrations within the space of LLMs and related tooling. It’s interesting to see it’s out of the box dashboards for LLM Observability, yet it’s clear that there’s a lot more to be aspired from a brand like Datadog here.

Pricing

Pricing as per the Metrics / Traces usage within Datadog

Things To Consider

Only supports OpenAI integrations currently with depth. There is only cloud option like the normal Datadog offering.

They do not support LLM experimentation or iterations like the other tools mentioned above.
Their OOB compatibility for integrations, frameworks and endpoints is limited to the top few options like OpenAI, Langchain.

‍

Benefits

If you are already using Datadog for tracing, then this can be enabled to setup OpenAI usage tracing with just a flag change.
They focused on associated infrastructure monitoring like vectordbs too.