Datadog Agent Agent not collecting custom metrics

Custom metrics are not configured properly or the agent lacks permissions to access the data source.

Understanding Datadog Agent

Datadog Agent is a lightweight software that runs on your hosts. It collects events and metrics from your hosts and sends them to Datadog, where you can analyze and visualize them. The agent is crucial for monitoring the performance of your infrastructure and applications.

Identifying the Symptom

One common issue users face is the Datadog Agent not collecting custom metrics. This can be frustrating as it prevents you from gaining insights into specific application behaviors or infrastructure performance that are not covered by default metrics.

Exploring the Issue

The primary reason for the agent not collecting custom metrics is often due to improper configuration or insufficient permissions. Custom metrics need to be explicitly defined and the agent must have access to the data source from which these metrics are derived.

Configuration Errors

Ensure that the custom metrics are correctly defined in the configuration files. Check the syntax and paths specified for any discrepancies.

Permission Issues

The agent must have the necessary permissions to access the data source. This could involve setting appropriate user permissions or network access rules.

Steps to Fix the Issue

Verify Custom Metric Configuration

First, check the configuration file where custom metrics are defined. This is typically located in the /etc/datadog-agent/conf.d/ directory. Ensure that the YAML syntax is correct and that the metrics are properly specified.

custom_metrics:
- name: my_custom_metric
type: gauge
value: "my_metric_value"

Check Permissions

Ensure the Datadog Agent has the necessary permissions to access the data source. This might involve setting the correct user permissions or ensuring the agent runs with sufficient privileges. You can verify the agent's permissions by checking the user under which it runs:

ps aux | grep datadog-agent

Ensure Data Source Accessibility

Verify that the data source is accessible from the host where the Datadog Agent is running. You can use network tools like curl or ping to test connectivity:

curl http://your-data-source-endpoint

Additional Resources

For more detailed guidance, refer to the official Datadog Agent documentation. If you continue to experience issues, consider reaching out to Datadog Support for further assistance.

Master

Datadog Agent

in Minutes — Grab the Ultimate Cheatsheet

(Perfect for DevOps & SREs)

Most-used commands
Real-world configs/examples
Handy troubleshooting shortcuts
Your email is safe with us. No spam, ever.

Thankyou for your submission

We have sent the cheatsheet on your email!
Oops! Something went wrong while submitting the form.

Datadog Agent

Cheatsheet

(Perfect for DevOps & SREs)

Most-used commands
Your email is safe with us. No spam, ever.

Thankyou for your submission

We have sent the cheatsheet on your email!
Oops! Something went wrong while submitting the form.

MORE ISSUES

Made with ❤️ in Bangalore & San Francisco 🏢

Doctor Droid