Effective monitoring and alerting are essential for any organization seeking to ensure the performance and availability of its systems and services. Datadog’s alerting system, built around flexible and customizable monitors, provides teams with the tools to detect and respond to potential issues before they escalate.

By leveraging Datadog’s comprehensive capabilities—such as recovery modes, template variables, and dynamic alerts—engineers can tailor their alerting strategies to meet the unique needs of their infrastructure and applications.

In this guide, we’ll explore the foundational concepts of Datadog alerting, how to create and configure monitors, and the advanced features that help ensure reliable, real-time monitoring across your environment.

Recovery thresholds are optional conditions that can be added to a monitor to specify an additional requirement for transitioning out of alert or warning states. These thresholds define when the monitor should mark the issue as resolved, adding an extra layer of control over the recovery process.

The recovery threshold ensures that a monitor only switches to a "recovered" state once the metric crosses this specific threshold. If no recovery threshold is set, the monitor will automatically recover when the alert condition is no longer met.

The recovery threshold is met when the defined recovery condition is satisfied, and it varies depending on the alert condition:

For more detailed guidance on recovery modes and configuring thresholds, visit Datadog's Recovery Thresholds Documentation.

Template variables enable you to dynamically filter one or more widgets within a dashboard. By using template variables, you can create saved views based on your selected variable filters, allowing for easy organization and navigation of visualizations through dropdown menus. This helps streamline how you explore and analyze data across different contexts within the dashboard.

A template variable consists of the following components:

• Tag or Attribute:
  • Tag: This refers to the key in a key-value tagging format (<KEY>:<VALUE>), where the template variable is based on the <KEY>.
  • Attribute: Instead of a tag, a facet or measure can be used as the template variable.
• Name: A unique identifier for the template variable that is used in dashboard queries. By default, the name is derived from the selected tag or attribute.
• Default Value: The value that automatically appears when the dashboard is first loaded. By default, this is set to * (all values).
• Available Values: The options that appear in the dropdown menu for the variable. By default, all values (*) are available for selection. This allows querying of all possible values for the tag or attribute.

How to add a Variable?

To add a template variable to a dashboard:

• Click on Add Variables.
• If there are existing template variables, hover over the dashboard header and click the Edit button to enter edit mode.
• In edit mode, click the + (plus) icon to create a new template variable.
• (Optional) After selecting a tag, click the + Configure Dropdown Values button to rename the variable, set default values, or adjust the available selections.

For more details on how to configure template variables in Datadog, check out the Datadog Template Variables Guide.

Datadog provides several types of monitors, each designed to track and alert on various aspects of infrastructure, services, and applications. These monitors help teams detect potential issues, enabling them to take proactive measures to maintain system health and performance.

Here are the primary types of monitors in Datadog:

1. Metric Monitors Metric monitors track specific metrics (such as CPU usage, memory, disk space, etc.) and trigger alerts when these metrics exceed predefined thresholds. They allow for continuous monitoring and alerting based on real-time data. You can set different alert conditions depending on the performance metrics relevant to your environment.

Use Case: Monitor server CPU usage and trigger an alert when it exceeds 80% for more than 10 minutes.

2. Event Monitors: Event monitors track events within your infrastructure, such as system logs, application logs, or specific events (e.g., service restarts). These monitors help you capture and respond to critical events in real-time.

Use Case: Alert on critical errors in logs or service restarts to prevent issues from escalating.

3. Service Check Monitors: Service check monitors monitor the status of services within your infrastructure, such as the availability of databases, web services, or other critical systems. They help ensure that essential services are up and running and will alert you if any service becomes unavailable.

Use Case: Check the health of a web service and send alerts if the service becomes unresponsive.

4. Log Monitors: Log monitors allow you to create alerts based on log data, tracking specific log patterns or anomalies. These are useful for detecting performance bottlenecks, security issues, or other application-related incidents.

Use Case: Trigger an alert when the occurrence of "500 Internal Server Error" messages exceeds a certain threshold in the logs.

5. APM (Application Performance Monitoring) Monitors: APM monitors are designed to track the performance of your applications by observing metrics such as request latency, error rates, and throughput. These monitors help ensure your applications are running smoothly and will alert you if any performance degradation occurs.

Use Case: Alert if the latency of a critical API endpoint exceeds a defined threshold for a prolonged period.

6. Synthetic Monitors: Synthetic monitors simulate user interactions with your application to test performance, functionality, and availability. They allow you to monitor your application's performance from the perspective of end users by running test scenarios on websites, APIs, and other services.

Use Case: Simulate a user login scenario and trigger an alert if the login process takes too long or fails.

7. Composite Monitors: Composite monitors combine multiple monitors into one, allowing you to create complex alerting conditions. This can help reduce noise and ensure that alerts are only triggered when multiple conditions are met.

Use Case: Create a composite alert that triggers only when both high CPU usage and high memory consumption occur simultaneously on a server.

8. Process Monitors: Process monitors allow you to track and alert on specific processes running on your infrastructure. They provide visibility into running processes and enable you to set alerts based on CPU or memory consumption per process.

Use Case: Monitor a specific process on a server and alert if it consumes too much memory.

Each type of monitor in Datadog serves a unique purpose, offering flexibility in how you configure and manage alerts based on the performance, health, and stability of your infrastructure and applications.

For a more detailed overview of the types of monitors, visit the official Datadog Monitors Documentation.

Metric Monitors in Datadog allow you to monitor specific performance metrics across your infrastructure, services, and applications in real time. By setting thresholds and triggering alerts when metrics deviate from expected values, you can maintain control over the health and performance of your systems.

What Are Metric Monitors?

Metric Monitors track key performance indicators (KPIs) such as CPU usage, memory consumption, disk space, and more. They allow you to set conditions under which an alert is triggered, ensuring that you're notified when performance issues arise long before they impact users.

Key Features of Metric Monitors:

1. Threshold Alerts: You can create threshold-based alerts, which trigger when a specific metric crosses a defined threshold (e.g., when CPU usage exceeds 80%).
2. Evaluation Period: Set how long the condition needs to be true before the alert is triggered (e.g., if the threshold is exceeded for more than 5 minutes).
3. Recovery Thresholds: Define conditions under which the system is considered recovered and the alert is cleared.
4. Multi-Alert Conditions: Metric Monitors support multiple alert conditions based on different criteria, allowing for more granular control.

How to Create a Metric Monitor:

Image Source

1. Navigate to Monitors > New Monitor and select the Metric monitor type.
2. Define the metric you want to monitor, such as CPU usage or memory allocation.
3. Set the threshold and alert conditions. For instance, if you want to trigger an alert when disk space drops below 10%, specify this in the threshold section.

Image Source

1. Choose the notification channels for alerting (e.g., email, Slack, or webhooks).

Example Use Case:

Suppose you want to monitor CPU usage across multiple servers. You can set a Metric Monitor to alert when CPU usage exceeds 80% for more than 10 minutes. This allows you to respond to high usage before it affects system performance.

For more detailed steps, check out the official Datadog Metric Monitors Documentation.