How to setup Alerts with dynamic thresholds using Doctor Droid PlayBooks?

Why do we need dynamic alerts?

Businesses need more than just static thresholds in their alerts to tell if something is wrong. While most monitoring tools and their alerting features offer configuring of a single threshold against a metric, in reality an engineer is looking for much more complex rules like having different thresholds based on seasonality of your product and usage of business, doing more checks in logs or database other than just the concerned metric before needing to be notified when it's breaking the trend. Doctor droids dynamic alerting feature caters to these needs and help a developer remove false positive from landing in their inbox or the channel.

How to setup Dynamic Alerting in Doctor Droid?

We navigate to the Dynamic Alerts page in the platform.

Basic Details

First step would be to define the name of the alert.

Then we can configure what is the metric we want to setup.

Next we can fill in all the details to define the required metric.

Configuration

Now comes the configuration, or where the magic happens.

We can setup a window in which we want to evaluate this metric. We can simply enter a value in minutes.

Next, we can even enter some conditions here after which only I should be notified.

Let's say whenever our average is greater than four, only then I want to be notified. And when do we want to be notified? I can actually enter a cron here. Let's say we want to only be notified on Tuesday and Wednesday, we can set that rule in cron.

Next, we can add as many conditions we want, at any time we want. Let's see how it looks like:

Notifications

Whenever our conditions are specified, we want to be notified in a source (for example, slack). We can just select it there, select the channel and add what message do we want to send in the case our conditions are specified and our metric has crossed a given threshold.

With this, we have completed the dynamic alerting setup and we can save and start. This will activate our dynamic alert and it will keep listening and only notify us if all our conditions are crossed for that specific metric.

How it works under the hood?

The concept of dynamic alert is essentially a workflow with a keep-alive configuration for the schedule and a playbook working together. We can see the workflow and the playbook created for a dynamic alert you create in the pages for both of them respectively.

If you're interested to try out PlayBooks, sign up here or clone our repo from Github.