Scalyr Alerts have a number of helpful features that can be used to test and diagnose missing alert notifications. This article covers some of the best practices we use.
Instead of waiting until your alert is triggered, you can test your alert's notifications immediately by setting the trigger to a boolean value of true
After 1-2 minutes, you should begin to see notification activity.
Note: If you are using the
#lastLogLines# token, no log lines will be displayed if you are using a trigger of
true (since no logs were actually processed)
Confirm your alert's status
Check for webhook errors
In our experience, configuring webhooks with JSON payloads can require more initial effort, due to their formatting requirements. One way to quickly spot a problem is to search for
If a faulty payload is causing your notifications to be rejected, you'll see log events similar to:
In the above example, I realized that I had inadvertently double escaped (\\\") my JSON payload in the alerts configuration, which resulted in the payload error (and no notifications being sent to Slack)
Review the alert's current status
If you're testing an alert, you can review its status and when it was triggered by searching for
In rare instances, a new alert may not trigger within the usual 2-3 minutes because its histograms are still being created. Once these histograms have been generated, the alert will function as expected.
$tag='alertState' is a good way to investigate a late / missing notification. If no matching log lines are returned, there's a good chance that histogram generation is the underlying cause.