Setting up monitoring and alerts

With Mackerel it’s possible to monitor your registered hosts by receiving notifications of alerts regarding the status of hosts.

Using the monitoring function effectively

  • In the event that you are not being notified of an alert, please check the alert settings page and user settings page to make sure everything is configured correctly.

Monitoring targeted hosts

  • Only hosts with their status set to “working” or “standby” will be monitored, whereas hosts with “maintenance” or “power off” status will not be targeted by monitoring.
  • Settings for the default starting status can be configured under the Settings tab in your organization’s page or it can be accessed directly through this link (https://mackerel.io/my?tab=setting). By setting the starting status as “working” you can begin receiving alert notifications immediately upon host registration.
  • Mackerel does not send alert notifications for hosts with a status of “standby”, however you can view these alerts on the web.
status monitoring notifications
working yes yes
standby yes no
maintenance no no
poweroff no no

About alerts when a host retires or the status changes

When a host retires, alerts related to that host will automatically be closed.

When a host’s status changes, automatic operations regarding the alert will not be performed.

Notifications will not be sent for alerts that occur when the host’s status is “standby”, even if the status is changed to “working”. In this case, either the status of the alert has changed afterward or the re-sending interval has been configured and a notification will be sent if the interval is passed.

In the event that a host’s status is changed to “maintenance” or “poweroff”, alerts that have already occurred will not close automatically, even if threshold abnormalities and communications are restored. Returning the status to “working” or “standby”, alerts will automatically close once the abnormal state recovers.

Connectivity Monitoring

Once mackerel-agent has been installed and launched, connectivity monitoring of hosts will be done automatically.

Connectivity monitoring works by looking at intervals of metric data from mackerel-agent. In the event that no data is being reported from a host, Mackerel will determine the source of the problem and send an alert.

Metric monitoring

Metric data (including custom metrics) can be monitored by setting thresholds, which are configured in two levels: warning and critical.

A threshold is generally configured as a limit greater than/less than a designated value, while CPU usage and memory usage thresholds will be set in relation to the percentage of the total value.

Monitors

The following items are included in monitoring rules.

  • Name
  • Metric targeted by monitoring
  • Warning parameters
    • Greater than/less than value
    • Threshold
  • Critical parameters
    • Greater than/less than value
    • Threshold
  • Duration of given conditions
    • The average value of conditions in this time span will be monitored and alerts will be generated
  • Services and roles to which the host being monitored belongs

A few things to keep in mind:

  • Monitoring threshold values can be changed at any time, but the target metric cannot be changed.
  • If a threshold is changed, alerts which were already raised will not be closed automatically, so it is necessary to close those alerts manually.
  • Hosts can be assigned to multiple services and roles. If no services or roles are designated, all hosts will be targeted by monitoring.
  • In the case that all roles to which a monitor’s target have been designated are deleted, the monitor’s target will become all hosts.
  • In the case of one metric being targeted by multiple monitors, each monitor will operate individually with its respective parameters.

Creating monitors

The list of all monitors can be viewed in the Monitors page. To create a new monitor, click the “New Monitor” button in the upper right hand corner, enter the new monitor’s parameters and then click “Create”.

Checking alerts

Alerts can be checked by going to the “Alerts” page.

Closing alerts

If an alert has come up but is no longer meeting the parameters set for that alert (i.e. if connectivity to a host has been restored, or if a metric has returned to a value within the threshold) that alert will automatically be closed. Alerts can also be closed manually by clicking the “Close this alert” button either in the “Alerts” page or in an alert dialogue window.

Disabling alert notifications

Alert notifications can be temporarily or permanently suspended by going to Monitors and opening the drop down menu by clicking the arrow next to the “Turn off notifications” button. If alert notification is set to “disabled” or “suspended”, notifications will not be sent in the event an alert is raised. (Alerts, however, will continue to function as normal)

Notifications

Mackerel will notify you in the event that an alert has been raised, resolved/closed, or if the status of an alert has changed.

Notification settings can be edited in Channels Settings. Supported notification channels other than email and Webhook include Slack, HipChat, Typetalk, ChatWork, PagerDuty, im.kayak.com and many more. We will continue to add more supported notification channels as need dictates.

With notification by email broadcast, all members belonging to your organization will be notified simultaneously (this can be changed in Personal Settings so as not to be notified). For details on setting up other notification channels, please refer to the Alerts section of the Help Documents sidebar.

To stop receiving notification emails (User Settings)

To stop receiving notification emails you can change your preferences in User Settings.

Notification Groups

With Mackerel’s notification groups, it’s possible to configure even more flexible custom notifications, for example setting alert notifications from only one specific service to be sent to one team’s notification channel. Just like regular notification channels, notification groups can also be created in the Channels Settings page.

Only notifications pertaining to the services and monitoring rules which have been configured will be sent to the channels included in that notification group.

Notification group that receives all notifications

In Mackerel there is a default notification group that will receive all notifications associated with that whole organization. When setting up a new notification channel it will automatically be included in the default notification group.

Ignore other notification groups

When designating a monitoring rule within a notification group, by checking the box next to “Ignore other notification groups.”, notifications from that monitoring rule will be sent only through that notification group, and your organization’s default notification group as well as any other notification groups will no longer receive notifications generated by that monitoring rule.

Example configuration

For example, the configuration for [alerts from connectivity monitoring rules are sent to the appropriate team and all other notifications go to email and Slack] would look like this:

Notification level

If Critical only is On, Warning notifications will not be sent. Detailed notification conditions are as follows.

  • Notifications will be sent for error levels of Critical or higher (Critical, Unknown)
  • OK notifications will be sent for alerts that have previously exceeded Critical

-

If you have any questions please contact our support team at support@mackerel.io