Using Mackerel in Auto Scaling environments

This page explains how to use Mackerel effectively in environments configured with autoscaling, which automatically increases and decreases servers in response to load.

Setting the host status to working when mackerel-agent starts

For hosts running in Auto Scaling environments, we recommend configuring the status to automatically be set to working upon startup so that they begin monitoring and sending notifications. This can be done by specifying the following settings in the mackerel-agent configuration file.

# /etc/mackerel-agent/mackerel-agent.conf
[host_status]
on_start = "working"

Even without assigning the above configurations, if the host's default status is set to working in your organization's settings, the same result will be achieved. For more information please refer to Setting up monitoring and alerts

Automatic host retirement

In Mackerel, a connectivity alert is generated if a host is suspended and mackerel-agent terminates. This is beneficial for detecting sudden interruptions for regular hosts.

However, host suspension isn’t considered abnormal behavior in Auto Scaling environments. So, let's make the arrangements so that when a host is suspended, it is retired accordingly in Mackerel and alerts are not generated. Please refer to the following for how to configure automatic host retirement.

Notes on auto retirement processing

• If auto retirement is enabled, the host will be retired not only when the OS shuts down, but also when mackerel-agent is manually stopped.
• The mechanism of auto retirement by mackerel-agent sends a retirement request to Mackerel when mackerel-agent stops. If OS shutdown processing takes priority over the agent stops before the request completes, retirement processing may not be performed successfully.
  • If you are using an AWS environment, you can also use automatic retirement via AWS Integration, so please consider this if necessary.

for Linux

Add the following line to the environment variable configuration file. For the location of the configuration file, please refer to Settings file for init script in the mackerel-agent specifications.

AUTO_RETIREMENT=1

If launching with a shell script

If you have not installed mackerel-agent as a package, or even if you have installed it as a package but are launching it with your own shell script rather than via an init script, configure automatic retirement processing to execute when the shell script terminates.

Setting up automatic host retirement can be carried out by using the retire subcommand mackerel-agent retire [-force] and the trap command. Now, insert the retire command as a signal banner using the trap command into the shell script that launches mackerel-agent as shown below.

trap 'mackerel-agent retire -conf /etc/mackerel-agent/mackerel-agent.conf -force' INT TERM
mackerel-agent -conf /etc/mackerel-agent/mackerel-agent.conf

As a result, when SIGINT or SIGTERM is sent, automatic retirement processing will be performed before termination, so host retirement processing will be automatically executed when an instance terminates.

for Windows

If using mackerel-agent on Windows, set the following value in the system environment variable.

MACKEREL_AUTO_RETIREMENT=1

You can configure this using the GUI or by executing the following command in the command prompt or PowerShell.

setx MACKEREL_AUTO_RETIREMENT 1 -m

for AWS Integration

If registering an EC2 host using AWS Integration, it can be automatically retired through AWS Integration.

When this function is enabled, the retirement process will automatically execute when the instance terminates. This function can be used in conjunction with the auto retirement function of mackerel-agent described above. This is an effective way to ensure that auto retirement is performed.

Viewing graphs for Auto Scaling

In Mackerel, hosts that are added, removed, or replaced in an Auto Scaling environment, including past hosts, will be displayed in role graphs as shown in the example above. Role graphs are graphs that display hosts grouped by role in each service screen.

Even in Auto Scaling environments, easy to view graphs can be automatically generated by assigning roles, so we hope you’ll give assigning services and roles a try. For more concrete information about service and role configurations please refer to the mackerel-agent specs.

Points to note when creating Machine Images installed with mackerel-agent

In Mackerel, in order to uniquely identify hosts, each one is issued a "Host ID" and is managed in association to it (reference). ID information is saved in the "var/lib/mackerel-agent/id" file for Linux OS hosts and in the "C:\Program Files\Mackerel\mackerel-agent\id" file inside the installation folder for Windows OS.

When creating a custom Machine Image installed with mackerel-agent as the start-up instance’s base image when scale-out with Auto Scaling occurs, make sure that the image does not contain the file with this host ID. (If a scale-out occurs and the ID file has not been deleted, Mackerel will identify all started servers as the same.)