Using Mackerel in Auto Scaling environments

Autoscaling structures such as AWS Auto Scaling and Heroku which automatically increase and decrease the number of servers in response to load are gradually becoming more widely used. Here we’d like to introduce and talk about some ways Mackerel can be best used in Auto Scaling environments. For those who are using Heroku, in addition to this guide please refer to our guide to Monitoring Heroku with Mackerel.

mackerel-agent config for Auto Scaling environments

First, we’ll introduce the recommended configurations for mackerel-agent in Auto Scaling environments. Not compatible with Windows OS environments

Setting the host status to working upon startup

For hosts running in Auto Scaling environments, we recommend configuring the status to automatically be set to working upon startup so that they begin monitoring and sending notifications. This can be done by specifying the following settings in the mackerel-agent configuration file.

# /etc/mackerel-agent/mackerel-agent.conf
[host_status]
on_start = "working"

Even without assigning the above configurations, if the host's default status is set to working in your organization's settings, the same result will be achieved. For more information please refer to Setting up monitoring and alerts

Automatic host retirement when scaling down

In Mackerel, a connectivity alert is generated if a host is suspended and mackerel-agent terminates. This is beneficial for detecting sudden interruptions for regular hosts.

However, host suspension isn’t considered abnormal behavior in Auto Scaling environments. So, let's make the arrangements so that when a host is suspended, it is retired accordingly in Mackerel and alerts are not generated. Depending on how mackerel-agent is launched, there are two different ways to do this. Both methods are described below, so please follow the one that fits your situation.

Installing with rpm/deb and launching with init script

For those using package install, automatic host retirement can be set up simply by appending one line into the environment variable configuration file for init script.

In the environment variable configuration file, located at /etc/sysconfig/mackerel-agent for yum/rpm, and /etc/default/mackerel-agent for deb/apt, append the following line.

AUTO_RETIREMENT=1

By making this specification, hosts will automatically retire upon shutdown (more specifically /etc/init.d/mackerel-agent stop ).

Caution Even if /etc/init.d/mackerel-agent stop is entered in manually, if this specification is made, the host will be retired. Please use /etc/init.d/mackerel-agent reload to stop/start the agent when updating the version, etc.

For more information, please refer to the mackerel-agent specs as well as “Auto host retirement and status change; automation for cold standby systems and auto scale”.

Launching with shell script

For those not using package install, or for those who are using package install but launching with a unique shell script rather than via init script, here's how to set up automatic host retirement when the shell script terminates.

Setting up automatic host retirement can be carried out by using the retire subcommand mackerel-agent retire [-force] and the trap command. Now, insert the retire command as a signal banner using the trap command into the shell script that launches mackerel-agent as shown below.

trap 'mackerel-agent retire -conf /etc/mackerel-agent/mackerel-agent.conf -force' INT TERM
mackerel-agent -conf /etc/mackerel-agent/mackerel-agent.conf

As a result, host retirement will be handled automatically and terminated when SIGINT or SIGTERM is sent. So, host registration / retirement processing will be automatically carried out according to the increase and decrease of hosts in heroku or AWS Auto scaling environments.

Viewing graphs for Auto Scaling

https://cdn-ak.f.st-hatena.com/images/fotolife/m/mackerelio/20150730/20150730191150_original.png

In Mackerel, hosts that are added, removed, or replaced in an Auto Scaling environment, including past hosts, will be displayed in role graphs as shown in the example above. Role graphs are graphs that display hosts grouped by role in each service screen.

Even in Auto Scaling environments, easy to view graphs can be automatically generated by assigning roles, so we hope you’ll give assigning services and roles a try. For more concrete information about service and role configurations please refer to the mackerel-agent specs.

Points to note when creating Machine Images installed with mackerel-agent

In Mackerel, in order to uniquely identify hosts, each one is issued a "Host ID" and is managed in association to it (reference). ID information is saved in the "var/lib/mackerel-agent/id" file for Linux OS hosts and in the "id" file inside the installation folder for Windows OS.

When creating a custom Machine Image installed with mackerel-agent as the start-up instance’s base image when scale-out with Auto Scaling occurs, make sure that the image does not contain the file with this host ID. (If a scale-out occurs and the ID file has not been deleted, Mackerel will identify all started servers as the same.)