Subscribed unsubscribe Subscribe Subscribe

Using Mackerel in Auto Scaling environments

Autoscaling structures such as AWS Auto Scaling and Heroku which automatically increase and decrease the number of servers in response to load are gradually becoming more widely used. Here we’d like to introduce and talk about some ways Mackerel can be best used in Auto Scaling environments. For those who are using Heroku, in addition to this guide please refer to our guide to Monitoring Heroku with Mackerel.

mackerel-agent config for Auto Scaling environments

First we’ll introduce our mackerel-agent recommended configurations for Auto Scaling environments.

Setting hosts’ statuses to working when they start up

With hosts running in an Auto Scaling environment, we recommend configuring their statuses to automatically be set to working upon startup so that they begin monitoring and sending notifications. This can be done by designating the following settings in the mackerel-agent configuration file.

# /etc/mackerel-agent/mackerel-agent.conf
[host_status]
on_start = "working"

Even without assigning the above configurations, it you have the default host status set to working in your organization settings, the same result will be achieved. For more information please refer to Setting up monitoring and alerts

Automatic host retirement when scaling down

In Mackerel, a connectivity alert will be created if a host is suspended and the mackerel-agent terminates. This is beneficial for detecting sudden interruptions of regular hosts.

On the other hand though, host suspension isn’t considered abnormal behavior in Auto Scaling environments, so in Mackerel we’ll make it so that when a host is suspended, it is retired accordingly and alerts are not created. Depending on how the mackerel-agent is launched there are two different ways to do this, which we will describe below, so please follow the one that suits your circumstances.

Installing with rpm/deb and launching with init script

For those using package install, automatic host retirement can be set up simply by appending one line into the settings file of the environment variable for init script.

In the environment variable settings file, located at /etc/sysconfig/mackerel-agent for yum/rpm, and /etc/default/mackerel-agent for deb/apt, we append the following line.

AUTO_RETIREMENT=1

By making this designation, hosts will be retired automatically upon shutdown (more specifically /etc/init.d/mackerel-agent stop ).

Caution Even if /etc/init.d/mackerel-agent stop is typed in manually, if this configuration has been assigned the host will still be retired. Please use /etc/init.d/mackerel-agent reload to stop/start the agent when updating the agent version, etc.

For more information please refer to the mackerel-agent specs as well as “Auto host retirement and status change; automation for cold standby systems and auto scale”.

Launching with shell script

For those not using package install, or for those who are using package install but launching with a unique shell script as opposed to via init script, we will make it so that host retirement is handled automatically when the shell script terminates.

Automatic handling of host retirement can be carried out by using a subcommand for retirement mackerel-agent retire [-force] and the trap command. Now in the shell script, we will insert as a signal handler the retire command that used the trap command as shown below.

trap 'mackerel-agent retire -conf /etc/mackerel-agent/mackerel-agent.conf -force' INT TERM
mackerel-agent -conf /etc/mackerel-agent/mackerel-agent.conf

With this, if SIGINT or SIGTERM is sent, host retirement will be handled automatically and terminated, so in heroku or AWS Auto Scaling environments host registration and retirement will be handled automatically in accordance to host increases and decreases.

Viewing graphs for Auto Scaling

https://f.st-hatena.com/images/fotolife/m/mackerelio/20150730/20150730191150_original.png

In Mackerel, hosts that are added, removed, or replaced in an Auto Scaling environment included past hosts, will be displayed in role graphs as shown in the example above. Role graphs are graphs that display hosts grouped by role in each service screen.

Even in Auto Scaling environments, by assigning roles, easy to view graphs can be automatically generated, so we hope you’ll give assigning services and roles a try. For more concrete information about service and role configurations please refer to the mackerel-agent specs.