Autoscaling structures such as AWS Auto Scaling and Heroku which automatically increase and decrease the number of servers in response to load are gradually becoming more widely used. Here we’d like to introduce and talk about some ways Mackerel can be best used in Auto Scaling environments. For those who are using Heroku, in addition to this guide please refer to our guide to Monitoring Heroku with Mackerel.
- mackerel-agent config for Auto Scaling environments
- Points to note when creating Machine Images installed with mackerel-agent
mackerel-agent config for Auto Scaling environments
First we’ll introduce our mackerel-agent recommended configurations for Auto Scaling environments.
Setting hosts’ statuses to
working when they start up
With hosts running in an Auto Scaling environment, we recommend configuring their statuses to automatically be set to
working upon startup so that they begin monitoring and sending notifications.
This can be done by designating the following settings in the mackerel-agent configuration file.
# /etc/mackerel-agent/mackerel-agent.conf [host_status] on_start = "working"
Even without assigning the above configurations, it you have the default host status set to
working in your organization settings, the same result will be achieved.
For more information please refer to Setting up monitoring and alerts
Automatic host retirement when scaling down
In Mackerel, a connectivity alert will be created if a host is suspended and the mackerel-agent terminates. This is beneficial for detecting sudden interruptions of regular hosts.
On the other hand though, host suspension isn’t considered abnormal behavior in Auto Scaling environments, so in Mackerel we’ll make it so that when a host is suspended, it is retired accordingly and alerts are not created. Depending on how the mackerel-agent is launched there are two different ways to do this, which we will describe below, so please follow the one that suits your circumstances.
Installing with rpm/deb and launching with init script
For those using package install, automatic host retirement can be set up simply by appending one line into the settings file of the environment variable for init script.
In the environment variable settings file, located at /etc/sysconfig/mackerel-agent for yum/rpm, and /etc/default/mackerel-agent for deb/apt, we append the following line.
By making this designation, hosts will be retired automatically upon shutdown (more specifically
/etc/init.d/mackerel-agent stop ).
Caution Even if
/etc/init.d/mackerel-agent stop is typed in manually, if this configuration has been assigned the host will still be retired. Please use
/etc/init.d/mackerel-agent reload to stop/start the agent when updating the agent version, etc.
For more information please refer to the mackerel-agent specs as well as “Auto host retirement and status change; automation for cold standby systems and auto scale”.
Launching with shell script
For those not using package install, or for those who are using package install but launching with a unique shell script as opposed to via init script, we will make it so that host retirement is handled automatically when the shell script terminates.
Automatic handling of host retirement can be carried out by using a subcommand for retirement
mackerel-agent retire [-force] and the trap command.
Now in the shell script, we will insert as a signal handler the retire command that used the trap command as shown below.
trap 'mackerel-agent retire -conf /etc/mackerel-agent/mackerel-agent.conf -force' INT TERM mackerel-agent -conf /etc/mackerel-agent/mackerel-agent.conf
With this, if SIGINT or SIGTERM is sent, host retirement will be handled automatically and terminated, so in heroku or AWS Auto Scaling environments host registration and retirement will be handled automatically in accordance to host increases and decreases.
Viewing graphs for Auto Scaling
In Mackerel, hosts that are added, removed, or replaced in an Auto Scaling environment, including past hosts, will be displayed in role graphs as shown in the example above. Role graphs are graphs that display hosts grouped by role in each service screen.
Even in Auto Scaling environments, by assigning roles, easy to view graphs can be automatically generated, so we hope you’ll give assigning services and roles a try. For more concrete information about service and role configurations please refer to the mackerel-agent specs.
Points to note when creating Machine Images installed with mackerel-agent
In Mackerel, in order to uniquely identify hosts, each one is issued a “Host ID” and is managed in association to it (reference). ID information is saved in the “var/lib/mackerel-agent/id” file for Linux OS hosts and in the “id” file inside the installation folder for Windows OS.
When creating a custom Machine Image installed with mackerel-agent as the start-up instance’s base image when scale-out with Auto Scaling occurs, make sure that the image does not contain the file with this host ID. (If a scale-out occurs and the ID file has not been deleted, Mackerel will identify all started servers as the same.)