Mackerel blog #mackerelio

The Official Blog of Mackerel

“Critical alerts only” now possible for notification groups etc.

Hello. Development team director id:daiksy here.

Recently, I opened my window before bed expecting a warm night, but was surprised by a very chilly morning. This kind of weather makes controlling my physical condition pretty difficult. Is everyone out there staying healthy?

Here is this week’s update.

“Critical alerts only” now possible for notification groups

For some time we’ve been receiving requests to be able to change notification destinations and to have only urgent Critical alerts be sent to chats when notifications of monitoring results are made.

In this week’s release, we added the item “notification level” to the notification group’s setting screen. When the Critical only is toggled on, alerts to the notification group will only be sent under the following conditions and Warning alerts will not be sent.

  • Notifications will be sent for error levels of Critical or higher (Critical, Unknown)
  • OK notifications will be sent for alerts that have previously exceeded Critical

By combining notification groups with limited notification levels, notification destinations can flexibly be set according to the notification level.

Check out the help page as well.

Notice for new mackerel-agent repositories

New package repositories have been released for yum and apt targeted for the new Linux distribution. Check out the link below for more information.

Releases for mackerel-agent and mackerel-agent-plugins

In a previous post, we announced the support of HTTP/2 in mackerel-agent, but due to reports of mackerel-agent not running stably with some network devices, we released a version of mackerel-agent with the HTTP/2 support reverted.

Other changes including detailed content are as follows.

  • mackerel-agent 0.43.2

    • HTTP/2 support revert
  • mackerel-agent-plugins 0.28.0

    • Addition of mackerel-plugin-solr to package 
    • Addition of mackerel-plugin-aws-dynamodb (not packaged)
    • Addition of mackerel-plugin-redash (not packaged)

Notice for mackerel-agent’s new package repositories for yum and apt

We recently released new package repositories for yum and apt targeted for the new Linux distribution. We will continue to maintain existing package repositories, so there is no need to immediately change your existing environment. Continue below for detailed installation methods for each.

In this entry, the existing package repository will be referred to as “v1” and the new repository as “v2”.

Regarding supported distributions

The supported distributions of v1, v2 are as follows. Amazon Linux has its own repository, so this change will not affect it.

Repository Supported distributions
yum v1 CentOS 6/7
yum v2 CentOS 7 or later
apt v1 Debian 7/8, Ubuntu 14.04/16.04
apt v2 Debian 8 or later、Ubuntu 16.04 or later

v2 repository features

The v2 repository has the following features.

  • Systemd direct support
  • Drastic volume reduction for plugin packages
  • gpg signature for SHA256
  • Only offered for 64bit

Systemd direct support

We got rid of the SysVinit format startup file and now support Systemd directly. This will make running mackerel-agent more stable.

Additionally, logs are now output to Journal. You can check the mackerel-agent log with the following command.

sudo journalctl -u mackerel-agent.service

Drastic volume reduction for plugin packages

The build method was revised and the file sizes for the mackerel-agent-plugins and mackerel-check-plugins packages have been reduced to about 1/10th of the size. The current versions are as follows.

Package v1 v2
mackerel-agent-plugins(v0.28.0) 56.6MB 4.23MB
mackerel-check-plugins(v0.10.4) 20.5MB 2.75MB

How to update the package repository from v1 to v2

If you would like to switch to an environment where mackerel-agent etc. is already installed to the v2 repository, you’ll first need to delete the package installed with the v1 repository, delete the v1 repository configuration body, then set up the v2 repository.

The procedure for each deletion is provided below.

Before deleting anything, we recommend that you backup the configuration file (/etc/mackerel-agent/mackerel-agent.conf, / etc / sysconfig / mackerel-agent, / etc / default / mackerel-agent) just in case.

yum v1 repository removal

sudo yum remove -y mackerel-agent mackerel-agent-plugins mackerel-check-plugins mkr
sudo rm /etc/yum.repos.d/mackerel.repo
sudo yum clean all

After completing this procedure, the configuration files will be renamed and saved as /etc/mackerel-agent/mackerel-agent.conf.rpmsave /etc/sysconfig/mackerel-agent.rpmsave.

apt v1 repository removal

- sudo apt-get remove -y mackerel-agent mackerel-agent-plugins mackerel-check-plugins mkr
- sudo rm /etc/apt/sources.list.d/mackerel.list
- sudo apt-key del C2B48821
- sudo apt-get clean

The configuration file will not be deleted by performing this procedure.

How to install the v2 repository

After completing the above procedures, install mackerel-agent. Please refer to the following link for installation instructions.

Regarding the Provisioning Toolchain

We will be supporting the officially offered Chef’s cookbook and Ansible’s Role in sequential order.

Specifying multiple conditions in log monitoring is now easier with check-log etc.

Last week from May 31st to June 2nd, Mackerel ran a booth at the AWS Summit Tokyo 2017.

We had so many visitors, it was a very successful three days! A big shout out to all of you who came by the Mackerel booth, thank you!

Anyways, here is this week’s update.

Specifying multiple conditions in log monitoring is now easier with check-log

go-check-plugins v0.10.4 has been released, making it easier to specify multiple conditions in log monitoring using the check-log plugin.

Now, multiple --pattern options can be specified. When specified, they are treated with the AND operator (only log output lines that satisfy all specified patterns are detected).

The following is a configuration example to have alerts occur when a log output line that includes the two strings “PRODUCTION” and “FATAL” appears.

command = "check-log --file /var/log/access.log --pattern PRODUCTION --pattern FATAL"

Also, check out the help page below for more information.

Specifications for ELB metrics integrated with AWS Integration have changed

As you all may know, Mackerel’s AWS Integration feature is implemented using the Amazon CloudWatch.

Using the CloudWatch API to obtain ELB (CLB / ALB) metrics in relation to each HTTP status response count (e.g. HTTP Code ELB Count), responses without metric points themselves can be obtained at times when there is no response such as 4xx or 5xx.

Mackerel’s AWS integration also followed the CloudWatch API response, but was incompatible with Mackerel’s specifications and we received inquiries from multiple customers about a problem with monitoring not starting unless metric points existed. With this release, if there is no metric points, ELB metrics in relation to the response count for each HTTP status are now registered as 0.

This makes checking graphs more intuitive and also resolves the problem in the past where target metrics could not be selected if there were no 4xx or 5xx responses.

Updates for mackerel-agent-plugins and more

In addition to the updates made for go-check-plugins above, the following items have also been updated.

mackerel-agent-plugins v0.27.2

  • AWS Kinesis Firehose has been added
  • [mackerel-plugin-php-opcache] minor adjustments were made
  • [aws-rds] minor adjustments were made
  • Now built with Go 1.8

mkr v0.10.4

  • Minor adjustments were made

Thank you to everyone who contributed Pull Requests!

Lambda function integration targets can now be narrowed down with tags in AWS Integration

Hello! Sales engineer id:a-know here.

Right now, Mackerel is busy running a booth at AWS Summit Tokyo 2017! Today is the last day, but we’ve already received so many visitors over the these past two days. Thank you all so very much!

Only a few hours left, but I want to tell as many people as possible about Mackerel!

Anyways, here is this week’s update information.

Lambda function integration targets can now be narrowed down with tags in AWS Integration

You might recall the recent support of tags being made the other day for AWS Lambda, but now, Lambda function hosts to be registered can be narrowed down with tags in Mackerel as well.

For more details about the Lambda function integration of AWS Integration, refer to the help page below.

Mackerel support staff to present at Customer Success Event

Here’s a little something just before the event. I, Inoue, also in charge of Mackerel’s customer support, will be presenting at the “Customer Success Event vol.3” to be held on Monday, June 5th at the office of Good Patch Co., Ltd. (Japanese only)

The theme for this event is “Customer Success for Engineer targeted SaaS”. And although I’m technically a “sales engineer”, I am constantly thinking about how to make day-to-day server management more efficient with Mackerel while interacting with customers. I believe that this way of thinking has something to say about customer success, and I hope to talk a little bit about it at the event. For those interested, by all means, come join us.

The period an API key has not been in use is now displayed

It looks like it might rain here in Tokyo today. Recently, it seems like we’ve been getting a lot of rain and it makes me wonder if the rainy season is already upon us. That’s probably just down toward Okinawa though. I guess I’m a little impatient.

Anyways, here is this week’s update information.

The period an API key has not been in use is now displayed

In Mackerel, you can create API keys in addition to the default API key. It is also possible to create read-only keys which are really useful. On the other hand, there have been issues where keys that aren’t being used get left-over or keys that are actually being used are accidentally deleted.

With this update, the period an API key hasn’t been in use will be displayed for each key.

The time is displayed below the API key string. Time periods of less than two days will not be displayed.

Hopefully this will make it easier to manage API keys.

【Notice】Mackerel booth at AWS Summit Tokyo 2017

We made this announcement last week, but Mackerel is set to run a booth next week at AWS Summit Tokyo 2017.

AWS Summit Tokyo 2017 |AWS

We’re scheduled to attend the event from Wednesday, May 31st to Friday, June 2nd. New members of the sales team will be there along with Mackerel application engineers.

I hope many of you can make it!

Release of invitation API・You can now create alerts for check monitoring that do not automatically close etc.

Hello. Mackerel sales engineer id:a-know here.

I transitioned to my summer wardrobe the other day, but the mornings and nights are still kind of chilly. Although, this may be the most comfortable time of the year.

Anyways, here is this week’s update information.

The invitation API has been released

You can now invite other user accounts to an organization with the API.

Invitations can be created by specifying an email address and user permission.

Use this for tasks like adding a large number of new members or when promoting automation in the flow of joining a company.

You can now create alerts for check monitoring that do not automatically close

By specifying the option prevent_alert_auto_close in mackerel-agent.conf, alerts opened for check monitoring will not automatically close.

A configuration example is shown below.

command = "check-log --file=/path/to/file --pattern=CRITICAL"
prevent_alert_auto_close = true

Alerts created for check monitoring with this option specified will not automatically close.

In log monitoring for example, alerts can be left open even if the problem string is no longer detected. Depending on the monitoring target and type, you can use this for different purposes.

Please note that with this option, max_check_attempts will always be treated as 1 when used with max_check_attempts .

Check out the following help page as well for more on check monitoring with Mackerel.

【Notice】Response check behavior of URL external monitoring will change

We made this announcement from this blog the other day, but currently, response body checks are not performed for 3xx type responses in Mackerel’s URL external monitoring feature.

Specification changes to perform this response body check will be carried out within the next week.

Following this specification change, for monitoring items with the “response body check” option set, it is probable that alerts will arise, so please take a moment to review the settings.

Mackerel booth at AWS Summit Tokyo 2017

Mackerel is set to run a booth the week after next at AWS Summit Tokyo 2017.

AWS Summit Tokyo 2017 |AWS

We are scheduled to be at the event from Wednesday, May 31st to Friday, June 2nd. Members of our sales team and even some of our engineers will be in attendance.

For any of you who want to talk about the technical side of Mackerel, by all means, stop on by!

New user permissions・Stoppage notice for several OS support etc.

The first week following the Golden Week holidays is almost over…

Refreshed from the holiday break, the Mackerel development team continues its’ daily feature development.

This week we released a fairly big change.

Here is this week’s update.

New user permissions

New user permissions have been released and were announced in the entry below.

The three user permissions used up until now; “Owner”, “Editor”, and “Viewer” have been changed to “Owner”, “Manager”, “Collaborator” and “Viewer”.

“Collaborator” is the newly established permission which has the authority to make changes to monitoring configurations, services and roles, etc., but has limited control of operations concerned with the organization as a whole.

For more detailed information about the concepts of each permission and the changes made, check out the announcement blog entry above or the help page below.

Support for several OS has been stopped

We previously made this announcement which you read in the blog entry below, but the following OS support has ended.

  • CentOS5
  • Debian6
  • Ubuntu12.04

Although support has ended, problems such as mackerel-agent suddenly not running or server monitoring failing will not occur. Additionally, there are no plans to delete package repositories etc. at the moment. However, we can not guarantee that mackerel-agent will run with future updates.

From now on as well, Mackerel plans to stop OS support according to the support end date of each OS. Regarding the supported environments of Mackerel and mackerel-agent, please refer to the following page.

services subcommand added to mkr

mkr 0.16.0 has been released.

services has been added to the mkr subcommands and service information can now be retrieved.

$ mkr services
        "name": "sample service",
        "roles": [
        "name": "mackerelio"
        "name": "personal-server",
        "roles": [

Other updates for mackerel-agent, mackerel-agent-plugins and more are as follows.

  • mackerel-agent 0.43.0
    • The problem regarding the Windows version agent retrieving the transfer amount rather than the access count for the IOPS has been fixed
    • HTTP/2 support
  • mackerel-agent-plugins 0.27.1
    • Minor modifications
  • mkr 0.16.0
    • services subcommand added