Mackerel blog #mackerelio

The Official Blog of Mackerel

Announcement concerning specification adjustments to the average value calculation method of metric monitoring

Mackerel Sub Producer id:Songmu here. As stated in the title, this is an announcement concerning adjustments made to the specifications of the average value calculation method for metric monitoring.


The average value calculation method for metric monitoring will be uniformly changed to the average value of "the number of points".

This will affect host metric monitoring as well as response time monitoring of URL external monitoring. The average values for these metrics are currently being calculated for "N minutes". Service metrics will not be affected.

Potentially affected users

As these adjustments are minor, there generally won’t be any big effects.

There is a possibility that users using host metric monitoring or external response time monitoring, furthermore, those with an average value monitor set for 2 minutes or more may be affected.

In particular, the likelihood of impact is higher for metrics posted at intervals other than 1 minute. For example, there are some AWS integration metrics posted at 5 minute intervals.

Date of change

February 8th 2018 (Thursday)

A concrete comparison of the current and new specifications

The average value calculation target for the following metric monitoring will change as shown below.

Current status New specifications
Host metric monitoring N minutes N points
Response time monitoring N minutes N points
Service metric monitoring N points N points

Differences between "N minutes" and "N points"

The average value for metrics targeted in “N minutes” is calculated on the premise that data be posted every minute. In some cases, this behavior is not intuitive, such as the following.

  • When data is posted at an interval other than 1 minute
    • For example, at an interval of 5 minutes, etc.
  • When metric data is temporary missing
    • For example, in plugins that performs diff calculation for the counter value, metrics are not calculated when the counter decreases

In these cases, when the average value monitor is set for 2 minutes or more, the average value is not calculated for the 1 minute of the value not being measured (null).

In other words, you can’t realistically configure an average value monitor of 2 minutes or more for 5 min interval host metrics and monitoring doesn’t work when data is temporarily missing and the average values of before and after can’t be calculated.

The following example specifically illustrates how the average of N minutes and the average of N points are calculated differently when there is missing data.

Time Raw metrics 3 minute average 3 point average
15:00 10 - -
15:01 11 - -
15:02 12 11 11
15:03 13 12 12
15:04 null null 12
15:05 14 null 13
15:06 15 null 14
15:07 16 15 15

This specification adjustment will unify the average value calculation in N points.

Additional information

With this adjustment, average value monitoring now works with metrics of arbitrary intervals. However, as with current service metrics, only metric points within the last 24 hours are eligible. In other words, if you are posting metrics on a daily basis, monitoring the average value of 2 points or more will not work well.

In cases where the posting interval is not constant, data is not weighted according to the interval and the average value is simply calculated.

Additionally, the shortest metric interval remains unchanged at 1 minute. In other words, it is not currently possible to save multiple metric points between one minute intervals. Even if multiple metrics are posted with a precision of less than 1 minute, they are rounded up to 1 minute and overwritten with the most recently posted value.

As we continue striving to improve our services, we appreciate your understanding and cooperation.

On 2/27 (Tuesday) the system will temporarily shutdown for database maintenance

Mackerel Product Owner id:Songmu here. As stated in the title, system maintenance is scheduled to be carried out this month.

Because this maintenance will require the system to be temporarily shutdown for a relatively long period of time, we understand that this will be an inconvenience for all of our users and we apologize. Nevertheless, this maintenance is indispensable to continuously providing better service in the future. We appreciate your understanding and cooperation.

Scheduled date and time

Tuesday, February 27th, 2018 from 2:30 p.m. - 5:30 p.m. (JST)

Implementation content

Database maintenance

Regarding the impact on the day of

  • The maintenance completion time stated above, is an estimate of the longest case scenario. The actual maintenance period will end once the work has been completed.
  • After maintenance has begun, the entire Mackerel system will shutdown for a short period of time
    • Web access to Mackerel, data posting by the agent, API access (including the CLI tool), alert notifications, etc. will be unavailable
  • As soon as the maintenance work is completed and operation confirmation is obtained, the system suspension will end and an announcement will be made
  • As for mackerel-agent metric posting, data will be buffered from mackerel-agent during the maintenance period and resent after maintenance has been completed
    • If resent correctly, graphs during the maintenance period will also be displayed

Regarding announcements on the day of

Announcements will be made from the Mackerel status page ( as well as from this blog (

Additionally, we’ll also be using our official Twitter account (

For inquiries related to this matter

Please send all inquiries regarding this matter to

Thank you again for your understanding and cooperation. And thank you for choosing Mackerel.

The metric retention period and the period to check metrics of 1 min granularity have been extended etc.

Mackerel users, thanks for waiting! The following large updates were made regarding metric retention with Mackerel.

  • The Standard plan’s metric retention period has been extended to【460 days】 (previously【400 days】).
  • The period to check metrics of 1 min granularity is now【460 days】 (previously【25 hours】)
    • Due to feature implementation timing, checking 1 min granularity metrics is only available for data after December 1st 2017.

I am very happy to be able to bring this update news to all of you. Now it’s even easier to check server load seasonality and apply capacity planning. Definitely give it a try!

Other updates follow below.

You can now set an email address to receive Organization related mail

As you can see in the image above, it’s now possible to add an email address to receive mail related to "Payment" and "Support Team contact".

Up until now, "payment related email" was sent to users with owner authority and “emails exchanged with the Mackerel support team” were sent to the user who contacted the support team. Now, in addition to having mail sent to each user, you can have mail sent to an address of your specification.

This can be configured in the Organization’s settings.

Updates for Mackerel related OSS

Updates for various Mackerel related OSS have been made. The details follow below.

mackerel-agent v0.51.0

  • [Windows] An issue where the memory pagefile value was mistakenly multiplied by 1024 has been fixed.

mackerel-agent-plugins v0.43.0

  • Plugin passwords can now be passed with environment variables. The target plugins and environment variable names are as follows.
    • mackerel-plugin-postgres
    • mackerel-plugin-openldap
    • mackerel-plugin-redis
    • mackerel-plugin-haproxy
    • mackerel-plugin-sidekiq
    • mackerel-plugin-mysql
    • mackerel-plugin-rabbitmq
    • mackerel-plugin-mongodb

mackerel-check-plugins v0.16.0

  • Plugin passwords can now be passed with environment variables. The target plugins and environment variable names are as follows.

mkr v0.26.0

  • [Windows] An issue where the .zip file was also copied to the bin directory when doing mkr plugin install has been fixed.

To everyone who contributed Pull Requests, thank you!

Mackerel Meetup #11

Mackerel Meetup #11, the first official event of 2018 will be held on February 5th! (Japanese only)

We are pleased to announce that two Mackerel user companies, Seesaa Co, Inc. and Makuake , Inc. (in order of presentation) will be giving guest presentations! There’s still plenty of room left for both general and blog participants. We’ll be talking about the future direction of the evolution of Mackerel, so you don’t want to miss out!

Command line tool・mkr now included in the package for Windows etc.

Hello! Mackerel team CRE Inoue (id:a-know) here.

I thought this week was going to start off with a big cold front, but it turned out warming up quite a bit, with weather comparable to that of the month of March. With these changes, it’s been difficult to regulate body temperature. I hope everyone is staying healthy out there.

Luckily, I personally haven’t had any big health problems this winter and I’d like to keep it that way. I’ll be sure to keep “monitoring” my various “metrics”... maybe I’ve already caught an occupational illness? (lol)

Now on to this week’s update information.

Command line tool・mkr now included in the package for Windows

With this week’s update, mkr, the command line tool that strongly supports the implementation of various operations regarding Mackerel via command line is now included in the package for Windows. Now it’s even easier to use mkr.

For more on the basic usages of mkr and various helpful cases using the tool, refer to the following help pages.

When using mkr in a server that is running mackerel-agent, the configuration file inside the server is automatically referenced, so it is not necessary to specify MACKEREL_APIKEY or your own host ID .

mkr status
mkr retire

Updates for mackerel-agent(v0.50.1)

The following updates have been made for mackerel-agent.

  • Command line tool・mkr is now included in the package for Windows (Described above)
  • Stability improved when using the v1 package
  • mackerel-agent once behavior changed
    • mackerel-agent once is a command that can check behavior by manually executing mackerel-agent once.
    • Until now, even if metric collection failed, the exit status would be 0 (normal handling).
    • With this update, failure to collect metrics will end with the exit status 1 (abnormal handling).

Notification content to PagerDuty improved・Updates for the agent and more

Happy New Year everybody! Mackerel team CRE Inoue (id:a-know) here.

Today’s release is the first of 2018. This year we are excited to bring all of our users lots of advanced and useful features.

Now on to this week’s update information.

Notification content to PagerDuty improved

The following items have been added to the json content that is included in the notification request for PagerDuty notifications, one of Mackerel’s notification channels.

  • In the event of an alert notification to the host, newly included items are:
    • Host ID, Host name, a list of roles to which the host belongs
  • In the event of an alert notification to the service, newly included items are:
    • Service name

Available with either PagerDuty API V1 or V2, this update makes notification content even more programmable. By all means, try it out.

mackerel-agent now supports plugin timeout setting

There are plugins with various possible configurations for mackerel-agent. Although convenient, up until now these plugins all had a uniform specification of 30 seconds to timeout.

With this update, it is now possible to set an arbitrary number of seconds for each plugin. For example, if you want to set the timeout to 45 seconds, write the following.

command = "ruby /path/to/get-metrics.rb"
timeout_seconds = 45

This can also be specified for check plugins.

command = "ruby /path/to/check-state.rb"
timeout_seconds = 45

By default, the plugin execution interval is set at one minute. Control to prevent plugins from running at the same time is not included in the specifications of mackerel-agent. Therefore, we recommend that you set a time for timeout_seconds that does not exceed the plugin execution interval. (otherwise processing for the following plugin will start before the previous is completed)

Updates for Mackerel related OSS

Updates for various Mackerel related OSS were made this week and the details follow below. We received a ton of Pull Requests over the New Year’s holidays and we would like the thank everyone who contributed!

mackerel-agent v0.49.0

  • mackerel-agent now supports plugin timeout setting (as described above)
  • retry processing for check monitoring notifications was improved
  • The problem of obtaining too much information and metrics with the Docker host was addressed
    • network interfaces beginning with veth are now excluded
    • devices beginning with dm- (devicemapper) are now excluded
    • file systems beginning with dm- (devicemapper) are now excluded
  • A startup problem that occurred if /var/lib/mackerel-agent/id was empty has been corrected
  • [Windows version] check-uptime.exe is now bundled in the package

mackerel-agent-plugins v0.42.0

  • [mongodb] A problem with some metrics not being obtained correctly with mongodb where replica-set is built was fixed
  • [haproxy] Unix Domain Socket can now be specified with the -socket option
  • [postgres] Can now run with the older version (9.3.14) Postgresql

go-check-plugins v0.15.0

  • [mysql] readonly subcommand was added to check if the target mysql server is readonly


  • mackerel-plugin-aws-ecs was added
    • This can be used like mkr plugin install mackerel-plugin-aws-ecs by using the plugin installer feature.
    • Some users have already tried out and wrote about it. Check out the following article! (Japanese only)

A thorough explanation of mackerel-plugin-accesslog

Mackerel Product Owner (and sub-producer) id:Songmu here. This is an article that I published on the ninth day of our 2017 Mackerel Advent Calendar.

This article is an introduction for the official plugin mackerel-plugin-accesslog. We worked hard on this plugin this year and we think it’s really useful.

This plugin aggregates and visualizes web server access logs.

How to install

mackerel-plugin-accesslog is included in the official metric plugin package and can be used once the package has been installed. For details on how to install the package, see the following help page.

Using the official plugin pack to visualize middleware metrics - Mackerel Docs

go get can be used in Go environments as shown below.

% go get

How to configure

Specifying the access log file name as a parameter is extremely easy. Simply specify the following in mackerel-agent.conf.

command = "mackerel-plugin-accesslog /path/to/access.log"

Supported formats

The following log formats are supported.

  • Normal Apache format logs (Common/Combined) used in Apache and Nginx
  • LTSV format logs

In addition to major Apache logs, LTSV (Labeled Tab-separated Values) access log format, a format commonly used in the Japanese web industry and throughout Hatena1, is also supported.

The LTSV access log must be in a format that uses the Recommended Label defined in ltsv.org2. Regarding configuration method, refer to the description in .

I wrote the access log parser3, and I am proud to say that most logs can be parsed. However, I do strongly recommend using the LTSV log because response latency can only be aggregated with LTSV. Moreover, LTSV seems to have better parse efficiency, though I think the difference is slight.

Graph items

The following items are aggregated and graphed.

  • Access count of each status code
  • Percentage of each status code
  • Latency (only for LTSV log)
    • Average
    • 90/95/99 Percentile

As you can see in the above screenshot, visualization of these metrics is fairly straight forward.

Incidentally, mackerel-plugin-accesslog records the position of the read log file at the time of last execution and continues to read the log from that time. Therefore, values are collected every 1 minute when regularly executed from the agent.

Even if the logs are rotated, mackerel-plugin-accesslog searches for potential files before being rotated and, if located, reads it to the end and then reads the new log file from the beginning. As a result, aggregation is more accurate. This can be suppressed with a library called postailer4.

Combining expression graphs and aggregating in one role

As it’s typical for web servers to have multiple configurations, you may want to have graphs aggregated by role.

Mackerel has an experimental feature that lets you display customized graphs using expressions (commonly called: expression graphs5). With this feature, you can display access log graphs aggregated by role.

Here, let's try displaying the access count and ratio of each status code aggregated in one role in a stacked graph. Also, please be aware that expression graphs are still an experimental feature and must be enabled in the organization settings.6

Displaying the status code access count in stacked graphs

The expression for displaying the above graph follows below. Please rewrite the role name and metric name as appropriate.


Displaying the status code access percentage in stacked graphs

The expression for displaying this graph is a bit complex, but follows below. Please rewrite role name and metric name as appropriate here as well.


Monitoring for expression

It’s also possible to monitor the value obtained by expression. Please refer to the Help page for more information.

Expression graphs is a powerful feature, and it is extremely satisfying if you can write a useful expression correctly, but I’m aware that this feature may still be difficult to use so I am working on improvements. Requests are always welcome.

Comparing use of fluentd

The official help page "Posting service metrics with fluentd" introduces a method to aggregate access logs using fluentd.7

mackerel-plugin-accesslog is extremely useful because it makes access log aggregation possible without the use of fluentd.

However, that does not mean that the method using fluentd is no longer necessary. mackerel-plugin-accesslog is specifically for aggregation of a single server. It may be more convenient to use fluentd for situations with multiple servers or when performing arbitrary aggregation. As a matter of fact, in recent years it’s increasingly common that logs are not written in the server local file system and to rely on a log collector such as fluentd. In such a case, mackerel-plugin-access log can not be used.


Mackerel-plugin-accesslog can be a useful feature, by all means, give it a try. I’m also looking forward to your issues and pull requests for improvements and amendments.

Our support window will be closed for the holidays

Both methods of contacting our support team, either through or the “contact our support team” option displayed in the upper right header when logged into Mackerel, will be closed during the following dates for the New Year’s holidays.

New Year’s holiday period :Thursday, December 28th, 2017 - Wednesday, January 3rd, 2018 (JST)

All inquiries received during this period will be handled sequentially starting from January 4th, 2018 (Thursday).