Mackerel blog #mackerelio

The Official Blog of Mackerel

You can now have notifications sent to channels for monitor rule operations etc.

Hello! Mackerel team CRE Inoue (id:a-know) here.

Already the 4th week in December, I’ve attended a end-of-the-year party everyday this week. I hope everyone is enjoying the holiday season as much as me.

This will be the last of this year’s weekly updates for Mackerel.

Since September of 2014, we’ve continued our tradition of “Consecutive weekly releases” and we’ll be aiming for 200 consecutive weeks in the coming year. We’re also excited to deliver various new functions/updates to everyone in 2018!

Now on to this week’s update information.

You can now have notifications sent to channels for monitor rule operations

It is now possible to have event notifications sent to notification channels when monitor rule related operations (creating / updating / deleting) are made.

If you select Webhook as the notification channel, changes made to the monitor rule will be included in the JSON format notification content. The specifics follow below.

  • Creating or updating a monitor rule
    • Monitor rule content will be notified after the operation
  • Deleting a monitor rule
    • Monitor rule content will be notified immediately before deletion

Posting graphs in notification channels now supported with LINE and Yammer

The graph posting feature allows you to post graphs to notification channels by clicking the camera icon in the graph menu. This is a useful feature because it lets you share metrics at any time, even those not configured for alerts. Now, LINE and Yammer are also supported with this feature.

Check out the following help page for more on this feature.

Updates for Mackerel related OSS

Updates for various Mackerel related OSS were made this week and the details follow below. As always, thank you to everyone who contributed!

mackerel-agent v0.48.2

  • [For Windows] The problem regarding address information of the network interface not being available was fixed.

mackerel-agent-plugins v0.41.1

  • [mysql] Sections of stats related to InnoDB can now be obtained, even the reader node of AWS Aurora for MySQL
  • [mysql] Other minor adjustments

Thanks for another great year!

Although I mentioned this at the beginning, this will be our last update announcement for this year. Thank you so much for supporting Mackerel through 2017.

We are excited for 2018 and continuing our weekly updates. Next year, we hope to deliver an even more advanced and efficient server monitoring/management experience to our users through Mackerel.

Thank you for choosing Mackerel. Until next year!

Environment variables can now be specified for command execution via plugin etc.

Hello! Mackerel team CRE Inoue (id:a-know) here.

Only half a month left in 2017 and I’m sure everyone’s busy with work and year-end parties. The Mackerel team is keeping busy as we continue with our consecutive weekly releases. Next week will be our last update of this year.

Let's survive this home stretch of 2017 so that we can relax and enjoy the end of the year.

Now on to this week’s update information.

Environment variables can now be specified for command execution via plugin

Mackerel plugins can be used by adding the following configuration to the conf file.

command = "ruby /path/to/mysql.rb --username user --password password"

The agent executes the character array specified in command, and a post is made to Mackerel based on the resulting stdout (or exit code). This is a very useful mechanism.

With this week's update, you can now specify environment variables for command execution with this plugin mechanism. Below is the specification method.

command = "ruby /path/to/mysql.rb"
env = { "MYSQL_USERNAME" = "user", "MYSQL_PASSWORD" = "password" }

We will be making updates, even for the official plugins that are currently provided, so that password configuration and such for middleware can be performed via environment variables.

This feature can be used with check plugins and metadata plugins as well!

The password strength policy was reviewed and now passwords with weak security strength can be used

With this week’s release, updates were made for the following two points.

  • The strength policy for password configuration was reviewed
  • Although it was previously impossible to set a password with weak security strength, this can now be ignored

Additionally, it’s currently possible to remove password authentication in Mackerel if configured with GitHub or Google. Hopefully you can utilize this in combination with the option to ignore the password strength check.

Updates for Mackerel related OSS

Updates for various Mackerel related OSS were made this week and the details follow below. As always, thank you to everyone who contributed!

mackerel-agent v0.48.1

  • Environment variables can now be specified for command execution via plugin (described above)
  • The cloud_platform option was added to select targets for gathering laaS meta information and obtaining custom_identifier

mackerel-agent-plugins v0.40.0

  • mackerel-plugin-h2o was added
  • [redis] The CONFIG command that is internally used by the plugin is now available even if it can not be used as is.
    • Configured with the -config-command option
  • [redis] Replication delay metrics are now obtainable

mkr v0.24.1

You can now configure the maximum number of attempts for host/service metric monitors etc.

Thanks to all of you, the Mackerel Advent Calendar that we talked about last week has been completely filled! (Japanese only)

Thank you!

Also, be sure to check out the advent calendar that Mackerel team CRE id:Soudai is working so hard on, not missing a single day and continuing to release blogs full of information and useful knowledge. (Japanese only)

I hope you enjoy these advent calendars leading up to Christmas.

Now onto this week’s update information.

Customized (expression) graphs can now be added to GraphBoard

"GraphBoard" is an interactive feature that lets the user freely arrange arbitrary role and service metrics graphs.

With this week's update, we’ve increased the number of types of graphs that can be added to this GraphBoard. Customized graphs (expression graphs) can now be added!

  • Overlay server load statuses for the current week and the previous week and display as one graph
  • Display the calculated maximum value/average value/minimum value of the server load belonging to a particular role

Customized graphs are extremely useful and allow you to implement the above procedures by describing the function expression. We believe that putting this to use will be even easier than before.

Displaying customized graphs is an experimental feature. For more details, check out the following help pages.

You can now configure the max number of attempts for host/service metric monitors

As you can see in the image above, the "Max number of attempts before an alert is triggered" can be configured with host metric and service metric monitors.

Up until now, only the monitoring of “Duration” (average of the past n minutes) was supported. For example, if you wanted to have an alert occur when a threshold of 5 is consecutively exceeded, you would configure the following.

  • Threshold of 5
  • Duration of 3 minutes

However, with this configuration, not only would a progression of 756 (a 3min average of 6) lead to an alert occurrence, but also for a progression such as 1161 (in this case, also a 3min average of 6). For this reason, sometimes unintentional alerts would occur.

With this update, you can eliminate cases such as the one above by utilizing the newly added "max_check_attempts". We hope that you can use this feature together with "Duration” monitoring to eliminate unintentional alerts.

Our support window will be closed for the holidays

Both methods of contacting our support team, either through or the “contact our support team” option displayed in the upper right header when logged into Mackerel, will be closed during the following dates for the New Year’s holidays.

New Year’s holiday period :Thursday, December 28th, 2017 - Wednesday, January 3rd, 2018 (JST)

All inquiries received during this period will be handled sequentially starting from January 4th, 2018 (Thursday).

Please review your server monitoring settings and contact us if you have any concerns before the end of the year holidays begin!

Target URLs can now be specified directly with the mkr plugin installer etc.

Hello! Mackerel team CRE Inoue (id:a-know) here.

It’s December! And around this time every year, what the IT/Web industry refers to as "Advent Calendars" bustles with various themed blog posts, and Mackerel is no exception. Here is Mackerel’s Advent Calendar! (Japanese only)

But wait, here’s another calendar! (Japanese only)

Be sure to check out this calendar as well. Fellow team CRE Sone (id:Soudai) is going all out!

Now for this week’s update information.

Target URLs can now be specified directly with the mkr plugin installer

We recently made an announcement regarding the recently released command line tool mkr plugin installer feature, including usage details/application methods.

With this week’s feature update, you can now directly specify a URL as shown below.

mkr plugin install

Be sure to try out this plugin installer feature which allows you to easily install useful third-party plugins.

You can now select the API V2 in the PagerDuty notification channel settings

In the integration settings for PagerDuty, one of the notification channels available with Mackerel, you can now select the V2 API.

Although there is no functional difference at this time, it is possible that options only available with V2 will be added in future feature expansion. If this applies to you, please consider making the transition.

Updates for Mackerel related OSS

Updates for various OSS have been released and are listed below. To all of you out there who contributed or reported a problem, thank you very much!

mackerel-agent v0.47.3

  • The output message that occurs when there is a defect in the mackerel-agent.conf plugin command configuration has been improved.
  • The issue of the interface value not being obtained when the network counter is 8 digits or more has been fixed.

mackerel-agent-plugins v0.39.0

  • [docker] The valued obtained by Docker Blkio IOPS has been normalized from per Minute to per Second
  • [jvm] The issue of the option to monitor the remote jvm not functioning properly was fixed.
  • [kinesis-streams] Regarding metrics for the number of records and processing bytes, outputting the "statistics of one record" was changed to "total amount per minute".

Mackerel × Twilio Hands-On Seminar #2

Earlier this year in April, the first Mackerel × Twilio Hands-On Seminar was held and went down as a huge success. Now, the second edition of the seminar is in motion and scheduled to be held on Wednesday, December 13th. (Japanese only)

For the venue of the event, we’ll be renting out the space of RecoChoku, a company which extensively uses the Mackerel - Twilio Integration. This will be a valuable opportunity to learn more about a variety of things ranging from the basics of why server monitoring is important to installing the agent and receiving phone notifications through Twilio.

The event is almost at capacity, but because applications as chosen through a lottery system, there’s still a good chance that you’ll be accepted even if you apply now. You definitely don’t want to miss out on this opportunity!

Azure Integration now supports Virtual Machines

I can’t believe that it’ll already be December next week. It feels like we just celebrated Halloween...

But with December comes the end of year Advent Calendar! Thanks to all of you, Mackerel’s Advent Calendar is filling up little by little. (Japanese only)

There are still spots available, so please join us!

Now onto this weeks update information.

Azure Integration now supports Virtual Machines

Mackerel's integration feature lets you monitor cloud services without having to introduce an agent, and Azure integration now supports Virtual Machines!

For information on available metrics, check out the the following help page.

This feature is also available if currently using virtual machines with the agent installed.

Even if you use both, they can be managed as one host (the metrics collected with the integration feature will be integrated as "custom metrics" of that host), so it’s possible to aggregate metrics in Mackerel that can’t be collected by the agent alone. And of course, a usage fee for only one host will be charged. Be sure to give it a try!

mkr third party plugin installer feature

In last week's release, we announced the addition of our plugin installer feature to the mkr command. This feature makes it easier for users to install their own third party plugins in the system.

Up until now, in order to install your own plugin on the system, you either had to arrange a method of installation by yourself or submit source code to the following central repository.

Using this new feature, it is now possible to install plugins created in your own GitHub repository using the mkr command. This allows the creator of the plugin the ability to continually perform maintenance.

Additionally, when installing a plugin by submitting code to the central repository, developing with Golang was required. However, with this feature, it’s now possible to develop using the language of your choice.

How to use

Using the mkr plugin install command, you can install the contents of the specified Mackerel plugin package on your system.

% sudo mkr plugin install <owner>/<repo>[@<release_tag>]

By specifying the Github owner name, repository name, as well as the tag name for Github Releases, the Mackerel plugin package is obtained from GitHub Releases and installed.

Mackerel plugin package format

The Mackerel plugin package is a simple zip file. There should be an executable file starting with mackerel-plugin- or check- stored in the archive. As long as it’s an executable file, any language can be used.

The mkr plugin install command regards these executable files as Mackerel plugins and places them in the specific directory. By default, they will be located under /opt/mackerel-agent/plugins/bin.

Uploading the Mackerel plugin package in GitHub Releases

% sudo mkr plugin install <owner>/<repo>[@<release_tag>]

When the above command is executed, mkr attempts to acquire the Mackerel plugin package of the following URL.<owner>/<repo>/releases/download/<release_tag>/<repo>_<GOOS>_<GOARCH>.zip

If the Mackerel plugin package is placed in GitHub Releases’ <release_tag> according to the naming convention <repo>_<GOOS>_<GOARCH>.zip, you’ll be able to download the Mackerel plugin package from the above URL. Take a look at the following with mackerel-plugin-sample used as an example.

The official plugin registry

Registering useful plugins in the official plugin registry makes it easier for other users to find plugins. Plugins registered in the plugin registry can be installed using the format mkr plugin install <plugin_name>.

Registration is done by sending a Pull Request to the plugin registry’s Github repository. For the contents of the Pull Request, simply create a file called <plugin_name>.json under the plugins/ directory. The JSON file format is as follows.

You can see a list of the plugins that are currently registered at

Understand that contribution authority and licensing for the plugins registered in the official plugin registry correspond with the referent repository.

Please do not hesitate to send us a Pull Request.

On a separate note, since package management registries like this often have cool names like CPAN, RubyGems, homebrew, we considered doing the same for our official plugin registry. Hatena CTO id:motemen even came up with the catchy name tsukiji, but it was ultimately dismissed for various reasons, and we settled with the safe name of plugin-registry.

Pull Requests to the existing central repository

Up until now, it was acceptable to register a new plugin with the above-mentioned central repository, but as a general rule from now on, adding new plugins this way will no longer be accepted. Adding a feature to an existing plugin or fixing a bug is of course welcome.

If you create a new plugin, you should create your own plugin repository on GitHub and register it in the official plugin registry.

If you would like to incorporate a registered plugin into Mackerel’s official plugin package distributed by rpm or deb, please contact us through a separate issue etc. Conversely, the Mackerel development team might reach out to plugin creators to ask for permission to incorporate registered packages into the official plugin package. In such a case, we would appreciate your cooperation.

More detailed documentation

For more details on how to install the third party plugin installer described in this entry or how to create a Mackerel plugin package, please refer to the following help pages.

Future expansion

We are considering the following expansions in the future.

  • A flexible template-like naming configuration, since naming for the Mackerel plugin package is currently defined strictly as <repo>_<GOOS>_<GOARCH>.zip
  • An update feature for installed plugins
  • Metadata plugin support

Of course, we welcome additional proposals and Pull Requests for this feature.

First, create your own plugin, then send a Pull Request to the official plugin registry.

You can now have notifications sent to notification channels for events such as host status changes etc. 

Hello! Mackerel team CRE Inoue (id:a-know) here.

It certainly seems as though winter has arrived as temperatures drop all around the country today. This morning I pulled out my winter gloves without evening thinking about it. The air seems really dry as well. I hope everyone out there is staying healthy.

Anyways, now for this week’s updates.

You can now have notifications sent to notification channels for events such as host status changes and more

A new field labeled "Notification events" is now provided as a configuration item for certain notification channels. With this, in addition to the alert notifications available up to now, it is now possible to have notifications sent the channel for the following events.

  • Host status changes
  • Hosts registered to the organization
  • Hosts retired

Not only is this feature useful for letting you know when events like those mentioned above occur, but combined with Webhook notifications and Amazon API Gateway + Lambda etc., it also gives the user the ability to build a mechanism that will execute arbitrary actions in accordance with the occurrence of these events.

We plan to expand the types of events that can be selected for notification in the future. We are also open to requests / opinions!

OpsGenie V2 REST API now supported

We previously made an announcement regarding the OpsGenie V2 REST API in the entry below, but with this week’s release, support has been completed.

We apologize for the inconvenience.

Plugins are being added one after another to Mackerel’s official plugin registry

As mentioned in last week’s release and yesterday’s explanation article, a third party plugin installer feature has been added to the CLI tool · mkr.

In order to make it easier for users to find third party plugins, we’ve made an official plugin registry.

Plugins are being registered one after another. You can register by sending a Pull Request to the plugin registry’s Github repository.

Even plugins such as mackerel-plugin-aws-batch are being added, so be sure to check it out!

【Only a few spots left】Monitoring Seminar in mercari

On Wednesday, November 29th, we are renting out the venue of Mercari (a heavy user of Mackerel) to hold a seminar titled "Monitoring Seminar in mercari"! (Japanese only)

This will be a great chance to exchange knowledge on not only Mackerel but also server monitoring in general. This will also be an opportunity to think about and discuss "server monitoring" in a more familiar way. By all means, join us (Only a few remaining seats)!