Mackerel blog #mackerelio

The Official Blog of Mackerel

An error regarding obtaining metrics with mackerel-plugin-mysql has been fixed

Hello. Mackerel team CRE Inoue (id:a-know) here. This is an announcement regarding an error that was fixed for mackerel-plugin-mysql of the official Mackerel plugin pack.

What was fixed?

The metrics obtainable with mackerel-plugin-mysql are Bytes_sent (mysql.traffic.Bytes_sent) and Bytes_received (mysql.traffic.Bytes_received). These metrics represent the number of bytes sent and received by all clients.

The plugin (1) retrieves the corresponding metrics recorded as the cumulative total value in MySQL and (2) calculates the difference and sends it to Mackerel. When browsing these metrics in Mackerel, the expected unit is bytes/sec. However, a value 60 times larger (the value of bytes/min) than the actual value was being sent. This has been corrected according to the Pull Request linked below. Now, a value that is 1/60 of the previous value will be sent.

What needs to be done?

Today we are updating Mackerel’s official plugin package mackerel-agent-plugins. Be sure to update the mackerel-agent-plugins package with your package management system (yum or apt).

With this update, the values of Bytes_sent (mysql.traffic.Bytes_sent) and Bytes_received (mysql.traffic.Bytes_received) will be sent as 1/60 of the previous values. If you’ve created a monitor for these metrics, please be sure to review the thresholds as well.

We apologize for any inconvenience this may have caused and thank you for your understanding.

Azure Database for MySQL / PostgreSQL now supported with Azure Integration!

Hello. Mackerel team CRE Inoue ( id:a-know) here with another update announcement.

Azure Database for MySQL / PostgreSQL now supported with Azure Integration


It is now possible to monitor Azure Database with Azure integration! Azure Database is Azure’s fully managed database service! Both MySQL and PostgreSQL are now supported.

Check out the help page linked below for configuration methods.

And be sure to check out each of the following help pages regarding obtaining metrics with this newly added integration.

The selected metric is now reflected when posting role metric graphs to notification channels

Now, when using the camera icon to post graphs to notification channels, the metric selection is reflected in the shared content. The same goes for host metric graphs and service metric graphs as well. We hope this is helpful!

Tips to help your inquiries to the support team get resolved quicker

Hello, Mackerel team CRE Nishiyama (id:tukaelu) here.

Have you ever searched through our FAQ (Frequently Asked Questions) or contacted our support team with questions regarding Mackerel? Well, the FAQ site and the Feedback form have both been recently updated!

And although the new feedback form is now more detailed than before, entering the correct information into the required fields is important if you want a speedy resolution.

Today we'll be taking a look at a few pointers that will help prompt a quick resolution when asking a question via the feedback form!

Make use of the ‘Subject’ field

The Subject field doesn’t need any new explanation per se, but one of the handy new features of the feedback form is that it now displays a list of suggested FAQ pages that seem to be relevant to the content entered in the subject line!

The suggested FAQ articles are listed like shown in the image above. Be sure to check for any articles that might be related!

Go into detail when filling in the Description

As for the Description of your inquiry, including the following points may lead to a quicker resolution.

Specify environment information

Including the following points detailing the environment of the host to which you are inquiring about will benefit the resolution turnaround.

  • OS type, version
  • Version of mackerel-agent
  • Plugin name, version (※If using an official plugin)
  • Service name for inquiries regarding various integration

Also, if you’re monitoring middleware with a plugin, inform us of the software name and version.

Specify the situation, status, etc.

Describe in detail the event or problem you are experiencing. Try to be as clear as possible about what operations were performed and what events occurred.

  • NG
    • mackerel-agent doesn’t start up. What are the possible causes?
  • OK
    • When mackerel-agent is restarted with the attached settings, the following error occurs and does not start up.

You can also attach files to the new feedback form. Attaching information like the following to your inquiry can help lead to a quicker resolution.

  • mackerel-agent logs
  • mackerel-agent.conf
    • Masking things like API keys is not a problem
  • User investigated content, executed commands, results, etc.
  • Screenshots that can be verified from the web console etc.

Include information that can be verified from the Mackerel Web Console

If what you are inquiring about can be verified from the Mackerel web console, please include the URL.

  • Host details URL
  • Alert details URL
  • Monitor details URL etc.

The review of your inquiry on our end will go a lot smoother if you can provide us with a URL that can be verified from the web console, such as a limited time graph URL. If this information is available, please make sure to attach it when contacting us.

Other Tips

In addition to the everything mentioned above, please consider the following points.

1 question per case

It may be quicker to have one question answered at a time.

If you have multiple questions that are all related, of course it’s better to consolidate them into one inquiry. But if this is not the case, you might try splitting up your questions into multiple inquiries.

An inquiry may be difficult to investigate if related to a retired host or deleted monitor

We recommend trying to reserve the resources (hosts, monitors, etc.) related to your inquiry as much as possible.

And please inform us if deleted or any changes were made.

You can add recipients (Cc) to receive a reply

You can ‘Cc’ additional recipients of the response by specifying email addresses separated by commas(,) in the "Additional email addresses to receive replies" field of the feedback form.

This can be useful for sharing the support team’s response among your team or people of concern.

Please inform us when the issue is resolved

If our support team’s response helps resolve your issue, please let us know and we can close the support case.

To close

This has been a look at a few useful pointers that can help get your inquiries to the support team resolved quicker!

If you have any questions or requests regarding Mackerel, please feel free to contact us!

An API for AWS Integration settings and more

Hello. Mackerel team CRE Inoue (id:a-know) here with another update announcement.

An API for AWS Integration settings!

Some users have been waiting expectantly for this update. The API documentation is linked below.

It is now possible to configure AWS Integration settings from the API. This includes specifying integration excludable metrics. Be sure to give this a try!

The Organizations page can now be sorted by number of alerts


The Organizations page shows a list of the organizations to which you belong. With this update, you can now rearrange this list by the number of occurred / continuing alerts. This is a nice update for those who manage a lot of organizations. We hope that you find it useful.

The Web Console’s Feedback form gets a new look!

Have you ever tried contacting our support team using the ‘Feedback’ button that’s located in the upper right corner of the web console? Well, we’ve updated the request form.

Now you can attach files to the new request form. Feel free to contact us using screenshots or configuration files that relate to your inquiry!

Taking a look at the top 10 most popular metric plugins on Mackerel

Mackerel team CRE Yoshida (id:syou6162) here. One of the great things about Mackerel is the abundance of plugins. So much so in fact, I’m sure there are plenty of people who find it rather difficult deciding which plugin they should use. On top of this, you can visualize middleware metrics after installing the plugin, but which metrics should you choose to monitor?

This post will hopefully help make these decisions easier. Today we’ll be taking a look at...

  • The top 10 most used metric plugins on Mackerel
  • Plugin metrics commonly used for monitoring

There are three types of Mackerel plugins (listed below), but in this entry we’ll be taking a look at the usage of metric plugins.

  • Metric plugins
    • Plugins that post statistics associated with the host as custom metrics
  • Check plugins
    • Plugins that determine OK / NG (CRITICAL or WARNING or UNKNOWN) within the host and post the results to Mackerel.
    • Refer here for more details
  • Metadata plugins
    • Plugins that register arbitrary JSON data for each host
    • Refer here for more details

Top 10 most used metric plugins on Mackerel!

Without further ado, here are the rankings! ...Ta-da!

Rank Plugin Usage Score*1
1 linux 7.3
2 mysql 4.4
3 nginx 3.4
4 apache2 3.3
5 docker 2.3
6 redis 1.2
7 accesslog 1.1
8 conntrack 1.1
9 jvm 1.1
10 inode 1

I’m sure a lot of you are already familiar with some of these plugins, but there might be a few you’ve never heard of. Let's take a look at each individual plugin, including what metrics are monitored.

#1: linux plugin

The top spot on our list goes to the linux plugin with a landslide victory over 2nd place by almost double the score. This plugin is essential if you use Linux. Metrics that are most commonly monitored include the number of logged-in users and network status.

Commonly used metrics
  • custom.linux.users.users
  • custom.linux.swap.pswpin


#2: mysql plugin

Coming in at #2, the mysql plugin is a must-have database for service operation. It's a popular RDBMS and understandably so. Metrics that are commonly monitored include the number of connections per status, deadlock detection, and slow query statistics.

Commonly used metrics
  • custom.mysql.seconds_behind_master.Seconds_Behind_Master
  • custom.mysql.threads.Threads_running
  • custom.mysql.capacity.PercentageOfConnections
  • custom.mysql.threads.Threads_connected
  • custom.mysql.table_locks.Slow_queries


#3: nginx plugin

The nginx plugin takes bronze with a respectable usage score of 3.4. Some commonly used monitoring items include the number of connections and queue status.

Commonly used metrics
  • custom.nginx.connections.connections
  • custom.nginx.requests.requests
  • custom.nginx.queue.waiting
  • custom.nginx.requests.handled
  • custom.nginx.requests.accepts

#4: apache2 plugin

Following #3 with another web server related plugin, apache2 plugin ranks 4th. Commonly used metrics are similar to those of the nginx plugin, and it seems that the number of connections and the status of workers are monitored well.

Commonly used metrics
  • custom.apache2.workers.busy_workers
  • custom.apache2.workers.idle_workers
  • custom.apache2.req.requests
  • custom.apache2.cpu.cpu_load

#5: docker plugin

Of course users want to monitor the containerization process, so it’s not surprising that the docker plugin ranks 5th on our list. Most of the actual monitoring items are related to memory usage and state.

Commonly used metrics
  • custom.docker.running.XXX.state
  • custom.docker.memory_used.XXX.memory_percentage
The `XXX` segment differs depending on the environment.

#6: redis plugin

The redis plugin comes in at #6 and marks the turning point for our list. On a related note, the similarly popular key-value store memcached just missed our list coming in at #11. Some metrics that are commonly monitored include the number of connected clients, memory usage, and so on.

Commonly used metrics
  • custom.redis.clients.connected_clients
  • custom.redis.connections.rejected_connections
  • custom.redis.capacity.percentage_of_memory
  • custom.redis.uptime.uptime_in_seconds
  • custom.redis.capacity.percentage_of_clients

#7: accesslog plugin

Breaking in at #7, accesslog is another web server-related plugin. This plugin aggregates logs in Apache and LTSV format and is also super useful for SLI / SLO measurement. Try using it together with the nginx and apache2 plugins.

Commonly used metrics
  • custom.accesslog.access_num.5xx_count
  • custom.accesslog.access_rate.5xx_percentage
  • custom.accesslog.access_rate.4xx_percentage
  • custom.accesslog.latency.99_percentile
  • custom.accesslog.access_num.2xx_count


#8: conntrack plugin

At #8, we have the conntrack plugin. This plugin monitors ip_conntrack, which is statistical information on iptables tracking. ip_conntrack has a tracking limit and if this limit is exceeded, it will count as a new session and network performance may deteriorate. So don't forget to add ip_conntrack tracking for servers that handle a lot of traffic (load balancers and cache servers).

Commonly used metrics
  • custom.conntrack.count.used


#9: jvm plugin

The jvm plugin hits our list at #9. When it comes to operating applications run with jvm, certain monitoring items are unavoidable. Some typical examples would be time spent in GC / number of events, heap usage, and so on.

Commonly used metrics
  • custom.jvm.bootstrap.gc_events.FGC
  • custom.jvm.bootstrap.gc_time.FGCT
  • custom.jvm.heap.used
  • custom.jvm.prodserverstart.gc_events.FGC
  • custom.jvm.elasticsearch.memorySpace.oldSpaceRate

#10: inode plugin

And finally, the last spot on our list goes to the inode plugin coming in at #10. Exhausting inodes can lead to problems like not being able to create new directories and files, so it is safe to say that monitoring can offer a little peace of mind.

Commonly used metrics
  • custom.inode.percentage.*.used

References: - inodeの監視 ~ mackerel-plugin-inodeを読み解く - そーだいなるらくがき帳 (Japanese only)

Final thoughts

  • There is so much information out there on each metric plugin and the different kinds of monitoring items. I learned a lot making this list.
  • Although they didn’t make our list this time, there are still lots of other popular plugins like multicore / postgres / fluentd. Hopefully we’ll get a chance to look at some of these in future blogs.

*1:The usage score is a relative value that represents the number of users when 10th place equals 1.

Monitoring the intermediate CA certificate expiration date now supported

Hello, Mackerel team CRE Nishiyama (id:tukaelu) here with another update announcement.

As you may know, Mackerel’s external URL monitoring enables you to monitor the SSL certificate expiration date. Incidentally, we’ve received numerous requests to include the same support for intermediate CA certificates as well. With this update, this is now possible.

Expiration dates of chained intermediate CA certificates can now be detected in advance, so be sure to check for this when external monitoring alerts occur.

You can check a certificate by executing a command like the one shown below. This command behaves in a similar fashion to Mackerel's external monitoring system. Make sure to replace the ‘’ sections in the following example with your target before running the command.

openssl s_client -servername -connect -showcerts </dev/null >c.pem
openssl crl2pkcs7 -nocrl -certfile c.pem | openssl pkcs7 -print_certs -text -noout

This feature was made possible thanks to the requests of Mackerel users.

Please feel free to contact our support team with any comments or questions regarding this update via the Mackerel Web Console.

We look forward to your feedback.

New metric targets added for AWS / NLB Integration and more

Hello. Mackerel team CRE Inoue (id:a-know) here with another update announcement.

New metric targets added for AWS / NLB Integration

With this update, new metric targets are now obtainable through AWS and NLB Integration. The following metrics relate to TLS termination in NLB.

  • Processed Bytes
    • nlb.bytes.tcp
    • nlb.bytes.tls
    • nlb.bytes.udp
  • Established Active Flow
    • nlb.established_active_flow.tcp
    • nlb.established_active_flow.tls
    • nlb.established_active_flow.udp
  • Consumed LCUs
    • nlb.consumed_lcus.all
    • nlb.consumed_lcus.tcp
    • nlb.consumed_lcus.tls
    • nlb.consumed_lcus.udp
  • New Flow
    • nlb.new_flow.tcp
    • nlb.new_flow.tls
    • nlb.new_flow.udp
  • TLS Negotiation Error
    • nlb.tls_negotiation_error.client

In addition to this, the maximum number of retrievable metrics has been increased from 6 + 2 × (number of target groups) to 21 + 2 × (number of target groups). So depending on the number of target groups, you may exceed the 30 metric limit in which case the excess will be converted into an additional micro host (FAQ・Handling of host conversion when plan limits are exceeded - Mackerel Docs).

Obtaining metrics with AWS Integration can be freely configured at your own discretion in settings. Although, metrics are retrieved by default, so please manage this as necessary.

HTTP API support for AWS / API Gateway Integration

I’m sure that the current development of an HTTP API available (GA) for Amazon API Gateway is news to everyone. (Japanese only)

With this update, it is now possible to monitor HTTP Protocol-Type API Gateway resources. Prior to this, only REST and WebSocket were supported, but now all currently existing Protocol-Type are available.

For more on integratable metrics, check out the documentation linked below.

Reflect a selected metric when sharing service metric graphs to notification channels

Have you ever used the graph sharing function that lets you post graphs to notification channels by clicking the camera icon located in the upper right corner of each graph?


Now, when sharing service metric graphs, you can select a particular metric to be reflected in the shared content.

By selecting a particular metric to share...

Only that selected metric will be shared to the notification channel!

This feature has been supported for system metrics and custom metrics, but is now available for service metric graphs as well. We hope this will be helpful for communicating and addressing specific areas.