Mackerel blog #mackerelio

The Official Blog of Mackerel

Taking a look at the top 10 most popular metric plugins on Mackerel

Mackerel team CRE Yoshida (id:syou6162) here. One of the great things about Mackerel is the abundance of plugins. So much so in fact, I’m sure there are plenty of people who find it rather difficult deciding which plugin they should use. On top of this, you can visualize middleware metrics after installing the plugin, but which metrics should you choose to monitor?

This post will hopefully help make these decisions easier. Today we’ll be taking a look at...

  • The top 10 most used metric plugins on Mackerel
  • Plugin metrics commonly used for monitoring

There are three types of Mackerel plugins (listed below), but in this entry we’ll be taking a look at the usage of metric plugins.

  • Metric plugins
    • Plugins that post statistics associated with the host as custom metrics
  • Check plugins
    • Plugins that determine OK / NG (CRITICAL or WARNING or UNKNOWN) within the host and post the results to Mackerel.
    • Refer here for more details
  • Metadata plugins
    • Plugins that register arbitrary JSON data for each host
    • Refer here for more details

Top 10 most used metric plugins on Mackerel!

Without further ado, here are the rankings! ...Ta-da!

Rank Plugin Usage Score*1
1 linux 7.3
2 mysql 4.4
3 nginx 3.4
4 apache2 3.3
5 docker 2.3
6 redis 1.2
7 accesslog 1.1
8 conntrack 1.1
9 jvm 1.1
10 inode 1

I’m sure a lot of you are already familiar with some of these plugins, but there might be a few you’ve never heard of. Let's take a look at each individual plugin, including what metrics are monitored.

#1: linux plugin

The top spot on our list goes to the linux plugin with a landslide victory over 2nd place by almost double the score. This plugin is essential if you use Linux. Metrics that are most commonly monitored include the number of logged-in users and network status.

Commonly used metrics
  • custom.linux.users.users
  • custom.linux.ss.ESTAB
  • custom.linux.ss.CLOSE-WAIT
  • custom.linux.swap.pswpin
  • custom.linux.ss.TIME-WAIT

References:

#2: mysql plugin

Coming in at #2, the mysql plugin is a must-have database for service operation. It's a popular RDBMS and understandably so. Metrics that are commonly monitored include the number of connections per status, deadlock detection, and slow query statistics.

Commonly used metrics
  • custom.mysql.seconds_behind_master.Seconds_Behind_Master
  • custom.mysql.threads.Threads_running
  • custom.mysql.capacity.PercentageOfConnections
  • custom.mysql.threads.Threads_connected
  • custom.mysql.table_locks.Slow_queries

References:

#3: nginx plugin

The nginx plugin takes bronze with a respectable usage score of 3.4. Some commonly used monitoring items include the number of connections and queue status.

Commonly used metrics
  • custom.nginx.connections.connections
  • custom.nginx.requests.requests
  • custom.nginx.queue.waiting
  • custom.nginx.requests.handled
  • custom.nginx.requests.accepts

#4: apache2 plugin

Following #3 with another web server related plugin, apache2 plugin ranks 4th. Commonly used metrics are similar to those of the nginx plugin, and it seems that the number of connections and the status of workers are monitored well.

Commonly used metrics
  • custom.apache2.workers.busy_workers
  • custom.apache2.workers.idle_workers
  • custom.apache2.req.requests
  • custom.apache2.cpu.cpu_load

#5: docker plugin

Of course users want to monitor the containerization process, so it’s not surprising that the docker plugin ranks 5th on our list. Most of the actual monitoring items are related to memory usage and state.

Commonly used metrics
  • custom.docker.running.XXX.state
  • custom.docker.memory_used.XXX.memory_percentage
The `XXX` segment differs depending on the environment.

#6: redis plugin

The redis plugin comes in at #6 and marks the turning point for our list. On a related note, the similarly popular key-value store memcached just missed our list coming in at #11. Some metrics that are commonly monitored include the number of connected clients, memory usage, and so on.

Commonly used metrics
  • custom.redis.clients.connected_clients
  • custom.redis.connections.rejected_connections
  • custom.redis.capacity.percentage_of_memory
  • custom.redis.uptime.uptime_in_seconds
  • custom.redis.capacity.percentage_of_clients

#7: accesslog plugin

Breaking in at #7, accesslog is another web server-related plugin. This plugin aggregates logs in Apache and LTSV format and is also super useful for SLI / SLO measurement. Try using it together with the nginx and apache2 plugins.

Commonly used metrics
  • custom.accesslog.access_num.5xx_count
  • custom.accesslog.access_rate.5xx_percentage
  • custom.accesslog.access_rate.4xx_percentage
  • custom.accesslog.latency.99_percentile
  • custom.accesslog.access_num.2xx_count

References:

#8: conntrack plugin

At #8, we have the conntrack plugin. This plugin monitors ip_conntrack, which is statistical information on iptables tracking. ip_conntrack has a tracking limit and if this limit is exceeded, it will count as a new session and network performance may deteriorate. So don't forget to add ip_conntrack tracking for servers that handle a lot of traffic (load balancers and cache servers).

Commonly used metrics
  • custom.conntrack.count.used
  • custom.conntrack.count.free

References:

#9: jvm plugin

The jvm plugin hits our list at #9. When it comes to operating applications run with jvm, certain monitoring items are unavoidable. Some typical examples would be time spent in GC / number of events, heap usage, and so on.

Commonly used metrics
  • custom.jvm.bootstrap.gc_events.FGC
  • custom.jvm.bootstrap.gc_time.FGCT
  • custom.jvm.heap.used
  • custom.jvm.prodserverstart.gc_events.FGC
  • custom.jvm.elasticsearch.memorySpace.oldSpaceRate

#10: inode plugin

And finally, the last spot on our list goes to the inode plugin coming in at #10. Exhausting inodes can lead to problems like not being able to create new directories and files, so it is safe to say that monitoring can offer a little peace of mind.

Commonly used metrics
  • custom.inode.percentage.*.used

References: - inodeの監視 ~ mackerel-plugin-inodeを読み解く - そーだいなるらくがき帳 (Japanese only)

Final thoughts

  • There is so much information out there on each metric plugin and the different kinds of monitoring items. I learned a lot making this list.
  • Although they didn’t make our list this time, there are still lots of other popular plugins like multicore / postgres / fluentd. Hopefully we’ll get a chance to look at some of these in future blogs.

*1:The usage score is a relative value that represents the number of users when 10th place equals 1.