Mackerel team CRE Yoshida (
id:syou6162) here. One of the great things about Mackerel is the abundance of plugins. So much so in fact, I’m sure there are plenty of people who find it rather difficult deciding which plugin they should use. On top of this, you can visualize middleware metrics after installing the plugin, but which metrics should you choose to monitor?
This post will hopefully help make these decisions easier. Today we’ll be taking a look at...
- The top 10 most used metric plugins on Mackerel
- Plugin metrics commonly used for monitoring
There are three types of Mackerel plugins (listed below), but in this entry we’ll be taking a look at the usage of metric plugins.
- Metric plugins
- Plugins that post statistics associated with the host as custom metrics
- Check plugins
- Plugins that determine OK / NG (CRITICAL or WARNING or UNKNOWN) within the host and post the results to Mackerel.
- Refer here for more details
- Metadata plugins
Top 10 most used metric plugins on Mackerel!
Without further ado, here are the rankings! ...Ta-da!
| Rank | Plugin | Usage Score*1 |
|---|---|---|
| 1 | linux | 7.3 |
| 2 | mysql | 4.4 |
| 3 | nginx | 3.4 |
| 4 | apache2 | 3.3 |
| 5 | docker | 2.3 |
| 6 | redis | 1.2 |
| 7 | accesslog | 1.1 |
| 8 | conntrack | 1.1 |
| 9 | jvm | 1.1 |
| 10 | inode | 1 |
I’m sure a lot of you are already familiar with some of these plugins, but there might be a few you’ve never heard of. Let's take a look at each individual plugin, including what metrics are monitored.
#1: linux plugin
The top spot on our list goes to the linux plugin with a landslide victory over 2nd place by almost double the score. This plugin is essential if you use Linux. Metrics that are most commonly monitored include the number of logged-in users and network status.
Commonly used metrics
custom.linux.users.userscustom.linux.ss.ESTABcustom.linux.ss.CLOSE-WAITcustom.linux.swap.pswpincustom.linux.ss.TIME-WAIT
References:
- Linuxの監視 ~ mackerel-plugin-linuxを読み解く - そーだいなるらくがき帳 (Japanese only)
- Former CRE
id:Soudai goes over installation and other metrics in detail.
- Former CRE
#2: mysql plugin
Coming in at #2, the mysql plugin is a must-have database for service operation. It's a popular RDBMS and understandably so. Metrics that are commonly monitored include the number of connections per status, deadlock detection, and slow query statistics.
Commonly used metrics
custom.mysql.seconds_behind_master.Seconds_Behind_Mastercustom.mysql.threads.Threads_runningcustom.mysql.capacity.PercentageOfConnectionscustom.mysql.threads.Threads_connectedcustom.mysql.table_locks.Slow_queries
References:
- MySQLの監視 ~ mackerel-plugin-mysqlを読み解く - そーだいなるらくがき帳 (Japanese only)
- InnoDBの監視 ~ mackerel-plugin-mysqlを読み解く その2 - そーだいなるらくがき帳 (Japanese only)
#3: nginx plugin
The nginx plugin takes bronze with a respectable usage score of 3.4. Some commonly used monitoring items include the number of connections and queue status.
Commonly used metrics
custom.nginx.connections.connectionscustom.nginx.requests.requestscustom.nginx.queue.waitingcustom.nginx.requests.handledcustom.nginx.requests.accepts
#4: apache2 plugin
Following #3 with another web server related plugin, apache2 plugin ranks 4th. Commonly used metrics are similar to those of the nginx plugin, and it seems that the number of connections and the status of workers are monitored well.
Commonly used metrics
custom.apache2.workers.busy_workerscustom.apache2.workers.idle_workerscustom.apache2.req.requestscustom.apache2.cpu.cpu_load
#5: docker plugin
Of course users want to monitor the containerization process, so it’s not surprising that the docker plugin ranks 5th on our list. Most of the actual monitoring items are related to memory usage and state.
Commonly used metrics
custom.docker.running.XXX.statecustom.docker.memory_used.XXX.memory_percentage
#6: redis plugin
The redis plugin comes in at #6 and marks the turning point for our list. On a related note, the similarly popular key-value store memcached just missed our list coming in at #11. Some metrics that are commonly monitored include the number of connected clients, memory usage, and so on.
Commonly used metrics
custom.redis.clients.connected_clientscustom.redis.connections.rejected_connectionscustom.redis.capacity.percentage_of_memorycustom.redis.uptime.uptime_in_secondscustom.redis.capacity.percentage_of_clients
#7: accesslog plugin
Breaking in at #7, accesslog is another web server-related plugin. This plugin aggregates logs in Apache and LTSV format and is also super useful for SLI / SLO measurement. Try using it together with the nginx and apache2 plugins.
Commonly used metrics
custom.accesslog.access_num.5xx_countcustom.accesslog.access_rate.5xx_percentagecustom.accesslog.access_rate.4xx_percentagecustom.accesslog.latency.99_percentilecustom.accesslog.access_num.2xx_count
References:
- A thorough explanation of mackerel-plugin-accesslog - Mackerel blog #mackerelio
- Webサーバーアクセスログの分析と監視 〜 mackerel-plugin-accesslog を読み解く - えいのうにっき (Japanese only)
#8: conntrack plugin
At #8, we have the conntrack plugin. This plugin monitors ip_conntrack, which is statistical information on iptables tracking. ip_conntrack has a tracking limit and if this limit is exceeded, it will count as a new session and network performance may deteriorate. So don't forget to add ip_conntrack tracking for servers that handle a lot of traffic (load balancers and cache servers).
Commonly used metrics
custom.conntrack.count.usedcustom.conntrack.count.free
References:
- iptablesの監視 ~ mackerel-plugin-conntrackを読み解く - そーだいなるらくがき帳 (Japanese only)
#9: jvm plugin
The jvm plugin hits our list at #9. When it comes to operating applications run with jvm, certain monitoring items are unavoidable. Some typical examples would be time spent in GC / number of events, heap usage, and so on.
Commonly used metrics
custom.jvm.bootstrap.gc_events.FGCcustom.jvm.bootstrap.gc_time.FGCTcustom.jvm.heap.usedcustom.jvm.prodserverstart.gc_events.FGCcustom.jvm.elasticsearch.memorySpace.oldSpaceRate
#10: inode plugin
And finally, the last spot on our list goes to the inode plugin coming in at #10. Exhausting inodes can lead to problems like not being able to create new directories and files, so it is safe to say that monitoring can offer a little peace of mind.
Commonly used metrics
custom.inode.percentage.*.used
References: - inodeの監視 ~ mackerel-plugin-inodeを読み解く - そーだいなるらくがき帳 (Japanese only)
Final thoughts
- There is so much information out there on each metric plugin and the different kinds of monitoring items. I learned a lot making this list.
- Although they didn’t make our list this time, there are still lots of other popular plugins like multicore / postgres / fluentd. Hopefully we’ll get a chance to look at some of these in future blogs.
