Mackerel blog #mackerelio

The Official Blog of Mackerel

Alert status can now be selected when an alert occurs with host connectivity monitors and more

Hello. Mackerel team CRE Inoue (id:a-know) here with another update announcement.

Alert status can now be selected when an alert occurs with host connectivity monitors

Among the various monitors that can be configured with Mackerel, ‘Host connectivity monitoring’ can detect when metric transmission from the agent (mackerel-agent) gets interrupted and issues an alert. However up until now, the status of this alert was fixed to critical. With this update, it is now possible to choose the status of an alert when it occurs.

Some users may not even be aware of host connectivity monitoring due to the fact that it’s already configured from the get-go. We recommend reviewing the page linked below.

support.mackerel.io

New obtainable metrics added for AWS/RDS Integration

With this update, now up to 52 metrics are available for RDS Aurora, up to 51 for Aurora Serverless, and up to 24 metrics are available for other services. Be sure to check out the differences detailed in the link below. Also, please note that some metrics are supported for certain database engines while others may not.

github.com

OSS Updates

mackerel-agent-plugins(v0.62.0)

A bug in mackerel-plugin-linux that prevented some OS versions (Debian 10 / RHEL 8 / Ubuntu 20.04) from retrieving disk stats was fixed.

mackerel-check-plugins(v0.36.0)

In check-log, {} is now supported for --file option specification (glob format). It is now possible to specify commands like the one below.

command = ["check-log", "--file", "/var/log/{secure,cron}", "-p", "timeout"]

Let us know what you think!

New cloud integration metrics and more

Hello. Mackerel team CRE Inoue (id:a-know) here with another update announcement.

New cloud integration metrics have been added

Newly obtainable metrics have been added for each of the managed services for cloud integration below.

Azure SQL Database

Metrics can now be retrieved according to price tier. See the help page below for more information on which metrics you can get at each price level.

mackerel.io

Amazon CloudFront

With the addition of eight real-time metrics in Amazon CloudWatch, Amazon CloudFront is now supported with Mackerel's AWS integration.

aws.amazon.com

The following metrics have been added.

  • ErrorRate
    • 401ErrorRate (cloudfront.error_rate.401_error_rate)
    • 403ErrorRate (cloudfront.error_rate.403_error_rate)
    • 404ErrorRate (cloudfront.error_rate.404_error_rate)
    • 502ErrorRate (cloudfront.error_rate.502_error_rate)
    • 503ErrorRate (cloudfront.error_rate.503_error_rate)
    • 504ErrorRate (cloudfront.error_rate.504_error_rate)
  • CacheHitRate
    • CacheHitRate (cloudfront.cache_hit_rate)
  • OriginLatency
    • OriginLatency (cloudfront.origin_latency)

Be sure to check out the help page below as well.

mackerel.io

OSS Updates

mackerel-agent-plugins (v0.61.0)

Regarding the mackerel-plugin-accesslog plugin, a parsing error will no longer be output for fields included in the access log that are not used by the plugin.

mackerel-check-plugins (v0.35.0)

The check-ssl-cert plugin now also monitors the expiration date of intermediate certificates.

mackerel-client-go

A bug was fixed that caused HTTP Keep-Alive to not perform under certain conditions.

We will be holding an online seminar to present the updates of the first half of 2020!

As we’ve previously stated on this blog, Mackerel is constantly making new updates. We’ve already made a ton of updates this year.

For this event, we’ll be presenting the updates made in the first half of 2020, and a Mackerel team CRE will introduce their top functions with a demo!

mackerelio.connpass.com (Only available in Japanese)

There may be some updates that you missed. Please consider participating!

An error regarding obtaining metrics with mackerel-plugin-mysql has been fixed

Hello. Mackerel team CRE Inoue (id:a-know) here. This is an announcement regarding an error that was fixed for mackerel-plugin-mysql of the official Mackerel plugin pack.

What was fixed?

The metrics obtainable with mackerel-plugin-mysql are Bytes_sent (mysql.traffic.Bytes_sent) and Bytes_received (mysql.traffic.Bytes_received). These metrics represent the number of bytes sent and received by all clients.

The plugin (1) retrieves the corresponding metrics recorded as the cumulative total value in MySQL and (2) calculates the difference and sends it to Mackerel. When browsing these metrics in Mackerel, the expected unit is bytes/sec. However, a value 60 times larger (the value of bytes/min) than the actual value was being sent. This has been corrected according to the Pull Request linked below. Now, a value that is 1/60 of the previous value will be sent.

github.com

What needs to be done?

Today we are updating Mackerel’s official plugin package mackerel-agent-plugins. Be sure to update the mackerel-agent-plugins package with your package management system (yum or apt).

mackerel.io

With this update, the values of Bytes_sent (mysql.traffic.Bytes_sent) and Bytes_received (mysql.traffic.Bytes_received) will be sent as 1/60 of the previous values. If you’ve created a monitor for these metrics, please be sure to review the thresholds as well.

We apologize for any inconvenience this may have caused and thank you for your understanding.

Azure Database for MySQL / PostgreSQL now supported with Azure Integration!

Hello. Mackerel team CRE Inoue ( id:a-know) here with another update announcement.

Azure Database for MySQL / PostgreSQL now supported with Azure Integration

f:id:mackerelio:20200702150259p:plain

It is now possible to monitor Azure Database with Azure integration! Azure Database is Azure’s fully managed database service! Both MySQL and PostgreSQL are now supported.

Check out the help page linked below for configuration methods.

mackerel.io

And be sure to check out each of the following help pages regarding obtaining metrics with this newly added integration.

mackerel.io

mackerel.io

The selected metric is now reflected when posting role metric graphs to notification channels

https://cdn-ak.f.st-hatena.com/images/fotolife/m/mackerelio/20200603/20200603123042.png

Now, when using the camera icon to post graphs to notification channels, the metric selection is reflected in the shared content. The same goes for host metric graphs and service metric graphs as well. We hope this is helpful!

Tips to help your inquiries to the support team get resolved quicker

Hello, Mackerel team CRE Nishiyama (id:tukaelu) here.

Have you ever searched through our FAQ (Frequently Asked Questions) or contacted our support team with questions regarding Mackerel? Well, the FAQ site and the Feedback form have both been recently updated!

And although the new feedback form is now more detailed than before, entering the correct information into the required fields is important if you want a speedy resolution.

Today we'll be taking a look at a few pointers that will help prompt a quick resolution when asking a question via the feedback form!

Make use of the ‘Subject’ field

The Subject field doesn’t need any new explanation per se, but one of the handy new features of the feedback form is that it now displays a list of suggested FAQ pages that seem to be relevant to the content entered in the subject line!

The suggested FAQ articles are listed like shown in the image above. Be sure to check for any articles that might be related!

Go into detail when filling in the Description

As for the Description of your inquiry, including the following points may lead to a quicker resolution.

Specify environment information

Including the following points detailing the environment of the host to which you are inquiring about will benefit the resolution turnaround.

  • OS type, version
  • Version of mackerel-agent
  • Plugin name, version (※If using an official plugin)
  • Service name for inquiries regarding various integration

Also, if you’re monitoring middleware with a plugin, inform us of the software name and version.

Specify the situation, status, etc.

Describe in detail the event or problem you are experiencing. Try to be as clear as possible about what operations were performed and what events occurred.

  • NG
    • mackerel-agent doesn’t start up. What are the possible causes?
  • OK
    • When mackerel-agent is restarted with the attached settings, the following error occurs and does not start up.

You can also attach files to the new feedback form. Attaching information like the following to your inquiry can help lead to a quicker resolution.

  • mackerel-agent logs
  • mackerel-agent.conf
    • Masking things like API keys is not a problem
  • User investigated content, executed commands, results, etc.
  • Screenshots that can be verified from the web console etc.

Include information that can be verified from the Mackerel Web Console

If what you are inquiring about can be verified from the Mackerel web console, please include the URL.

  • Host details URL
  • Alert details URL
  • Monitor details URL etc.

The review of your inquiry on our end will go a lot smoother if you can provide us with a URL that can be verified from the web console, such as a limited time graph URL. If this information is available, please make sure to attach it when contacting us.

mackerel.io

Other Tips

In addition to the everything mentioned above, please consider the following points.

1 question per case

It may be quicker to have one question answered at a time.

If you have multiple questions that are all related, of course it’s better to consolidate them into one inquiry. But if this is not the case, you might try splitting up your questions into multiple inquiries.

An inquiry may be difficult to investigate if related to a retired host or deleted monitor

We recommend trying to reserve the resources (hosts, monitors, etc.) related to your inquiry as much as possible.

And please inform us if deleted or any changes were made.

You can add recipients (Cc) to receive a reply

You can ‘Cc’ additional recipients of the response by specifying email addresses separated by commas(,) in the "Additional email addresses to receive replies" field of the feedback form.

This can be useful for sharing the support team’s response among your team or people of concern.

Please inform us when the issue is resolved

If our support team’s response helps resolve your issue, please let us know and we can close the support case.

To close

This has been a look at a few useful pointers that can help get your inquiries to the support team resolved quicker!

If you have any questions or requests regarding Mackerel, please feel free to contact us!

An API for AWS Integration settings and more

Hello. Mackerel team CRE Inoue (id:a-know) here with another update announcement.

An API for AWS Integration settings!

Some users have been waiting expectantly for this update. The API documentation is linked below.

mackerel.io

It is now possible to configure AWS Integration settings from the API. This includes specifying integration excludable metrics. Be sure to give this a try!

The Organizations page can now be sorted by number of alerts

f:id:mackerelio:20200612183053p:plain

The Organizations page shows a list of the organizations to which you belong. With this update, you can now rearrange this list by the number of occurred / continuing alerts. This is a nice update for those who manage a lot of organizations. We hope that you find it useful.

The Web Console’s Feedback form gets a new look!

Have you ever tried contacting our support team using the ‘Feedback’ button that’s located in the upper right corner of the web console? Well, we’ve updated the request form.

Now you can attach files to the new request form. Feel free to contact us using screenshots or configuration files that relate to your inquiry!

Taking a look at the top 10 most popular metric plugins on Mackerel

Mackerel team CRE Yoshida (id:syou6162) here. One of the great things about Mackerel is the abundance of plugins. So much so in fact, I’m sure there are plenty of people who find it rather difficult deciding which plugin they should use. On top of this, you can visualize middleware metrics after installing the plugin, but which metrics should you choose to monitor?

This post will hopefully help make these decisions easier. Today we’ll be taking a look at...

  • The top 10 most used metric plugins on Mackerel
  • Plugin metrics commonly used for monitoring

There are three types of Mackerel plugins (listed below), but in this entry we’ll be taking a look at the usage of metric plugins.

  • Metric plugins
    • Plugins that post statistics associated with the host as custom metrics
  • Check plugins
    • Plugins that determine OK / NG (CRITICAL or WARNING or UNKNOWN) within the host and post the results to Mackerel.
    • Refer here for more details
  • Metadata plugins
    • Plugins that register arbitrary JSON data for each host
    • Refer here for more details

Top 10 most used metric plugins on Mackerel!

Without further ado, here are the rankings! ...Ta-da!

Rank Plugin Usage Score*1
1 linux 7.3
2 mysql 4.4
3 nginx 3.4
4 apache2 3.3
5 docker 2.3
6 redis 1.2
7 accesslog 1.1
8 conntrack 1.1
9 jvm 1.1
10 inode 1

I’m sure a lot of you are already familiar with some of these plugins, but there might be a few you’ve never heard of. Let's take a look at each individual plugin, including what metrics are monitored.

#1: linux plugin

The top spot on our list goes to the linux plugin with a landslide victory over 2nd place by almost double the score. This plugin is essential if you use Linux. Metrics that are most commonly monitored include the number of logged-in users and network status.

Commonly used metrics
  • custom.linux.users.users
  • custom.linux.ss.ESTAB
  • custom.linux.ss.CLOSE-WAIT
  • custom.linux.swap.pswpin
  • custom.linux.ss.TIME-WAIT

References:

#2: mysql plugin

Coming in at #2, the mysql plugin is a must-have database for service operation. It's a popular RDBMS and understandably so. Metrics that are commonly monitored include the number of connections per status, deadlock detection, and slow query statistics.

Commonly used metrics
  • custom.mysql.seconds_behind_master.Seconds_Behind_Master
  • custom.mysql.threads.Threads_running
  • custom.mysql.capacity.PercentageOfConnections
  • custom.mysql.threads.Threads_connected
  • custom.mysql.table_locks.Slow_queries

References:

#3: nginx plugin

The nginx plugin takes bronze with a respectable usage score of 3.4. Some commonly used monitoring items include the number of connections and queue status.

Commonly used metrics
  • custom.nginx.connections.connections
  • custom.nginx.requests.requests
  • custom.nginx.queue.waiting
  • custom.nginx.requests.handled
  • custom.nginx.requests.accepts

#4: apache2 plugin

Following #3 with another web server related plugin, apache2 plugin ranks 4th. Commonly used metrics are similar to those of the nginx plugin, and it seems that the number of connections and the status of workers are monitored well.

Commonly used metrics
  • custom.apache2.workers.busy_workers
  • custom.apache2.workers.idle_workers
  • custom.apache2.req.requests
  • custom.apache2.cpu.cpu_load

#5: docker plugin

Of course users want to monitor the containerization process, so it’s not surprising that the docker plugin ranks 5th on our list. Most of the actual monitoring items are related to memory usage and state.

Commonly used metrics
  • custom.docker.running.XXX.state
  • custom.docker.memory_used.XXX.memory_percentage
The `XXX` segment differs depending on the environment.

#6: redis plugin

The redis plugin comes in at #6 and marks the turning point for our list. On a related note, the similarly popular key-value store memcached just missed our list coming in at #11. Some metrics that are commonly monitored include the number of connected clients, memory usage, and so on.

Commonly used metrics
  • custom.redis.clients.connected_clients
  • custom.redis.connections.rejected_connections
  • custom.redis.capacity.percentage_of_memory
  • custom.redis.uptime.uptime_in_seconds
  • custom.redis.capacity.percentage_of_clients

#7: accesslog plugin

Breaking in at #7, accesslog is another web server-related plugin. This plugin aggregates logs in Apache and LTSV format and is also super useful for SLI / SLO measurement. Try using it together with the nginx and apache2 plugins.

Commonly used metrics
  • custom.accesslog.access_num.5xx_count
  • custom.accesslog.access_rate.5xx_percentage
  • custom.accesslog.access_rate.4xx_percentage
  • custom.accesslog.latency.99_percentile
  • custom.accesslog.access_num.2xx_count

References:

#8: conntrack plugin

At #8, we have the conntrack plugin. This plugin monitors ip_conntrack, which is statistical information on iptables tracking. ip_conntrack has a tracking limit and if this limit is exceeded, it will count as a new session and network performance may deteriorate. So don't forget to add ip_conntrack tracking for servers that handle a lot of traffic (load balancers and cache servers).

Commonly used metrics
  • custom.conntrack.count.used
  • custom.conntrack.count.free

References:

#9: jvm plugin

The jvm plugin hits our list at #9. When it comes to operating applications run with jvm, certain monitoring items are unavoidable. Some typical examples would be time spent in GC / number of events, heap usage, and so on.

Commonly used metrics
  • custom.jvm.bootstrap.gc_events.FGC
  • custom.jvm.bootstrap.gc_time.FGCT
  • custom.jvm.heap.used
  • custom.jvm.prodserverstart.gc_events.FGC
  • custom.jvm.elasticsearch.memorySpace.oldSpaceRate

#10: inode plugin

And finally, the last spot on our list goes to the inode plugin coming in at #10. Exhausting inodes can lead to problems like not being able to create new directories and files, so it is safe to say that monitoring can offer a little peace of mind.

Commonly used metrics
  • custom.inode.percentage.*.used

References: - inodeの監視 ~ mackerel-plugin-inodeを読み解く - そーだいなるらくがき帳 (Japanese only)

Final thoughts

  • There is so much information out there on each metric plugin and the different kinds of monitoring items. I learned a lot making this list.
  • Although they didn’t make our list this time, there are still lots of other popular plugins like multicore / postgres / fluentd. Hopefully we’ll get a chance to look at some of these in future blogs.

*1:The usage score is a relative value that represents the number of users when 10th place equals 1.