Mackerel blog #mackerelio

The Official Blog of Mackerel

New metric targets added for AWS / NLB Integration and more

Hello. Mackerel team CRE Inoue (id:a-know) here with another update announcement.

New metric targets added for AWS / NLB Integration

With this update, new metric targets are now obtainable through AWS and NLB Integration. The following metrics relate to TLS termination in NLB.

  • Processed Bytes
    • nlb.bytes.tcp
    • nlb.bytes.tls
    • nlb.bytes.udp
  • Established Active Flow
    • nlb.established_active_flow.tcp
    • nlb.established_active_flow.tls
    • nlb.established_active_flow.udp
  • Consumed LCUs
    • nlb.consumed_lcus.all
    • nlb.consumed_lcus.tcp
    • nlb.consumed_lcus.tls
    • nlb.consumed_lcus.udp
  • New Flow
    • nlb.new_flow.tcp
    • nlb.new_flow.tls
    • nlb.new_flow.udp
  • TLS Negotiation Error
    • nlb.tls_negotiation_error.client

In addition to this, the maximum number of retrievable metrics has been increased from 6 + 2 × (number of target groups) to 21 + 2 × (number of target groups). So depending on the number of target groups, you may exceed the 30 metric limit in which case the excess will be converted into an additional micro host (FAQ・Handling of host conversion when plan limits are exceeded - Mackerel Docs).

Obtaining metrics with AWS Integration can be freely configured at your own discretion in settings. Although, metrics are retrieved by default, so please manage this as necessary.

HTTP API support for AWS / API Gateway Integration

I’m sure that the current development of an HTTP API available (GA) for Amazon API Gateway is news to everyone. (Japanese only)

With this update, it is now possible to monitor HTTP Protocol-Type API Gateway resources. Prior to this, only REST and WebSocket were supported, but now all currently existing Protocol-Type are available.

For more on integratable metrics, check out the documentation linked below.

Reflect a selected metric when sharing service metric graphs to notification channels

Have you ever used the graph sharing function that lets you post graphs to notification channels by clicking the camera icon located in the upper right corner of each graph?


Now, when sharing service metric graphs, you can select a particular metric to be reflected in the shared content.

By selecting a particular metric to share...

Only that selected metric will be shared to the notification channel!

This feature has been supported for system metrics and custom metrics, but is now available for service metric graphs as well. We hope this will be helpful for communicating and addressing specific areas.

You can now use an API to get host monitoring statuses and more

Hello. Mackerel team CRE Inoue (id:a-know) here with another update announcement.

You can now use an API to get host monitoring statuses

With this update, it is now possible to retrieve the monitor associated with the host and its status (monitoring status) using an API. The following are supported.

  • Connectivity monitoring
  • Metric monitoring
  • Check monitoring
  • Anomaly Detection for Roles

Check out the API documentation in the link below.

This API lets you retrieve the information shown in the screenshot of the Web console below.


The purpose of this function is to help you check whether or not your host is being monitored as intended. Be sure to give it a try.

Units can now specified in custom dashboards / value widgets


The value widget is super convenient as it displays the current value of the target metric at an easy-to-read size. With this update, you can now specify an arbitrary unit for this value as well, making it easier to understand what kind of value it is.

The image above uses response time as an example, but posting something like the number of newly registered members or even sales figures on the dashboard might be more interesting. Please give it a try!

mackerel-agent-plugins has been updated(v0.60.2)

This update includes the following revisions. If you're using any of the corresponding plugins, please consider updating.

  • mackerel-plugin-accesslog
    • Cases that have multiple values with retry are now supported in LTSV format log and access log format including apptime:$upstream_response_time.
  • mackerel-plugin-sidekiq
    • Fixed a bug that caused an Invalid value error.

Thank you to everyone who contributed and reported issues!

Mackerel now offers a Telework Support Program

This announcement was previously posted on this blog the other day. (Japanese only)

Due to the current situation, a lot of people have been forced to work in a completely different environment than what they might be used to, and this can be difficult. We want to help such people, so if you’re eligible, please feel free to apply.

Azure Load Balancer now supported with Azure Integration and more

Hello. Mackerel team CRE Inoue (id:a-know) here with another update announcement.

Azure Load Balancer now supported with Azure Integration

With this release, various Azure Load Balancer metrics can now be monitored with Azure Integration!

Billable targets are determined using the conversion 1 resource = 1 Micro Host. Available metrics are listed in the help page linked below.

I know that many of our Azure users out there utilize ALB and hopefully this will help with service monitoring!

The URL character limit for expression graphs has been significantly relaxed

Do you use Mackerel’s expression graphs? If you just recently started using Mackerel, you may not know about this feature.

Expression graphs let you perform various calculations for metrics using functions and display the results in a flexible graph. For more details, check out the help page linked below.

To use expression graphs, you write an expression in the URL query parameter like the one shown below.<organization name>/advanced-graph?query=<metric>&unit=<unit>&title=<title>

Up until now, the number of characters for this URL was limited to 2048. This number has now been increased 4 fold to a limit of 8192 characters.

With this, it is now possible to use the lengthy expressions that could not be specified before. If you’ve given up on graphing something due to this character limit in the past, please give it another shot.

Finally, please note that the expression graph function is currently listed as an experimental feature.

Mackerel’s first online seminar!

This will be the first Mackerel-hosted event to be held as an online seminar! (Japanese only)

For this event, we would like to provide the opportunity to get to know Mackerel starting from the basics. The topic of discussion titled "Getting Started with Mackerel - An Introduction to Cloud Monitoring".

We’ll be using Zoom to facilitate the online seminar. Please feel free to join us!

Regarding the agent output error “x509: certificate signed by unknown authority”

Hello. Mackerel team CRE Inoue (id:a-know) here.

We are currently updating the certificate, and during this time, we have confirmed cases of the Mackerel agent outputting the error x509: certificate signed by unknown authority.

The Mackerel agent output log will show the following.

Failed to post metrics value (will retry): Post x509: certificate signed by unknown authority

In response to this, we have created a new FAQ page.

We have confirmed that this event may occur when the CA certificate of the host on which Mackerel agent is installed is outdated (when using CentOS 6 etc.). If this is the case, please update the ca-certificates package or update Windows as described in the FAQ page above.

In light of this occurrence, we will temporarily suspend renewing the certificate and plan to resume on Tuesday, May 12th, 2020 (JST).

This event of a CA certificate becoming outdated and failing to establish SSL/TLS communication can occur in any case where SSL/TLS communication is performed from a host, and is not limited to Mackerel. So we recommend that you update your host’s CA certificate regularly.

3 Mackerel functions to help make server monitoring from home easier

Mackerel team CRE Yoshida (id:syou6162) here. And today I’d like to highlight 3 Mackerel features that really come in handy when working remotely.

Sharing graphs with team members by posting to Slack

When working / developing with a team while physically together in the same room, it’s probably most common to have the Mackerel dashboard displayed on a large screen for everyone to see (like in the WILLER case study). If everyone’s looking at the same screen, the discussion can progress. However, this approach might need to change when the situation calls for working remotely through applications such as Google Meet or Zoom, and depending on certain variables like connection status and bandwidth, sharing the same screen with every member may not be an option.

For situations such as this, we definitely recommend using the graph sharing function. You can easily post graphs to notification channels by simply clicking the camera button on the graph screen.


This function lets you share a graph that you are currently viewing in Mackerel with notification channels. With just a few clicks, you can share any graph that you might want to discuss with team members without having to share the screen itself. We believe that this can be really useful and hope that you put it to good use while working remotely.

Graph Annotations

When working from home, I personally find it more difficult for communication to go smoothly compared to working face to face. And if the situation necessitates more communication, a higher chat flow is often inevitable. I’m sure many of us who regularly post notifications about server operations in congested chats can understand, keeping track of what happened throughout the day can be a struggle.

For this, Mackerel’s graph annotation function can be really useful. Graph annotations allow you to write and leave information for services and roles like details of when certain operations were performed.

Leaving graph annotations makes it easier to track the changes made to the service. We highly recommend adding a link to the pull request or a Git commit hash in the description as this will make it much easier to understand the situation. Also, being able to see the metrics and the annotation together is really helpful in understanding what caused the change.

Additionally, you can implement this for all kinds of things depending on the situation. Fellow CRE id:a-know wrote a blog post about leaving output lines detected by log monitoring as graph annotations. Check that out in the link below.

(Japanese only)

You can also post annotations using the mkr command, which is Mackerel's official CLI tool, or with the API. On top of that, you can have annotations posted automatically when the release is completed by incorporating these in the CI/CD tool application release flow.

When it comes to server operations, you might have more to think about when working remotely compared to in the office, but we hope that using graph annotations can help lighten that load a little.

An easy to read user interface even from your smartphone

One of the obvious perks of working from home is not having to commute. But for some, that time is used to mentally switch on/off from work. I can sympathize. After work, I want to close my PC and just relax, but more often than not, an alert or a problem will pop up and drag me right back. Even if I tell myself I’m just checking an alert, if I reopen my PC, it’s easy to inadvertently get sucked back in.

Mackerel’s user interface was designed to be user friendly and easy to read on PC, but the same goes for smartphones as well. The graph sharing function, which we covered in the beginning, can easily be utilized from your smartphone and checking alerts is quick and easy. If it’s not a problem that needs to be dealt with right away, you can just stay lying down and relax. Being able to use Mackerel from your phone while relaxing might help relieve some stress.

Bonus: Monitor your own room environment with service metrics

When stuck at home all day, sometimes it’s easy to get caught up in work and forget about important things like ventilation. If the CO2 level gets too high and your head starts feeling a bit foggy, that’s a situation we definitely want to avoid.

Some users monitor the carbon dioxide and humidity levels in their rooms using IoT devices with Mackerel service metrics. It's relatively easy to install, so if you're interested, why not give it a try?

(Japanese only)

To close

For many of us, remote work is becoming the norm and there’s going to be a lot that needs to change. But with the help of Mackerel’s intuitive functions, I think that service monitoring can still run smoothly even while working remotely. If you’ve been meaning to try any of these functions out but haven’t had the chance, by all means, now is as good of a time as any.

AWS WAF (v1 / v2) now supported with AWS Integration

Hello. Mackerel team CRE Inoue (id:a-know) here with another update announcement.

AWS WAF (v1 / v2) now supported with AWS Integration

Various WAF metrics can now be monitored with AWS integration!

Billable targets are determined using a conversion of 1 Web Access Control List = 1 Micro Host. Available metrics are listed in the help page linked below.

Now, it’s possible to monitor requests processed by Web Application Firewall and more. We hope this will help with monitoring services and various metrics of web applications under the protection of WAF!

Specifications changed so that values larger than 100% are no longer registrable for thresholds of metric rate monitors

This was previously announced in an earlier post on this blog. For more details, check out the notice linked below.

Content added to alert notifications of Amazon EventBridge and Webhook

In Mackerel, you can send alert notifications not only by email or Slack, but with other services like Amazon EventBridge and Webhook as well. With this update, the following items are now included in notifications sent with Amazon EventBridge or Webhook.

  • (EventBridge notifications only)Instance ID
  • Alert ID

Amazon EventBridge and Webhook are both extremely useful features regarding all sorts of automation. Of course, it’s best to look into the cause of a problem when they occur, but if that’s not possible, automating the response can be really helpful. We highly recommended that you take a moment and consider whether or not you could benefit from a little automation.

Windows version of mackerel-agent installer change to 64bit version

The Windows version of mackerel-agent installer that can be downloaded from the New Host Registration Screen in the Mackerel Web console has been changed to the 64-bit version. The installation folder has been changed to C:\Program Files\Mackerel. Even if you are currently using the 32-bit version, you can upgrade by using this.

If you wish to continue using the 32-bit mackerel-agent installer, you can download it from the official document linked below.

【Spec Change Notice】Values larger than 100% no longer registrable for thresholds of metric rate monitors

Mackerel team CRE Inoue (id:a-know) here. Today’s announcement concerns changes to specifications.

As stated in the title, it will no longer be possible to register values that are larger than 100% for metric rate monitor thresholds. These changes will be made to the specifications for the Web console and the API.

About metric rate monitors

First, be sure to refer to the help page linked below for a more detailed write-up on metric rate monitors.

Specification changes

As described in the help page above, threshold values for metric rate monitors are specified as "percentages", and the maximum value was originally intended to be 100%, but up until now, it has been possible to enter a value over 100.

This change will make it so that a value larger than 100 cannot be registered. We apologize for the oversight on our part by allowing a value that is practically invalid from the viewpoint of monitoring to be set.


The effects of this specification change are as follows.

  • Following this release, a value larger than 100 will no longer be registrable as a threshold for metric rate monitors.
    • An error will occur when registering from the Web console and also when registering with the API.
  • Monitors that have already been registered with a value larger than 100 will be left as they are.
    • As mentioned above, this state is practically invalid from a monitoring viewpoint, so we recommend updating the settings to appropriate values.

Scheduled release date

The release that will include these specification changes is scheduled for Tuesday, April 21, 2020 (JST).

We greatly appreciate your understanding and cooperation.