Mackerel blog #mackerelio

The Official Blog of Mackerel

Email notifications now available for events such as host registration and monitor changes and more

Hello. Mackerel team CRE Inoue (id:a-know) here with another update announcement.

Email notifications now available for events such as host registration and monitor changes

Up until now, event notifications for things like registering a host in Mackerel and changes to monitors were only supported for some notification channels such as Slack and Webhook.

Now, these notifications are also available for email as well.

We hope that email notifications will help you stay on top of the unexpected.

New metric targets added for AWS・S3 Integration

Two types of metrics have been added, Bucket Size and Number Of Objects. See the help page linked below.

mackerel.io

Please note that both are daily metrics.

OSS Updates

Today’s update covers a wide range of OSS products.

mackerel-agent-plugins(v0.63.2)

mackerel-check-plugins(v0.38.0)

  • [check-postgresql] Added support for the sslrootcert option used to encrypt plugin communication

mackerel-container-agent(v0.5.0)

  • CPU / memory metrics for container-agent now available with Kubernetes 1.18 and later.
    • Additionally, a get operation (verb) is no longer required for the nodes/spec resource in the RBAC configuration of Kubernetes (EKS on Fargate).

mkr(v0.42.0)

  • [monitors subcommand] Support added for alertStatusOnGone (specifying the alert status when an alert occurs) with push pull.

Thank you to everyone who contributed!

Support availability during the year-end and New Year holidays

The end of the year is fast approaching with less than three weeks left in 2020. And thank you for choosing Mackerel this year.

Mackerel’s official support window will be closed from December 28, 2020 (Thursday) until January 3, 2021 (Sunday).

For those of you planning to look over your Mackerel settings etc. before the end of the year and New Year holidays, we recommend you do so as soon as possible. And if you have any questions or concerns, please do not hesitate to contact us.

Mackerel’s Official Support Request Form

mackerel-container-agent is now available on Amazon ECR public

Amazon ECR public was just released at AWS’s annual event re:Invent 2020.

To go along with this, we’ve released mackerel-container-agent on Amazon ECR public.

gallery.ecr.aws

The container image for mackerel-container-agent was previously only available from Docker Hub. Now, the mackerel-container-agent container image is also available from Amazon ECR public. Now you have multiple options to choose from.

Billing data can now be monitored with AWS Integration and more

Hello. Mackerel team CRE Inoue (id:a-know) here with another update announcement.

Billing data can now be monitored with AWS Integration

2020-11-20 Postscript

We’ve recently confirmed the occurrence of hosts being registered in Mackerel, but not posting graphs if the budgets name is longer than a certain length on the AWS side.

We apologize for the inconvenience and appreciate your cooperation as we proceed with further investigation to resolve this event.

End of Postscript

f:id:mackerelio:20201118143501p:plain

It is now possible to monitor Billing data with AWS Integration. With this, you can now monitor/visualize billing and budget metrics for your AWS account. If using AWS Organizations for consolidated billing, you can also retrieve metrics for billing amounts per member account. Prices are calculated using the conversion 1 management account = 1 micro host.

Account IDs are used for host names. And Budget names are incorporated as part of metric names, but because Japanese characters cannot be used for metric names in Mackerel, if such a character string is included in a Budget name, those characters are encoded as shown below.

f:id:mackerelio:20201119110122p:plain

Check out the help page below for more details.

mackerel.io

For information regarding how to set up AWS integration, you can refer to the following help page (If using an existing IAM role, setting an additional AWSBudgetsReadOnlyAccess policy is required).

mackerel.io

For a while now, we’ve received a lot of interest in being able to use Mackerel to aggregate and monitor not only the statuses of various infrastructure resources, but also billing data such as this. This release is in response to those requests. Be sure to give it a try!

Support added for 1000+ registered queues in AWS / SQS Integration.

mackerel.io

Up until now, only up to 1000 items could be integrated with AWS / SQS Integration, but with this update, that number is now unlimited.

This will probably only be relevant to a few special use cases, but if applicable to you, be sure to have a look.

Azure Application Gateway now supported with Azure Integration and more!

Hello. Mackerel team CRE Inoue (id:a-know) here with another update announcement.

Azure Application Gateway now supported with Azure Integration

f:id:mackerelio:20201106145512p:plain

Azure Application Gateway can now be monitored with Azure Integration! Application Gateway is Azure's managed load balancer service that enables you to do application layer (OSI layer 7) load balancing.

Check out the help page linked below for configuration methods.

mackerel.io

And the following link for obtainable metrics with Application Gateway.

mackerel.io

Notification content expanded for alerts to PagerDuty

The content that can be included in alert notifications sent from Mackerel to PagerDuty has been expanded. The following items have been added.

  • Monitor notes
  • A thumbnail image of the target metric
  • custom_identifier

Now when an alert occurs, more information can be obtained when looking at PagerDuty. You may want to take this opportunity to consider using PagerDuty or at least review the troubleshooting flow.

Also, please note that this expansion is only available with the Event API v2. For more information regarding alert notifications from Mackerel to PagerDuty, be sure to take a look at the following help page.

mackerel.io

Monitor memos now included in Webhook notification payload

Similar to the previous update regarding Mackerel supported notification channels, Webhook notifications have added monitor memos to its payload.

A variety of information can be included in monitor memos. For example:

  • Information regarding the initial response to an alert (commands or emergency measures)
  • An explanation for or the reason behind the monitor
  • URLs of related documents, etc.

This information is also included in Webhook requests, potentially enabling you to respond more smoothly and reduce recovery time. So be sure to give it a try!

Arm64 package release・Following redirects now possible for external monitoring and more

Hello. Mackerel team CRE Inoue (id:a-know) here with another update announcement.

Arm64 package released for mackerel-agent and more

An Arm64 package for mackerel-agent and various plugin packages is now officially available and can be found in the apt/yum repository (v2 repository).

Some of you may have noticed the recent increase in support for Arm architecture, for example, the release of the A1 instance by Amazon EC2. Feel free to give this a try.

Redirects can now be followed in external monitoring

External URL Monitoring lets you monitor response status codes and response times by simply specifying a URL in the Mackerel Web console. In fact, I think a lot of our users start configuring this monitoring first because it’s not necessary to install the agent.

mackerel.io

Up until now, Mackerel's external URL monitoring specifications were to "recognize 2xx or 3xx-type responses as normal responses and not as errors" and "not to follow redirects", but this behavior has been changed. This can now be changed in settings.

Check out the help page linked below for more details.

mackerel.io

An error message is now displayed when there is a discrepancy between the agent installation script and the OS type or version

f:id:mackerelio:20201030140824p:plain

Here’s an example of what it looks like when running Amazon Linux installation script for an Amazon Linux 2 environment.

Our support team has received numerous inquiries regarding this discrepancy, so we’ve added this as a response. Please be advised when installing.

The official command line tool mkr has been updated

Do you use mkr, Mackerel's official command line tool?

mackerel.io

Well, it has now been updated to v0.41.0. The following changes have been made.

  • Made it easier to understand the output when running mkr hosts and no hosts exist
  • An armhf package for mkr is now generated in GitHub Releases

Thank you to all who contributed!

The system will temporarily shut down for scheduled maintenance on December 2nd (Wed.)

Thank you for choosing Mackerel.

In order to carry out scheduled maintenance, the system will temporarily shut down during the time designated below. We apologize for the inconvenience, and thank you for your understanding.

This maintenance was previously announced as the second of two operations and was originally scheduled for November, but has been moved back.

mackerel.io

Scheduled date and time

  • December 2, 2020 (Wednesday) 1:00pm - 6:00pm (JST)
  • The system will be temporarily shut down within the above time period
  • This time period was set in anticipation of handling issues as they occur. If everything proceeds normally, this time period is expected to end earlier than scheduled

Maintenance subject

Areas affected during maintenance period

  • The scheduled time frame is the longest case scenario and will end as soon as the work has been completed
    • Once maintenance work has begun, the entire Mackerel system will shut down for a short period of time
  • Web access to Mackerel, data posting by the agent, API access (including the CLI tool), alert notifications, etc. will be unavailable during this time
  • As soon as the scheduled maintenance has been completed and working operation has been confirmed, the system shutdown will end and an announcement will be made
  • As for mackerel-agent metric posting, data will be buffered from mackerel-agent during the maintenance period and resent after maintenance has been completed
    • If resent properly, graphs during the maintenance period will also be displayed
    • Please note that if mackerel-agent is stopped or restarted before resending is complete, buffering data will be lost
    • Trying to start up mackerel-agent during maintenance will fail because communication with Mackerel cannot be made
    • Trying to transmit to mackerel during maintenance will fail and an error message will be output in the error log
    • If metric thresholds are exceeded during the maintenance time period, alerts will not be issued retroactively following maintenance completion
    • Check monitoring alerts will be issued for up to 6 hours for events detected during the maintenance time period
  • External monitoring and cloud integration graphs may be incomplete during the maintenance period

Why we chose this time schedule

Due to the fact that the Mackerel service is relied upon for system monitoring, we came to the conclusion that the best time for a temporary shutdown would be during the day when the majority of Mackerel users could be available to respond if necessary, rather than in the middle of the night. For this reason, we’ve scheduled the maintenance to take place during a weekday in Japan Standard Time.

On the other hand, we understand that it might be difficult to perform server operations and deploy applications during the daytime and we sincerely apologize for this inconvenience.

Consideration for our users is essential in determining an appropriate maintenance timeframe and something that we want to continue to improve. If you have any feedback, please let us know.

Announcements

Situation reports will be made on the Mackerel status page (http://status.mackerel.io).

We will also make periodical updates on our official Twitter account (https://twitter.com/mackerelio_jp).

Related inquiries

Please send all inquiries regarding this matter to support@mackerel.io using the forms provided below.

Again, thank you for your understanding and cooperation.

Newly obtainable metrics added for Azure / SQL Database Integration and more

Hello. Mackerel team CRE Inoue (id:a-know) here with another update announcement.

Newly obtainable metrics added for Azure / SQL Database Integration

5 new obtainable metrics have been added and they are as follows.

  • SQL Server process core percent
    • azure.sql_database.sqlserver_process_core_percent.percent
  • SQL Server process memory percent
    • azure.sql_database.sqlserver_process_memory_percent.percent
  • Tempdb Data File Size
    • azure.sql_database.tempdb_data_size.data_size
  • Tempdb Log File Size
    • azure.sql_database.tempdb_log_size.log_size
  • Tempdb Percent Log Used
    • azure.sql_database.tempdb_log_used_percent.percent

Be sure to check out the documentation linked below as well.

mackerel.io

Spec changes made related to posting time and posting interval for check monitoring results.

A lot of our users are using plugins to perform check monitoring for Mackerel.

mackerel.io

These check monitoring results are sent to Mackerel together with a timestamp of when the check process was performed, then an alert is issued according to those results. Up until now, this was handled the same way regardless of how much time had passed. Now, specifications have been changed so that this function is only available up to 6 hours.

In addition to this, you also have maxCheckAttemps, which sets the maximum number of attempts as a mackerel-agent option for check monitoring. However, specifications have been changed so that the range for evaluating this is within 6 hours. (The previous range was within 1 hour.)

Mackerel’s own Watanabe will be presenting at an event on October 28th (Wednesday)!

Mr. Watanabe, Mackerel's very own Product Manager, is scheduled to present at an event titled "Bringing agility to business with SaaS + Serverless" hosted by Amazon Web Services Japan K.K.

aws-serverless.connpass.com (Japanese only)

The theme of this event focuses on Amazon EventBridge (which was recently supported by Mackerel) and Mr. Watanabe will be talking about how to best use Mackerel in serverless environments. Please consider joining us!