Mackerel blog #mackerelio

The Official Blog of Mackerel

mackerel-container-agent is now available on Amazon ECR public

Amazon ECR public was just released at AWS’s annual event re:Invent 2020.

To go along with this, we’ve released mackerel-container-agent on Amazon ECR public.

gallery.ecr.aws

The container image for mackerel-container-agent was previously only available from Docker Hub. Now, the mackerel-container-agent container image is also available from Amazon ECR public. Now you have multiple options to choose from.

Billing data can now be monitored with AWS Integration and more

Hello. Mackerel team CRE Inoue (id:a-know) here with another update announcement.

Billing data can now be monitored with AWS Integration

2020-11-20 Postscript

We’ve recently confirmed the occurrence of hosts being registered in Mackerel, but not posting graphs if the budgets name is longer than a certain length on the AWS side.

We apologize for the inconvenience and appreciate your cooperation as we proceed with further investigation to resolve this event.

End of Postscript

f:id:mackerelio:20201118143501p:plain

It is now possible to monitor Billing data with AWS Integration. With this, you can now monitor/visualize billing and budget metrics for your AWS account. If using AWS Organizations for consolidated billing, you can also retrieve metrics for billing amounts per member account. Prices are calculated using the conversion 1 management account = 1 micro host.

Account IDs are used for host names. And Budget names are incorporated as part of metric names, but because Japanese characters cannot be used for metric names in Mackerel, if such a character string is included in a Budget name, those characters are encoded as shown below.

f:id:mackerelio:20201119110122p:plain

Check out the help page below for more details.

mackerel.io

For information regarding how to set up AWS integration, you can refer to the following help page (If using an existing IAM role, setting an additional AWSBudgetsReadOnlyAccess policy is required).

mackerel.io

For a while now, we’ve received a lot of interest in being able to use Mackerel to aggregate and monitor not only the statuses of various infrastructure resources, but also billing data such as this. This release is in response to those requests. Be sure to give it a try!

Support added for 1000+ registered queues in AWS / SQS Integration.

mackerel.io

Up until now, only up to 1000 items could be integrated with AWS / SQS Integration, but with this update, that number is now unlimited.

This will probably only be relevant to a few special use cases, but if applicable to you, be sure to have a look.

Azure Application Gateway now supported with Azure Integration and more!

Hello. Mackerel team CRE Inoue (id:a-know) here with another update announcement.

Azure Application Gateway now supported with Azure Integration

f:id:mackerelio:20201106145512p:plain

Azure Application Gateway can now be monitored with Azure Integration! Application Gateway is Azure's managed load balancer service that enables you to do application layer (OSI layer 7) load balancing.

Check out the help page linked below for configuration methods.

mackerel.io

And the following link for obtainable metrics with Application Gateway.

mackerel.io

Notification content expanded for alerts to PagerDuty

The content that can be included in alert notifications sent from Mackerel to PagerDuty has been expanded. The following items have been added.

  • Monitor notes
  • A thumbnail image of the target metric
  • custom_identifier

Now when an alert occurs, more information can be obtained when looking at PagerDuty. You may want to take this opportunity to consider using PagerDuty or at least review the troubleshooting flow.

Also, please note that this expansion is only available with the Event API v2. For more information regarding alert notifications from Mackerel to PagerDuty, be sure to take a look at the following help page.

mackerel.io

Monitor memos now included in Webhook notification payload

Similar to the previous update regarding Mackerel supported notification channels, Webhook notifications have added monitor memos to its payload.

A variety of information can be included in monitor memos. For example:

  • Information regarding the initial response to an alert (commands or emergency measures)
  • An explanation for or the reason behind the monitor
  • URLs of related documents, etc.

This information is also included in Webhook requests, potentially enabling you to respond more smoothly and reduce recovery time. So be sure to give it a try!

Arm64 package release・Following redirects now possible for external monitoring and more

Hello. Mackerel team CRE Inoue (id:a-know) here with another update announcement.

Arm64 package released for mackerel-agent and more

An Arm64 package for mackerel-agent and various plugin packages is now officially available and can be found in the apt/yum repository (v2 repository).

Some of you may have noticed the recent increase in support for Arm architecture, for example, the release of the A1 instance by Amazon EC2. Feel free to give this a try.

Redirects can now be followed in external monitoring

External URL Monitoring lets you monitor response status codes and response times by simply specifying a URL in the Mackerel Web console. In fact, I think a lot of our users start configuring this monitoring first because it’s not necessary to install the agent.

mackerel.io

Up until now, Mackerel's external URL monitoring specifications were to "recognize 2xx or 3xx-type responses as normal responses and not as errors" and "not to follow redirects", but this behavior has been changed. This can now be changed in settings.

Check out the help page linked below for more details.

mackerel.io

An error message is now displayed when there is a discrepancy between the agent installation script and the OS type or version

f:id:mackerelio:20201030140824p:plain

Here’s an example of what it looks like when running Amazon Linux installation script for an Amazon Linux 2 environment.

Our support team has received numerous inquiries regarding this discrepancy, so we’ve added this as a response. Please be advised when installing.

The official command line tool mkr has been updated

Do you use mkr, Mackerel's official command line tool?

mackerel.io

Well, it has now been updated to v0.41.0. The following changes have been made.

  • Made it easier to understand the output when running mkr hosts and no hosts exist
  • An armhf package for mkr is now generated in GitHub Releases

Thank you to all who contributed!

The system will temporarily shut down for scheduled maintenance on December 2nd (Wed.)

Thank you for choosing Mackerel.

In order to carry out scheduled maintenance, the system will temporarily shut down during the time designated below. We apologize for the inconvenience, and thank you for your understanding.

This maintenance was previously announced as the second of two operations and was originally scheduled for November, but has been moved back.

mackerel.io

Scheduled date and time

  • December 2, 2020 (Wednesday) 1:00pm - 6:00pm (JST)
  • The system will be temporarily shut down within the above time period
  • This time period was set in anticipation of handling issues as they occur. If everything proceeds normally, this time period is expected to end earlier than scheduled

Maintenance subject

Areas affected during maintenance period

  • The scheduled time frame is the longest case scenario and will end as soon as the work has been completed
    • Once maintenance work has begun, the entire Mackerel system will shut down for a short period of time
  • Web access to Mackerel, data posting by the agent, API access (including the CLI tool), alert notifications, etc. will be unavailable during this time
  • As soon as the scheduled maintenance has been completed and working operation has been confirmed, the system shutdown will end and an announcement will be made
  • As for mackerel-agent metric posting, data will be buffered from mackerel-agent during the maintenance period and resent after maintenance has been completed
    • If resent properly, graphs during the maintenance period will also be displayed
    • Please note that if mackerel-agent is stopped or restarted before resending is complete, buffering data will be lost
    • Trying to start up mackerel-agent during maintenance will fail because communication with Mackerel cannot be made
    • Trying to transmit to mackerel during maintenance will fail and an error message will be output in the error log
    • If metric thresholds are exceeded during the maintenance time period, alerts will not be issued retroactively following maintenance completion
    • Check monitoring alerts will be issued for up to 6 hours for events detected during the maintenance time period
  • External monitoring and cloud integration graphs may be incomplete during the maintenance period

Why we chose this time schedule

Due to the fact that the Mackerel service is relied upon for system monitoring, we came to the conclusion that the best time for a temporary shutdown would be during the day when the majority of Mackerel users could be available to respond if necessary, rather than in the middle of the night. For this reason, we’ve scheduled the maintenance to take place during a weekday in Japan Standard Time.

On the other hand, we understand that it might be difficult to perform server operations and deploy applications during the daytime and we sincerely apologize for this inconvenience.

Consideration for our users is essential in determining an appropriate maintenance timeframe and something that we want to continue to improve. If you have any feedback, please let us know.

Announcements

Situation reports will be made on the Mackerel status page (http://status.mackerel.io).

We will also make periodical updates on our official Twitter account (https://twitter.com/mackerelio_jp).

Related inquiries

Please send all inquiries regarding this matter to support@mackerel.io using the forms provided below.

Again, thank you for your understanding and cooperation.

Newly obtainable metrics added for Azure / SQL Database Integration and more

Hello. Mackerel team CRE Inoue (id:a-know) here with another update announcement.

Newly obtainable metrics added for Azure / SQL Database Integration

5 new obtainable metrics have been added and they are as follows.

  • SQL Server process core percent
    • azure.sql_database.sqlserver_process_core_percent.percent
  • SQL Server process memory percent
    • azure.sql_database.sqlserver_process_memory_percent.percent
  • Tempdb Data File Size
    • azure.sql_database.tempdb_data_size.data_size
  • Tempdb Log File Size
    • azure.sql_database.tempdb_log_size.log_size
  • Tempdb Percent Log Used
    • azure.sql_database.tempdb_log_used_percent.percent

Be sure to check out the documentation linked below as well.

mackerel.io

Spec changes made related to posting time and posting interval for check monitoring results.

A lot of our users are using plugins to perform check monitoring for Mackerel.

mackerel.io

These check monitoring results are sent to Mackerel together with a timestamp of when the check process was performed, then an alert is issued according to those results. Up until now, this was handled the same way regardless of how much time had passed. Now, specifications have been changed so that this function is only available up to 6 hours.

In addition to this, you also have maxCheckAttemps, which sets the maximum number of attempts as a mackerel-agent option for check monitoring. However, specifications have been changed so that the range for evaluating this is within 6 hours. (The previous range was within 1 hour.)

Mackerel’s own Watanabe will be presenting at an event on October 28th (Wednesday)!

Mr. Watanabe, Mackerel's very own Product Manager, is scheduled to present at an event titled "Bringing agility to business with SaaS + Serverless" hosted by Amazon Web Services Japan K.K.

aws-serverless.connpass.com (Japanese only)

The theme of this event focuses on Amazon EventBridge (which was recently supported by Mackerel) and Mr. Watanabe will be talking about how to best use Mackerel in serverless environments. Please consider joining us!

Release of check-aws-cloudwatch-logs-insights (beta version)

Hello. Mackerel team’s id:astj here.

Today we are releasing a beta version of check-aws-cloudwatch-logs-insights on GitHub. check-aws-cloudwatch-logs-insights is a check plugin that monitors logs in Amazon CloudWatch Logs. This plugin was created so that high-flow log groups could be monitored utilizing the CloudWatch Logs Insights API.

github.com

Some of you might already be familiar with check-aws-cloudwatch-logs *1, which is a pre-existing official Mackerel check plugin used to monitor logs in Amazon CloudWatch Logs. This plugin works well for low-flow log groups, but struggles when targeting high-flow groups. In some cases, a timeout may occur and inhibit monitoring altogether. check-aws-cloudwatch-logs-insights is intended to enable log monitoring for high capacity log groups using the CloudWatch Logs Insights API.

How to use

Follow the process described below on how to use check-aws-cloudwatch-logs-insights.

Getting the plugin

The plugin can be obtained by using mkr plugin install or by downloading it directly from GitHub. When using mkr plugin install, the following command will extract the v0.0.2 executable file under /opt/mackerel-agent/plugins/bin/. (check the latest version on GitHub)

sudo mkr plugin install mackerelio-labs/check-aws-cloudwatch-logs-insights@v0.0.2

You can also download the executable file from GitHub Releases. Be sure to download the version that matches your OS / architecture.

Releases · mackerelio-labs/check-aws-cloudwatch-logs-insights · GitHub

Using commands

See the README linked below for a more detailed writeup.

github.com

The query commands in CloudWatch Logs Insights Query Syntax - Amazon CloudWatch Logs can be used with the --filter option to filter logs.

And because this plugin uses the AWS API, you’ll need to make sure that it can access the credentials of IAM users / roles that can execute the following 3 actions for the target log group: logs:GetQueryResults logs:StartQuery logs:StopQuery.

The use of EC2 instance profiles, named profiles with the environment variable AWS_PROFILE, and direct access key specification with the environment variables AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY are also supported.

Here are a few examples.

The example below returns a WARNING status when one or more logs containing the character string "ERROR" are found in the log stream /aws/lambda/some-lambda-function, and CRITICAL when 10 or more logs are found.

% AWS_REGION=ap-northeast-1 /path/to/check-aws-cloudwatch-logs-insights --log-group-name=/aws/lambda/some-lambda-function --filter='filter @message like /ERROR/' -warning-over 1 -critical-over 10

This is what it looks like when writing the same monitoring configuration as above in the mackerel-agent configuration file.

[plugin.checks.aws-cloudwatch-logs-insights-sample]
command = ["/path/to/check-aws-cloudwatch-logs-insights", "--log-group-name", "/aws/lambda/some-lambda-function", "--filter", "filter @message like /ERROR/", "--critical-over", "10", "--warning-over", "1"]
env = { AWS_REGION = "ap-northeast-1" }

You can also utilize the CloudWatch Logs Insights query syntax for even more advanced filtering. The next example shows using the level key in JSON format to target any string containing "error".

% AWS_REGION=ap-northeast-1 /path/to/check-aws-cloudwatch-logs-insights --log-group-name=... --filter='filter level = "error"' ...

The difference between check-aws-cloudwatch-logs

As previously mentioned, this plugin was developed to address the issue of unreliability when using check-aws-cloudwatch-logs to monitor high-flow log groups. Now we’ll compare the two and point out the differences and restrictions between this plugin and check-aws-cloudwatch-logs.

  • API requests for CloudWatch Logs Insights incur costs in AWS
  • Insights is less real-time than check-aws-cloudwatch-logs
  • Different expressions are used for filtering logs

API requests for CloudWatch Logs Insights incur costs in AWS

The original check-aws-cloudwatch-logs uses CloudWatch Logs' FilterLogEvents API which does not incur costs in AWS. However, check-aws-cloudwatch-logs-insights uses CloudWatch Logs Insights which incurs costs for the amount of scanned log data. See the AWS page linked below for more details regarding costs. (As of October 2020 in the Tokyo region, 1GB costs 0.005 USD)

aws.amazon.com

Insights is less real-time than check-aws-cloudwatch-logs

Due to the specifications of the CloudWatch Logs API, this plugin monitors logs up to 5 minutes prior the current time. check-aws-cloudwatch-logs is not subject to this limitation, allowing for more real-time monitoring.

Different expression are used for filtering logs

check-aws-cloudwatch-logs uses the CloudWatch Logs FilterLogEvents API which has its own syntax for writing filter conditions.

docs.aws.amazon.com

As mentioned in the ‘Using commands’ section, check-aws-cloudwatch-logs-insights uses the CloudWatch Logs Insights API which requires a different syntax.

docs.aws.amazon.com

Notes / requests

This plugin is currently in beta. We are publishing it as is for the time being and we plan on making improvements based on user feedback. If you have any requests regarding the specifications of this plugin, or regarding CloudWatch Logs monitoring (not limited to this plugin), we would love to hear from you. To send feedback, you can use an issue in the GitHub repository or contact us via Mackerel Support or using the Mackerel User Group channel #check-aws-cloudwatch-logs-insights on Slack.

support.mackerel.io

You can join the Mackerel User Group with this URL.

mackerel-ug-slackin.herokuapp.com

When we release the official version of this plugin in the future, we will also publish an official usage guide.

In addition to this, the specifications of this plugin are subject to change without notice during this beta period. Please check the GitHub repository for the latest information.

Give it a try

We are super excited about the beta release of this new plugin. If you're interested in monitoring logs on Amazon CloudWatch Logs, particularly those with high-flow, we definitely recommend giving this plugin a try.

About mackerelio-labs

Many of the OSS repositories that Mackerel publishes such as mackerel-agent, can be found at https://github.com/mackerelio/. However, the Mackerel team also publishes experimental products and those still under-development at https://github.com/mackerelio-labs/. check-aws-cloudwatch-logs-insights is one of these products. When we release the official version in the future, we will release it in the mackerelio organization.