Mackerel blog #mackerelio

The Official Blog of Mackerel

The system will temporarily shut down for scheduled maintenance on December 2nd (Wed.)

Thank you for choosing Mackerel.

In order to carry out scheduled maintenance, the system will temporarily shut down during the time designated below. We apologize for the inconvenience, and thank you for your understanding.

This maintenance was previously announced as the second of two operations and was originally scheduled for November, but has been moved back.

mackerel.io

Scheduled date and time

  • December 2, 2020 (Wednesday) 1:00pm - 6:00pm (JST)
  • The system will be temporarily shut down within the above time period
  • This time period was set in anticipation of handling issues as they occur. If everything proceeds normally, this time period is expected to end earlier than scheduled

Maintenance subject

Areas affected during maintenance period

  • The scheduled time frame is the longest case scenario and will end as soon as the work has been completed
    • Once maintenance work has begun, the entire Mackerel system will shut down for a short period of time
  • Web access to Mackerel, data posting by the agent, API access (including the CLI tool), alert notifications, etc. will be unavailable during this time
  • As soon as the scheduled maintenance has been completed and working operation has been confirmed, the system shutdown will end and an announcement will be made
  • As for mackerel-agent metric posting, data will be buffered from mackerel-agent during the maintenance period and resent after maintenance has been completed
    • If resent properly, graphs during the maintenance period will also be displayed
    • Please note that if mackerel-agent is stopped or restarted before resending is complete, buffering data will be lost
    • Trying to start up mackerel-agent during maintenance will fail because communication with Mackerel cannot be made
    • Trying to transmit to mackerel during maintenance will fail and an error message will be output in the error log
    • If metric thresholds are exceeded during the maintenance time period, alerts will not be issued retroactively following maintenance completion
    • Check monitoring alerts will be issued for up to 6 hours for events detected during the maintenance time period
  • External monitoring and cloud integration graphs may be incomplete during the maintenance period

Why we chose this time schedule

Due to the fact that the Mackerel service is relied upon for system monitoring, we came to the conclusion that the best time for a temporary shutdown would be during the day when the majority of Mackerel users could be available to respond if necessary, rather than in the middle of the night. For this reason, we’ve scheduled the maintenance to take place during a weekday in Japan Standard Time.

On the other hand, we understand that it might be difficult to perform server operations and deploy applications during the daytime and we sincerely apologize for this inconvenience.

Consideration for our users is essential in determining an appropriate maintenance timeframe and something that we want to continue to improve. If you have any feedback, please let us know.

Announcements

Situation reports will be made on the Mackerel status page (http://status.mackerel.io).

We will also make periodical updates on our official Twitter account (https://twitter.com/mackerelio_jp).

Related inquiries

Please send all inquiries regarding this matter to support@mackerel.io using the forms provided below.

Again, thank you for your understanding and cooperation.

Newly obtainable metrics added for Azure / SQL Database Integration and more

Hello. Mackerel team CRE Inoue (id:a-know) here with another update announcement.

Newly obtainable metrics added for Azure / SQL Database Integration

5 new obtainable metrics have been added and they are as follows.

  • SQL Server process core percent
    • azure.sql_database.sqlserver_process_core_percent.percent
  • SQL Server process memory percent
    • azure.sql_database.sqlserver_process_memory_percent.percent
  • Tempdb Data File Size
    • azure.sql_database.tempdb_data_size.data_size
  • Tempdb Log File Size
    • azure.sql_database.tempdb_log_size.log_size
  • Tempdb Percent Log Used
    • azure.sql_database.tempdb_log_used_percent.percent

Be sure to check out the documentation linked below as well.

mackerel.io

Spec changes made related to posting time and posting interval for check monitoring results.

A lot of our users are using plugins to perform check monitoring for Mackerel.

mackerel.io

These check monitoring results are sent to Mackerel together with a timestamp of when the check process was performed, then an alert is issued according to those results. Up until now, this was handled the same way regardless of how much time had passed. Now, specifications have been changed so that this function is only available up to 6 hours.

In addition to this, you also have maxCheckAttemps, which sets the maximum number of attempts as a mackerel-agent option for check monitoring. However, specifications have been changed so that the range for evaluating this is within 6 hours. (The previous range was within 1 hour.)

Mackerel’s own Watanabe will be presenting at an event on October 28th (Wednesday)!

Mr. Watanabe, Mackerel's very own Product Manager, is scheduled to present at an event titled "Bringing agility to business with SaaS + Serverless" hosted by Amazon Web Services Japan K.K.

aws-serverless.connpass.com (Japanese only)

The theme of this event focuses on Amazon EventBridge (which was recently supported by Mackerel) and Mr. Watanabe will be talking about how to best use Mackerel in serverless environments. Please consider joining us!

Release of check-aws-cloudwatch-logs-insights (beta version)

Hello. Mackerel team’s id:astj here.

Today we are releasing a beta version of check-aws-cloudwatch-logs-insights on GitHub. check-aws-cloudwatch-logs-insights is a check plugin that monitors logs in Amazon CloudWatch Logs. This plugin was created so that high-flow log groups could be monitored utilizing the CloudWatch Logs Insights API.

github.com

Some of you might already be familiar with check-aws-cloudwatch-logs *1, which is a pre-existing official Mackerel check plugin used to monitor logs in Amazon CloudWatch Logs. This plugin works well for low-flow log groups, but struggles when targeting high-flow groups. In some cases, a timeout may occur and inhibit monitoring altogether. check-aws-cloudwatch-logs-insights is intended to enable log monitoring for high capacity log groups using the CloudWatch Logs Insights API.

How to use

Follow the process described below on how to use check-aws-cloudwatch-logs-insights.

Getting the plugin

The plugin can be obtained by using mkr plugin install or by downloading it directly from GitHub. When using mkr plugin install, the following command will extract the v0.0.2 executable file under /opt/mackerel-agent/plugins/bin/. (check the latest version on GitHub)

sudo mkr plugin install mackerelio-labs/check-aws-cloudwatch-logs-insights@v0.0.2

You can also download the executable file from GitHub Releases. Be sure to download the version that matches your OS / architecture.

Releases · mackerelio-labs/check-aws-cloudwatch-logs-insights · GitHub

Using commands

See the README linked below for a more detailed writeup.

github.com

The query commands in CloudWatch Logs Insights Query Syntax - Amazon CloudWatch Logs can be used with the --filter option to filter logs.

And because this plugin uses the AWS API, you’ll need to make sure that it can access the credentials of IAM users / roles that can execute the following 3 actions for the target log group: logs:GetQueryResults logs:StartQuery logs:StopQuery.

The use of EC2 instance profiles, named profiles with the environment variable AWS_PROFILE, and direct access key specification with the environment variables AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY are also supported.

Here are a few examples.

The example below returns a WARNING status when one or more logs containing the character string "ERROR" are found in the log stream /aws/lambda/some-lambda-function, and CRITICAL when 10 or more logs are found.

% AWS_REGION=ap-northeast-1 /path/to/check-aws-cloudwatch-logs-insights --log-group-name=/aws/lambda/some-lambda-function --filter='filter @message like /ERROR/' -warning-over 1 -critical-over 10

This is what it looks like when writing the same monitoring configuration as above in the mackerel-agent configuration file.

[plugin.checks.aws-cloudwatch-logs-insights-sample]
command = ["/path/to/check-aws-cloudwatch-logs-insights", "--log-group-name", "/aws/lambda/some-lambda-function", "--filter", "filter @message like /ERROR/", "--critical-over", "10", "--warning-over", "1"]
env = { AWS_REGION = "ap-northeast-1" }

You can also utilize the CloudWatch Logs Insights query syntax for even more advanced filtering. The next example shows using the level key in JSON format to target any string containing "error".

% AWS_REGION=ap-northeast-1 /path/to/check-aws-cloudwatch-logs-insights --log-group-name=... --filter='filter level = "error"' ...

The difference between check-aws-cloudwatch-logs

As previously mentioned, this plugin was developed to address the issue of unreliability when using check-aws-cloudwatch-logs to monitor high-flow log groups. Now we’ll compare the two and point out the differences and restrictions between this plugin and check-aws-cloudwatch-logs.

  • API requests for CloudWatch Logs Insights incur costs in AWS
  • Insights is less real-time than check-aws-cloudwatch-logs
  • Different expressions are used for filtering logs

API requests for CloudWatch Logs Insights incur costs in AWS

The original check-aws-cloudwatch-logs uses CloudWatch Logs' FilterLogEvents API which does not incur costs in AWS. However, check-aws-cloudwatch-logs-insights uses CloudWatch Logs Insights which incurs costs for the amount of scanned log data. See the AWS page linked below for more details regarding costs. (As of October 2020 in the Tokyo region, 1GB costs 0.005 USD)

aws.amazon.com

Insights is less real-time than check-aws-cloudwatch-logs

Due to the specifications of the CloudWatch Logs API, this plugin monitors logs up to 5 minutes prior the current time. check-aws-cloudwatch-logs is not subject to this limitation, allowing for more real-time monitoring.

Different expression are used for filtering logs

check-aws-cloudwatch-logs uses the CloudWatch Logs FilterLogEvents API which has its own syntax for writing filter conditions.

docs.aws.amazon.com

As mentioned in the ‘Using commands’ section, check-aws-cloudwatch-logs-insights uses the CloudWatch Logs Insights API which requires a different syntax.

docs.aws.amazon.com

Notes / requests

This plugin is currently in beta. We are publishing it as is for the time being and we plan on making improvements based on user feedback. If you have any requests regarding the specifications of this plugin, or regarding CloudWatch Logs monitoring (not limited to this plugin), we would love to hear from you. To send feedback, you can use an issue in the GitHub repository or contact us via Mackerel Support or using the Mackerel User Group channel #check-aws-cloudwatch-logs-insights on Slack.

support.mackerel.io

You can join the Mackerel User Group with this URL.

mackerel-ug-slackin.herokuapp.com

When we release the official version of this plugin in the future, we will also publish an official usage guide.

In addition to this, the specifications of this plugin are subject to change without notice during this beta period. Please check the GitHub repository for the latest information.

Give it a try

We are super excited about the beta release of this new plugin. If you're interested in monitoring logs on Amazon CloudWatch Logs, particularly those with high-flow, we definitely recommend giving this plugin a try.

About mackerelio-labs

Many of the OSS repositories that Mackerel publishes such as mackerel-agent, can be found at https://github.com/mackerelio/. However, the Mackerel team also publishes experimental products and those still under-development at https://github.com/mackerelio-labs/. check-aws-cloudwatch-logs-insights is one of these products. When we release the official version in the future, we will release it in the mackerelio organization.

Newly obtainable metrics added for AWS / DynamoDB Integration and more

Hello. Mackerel team CRE Inoue ( id:a-know ) here with another update announcement.

Newly obtainable metrics added for AWS / DynamoDB Integration

Two new metrics related to DynamoDB transaction conflict have been added. These metrics follow CloudWatch API metric support.

  • TransactionConflict
    • dynamodb.transaction_conflict.item_level
    • dynamodb.transaction_conflict.request_level

Support for API names with non-ASCII characters in AWS / API Gateway Integration

Up until now, integration with Mackerel wasn’t possible If the target API Gateway name contained non-ASCII characters. With this release, the characters will now be replaced as described below and integration is now possible.

  • If non-ASCII characters are included along with ASCII characters, the non-ASCII characters will be converted to -.
  • If composed of only non-ASCII characters, it will be set as the API ID.

Check out the help page linked below as well.

mackerel.io

OSS Updates

Changes are as follows.

mackerel-check-plugins

  • [check-http] New options added such as -m ( --method ) and -d ( --body ). See here for more details.
  • [check-load] A bug was fixed that caused the metric value and the command output to differ when using the --percpu option.

Thank you to everyone who contributed!

Check out the help page linked below for more on mackerel-check-plugins.

mackerel.io

Google Cloud Integration Release!

Hello. Mackerel team CRE Inoue (id:a-know) here. Last Thursday was one that should have been commemorated. The 17th marked the sixth year of Mackerel’s official release.

And to celebrate... well not really, but we’ve got a big release announcement.

The long-awaited Google Cloud Integration has been released! It’s available for use starting today!

mackerel.io

Google Cloud Integration Release

You can configure Google Cloud Integration from the ‘Google Cloud Integration’ tab in your organization’s details page.

In order to use Google Cloud Integration, you’ll need to set up a service account. You can refer to the help page linked below for more details.

mackerel.io

With today's release, you can now integrate and monitor the following 3 Google Cloud Platform services.

Check out the help pages linked below to see which metrics are available for each service.

mackerel.io

mackerel.io

mackerel.io

We plan on expanding the list of services in the future. And in order to do this, we believe that feedback / requests from our users is essential to the development process. So please feel free to contact us via the inquiry form linked below!

support.mackerel.io

The system will be temporarily shut down for scheduled maintenance in October and November

Thank you for choosing Mackerel.

As stated in the title, the system is scheduled to be temporarily shut down in order to carry out maintenance. The maintenance has been broken up into two intervals and the time schedule is laid out below. We apologize for the inconvenience, and thank you for your understanding.

October shutdown schedule

Date and time

  • October 14, 2020 (Wednesday) 2:30pm -4:30pm (JST)
    • The system will be temporarily shut down within the above time period

※This time window is the longest case scenario. The system will restart as soon as maintenance is complete.

Content

  • Migration of Redis to Amazon ElastiCache

Areas affected during maintenance period

  • The scheduled time frame is the longest case scenario and will end as soon as the work has been completed
    • Once maintenance work has begun, the entire Mackerel system will shut down for a short period of time
  • Web access to Mackerel, data posting by the agent, API access (including the CLI tool), alert notifications, etc. will be unavailable during this time
  • As soon as the scheduled maintenance has been completed and working operation has been confirmed, the system shutdown will end and an announcement will be made
  • As for mackerel-agent metric posting, data will be buffered from mackerel-agent during the maintenance period and resent after maintenance has been completed
    • If resent properly, graphs during the maintenance period will also be displayed
    • Please note that if mackerel-agent is stopped or restarted before resending is complete, buffering data will be lost
    • Trying to start up mackerel-agent during maintenance will fail because communication with Mackerel cannot be made
    • Trying to transmit to mackerel during maintenance will fail and an error message will be output in the error log
  • External monitoring and cloud integration graphs may be incomplete during the maintenance period

November shutdown schedule

Date and time

  • November 25, 2020 (Wednesday) (JST)

※ The time will be announced one month before the scheduled date. Similar to the maintenance in October, we expect to carry out this maintenance during daytime hours.

Content

Areas affected during maintenance period

  • The areas affected during November’s maintenance are the same as that of October.

Why we chose this time schedule

Due to the fact that users rely upon the Mackerel service for system monitoring, we came to the conclusion that the best time for a temporary shutdown is during the day when the majority of users are available to respond if necessary, rather than in the middle of the night. For this reason, we’ve scheduled the maintenance to take place during a weekday in Japan Standard Time.

On the other hand, we understand that it will be difficult to perform server operations and deploy applications during the daytime and we sincerely apologize for this inconvenience.

Consideration for our users is essential in determining an appropriate maintenance timeframe and something that we want to continue to improve. If you have any feedback, please let us know.

Announcements

Situation reports will be made on the Mackerel status page (http://status.mackerel.io).

We will also make periodical updates on our official Twitter account (https://twitter.com/mackerelio_jp).

Related inquiries

Please send all inquiries regarding this matter to support@mackerel.io using the forms provided below.

Again, thank you for your understanding and cooperation.

Newly obtainable metrics added for AWS / ElastiCache Integration and more

Hello. Mackerel team CRE Inoue ( id:a-know) here with another update announcement.

Newly obtainable metrics added for AWS / ElastiCache Integration

These specification changes were announced on this blog a couple of weeks ago, and as scheduled, were released today the 7th. Check out the blog post linked below for details regarding the content.

mackerel.io

Graph image URL added to the Webhook notification payload

Alerts that are detected/opened with Mackerel can be linked to various chat services and incident management tools by configuring notification channels. Webhook is one of the notification channels supported by Mackerel.

mackerel.io

This mechanism lets you make a POST request for an arbitrary URL using JSON formatted information of the detected alert as the payload.

With this update, the graph image URL imageUrl is now included in the payload. If you’ve customized your notifications with our Webhook notification channel, we hope this helps you make your content even richer!