check-aws-cloudwatch-logs-insights is now officially available

Hello. Mackerel team’s id:astj here.

Last October, we had released a beta version of check-aws-cloudwatch-logs-insights, a new check monitoring plugin to monitor logs on Amazon CloudWatch Logs. We would like to take this opportunity to thank all the users who have tried the beta version.

Today, we are pleased to announce the official release of this plugin. Compared to v0.0.2, which was released as the beta version, there are no changes in the internal implementation, but the installation method using mkr plugin install has been partially changed. The upgrade procedure from the beta version to the official version is also described in this article.

Although there is some overlap with the content of the article published when the beta version was released, the following is a description of the plugin and how to install it. We hope you will read this article whether you have been using the beta version or are considering using it for the first time.

Some of you might already be familiar with check-aws-cloudwatch-logs *1, which is a pre-existing official Mackerel check plugin used to monitor logs in Amazon CloudWatch Logs. This plugin works well for low-flow log groups, but struggles when targeting high-flow groups. In some cases, a timeout may occur and inhibit monitoring altogether. check-aws-cloudwatch-logs-insights is intended to enable log monitoring for high capacity log groups using the CloudWatch Logs Insights API.

How to use

Follow the process described below on how to use check-aws-cloudwatch-logs-insights.

Getting the plugin

The plugin can be obtained by using mkr plugin install or by downloading it directly from GitHub. When using mkr plugin install, the following command will extract the latest executable file under /opt/mackerel-agent/plugins/bin/. (check the latest version on GitHub)

sudo mkr plugin install check-aws-cloudwatch-logs-insights

You can also download the executable file from GitHub Releases. Be sure to download the version that matches your OS / architecture.

Releases · mackerelio/check-aws-cloudwatch-logs-insights · GitHub

Upgrading from the beta version to the official version

If you have already installed the beta version and want to upgrade to the official version, you can upgrade by following the steps below. Note that unlike the normal installation, you need to run mkr plugin install with the --overwrite option.

sudo mkr plugin install --overwrite check-aws-cloudwatch-logs-insights

Using commands

See the README linked below for a more detailed writeup.

The query commands in CloudWatch Logs Insights query syntax - Amazon CloudWatch Logs can be used with the --filter option to filter logs.

And because this plugin uses the AWS API, you’ll need to make sure that it can access the credentials of IAM users / roles that can execute the following 3 actions for the target log group: logs:GetQueryResults logs:StartQuery logs:StopQuery.

The use of EC2 instance profiles, named profiles with the environment variable AWS_PROFILE, and direct access key specification with the environment variables AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY are all supported.

Here are a few examples.

The example below returns a WARNING status when one or more logs containing the character string "ERROR" are found in the log stream /aws/lambda/some-lambda-function, and CRITICAL when 10 or more logs are found.

% AWS_REGION=ap-northeast-1 /path/to/check-aws-cloudwatch-logs-insights --log-group-name=/aws/lambda/some-lambda-function --filter='filter @message like /ERROR/' -warning-over 1 -critical-over 10

This is what it looks like when writing the same monitoring configuration as above in the mackerel-agent configuration file.

command = ["/path/to/check-aws-cloudwatch-logs-insights", "--log-group-name", "/aws/lambda/some-lambda-function", "--filter", "filter @message like /ERROR/", "--critical-over", "10", "--warning-over", "1"]
env = { AWS_REGION = "ap-northeast-1" }

You can also utilize the CloudWatch Logs Insights query syntax for even more advanced filtering. The next example shows using the level key in JSON format to target any string containing "error".

% AWS_REGION=ap-northeast-1 /path/to/check-aws-cloudwatch-logs-insights --log-group-name=... --filter='filter level = "error"' ...

The difference between check-aws-cloudwatch-logs

As previously mentioned, this plugin was developed to address the issue of unreliability when using check-aws-cloudwatch-logs to monitor high-flow log groups. Now we’ll compare the two and point out the differences and restrictions between this plugin and check-aws-cloudwatch-logs.

  • API requests for CloudWatch Logs Insights incur costs in AWS
  • Insights is less real-time than check-aws-cloudwatch-logs
  • Different expressions are used for filtering logs

API requests for CloudWatch Logs Insights incur costs in AWS

The original check-aws-cloudwatch-logs uses CloudWatch Logs' FilterLogEvents API which does not incur costs in AWS. However, check-aws-cloudwatch-logs-insights uses CloudWatch Logs Insights which incurs costs for the amount of scanned log data. See the AWS page linked below for more details regarding costs. (As of June 2021 in the Tokyo region, 1GB costs 0.0076 USD)

Insights is less real-time than check-aws-cloudwatch-logs

Due to the specifications of the CloudWatch Logs API, this plugin monitors logs up to 5 minutes prior the current time. check-aws-cloudwatch-logs is not subject to this limitation, allowing for more real-time monitoring.

Different expression are used for filtering logs

check-aws-cloudwatch-logs uses the CloudWatch Logs FilterLogEvents API which has its own syntax for writing filter conditions.

As mentioned in the ‘Using commands’ section, check-aws-cloudwatch-logs-insights uses the CloudWatch Logs Insights API which requires a different syntax.

Give it a try

We are very excited about the official release of this new plugin. If you're interested in monitoring logs on Amazon CloudWatch Logs, particularly those with high-flow, we definitely recommend giving this plugin a try.