Hello. Mackerel team’s 
id:astj here.
Today we are releasing a beta version of check-aws-cloudwatch-logs-insights on GitHub. check-aws-cloudwatch-logs-insights is a check plugin that monitors logs in Amazon CloudWatch Logs. This plugin was created so that high-flow log groups could be monitored utilizing the CloudWatch Logs Insights API.
Some of you might already be familiar with check-aws-cloudwatch-logs *1, which is a pre-existing official Mackerel check plugin used to monitor logs in Amazon CloudWatch Logs. This plugin works well for low-flow log groups, but struggles when targeting high-flow groups. In some cases, a timeout may occur and inhibit monitoring altogether. check-aws-cloudwatch-logs-insights is intended to enable log monitoring for high capacity log groups using the CloudWatch Logs Insights API.
How to use
Follow the process described below on how to use check-aws-cloudwatch-logs-insights.
Getting the plugin
The plugin can be obtained by using mkr plugin install or by downloading it directly from GitHub.
When using mkr plugin install, the following command will extract the v0.0.2 executable file under /opt/mackerel-agent/plugins/bin/. (check the latest version on GitHub)
sudo mkr plugin install mackerelio-labs/check-aws-cloudwatch-logs-insights@v0.0.2
You can also download the executable file from GitHub Releases. Be sure to download the version that matches your OS / architecture.
Releases · mackerelio-labs/check-aws-cloudwatch-logs-insights · GitHub
Using commands
See the README linked below for a more detailed writeup.
The query commands in CloudWatch Logs Insights Query Syntax - Amazon CloudWatch Logs can be used with the --filter option to filter logs.
And because this plugin uses the AWS API, you’ll need to make sure that it can access the credentials of IAM users / roles that can execute the following 3 actions for the target log group: logs:GetQueryResults logs:StartQuery logs:StopQuery.
The use of EC2 instance profiles, named profiles with the environment variable AWS_PROFILE, and direct access key specification with the environment variables AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY are also supported.
Here are a few examples.
The example below returns a WARNING status when one or more logs containing the character string "ERROR" are found in the log stream /aws/lambda/some-lambda-function, and CRITICAL when 10 or more logs are found.
% AWS_REGION=ap-northeast-1 /path/to/check-aws-cloudwatch-logs-insights --log-group-name=/aws/lambda/some-lambda-function --filter='filter @message like /ERROR/' -warning-over 1 -critical-over 10
This is what it looks like when writing the same monitoring configuration as above in the mackerel-agent configuration file.
[plugin.checks.aws-cloudwatch-logs-insights-sample]
command = ["/path/to/check-aws-cloudwatch-logs-insights", "--log-group-name", "/aws/lambda/some-lambda-function", "--filter", "filter @message like /ERROR/", "--critical-over", "10", "--warning-over", "1"]
env = { AWS_REGION = "ap-northeast-1" }
You can also utilize the CloudWatch Logs Insights query syntax for even more advanced filtering. The next example shows using the level key in JSON format to target any string containing "error".
% AWS_REGION=ap-northeast-1 /path/to/check-aws-cloudwatch-logs-insights --log-group-name=... --filter='filter level = "error"' ...
The difference between check-aws-cloudwatch-logs
As previously mentioned, this plugin was developed to address the issue of unreliability when using check-aws-cloudwatch-logs to monitor high-flow log groups. Now we’ll compare the two and point out the differences and restrictions between this plugin and check-aws-cloudwatch-logs.
- API requests for CloudWatch Logs Insights incur costs in AWS
- Insights is less real-time than check-aws-cloudwatch-logs
- Different expressions are used for filtering logs
API requests for CloudWatch Logs Insights incur costs in AWS
The original check-aws-cloudwatch-logs uses CloudWatch Logs' FilterLogEvents API which does not incur costs in AWS. However, check-aws-cloudwatch-logs-insights uses CloudWatch Logs Insights which incurs costs for the amount of scanned log data. See the AWS page linked below for more details regarding costs. (As of October 2020 in the Tokyo region, 1GB costs 0.005 USD)
Insights is less real-time than check-aws-cloudwatch-logs
Due to the specifications of the CloudWatch Logs API, this plugin monitors logs up to 5 minutes prior the current time. check-aws-cloudwatch-logs is not subject to this limitation, allowing for more real-time monitoring.
Different expression are used for filtering logs
check-aws-cloudwatch-logs uses the CloudWatch Logs FilterLogEvents API which has its own syntax for writing filter conditions.
As mentioned in the ‘Using commands’ section, check-aws-cloudwatch-logs-insights uses the CloudWatch Logs Insights API which requires a different syntax.
Notes / requests
This plugin is currently in beta. We are publishing it as is for the time being and we plan on making improvements based on user feedback. If you have any requests regarding the specifications of this plugin, or regarding CloudWatch Logs monitoring (not limited to this plugin), we would love to hear from you. To send feedback, you can use an issue in the GitHub repository or contact us via Mackerel Support or using the Mackerel User Group channel #check-aws-cloudwatch-logs-insights on Slack.
You can join the Mackerel User Group with this URL.
mackerel-ug-slackin.herokuapp.com
When we release the official version of this plugin in the future, we will also publish an official usage guide.
In addition to this, the specifications of this plugin are subject to change without notice during this beta period. Please check the GitHub repository for the latest information.
Give it a try
We are super excited about the beta release of this new plugin. If you're interested in monitoring logs on Amazon CloudWatch Logs, particularly those with high-flow, we definitely recommend giving this plugin a try.
About mackerelio-labs
Many of the OSS repositories that Mackerel publishes such as mackerel-agent, can be found at https://github.com/mackerelio/. However, the Mackerel team also publishes experimental products and those still under-development at https://github.com/mackerelio-labs/. check-aws-cloudwatch-logs-insights is one of these products. When we release the official version in the future, we will release it in the mackerelio organization.
