Mackerel blog #mackerelio

The Official Blog of Mackerel

【Important】Scheduled maintenance in early August

Dear Mackerel users,

Thank you for your continued support. I’m id:sugiyama88, Mackerel’s project manager.

Thanks to all of you, Mackerel has grown a lot. And in order to prepare for even further leaps forward in the future, we have decided to schedule some maintenance. The main objective of this maintenance will be performance and feature expansions of the time series database.

At the time of scheduled maintenance, we will put maximum effort into minimizing the impact as much as possible. We are making this announcement in advance because users may be affected. Additionally, the maintenance is scheduled to be divided into two stages.

This is an important measure in order to be able to continually offer a better service and we thank you for your understanding and cooperation.

Maintenance schedule and effects

Maintenance #1

August 7th, 2017 (Monday) 2:30 p.m. - 4:30 p.m. (JST)

  • Access to “mackerel.io” will be unstable during the above mentioned time period
    • Actions such as accessing Mackerel via the Web, Mackerel agent data posting, API access (including the CLI tool), and alert notifications may become unavailable
    • Temporary buffering and resending are expected regarding metric postings from mackerel-agent

Maintenance #2

Scheduled for mid-September 2017 (a separate announcement will be made)

The source IP address used for URL external monitoring, AWS integration, Azure integration, and Mackerel notification channels such as Webhook, etc. will change. There will be no change concerning mail notifications.

Future announcement plans

Notices will be made according the the schedule below, but please be aware that some fluctuation is possible depending on the circumstances. Announcements will be re-made regardless of change.

Maintenance #1

  1. 7/11 (Tuesday) An announcement will be made via blog and newsletter (1 month in advance)
  2. 7/24 (Monday) An announcement will be made via newsletter(2 weeks in advance)
  3. 7/31 (Monday) An announcement will be made via newsletter(1 week in advance)
  4. 8/4 (Friday) An announcement will be made via newsletter (3 days in advance)
  5. 8/7 (Monday) An announcement will be made before and after maintenance via newsletter and Status.io (status.mackerel.io)

We will keep you posted from our blog as soon as updated information is available.

Maintenance #2

A separate announcement will be made.

Changes and requests

Maintenance #1

The IP address of mackerel.io will change. For customers who are subject to IP address restrictions, add the new IP address in addition to the current IP address to the transmission permission target by August 7th (Monday).

※Due to a possibility of roll backs after maintenance completion, we suggest allowing the current IP address to remain for a few months. We will make a separate announcement regarding deletion of the current IP address.

Current IP address

  • 59.106.194.79/32
  • 59.106.194.84/30

New IP address

  • 13.113.31.156/32
  • 54.92.89.142/32
  • 52.196.35.56/32
  • 52.69.239.167/32
  • 52.198.226.45/32
  • 52.198.218.152/32
  • 13.113.245.173/32
  • 52.199.207.168/32
  • 52.193.231.228/32

Maintenance #2

The source IP address used for URL external monitoring, AWS integration, Azure integration, and Mackerel notification channels such as Webhook, etc. will change. There will be no change concerning mail notifications.

For customers who are subject to IP address restrictions, add the new IP address in addition to the current IP address to the transmission permission target by the day of maintenance (A separate announcement will be made).

※Due to a possibility of roll backs after maintenance completion, we suggest allowing the current IP address to remain for a few months. We will make a separate announcement regarding deletion of the current IP address.

In addition, since some variation is expected in URL external monitoring HTTP response times, users with strict settings should adjust threshold values after maintenance.

Current IP address

  • 59.106.108.64/26

New IP address

  • 52.193.111.118/32
  • 52.196.125.133/32
  • 13.113.213.40/32
  • 52.197.186.229/32
  • 52.198.79.40/32
  • 13.114.12.29/32
  • 13.113.240.89/32
  • 52.68.245.9/32
  • 13.112.142.176/32

For any inquiries concerning this matter, contact us at support@mackerel.io.

Thank you for choosing Mackerel.

AWS Integration for RDS now supports Amazon Aurora etc.

Today is Tanabata, a day celebrated on July 7th here in Japan. Also referred to as the Star Festival, children and adults hang written wishes on bamboo branches and look up to the stars in hopes that their wish comes true. It might be good to look up at the night sky this evening and make a wish.

Also, a typhoon hit islands off the coast of Japan this week. Looks like we’re about to enter typhoon season, so everyone stay safe!

Anyways, here is this week’s update information.

AWS Integration for RDS now supports Amazon Aurora

AWS Integration now supports Amazon Aurora as a metric target of RDS Integration.

aws.amazon.com

Since its introduction, this feature has been highly requested. Sorry to keep you all waiting!

Check out the list of obtainable metric items in the help page below.

mackerel.io

Mackerel Drink Up #5 Tokyo!

Mackerel Drink Up is an official event where users are invited to come and interact with Mackerel staff and users from other companies while enjoying good food and drinks. Mackerel Drink Up #5 Tokyo will be held on Wednesday, July 19th (JST)!

mackerelio.connpass.com (Japanese only)

Many members of the Mackerel team will be there, including the director, producer, and myself. This is a good opportunity to ask all your Mackerel related questions, so by all means, join us!

Updates for mackerel-agent-plugins and fluent-plugin-mackerel

Hello. Mackerel sales engineer Inoue (id:a-know) here.

Tomorrow marks the beginning of July, and just like that, half the year has already come and gone. How about this weather though? The seasonal wave of humidity is definitely upon us. Food spoils easily this time of year so be careful everyone.

Anyways, here is this week’s update information.

mackerel-agent-plugins updates

mackerel-agent-plugins has been updated and is now version 0.28.1. There is quite a lot of new content, so we’ll go over this topic below.

mackerel-plugin-accesslog has been added

mackerel-plugin-accesslog has been added to the mackerel-agent-plugins package. By using this, it is possible to classify and aggregate the access logs for each status code output in Apache log format or LTSV format, and visualize the breakdown.

In Posting Service Metrics with fluentd - Mackerel Docs, the example “Post the number of requests for each status code from Nginx’s access log to service metrics” uses fluentd and fluent-plugin-mackerel. Aggregation like this can easily be done now with this plugin. By all means, try it out!

mackerel-plugin-postgres now supports PostgreSQL 9.6

Updates were made for mackerel-plugin-postgres and PostgreSQL 9.6 is now supported. To everyone who submitted requests, sorry to keep you waiting!

Other updates

  • [postgres] metric-key-prefix option supported
  • [aws-dynamodb] package added
  • Other minor adjustments

fluent-plugin-mackerel has been updated

fluent-plugin-mackerel has been updated to version 1.0.0.

github.com

fluentd v.0.14.x versions are now supported with this update. In other words, since the latest version of fluent-plugin-mackerel is not available for earlier fluentd versions, either update your version of fluentd or stay with fluentd-plugin-mackerel version 0.1.3.

A huge thank you to everyone who contributed to mackerel-agent-plugins and fluent-plugin-mackerel!!

Azure Integration release

Hello, Sales engineer Inoue (id:a-know) here.

I usually work in our Tokyo office and Japan has been in the rainy season for a short while now, but compared to a normal year, I feel like we’ve had less rainy days and more overcast ones. I recently learned that in forecast terminology, a rainy season like this is referred to as a “negative rainy season”.

I don’t really do well in the heat, so if the temperature stays cool, I’m fine with it.

Anyways, here is this week’s update information.

Azure Integration release

We announced its development at Mackerel Meetup #9, but now Azure Integration has been released.

Mackerel is a push-type server monitoring service that collects and monitors metrics usually by installing agent software (mackerel-agent), but this agent cannot be installed for cloud platform managed services.

Mackerel’s integration feature is used for aggregating information of such a service, and this time, Azure SQL Database is the target. Of course, we plan to expand this target in the future.

azure.microsoft.com

Each Azure cloud product will be registered as one host in Mackerel and therefore be counted as a billable host. Additionally, the API of Azure will be called every 5 minutes for each targeted metric to be obtained. Please take note, for this reason, an Azure Monitor API usage fee may occur.

Metrics obtained with Azure integration are handled the same way as the host metrics obtained by the agent and monitor configurations are also possible.

You can setup the Azure integration here. In addition, we’ve prepared a detailed help page in regards to the setup process, so definitely check that out (this function is only offered in the Trial plan and Standard plan).

mackerel.io

“Critical alerts only” now possible for notification groups etc.

Hello. Development team director id:daiksy here.

Recently, I opened my window before bed expecting a warm night, but was surprised by a very chilly morning. This kind of weather makes controlling my physical condition pretty difficult. Is everyone out there staying healthy?

Here is this week’s update.

“Critical alerts only” now possible for notification groups

For some time we’ve been receiving requests to be able to change notification destinations and to have only urgent Critical alerts be sent to chats when notifications of monitoring results are made.

In this week’s release, we added the item “notification level” to the notification group’s setting screen. When the Critical only is toggled on, alerts to the notification group will only be sent under the following conditions and Warning alerts will not be sent.

  • Notifications will be sent for error levels of Critical or higher (Critical, Unknown)
  • OK notifications will be sent for alerts that have previously exceeded Critical

By combining notification groups with limited notification levels, notification destinations can flexibly be set according to the notification level.

Check out the help page as well. mackerel.io

Notice for new mackerel-agent repositories

New package repositories have been released for yum and apt targeted for the new Linux distribution. Check out the link below for more information.

mackerel.io

Releases for mackerel-agent and mackerel-agent-plugins

In a previous post, we announced the support of HTTP/2 in mackerel-agent, but due to reports of mackerel-agent not running stably with some network devices, we released a version of mackerel-agent with the HTTP/2 support reverted.

Other changes including detailed content are as follows.

  • mackerel-agent 0.43.2

    • HTTP/2 support revert
  • mackerel-agent-plugins 0.28.0

    • Addition of mackerel-plugin-solr to package 
    • Addition of mackerel-plugin-aws-dynamodb (not packaged)
    • Addition of mackerel-plugin-redash (not packaged)

Notice for mackerel-agent’s new package repositories for yum and apt

We recently released new package repositories for yum and apt targeted for the new Linux distribution. We will continue to maintain existing package repositories, so there is no need to immediately change your existing environment. Continue below for detailed installation methods for each.

https://mackerel.io/my/instruction-agent

In this entry, the existing package repository will be referred to as “v1” and the new repository as “v2”.

Regarding supported distributions

The supported distributions of v1, v2 are as follows. Amazon Linux has its own repository, so this change will not affect it.

Repository Supported distributions
yum v1 CentOS 6/7
yum v2 CentOS 7 or later
apt v1 Debian 7/8, Ubuntu 14.04/16.04
apt v2 Debian 8 or later、Ubuntu 16.04 or later

v2 repository features

The v2 repository has the following features.

  • Systemd direct support
  • Drastic volume reduction for plugin packages
  • gpg signature for SHA256
  • Only offered for 64bit

Systemd direct support

We got rid of the SysVinit format startup file and now support Systemd directly. This will make running mackerel-agent more stable.

Additionally, logs are now output to Journal. You can check the mackerel-agent log with the following command.

sudo journalctl -u mackerel-agent.service

Drastic volume reduction for plugin packages

The build method was revised and the file sizes for the mackerel-agent-plugins and mackerel-check-plugins packages have been reduced to about 1/10th of the size. The current versions are as follows.

Package v1 v2
mackerel-agent-plugins(v0.28.0) 56.6MB 4.23MB
mackerel-check-plugins(v0.10.4) 20.5MB 2.75MB

How to update the package repository from v1 to v2

If you would like to switch to an environment where mackerel-agent etc. is already installed to the v2 repository, you’ll first need to delete the package installed with the v1 repository, delete the v1 repository configuration body, then set up the v2 repository.

The procedure for each deletion is provided below.

Before deleting anything, we recommend that you backup the configuration file (/etc/mackerel-agent/mackerel-agent.conf, / etc / sysconfig / mackerel-agent, / etc / default / mackerel-agent) just in case.

yum v1 repository removal

sudo yum remove -y mackerel-agent mackerel-agent-plugins mackerel-check-plugins mkr
sudo rm /etc/yum.repos.d/mackerel.repo
sudo yum clean all

After completing this procedure, the configuration files will be renamed and saved as /etc/mackerel-agent/mackerel-agent.conf.rpmsave /etc/sysconfig/mackerel-agent.rpmsave.

apt v1 repository removal

- sudo apt-get remove -y mackerel-agent mackerel-agent-plugins mackerel-check-plugins mkr
- sudo rm /etc/apt/sources.list.d/mackerel.list
- sudo apt-key del C2B48821
- sudo apt-get clean

The configuration file will not be deleted by performing this procedure.

How to install the v2 repository

After completing the above procedures, install mackerel-agent. Please refer to the following link for installation instructions.

https://mackerel.io/my/instruction-agent

Regarding the Provisioning Toolchain

We will be supporting the officially offered Chef’s cookbook and Ansible’s Role in sequential order.

Specifying multiple conditions in log monitoring is now easier with check-log etc.

Last week from May 31st to June 2nd, Mackerel ran a booth at the AWS Summit Tokyo 2017.

We had so many visitors, it was a very successful three days! A big shout out to all of you who came by the Mackerel booth, thank you!

Anyways, here is this week’s update.

Specifying multiple conditions in log monitoring is now easier with check-log

go-check-plugins v0.10.4 has been released, making it easier to specify multiple conditions in log monitoring using the check-log plugin.

Now, multiple --pattern options can be specified. When specified, they are treated with the AND operator (only log output lines that satisfy all specified patterns are detected).

The following is a configuration example to have alerts occur when a log output line that includes the two strings “PRODUCTION” and “FATAL” appears.

[plugin.checks.access_log]
command = "check-log --file /var/log/access.log --pattern PRODUCTION --pattern FATAL"

Also, check out the help page below for more information.

mackerel.io

Specifications for ELB metrics integrated with AWS Integration have changed

As you all may know, Mackerel’s AWS Integration feature is implemented using the Amazon CloudWatch.

Using the CloudWatch API to obtain ELB (CLB / ALB) metrics in relation to each HTTP status response count (e.g. HTTP Code ELB Count), responses without metric points themselves can be obtained at times when there is no response such as 4xx or 5xx.

Mackerel’s AWS integration also followed the CloudWatch API response, but was incompatible with Mackerel’s specifications and we received inquiries from multiple customers about a problem with monitoring not starting unless metric points existed. With this release, if there is no metric points, ELB metrics in relation to the response count for each HTTP status are now registered as 0.

This makes checking graphs more intuitive and also resolves the problem in the past where target metrics could not be selected if there were no 4xx or 5xx responses.

Updates for mackerel-agent-plugins and more

In addition to the updates made for go-check-plugins above, the following items have also been updated.

mackerel-agent-plugins v0.27.2

  • AWS Kinesis Firehose has been added
  • [mackerel-plugin-php-opcache] minor adjustments were made
  • [aws-rds] minor adjustments were made
  • Now built with Go 1.8

mkr v0.10.4

  • Minor adjustments were made

Thank you to everyone who contributed Pull Requests!