Mackerel blog #mackerelio

The Official Blog of Mackerel

The release of Mackerel container agent (public beta) and more

Mackerel team CRE Miura (id:missasan) here.

The long-awaited Mackerel container agent (public beta) has finally been released!

Also, Mackerel Meetup #13 Tokyo is scheduled to be held on Friday, March 1st at the Tokyo office of Cybozu, Inc. At the event, a lot of the topics will focus on the operation and monitoring of containers. This will be a great opportunity to get the story behind the development and some know-how regarding using Mackerel container agent.You don’t want to miss it!

Now on to this week’s update information.

The release of Mackerel container agent (public beta)

Mackerel container agent has been released. Use it to monitor containers on container orchestration platforms. Currently, the following platforms are supported.

Executable as a task/Pod sidecar, you can post CPU, memory, and network interface metrics as system metrics for each container. You can also configure the monitoring rate based on the obtained metric in the Monitor Settings screen.

For details regarding specifications or the setup process for Mackerel container agent, check out the Help page linked below.

mackerel.io

Mackerel container agent is still in the development process in preparation for its official release. So please give it a try and let us know what you think. Incompatible changes will not be made without an announcement, however, depending on demand, changes may be made with advance notice.

For Amazon ECS / AWS Fargate, 1 task will count as one host. For Kubernetes, 1 Pod will count as 1 host. The number of hosts will be calculated using a moving average of the previous month. For more details, please refer to FAQ · Calculating the number of hosts.

※The billing system for registered hosts with container agent will change in the future.

Mackerel Meetup #13 Tokyo on March 1st (Fri)!

We’ll be borrowing Cybozu Inc.’s Tokyo office seminar room to hold Mackerel Meetup #13 Tokyo! Our very own Imai (id:hayajo_77), the developer behind Mackerel container agent, is scheduled to speak at the event. The presentation will go over the agent’s specifications and features, as well as design ideas and implementation methods.

Click on the link below to apply.

mackerelio.connpass.com (Japanese only)

Event details

  • Date and time: March 1, 2019 (Fri) at 4:30 p.m. ~ 9:00 p.m. (JST) (Reception starts at 4:00 p.m.)
  • Venue: Cybozu Inc. Tokyo Office
  • Address: 〒103-6028 Tokyo, Chuo-ku, Nihonbashi 2-7-1, Tokyo Nihonbashi Tower 27th floor (Reception 7F)
  • Access: Tokyo Office Access Map | Cybozu Inc.
  • Cost: Free

Microsoft Teams added to notification channels

Mackerel team CRE Miura (id:missasan) here.

This week’s release will come as welcome news for those using Microsoft Teams for team communication.

In the past, we’ve heard from many users requesting the ability to send alert notifications to Microsoft Teams channels. With this release, alert notifications for Microsoft Teams is now supported as a standard function. It’s easy to setup and the alert content is easy to see. Definitely give it a try!

Now on to this week’s update information.

Microsoft Teams added to notification channels

When attempting to integrate alert notifications in Microsoft Teams in the past, mail notifications had to be sent to the email address issued by Microsoft Teams. Although this method worked, we received feedback that the display was partially distorted and it was difficult to use. With this release, the ability to integrate notifications to Microsoft Teams is now available as a standard function. Problems regarding visibility and image loss have also been resolved.

Configurations can be made from the Channel Settings screen.

When an alert occurs, it will be displayed like shown below.

For more details, refer to the Help page. mackerel.io

We look forward to your thoughts and feedback.

The system shutdown for database maintenance scheduled for Feb. 7th (Thur) has been cancelled

Thank you for choosing Mackerel.

mackerel.io

Regarding our previous announcement and the urgent maintenance scheduled to take place on February 7th (Thur) at 2:30 pm, a resolution for the problem has been found and the scheduled maintenance has been cancelled. The use of Mackerel will continue as usual during that aforementioned time period.

Reason for cancellation

We’ve confirmed the restoration of autovacuum in PostgreSQL (RDS), the data store used by Mackerel. And since the problem of the transaction ID being depleted has also been resolved, we have decided that emergency maintenance is unnecessary.

We will continue monitor the situation and make every effort to ensure that the operation of Mackerel is stable in the future.

We apologize for any inconvenience this may have caused.

Thank you for understanding and cooperation.

New information added that can be obtained with the user list API and more

Mackerel team CRE Miura (id:missasan) here.

As previously announced, several fascinating presentations are lined-up for Mackerel UG Kansai Meetup # 1. Itec Hankyu Hanshin will be presenting on "Introducing Mackerel in fully managed hosting" and Beyond Co., Ltd. will be talking about "Comparing server monitoring contents with Mackerel and Zabbix”. Right now it seems like there are still some spots available, so please come and join us if you’re in the Kansai area!

mackerel-ug.connpass.com (Japanese only)

Now on to this week’s update information.

New information added that can be obtained with the user list API

The added information is as follows.

  • isInRegistrationProcess
  • isMFAEnabled
  • authenticationMethods
  • joinedAt

For more details, refer to the help page linked below.

mackerel.io

Specification changes for the host update API

As was announced in the following entry, specifications of the host update API have changed. For more details, see the entry below.

mackerel.io

(Appended)【Urgent Maintenance】The system will temporarily shut down for database maintenance on Feb. 2 (Thur) at 2:30 pm

Thank you for choosing Mackerel.

【Added content】

Regarding the scheduled maintenance, the problem is being resolved and the maintenance has been cancelled. For more details, refer to the following entry.

mackerel.io

【End of added content】

The system will be temporarily suspended for maintenance according to the schedule below. We apologize for any inconvenience that this may cause and appreciate your understanding.

Moreover, we apologize for such a short notice due to the urgency of this maintenance.

Scheduled date and time

  • February 7th (Thur) 2:30 pm - 4:30 pm (JST)
    • This time window is the longest case scenario. A breakdown of the procedure of operations will follow below.

Content

  • Database maintenance

Regarding impact on the day of

  • The above time window is an estimate of the longest case scenario. The actual maintenance period will end once the work has been completed.
    • Once maintenance has begun, the entire Mackerel system will shutdown for a short period of time.
  • Web access to Mackerel, data posting by the agent, API access (including the CLI tool), alert notifications, etc. will be unavailable
  • As soon as the maintenance work is completed and operation is confirmed, the system shutdown will end and an announcement will be made
  • As for mackerel-agent metric posting, data will be buffered from mackerel-agent during the maintenance period and resent after maintenance has been completed
    • If resent properly, graphs during the maintenance period will also be displayed.

Reason for maintenance

In PostgreSQL (RDS), a data store used by Mackerel, autovacuum isn’t running properly. In the current state, service operations have not been affected, but if this state continues, the transaction ID will be exhausted and the update function for the database will be forcibly stopped. For this reason, we’ve decided that urgent maintenance is necessary.

Related to this, a similar Bug Report (https://www.postgresql-archive.org/found-xmin-from-before-relfrozenxid-on-pg-catalog-pg-authid-td6011810.html ) was submitted and found that the problem can be restored by rebooting.

We will try this during the maintenance period. In addition, as a result of preliminary inspection, we’ve confirmed that autovacuum can be run on DB restored from snapshot. And if the problem can’t be resolved by rebooting, that will be the next issue addressed.

Procedure of operations

  • Temporary system shutdown
  • RDS reboot
    • If recovery is confirmed here, maintenance will end
  • Replace with DB restored from snapshot
  • After recovery is confirmed, maintenance will end

The 2 hour maintenance time window is an estimation of recovery time based on the snapshot. If the issue is resolved by rebooting, maintenance will end at that point.

Regarding Announcements

Situation reports will be made from the Mackerel status page (http://status.mackerel.io) as well as from this blog (https://mackerel.io/blog/).

Additionally, we’ll also be using our official Twitter account (https://twitter.com/mackerelio_jp).

For inquiries related to this matter

Please send all inquiries regarding this matter to support@mackerel.io.

Thank you again for your understanding and cooperation. And thank you for choosing Mackerel.

AWS Integration now supported with Elasticsearch Service and more

Mackerel team CRE Miura (id:missasan) here.

The first event for the Kansai branch of the Mackerel UG is going to be held at the Osaka office of Aiming Inc.! The Mackerel team is planning to mix and mingle at the event so if your in the Kansai area, please join us!

Now on to this week’s update information.

AWS Integration now supported with Elasticsearch Service

Check out the following help page for the details.

mackerel.io

Aurora Serverless clusters can now be integrated with AWS Integration RDS

Managing database instances is unnecessary in Aurora Serverless. With Aurora Serverless enabled with AWS Integration, clusters are counted as one host.

Obtainable metrics have been added as well. Check out more of the details in the help page linked below.

mackerel.io

Mackerel UG Kansai Meetup #1!

Several presentations are planned for this event. Itec Hankyu Hanshin will be presenting on "Introducing Mackerel in fully managed hosting", the same as there presentation at Mackerel Meetup # 12 in Tokyo. And Beyond Co., Ltd. will be talking about "Comparing server monitoring contents with Mackerel and Zabbix”. For those of you who are too far away to come to the Meetups in Tokyo, be sure to join us and the Mackerel UG Kansai branch.

Check out the event page below for more details. (Japanese only)

mackerel-ug.connpass.com

Event details

【Advanced release notice】Specification changes for the host update API

Thank you for choosing Mackerel.

With the release scheduled for January 30, 2019 (Wednesday), changes will be made to the specifications of the host update API in order to make posting check reports with the API even more convenient.

The contents of these changes follow below.

Before the changes

With the host update API, in addition to the host's own information, you can send a list of check monitors that monitor the host and that list is synchronized with the check monitors in Mackerel.

Under the current specifications, if reports of check monitors not included in this list are posted to the host, the reports will be deleted and any open alerts will be closed automatically when the host information is updated.

The agent periodically updates the host’s information internally using the host update API and sends it together with a list of the check monitors described in the agent configuration file. With this mechanism, the configuration file and the check monitors in Mackerel are synchronized, and the check monitors that are deleted when the configuration file is updated are also deleted in Mackerel.

After the changes

The specification that synchronizes the host update API and the list of check monitors will change in the following ways:

When there are reports of check monitors that are not included in the list that was previously posted to the host...

  • If an alert related to that check monitor is opened, then the check monitor result will not be deleted in Mackerel and the alert will remain open.
  • If an alert related to that check monitor has not occurred, as with the current specifications, the check monitor results will be deleted in Mackerel.

Intention of the specification changes

As previously mentioned above, the agent internally uses the host update API to periodically synchronize the check monitors described in the agent configuration file with Mackerel. For this reason, according to the current specifications, if using the check report posting API from outside the agent to post reports to the host, and if check monitors are not included in the agent configuration file, deletions periodically happen regardless of whether or not an alert has occurred. And because of this, the display of check reports in Mackerel and use of the alert function can not be used normally.

With the specification change, even if check reports are posted from outside the agent, even if check monitors are not included in the agent configuration file, reports will no longer be deleted in Mackerel when an alert has occurred and the alert function can be used normally.

As a reminder, if there are no open alerts, the check reports will be deleted in Mackerel, same as with the current specifications. For this reason, there may be a difference in the display of check reports in the host detail screen, depending on the timing of the host information update and when check reports are posted from outside the agent. Please take this into consideration before using the API.

Thank you for your cooperation.