Mackerel blog #mackerelio

The Official Blog of Mackerel

Updates for mackerel-agent-plugins and more

Well we’re halfway through September and it’s finally starting to feel like the fall season. It’s supposed to get chilly in the coming days. So be careful not to catch a cold!

Anyways, here is this week’s update information.

Host metrics posted at 5-minute intervals can now be checked more than 25 hours retroactively

Up until now in Mackerel, when posting host metrics, which were supposed to be posted every minute, at an interval of 5 minutes or more, data rounded over 25 hours would be lost.

With this week’s update, it is now possible to check back on data more than 25 hours retroactively. As a result, past changes among metrics obtained with AWS Integration can be checked from Mackerel, even data with 5-minute granularity.

This is a part of the development of the “New Time Series Data System”. We are continually making improvements, so look forward to future updates.

The role name of the official role registered in Ansible Galaxy has been changed

The official role registered in Ansible Galaxy has been updated. The new version is 0.7.0. Along with this, we are changing the role name from mackerel to mackerelio. Please be aware that it will no longer be available under the old name.

Also with this update, check plugin options can be specified. Give it a try.

Updates for mackerel-agent-plugins and more

Updates have been made for mackerel-agent-plugins and more. Continue below for the details.

mackerel-agent-plugins v0.32.0

  • [memcached]evicted.reclaimed and evicted.nonzero_evictions metrics were added.
  • [accesslog] Adjusted to be able to scan long logs
  • [mysql] Minor corrections were made.
  • [Redis] Minor corrections were made.

mkr v0.18.0

  • [mkr create] Be able to specify custom_identifier by --customIdentifier option.

cookbook-mackerel-agent

A big thanks to all of you who contributed!

Mackerel at the AWS Cloud Roadshow 2017 Osaka!

We made this announcement last week, but Mackerel will be running a booth at the AWS Cloud Roadshow 2017 Osaka starting next week, on September 21st!

AWS Cloud Roadshow 2017 大阪 powered by Intel® 広島、大阪、名古屋、福岡の 4 都市を巡る無料クラウドカンファレンス開催! | アマゾン ウェブ サービス

If you’re planning on attending the event, definitely stop-by our booth!

Specific metrics can now be specified when sharing graphs etc.

Hello, Mackerel Team Director Kasuya (id:daiksy) here.

We’ve always made our weekly feature announcements from this blog on Friday, but this week we’re publishing on Thursday.

Last week, participants from the Hatena Summer Internship Program joined the Mackerel development team and started working on feature development. Tomorrow is the last day for our interns and each team will make a presentation to show what they’ve accomplished.

So, this week we’re announcing our new features one day early in preparation for the interns’ presentations tomorrow.

All new features announced this week will be those that the interns have worked on with support from the development team!

Anyways, here is this week’s update information.

Specific metrics can now be specified when posting graphs to notification channels

Up until now, sharing a graph with Mackerel’s graph posting feature meant displaying all metrics included in that graph. For example, even if you only wanted to discuss memory.used trends, you could only share a graph that displayed all the memory metrics

With this update, graphs can now be shared with specific metrics selected in the host metric graph.

The screenshot below shows graphs with only memory.used selected shared in Slack.

And of course, the permalink for this shared graph will also only have memory.used selected.

Notification channels are now displayed in the log of the alert details screen

All channels to which notifications will be sent are now displayed for each log in the alert details screen.

There are three types of icons to verify the notification status; one to show that the notification was successful, one for failed and currently re-sending, and one to show that the notification attempted to re-send the specified number of times, but failed.

Alerts can now be filtered by monitor rule in the alert list

“Monitor rule” has been added as a condition to filter alerts in the alert list.

For example, now it’s easier to do things like display a list of only closed check monitoring alerts and obtain an application’s rate of operation or narrow the list down to external monitoring alerts only and check the Web application status.

When manually closing alerts, the closer and the reason for closing can now be noted

In the notification for when an alert is closed manually, the person who closed the alert and the reason for closing it can now be noted.

With this, now you can get a better understanding of the manual closing without having to go the the alert details screen.

Upcoming Mackerel events

The Mackerel team is working hard to create opportunities like events and booths where we can meet all our users.

September 21st AWS Cloud Roadshow 2017 Osaka

Mackerel is sponsoring the AWS Cloud Roadshow in Osaka on September 21st. Be sure to stop by if you’re in the area! !

AWS Cloud Roadshow 2017 大阪 powered by Intel® 広島、大阪、名古屋、福岡の 4 都市を巡る無料クラウドカンファレンス開催! | アマゾン ウェブ サービス (Japanese only)

September 26th Drinkup Osaka

Mackerel’s Drinkup event has often been held in Tokyo, but this month it’s being held in Osaka.

mackerelio.connpass.com

I’m looking forward to seeing all of you in the Kansai area!

The namespace list can now be obtained with the metadata API etc.

Hello, Mackerel Team Director Kasuya (id:daiksy) here.

Here at Hatena, it wouldn’t feel like summer without our summer internship program. And this year’s new interns are fitting right in.

Hatena Summer Intership 2017 (Japanese only)

Starting this week, the student interns who’ve finished the first half lecture part of their internship have each been assigned to various teams and are trying their hands at feature development. I’m excited to see what features they’ll release this year.

By the way, here are features developed by the student interns who were assigned to the Mackerel team last year.

mackerel.io

mackerel.io

Anyways, here is this week’s update information.

The namespace list can now be obtained with the metadata API

An API to obtain the namespace list of metadata registered to the host has been implemented in the metadata API.

Combining this with the metadata acquisition API makes it easier to manage the information registered to the host.

Check out the help page below for more information. mackerel.io

By the way, this is a feature implemented by this year’s interns under the support of our development team! !

Log levels have been changed for when metric posting fails in mackerel-agent

Up until now, if metric posting in mackerel-agent failed, it was output as an error in the agent’s log. However, the failed posting would be retried and, unless the agent stopped, posted to Mackerel without losing any data.

For this reason, we’ve changed the log level to Warning.

Other updates for mackerel-agent and mackerel-agent-plugins are as follows.

mackerel-agent 0.44.2

  • Log levels changed for when when metric posting fails
  • Linux/MIPS CPU/SoC model name is now displayed in host details

mackerel-agent-plugins 0.31.1

  • [redis] the method to obtain queries metrics was corrected
  • [plack] errors no longer occur when JSON parsing fails
  • [jmx-jolokia] obtain metrics for Jmx Threading graph added

Mackerel at AWS Cloud Roadshow Osaka

Mackerel will be sponsoring and running a booth at the AWS Cloud Roadshow Osaka to be held on September 21st. Drop by if you’re in the area!!

AWS Cloud Roadshow 2017 大阪 powered by Intel® 広島、大阪、名古屋、福岡の 4 都市を巡る無料クラウドカンファレンス開催! | アマゾン ウェブ サービス (Japanese only)

A new feature, “GraphBoards” has been released! etc.

Hello! Mackerel team’s CRE Inoue (id:a-know) here.

Last Saturday, Mackerel successfully held its’ first Hackathon event, Mackerel Plugin Hackathon #1.

Anyways, here is this week’s update information.

A new feature, “GraphBoards” has been released!

With this update, we’ve released a new feature called “GraphBoards”!

So what does this feature do?

This feature is like dashboards, you can select any role graph or service metrics graph of that service and freely rearrange the layout. It’s ideal for users who have numerous roles or are using a lot of service metrics.

It’s also possible to create a more intuitive layout just by dragging and dropping!

Use of the automatic update feature and switching between display intervals is possible for any of the arranged graphs without having to change to another screen.

Check out the help page below for information on how to create and configure GraphBoards.

mackerel.io

Notes attached to monitoring rules are now included in the body of mail notifications

For monitoring rules created in Mackerel, you can leave notes such as “If this alert goes off, check —— first” or “If —– or —— occurs, this alert will occur”.

mackerel.io

Display for the content of these notes in alert notifications is already supported with some channels such as Slack and HipChat, but with this update, the note content is now also included in the body of mail notifications. We believe that this will help make troubleshooting smoother than before.

Updates for mackerel-agent and more

mackerel-agent v0.44.1

  • Minor corrections were made.

mackerel-agent-plugins v0.30.0

  • mackerel-plugin-openldap was newly added.
  • [aws-ec2-ebs] Burst Balance metrics were added.

go-check-plugins v0.12.0

  • check-disk plugin was newly added.
  • [postgresql] The database name can now be specified with the database option.

mkr v0.17.0

  • [dashboards] unit and title parameter support was added for expression graphs

Mackerel User Group Meeting Vol.3

The Mackerel User Group, a group of Mackerel user volunteers successfully held their third event, the Mackerel User Group Meeting Vol.3, on Wednesday, August 23rd!

mackerel-ug.connpass.com

On the day of, Lightning Talk consisted of presentations about useful tools and even a Mackerel game! It sounds like the event got pretty exciting.

Here are a few reports written by those who participated! (Japanese only)

For those of you who unfortunately couldn’t make it this time, please join us at the next User Group Meeting!

Mackerel at CEDEC 2017 August 30th - September 1st

Mackerel is scheduled to run a booth at CEDEC 2017 from Wednesday to Friday next week.

The event will be held at Pacifico Yokohama. We will be exhibiting on the third floor of the venue. Additionally, Sone (id:Soudai) , Mackerel CRE, will be speaking in the session “Enjoyable performance monitoring with Mackerel”.

If you’re planning on attending the event, definitely stop-by our booth!

Member invitations can now be canceled with the API etc.

Up until the middle of this week, our support window took some time off for the summer holidays (except for emergencies), but they’re back at it now. Sorry for any inconvenience and we appreciate your cooperation.

Anyways, here is this week’s update information.

Member invitations can now be canceled with the API

Using the invitation API, you can invite members to an organization by specifying an email address, but with this week’s update you can now cancel invitations with the API.

Up until now, it was only possible to cancel invitations from the Web UI, but now the same operation can be done from the API. You can also use it when you want to re-invite a member with different permissions. By all means, try it out.

For details regarding how to use the API, check out the help page below.

mackerel.io

Date and time of the latest update can now be obtained with the metadata API

Various information can be registered to the host using the metadata API, and now you can obtain the date and time of its latest update.

% curl --dump-header - https://mackerel.io/api/v0/hosts/<host_id>/metadata/<namespace> -H "X-Api-Key: <API_KEY>"
HTTP/1.1 200 OK
(OMTD)
Last-Modified: Sat, 01 Jul 2017 13:25:08 GMT

For more regarding the metadata API, check out the below help page.

mackerel.io

About the second stage of maintenance on Monday (8/21)

This is another announcement regarding the second stage of our platform transition maintenance which will be carried out early next week from 2: 30 - 3: 30 p.m. (JST) on Monday, August 21st, 2017.

This will be the last stage of scheduled maintenance related to this platform transition. Your cooperation is appreciated. Please see the entry below for details.

mackerel.io

A book on Mackerel coming soon!

A book on Mackerel will be released on the 26th of this month!

This book isn’t just about how to use Mackerel, it also talks about the basics of server monitoring and serves as an introduction to infrastructure management. By all means, take a look!

gihyo.jp

Mackerel User Group Meeting Vol.3 coming soon!

We made an announcement about this in a previous blog post, but the event is almost here! The “Mackerel User Group”, a group put together by volunteer Mackerel users, is holding their next event, Mackerel User Group Meeting Vol. 3!

mackerel-ug.connpass.com

Even though this is a User Group event, id:Soudai and id:Songmu, CRE and sub-producer of the Mackerel team are scheduled to take the stage!

On top of that, this event has a line up of presentations that are sure to be really interesting, so you’ll definitely want to join us!

Regarding stage 2 maintenance on 8/21(Mon.)

Mackerel sub-producer id:Songmu here.

The first stage of maintenance was carried out this week on Monday, August 7th. As previously announced, in continuation with this maintenance, the second stage will take place on Monday, August 21st.

This time, only relocation of the subsystems will be done and since the database and such will not be relocated, service will not be stopped.

With this, maintenance related to this platform transition will come to an end. Your cooperation is appreciated.

Although mentioned in the entry below, we’ll take you through the details again.

Date and time

Monday, August 21, 2017 2:30 - 3:30 p.m. (JST)

Content

The URL external monitoring, AWS integration, and Azure integration subsystems will be relocated.

Extent of impact

There will be no stoppage of service. However, there is a possibility that monitoring with the above-mentioned subsystems may occasionally stop working.

After maintenance is completed, the source IP address used for notification channels from Mackerel such as URL external monitoring, AWS integration, Azure integration, and Webhook will be changed. Email notifications will not change.

Additionally, in regards to URL monitoring, it’s possible that measured response time trends may fluctuate slightly in order to for the observation conditions, including location, to change.

For users who are subject to IP address restrictions, please add the new IP address in addition to the current IP address to the transmission permission targets by the maintenance date.

※Due to the possibility that rollbacks may occur after maintenance is completed, please keep the current IP address concurrently for a few months. We will make a separate announcement regarding deletion of the current IP address.

Current IP address

  • 59.106.108.64/26

New IP address

  • 52.193.111.118/32
  • 52.196.125.133/32
  • 13.113.213.40/32
  • 52.197.186.229/32
  • 52.198.79.40/32
  • 13.114.12.29/32
  • 13.113.240.89/32
  • 52.68.245.9/32
  • 13.112.142.176/32

Regarding announcements on the day of

Announcements will be made from our Status Page (http://status.mackerel.io).

We’ll be making occasional status update announcements from the status page. In the unlikely event that maintenance should be extended, we will inform you from the status page.

In addition, after maintenance is completed, we will update our blog with details including actual times of implementation.

For inquiries regarding this matter please email us at support@mackerel.io.

Thank you for your cooperation and thank you for choosing Mackerel.

Maintenance completion notice + an apology in relation to data loss and custom dashboard/expression monitoring failures

I’m id:Songmu, Mackerel’s sub-producer. Previously announced maintenance began today (Monday, August 8th) at 2:30 p.m. (JST) and was completed at 6:00 p.m. Thank you to all our users for your cooperation.

In this entry, we’ll report on some of the details related to this event. All the times listed below will be in Japan Standard Time.

Period of no-access

The entire system was inaccessible from 2:30-2:50 p.m. as well as 3:30-3:33 p.m.

Regarding data loss

During maintenance, an unexpected loss of data occurred and we sincerely apologize for any inconvenience this may have caused.

Data created in the Mackerel database during the time of 2:30-3:31 p.m. excluding times series data, was lost. Specifically, data including alert and host information registered during the above time period was lost.

Regarding the cause of the incident and our response

The cause of this incident was due to an unintended database failover during the time of maintenance. Data was lost from the start of maintenance until when the failover occurred. Originally, preventative measures were taken so that data would not be lost if a failover occurred, but failed due to a unique circumstance while work was in-progress.

Attempts were made to recover said data, but due to difficulty in maintaining consistency with the data updated after failover, the recovery was abandoned.

After the failover, we carefully reexamined the redundant configuration of the database and made sure that a similar problem would not occur. Technical details regarding this matter will be posted at another time.

Regarding work requests for hosts registered during the period of data loss

Hosts that were newly registered with mackerel-agent during the above mentioned time period (2:30-3:31 p.m.) can not post data correctly. We’re sorry for the inconvenience, but please re-register the host.

Specifically, delete the id file ( /var/lib/mackerel-agent/id ), then restart mackerel-agent.

Regarding custom dashboard failures (fixed 8/8)

Graph display problems have occurred for custom dashboards created or updated since the start of maintenance. This issue is scheduled to be fixed by Tuesday, August 8th.

(Fixed 8/8) Corrections were complete with this release on Tuesday, August 8th.

Regarding expression monitoring failures (fixed 8/8)

We have confirmed the occurrence of expression monitoring failures in several organizations after the maintenance. We sincerely apologize, but we are unable to respond immediately. Individual announcements will be made to the owners of the respective organizations.

(Fixed 8/8) Corrections were made and the situation resolved with this release on Tuesday, August 8th.

Again, we apologize for any inconvenience this may have caused our users. We are committed to further improving our quality of service in the future and we thank you for your patience and appreciate your continued use of Mackerel.