Mackerel blog #mackerelio

The Official Blog of Mackerel

Some changes have been made to our Help page regarding IAM policies required for AWS・Lambda Integration to go along with the announcement from AWS

Hello. Mackerel team CRE Inoue (id:a-know) here again wishing you a happy new year! Here’s to hoping 2021 is a great one!

Without further ado, let’s get to today’s announcement.

Some changes have been made to our Help page regarding IAM policies required for AWS・Lambda Integration to go along with the announcement from AWS

Have you seen the recent announcement from AWS?

docs.aws.amazon.com

To go along with this announcement, we are re-examining the Mackerel Help page regarding AWS Integration.

  • Before
    • AWSLambdaReadOnlyAccess
  • After
    • AWSLambda_ReadOnlyAccess

The old policy AWSLambdaReadOnlyAccess will become deprecated after January 25th, 2021 and will no longer be attachable. Instead please use the new policy AWSLambda_ReadOnlyAccess (Note that policies that have already been attached should still be available).

Metrics added / replaced for Kinesis Data Streams / Step Functions AWS Integration and more

Hello. Mackerel team CRE Inoue (id:a-know) here with the last update announcement of 2020. Thank you for choosing Mackerel this year.

Metrics added / replaced for Kinesis Data Streams AWS Integration

The following metrics have been added / replaced for Kinesis Data Streams.

Before

  • Records
    • GetRecords.Records (kinesis.records.get_records)
    • IncomingRecords (kinesis.records.incoming)
    • PutRecords.Records (kinesis.records.put_records)

Now

  • Records
    • GetRecords.Records (kinesis.records.get_records)
    • IncomingRecords (kinesis.records.incoming)
    • PutRecords.SuccessfulRecords (kinesis.records.put_records_successful)
    • PutRecords.TotalRecords (kinesis.records.put_records_total)
    • PutRecords.FailedRecords (kinesis.records.put_records_failed)
    • PutRecords.ThrottledRecords (kinesis.records.put_records_throttled)

The metric previously recorded as PutRecords.Records (kinesis.records.put_records) is now PutRecords.SuccessfulRecords (kinesis.records.put_records_successful). Three new types of records data are now integrated as well. The name change from kinesis.records.put_records to kinesis.records.put_records_successful should be noted in particular because it may affect things such as monitors.

Also, the maximum number of metrics per host has been increased from 31 to 34.

Metrics added for AWS Integration with Step Functions

  • Service Integrations
    • ServiceIntegrationsFailed (states.service_integrations.SERVICE_INTEGRATION_NAME.failed)
    • ServiceIntegrationsScheduled (states.service_integrations.SERVICE_INTEGRATION_NAME.scheduled)
    • ServiceIntegrationsStarted (states.service_integrations.SERVICE_INTEGRATION_NAME.started)
    • ServiceIntegrationsSucceeded (states.service_integrations.SERVICE_INTEGRATION_NAME.succeeded)
    • ServiceIntegrationsTimedOut (states.service_integrations.SERVICE_INTEGRATION_NAME.timedout)
  • Service Integration Time
    • ServiceIntegrationRunTime (states.service_integration_time.SERVICE_INTEGRATION_NAME.run)
    • ServiceIntegrationScheduleTime (states.service_integration_time.SERVICE_INTEGRATION_NAME.schedule)
    • ServiceIntegrationTime (states.service_integration_time.SERVICE_INTEGRATION_NAME.all)

The service integration name replaces SERVICE_INTEGRATION_NAME.

With this addition, the maximum number of metrics per host is now calculated using the formula 7 + 9 × (number of activities) + 8 × (number of Lambda functions) + 8 × (number of service integrations). Please be sure to take note of the number of metrics as this may affect your micro hosts.

The same applies to the above-mentioned Kinesis Data Streams integration. If necessary, you can select integration targets using the optional selection function.

Network interface metrics can now be excluded with mackerel-agent

You can now exclude network interface metrics by writing the following in the mackerel-agent.conf.

[interfaces]
ignore = "bond.*"

Using a regular expression, metrics from the specified interface will not be collected. For Windows Server, specify the interface name that corresponds to that in Device Manager.

See the help page below for more information.

mackerel.io

Along with this update, the mkr command line tool has been updated from 0.42.0 to 0.43.0.

github.com

Official support for mackerel-agent in CentOS6 environments has ended

With CentOS6 ending their support on November 30th of this year, Mackerel too has officially ended support for CentOS6 environments.

Even if an environment is kept the same, surrounding environments are always changing (for example, managing old versions of TLS communication and host CA certificate renewal etc.), and so we believe that the best course of action is to update and switch environments. We apologize for the inconvenience and thank you for your cooperation.

Email notifications now available for events such as host registration and monitor changes and more

Hello. Mackerel team CRE Inoue (id:a-know) here with another update announcement.

Email notifications now available for events such as host registration and monitor changes

Up until now, event notifications for things like registering a host in Mackerel and changes to monitors were only supported for some notification channels such as Slack and Webhook.

Now, these notifications are also available for email as well.

We hope that email notifications will help you stay on top of the unexpected.

New metric targets added for AWS・S3 Integration

Two types of metrics have been added, Bucket Size and Number Of Objects. See the help page linked below.

mackerel.io

Please note that both are daily metrics.

OSS Updates

Today’s update covers a wide range of OSS products.

mackerel-agent-plugins(v0.63.2)

mackerel-check-plugins(v0.38.0)

  • [check-postgresql] Added support for the sslrootcert option used to encrypt plugin communication

mackerel-container-agent(v0.5.0)

  • CPU / memory metrics for container-agent now available with Kubernetes 1.18 and later.
    • Additionally, a get operation (verb) is no longer required for the nodes/spec resource in the RBAC configuration of Kubernetes (EKS on Fargate).

mkr(v0.42.0)

  • [monitors subcommand] Support added for alertStatusOnGone (specifying the alert status when an alert occurs) with push pull.

Thank you to everyone who contributed!

Support availability during the year-end and New Year holidays

The end of the year is fast approaching with less than three weeks left in 2020. And thank you for choosing Mackerel this year.

Mackerel’s official support window will be closed from December 28, 2020 (Thursday) until January 3, 2021 (Sunday).

For those of you planning to look over your Mackerel settings etc. before the end of the year and New Year holidays, we recommend you do so as soon as possible. And if you have any questions or concerns, please do not hesitate to contact us.

Mackerel’s Official Support Request Form

mackerel-container-agent is now available on Amazon ECR public

Amazon ECR public was just released at AWS’s annual event re:Invent 2020.

To go along with this, we’ve released mackerel-container-agent on Amazon ECR public.

gallery.ecr.aws

The container image for mackerel-container-agent was previously only available from Docker Hub. Now, the mackerel-container-agent container image is also available from Amazon ECR public. Now you have multiple options to choose from.

Billing data can now be monitored with AWS Integration and more

Hello. Mackerel team CRE Inoue (id:a-know) here with another update announcement.

Billing data can now be monitored with AWS Integration

2020-11-20 Postscript

We’ve recently confirmed the occurrence of hosts being registered in Mackerel, but not posting graphs if the budgets name is longer than a certain length on the AWS side.

We apologize for the inconvenience and appreciate your cooperation as we proceed with further investigation to resolve this event.

End of Postscript

f:id:mackerelio:20201118143501p:plain

It is now possible to monitor Billing data with AWS Integration. With this, you can now monitor/visualize billing and budget metrics for your AWS account. If using AWS Organizations for consolidated billing, you can also retrieve metrics for billing amounts per member account. Prices are calculated using the conversion 1 management account = 1 micro host.

Account IDs are used for host names. And Budget names are incorporated as part of metric names, but because Japanese characters cannot be used for metric names in Mackerel, if such a character string is included in a Budget name, those characters are encoded as shown below.

f:id:mackerelio:20201119110122p:plain

Check out the help page below for more details.

mackerel.io

For information regarding how to set up AWS integration, you can refer to the following help page (If using an existing IAM role, setting an additional AWSBudgetsReadOnlyAccess policy is required).

mackerel.io

For a while now, we’ve received a lot of interest in being able to use Mackerel to aggregate and monitor not only the statuses of various infrastructure resources, but also billing data such as this. This release is in response to those requests. Be sure to give it a try!

Support added for 1000+ registered queues in AWS / SQS Integration.

mackerel.io

Up until now, only up to 1000 items could be integrated with AWS / SQS Integration, but with this update, that number is now unlimited.

This will probably only be relevant to a few special use cases, but if applicable to you, be sure to have a look.

Azure Application Gateway now supported with Azure Integration and more!

Hello. Mackerel team CRE Inoue (id:a-know) here with another update announcement.

Azure Application Gateway now supported with Azure Integration

f:id:mackerelio:20201106145512p:plain

Azure Application Gateway can now be monitored with Azure Integration! Application Gateway is Azure's managed load balancer service that enables you to do application layer (OSI layer 7) load balancing.

Check out the help page linked below for configuration methods.

mackerel.io

And the following link for obtainable metrics with Application Gateway.

mackerel.io

Notification content expanded for alerts to PagerDuty

The content that can be included in alert notifications sent from Mackerel to PagerDuty has been expanded. The following items have been added.

  • Monitor notes
  • A thumbnail image of the target metric
  • custom_identifier

Now when an alert occurs, more information can be obtained when looking at PagerDuty. You may want to take this opportunity to consider using PagerDuty or at least review the troubleshooting flow.

Also, please note that this expansion is only available with the Event API v2. For more information regarding alert notifications from Mackerel to PagerDuty, be sure to take a look at the following help page.

mackerel.io

Monitor memos now included in Webhook notification payload

Similar to the previous update regarding Mackerel supported notification channels, Webhook notifications have added monitor memos to its payload.

A variety of information can be included in monitor memos. For example:

  • Information regarding the initial response to an alert (commands or emergency measures)
  • An explanation for or the reason behind the monitor
  • URLs of related documents, etc.

This information is also included in Webhook requests, potentially enabling you to respond more smoothly and reduce recovery time. So be sure to give it a try!

Arm64 package release・Following redirects now possible for external monitoring and more

Hello. Mackerel team CRE Inoue (id:a-know) here with another update announcement.

Arm64 package released for mackerel-agent and more

An Arm64 package for mackerel-agent and various plugin packages is now officially available and can be found in the apt/yum repository (v2 repository).

Some of you may have noticed the recent increase in support for Arm architecture, for example, the release of the A1 instance by Amazon EC2. Feel free to give this a try.

Redirects can now be followed in external monitoring

External URL Monitoring lets you monitor response status codes and response times by simply specifying a URL in the Mackerel Web console. In fact, I think a lot of our users start configuring this monitoring first because it’s not necessary to install the agent.

mackerel.io

Up until now, Mackerel's external URL monitoring specifications were to "recognize 2xx or 3xx-type responses as normal responses and not as errors" and "not to follow redirects", but this behavior has been changed. This can now be changed in settings.

Check out the help page linked below for more details.

mackerel.io

An error message is now displayed when there is a discrepancy between the agent installation script and the OS type or version

f:id:mackerelio:20201030140824p:plain

Here’s an example of what it looks like when running Amazon Linux installation script for an Amazon Linux 2 environment.

Our support team has received numerous inquiries regarding this discrepancy, so we’ve added this as a response. Please be advised when installing.

The official command line tool mkr has been updated

Do you use mkr, Mackerel's official command line tool?

mackerel.io

Well, it has now been updated to v0.41.0. The following changes have been made.

  • Made it easier to understand the output when running mkr hosts and no hosts exist
  • An armhf package for mkr is now generated in GitHub Releases

Thank you to all who contributed!