Mackerel blog #mackerelio

The Official Blog of Mackerel

A thorough explanation of mackerel-plugin-accesslog

Mackerel Product Owner (and sub-producer) id:Songmu here. This is an article that I published on the ninth day of our 2017 Mackerel Advent Calendar.

This article is an introduction for the official plugin mackerel-plugin-accesslog. We worked hard on this plugin this year and we think it’s really useful.

This plugin aggregates and visualizes web server access logs.

How to install

mackerel-plugin-accesslog is included in the official metric plugin package and can be used once the package has been installed. For details on how to install the package, see the following help page.

Using the official plugin pack to visualize middleware metrics - Mackerel Docs

go get can be used in Go environments as shown below.

% go get github.com/mackerelio/mackerel-agent-plugins/mackerel-plugin-accesslog

How to configure

Specifying the access log file name as a parameter is extremely easy. Simply specify the following in mackerel-agent.conf.

[plugin.metrics.accesslog]
command = "mackerel-plugin-accesslog /path/to/access.log"

Supported formats

The following log formats are supported.

  • Normal Apache format logs (Common/Combined) used in Apache and Nginx
  • LTSV format logs

In addition to major Apache logs, LTSV (Labeled Tab-separated Values) access log format, a format commonly used in the Japanese web industry and throughout Hatena1, is also supported.

The LTSV access log must be in a format that uses the Recommended Label defined in ltsv.org2. Regarding configuration method, refer to the description in ltsv.org .

I wrote the access log parser3, and I am proud to say that most logs can be parsed. However, I do strongly recommend using the LTSV log because response latency can only be aggregated with LTSV. Moreover, LTSV seems to have better parse efficiency, though I think the difference is slight.

Graph items

The following items are aggregated and graphed.

  • Access count of each status code
  • Percentage of each status code
  • Latency (only for LTSV log)
    • Average
    • 90/95/99 Percentile

As you can see in the above screenshot, visualization of these metrics is fairly straight forward.

Incidentally, mackerel-plugin-accesslog records the position of the read log file at the time of last execution and continues to read the log from that time. Therefore, values are collected every 1 minute when regularly executed from the agent.

Even if the logs are rotated, mackerel-plugin-accesslog searches for potential files before being rotated and, if located, reads it to the end and then reads the new log file from the beginning. As a result, aggregation is more accurate. This can be suppressed with a library called postailer4.

Combining expression graphs and aggregating in one role

As it’s typical for web servers to have multiple configurations, you may want to have graphs aggregated by role.

Mackerel has an experimental feature that lets you display customized graphs using expressions (commonly called: expression graphs5). With this feature, you can display access log graphs aggregated by role.

Here, let's try displaying the access count and ratio of each status code aggregated in one role in a stacked graph. Also, please be aware that expression graphs are still an experimental feature and must be enabled in the organization settings.6

Displaying the status code access count in stacked graphs

The expression for displaying the above graph follows below. Please rewrite the role name and metric name as appropriate.

group(
  stack(alias(sum(
    role(SugoiService:web,custom.accesslog.access_num.2xx_count)),
    '2xx_count')),
  stack(alias(sum(
    role(SugoiService:web,custom.accesslog.access_num.3xx_count)),
    '3xx_count')),
  stack(alias(sum(
    role(SugoiService:web,custom.accesslog.access_num.4xx_count)),
    '4xx_count')),
  stack(alias(sum(
    role(SugoiService:web,custom.accesslog.access_num.5xx_count)),
    '5xx_count')))

Displaying the status code access percentage in stacked graphs

The expression for displaying this graph is a bit complex, but follows below. Please rewrite role name and metric name as appropriate here as well.

group(
  stack(alias(scale(
    divide(sum(role(SugoiService:web,custom.accesslog.access_num.2xx_count)),
    sum(role(SugoiService:web,custom.accesslog.access_num.total_count))),100),
    '2xx_rate')),
  stack(alias(scale(
    divide(sum(role(SugoiService:web,custom.accesslog.access_num.3xx_count)),
    sum(role(SugoiService:web,custom.accesslog.access_num.total_count))),100),
    '3xx_rate')),
  stack(alias(scale(
    divide(sum(role(SugoiService:web,custom.accesslog.access_num.4xx_count)),
    sum(role(SugoiService:web,custom.accesslog.access_num.total_count))),100),
    '4xx_rate')),
  stack(alias(scale(
    divide(sum(role(SugoiService:web,custom.accesslog.access_num.5xx_count)),
    sum(role(SugoiService:web,custom.accesslog.access_num.total_count))),100),
    '5xx_rate')))

Monitoring for expression

It’s also possible to monitor the value obtained by expression. Please refer to the Help page for more information.

Expression graphs is a powerful feature, and it is extremely satisfying if you can write a useful expression correctly, but I’m aware that this feature may still be difficult to use so I am working on improvements. Requests are always welcome.

Comparing use of fluentd

The official help page "Posting service metrics with fluentd" introduces a method to aggregate access logs using fluentd.7

mackerel-plugin-accesslog is extremely useful because it makes access log aggregation possible without the use of fluentd.

However, that does not mean that the method using fluentd is no longer necessary. mackerel-plugin-accesslog is specifically for aggregation of a single server. It may be more convenient to use fluentd for situations with multiple servers or when performing arbitrary aggregation. As a matter of fact, in recent years it’s increasingly common that logs are not written in the server local file system and to rely on a log collector such as fluentd. In such a case, mackerel-plugin-access log can not be used.

Summary

Mackerel-plugin-accesslog can be a useful feature, by all means, give it a try. I’m also looking forward to your issues and pull requests for improvements and amendments.

https://github.com/mackerelio/mackerel-agent-plugins/tree/master/mackerel-plugin-accesslog

Our support window will be closed for the holidays

Both methods of contacting our support team, either through support@mackerel.io or the “contact our support team” option displayed in the upper right header when logged into Mackerel, will be closed during the following dates for the New Year’s holidays.

New Year’s holiday period :Thursday, December 28th, 2017 - Wednesday, January 3rd, 2018 (JST)

All inquiries received during this period will be handled sequentially starting from January 4th, 2018 (Thursday).

You can now have notifications sent to channels for monitor rule operations etc.

Hello! Mackerel team CRE Inoue (id:a-know) here.

Already the 4th week in December, I’ve attended a end-of-the-year party everyday this week. I hope everyone is enjoying the holiday season as much as me.

This will be the last of this year’s weekly updates for Mackerel.

Since September of 2014, we’ve continued our tradition of “Consecutive weekly releases” and we’ll be aiming for 200 consecutive weeks in the coming year. We’re also excited to deliver various new functions/updates to everyone in 2018!

Now on to this week’s update information.

You can now have notifications sent to channels for monitor rule operations

It is now possible to have event notifications sent to notification channels when monitor rule related operations (creating / updating / deleting) are made.

If you select Webhook as the notification channel, changes made to the monitor rule will be included in the JSON format notification content. The specifics follow below.

  • Creating or updating a monitor rule
    • Monitor rule content will be notified after the operation
  • Deleting a monitor rule
    • Monitor rule content will be notified immediately before deletion

Posting graphs in notification channels now supported with LINE and Yammer

The graph posting feature allows you to post graphs to notification channels by clicking the camera icon in the graph menu. This is a useful feature because it lets you share metrics at any time, even those not configured for alerts. Now, LINE and Yammer are also supported with this feature.

Check out the following help page for more on this feature.

mackerel.io

Updates for Mackerel related OSS

Updates for various Mackerel related OSS were made this week and the details follow below. As always, thank you to everyone who contributed!

mackerel-agent v0.48.2

  • [For Windows] The problem regarding address information of the network interface not being available was fixed.

mackerel-agent-plugins v0.41.1

  • [mysql] Sections of stats related to InnoDB can now be obtained, even the reader node of AWS Aurora for MySQL
  • [mysql] Other minor adjustments

Thanks for another great year!

Although I mentioned this at the beginning, this will be our last update announcement for this year. Thank you so much for supporting Mackerel through 2017.

We are excited for 2018 and continuing our weekly updates. Next year, we hope to deliver an even more advanced and efficient server monitoring/management experience to our users through Mackerel.

Thank you for choosing Mackerel. Until next year!

Environment variables can now be specified for command execution via plugin etc.

Hello! Mackerel team CRE Inoue (id:a-know) here.

Only half a month left in 2017 and I’m sure everyone’s busy with work and year-end parties. The Mackerel team is keeping busy as we continue with our consecutive weekly releases. Next week will be our last update of this year.

Let's survive this home stretch of 2017 so that we can relax and enjoy the end of the year.

Now on to this week’s update information.

Environment variables can now be specified for command execution via plugin

Mackerel plugins can be used by adding the following configuration to the conf file.

[plugin.metrics.mysql]
command = "ruby /path/to/mysql.rb --username user --password password"

The agent executes the character array specified in command, and a post is made to Mackerel based on the resulting stdout (or exit code). This is a very useful mechanism.

With this week's update, you can now specify environment variables for command execution with this plugin mechanism. Below is the specification method.

[plugin.metrics.mysql]
command = "ruby /path/to/mysql.rb"
env = { "MYSQL_USERNAME" = "user", "MYSQL_PASSWORD" = "password" }

We will be making updates, even for the official plugins that are currently provided, so that password configuration and such for middleware can be performed via environment variables.

This feature can be used with check plugins and metadata plugins as well!

The password strength policy was reviewed and now passwords with weak security strength can be used

With this week’s release, updates were made for the following two points.

  • The strength policy for password configuration was reviewed
  • Although it was previously impossible to set a password with weak security strength, this can now be ignored

Additionally, it’s currently possible to remove password authentication in Mackerel if configured with GitHub or Google. Hopefully you can utilize this in combination with the option to ignore the password strength check.

Updates for Mackerel related OSS

Updates for various Mackerel related OSS were made this week and the details follow below. As always, thank you to everyone who contributed!

mackerel-agent v0.48.1

  • Environment variables can now be specified for command execution via plugin (described above)
  • The cloud_platform option was added to select targets for gathering laaS meta information and obtaining custom_identifier

mackerel-agent-plugins v0.40.0

  • mackerel-plugin-h2o was added
  • [redis] The CONFIG command that is internally used by the plugin is now available even if it can not be used as is.
    • Configured with the -config-command option
  • [redis] Replication delay metrics are now obtainable

mkr v0.24.1

You can now configure the maximum number of attempts for host/service metric monitors etc.

Thanks to all of you, the Mackerel Advent Calendar that we talked about last week has been completely filled!

qiita.com (Japanese only)

Thank you!

Also, be sure to check out the advent calendar that Mackerel team CRE id:Soudai is working so hard on, not missing a single day and continuing to release blogs full of information and useful knowledge.

qiita.com (Japanese only)

I hope you enjoy these advent calendars leading up to Christmas.

Now onto this week’s update information.

Customized (expression) graphs can now be added to GraphBoard

"GraphBoard" is an interactive feature that lets the user freely arrange arbitrary role and service metrics graphs.

mackerel.io

With this week's update, we’ve increased the number of types of graphs that can be added to this GraphBoard. Customized graphs (expression graphs) can now be added!

  • Overlay server load statuses for the current week and the previous week and display as one graph
  • Display the calculated maximum value/average value/minimum value of the server load belonging to a particular role

Customized graphs are extremely useful and allow you to implement the above procedures by describing the function expression. We believe that putting this to use will be even easier than before.

Displaying customized graphs is an experimental feature. For more details, check out the following help pages.

mackerel.io

mackerel.io

You can now configure the max number of attempts for host/service metric monitors

As you can see in the image above, the "Max number of attempts before an alert is triggered" can be configured with host metric and service metric monitors.

Up until now, only the monitoring of “Duration” (average of the past n minutes) was supported. For example, if you wanted to have an alert occur when a threshold of 5 is consecutively exceeded, you would configure the following.

  • Threshold of 5
  • Duration of 3 minutes

However, with this configuration, not only would a progression of 756 (a 3min average of 6) lead to an alert occurrence, but also for a progression such as 1161 (in this case, also a 3min average of 6). For this reason, sometimes unintentional alerts would occur.

With this update, you can eliminate cases such as the one above by utilizing the newly added "max_check_attempts". We hope that you can use this feature together with "Duration” monitoring to eliminate unintentional alerts.

Our support window will be closed for the holidays

Both methods of contacting our support team, either through support@mackerel.io or the “contact our support team” option displayed in the upper right header when logged into Mackerel, will be closed during the following dates for the New Year’s holidays.

New Year’s holiday period :Thursday, December 28th, 2017 - Wednesday, January 3rd, 2018 (JST)

All inquiries received during this period will be handled sequentially starting from January 4th, 2018 (Thursday).

Please review your server monitoring settings and contact us if you have any concerns before the end of the year holidays begin!

Target URLs can now be specified directly with the mkr plugin installer etc.

Hello! Mackerel team CRE Inoue (id:a-know) here.

It’s December! And around this time every year, what the IT/Web industry refers to as "Advent Calendars" bustles with various themed blog posts, and Mackerel is no exception. Here is Mackerel’s Advent Calendar!

qiita.com (Japanese only)

But wait, here’s another calendar!

qiita.com (Japanese only)

Be sure to check out this calendar as well. Fellow team CRE Sone (id:Soudai) is going all out!

Now for this week’s update information.

Target URLs can now be specified directly with the mkr plugin installer

We recently made an announcement regarding the recently released command line tool mkr plugin installer feature, including usage details/application methods.

mackerel.io

With this week’s feature update, you can now directly specify a URL as shown below.

mkr plugin install https://www.example.com/mackerel-plugin-foobar.zip

Be sure to try out this plugin installer feature which allows you to easily install useful third-party plugins.

You can now select the API V2 in the PagerDuty notification channel settings

In the integration settings for PagerDuty, one of the notification channels available with Mackerel, you can now select the V2 API.

Although there is no functional difference at this time, it is possible that options only available with V2 will be added in future feature expansion. If this applies to you, please consider making the transition.

Updates for Mackerel related OSS

Updates for various OSS have been released and are listed below. To all of you out there who contributed or reported a problem, thank you very much!

mackerel-agent v0.47.3

  • The output message that occurs when there is a defect in the mackerel-agent.conf plugin command configuration has been improved.
  • The issue of the interface value not being obtained when the network counter is 8 digits or more has been fixed.

mackerel-agent-plugins v0.39.0

  • [docker] The valued obtained by Docker Blkio IOPS has been normalized from per Minute to per Second
  • [jvm] The issue of the option to monitor the remote jvm not functioning properly was fixed.
  • [kinesis-streams] Regarding metrics for the number of records and processing bytes, outputting the "statistics of one record" was changed to "total amount per minute".

Mackerel × Twilio Hands-On Seminar #2

Earlier this year in April, the first Mackerel × Twilio Hands-On Seminar was held and went down as a huge success. Now, the second edition of the seminar is in motion and scheduled to be held on Wednesday, December 13th.

mackerelio.connpass.com (Japanese only)

For the venue of the event, we’ll be renting out the space of RecoChoku, a company which extensively uses the Mackerel - Twilio Integration. This will be a valuable opportunity to learn more about a variety of things ranging from the basics of why server monitoring is important to installing the agent and receiving phone notifications through Twilio.

The event is almost at capacity, but because applications as chosen through a lottery system, there’s still a good chance that you’ll be accepted even if you apply now. You definitely don’t want to miss out on this opportunity!

Azure Integration now supports Virtual Machines

I can’t believe that it’ll already be December next week. It feels like we just celebrated Halloween...

But with December comes the end of year Advent Calendar! Thanks to all of you, Mackerel’s Advent Calendar is filling up little by little.

qiita.com (Japanese only)

There are still spots available, so please join us!

Now onto this weeks update information.

Azure Integration now supports Virtual Machines

Mackerel's integration feature lets you monitor cloud services without having to introduce an agent, and Azure integration now supports Virtual Machines!

For information on available metrics, check out the the following help page.

mackerel.io

This feature is also available if currently using virtual machines with the agent installed.

Even if you use both, they can be managed as one host (the metrics collected with the integration feature will be integrated as "custom metrics" of that host), so it’s possible to aggregate metrics in Mackerel that can’t be collected by the agent alone. And of course, a usage fee for only one host will be charged. Be sure to give it a try!