Posting Service Metrics with fluentd

"Service Metrics" is a construct used for visualizing and monitoring metrics that are not directly tied to any specific server, making it possible, for example, to visualize an entire web service's response time and status code distribution, and check if response time is slowing down or if the error rate is going up.

Using fluent-plugin-mackerel service metrics can be posted to Mackerel via fluentd.

Next we'll introduce two concrete examples of posting service metrics.

Setup

First we'll need to install fluentd and fluent-plugin-mackerel:

$ gem install fluentd fluent-plugin-mackerel

Posting service metrics

Settings for posting service metrics with fluent-plugin-mackerel are as follows.

<match ...>
  @type mackerel
  api_key <API key of the organization that will be posted to>
  service <name of the service that will be posted to>
  remove_prefix
  metrics_name http_status.${out_key}
  out_keys <keys of the metrics that will be posted, e.g. 2xx_count,3xx_count,4xx_count,5xx_count>
</match>

Posting custom host metrics

With fluent-plugin-mackerel you can post custom host metrics. The settings for which are as follows.

If directly specifying the host ID

If you will be writing the host ID (view glossary) in the configuration file, your settings will need to be as follows.

<match ...>
  @type mackerel
  api_key <API key of the organization that will be posted to>
  hostid xyz
  metrics_name http_status.${out_key}
  out_keys <keys of the metrics that will be posted, e.g. 2xx_count,3xx_count,4xx_count,5xx_count>
</match>

It's also possible to specify a file where the host ID is saved. This is convenient when posting to Mackerel directly from a host on which the mackerel-agent is operating.

<match ...>
  @type mackerel
  api_key <API key of the organization that will be posted to>
  hostid_path /var/lib/mackerel-agent/id
  metrics_name http_status.${out_key}
  out_keys <keys of the metrics that will be posted, e.g. 2xx_count,3xx_count,4xx_count,5xx_count>
</match>

Example usages

Here we will introduce two concrete examples of posting service metrics.

Post the number of requests for each status code from Nginx's access log to service metrics.

In this example we'll make a graph like the one shown below that counts the number of requests for each status code.

f:id:mackerelio:20140917011215p:plain

First, we'll make it so that the access log from Nginx outputs in LTSV format.

log_format ltsv "time:$time_local"
                "\thost:$remote_addr"
                "\tforwardedfor:$http_x_forwarded_for"
                "\treq:$request"
                "\tstatus:$status"
                "\tsize:$body_bytes_sent"
                "\treferer:$http_referer"
                "\tua:$http_user_agent"
                "\treqtime:$request_time"
                "\tcache:$upstream_http_x_cache"
                "\truntime:$upstream_http_x_runtime"
                "\tvhost:$host";

access_log  /var/log/nginx/access.log  ltsv;

fluent-plugin-datacounter will be used so that the number of requests are counted for each status code, so we will need to install this plugin.

gem install fluent-plugin-datacounter

Then we will prepare the configurations for the fluentd that will read and aggregate Nginx's access log and post it to Mackerel.

# Reads LTSV format logs
<source>
  @type tail
  format ltsv
  time_format %d/%b/%Y:%H:%M:%S %z
  path /var/log/nginx/access.log
  pos_file /var/log/nginx/access_log.pos
  tag access.nginx
</source>

# Aggregates for each status code with fluent-plugin-datacounter
<match access.nginx>
  @type datacounter
  count_interval 1m
  count_key status
  aggregate all
  tag nginx.status
  pattern1 2xx ^2\d\d$
  pattern2 3xx ^3\d\d$
  pattern3 4xx ^4\d\d$
  pattern4 5xx ^5\d\d$
</match>

# Post to service metrics with fluent-plugin-mackerel
<match nginx.status.**>
  @type mackerel
  api_key <API key of the organization that will be posted to>
  service <name of the service that will be posted to>
  remove_prefix
  metrics_name access_num.${out_key}
  out_keys 2xx_count,3xx_count,4xx_count,5xx_count
</match>

And that's it! We just generated the graph shown below. Now by clicking the cog in the upper right-hand corner of the graph, you can configure the graph to your liking, for example changing it from a line graph to a stacked graph.

f:id:mackerelio:20140917011221p:plain

Post ELB-related metrics from AWS CloudWatch's metrics to service metrics

With AWS CloudWatch we can retrieve the number of ELB requests (in total as well as for each status code) and metric data such as response time. By posting that data as service metrics, the following graphs can be obtained.

fluent-plugin-cloudwatch will be used to read the metric data from AWS CloudWatch, so we will need to install this plugin.

gem install fluent-plugin-cloudwatch

Next we will prepare the configurations for the fluentd that will retrieve the ELB data from CloudWatch and post it to Mackerel.

# Retrieves data from CloudWatch using fluent-plugin-cloudwatch
<source>
  @type cloudwatch
  tag  cloudwatch-latency.elb01
  aws_key_id <aws-key>
  aws_sec_key <aws-sec-key>
  cw_endpoint monitoring.ap-northeast-1.amazonaws.com

  namespace AWS/ELB
  metric_name Latency
  dimensions_name LoadBalancerName
  dimensions_value elb01
  interval         60
  period           60
  delayed_start    true
</source>

<source>
  @type cloudwatch
  tag  cloudwatch-status.elb01
  aws_key_id <aws-key>
  aws_sec_key <aws-sec-key>
  cw_endpoint monitoring.ap-northeast-1.amazonaws.com

  namespace AWS/ELB
  statistics Sum
  metric_name HTTPCode_Backend_2XX,HTTPCode_Backend_3XX,HTTPCode_Backend_4XX,HTTPCode_Backend_5XX,RequestCount
  dimensions_name LoadBalancerName
  dimensions_value elb01
  interval         60
  period           60
  delayed_start    true
</source>

# Post them to Mackerel as service metrics
<match cloudwatch-status.*>
  @type copy
  <store>
    @type            mackerel
    api_key          <api-key>
    service          <service name to post metrics>
    remove_prefix
    metrics_name     ${[1]}-status.${out_key}
    out_keys         HTTPCode_Backend_2XX,HTTPCode_Backend_3XX,HTTPCode_Backend_4XX,HTTPCode_Backend_5XX
  </store>
  <store>
    @type            mackerel
    api_key          <api-key>
    service          <service name to post metrics>
    remove_prefix
    metrics_name     ${[1]}-count.${out_key}
    out_keys         RequestCount
  </store>
</match>

<match cloudwatch-latency.*>
  @type            mackerel
  api_key          <api-key>
  service          <service name to post metrics>
  remove_prefix
  metrics_name     ${[1]}-latency.${out_key}
  out_keys         Latency
</match>

-

If you have any questions please contact our support team at support@mackerel.io