Posting user-defined custom metrics

In addition to standard host metrics which are automatically posted to Mackerel by the agent such as loadavg5, cpu.user, etc., any user-defined values or “custom metrics” can be configured to be periodically posted to Mackerel.

By entering a measured value output command in a format compatible with sensu (a plugin which will be mentioned later), that output will be sent together with default metrics to Mackerel and will be displayed in a separate graph.

We also are offering an official plugin pack which you can learn more about here: Using the official plugin pack to visualize middleware metrics

Additionally, to develop a plugin using github.com/mackerelio/go-mackerel-plugin-helper (a helper library that is used in our official plugins), please refer to creating custom metrics plugins using go-mackerel-plugin-helper.

Configuration

Add the item in the agent configuration file like the example below.

[plugin.metrics.vmstat]
command = "ruby /path/to/vmstat-metrics.rb"
user = "SOME_USER_NAME" # optional
  • Item name: Used only in the configuration file and not used by the agent. It is necessary that the item name both begins with “plugin.metrics.” and contains exactly two dots (.)
  • command: A command which mackerel-agent runs periodically and whose output is used as metric data. This command must follow the specifications mentioned below.
    • It is also possible to transfer an array to command. As shown in the example above, you can specify command as: command = [ "ruby", "/path/to/vmstat-metrics.rb" ]
    • If an array is transferred, it will not run via a shell and escaping will be unnecessary.
  • user(optional): The user to run command can be specified with command. Specifying this user is not necessary. If a user is not specified, the executing user of mackerel-agent will also be the executing user of command.

Caution: Using PowerShell Script

Depending on the Windows PowerShell Script execution policy, plugins written in PowerShell Script may not be able to be executed. In that case, prepare a batch file as shown below and assign it to the plugin’s command and it will then be able to be executed.

@echo off
cd %~dp0

powershell Set-ExecutionPolicy RemoteSigned
powershell .\sample-plugin.ps1

Posting metrics

The command should output in the following format (\t is the tab character):

{metric name}\t{metric value}\t{epoch seconds}
  • Metrics whose names are the same up to the last dot (.) will be grouped together and charted in one graph.
  • All custom metrics are given the prefix “custom.” automatically.
  • Characters which can be used in custom metric names include any alphanumeric characters as well as hyphens (-), underscores (_), and dots (.).

Example: if posting metrics named example.foo and example.bar:

A graph named custom.example.* will appear in the host details page displaying data from both example.foo and example.bar.

Defining Graph Schema

You can specify the display settings of graphs created by posted custom metrics with the plugin’s command executed by the agent in JSON format.

Upon booting up mackerel-agent runs once for each of the custom metric commands specified in the configuration file with environment variable MACKEREL_AGENT_PLUGIN_META set to “1”. If the command’s output is in the format described below, the agent will recognize the output as the plugin’s meta information and will send the graph definition to Mackerel. With this feature, you can configure graph display settings without having to manage them on the web UI.

The first line of the output must be as shown below, otherwise mackerel-agent will assume that this plugin does not output the meta-information and won’t be able to set graph definition.

# mackerel-agent-plugin

Moving on from there, we will output the following configuration in JSON.

{
  "graphs": {
    {graph}: {
      "label": GRAPH_LABEL,
      "unit": UNIT_TYPE
      "metrics": [
        {
          "name": METRIC_NAME,
          "label": METRIC_LABEL
        },
        ...
      ]
    },
    GRAPH_NAME: ...
  }
}
…

The following is a description of each item.

Key Description
graphs.{graph}.label The label for the graph corresponding to custom metrics {graph}.* . {graph} may contain dots (.).
graphs.{graph}.unit The type of value displayed in the graph corresponding to custom metrics {graph}.* . Must be one of the following: “float”, “integer”, “percentage”, “bytes”, “bytes/sec”, or “iops”.
graphs.{graph}.metrics An array of metric definitions of custom metrics {graph}.* .

Metric definitions contain the following items:

Key Description
name Specifies that this definition corresponds to custom metric {graph}.{name} . {name} cannot contain dots (.). Any alphanumeric characters, hyphen (-), or underscore (_) can be used(/[-a-zA-Z0-9_]/). Additionally, wildcard characters (* and #) can also be used. For more details, refer to the API specs (v0) / Posting graph definitions help page.
label The label of the series corresponding to custom metric {graph}.{name} .
stacked Indicates whether or not the series of custom metrics {graph}.{name} is in stacked display mode. If false it will be displayed in line segment mode.

Example in Ruby

Here is a little example in Ruby, which posts the rolls of 6-sided and 20-sided dice to Mackerel.

if ENV['MACKEREL_AGENT_PLUGIN_META'] == '1'
  require 'json'

  meta = {
    :graphs => {
      'super.dice' => {
        :label   => 'My Dice',
        :unit    => 'integer',
        :metrics => [
          {
            :name  => 'd6',
            :label => 'Die (d6)'
          }, {
            :name  => 'd20',
            :label => 'Die (d20)'
          }
        ]
      }
    }
  }

  puts '# mackerel-agent-plugin'
  puts meta.to_json
  exit 0
end

puts [ 'super.dice.d6',  rand( 6)+1, Time.now.to_i ].join("\t")
puts [ 'super.dice.d20', rand(20)+1, Time.now.to_i ].join("\t")

With this plugin you can create a graph like the one shown here.

f:id:mackerelio:20140807163034p:plain

Update History

  • 2014-11-17
    • Available characters specified for metric name.
  • 2014-10-02
    • Removal of “experimental” from “Defining Graph Schema”.
  • 2014-09-05
    • Deletion of “type: designate a metric” in settings.
    • Addition of descriptions for official plugin package.
  • 2014-08-22
    • Version 0.12 support for content in “Defining Graph Schema(experimental)”.
  • 2014-08-07
    • Addition of “Defining Graph Schema(experimental)”.
  • 2014-05-27
    • Item name in “configuration” changed from “sensu.checks” to “plugin.metrics”.
    • Update “Posting metrics” to meet the specification change where metric names are given the “custom.” prefix.

-

If you have any questions please contact our support team at support@mackerel.io