Check plugins - check-log

check-log is a plug-in that monitors log files on the server, including applications and middleware.

Pattern matching using regular expressions is performed on the output differences of the monitored log files, and alerts are notified.

Monitoring Specifications

Logs subject to monitoring

The check-log plugin records the number of bytes read from the monitored file and the inode number (Linux only) in the State file each time it runs. On subsequent runs, it monitors only the differences since the previous run.

Regarding the corresponding rotation system

The check-log plugin supports the create method (creating a new file with the same name after moving the file) as the rotation method for monitored logs. For the copytruncate method (deleting the contents of the source file after copying), it does not track the copied file during rotation and only checks the contents of the original file from the beginning after its contents have been deleted.

Configurable options

Option Short Description Default
--file -f Specify the path of the file to be monitored (can be specified in glob format) *1
--pattern -p Specify the pattern of error wording to be detected with a regular expression *2
--exclude -E Specify patterns to exclude from detection using regular expressions *2
--warning-over -w Raise a Warning alert if the number of rows matching the detection pattern exceeds the specified value 0
--critical-over -c Raise a Critical alert if the number of rows matching the detection pattern exceeds the specified value 0
--warning-level Raise a Warning alert if the value extracted by the detection pattern exceeds the specified value
--critical-level Raise a Critical alert if the value extracted by the detection pattern exceeds the specified value
--return -r Notify of log lines that match the pattern with an alert *3
--search-in-directory Specify the directory path where the monitored files are located (in Windows environment, use with --file-pattern)
--file-pattern -F Specify files to be monitored by regular expression *4
--icase -i Case-insensitive matching
--state-dir -s Specify the directory path where the State file is to be saved
--no-state All logs are checked without State file
--encoding Specify character encoding of monitored files
--missing Specify alert level if monitored files are missing *5 UNKNOWN
--check-first Check all contents at the time of the first check of the file *6
--suppress-pattern Suppress the display of detection patterns on the host detail screen
  • *1
    • When executing via the command line or using string format for command specification in mackerel-agent.conf, glob patterns like --file "/var/log/*.log" must be enclosed in single quotes.
  • *2
  • *3
    • To include detected log lines in alerts, add the --return option.
    • Content exceeding 1024 characters will be truncated by mackerel-agent.
  • *4
    • Directory paths cannot be specified using regular expressions.
  • *5
    • This is ignored if --search-in-directory and --file-pattern are used to specify the monitoring target.
  • *6

About state file

If the --state-dir option is not specified, the agent creates a directory hierarchy matching the monitored log file's directory structure under the following default locations for each OS, and creates files in the format {monitored file}-<hash string>.json. For example, if logs under /var/log are being monitored, they will be saved under the directory /var/tmp/mackerel-agent/check-log/var/log.

  • For Linux
    • When executed via mackerel-agent
      • /var/tmp/mackerel-agent/check-log
    • When executed manually
      • /tmp/check-log
  • For Windows
    • When executed via mackerel-agent
      • C:\Windows\Temp\check-log
    • When executed manually
      • The check-log folder under the logged-in user's Temp folder (Check the location of the Temp folder via the Windows environment variable TEMP)

Example configurations

for Linux

The configuration for detecting ERROR output to /var/log/access.log is as follows

[plugin.checks.access_log]
command = ["check-log", "--file", "/var/log/access.log", "--pattern", "ERROR", "--return"]

To verify the above settings by executing them directly from the terminal or similar, run the following command:

check-log --file /var/log/access.log --pattern "ERROR" --return --no-state

for Windows

The configuration to detect when a log containing the string "ERROR" is output to C:\log\access.log is as follows. Although the path separator character \ in Windows environments is normally a single character, you must specify two characters when configuring it in mackerel-agent.conf.

[plugin.checks.access_log]
command = ["check-log", "--file", "C:\\log\\access.log", "--pattern", "ERROR", "--return"]

To verify the above settings by executing them directly from the Command Prompt or similar, run the following:

check-log --file "C:\log\access.log" --pattern "ERROR" --return --no-state

Tips

Specifying thresholds and exclusion patterns for occurrence frequency

Using --warning-over or --critical-over allows alerts to be triggered only when the number of logs output since the previous run exceeds the specified number of lines. The following example triggers a WARNING if ERROR appears in /var/log/nginx/error.log three or more times, and a CRITICAL if it appears ten or more times.

[plugin.checks.access_status]
command = ["check-log", "--file", "/var/log/nginx/error.log", "--pattern", "ERROR","--warning-over", "3", "--critical-over", "10", "--return"]

You can also specify exclusion patterns using the --exclude option. For example, the following configuration monitors Nginx access logs to check for 4xx and 5xx errors. Specifying --exclude excludes access to "robots.txt".

[plugin.checks.access_status]
command = ["check-log", "--file", "/var/log/nginx/access.log", "--pattern", "HTTP/1\.[01]\" [45][0-9][0-9] ", "--exclude", "GET .*?robots\.txt HTTP/1\.[01]", "--return"]

Alerts when multiple patterns are matched by AND conditions

Specifying multiple --pattern or --exclude options creates an AND condition, allowing you to target logs matching all specified conditions for alerts.

The following example triggers an alert when logs containing the strings "PRODUCTION" and "ERROR" are detected.

[plugin.checks.access_log]
command = ["check-log", "--file", "/var/log/access.log", "--pattern", "PRODUCTION", "--pattern", "ERROR", "--return"]

Alerts when any of the OR conditions are matched

When specifying multiple conditions separated by pipes | with --pattern or --exclude, they are treated as OR conditions, allowing you to target logs matching any specified condition for alerts.

The following example triggers an alert when logs containing the strings "FATAL" or "ERROR" are detected.

[plugin.checks.access_log]
command = ["check-log", "--file", "/var/log/access.log", "--pattern", "FATAL|ERROR", "--return"]

Treating Regular Expression Metacharacters as Literals

When using --pattern or --exclude, if you want to treat regular expression metacharacters (characters with special meanings like []) as literal characters, you must escape them by placing the escape character \ before the metacharacter.

Below is a configuration example for detecting logs containing the string [ERROR].

When the --pattern argument is enclosed in single quotes ', a single escape character (\) is required before each metacharacter.

command = ["check-log", "--file", "/var/log/access.log", "--pattern", "\[ERROR\]", "--return"]

If the --pattern argument is enclosed in double quotation marks ", two escape characters (\\) are required before each metacharacter.

command = ["check-log", "--file", "/var/log/access.log", "--pattern", "\\[ERROR\\]", "--return"]

Additionaly if the pattern itself contains a double quotation mark ", and the --pattern argument is enclosed in double quotation marks ", three escape characters (\\\) are required before each double quotation mark contained --pattern.
The following example detects logs containing the string "ERROR".

command = ["check-log", "--file", "/var/log/access.log", "--pattern", "\\\"ERROR\\\"", "--return"]

Targeting multiple log files

When targeting multiple files instead of a single file, the --file option supports glob patterns, while the --file-pattern option supports regular expressions. The following example targets log files matching /var/log/*.log.

[plugin.checks.access_log]
command = ["check-log", "--file", "/var/log/*.log", "--pattern", "FATAL"]

Additionally, you can specify the conditions for filenames to monitor using regular expressions with the --file-pattern option. Note that you cannot specify directory paths with regular expressions. Set monitors for each directory.

The following example targets files matching the regular expression /var/log/access.log.\\d{4}-\\d{2}-\\d{2} (such as /var/log/access.log.2014-09-17).

[plugin.checks.access_log]
command = ["check-log", "--file-pattern", "/var/log/access.log.\\d{4}-\\d{2}-\\d{2}", "--pattern", "FATAL"]

When using regular expressions with the --file-pattern option to specify files to monitor in a Windows environment, use the --search-in-directory option to avoid conflicts between the directory separator \ and regular expression escaping.

[plugin.checks.access_log]
command = ["check-log", "--search-in-directory", "C:\\log\\", "--file-pattern", "access.log.\\d{4}-\\d{2}-\\d{2}", "--pattern", "ERROR", "--return"]

To trigger only warning alerts

By default, check-log sets --warning-over and --critical-over to 0. Since the Critical alert level takes precedence, if these options are not specified, a Critical alert will be triggered upon detecting even a single log matching the conditions.

To ensure all alerts are triggered at the Warning level, specify a value for --critical-over that is absolutely not exceeded.

command = ["check-log", "--file", "/var/log/access.log", "--pattern", "ERROR", "--critical-over", "9999", "--return"]

Troubleshooting

Log lines matching the condition are output but not detected

Please review your settings, etc., as the following cases are often seen.

  • Make sure the --check-first option is specified.
    • If not specified, the contents of the monitored file will not be checked the first time it is checked.
    • The same applies to the initial check when a file is switched due to log rotation, etc.
  • Make sure that the detection pattern specified in the --pattern option is correct.
  • Make sure that the create method is used as the rotation method.
    • If logs are rotated using the copytruncate method, it is not possible to check for logs added from the plugin is run before rotation to rotation is run.
  • Please verify that the relevant log lines contain line breaks.
    • Logs without a line break code at the end of the line cannot be detected.

Also, if the file size when check-log is executed after rotation is larger than the file size when check-log is executed before rotation, rotation cannot be detected, and detection cannot be performed for logs written up to the size checked by check-log before rotation.

"LOG UNKNOWN: unexpected end of JSON input" alert is issued

The State file in which the check-log plugin records the monitoring status may be corrupted.

Deleting or renaming the State file may solve the problem. Please refer to About the State file for details. Also, since the monitoring state is reset after this action, the behavior will be the same as when it is executed for the first time.

Repository

https://github.com/mackerelio/go-check-plugins/tree/master/check-log