Mackerel team engineer id:itchyny here (Mackerel is a server monitoring service and mackerelagent is a daemon program to collect and post server's metrics).
”When mackerelagent is installed, the loadavg rises every 7 hours”
Recently, we’ve received several inquiries like the one above from multiple customers. So I tried it out for myself, and sure enough, this issue was reproduced. I installed mackerelagent on EC2t2.micro, configured basic log and process monitoring, and left it for a few days.
Indeed, the loadavg rose approximately every 7 hours. I did not configure cron for this cycle and no processing was done even within mackerelagent. However, the loadavg's peak value increased the more plugins were added.
This entry explains the cause of this phenomenon.
To understand the reasons why the loadavg rises, we first need to understand how the loadavg itself is calculated. So let’s first go over how Linux calculates loadavg.
The Linux loadavg is an exponential moving average value of the total number of “runnable” processes in the run queue and disk I/O waited (uninterruptible) processes. The following are statuses of the Linux process.
% man ps  grep A 10 "^PROCESS STATE" PROCESS STATE CODES Here are the different values that the s, stat and state output specifiers (header "STAT" or "S") will display to describe the state of a process: D uninterruptible sleep (usually IO) R running or runnable (on run queue) S interruptible sleep (waiting for an event to complete) T stopped, either by a job control signal or because it is being traced W paging (not valid since the 2.6.xx kernel) X dead (should never be seen) Z defunct ("zombie") process, terminated but not reaped by its parent
The loadavg is the value that represents the number of processes with the statuses R
and D
smoothed by the exponential weighted average.
Let's express the total number of running and uninterruptible processes subject to the loadavg as with as a function for time. If the sampling interval is and the time constant is , the exponential moving average of the number of processes, that is, the exponentially decaying weighted average can be written as follows ^{1}. $$ \begin{align} L(t) &= \big(1  e^{\Delta t / T}\big) \big( p(t) + e^{\Delta t / T} p(t  \Delta t) + e^{2\Delta t / T} p(t  2 \Delta t) + \cdots \big) \\ &= \big(1  e^{\Delta t / T}\big) p(t) + e^{\Delta t / T} L(t  \Delta t) \end{align} $$ The exponential moving average value of accumulated over time can be expressed with the weighted average of the previously calculated value and the current value . Even if you haven’t kept a history of the number of processes up until this point, we can calculate a new loadavg from the previous loadavg and the current number of processes.
Based on this recurrence formula, by selecting an appropriate constant and comparing the simulated value () and the actual value from Linux (Run yes > /dev/null&
on EC2 t2.micro and drop that process after 10 minutes), the following results were obtained.
Let’s confirm that the loadavg is actually being calculated based on the above recurrence formula while referring to the Linux source code. First, let’s look into kernel/sched/loadavg.c.
/* Variables and functions for calc_load */ atomic_long_t calc_load_tasks; unsigned long calc_load_update; unsigned long avenrun[3]; void calc_global_load(unsigned long ticks) { unsigned long sample_window; long active, delta; sample_window = READ_ONCE(calc_load_update); if (time_before(jiffies, sample_window + 10)) return; // ... active = atomic_long_read(&calc_load_tasks); active = active > 0 ? active * FIXED_1 : 0; avenrun[0] = calc_load(avenrun[0], EXP_1, active); avenrun[1] = calc_load(avenrun[1], EXP_5, active); avenrun[2] = calc_load(avenrun[2], EXP_15, active); WRITE_ONCE(calc_load_update, sample_window + LOAD_FREQ); // ... }
The loadavg values are stored in avenrun
. We can see that in order for this value to periodically update, the value of calc_load_update
is compared against jiffies
(a variable representing time, increasing by 1 every tick) while increasing by LOAD_FREQ
. In other words, loadavg is updated every LOAD_FREQ.
As described in include/linux/sched/loadavg.h, this value is 5*HZ+1
. Since HZ
is the value of how many ticks increase per second ^{2}, we can see that the loadavg is updated every 5 seconds plus one tick.
Now let’s look at calc_load
which is the actual value calculation.
/* * a1 = a0 * e + a * (1  e) */ static unsigned long calc_load(unsigned long load, unsigned long exp, unsigned long active) { unsigned long newload; newload = load * exp + active * (FIXED_1  exp); if (active >= load) newload += FIXED_11; return newload / FIXED_1; }
Ignore the branching statement and write it in one expression ^{3}.
avenrun[0] = ((FIXED_1  EXP_1) * active + EXP_1 * avenrun[0]) / FIXED_1;
Here, according to include/linux/sched/loadavg.h, EXP_1 = 1884
and FIXED_1 = (1<<11)
. By substituting for the recurrence formula obtained above, we get following.
$$
\begin{align}
L(t) = \big(1  e^{1/12}\big) p(t) + e^{1/12} L(t  \Delta t)
\end{align}
$$
Taking it into account that Linux uses values shifted by 11 bits (i.e. scaled by FIXED_1
) to calculate only by integral operations, and equals to roughly 1884, we can see that the calculation in Linux indeed matches the recurrence formula.
Loadavg 5, 15 can be checked in the same way.
The loadavg is updated approximately every 5 seconds using the number of processes in the run queue at that moment (calc_load_tasks
) and the recurrence formula. To be exact, it is calculated every 5*HZ+1
ticks. This +1
is the cause for the loadavg rising roughly every 7 hours.
mackerelagent opens the plugin processes to collect metrics every minute. Since recalculation of the loadavg takes place approximately every 5 seconds, the timing of when mackerelagent starts collecting metrics and the timing of the loadavg recalculation will periodically be in sync.
In an environment where HZ
is 1000, the loadavg is updated every 5.001 seconds. When multiplied by 5000, the cycle is 25005 seconds (or 6 hours, 56 minutes and 45 seconds) and the +1
accumulates and coincides with 5*HZ
. This cycle is not a multiple of 60 seconds which is the metric collection interval of mackerelagent, however, a phenomenon occurs when the timing of opening processes and the loadavg update roughly overlap (off by 0.003 seconds) every 6 hours and 57 minutes.
In an environment where HZ
is 250, repeating 5*HZ+1
1250 times comes out to 6255 seconds (or 1 hour 44 minutes 15 seconds). It’s normal for the loadavg to go up during this cycle, but we can’t ignore the fact that the metric collection timing for mackerelagent shifts by 0.012 seconds each cycle. The value is multiplied by 4, that is, 6 hours and 57 minutes, which is a multiple of 60 seconds and this overlaps perfectly with mackerelagent 's metrics collection timing. In my own experience HZ
equaled 250
. If you look at the graph at the beginning of this entry, you can see that there are peaks of uniform height with a cycle of 6 hours 57 minutes. You can also see that there are smaller peaks 1 hour 44 minutes before and after these peaks. This is a phenomenon derived from HZ
being 250.
So why isn’t the loadavg update interval exact 5 seconds?
Well, in Linux, various processes are running in their own cycle.
And if a process with a 5 second interval exists, there is a possibility that it may sync with the loadavg recalculation timing. In order to prevent this phenomenon of the loadavg unintentionally rising due to this timing coincidence, the cycle is slightly shifted from 5 seconds. For more details on this, take a look at the mailing list or the patch commit message where 1
was added to 5*HZ
.
The loadavg update interval in Linux is intentionally slightly deviated from 5 seconds. This slight deviation accumulates and is a multiple of 60 seconds in a cycle of about 6 hours 57 minutes. mackerelagent opens plugin processes every 60 seconds, and the timing of this periodically coincides with the loadavg recalculation. This is the cause of the phenomenon in which the loadavg periodically spikes when mackerelagent is installed. This phenomenon has also been reported with other monitoring tools such as collectd and Telegraf
References
 Avast Blog: Investigation of regular high load on unused machines every 7 hours
 This blog post goes over the same phenomenon that occurs in environments using collectd.
 Linux Load Averages: Solving the Mystery
 Here, the reason why the number of uninterruptible processes is included in the loadavg is explained following a little bit of back story.

You can verify that the sum of the coefficients is 1. This is a necessary condition for being a weighted average.↩

In this entry
HZ
refers to the kernel timer frequencyCONFIG_HZ
, which must be distinguished from the userland frequencyUSER_HZ
.USER_HZ
is the value obtained fromgetconf CLK_TCK
and/proc/stat
, which is defined as 100 on x86. The kernel timer frequency can be checked withgrep "CONFIG_HZ=" /boot/config$(uname r)
or by watching the increment ofwatch n1 "grep ^jiffies /proc/timer_list  head n1"
.↩ 
The branching procedure is required so that a system with no processes has a loadavg of 0.0 and a system with one process has 1.0. If this procedure does not exist, the loadavg will never reach 1.0, and if added without branching, it will not reach 0.0 on idle. It used to be that the value was rounded off, but the problem of not reaching 0.0 in an idle state was identified and it now the value is rounded up when the number of processes increases. For details, refer to the patch and the mailing list. ↩