New Relic – CPU utilization (%) and Load Common

At AWS re:Invent 2025, I had the chance to present a quick demonstration on the New Relic sales space that opened the stage to some AWS Heroes. In Linux environments, directors usually depend on CPU utilization and cargo common metrics to find out whether or not an occasion is appropriately sized. An outsized occasion that sits idle wastes assets and drives up the cloud invoice, whereas an undersized occasion pushed to its limits can degrade utility efficiency—finally impacting an organization’s income.

To set the stage for my demo, I started with the Abstract tab’s first graph, which shows CPU utilization as a proportion. I began a easy CPU-bound process utilizing the sure command, which repeatedly outputs the character “y”. Then, I moved on to a extra advanced workload: fio with 32 threads performing synchronous, direct I/O disk writes:

fio --rw=write --ioengine=psync --direct=1 --bs=1M --numjobs=32 --name=check --filename=/tmp/x --size=10G --thread

One might count on the sure command to saturate the CPU, and this fio workload to be I/O-bound, however surprisingly, each duties present 100% CPU utilization:

The load common graph would not assist and reveals the 32 threads on the CPU:

We do not have extra particulars on the load common. The Course of tab reveals sure, however with solely low CPU utilization, and never fio:

What might assist element the load common is the method state, not gathered right here. The R state is actual CPU utilization, for operating duties, just like the 12.5% of sure. The fio processes are largely doing disk I/O within the D state, which does not make the most of CPU, regardless of what’s displayed right here.

Again to the CPU utilization graph, I can get extra particulars. I open the question and as an alternative of cpuPercent I show the main points:

SELECT --average(cpuPercent),
 common(cpuUserPercent), common(cpuSystemPercent),  -- operating
 common(cpuStealPercent),                            -- hypervisor
 common(cpuIdlePercent), common(cpuIOWaitPercent)   -- idle (incl. wait I/O)
FROM SystemSample WHERE (entityGuid = 'NzM2NzA3MXxJTkZSQXxOQXw0MDAyNTY2MTYyODgwNDkyMzM0') 
TIMESERIES AUTO SINCE 5 minutes in the past UNTIL now

The 100% CPU utilization was in “Steal” for the operating sure command as a result of I’ve run that on an overprovisioned digital machine the place the hypervisor provides just one/4th of the CPU cycles, and was in “IO Wait” for fio when it was ready for IO completion relatively than operating in CPU:

To clarify this “IO Wait” and that it is rather like “Idle”, I’ve began sure once more whereas fio was operating and the “IO Wait” disappeared:

The reason being that “IO Wait” is accounted when a CPU is idle, as a result of the method that was operating waits on an IO name, and no different course of needed to run on CPU. Then, the method stays scheduled on the CPU with this state. But when one other course of involves run in CPU then the “IO Wait” will not be accounted anymore. The CPU utilization (%) will not be the state of the processes, however the state of the processor:

• when a course of is operating on a processor’s threads, it’s in R state and counts in cpuUserPercent or cpuSystemPercent relying whether it is operating in userspace (the applying) or kernel (system name). If the hypervisor preempted the CPU cycles, it’s reported as “Steal”
  • when a course of is on an uninterruptible name and never scheduled out of the processor’s thread, it’s in D state and counts as “IO Wait”
  • when the method within the processor’s thread it’s ready on one thing else, it’s accounted as “Idle”

Again to the load common, the rationale why the D state (“IO Wait”) is accounted along with the R state (“Consumer”, “Sys” and “Steal”) is seen within the loadavg.c code:

 * The world load common is an exponentially decaying common of nr_running +
 * nr_uninterruptible.
 *
 * As soon as each LOAD_FREQ:
 *
 *   nr_active = 0;
 *   for_each_possible_cpu(cpu)
 *  nr_active += cpu_of(cpu)->nr_running + cpu_of(cpu)->nr_uninterruptible;
 *
 *   avenrun[n] = avenrun[0] * exp_n + nr_active * (1 - exp_n)

Nonetheless, an important remark is the one which explains that it’s a foolish quantity:

/*
 * kernel/sched/loadavg.c
 *
 * This file accommodates the magic bits required to compute the worldwide loadavg
 * determine. Its a foolish quantity however individuals assume its essential. We undergo
 * nice pains to make it work on massive machines and tickless kernels.
 */

The load common metric traces again to early Unix techniques (Seventies), the place it mirrored how busy a system was by counting processes within the run queue. At the moment, computer systems used periodic scheduling ticks—even whereas idle—to maintain observe of time and processes. Sysadmins would see a load common of 1 on a single CPU as one course of operating or ready, thus equating it to CPU utilization. At the moment, with multi-core processors, subtle scheduling, and tickless kernels, the load common is a far much less dependable indicator of actual‑time CPU utilization and is usually misunderstood with out contemplating the actual state of the processes.

The uninterruptible state D will not be essentially linked to disk exercise. For instance, asynchronous disk I/O operations that accumulate accomplished IO don’t enter the D state. I demonstrated this by altering the fio job from psync to async, observing an identical I/O throughput and charge however much less “IO Wait” and a decrease load common. Moreover, some system calls can seem as “IO Wait” and improve the load common, even when they’re idle and innocent. I additionally confirmed this by launching a thousand processes with {vfork();sleep(300);}.

Step one when utilizing the New Relic dashboard for Linux is to switch ‘cpuPercent’ within the CPU utilization charts with extra detailed metrics:

• cpuUserPercent and cpuSystemPercent for duties operating in CPU, respectively, for consumer (utility) and system (kernel) code.
• cpuStealPercent, cpuIOWaitPercent, cpuIdlePercent for idle CPU, as a result of the hypervisor did not enable duties to run (and steals CPU cycles), or no duties has one thing to run (with or with out a process ready on uninterruptible name)

Keep in mind, the load common will not be a dependable metric as a result of it doesn’t completely replicate actively operating processes. It could additionally embody varied varieties of waits, however not all. The ‘IO wait’ proportion doesn’t point out excessive IO exercise. As an alternative, it reveals the CPU is idle whereas many processes are ready for IO operations. In cloud environments the place minimizing prices by lowering idle CPU time is essential, you must concentrate on CPU utilization for consumer and system processes, excluding ready duties, when sizing an occasion. An inexperienced consumer would possibly misread the 100% utilization proven in my fio demo as an indication the occasion is simply too small, whereas the truth is, it is the alternative.