There's More Than CPU and Average

WhatThe percentage of time the CPU is idle but stalled waiting on an I/O operation to finish.

Why it mattersHigh CPU wait looks like a CPU problem on a graph, but it isn't one.

Best statisticMax and 95th percentile over short intervals — averages hide brief spikes.

Might really meanThe disk, network, or a downstream database is slow. Don't add CPU to fix this.

WhatTime a virtual CPU wanted to run but the hypervisor gave the cycles to another tenant instead.

Why it mattersOnly exists in virtualized/cloud environments, and is invisible unless you look for it.

Best statisticAny value consistently above 0% is worth investigating; max during peak periods.

Might really meanNoisy-neighbor contention on shared infrastructure — not your app, not your code.

WhatHow many processes are waiting for a turn on the CPU, not just how busy the CPU is.

Why it mattersA CPU can show 60% utilized and still have a long queue if work isn't scheduled evenly.

Best statisticMax, and count of time spent above (core count).

Might really meanUnder-provisioned compute, or a single thread-bound process blocking others.

WhatNot just "how much RAM is used," but how much is actually available for new work (accounting for cache/buffers).

Why it matters"Used" memory numbers are often misleading because OSes use free RAM for caching.

Best statisticMinimum available memory, not average used.

Might really meanNothing yet — but watch swap closely if this trends down.

WhatHow much data has been pushed out of RAM onto disk because memory ran out.

Why it mattersDepends entirely on what the server does. On a general-purpose system, if you never swap, you likely have more memory than you need. On a database server, ANY swap activity severely degrades performance — the same metric, read completely differently, depending on the server's mission.

Best statisticMax, and any non-zero sustained value (weighted by server type).

Might really meanOn a DB server specifically — this is often confused with a "disk is slow" problem, because swapping makes disk I/O spike as a side effect. The real problem is memory.

WhatHow often the system has to fetch data from disk because it wasn't in memory.

Why it mattersA rising trend here often precedes visible slowness by hours or days.

Best statisticRate of change over time, not a single snapshot.

Might really meanMemory pressure building before it becomes obvious anywhere else.

WhatNumber of I/O requests waiting for the disk to become available.

Why it mattersA disk can report low "utilization %" and still have requests backing up.

Best statisticMax and 95th percentile.

Might really meanThe storage backend (SAN, cloud volume) is saturated — or, less obviously, a misconfigured network path to that storage (a mismatched router port, a failing I/O controller). Not every disk queue problem is the disk.

WhatHow long each I/O request actually takes to complete.

Why it mattersThroughput can look fine while individual requests are quietly getting slower.

Best statistic95th/99th percentile — this is exactly the kind of metric averages hide.

Might really meanShared storage contention, a failing disk, or a misconfigured volume.

WhatPackets that had to be resent because they weren't acknowledged the first time.

Why it mattersA small percentage sounds harmless but can represent significant repeated work.

Best statisticRate over short intervals, and spikes correlated with load.

Might really meanA couple of possibilities, and they point in different directions. It can mean the receiving application isn't accepting connections fast enough, and the sender is timing out. Or it can be a genuinely "flaky" path — one common, easy-to-miss cause: a packet size mismatch on a VPN'd circuit. A standard 1500-byte packet hitting a VPN gets split in two once the encryption header is added — one full-size packet and one tiny fragment — which shows up as retransmits and odd latency. Setting MTU to around 1400 on that path leaves room for the encryption header and avoids the split.

WhatNumber of incoming connection requests waiting to be accepted by the OS or app.

Why it mattersIf this queue fills, new connections get refused or dropped — silently, from the client's point of view.

Best statisticMax.

Might really meanThe application isn't accepting connections fast enough (see Idle Workers below — often the same problem seen from a different angle). It can also mean something adversarial or accidental is eating up available connections: a denial-of-service attempt, or an application that's slow to disconnect — or simply fails to disconnect — leaving connections tied up that should have been freed.

WhatHow many of the application's configured worker threads are free to pick up new work right now.

Why it mattersThis is the one most people don't check, and it's often the most direct measure of "can this app actually do work right now." It doesn't matter how much CPU, memory, or disk you have left if every worker is busy — the app is effectively down.

Best statisticMinimum, not average. An average of 40 idle workers can hide a period where it was zero for hours.

Might really meanIf workers are pegged busy but CPU/memory/disk all look normal, look upstream or downstream — something is flooding this app with connections or work, or something downstream is holding connections open too long. Worth asking directly: how does this application behave when idle workers hit zero? Most newer applications handle this very poorly until that specific failure mode has been deliberately coded for — it's rarely something that gets tested until it happens in production.

WhatWork items waiting for a thread to become free inside the application itself.

Why it mattersSimilar to idle workers, but measured inside the app rather than at the OS/network layer.

Best statisticMax and time-above-threshold.

Might really meanA slow downstream dependency (database, external API) is holding threads longer than normal, starving everything else.

WhatHow many of the app's available database connections are currently checked out and in use.

Why it mattersWhen this pool is exhausted, requests queue or fail even though the database itself may be perfectly healthy.

Best statisticMax, and time spent near the pool limit.

Might really meanEither genuine load growth, or connections not being released properly (a leak) — worth distinguishing before resizing the pool.

WhatHow often, and for how long, the runtime pauses the application to reclaim memory.

Why it mattersLong or frequent GC pauses look exactly like random, unexplained slowness or timeouts. Effective garbage collection tuning is its own skill — not a settings checkbox — and it's easy to under-invest in.

Best statisticMax pause duration, and total time spent in GC per interval.

Might really meanMemory pressure, a memory leak, or an application sized too small for its workload. Rule of thumb: if a restart is the only thing that ever fixes the slowness, look closely at garbage collection.

WhatThe response time experienced by your slowest 5% or 1% of requests.

Why it mattersEvery technology decision is ultimately a balance between cost and customer expectation — and response time is usually the stand-in for "customer expectation." The trouble is, it's genuinely hard to measure well: a meaningful average is close to impossible to define, and true end-user experience (their network, their device) is impossible to measure directly from the server side. Performance/SLA contracts almost always specify some response-time threshold anyway, however imperfectly measured — so it's worth making sure whatever you're measuring is actually the thing the contract refers to.

Best statistic95th and 99th percentile, tracked separately from average.

Might really meanIntermittent contention somewhere in the stack (any of the above) that only shows up under specific conditions.

WhatPercentage of requests failing outright, as opposed to just being slow.

Why it mattersObvious in isolation, but rarely looked at alongside the resource metrics above — the correlation is often the useful part.

Best statisticRate over short windows, correlated in time with the metrics above.

Might really meanWhatever's exhausted upstream (workers, connections, threads) is now hard-failing requests instead of queuing them.

WhatHow many users or sessions are active at a given moment, as opposed to total daily volume.

Why it mattersTwo days with the same total traffic can have very different peak concurrency, and concurrency — not volume — is usually what breaks things. Think morning login rush, everyone returning from lunch at once, or the start/end-of-day reporting crunch — short, predictable peaks that a daily average will never show.

Best statisticMax, at the finest time resolution you can get (seconds, not hours).

Might really meanA short, sharp peak your capacity was never sized for, even if the daily average looks unremarkable.

WhatRequests or transactions per second, viewed at its extremes rather than its average.

Why it mattersAn hourly average of "5 per second" can be hiding a burst of 40 per second for two minutes — often tied to the same predictable moments as above (morning login, lunch return, reporting deadlines).

Best statisticMax at the shortest interval your monitoring supports.

Might really meanNothing on its own — but it's a prime target worth correlating against the metrics above when investigating an incident.