We Audited a 7-Cluster Kubernetes Platform: Here Is Where the Money Was Leaking

Kubernetes bills rarely leak in dramatic ways. There is no single line item that reads “wasted.” Instead the money drains out through defaults nobody revisited, resource requests copied from one service to the next, and failure modes that get misdiagnosed and then papered over with more replicas. We ran a structured audit of a 7-cluster estate we operate, spanning development, staging, and production, and the pattern was consistent. The waste was not exotic. It was ordinary, repeated, and invisible until someone measured it.

Here is what we found, framed the way the finance team cares about: as money.

Autoscaling Was Off by Default

The platform Helm chart shipped with horizontal pod autoscaling (HPA) disabled by default. Every service inherited that default, and almost nobody turned it back on. The result is the worst of both worlds. Under load, services could not scale out, so they throttled or fell over. Under normal traffic, they ran a fixed replica count sized for a peak that arrives a few hours a day, which means most of the day you are paying for idle pods that cannot even shed capacity when demand drops.

A disabled HPA is not a neutral default. It is a standing decision to over-provision for peak and eat the cost during every trough. Turning it on, with sane minimum and maximum replica counts, is often the single highest-leverage change available, and it costs nothing but attention.

Resource Requests Were Copy-Paste, and Backwards

The second finding was structural. Resource requests and limits were hardcoded and copied across services with almost no per-workload thinking. A lightweight API and a heavy batch consumer carried the same numbers because someone had pasted a working block and moved on.

When we compared requests against actual utilization, the mismatch ran in both directions at once:

• CPU was systematically over-provisioned. Services reserved far more CPU than they ever used. In Kubernetes, a request is a reservation: the scheduler sets that capacity aside whether the pod uses it or not. Over-requested CPU is capacity you have paid for and fenced off so nothing else can use it.
• Memory was under-provisioned. The same copy-paste left several services with memory requests below their real working set, which is how you end up with OOMKills and eviction risk on the exact workloads that looked “fine” on paper.

The lesson is that a request is a bill, not a hint. Every over-stated CPU request shrinks how much you can pack onto a node, which forces more nodes, which is the actual invoice.

The “Memory Leak” That Was CPU Throttling

One service crash-looped for weeks. The received wisdom on the team was that it had a memory leak, and every restart reinforced the story. It was not a memory leak.

The service had a 150 millicore CPU limit. One of its endpoints was a list route that was CPU-bound and unpaginated: it deserialized a growing dataset, sorted it, and mapped it on every call. As the data grew, the work per request grew with it, and against a 150 millicore ceiling the container spent most of its time being throttled by the kernel scheduler rather than doing work. Requests piled up, health checks timed out, and Kubernetes restarted the pod. That restart looks exactly like an OOM crash loop if you are only watching the pod restart count and not the throttling metrics.

The fix was two lines in the tenant override: raise the CPU limit and raise the HPA maximum replicas. After the change, restarts went to zero and stayed there. No code change to the endpoint was needed to stop the bleeding, though pagination is on the follow-up list.

The takeaway is worth internalizing because it is expensive to get wrong: before you believe “memory leak,” look at container_cpu_cfs_throttled_periods_total. A throttled CPU-bound service and a leaking one produce the same restart graph and require opposite fixes. Weeks of engineering time went into chasing the wrong one.

A Node Evicting Pods for 68 Days, Unwatched

The last finding was an operational blind spot rather than a config mistake. One node had been under chronic disk pressure and had crossed the kubelet eviction threshold 509 times over 68 days. Nobody was watching that signal, so pods were being evicted and rescheduled quietly, adding latency and churn that never surfaced as an alert.

On the same estate, two worker nodes sat at 95 to 98 percent CPU requests, not usage. That distinction matters: the scheduler places pods based on requests, so a node that is 95 percent requested is effectively full and will refuse new pods even while its actual CPU sits mostly idle. Combine over-stated requests from the previous finding with unwatched node pressure and you get a cluster that feels full, schedules poorly, and costs more, all without a single dashboard turning red.

Why This Is a Money Problem, Not Just a Tidiness Problem

None of these findings are unusual, and that is the point. Over-provisioned requests are paid-for-but-idle capacity, every single hour. And the industry data says almost nobody is measuring it. The CNCF’s 2024 survey on Kubernetes cost found that 68 percent of organizations either do not monitor Kubernetes spend at all or rely on manual estimates (CNCF, “The FinOps for Kubernetes microsurvey,” 2024). If two thirds of teams are estimating by hand, the over-provisioning above is not an edge case, it is the norm.

The scale of the loss lines up. In the FinOps Foundation’s State of FinOps 2025, practitioners and executives estimated that roughly 30 percent of cloud compute spend is wasted (FinOps Foundation, State of FinOps 2025). On a Kubernetes estate the size of the one we audited, 30 percent is not a rounding error. It is a headcount.

Where to Start

If you run Kubernetes in production and have not audited it against actual utilization recently, the highest-value moves are boring and fast: turn HPA on with real limits, compare requests to measured usage instead of copied defaults, and put a signal on node pressure and CPU throttling so misdiagnosis does not cost you weeks.

We package exactly this work as a fixed-scope engagement. Our cost and reliability audit is a fixed-price review that finds the leaks above and hands you the specific overrides to close them, and our monthly health report keeps the signals watched so a node does not spend 68 days evicting pods in the dark. If either sounds like your estate, get in touch.