A Kubernetes Cluster Health Check in 10 Signals

Most cluster problems are visible days before they become incidents, if you know which signals to read. This is the checklist we run against production clusters, one signal at a time. For each one: what to look at, the command or metric, and what “bad” actually looks like. None of it requires a fancy platform. Most of it is kubectl and a Prometheus query.

1. Pending and Evicted Pods, and Node Pressure

What to look at: pods that cannot schedule, and nodes reporting resource pressure.

Command: kubectl get pods -A --field-selector=status.phase=Pending and kubectl describe node <node> | grep -i pressure.

What bad looks like: pods stuck Pending with FailedScheduling events usually mean the scheduler cannot satisfy resource requests, not that the cluster is out of raw capacity. Watch for MemoryPressure or DiskPressure conditions on nodes. We once found a node that had crossed the disk-pressure eviction threshold hundreds of times over a couple of months, quietly evicting and rescheduling pods, with no alert on it. Eviction churn shows up as latency and restarts long before anyone connects it to a full disk.

2. Restart Counts and CrashLoopBackOff Causes

What to look at: which pods are restarting, and crucially, why.

Command: kubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount' then kubectl describe pod and check lastState.terminated.reason.

What bad looks like: a climbing restart count is a symptom, not a diagnosis. The reason field is where the truth lives. OOMKilled means the container exceeded its memory limit and needs more memory or less work. But a CPU-bound container hitting its CPU limit gets throttled by the kernel, times out its health checks, and gets restarted, producing an identical restart graph with a completely different cause. We have watched a team chase a “memory leak” for weeks that was actually a 150 millicore CPU limit on an unpaginated list endpoint. Always separate OOMKilled from CPU throttling before you pick a fix.

3. The Requests-Versus-Utilization Gap

What to look at: what pods reserve versus what they actually use.

Metric: compare kube_pod_container_resource_requests against container_cpu_usage_seconds_total and container_memory_working_set_bytes.

What bad looks like: CPU requests several times higher than measured usage means you are paying for reserved capacity nobody uses, and packing fewer pods per node than you could. The opposite failure is memory requests below the real working set, which invites OOMKills. In estates where requests were copied service-to-service, we routinely see both at once: CPU over-provisioned, memory under-provisioned, on the same pod. The request is a reservation the scheduler honors whether the pod uses it or not, so this gap is a direct line to your node count.

4. HPA Presence and Sanity

What to look at: whether horizontal pod autoscalers exist, and whether they are doing anything.

Command: kubectl get hpa -A.

What bad looks like: the first failure is absence. We have found platform Helm charts that ship with HPA disabled by default, so every service runs a fixed replica count sized for peak. The second failure is a misconfigured HPA: current metric stuck at <unknown> (usually a missing metrics-server or missing resource requests), or minReplicas equal to maxReplicas, which is an autoscaler that cannot scale. An HPA that never changes replica count is decoration.

5. PDB Coverage Versus Cluster Upgrades

What to look at: pod disruption budgets, and whether they will let a node drain.

Command: kubectl get pdb -A.

What bad looks like: two opposite failures. No PDB on a critical multi-replica service means a node drain can take all its pods at once. But an over-strict PDB is worse during maintenance: a minAvailable set equal to the replica count, or a single-replica deployment with a PDB, means kubectl drain blocks forever and your cluster upgrade stalls. We have seen node upgrades stuck for hours because a PDB would not allow a single eviction. PDBs need to protect availability without making the cluster un-drainable.

6. Certificate Expiry

What to look at: kubelet and control-plane certs, ingress TLS, and internal mesh certs.

Command: kubeadm certs check-expiration on managed control planes where you have access, plus checking your ingress and cert-manager Certificate resources: kubectl get certificate -A.

What bad looks like: any cert inside a few weeks of expiry with no automated renewal path. Expired certs are among the highest-severity, most avoidable outages in Kubernetes, because they take down authentication or ingress cluster-wide and the error messages rarely say “expired certificate” in plain language.

7. Image Pull and Registry Health

What to look at: pods failing to pull images, and where your images come from.

Command: kubectl get events -A --field-selector reason=Failed | grep -i image and watch for ImagePullBackOff / ErrImagePull.

What bad looks like: ImagePullBackOff from expired registry credentials, rate-limited public registries, or a :latest tag that moved under you. Pinning to :latest is a reliability bug, not just a style one: two nodes can end up running two different builds of the “same” image. Sustained pull failures on a rollout mean your registry or its credentials are now a single point of failure.

8. Control Plane and etcd Basics

What to look at: API server responsiveness and etcd health.

Command: kubectl get componentstatuses where available, kubectl -n kube-system get pods, and on self-managed clusters etcdctl endpoint health.

What bad looks like: rising API server request latency, etcd database size creeping toward its quota, or high etcd leader-election churn. On managed control planes you see less of this directly, but slow or intermittently failing kubectl calls are the tell. A struggling etcd degrades everything above it, so this is the signal to check when “the whole cluster feels slow” and nothing specific is broken.

9. Namespace Resource Quotas

What to look at: whether namespaces have quotas, and whether they are near the limit.

Command: kubectl get resourcequota -A and kubectl describe resourcequota -n <namespace>.

What bad looks like: in a multi-tenant cluster, missing quotas mean one namespace can starve every other one, the classic noisy-neighbor failure. The opposite is a namespace pinned at 100 percent of its quota, silently rejecting new pods with quota-exceeded errors that look like scheduling failures. Quotas are how you keep one tenant’s bad day from becoming everyone’s.

10. Alert Fatigue and Silent Alerts

What to look at: your alerting itself, as a system that can fail.

Metric: review firing and recently-resolved alerts in Alertmanager, and confirm the delivery path actually works end to end.

What bad looks like: two failure modes, and both are dangerous. Alert fatigue is a wall of low-value alerts that trains everyone to ignore the channel, so the one that matters scrolls past. The quieter, worse failure is the silent alert: a rule that never fires because the metric stopped being scraped, or a Slack or Teams webhook that broke months ago and nobody noticed because absence of alerts feels like health. Send yourself a deliberate test alert. An alerting pipeline you have not exercised is not a safety net, it is a decoration you are trusting with production.

Running This Regularly Is the Whole Point

Any one of these signals read once tells you little. Read together, on a schedule, they tell you where a cluster is drifting before it breaks. The hard part is not the commands, it is doing them every month when nothing is on fire and attention has moved elsewhere.

That is the productized version of this checklist. Our monthly health report runs these ten signals automatically against your clusters, then a human reviews the output so you get findings and fixes, not another dashboard to ignore. If you would rather not remember to run this yourself, get in touch.