A Kubernetes cluster can show green dashboards while slowly degrading: pods restart a little more often than usual, disk fills up gradually, HPA replicas creep toward the ceiling - and none of these signals looks critical on its own. Most teams' problem isn't a lack of monitoring but too much of it: hundreds of metrics are collected, dashboards grow to dozens of panels, but the on-call engineer can't quickly tell background noise from an early sign of an incident. As a result, alerts are either ignored from false-positive fatigue, or the real problem drowns among secondary charts. Let's break down which indicators really define cluster health, why many production dashboards show the wrong things, and how to build observability that actually shortens incident response rather than just pleasing the eye with pretty graphs.
Pod resources: between request and limit
The most common mistake is monitoring only the fact of a limit breach while ignoring the dynamics between request and limit. Most performance problems that never reach alerts but constantly erode resilience live in that gap. Teams usually approach CPU and RAM optimization exactly by analyzing it:
- CPU and Memory requests vs actual usage. A healthy range is 60-80% of the declared request at the 90th percentile of load. Lower means overpaying for unused resources; higher means the risk of node contention and eviction of neighboring pods.
- CPU and Memory limits vs actual usage. Aim for about 80% of the limit at the 90th percentile. Exceeding CPU turns into throttling and rising latency; exceeding memory into a hard OOMKilled and loss of application state.
- Share of unavailable pods. Any non-zero value here is already a deviation, not "acceptable noise." The alert threshold should be tuned to each service's criticality, not a single value for the whole cluster - otherwise background restarts of secondary components mask degradation of the ones that matter.
Tracking these three together gives a far more accurate picture than the sum of separate CPU, memory and restart alerts. For example, a rising share of unavailable pods against normal resource usage almost always points not to a capacity shortage but to a problem with application readiness, a network policy or a dependency on an external service - and without correlating the metrics that cause stays invisible until it grows into full downtime.
Node health and predictable autoscaling
Pod-level metrics are useless if the node itself is degrading. Engineers often learn about infrastructure-level problems last - after users have already noticed. To avoid that, keep node conditions and horizontal scaling parameters under watch:
- Node condition checks. Ready, DiskPressure, MemoryPressure, PIDPressure and NetworkUnavailable should be checked with zero tolerance - a faulty node shouldn't stay in the cluster longer than it takes to automatically drop it from scheduling.
- HPA desired replicas relative to the maximum. Set the alert at 85% of the configured replica maximum - that gives time for manual intervention before the autoscaler hits the ceiling under growing load.
- Persistent Volume utilization. What matters is not current disk fill but the growth trajectory: a linear forecast lets you plan volume expansion in advance rather than firefighting an overflow after the fact.
The combination of node health checks and a predictive HPA metric covers most incidents that would otherwise surface only through user complaints. A node rarely fails instantly - it's usually preceded by a gradual worsening of disk or memory pressure, and that degradation window is exactly what gives time for a planned node replacement without affecting availability, if observability is set up to get ahead of the failure rather than record it after the fact.
How to choose a cluster monitoring tool
The choice of monitoring stack depends on infrastructure scale and on how much time the team is willing to spend maintaining the observability tools themselves. Through platform monitoring and metrics it's convenient to compare several approaches at once without switching between disparate systems:
- Prometheus and kube-state-metrics. An open pull model - flexible and free, but requiring constant alert-rule tuning and manual dashboard updates as the cluster grows; without dedicated time for this work the configuration quickly goes stale and stops reflecting the real service topology.
- Full-stack visibility. Monitoring must cover not only Kubernetes but the whole microservice chain - otherwise an engineer sees a healthy pod next to a degrading external API and can't tell where the problem actually came from.
- Automatic root-cause analysis. AI-driven platforms like Dynatrace can link a symptom to its root cause on their own, which is critical with many services and a short response window for the on-call engineer.
- Hybrid infrastructure support. If part of the load stays off-cloud, the tool should show local and cloud cluster metrics equally clearly in one interface, without switching between different systems and dashboard formats per site.
For a small team, Prometheus and Grafana remain a reasonable starting point, but as the number of services grows the cost of maintaining a home-grown monitoring stack starts to rival ready-made platforms. Every new alert rule, every dashboard for a new service and every exporter update takes engineer time - and that too is part of the real cost of observability, rarely counted at the start but accumulating with the cluster.
Kubernetes monitoring works not when every possible metric is collected, but when the few that actually predict an incident are selected from them. Pod resources, node health, the autoscaling ceiling and storage capacity are the minimal set worth putting on the on-call engineer's main dashboard, leaving the rest for deeper analysis after an alert fires. This approach doesn't remove the need for deep diagnostics, but it frees the team from constant noise and lets it react to a problem before users see it, not after complaints reach support.