Probes are how the kubelet decides whether a container is healthy and whether traffic can be sent to it. The idea is simple, yet probes are exactly where most teams stumble: a misconfigured probe doesn't protect your service, it takes it down itself - either through cascading restarts or by routing traffic into a pod that is known to be broken. Let's go through what each of the three probes does, the mistakes that come up most often, and how to set them up so they work for you rather than against you.
The three probes and what they actually do
Kubernetes has three probes, and confusing them is the number-one source of trouble. The key difference isn't how they check (an HTTP request, a TCP connection, a command inside the container - the mechanism is identical), but what happens on failure.
- liveness answers «is the container still alive or stuck?». Failure = the kubelet restarts the container. That's a heavy action: the pod loses in-memory state and connections are dropped.
- readiness answers «can this pod serve requests right now?». Failure = the pod is removed from the Service endpoints, no traffic reaches it, but the container is not restarted. As soon as the probe passes again, the pod returns to rotation.
- startup is for apps with a slow boot (cache warm-up, migrations, JVM). Until the startup probe succeeds, liveness and readiness don't run at all - so the pod won't be killed for «not responding» while it's still loading.
Five mistakes that take prod down
- The same logic in liveness and readiness. Under load the pod starts responding slowly - readiness pulls it out of traffic (correct), but the same check in liveness restarts the container at the same time (wrong). Instead of relief, you get a restart under load.
- liveness checking external dependencies. A classic: liveness hits an endpoint that talks to the database. The DB stalls for a second - liveness fails in every pod at once - Kubernetes restarts the whole service. One slow dependency turns into a restart avalanche.
- Thresholds that are too aggressive.
timeoutSeconds: 1andfailureThreshold: 1mean one slow response equals a restart. A single GC pause or latency spike is enough to land inCrashLoopBackOffout of nowhere. - No startup probe for a slow app. Without it liveness starts immediately and you have to guess
initialDelaySeconds. Set it too low - the pod is killed at boot; set it too high - real hangs are caught late. A startup probe removes that tradeoff. - A readiness probe that never fails. The probe returns 200 even when the app can't serve requests (lost its dependency connection, pool exhausted). Traffic keeps flowing into the broken pod and users get errors.
How to configure it right
The rule that closes most problems: liveness should be cheap and local, while readiness may look at dependencies.
- liveness - only «the process isn't deadlocked». A separate lightweight endpoint that returns 200 if the event loop is alive. No calls to the DB, cache or neighboring services.
- readiness - «can I serve a request». Here it's fine to check critical dependencies: is there a DB connection, is the pool warm. If a dependency is down, the pod leaves traffic but stays alive and comes back on its own.
- startup - budget the time:
failureThreshold × periodSecondsshould exceed your longest honest boot. For example,failureThreshold: 30andperiodSeconds: 5give 150 seconds to load, after which the pod is considered stuck.
A quick cheat sheet on parameters: initialDelaySeconds - the pause before the first check (barely needed for liveness if you have a startup probe); periodSeconds - how often to check; timeoutSeconds - how long to wait for a response; failureThreshold - how many consecutive failures count as a failure; successThreshold - how many successes to recover (always 1 for liveness).
How to diagnose
If a pod restarts endlessly, check the events first: kubectl describe pod <name> - the Events section will show a line like Liveness probe failed: ... with the concrete reason (timeout, response code, connection refused). A growing RESTARTS in kubectl get pods and a CrashLoopBackOff status almost always point at liveness. And if a pod sits at READY 0/1 for a long time but doesn't restart - it's readiness: the container is alive but not admitting traffic. Splitting «restart versus traffic» is the fastest way to know which probe to fix.
One caveat: a Last State: Terminated - Reason: OOMKilled message has nothing to do with probes - it's out of memory, and probes are not involved. Don't conflate the two, or you'll be tuning timeoutSeconds where you actually need to raise the memory limit.
When you don't have to write probes by hand
Everything above is manual work that gets repeated in every new service, with the same mistakes made in the same places. Internal Developer Platforms handle this differently: probes are generated from a healthcheck path or from the team's standard template, rather than hand-tuned on every deploy. If you're curious what that looks like in practice, we describe the approach in the Opsy platform overview. But even without a platform, the rule «liveness cheap and local, readiness looks at dependencies, startup for slow boots» removes most of the pain.