Kubernetes health check probes are the mechanism that lets Kubernetes verify whether your containers are actually working, not just running. Without them, Kubernetes has no way to distinguish a pod that's genuinely serving traffic from one that's silently deadlocked. Probes give the kubelet agent on each node a continuous signal: is this container alive, ready, and able to handle requests?
The cost of skipping or misconfiguring health checks is measurable and immediate. A pod that appears healthy but can't process requests keeps receiving traffic, resulting in timeouts and user-facing errors. A slow-starting application without a startup probe risks being killed during initial startup. Getting this right is foundational to building self-healing infrastructure that recovers from failures without manual intervention.
This guide walks you through the three probe types, implementation patterns for each mechanism, and troubleshooting strategies that reduce mean time to resolution.
Key takeaways
- Kubernetes offers three probe types: liveness (failure recovery), readiness (traffic control), and startup (slow initialization). Each serves a distinct purpose and should be configured independently.
- Misconfigured timing parameters such as
initialDelaySecondsandfailureThresholdare among the most common sources of avoidable downtime in containerized environments. - HTTP, command, and TCP probe mechanisms suit different application architectures. Choosing the right one reduces overhead and improves diagnostic accuracy.
- Correlating probe failures with application metrics, logs, and traces is what separates fast root cause analysis from prolonged incident response.
What are Kubernetes health checks, and why do they matter?
A Kubernetes health check is a configured probe that kubelet runs against your container to determine its state. Probes don't just monitor passively; they trigger automated responses.
- A failed liveness probe restarts the container
- A failed readiness probe removes the pod from service endpoints
- A startup probe gates all other checks until initialization completes
The business impact of poor health check configuration compounds quickly in distributed systems. A single misconfigured probe can cause premature container restarts under load, route traffic to pods that aren't ready, or allow genuinely broken containers to keep serving requests. Any of these conditions can cascade into broader outages across dependent services.
Properly configured probes are what make a Kubernetes cluster self-healing in practice, not just in theory. They're the difference between a cluster that recovers automatically and one that requires an engineer to intervene.
Types of Kubernetes health checks
Kubernetes uses three probe types to monitor container health at different lifecycle stages. Understanding when to use each one prevents the most common pitfalls, such as premature restarts, traffic routed to unready pods, and slow-starting applications killed before they finish initializing.
Liveness probes
Liveness probes answer one question: Is this container still functional? When the probe fails, the kubelet restarts the container automatically. This is your recovery mechanism for deadlocks, infinite loops, or any failure state where the process is running but can't serve requests.
Configure liveness probes conservatively. Overly aggressive settings cause restart loops during high-load periods or temporary slowdowns, which can cascade across your cluster. Set initialDelaySeconds long enough for your application to finish starting, and use a failureThreshold that tolerates brief hiccups without immediately triggering a restart. A threshold of 3 is the default starting point for most services.
Note that a liveness probe may not be necessary if the process in your container is able to crash on its own in the event of an issue or if it becomes unhealthy, as the kubelet will carry out the appropriate action configured in the Pod’s restartPolicy.
Readiness probes
Readiness probes control traffic routing and load balancing. When a readiness probe fails, Kubernetes removes the pod from service endpoints until the probe succeeds again and is ready to receive traffic, but keeps the container running.
Use readiness probes whenever your application needs time to warm up, load data, or establish connections before it can handle requests. They also run continuously throughout the container's lifecycle, so you can use them to temporarily pull a pod out of rotation during maintenance or when a dependency becomes unavailable. For zero-downtime deployments, readiness probes ensure new pods are fully operational before old ones are terminated.
Startup probes
Startup probes exist specifically for slow-starting containers. Once configured, a startup probe disables liveness and readiness checks until initialization succeeds, giving your application as much time as it needs without risking premature termination.
This is particularly valuable for legacy applications, large JVM-based services, or anything that performs extensive data loading at boot time from its container image. Without startup probes, you'd need artificially high initialDelaySeconds values on your liveness probes, which delays failure detection for the entire container lifecycle, not just startup. The product of failureThreshold and periodSeconds defines your maximum startup window. For example, failureThreshold: 30 with periodSeconds: 10 gives your application up to 5 minutes to initialize.
The practical decision framework is straightforward: use startup probes for initialization, readiness probes for traffic management, and liveness probes for runtime failure recovery. These three concerns are distinct and should be configured independently.
How to implement Kubernetes health check probes
Every probe type shares the same set of timing parameters that control how kubelet behaves:
initialDelaySeconds:How long the kubelet waits after container startup before running the first probeperiodSeconds:How frequently kubelet runs the probe (default: 10 seconds)timeoutSeconds:How long kubelet waits for a response before marking the probe failed (default: 1 second)failureThreshold:Consecutive failures before kubelet takes action (default: 3)successThreshold:Consecutive successes required after a failure to restore healthy status (default: 1)
Getting these values right requires knowing your application's actual startup time, expected response latency under load, and tolerance for transient failures. Defaults are rarely optimal for production workloads.
HTTP requests
HTTP probes are the most common choice for web services and APIs. Kubelet sends a GET request to a specified path and port on the container's IP. Any response with a status code between 200 and 399 is healthy; anything else is a failure.
apiVersion: v1
kind: Pod
metadata:
labels:
test: liveness
name: liveness-http
spec:
containers:
- name: liveness
image: k8s.gcr.io/liveness
args:
- /server
livenessProbe:
httpGet:
path: /healthz
port: 8080
httpHeaders:
- name: Custom-Header
value: Awesome
initialDelaySeconds: 15
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
successThreshold: 1Keep your health endpoint lightweight. It should verify critical dependencies, such as database connections and cache availability, without running expensive operations. A probe that consistently takes close to its timeoutSeconds value will generate false failures under load.
Commands
Command, or exec, probes execute a command inside the container, similar to how one might interact with a Docker container. A zero exit code means healthy; anything else is a failure. This approach works well for batch jobs, background workers, or legacy systems without HTTP interfaces.
apiVersion: v1
kind: Pod
metadata:
labels:
test: liveness
name: liveness-exec
spec:
containers:
- name: liveness
image: k8s.gcr.io/busybox
args:
- /bin/sh
- -c
- touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600
livenessProbe:
exec:
command:
- cat
- /tmp/healthy
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 1
failureThreshold: 3
successThreshold: 1Each command probe spawns a new process inside the container, which adds CPU overhead that HTTP and TCP probes don't. In high-density deployments with frequent probe intervals, this adds up. Keep command probes simple: file existence checks and quick status queries are ideal.
TCP connections
TCP probes verify that your application is accepting connections on a specified port. If kubelet can establish the connection, the container is healthy. This is the right choice for gRPC services, message queues, database servers, or any TCP-based application that doesn't expose HTTP.
apiVersion: v1
kind: Pod
metadata:
name: goproxy
labels:
app: goproxy
spec:
containers:
- name: goproxy
image: k8s.gcr.io/goproxy:0.1
ports:
- containerPort: 8080
readinessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 2
livenessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 15
periodSeconds: 20
timeoutSeconds: 2
failureThreshold: 3TCP probes are faster than HTTP probes but less informative. A successful connection confirms the port is open, not that the application logic behind it is working correctly. For this reason, many production deployments pair TCP liveness probes with HTTP readiness probes to get both speed and diagnostic depth.
Troubleshooting common Kubernetes health check issues
Most probe failures in production can be traced to a small set of configuration mistakes. Knowing where to look significantly reduces debugging time.
Debugging failed health check responses
Start with kubectl describe pod <pod-name>. The Events section shows probe failure messages with the specific error, including connection timeouts, non-2xx HTTP responses, and non-zero command exit codes. For HTTP probes, temporarily exec into the container and curl the health endpoint directly to verify it responds correctly within the container's network namespace.
Common misconfigurations to check first: initialDelaySeconds set too low for the actual startup time, timeoutSeconds shorter than the endpoint's p99 response time, and health endpoints that perform expensive dependency checks instead of lightweight status reads. If probes are failing intermittently rather than consistently, the issue is usually a timeout that's too tight for load conditions rather than a genuine application failure.
Optimizing health check performance
In high-traffic environments, probe overhead accumulates. Every probe execution consumes kubelet resources and, for HTTP probes, application thread capacity. A cluster running 500 pods with ten-second probe intervals generates thousands of health check requests per minute across your nodes.
Tune periodSeconds based on your actual recovery time objectives. If your alerting SLA is five minutes, running probes every five seconds provides no additional value over every 30 seconds, but it does add measurable overhead. For command probes specifically, profile the command's resource consumption before deploying to production at scale. Use successThreshold greater than one for readiness probes in high-churn environments to prevent pods from rapidly cycling in and out of service rotation during transient load spikes.
Monitor Kubernetes health checks with observability
Probes tell Kubernetes whether containers are healthy, but they don't explain why a probe failed or what impact it had on users. That context lives in your application metrics, logs, and traces, and correlating it with probe events is where the real diagnostic value comes from.
A unified observability platform like New Relic eliminates context switching between tools, which slows incident response. Instead of checking probe status in one place and investigating latency spikes in another, you can see how a readiness probe failure correlates with connection pool exhaustion or a downstream service degradation in a single view.
New Relic's Kubernetes monitoring connects health-check signals to the full telemetry stack. When a liveness probe triggers a restart, you can immediately see whether the restart resolved the underlying issue or masked a deeper problem. AI-powered anomaly detection surfaces patterns in probe failure data before they cascade into outages, shifting your posture from reactive troubleshooting to proactive health management.
Book a demo to explore the platform and see how correlating probe data with application telemetry can reduce your mean time to resolution during health-check failures.
FAQs about Kubernetes health checks
How often should Kubernetes health checks run in production environments?
Most production services work well with periodSeconds between 10 and 30 seconds. Start with the default of 10 seconds and increase it if probe overhead becomes measurable. The right interval depends on your recovery time objective: if you need to detect failures within 30 seconds, a 10-second interval with failureThreshold: 3 gives you a 30-second detection window.
What are the tradeoffs between aggressive vs conservative health check configurations?
Aggressive configurations (short intervals, low thresholds) detect failures faster but increase false positives during transient load spikes, which can trigger unnecessary restarts and destabilize healthy pods. Conservative configurations reduce noise but slow failure detection. For liveness probes, err on the conservative side. For readiness probes, you can afford to be more responsive since failed probes don't restart containers.
How do Kubernetes health checks impact application performance and resource usage?
HTTP and TCP probes add minimal overhead in most deployments. Command probes are the exception: each execution spawns a process inside the container, which consumes CPU. At scale, frequent command probes across many containers can create measurable resource pressure. Monitor kubelet CPU usage on your nodes if you're running command probes at high frequency across a large cluster.
As opiniões expressas neste blog são de responsabilidade do autor e não refletem necessariamente as opiniões da New Relic. Todas as soluções oferecidas pelo autor são específicas do ambiente e não fazem parte das soluções comerciais ou do suporte oferecido pela New Relic. Junte-se a nós exclusivamente no Explorers Hub ( discuss.newrelic.com ) para perguntas e suporte relacionados a esta postagem do blog. Este blog pode conter links para conteúdo de sites de terceiros. Ao fornecer esses links, a New Relic não adota, garante, aprova ou endossa as informações, visualizações ou produtos disponíveis em tais sites.
