Kubernetes: probes, resources and HPA
Liveness, readiness and startup probes, requests and limits, CPU and memory units, QoS classes, HorizontalPodAutoscaler
There are two questions Kubernetes needs every container to answer: is it alive and ready to accept traffic (probes) and how many resources does it need (requests and limits). Autoscaling also relies on resources: a HorizontalPodAutoscaler changes the number of replicas based on load relative to requests.
All of these are manifest fields. How probes and limits actually behave can only be seen in a running cluster: in the sandbox you can write them and validate them against the schema, but not run them.
Syntax
apiVersion: v1
kind: Pod
metadata:
name: web
spec:
containers:
- name: web
image: nginx:1.27
ports:
- {name: http, containerPort: 80}
resources:
requests: {cpu: 100m, memory: 64Mi} # guaranteed, used for scheduling
limits: {memory: 128Mi} # the ceiling
readinessProbe: # ready to accept traffic?
httpGet: {path: /, port: http}
livenessProbe: # alive? if not — restart
httpGet: {path: /, port: http}
periodSeconds: 10
failureThreshold: 3
Three kinds of probes
| Probe | What happens on failure | When you need it |
|---|---|---|
readinessProbe | the pod is removed from the Service's addresses — no traffic goes to it; the container is not restarted | almost always: warm-up, temporary overload, a lost dependency |
livenessProbe | kubelet restarts the container | the app can hang in a way where it doesn't crash on its own |
startupProbe | until it passes, the other probes don't run; not passing within the allotted time — restart | slow startup: gives it time without weakening liveness |
Without probes a container is considered alive until its process exits, and ready right after start.
Check method
Exactly one per probe:
| Field | Success if… | Parameters |
|---|---|---|
httpGet | the response code is from 200 to 399 | path (default /), port (a number or port name), scheme: HTTP|HTTPS, host, httpHeaders: [{name, value}] |
tcpSocket | a TCP connection was established | port |
exec | the command exited with code 0 | command: [list] — no shell; if you need a shell — ["sh", "-c", "…"] |
grpc | the gRPC health check returned SERVING | port, service |
Probe timings
| Field | Meaning | Default |
|---|---|---|
initialDelaySeconds | pause after the container starts before the first check | 0 |
periodSeconds | interval between checks | 10 |
timeoutSeconds | how long to wait for a response | 1 |
failureThreshold | how many failures in a row count as a probe failure | 3 |
successThreshold | how many successes in a row are needed after a failure | 1; for liveness and startup — only 1 |
terminationGracePeriodSeconds | its own shutdown time on a liveness/startup failure | from the pod |
Reaction time ≈ failureThreshold × periodSeconds: with the defaults, liveness restarts a hung container after about 30 seconds. A startup probe with failureThreshold: 30 and periodSeconds: 10 gives the app up to 300 seconds to start.
Resources: requests and limits
The resources field of each container:
| Field | Meaning | What happens when exceeded |
|---|---|---|
requests.cpu | the CPU share the scheduler reserves on the node | can use more if the node is idle |
requests.memory | memory reserved on the node | can use more, but a pod over its request is the first candidate for eviction when the node runs out of memory |
limits.cpu | the CPU ceiling | the process is throttled, not killed |
limits.memory | the memory ceiling | the container is killed: status OOMKilled, then a restart |
A pod lands only on a node where the sum of requests of already placed pods plus its own fits into the available resources. limits don't affect scheduling. If only limits is set, requests are taken equal to it.
Units
| Notation | Value |
|---|---|
CPU 1 | one core (vCPU) |
CPU 500m = 0.5 | half a core; m means thousandths, minimum 1m |
memory 128Mi, 1Gi | binary: Ki = 1024, Mi = 1024², Gi = 1024³ |
memory 128M, 1G | decimal: k = 1000, M = 1000², G = 1000³ (128M ≈ 122 Mi) |
memory 128m | 0.128 bytes — the m suffix means "milli" here too; almost always a typo |
QoS classes
Kubernetes assigns a pod a class itself (status.qosClass) — it determines who gets evicted first when a node runs out of memory:
| Class | Condition | Evicted |
|---|---|---|
Guaranteed | every container has CPU and memory requests and limits, and they're equal | last |
Burstable | not Guaranteed, but at least one container has requests or limits | after BestEffort |
BestEffort | no container has any requests or limits | first |
HorizontalPodAutoscaler (autoscaling/v2)
Every ~15 seconds the HPA compares the metric with the target and changes replicas of a Deployment or StatefulSet: desired count ≈ ceil(current_replicas × current_value / target). CPU and memory metrics require metrics-server in the cluster.
| Field | Meaning | Default / notes |
|---|---|---|
spec.scaleTargetRef | {apiVersion: apps/v1, kind: Deployment, name: web} — what to scale | required |
spec.minReplicas | lower bound | 1 |
spec.maxReplicas | upper bound | required |
spec.metrics[] | which metrics to use; with several, the maximum of the desired counts is taken | no metrics — CPU 80% |
metrics[].type: Resource | resource: {name: cpu|memory, target: …} | |
target: {type: Utilization, averageUtilization: 70} | average usage as a percentage of requests | containers must have requests |
target: {type: AverageValue, averageValue: 500Mi} | average absolute value per pod | |
metrics[].type: Pods / Object / External | custom and external metrics (via a metrics adapter) | |
spec.behavior.scaleUp / scaleDown | stabilizationWindowSeconds, policies: [{type: Pods|Percent, value, periodSeconds}], selectPolicy: Max|Min|Disabled | stabilization window: down — 300 s, up — 0 |
Examples
# requests and limits in a ready manifest — no cluster needed
kubectl create deployment web --image=nginx:1.27 --port=80 --dry-run=client -o yaml > deploy.yaml
kubectl set resources -f deploy.yaml --requests=cpu=100m,memory=128Mi \
--limits=cpu=500m,memory=256Mi --local -o yaml > deploy-res.yaml
# probes: there's no generator, so add them with yq
yq -i '.spec.template.spec.containers[0].readinessProbe = {"httpGet": {"path": "/", "port": 80}, "periodSeconds": 5}' deploy-res.yaml
yq -i '.spec.template.spec.containers[0].livenessProbe = {"tcpSocket": {"port": 80}, "initialDelaySeconds": 10}' deploy-res.yaml
yq '.spec.template.spec.containers[0] | {"resources": .resources, "probes": [.readinessProbe, .livenessProbe]}' deploy-res.yaml
# HPA: kubectl autoscale needs a cluster, so write the manifest yourself
cat > hpa.yaml <<'EOF'
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: web
spec:
scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: web}
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target: {type: Utilization, averageUtilization: 70}
EOF
# since the HPA manages replicas, remove replicas from the Deployment
yq -i 'del(.spec.replicas)' deploy-res.yaml
kubeconform -strict -summary deploy-res.yaml hpa.yaml
# in a cluster (won't work here):
# kubectl get hpa web --watch
# kubectl top pods
# kubectl describe pod web-… # Unhealthy events, Liveness probe failed
# kubectl get pod web-… -o jsonpath='{.status.qosClass}'
Slow startup: a startup probe allows up to 5 minutes, then liveness and readiness take over; an exec probe for a background worker without HTTP:
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
replicas: 2
selector:
matchLabels: {app: api}
template:
metadata:
labels: {app: api}
spec:
containers:
- name: api
image: registry.example.com/api:3.1
ports:
- {name: http, containerPort: 8080}
resources: # requests = limits → Guaranteed class
requests: {cpu: 250m, memory: 256Mi}
limits: {cpu: 250m, memory: 256Mi}
startupProbe:
httpGet: {path: /healthz, port: http}
periodSeconds: 10
failureThreshold: 30 # 30 × 10 s = up to 5 minutes to start
livenessProbe:
httpGet: {path: /healthz, port: http}
periodSeconds: 10
timeoutSeconds: 2
readinessProbe:
httpGet: {path: /ready, port: http}
periodSeconds: 5
failureThreshold: 2
- name: worker
image: registry.example.com/worker:3.1
resources:
requests: {cpu: 100m, memory: 128Mi}
limits: {cpu: 100m, memory: 128Mi}
livenessProbe:
exec:
command: ["sh", "-c", "test $(( $(date +%s) - $(stat -c %Y /tmp/heartbeat) )) -lt 60"]
periodSeconds: 30
An HPA on CPU and memory with gradual scale-down:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: api
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: api
minReplicas: 2
maxReplicas: 20
metrics:
- type: Resource
resource:
name: cpu
target: {type: Utilization, averageUtilization: 60}
- type: Resource
resource:
name: memory
target: {type: AverageValue, averageValue: 200Mi}
behavior:
scaleDown:
stabilizationWindowSeconds: 600
policies:
- {type: Pods, value: 1, periodSeconds: 60} # remove at most 1 pod per minute
Pitfalls
- Liveness checking dependencies. If the liveness probe hits the database, then when the database goes down kubelet starts restarting all of the app's pods — this won't help and only adds load. Liveness means "is the process itself alive"; dependencies belong in readiness.
- Identical liveness and readiness without startup on a slow-starting app: liveness kills the container before it manages to come up — an endless
CrashLoopBackOff. Add astartupProbeorinitialDelaySeconds. timeoutSeconds: 1by default. Under load a response slower than a second counts as a failure, and liveness starts restarting pods that are alive but busy.- An
execprobe without a shell.command: ["test -f /tmp/ok"]looks for a program namedtest -f /tmp/ok. Arguments go as separate elements, and$(…),&&,|work only via["sh", "-c", "…"]. - The probe port is the container port, not the Service's. A port name (
port: http) works only if the container'sportshasname: http. successThresholdgreater than 1 for liveness and startup is rejected by the API, butkubeconformlets it through.128mof memory is 0.128 bytes. The schema doesn't catch this either: it doesn't parse thequantityformat. Write memory withMi/Gi.MandMiare different units:512Mis roughly488Mi. Stick to one system within a project.- The memory limit is hard. When exceeded, the container is killed with
OOMKilled(code 137) rather than "throttled" as with CPU. - A low
limits.cpucauses slowdowns without errors: throttling isn't visible in logs or status, only in metrics. Many teams don't set a CPU limit at all, leaving onlyrequests. - An HPA on
Utilizationwithoutrequestsdoesn't work: there's nothing to compute the percentage from, and the HPA events showmissing request for cpu. replicasin the Deployment manifest while an HPA is running — everykubectl applyresets the replica count to what's written. Remove the field from the manifest.- The HPA doesn't react instantly: scale-down waits for the stabilization window (300 s by default), and changes within ~10% of the target are ignored.
- Without metrics-server the HPA shows the target as
<unknown>and scales nothing.
See also
kubectl · kubeconform · yq · k8s-workloads · k8s-networking · k8s-config · curl · top