← Full reference

Kubernetes: probes, resources and HPA

Liveness, readiness and startup probes, requests and limits, CPU and memory units, QoS classes, HorizontalPodAutoscaler

There are two questions Kubernetes needs every container to answer: is it alive and ready to accept traffic (probes) and how many resources does it need (requests and limits). Autoscaling also relies on resources: a HorizontalPodAutoscaler changes the number of replicas based on load relative to requests.

All of these are manifest fields. How probes and limits actually behave can only be seen in a running cluster: in the sandbox you can write them and validate them against the schema, but not run them.

Syntax

apiVersion: v1
kind: Pod
metadata:
  name: web
spec:
  containers:
    - name: web
      image: nginx:1.27
      ports:
        - {name: http, containerPort: 80}
      resources:
        requests: {cpu: 100m, memory: 64Mi}   # guaranteed, used for scheduling
        limits: {memory: 128Mi}               # the ceiling
      readinessProbe:                         # ready to accept traffic?
        httpGet: {path: /, port: http}
      livenessProbe:                          # alive? if not — restart
        httpGet: {path: /, port: http}
        periodSeconds: 10
        failureThreshold: 3

Three kinds of probes

ProbeWhat happens on failureWhen you need it
readinessProbethe pod is removed from the Service's addresses — no traffic goes to it; the container is not restartedalmost always: warm-up, temporary overload, a lost dependency
livenessProbekubelet restarts the containerthe app can hang in a way where it doesn't crash on its own
startupProbeuntil it passes, the other probes don't run; not passing within the allotted time — restartslow startup: gives it time without weakening liveness

Without probes a container is considered alive until its process exits, and ready right after start.

Check method

Exactly one per probe:

FieldSuccess if…Parameters
httpGetthe response code is from 200 to 399path (default /), port (a number or port name), scheme: HTTP|HTTPS, host, httpHeaders: [{name, value}]
tcpSocketa TCP connection was establishedport
execthe command exited with code 0command: [list] — no shell; if you need a shell — ["sh", "-c", "…"]
grpcthe gRPC health check returned SERVINGport, service

Probe timings

FieldMeaningDefault
initialDelaySecondspause after the container starts before the first check0
periodSecondsinterval between checks10
timeoutSecondshow long to wait for a response1
failureThresholdhow many failures in a row count as a probe failure3
successThresholdhow many successes in a row are needed after a failure1; for liveness and startup — only 1
terminationGracePeriodSecondsits own shutdown time on a liveness/startup failurefrom the pod

Reaction time ≈ failureThreshold × periodSeconds: with the defaults, liveness restarts a hung container after about 30 seconds. A startup probe with failureThreshold: 30 and periodSeconds: 10 gives the app up to 300 seconds to start.

Resources: requests and limits

The resources field of each container:

FieldMeaningWhat happens when exceeded
requests.cputhe CPU share the scheduler reserves on the nodecan use more if the node is idle
requests.memorymemory reserved on the nodecan use more, but a pod over its request is the first candidate for eviction when the node runs out of memory
limits.cputhe CPU ceilingthe process is throttled, not killed
limits.memorythe memory ceilingthe container is killed: status OOMKilled, then a restart

A pod lands only on a node where the sum of requests of already placed pods plus its own fits into the available resources. limits don't affect scheduling. If only limits is set, requests are taken equal to it.

Units

NotationValue
CPU 1one core (vCPU)
CPU 500m = 0.5half a core; m means thousandths, minimum 1m
memory 128Mi, 1Gibinary: Ki = 1024, Mi = 1024², Gi = 1024³
memory 128M, 1Gdecimal: k = 1000, M = 1000², G = 1000³ (128M ≈ 122 Mi)
memory 128m0.128 bytes — the m suffix means "milli" here too; almost always a typo

QoS classes

Kubernetes assigns a pod a class itself (status.qosClass) — it determines who gets evicted first when a node runs out of memory:

ClassConditionEvicted
Guaranteedevery container has CPU and memory requests and limits, and they're equallast
Burstablenot Guaranteed, but at least one container has requests or limitsafter BestEffort
BestEffortno container has any requests or limitsfirst

HorizontalPodAutoscaler (autoscaling/v2)

Every ~15 seconds the HPA compares the metric with the target and changes replicas of a Deployment or StatefulSet: desired count ≈ ceil(current_replicas × current_value / target). CPU and memory metrics require metrics-server in the cluster.

FieldMeaningDefault / notes
spec.scaleTargetRef{apiVersion: apps/v1, kind: Deployment, name: web} — what to scalerequired
spec.minReplicaslower bound1
spec.maxReplicasupper boundrequired
spec.metrics[]which metrics to use; with several, the maximum of the desired counts is takenno metrics — CPU 80%
metrics[].type: Resourceresource: {name: cpu|memory, target: …}
target: {type: Utilization, averageUtilization: 70}average usage as a percentage of requestscontainers must have requests
target: {type: AverageValue, averageValue: 500Mi}average absolute value per pod
metrics[].type: Pods / Object / Externalcustom and external metrics (via a metrics adapter)
spec.behavior.scaleUp / scaleDownstabilizationWindowSeconds, policies: [{type: Pods|Percent, value, periodSeconds}], selectPolicy: Max|Min|Disabledstabilization window: down — 300 s, up — 0

Examples

# requests and limits in a ready manifest — no cluster needed
kubectl create deployment web --image=nginx:1.27 --port=80 --dry-run=client -o yaml > deploy.yaml
kubectl set resources -f deploy.yaml --requests=cpu=100m,memory=128Mi \
  --limits=cpu=500m,memory=256Mi --local -o yaml > deploy-res.yaml

# probes: there's no generator, so add them with yq
yq -i '.spec.template.spec.containers[0].readinessProbe = {"httpGet": {"path": "/", "port": 80}, "periodSeconds": 5}' deploy-res.yaml
yq -i '.spec.template.spec.containers[0].livenessProbe = {"tcpSocket": {"port": 80}, "initialDelaySeconds": 10}' deploy-res.yaml
yq '.spec.template.spec.containers[0] | {"resources": .resources, "probes": [.readinessProbe, .livenessProbe]}' deploy-res.yaml

# HPA: kubectl autoscale needs a cluster, so write the manifest yourself
cat > hpa.yaml <<'EOF'
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web
spec:
  scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: web}
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: {type: Utilization, averageUtilization: 70}
EOF

# since the HPA manages replicas, remove replicas from the Deployment
yq -i 'del(.spec.replicas)' deploy-res.yaml

kubeconform -strict -summary deploy-res.yaml hpa.yaml

# in a cluster (won't work here):
# kubectl get hpa web --watch
# kubectl top pods
# kubectl describe pod web-…        # Unhealthy events, Liveness probe failed
# kubectl get pod web-… -o jsonpath='{.status.qosClass}'

Slow startup: a startup probe allows up to 5 minutes, then liveness and readiness take over; an exec probe for a background worker without HTTP:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 2
  selector:
    matchLabels: {app: api}
  template:
    metadata:
      labels: {app: api}
    spec:
      containers:
        - name: api
          image: registry.example.com/api:3.1
          ports:
            - {name: http, containerPort: 8080}
          resources:                    # requests = limits → Guaranteed class
            requests: {cpu: 250m, memory: 256Mi}
            limits: {cpu: 250m, memory: 256Mi}
          startupProbe:
            httpGet: {path: /healthz, port: http}
            periodSeconds: 10
            failureThreshold: 30        # 30 × 10 s = up to 5 minutes to start
          livenessProbe:
            httpGet: {path: /healthz, port: http}
            periodSeconds: 10
            timeoutSeconds: 2
          readinessProbe:
            httpGet: {path: /ready, port: http}
            periodSeconds: 5
            failureThreshold: 2
        - name: worker
          image: registry.example.com/worker:3.1
          resources:
            requests: {cpu: 100m, memory: 128Mi}
            limits: {cpu: 100m, memory: 128Mi}
          livenessProbe:
            exec:
              command: ["sh", "-c", "test $(( $(date +%s) - $(stat -c %Y /tmp/heartbeat) )) -lt 60"]
            periodSeconds: 30

An HPA on CPU and memory with gradual scale-down:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: {type: Utilization, averageUtilization: 60}
    - type: Resource
      resource:
        name: memory
        target: {type: AverageValue, averageValue: 200Mi}
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 600
      policies:
        - {type: Pods, value: 1, periodSeconds: 60}   # remove at most 1 pod per minute

Pitfalls

  • Liveness checking dependencies. If the liveness probe hits the database, then when the database goes down kubelet starts restarting all of the app's pods — this won't help and only adds load. Liveness means "is the process itself alive"; dependencies belong in readiness.
  • Identical liveness and readiness without startup on a slow-starting app: liveness kills the container before it manages to come up — an endless CrashLoopBackOff. Add a startupProbe or initialDelaySeconds.
  • timeoutSeconds: 1 by default. Under load a response slower than a second counts as a failure, and liveness starts restarting pods that are alive but busy.
  • An exec probe without a shell. command: ["test -f /tmp/ok"] looks for a program named test -f /tmp/ok. Arguments go as separate elements, and $(…), &&, | work only via ["sh", "-c", "…"].
  • The probe port is the container port, not the Service's. A port name (port: http) works only if the container's ports has name: http.
  • successThreshold greater than 1 for liveness and startup is rejected by the API, but kubeconform lets it through.
  • 128m of memory is 0.128 bytes. The schema doesn't catch this either: it doesn't parse the quantity format. Write memory with Mi/Gi.
  • M and Mi are different units: 512M is roughly 488Mi. Stick to one system within a project.
  • The memory limit is hard. When exceeded, the container is killed with OOMKilled (code 137) rather than "throttled" as with CPU.
  • A low limits.cpu causes slowdowns without errors: throttling isn't visible in logs or status, only in metrics. Many teams don't set a CPU limit at all, leaving only requests.
  • An HPA on Utilization without requests doesn't work: there's nothing to compute the percentage from, and the HPA events show missing request for cpu.
  • replicas in the Deployment manifest while an HPA is running — every kubectl apply resets the replica count to what's written. Remove the field from the manifest.
  • The HPA doesn't react instantly: scale-down waits for the stabilization window (300 s by default), and changes within ~10% of the target are ignored.
  • Without metrics-server the HPA shows the target as <unknown> and scales nothing.

See also

kubectl · kubeconform · yq · k8s-workloads · k8s-networking · k8s-config · curl · top