>_ IzzyLab

Hands-on task

This task is step 19 of the course “Processes and signals”

The exporter hung and ignores TERM

medium~10 min

Prometheus has stopped getting metrics: a request to http://localhost:9100/metrics hangs with no response. The exporter process on port 9100 is alive but stuck. Find it, stop it, and start the exporter again from /opt/exporter/start.sh in the background so that curl -m 3 http://localhost:9100/metrics serves metrics again.

What you'll practise:
  • hung process
  • ss -p
  • SIGTERM and SIGKILL
  • restarting a daemon
  • curl -m

Topic breakdown

A hung process differs from a crashed one: it's in ps, it holds the socket, the kernel even accepts connections into the listen queue, but there's no answer — so the client hangs instead of being refused. Diagnosis is a request with a timeout plus ss -tlnp to find who holds the port. A process that is stuck or handles signals itself may not react to SIGTERM and SIGHUP: the handler isn't called or the signal is ignored. SIGKILL can't be caught — the kernel removes the process without its cooperation, so it's kept as the last resort when soft signals haven't worked.

What's checked

  1. the exporter serves metrics again
  2. the hung process is stopped

What it looks like