>_ IzzyLab

Hands-on task

This task is step 18 of the course “Monitoring and metrics”

Alert with hysteresis

medium~10 min

Write a script /root/alert.sh that takes a number (a metric) as an argument and prints ALERT or OK with hysteresis: the alarm turns on when the value is ≥ 80 and stays on until the value drops to ≤ 60 (between 60 and 80 the state does not change). Persist the state between runs in the file /root/.alert_state.

What you'll practise:
  • hysteresis
  • bash script
  • state in a file
  • thresholds
  • conditional branching

Topic breakdown

Hysteresis is a technique against alert "flapping": two thresholds instead of one, a high one to turn the alarm on and a low one to reset it, with the state holding its previous value in between. The key point is that the state must survive process termination, so it is stored not in a variable but in a file on disk, read at startup and written back updated before exit. The logic reduces to inspecting the current state and the value of argument $1 via conditional branching, and you must print the new state.

What's checked

  1. hysteresis works on the sequence 50,70,85,70,60,75,80

What it looks like