>_ IzzyLab

Hands-on task

This task is step 13 of the course “Logs: analysis and housekeeping”

Level metrics from a worker log

medium~10 min

/var/log/worker/worker.log has lines of the form TIME LEVEL text, the level is the second field: INFO, WARN or ERROR. Count the lines of each level and write three Prometheus metrics to /root/worker_levels.prom: worker_log_events_total{severity="info"} N, and the same for warn and error (label value in lowercase).

What you'll practise:
  • prometheus
  • awk
  • associative array
  • tolower
  • labels

Topic breakdown

Prometheus counters in the text format are lines name{label="value"} number; the same worker_log_events_total with different severity values are different time series. Counting by field value is an awk associative array keyed by the field itself. The catch with logs is that the keyword often appears in the text too ("retry after ERROR from upstream"), so a substring search over the whole line inflates the count: it is more reliable to look at a specific field. Label values are conventionally lowercase, and it's better to convert the case explicitly than by hand.

What's checked

  1. error counter is correct
  2. warn and info counters are correct

What it looks like