>_ IzzyLab

Hands-on task

This task is step 8 of the course “Monitoring and metrics”

Counters by log level

medium~10 min

In /root/app.log lines start with a level: ERROR, WARN or INFO. Count how many lines of each level there are and write to /root/levels.prom the Prometheus metrics log_lines_total{level="error"} N, …{level="warn"} N, …{level="info"} N (level in lowercase).

What you'll practise:
  • prometheus
  • awk
  • counting by level
  • labels
  • case conversion

Topic breakdown

Here the frequencies must not just be counted but formatted as Prometheus metrics with a label: name{label="value"} number, where the label in curly braces makes log_lines_total with different level values into different time series. Counting by the first field of the line is the familiar associative array in awk or separate grep -c calls. Two gotchas: the label value is required in lowercase (in the log the levels are uppercase — convert explicitly via tolower()/tr), and the quotes around the value are a mandatory part of the format.

What's checked

  1. error count is correct
  2. warn and info counts are correct

What it looks like