>_ IzzyLab

Hands-on task

This task is step 8 of the course “Containers from the inside: namespaces and cgroups”

A report that eats memory: memory.max

medium~12 min

On bad data the quarterly report /opt/report/build.py starts eating memory without limit. Last time the kernel's OOM killer picked not the report but the database on the same server as its victim. While the developers fix the script, restrict it with the kernel's own tools:

  1. Create a cgroup v2 report (a directory in /sys/fs/cgroup) with a memory limit of 64 MiB and forbid swap for the group.
  2. Run /opt/report/build.py inside that group and make sure the kernel killed it inside the group — and the rest of the server didn't notice.

Don't run the script without a limit: on a production server it would eat all the memory. In the lab it has a safety fuse — outside a group with a small limit it stops by itself at 256 MiB, but then no OOM kill happens inside the group. cgroup v2 is mounted at /sys/fs/cgroup, and the controllers for child groups are already enabled.

What you'll practise:
  • cgroups
  • cgroup v2
  • memory.max
  • OOM killer

Topic breakdown

A cgroup (control group) is a kernel mechanism that limits and accounts for the resources of a group of processes: memory, CPU, I/O, number of processes. In cgroup v2 all groups form one directory tree in /sys/fs/cgroup: creating a group is mkdir, moving a process is writing its PID to cgroup.procs, and its descendants are then born in the same group. memory.max sets the memory limit. When a group hits its limit and memory can't be freed, the OOM killer fires inside the group: it kills a process from it rather than whatever happens to be on the server. The oom and oom_kill counters in memory.events show that it happened. The --memory flag of docker run and limits.memory in Kubernetes are the same memory.max of the container's group.

What's checked

  1. The group report is created
  2. The group's memory limit is 64 MiB
  3. Swap is forbidden for the group
  4. The kernel killed a process inside the group over the memory limit
  5. The report isn't running outside the group report

What it looks like