>_ IzzyLab

Hands-on task

This task is step 11 of the course “Containers from the inside: namespaces and cgroups”

Thumbnails are slow, but no limits in sight

medium~15 min

The thumbnail generator thumbs (a Docker container) works through its queue several times slower than on the neighboring server. docker stats shows it at about 5% CPU even though the server is idle, docker inspect shows no CPU limits — and docker update --cpus changes nothing.

Find what's really squeezing the container and lift the restriction without restarting or recreating the container: the generator must get at least a whole core. Leave the other containers alone.

What you'll practise:
  • cgroups
  • cpu.max
  • docker
  • troubleshooting

Topic breakdown

cgroup v2 groups form a tree, and a parent's limit applies to all its descendants together: if a group's cpu.max is 5% of a core, its child groups share those 5% among themselves, however much their own cpu.max allows. So "no limits" in docker inspect only means Docker didn't put a limit on the container's own group. The container may have landed in someone else's group via --cgroup-parent (that's done to keep background jobs in a shared "cage"), and the limit hangs on that group. It's easiest to search bottom-up: the process's group path is in /proc/<PID>/cgroup, and a growing nr_throttled in cpu.stat shows at which level the kernel is throttling.

What's checked

  1. The thumbs container is running and hasn't been restarted
  2. By the cgroup settings the generator may use a whole core
  3. The generator really gets the CPU

What it looks like