>_ IzzyLab

Hands-on task

This task is step 14 of the course “Running a server”

The queue worker's log ate the disk

medium~8 min

The queue worker's log /var/log/queue/worker.log has grown to 30 MB. The worker keeps the file open, and it can't be restarted. Empty the log right now without deleting or recreating the file, and for the future create the config /etc/logrotate.d/queue-worker: size-based rotation, and since the worker can't reopen its log — using copytruncate.

What you'll practise:
  • truncate
  • inode
  • logrotate
  • copytruncate
  • open descriptors

Topic breakdown

A process that opened a file writes into the inode, not the name: if you delete the log and create a new one, the worker keeps writing into the deleted file, and the space doesn't come back until a restart. So in an emergency space is freed by truncating the same file to zero — truncate -s 0 or redirecting nothing into it, and the inode doesn't change. For the future, rotation is handed to logrotate: a block in /etc/logrotate.d/ describes the path, the size threshold and the number of copies rotate. Ordinary rotation renames the file, and a program that can't reopen its log keeps writing into the old one; the copytruncate directive instead copies the contents and truncates the original file in place.

What's checked

  1. the log is emptied
  2. the file was emptied, not recreated
  3. logrotate: size-based rotation with copytruncate
  4. the logrotate config has no errors

What it looks like