>_ IzzyLab

Hands-on task

This task is step 19 of the course “Troubleshooting and recovering a Linux server”

File deleted, but space not freed

hard~12 min

The log /var/log/svc/big.log was deleted and the directory is empty — but the disk space wasn't freed: some process still holds the file open. Several background workers are running on the server, and the process list doesn't show which of them is to blame. Find the PID of the process holding the deleted file, write it (just the number) to /root/holder.txt, and stop only that process — so the space comes back and the other workers keep running.

What you'll practise:
  • /proc/PID/fd
  • file descriptors
  • deleted files
  • kill
  • freeing disk space

Topic breakdown

Deleting a file (unlink) removes only the name from the directory; the data itself lives on while the inode has open descriptors. So "I deleted it and it didn't help" is normal behavior, not a malfunction: the space comes back when the last descriptor is closed, i.e. when the process exits or reopens the file. You can find the holder via /proc/<PID>/fd, where the kernel marks such targets as (deleted) (lsof +L1 and fuser show the same, if installed). Hence the practice for logs: don't rm a log of a running service, use truncate -s 0 or rotation with a reopen signal — then the descriptor stays valid.

What's checked

  1. holder.txt contains the holder's PID
  2. the deleted file is no longer held by anyone
  3. the other workers are running

What it looks like