>_ IzzyLab

Hands-on task

This task is step 17 of the course “Disks, links and free space”

Deduplicate: keep one copy of each

hard~10 min

Over the years /srv/docs has piled up copies of the same documents under different names (including names with spaces and parentheses). Keep one copy of each unique file, deleting the rest. The kept file's name doesn't matter — what matters is that no unique content is lost.

What you'll practise:
  • md5sum
  • checksums
  • awk
  • associative array
  • names with spaces

Topic breakdown

Finding duplicates relies on a checksum of the content: identical data gives an identical hash regardless of the file's name, timestamps and permissions. The classic pipeline is to compute the sums, sort them so matches end up next to each other, and keep the first line of each group. In awk an associative array plays the role of "already seen": the postfix seen[$1]++ returns 0 the first time a key appears and a non-zero value afterwards, which separates the original from the copies. File names are a separate difficulty: md5sum separates the hash and the name with two spaces, and the name itself may contain spaces, so splitting by fields breaks and you need to cut off a fixed prefix instead.

What's checked

  1. one file left per distinct content
  2. no duplicates among the remaining files
  3. all distinct contents are preserved

What it looks like