>_ IzzyLab

Hands-on task

This task is step 16 of the course “Disks, links and free space”

Find duplicates by content

hard~12 min

In /root/files there are files, and exactly two of them have identical content (albeit with different names). Find this pair and write their names (basename), one per line, sorted alphabetically, to /root/answer.txt.

What you'll practise:
  • checksums
  • md5sum
  • duplicate detection
  • uniq
  • content comparison

Topic breakdown

Identical names or sizes don't yet mean identical content — you have to compare the data itself. Byte by byte that's expensive, so you take a checksum: files with identical content will have matching hashes (md5sum), and different ones will almost certainly diverge. Having computed the sums of all files, you look for the repeated hash (sorting plus selecting non-unique lines by the hash field) and output the files with it. This is the standard deduplication technique — comparing by hash instead of directly comparing content.

What's checked

  1. duplicates found by content

What it looks like