>_ IzzyLab

Hands-on task

This task is step 18 of the course “Text in the terminal: grep, sed, awk”

Remove duplicates, preserve order

medium~5 min

/root/emails.txt has a list of addresses with duplicates. Create /root/unique.txt, where each address appears once, and the order matches the first occurrences in the source file (a plain sort -u won't do — it breaks the order).

What you'll practise:
  • awk
  • associative arrays
  • deduplication
  • pipeline
  • redirection

Topic breakdown

The task is about the difference between deduplication and sorting: sort -u removes duplicates but reorders lines alphabetically, whereas here you need to preserve the order of first appearance. So sorting is not allowed — you must go through the file top to bottom and remember already-seen lines in a set (associative array), printing a line only on its first encounter. This is a classic pattern for awk: it has arrays keyed by a string and a "condition → print" model into which the whole logic fits compactly. The result is redirected into a new file.

What's checked

  1. /root/unique.txt has all unique addresses
  2. duplicates removed
  3. order of first occurrences preserved

What it looks like