1. Home
  2. Docs
  3. Beginning UNIX on Raven
  4. Working with text files
  5. Finding unique lines

Finding unique lines

The uniq command compares adjacent lines in a file and collapses identical lines into a single line.  When combined with the sort command it is a way of identifying the unique lines in a file:

grep Chevy americanpie.txt | uniq

Drove my Chevy to the levee, but the levee was dry

Applied to our American Pie lyrics, we can see that the uniq command has no effect at all on the output compared to a simple cat display:

uniq americanpie.txt

But combination with sort, will output the unique lines of our lyrics:

sort americanpie.txt | uniq

A few extra tricks allow us to return only repeated lines using the –d flag:

sort americanpie.txt | uniq -d

And by combining the command with the –c (count) flag and an additional numerical sort, we can get a table of the number of times each line of lyrics is repeated:

sort americanpie.txt | uniq -c | sort -n