The uniq command compares adjacent lines in a file and collapses identical lines into a single line. When combined with the sort command it is a way of identifying the unique lines in a file:
grep Chevy americanpie.txt | uniq
Drove my Chevy to the levee, but the levee was dry
Applied to our American Pie lyrics, we can see that the uniq command has no effect at all on the output compared to a simple cat display:
uniq americanpie.txt
But combination with sort, will output the unique lines of our lyrics:
sort americanpie.txt | uniq
A few extra tricks allow us to return only repeated lines using the –d flag:
sort americanpie.txt | uniq -d
And by combining the command with the –c (count) flag and an additional numerical sort, we can get a table of the number of times each line of lyrics is repeated:
sort americanpie.txt | uniq -c | sort -n
