Piping is a very important concept in UNIX. Put simply it allows you to send the output of a command/program directly into the input for another one. This allows the creation of chains of commands which allow some quite powerful manipulation of data files without the need to resort to a programming based solution.
Linked to piping is the concept of redirection. By default, the output of many commands is printed onto the screen. By using redirection we can send this output to a file, either writing/overwriting a file or appending the output to an existing file.
Piping is achieved using the | syntax. As an example we can string together 3 different commands to pull out the 2 classic lines from our American Pie lyrics.
cat americanpie.txt | tail -n 4 | head -n 2
Working through the logic of what we have done. The command cat displays the file, which is then piped into tail to return the final 4 lines. These lines are then piped into head which takes the top 2 lines (of the final 4 we already selected) and returns these to the display.
If we want to save the results of our previous command to a file, we can just append a > character followed by our desired filename.
cat americanpie.txt | tail -n 4 | head -n 2 > junk.txt
Occasionally we might wish to redirect our output to file, but also have it displayed on screen as well to check the contents. There are a number of ways to achieve this in a single line command. Using the ; character, we can string together a number of lines of code which are executed sequentially:
head americanpie.txt > junk.txt; cat junk.txt
Alternatively (and more neatly), we can use the tee command to fork the output stream to both the display and a file:
americanpie.txt | tee junk.txt
Finally we may wish to append to data to a file. For instance we can create an output log using the date command and the >> append syntax:
date > junk.txt
head americanpie.txt >> junk.txt
Summary of piping and redirection commands
||Pipe the output of one command to another|
||Redirect the standard output to a file|
||Append the standard output to a file|
||Fork the output to both screen and a file|