Unix: Redirection
Standard input, output and error
In the previous chapter, you learnt to use the command
cat to show the content of a file on the screen. The
name of the file to show was given as a command line argument like
this:
$ cat unixpast.txt
Now try to run cat without specifying an input file, like this:
$ cat
Then type a few words on the keyboard, and press the
[Enter] key. The words you just typed is echoed on the
screen. Now hit [Ctrl]-D. This should make the command
line prompt reappear.
What happened?
When you run the cat command without specifying a file
to read on the command line, it defaults to reading from the standard
input (i.e. the keyboard). It keeps on reading line after line, until
it receives the character [Ctrl]-D, which in Unix means
end of file.
Also, whatever input cat receives is written on the
standard output (the terminal). This makes each line appear twice on
the screen, because it appears as both input and output.
The command cat, like most standard Unix commands
and programs is by default connected to three streams:
the standard input,
the standard output, and
the standard error.
Unix programs may of course use more streams of data than these
three, but these three are almost universal. By default, the
standard input reads from the keyboard, while the standard output
and standard error writes to the terminal.
Redirecting the standard output
Having access to standard input, output and error available with nearly every process is handy, because Unix also offer simple mechanisms to redirect all three streams.
To redirect the output of a Unix command, we use the
> character followed by the name of the file we want
to redirect the input to.
For example, to create a file called dogs1.txt
containing a list of dog breeds, you may start cat and then type in the names of some breeds.
After typing in a breed, press [Enter] to get to the next line.
Terminate input the by typing [Ctrl]-D (end of file).
$ cat > dogs1.txt spaniel terrier dalmatian [Ctrl]-D
In this case, no duplicate output appears on the screen. The
cat command still reads the standard input (the
keyboard). But because we've redirected the output, it now
goes into the file dogs1.txt instead of to the terminal.
After you get the command line prompt back, you ca check the contents of this file by typing:
$ cat dogs1.txt
The same list of dog breeds as those you typed in should be listed.
Exercise
Using the above method, create another file called
dogs2.txt with the following breeds:
“, dogo, bulldog, pit-bull, mastiff” (only one breed per
line). Use cat to view the contents of
dogs2.txt.
Appending to a file
Redirection also let us append text lines to an existing file. This is done by using two characters >> instead of one. So to add more dog breeds to the file dogs1.txt, you should type: something like:
$ cat >> dogs1.txt beagle corgi basset [Ctrl]-D
To view the contents of the file dogs1.txt on the screen, type:
$ cat dogs1.txt dogs2.txt
You should now have two files. The file dogs1.txt contains six dog breeds, and file dogs2.txt contains four dog breeds.
The cat command just keeps reading files that are named on the command line until the contents of all files that have been named have been read.
Concatenating files
The next thing to try out is to use the cat command to
concatenate (join together) the contents of dogs1.txt and
dogs2.txt into a new file called dogbreeds.txt.
To do this, type:
$ cat dogs1.txt dogs2.txt > dogbreeds.txt
What this command does is to read the contents of dogs1.txt and dogs2.txt and at the same time redirect the output to the file named dogbreeds.txt.
To view the contents of the new file, type:
$ cat dogbreeds.txt
It should now be a list of 10 dog breeds.
Redirecting the standard input
The command sort (introduced in the previous chapter) sorts a list alphabetically or numerically.
Like cat, it will by default read from the standard
input and write to the standard output. Just to try it out, start it
and type in the string “----” (four hyphen) followed by
four car names. As always when reading from standard input, you
signal end of file by hitting [Ctrl]-D. This is what to
type:
$ sort ---- ford fiat audi opel [Ctrl]-D
As soon as you hit [Ctrl]-D, the program will process
the input. In this case, this entails sorting the list typed in
alphabetically, and write the following result on the screen:
---- audi fiat ford opel
Now, having to type in input by hand every time we want to sort something is not practical. Instead, to sort the contents of a file, you should redirect input to come from a file instead of from the keyboard.
This is done by using the character < on the
command line. For example, to sort the concatenated list of dog
breeds, type:
$ sort < dogbreeds.txt
The list of breeds will be alphabetically sorted, and will appear on the terminal .
To output the sorted list to a file instead, redirect the output as well. Type:
$ sort < dogbreeds.txt > sortedlist.txt
To view the contents of sortedlist.txt, use cat, more, or less.
Redirecting the standard error
Now, try to use ls to list something that does not
exist. You'll get a similar response to this;
$ ls nosuchthing ls: cannot access nosuchthing: No such file or directory
Say you want to capture the error message in file name
error.log, to document the problem. A first approach is to
redirect standard output. Afterwards, you want to use wc
to check that the error message is in the log file. Try this:
$ ls nosuchthing > error.log ls: cannot access nosuchthing: No such file or directory $ wc error.log 0 0 0 error.log
Unfortunately, this did not work. The error message appeared on
the terminal, so it was obviously not redirected, and wc
reports that the log is empty.
The reason for failure was that the error message was not written
to standard output, but to standard error. This is a stream Unix
processes use for error messages as opposed to standard output. To
redirect standard output, we need remove the useless log file, then
try again, this time redirecting standard error to the file by using
2> on the command line:
$ rm error.log $ ls nosuchthing 2> error.log $ wc error.log 1 9 57 error.log
This is much better. We now have 1 line, 9 words and 57 bytes in the log file.
Pipes
Assuming we want to count unique words in the text file named unixpast.txt, the following three lines will get the job done:
$ tr -sc 'A-Za-z' '\n' < unixpast.txt > a.tmp $ sort -uf < a.tmp > b.tmp $ wc -w < b.tmp 502
The first line redirects input to come from the file
unixpast.txt and translates anything that isn't an ASCII
character into a newline, squeezing multiple newlines into a single
one (look up the tr manual page if that isn't instantly
obvious to you). It redirects the output of all this to a temporary
file. The second line redirects the input to read from this temporary
file, sorts the stream alphabetically, while removing duplicates and
ignoring case. It redirects output to a second temporary file. The
last line just counts the words in this second temporary file,
reporting that there is 502 of them.
So we have the answer (502 unique words). But the solution is kind of messy, with two temporary files are left behind. What we really want to do is to connect the data streams from these programs together, and not be bothered with temporary files. Unix let you do this, with pipes.
Pipes is how Unix connects the output stream of one program to the input stream of another, sometimes to great effect. Below is an alternative solution to the word counting problem. It is now only a single line, and no temporary files are created:
$ tr -sc 'A-Za-z' '\n' < unixpast.txt | sort -uf | wc -w 502
The symbol for a pipe is the vertical bar (|). put it between two commands on a single line, and the output stream of the first is piped to the input stream of the second.
Exercise
Using grep and sort, display all lines of dogbreeds.txt containing the letter “o”, and sort the result alphabetically.
Summary
| Command | Meaning |
|---|---|
command < file |
redirect standard input from file |
command > file |
redirect standard output to file |
command >> file |
append standard output to file |
cat file1 file2 > file3 |
concatenate file1 and file2 to create file3 |
command 2> file |
redirect standard error to file |
command1 | command2 |
pipe the output of command1 to the input of command2 |
