Standard Streams is one of the most fundamental topics in Linux, and many Linux users get confused about these. However, having the right domain about stdout, stderr, stdin and redirection give us powers that we can use in the terminal and shell scripts.

This one of two tutorials about streams and redirections in Linux.

What is a Stream in Linux?

Looking for the noun stream in the dictionary, we probably get a description like the following:

Stream - a continuous flow of liquid, air, or gas.

A good example of a stream is a river's stream. A river has a stream of water in the same way as programs have streams of data.

So, we can say that a stream in Linux is a continuous flow of data, which can be used by programs.

Types of Standard Streams

Linux programs have attached three standard streams by default:

• Standard Output (stdout)
• Standard Error (stderr)
• Standard Input (stdin)

So now let us start explaining the Standard Output Stream.

The Standard Output Stream (stdout)

In Linux, the Standard Output Stream technically is referred to as stdout. So, what is the purpose of the stdout in a computer's program?. To explain this, let us remember that a computer program typically has an input and an output:

So, the Linux programs generally send their output to the output stream. Let's see a concrete example using the Linux terminal and the echo program.

Let's check what the value of our HOME environment variable is. For that we're going to type the following command in our Linux terminal:

echo $HOME

The output of this command is the value of our environment variable HOME, which normally is the home directory of the user. For example, if the username is the word hacker, then the output for the above command could be a path like:

~$ echo $HOME
/home/thehacker

In the example above, the program echo sent its output to the stdout, and Linux displayed the data of this stream in the terminal's display.

Linux by default sends the data of the stdout to the terminal display. However, we can redirect the stdout to a different target, but before to explain redirection, let's see how the Standard Error works.

The Standard Error Stream (stderr)

The Standard Error Stream technically is referred to as stderr. The behavior of this stream is very similar to the stdout, but, as probably you are thinking, programs use this stream for their error output.

Let's make an example of the stderr using the command ls with the invalid option -9:

~$ ls -9
ls: invalid option -- '9'
Try 'ls --help' for more information.

The ls program produces an error, and then the error is sent to the stderr. Linux by default, just like the stdout, sends the stderr to the terminal display.

The stderr can also be redirected to a different target than the terminal display. So, in the next section, we are going to see how redirection works.

Stream Redirection

Stream redirection is one of the most powerful features in Linux. Using redirection, we can change the flow of a program's data stream using the Linux terminal or a shell script.

Redirecting the stdout using greater than > redirector

Linux terminal uses the greater than > character to redirect the stdout stream to a file:

~$ echo $HOME > myhome.txt
~$

The command above sends the stdout stream to the file myhome.txt. In case the file doesn't exist, then the file is created. Otherwise, if the file already exists, then the file is overwritten. Note that the terminal doesn't show any output in the terminal display.

The stdout can also be redirected using the number one followed by the greater than character: 1>. It is because Linux's programs have two output streams. The number one (1) identifies the stdout, and, as we'll see in the next section, the number two identifies the standard error stream.

So we can redirect standard output using any of the two following characters: >, 1>.

The following commands are the same:

echo $PATH > mypath.txt

echo $PATH 1> mypath.txt

Redirecting the stdout using double greater than >> redirector

As we saw in the section above, the greater than > redirector creates a new file or overwrites an existing one. However, you can append the stdout to an existing file, instead of overwriting it, using the double greater than >> redirector.

~$ echo $HOME > myhome.txt
~$ cat myhome.txt
thehacker
~$ echo $HOME >> myhome.txt
~$ cat myhome.txt
thehacker
thehacker

Redirecting Standard Error Stream (stderr)

So, as we said above, the stderr is identified in Linux with the number two. So, to redirect the stderr, we use the number two followed by the greater than character: 2>.

To see stderr working, let's intentionally pass the invalid option -9 to the ls command, and then redirect the stderr to a file called error.log:

~$ ls -9 2> error.log
~$

Since we are redirecting using the 2> characters, the command sends the stderr to the error.log file.

Like with the stdout, we can use the double greater than >> redirector to append stderr. For that, we put the number 2 before the redirector.

~$ ls -9 2> error.log
~$ cat error.log
ls: invalid option -- '9'
Try 'ls --help' for more information.
~$ ls -9 2>> error.log
~$ cat myhome.txt
ls: invalid option -- '9'
Try 'ls --help' for more information.
ls: invalid option -- '9'
Try 'ls --help' for more information.

Redirecting stdout and stderr at the same time

If a command-line, which only is redirecting the stdout, produces an error, then the stderr is sent to the terminal display; however, an empty file is also created.

~$ ls -9 > files.txt
ls: invalid option -- '9'
Try 'ls --help' for more information.

The reason why the command above creates an empty file is that Linux resolves redirection before to execute the command.

The same happens if we make the opposite, redirecting the stderr with a command which doesn't produce an error:

~$ echo $HOME 2> error.log
/home/thehacker

What if we need to save both output streams to files? For that we can use both redirects in the same command line:

~$ echo $HOME 1> myhome.txt 2> error.log
~$

Probably now you are wondering, what if we need to redirect both outputs to the same file? We are going to answer this question in the next section.

Redirecting stdout and stderr to the same file

There are circumstances where we would wish send both outputs (stderr and stdout) to the same file. To see how to do it in practice, let's create a bash script which produces both outputs.

~$ echo "echo $HOME; ls -9" > both.sh
~$ chmod u+x both.sh

We created a script called both.sh which executes two command statements, the first statement echo $HOME produces a stdout, and the second statement ls -9 produces a stderr. Then we use the command chmod u+x both.sh to set the execution permission.

So, we execute the new both.sh script, and then we get both outputs in the terminal display:

~$ ./both.sh
/home/thehacker
ls: invalid option -- '9'
Try 'ls --help' for more information.

If we want to save these outputs in a file called debug.log we can proceed with the command below:

~$ ./both.sh 1> debug.log 2>&1
~$

The syntax above might seem a bit weird. So, let's see how it works.

The above image shows that stderr is being redirected to the stdout; however, stdout has already been redirected to debug.log.

So we can produce the same result changing the order in which the streams are redirected.

Additionally, we can redirect both stdout and stderr in a simpler way using an ampersand before the greater than character, as you can see in the following example:

~$ ./both.sh &> debug.log
~$

Redirecting stdin using less than < redirector

We can redirect the content of a file to the stdin of a command using the less than < redirector. Check the following example:

~$ echo $HOME > myhome.txt
~$ cat < myhome.txt
thehacker

As you see in the example above, we are redirecting the content of myhome.txt to the stdin of the cat command. So the cat command shows the content of the file.

However, maybe you are surprised that this redirector is producing the same result as when we pass the filename, as an argument, to the cat command:

~$ echo $HOME > myhome.txt
~$ cat myhome.txt
thehacker

So, maybe you don't see any sense to use the stdin redirector, but there are some other commands, like mail or mysql which doesn't have filename argument, and they need sending file input in the stdin.

The double less than << redirector

There is a double less than << redirector that we can use to redirect the terminal input to the stdin of some command. See the next example:

~$ wc -c << EOF
> This is a text which I would like to know how many characters there are here
> EOF
77

In the previous example, the command wc -c counts the number of characters of a file or another input. In this particular case, we are using some text entered in the terminal as the input. With the double less than << redirector and the word EOF we are indicating that we are going to start entering some text, and the text finish with the word EOF.

You can use any other word as the delimiter, instead of using `EOF:

~$ wc -c << EndOfFile
> This is a strange text
> with two lines
> EndOfFile
38

The triple less than <<< redirector

The triple less than redirector <<< is very similar to the double less than <<, but triple is normally used to redirect just one line of text:

~$ wc -c <<< 'this is just one line of text'
30

The Pipe | redirector

Streams can also be redirected to a command instead of a file, using the Pipe ( | ) redirector.

To see how this works let us create a file with three lines using the printf command:

~$ printf "Ubuntu 18.04\nCentos 7.3\nUbuntu 16.04\n" > dist.txt
~$ cat dist.txt
Ubuntu 18.04
Centos 7.3
Ubuntu 16.04

Now we can search in the content of the dist.txt file, using pipe redirector and the command grep -i, for the Ubuntu distributions:

~$ cat dist.txt | grep Ubuntu
Ubuntu 18.04
Ubuntu 16.04

The previous sentence is redirecting the output of cat dist.txt to the stdin of grep -i ubuntu command:

Note: the -i parameter of the grep command, is used to do a case-insensitive search.

The tee command

The tee command used with the pipe redirector | has the same behavior like the > redirector, sending the stdout to a file; however, this also sends the stdout to the terminal.

~$ cat dists.txt | tee names.txt
Ubuntu 18.04
Centos 7.3
Ubuntu 16.04
~$ cat names.txt
Ubuntu 18.04
Centos 7.3
Ubuntu 16.04

A good utility of tee is using it with a second | pipe, to send the stdout to another command:

~$ cat dists.txt | tee names.txt | grep -i ubuntu
Ubuntu 18.04
Ubuntu 16.04
~$ cat names.txt
Ubuntu 18.04
Centos 7.3
Ubuntu 16.04

You can see the workflow of the command above in the following image:

The xargs command

The xargs command splits a stdout to use each split element as an argument for another command.

Before to explain an example of the command xargs, first, let's see how the command printf works.

With the command printf we can format some text.

For example, suppose you want to prefix each distribution of the file dist.txt with the word Linux. We can do that with the following command:

~$ cat dists.txt | xargs -d '\n' printf "Linux %s\n"
Linux Ubuntu 18.04
Linux Centos 7.3
Linux Ubuntu 16.04

The stdout above shows each distribution of the file dists.txt with the prefix Linux. It has been possible because the xargs -d '\n' command has split each line of the file dists.txt to be used as arguments for the command printf "Linux %s\n":

Note: The argument -d '\n' of the command xargs is used to indicate that the file dists.txt is split using the new line \n character as the delimiter.

The argument "Linux %s\n" of the command printf formats the output for each line of the file dists.txt.

Summary

• Types of Standard Streams:
• • Standard Output (stdout).
  • Standard Error (stderr).
  • Standard Input (stdin).
• > or 1> redirects the stdout.
• 2> redirects the stderr.
• 2>&1 redirects stderr to stdout.
• &> redirects stdout and stderr.
• < redirects to stdin.
• << redirects text to stdin using arbitrary delimiters like EOF.
• <<< redirects text to stdin.
• | redirects the stdout to the stdin of a command.
• | tee redirects the stdout to a file and the terminal.
• | xargs splits the stdout to use each split element as an argument for some command.