What a pipe actually does

In shell syntax, | occupies one character. At runtime it is plumbing: the shell creates kernel pipe objects, starts commands with selected file descriptors attached, and tracks their completion.

cat access.log | grep ' 500 ' | sort | uniq -c

Model boundary: The moving records in the figure are a reading aid. A Linux pipe is a byte stream without message boundaries 3. Programs may read, write, and internally buffer different byte counts; the kernel does not know that one visual token represents one log line.

Three conventional descriptors

A process normally begins with three open file descriptors:

Number	Name	Conventional direction
0	standard input	bytes read by the process
1	standard output	ordinary result bytes written by the process
2	standard error	diagnostics written separately

These are integer references to open file descriptions, not permanent device names. A shell can arrange for descriptor 1 to refer to a terminal, regular file, socket, or the write end of a pipe. Descriptor duplication makes two numbers refer to the same open file description 4.

The pipeline operator connects one command’s standard output to the next command’s standard input. Standard error is not included by plain |:

cat stdout ──pipe──▶ grep stdin
cat stderr ───────────────────▶ terminal

Bash has |& as shorthand for piping both streams after other command redirections, but it is a shell extension; portable scripts should spell out their intended descriptor routing and test it in their target shell.

The shell builds a connected process graph

For the example pipeline, the useful conceptual setup is:

access.log → cat → [pipe A] → grep → [pipe B] → sort → [pipe C] → uniq → terminal

The shell creates pipe endpoints, starts each command with the relevant endpoint on descriptor 0 or 1, and closes copies it no longer needs. Closing those extra copies matters: a reader sees end-of-file only after every write endpoint for that pipe is closed.

Pipeline commands can run concurrently. The shell does not generally wait for cat to finish, save all its output, then start grep. Bash explicitly defines a pipeline as commands connected by pipes and waits according to foreground or background syntax 1; POSIX defines the portable shell contract 5.

A safe timing experiment makes concurrency visible:

{ printf 'producer started\n' >&2; sleep 1; printf 'one record\n'; } |
  { printf 'consumer started\n' >&2; IFS= read -r record; printf 'got: %s\n' "$record"; }

Both “started” diagnostics can appear before the one-second producer delay ends. They travel on stderr to the terminal; only one record travels through the pipe.

Buffers decouple, then apply backpressure

The kernel holds unread pipe bytes in a bounded buffer. This lets a producer run slightly ahead of a consumer. It does not provide infinite storage.

• A blocking read from an empty pipe waits while a writer still exists.
• A blocking write to a full pipe waits for a reader to consume data.
• If all read endpoints are closed, a writer receives SIGPIPE or a failed write.
• If all write endpoints are closed, a reader eventually observes end-of-file.

Capacity has varied across Linux versions and can be queried or adjusted within limits; robust programs depend on blocking semantics, not one memorized byte count 3.

This bounded experiment deliberately gives the consumer a one-second late start:

seq 1 200000 | { sleep 1; wc -l; }

The command should finish with 200000. During the sleep, seq can fill the pipe and block. After wc begins reading, backpressure clears. It is a flow control mechanism—not a deadlock by itself.

Application buffering is a separate layer. A C library may buffer stdout more aggressively when it is a pipe than when it is a terminal. That can explain why output arrives in bursts even when the kernel pipe still works correctly.

Redirections are ordered operations

Shell redirections change descriptors from left to right 2. These two command tails are therefore different:

>combined.log 2>&1   stdout → file, then stderr → current stdout (same file)
2>&1 >output.log     stderr → current stdout, then stdout → file

In the second form, stderr keeps pointing where stdout pointed before stdout moved. “Send 1 and 2 somewhere” is an insufficient model; each token mutates the descriptor table at that moment.

Caution: Output redirections create or truncate files. Before applying either form, verify the exact destination; use a disposable path for practice.

When a redirection appears on a pipeline command, it can override the endpoint the pipeline would otherwise install. Parentheses and braces also affect which process or compound command owns a redirection. Draw the descriptors when syntax gets dense.

Whose exit status is it?

Every process has its own termination status, but a shell pipeline must expose a single status to if, &&, or $?. By default, Bash reports the last command’s status. With Bash pipefail, the pipeline fails if any command fails, using the rightmost non-zero status; if all succeed, it is zero 1.

sh -c 'exit 7' | sh -c 'exit 0'
printf 'default pipeline status: %s\n' "$?"

bash -o pipefail -c "sh -c 'exit 7' | sh -c 'exit 0'"
printf 'Bash pipefail status: %s\n' "$?"

The first status is normally 0; the second is 7. pipefail is not in the portable POSIX shell language. Bash implements it, and current Dash and BusyBox ash releases may implement it too; older or different shells may not. Test the chosen interpreter and version. Also remember that expected early exits—such as a producer receiving SIGPIPE after head has enough input—can become visible failures.

In interactive Bash, PIPESTATUS preserves each component status immediately after the pipeline:

printf 'alpha\nbeta\n' | grep beta | wc -l
printf 'component statuses: %s\n' "${PIPESTATUS[*]}"

Reading another command can replace that array, so capture it promptly.

Is cat file | grep wrong?

grep pattern file can open a regular file itself, so the leading cat often adds a process and a pair of descriptors without adding capability. Removing it can be a reasonable simplification.

But “useless use of cat” is not a correctness law. A producer may be a remote stream, decompressor, generator, or deliberate abstraction. cat can make a teaching data source explicit or concatenate several inputs. The sharper review question is: what role does each process play, and is that role worth its cost?

Static trace of the example

1. cat reads bytes from access.log; its stdout enters pipe A.
2. grep reads pipe A. Non-matching bytes do not reach its stdout; diagnostics still use stderr.
3. sort reads all matching lines before it can promise final order; it may use memory and temporary files according to input size and implementation.
4. uniq -c counts adjacent equal lines, so sorting first groups equal records.
5. If uniq slows down, pipe C fills, then sort blocks on write. Sustained pressure can propagate backward through earlier pipes.
6. The shell derives one pipeline status using its configured rule.

That is the durable model: concurrent processes, mutable descriptor tables, bounded byte streams, and explicit status policy.