Pipes and filters

Pipes and filters is a software design pattern invented by Douglas McIlroy. Its inclusion on the Unix operating system was the first implementation of his vision of software componentry. The main difference between pipes and filters and newer software component technologies is that it only allows information to flow in one direction, like water flows in a pipe. In the context of Unix operating systems, a pipe signifies that the output of one program feeds directly as input to another program. \nA file can be used to provide input or to hold the output (in the pipe metaphor, a file would be a container).\nThe Unix shell uses the pipe character (|) to join programs together. A sequence of commands joined together by pipes is known as a pipeline. For creating this mechanism, all Unix tools have access to three distinct special files:

• stdin—the standard input file,\n* stdout—the standard output file, and\n* stderr—the standard error file.

By joining one tool's stdout to another tool's stdin, a pipeline is formed. Errors are sent to a side track and accumulated. This pattern is extensively used in Unix, but was also ported to DOS, OS/2, Windows NT, BeOS, MacOS X and other operating systems with a text console. Often filter programs form the constituent programs in a pipeline, performing their bits of data massaging. An example of a pipeline:

cat * | grep "alice" | grep -v "wonderland" | wc -l

will print out the number of lines in all files in the current directory which contain the text "alice", but not the text "wonderland". The pipeline has four parts:\n* cat * concatenates the text of all files to its stdout;\n* grep "alice" reads its stdin as lines, and prints on its stdout only those lines which contain the word "alice";\n* grep -v "wonderland" reads its stdin and prints on its stdout only those remaining lines which do not contain the word "wonderland" (mnemonic: -v inverts the selection);\n* wc -l counts the lines on its stdin, and prints a line count on its stdout. Pipes and filters can be viewed as a form of functional programming, using byte streams as data objects.\nThis pattern encourages the use of text streams as the input and output of programs.\nThis reliance on text has to be accounted when creating graphic shells to text programs. Tools like netcat can connect pipes to TCP/IP sockets, following the Unix philosophy of "everything is a file". An attempt to bring together the text-stream interface and hypermedia is XMLTerm, where the input and the output is XML and can be interpreted graphically.