Pipeline

\nThis article refers to the mechanical, electrical, and software systems meaning of pipeline. For pipelines used to transport fluids like water or petroleum, see pipeline transport.

---

The term pipeline has meaning in electrical and mechanical systems, as well as in software. \nIn general, the term represents the concept of splitting a job into subprocesses in which the output of one subprocess feeds into the next (much like water flows from one pipe segment to the next).

Table of contents

1 Mechanical analogy 2 Pipelined processors 3 Software pipelines 3.1 Hartmann pipelines

Mechanical analogy

\nA mechanical example of a pipeline is a washer/dryer system for clothing. \nInstead of having one unit that both washes and dries, we have two units that together form a pipeline (the output of the washer enters the drier). \nIf washing takes 1 hour and drying takes 1 hour, the pipeline allows us to finish a full load of laundry every hour, compared to every 2 hours if you had a single (non-pipelined) unit that washed and then dried. \nIt still requires two hours for an item of clothing to complete its wash/dry cycle of course.

Pipelined processors

\nElectrically, pipelines are used in microprocessors to allow complex logic sequences to execute at faster speeds. Pipelines are related to the engineering concepts of throughput and latency. \nSee Instruction pipeline and Classic RISC pipeline for a better discussion.

Software pipelines

\nIn computer software, a pipeline is a command line feature prevalent in UNIX and other UNIX-like operating systems. \nDouglas McIlroy, one of the authors of the early UNIX command shells, noticed that much of the time they were processing the output of one program as the input to another. The UNIX pioneers established a means of chaining the running programs together as co-processes so that the output of the first program becomes the input to the second. \nThis was to become the famous pipes and filters design pattern. \nA pipeline may be extended to any number of commands with the output of one serving as the input to the next.\n

Unix pipes

\nCommonly filter programs are used in a UNIX pipeline and they usually obey a few conventions: line structured records, reading data from the standard\ninput, and writing to the standard output. Below is an example of a pipeline that implements a kind of spell checker for this page.

curl http://www.wikipedia.org/wiki/Pipeline |\n sed 's/[^a-zA-Z ]//g' |\n tr 'A-Z ' 'a-z\\n' |\n grep '[a-z]' |\n sort -u |\n comm -23 - /usr/dict/words

Here is an explanation of the pipeline:

• First the curl program obtains the contents of this web page.\n*The contents of this page are piped through sed, which removes all characters which are not spaces or letters.\n*tr then changes all of the uppercase letters into their corresponding lowercase counterparts, and converts the spaces in the lines of text to newlines.\n*Each 'word' is now on a separate line.\n*grep is used to remove lines of whitespace.\n*sort sorts the list of 'words' into alphabetical order, and removes duplicates.\n*Finally, comm finds which of the words in the list are not in the given dictionary file (in this case, /usr/dict/words).

Hartmann pipelines

\nJohn Hartmann, a Danish engineer with IBM,\nextended the basic pipes and filters paradigm in a number of useful ways.\nHis product, a/k/a CMS Pipelines, is available on a number of IBM platforms. Some of the salient characteristics that distinguish Hartmann Pipeline from ordinary Unix pipes are:

• Filters may have multiple inputs and multiple outputs. For example, a selection filter can send the found records down one output pipe and the not found records down another.\n*A linear notation for representing pipeline networks.\n*An interface that allows REXX programs to act as filters.\n*A pacing strategy in the Pipeline supervisor that allows, for example, a stream to be split, say by a selection filter, and the records on the output legs to be processed by other filters, then merged by a join filter and have the record order preserved in result stream.

The utility of the many filters supplied with the program is exemplified by the LOOKUP filter: This arrangement allows one to use other filters to prepare the dictionary, \nor master records for input to LOOKUP from whatever source is required.\nThe many Input/Output filters, or drivers, \nallow a Hartmann Pipe to interact directly with a variety data sources,\nfrom files, to the system itself, and such things as TCP/IP ports.\nThe repertoir of filters and drivers is rich enough that one could,\nfor example, write a server that consisted solely of a Hartmann pipeline. Category:Computer terminology