Pipeline (software)

A pipe or pipeline (English pipe ) is a data stream between two processes through a buffer with the principle of First In - First Out (FIFO). This simply means that a result of a computer program is used as input of another program. Pipes were invented in 1973 by Douglas McIlroy for the Unix operating system.

Example

The following example is counted by means of a pipe, how many files are stored with specific file name extension in a directory:

Ls- R Pictures | grep -ci '\ jpg $. ' The ls program searches the directory " images " and its subdirectories with recursion, the names of all the files from. Instead of listing them directly on the screen is determined by the pipe symbol | is started automatically the grep program, which takes the result of ls as input. grep searches in the list generated by ls using a regular expression to all. " jpg " ending name and returns the number of. By adding additional pipes further processing can be performed to calculate, for example, the space occupied by these files storage.

Creating a pipe

Under most operating systems and Programmiersprachern be the requirement of a pipe by the system call pipe () from the operating system two access identifiers (English: handles) returned that are needed for writing into and reading from the pipe. Even child processes inherit the access to these handles. With the termination of the last process which has access to an active pipe, it is terminated by the operating system.

Pipe variants

There are anonymous and named pipes.

Anonymous pipes are three significant limitations:

• They can be used in only one direction: one process writes, reads another.
• They can be used only for communication between closely related processes.
• The maximum amount of data, which may include a pipe, is relatively small.

Named Pipes ( Named Pipes ), however, can also be used for communication between processes that are not related to each other and may reside on different computers in a network beyond. They are more flexible than anonymous pipes and are suitable for so-called client-server applications ( it can also be implemented RPCs ). Named Pipes allow communication in both directions, that is, data can be exchanged in full-duplex operation between the processes.

Any process that knows the name of a named pipe, using this name can connect to the pipe and thus connect to other processes.

Pipes in operating systems

Pipes are implemented in different operating systems, most offer both anonymous and named pipes.

OS / 2

OS / 2 knows anonymous and named pipes. Named pipes are among the most powerful IPC methods that OS / 2 has to offer. If a server process creates a named pipe, it can generate multiple instances of this pipe, all of which are addressed under the same name: A named pipe can also operate in multiplex mode, so that a single server process can service multiple clients simultaneously.

Unix

Anonymous pipes

Pipes are on Unix one of the most powerful tools to allow the sequential execution of commands on a given dataset.

In an anonymous pipe communication is limited to several processes of the same origin. This ( original ) relationship is formed mostly by Forks. In the shell, an anonymous pipe is the start time of the program by entering a | created character " ." The shell is then the ( common ) parent process of all processes and does the Forks automatically.

Example:

Grep '. sshd * Invalid user. ' / var / log / messages | awk ' { print $ NF } ' | sort-u Here is the system log file after the keyword " sshd " is searched, the same line of text "Invalid user" follows. Then the last field cut out of the message (" awk ...") that contains the IP address of the computer went out from which the ssh access. Finally, these IP addresses are sorted so that they do not occur more than once ( "sort - unique" ).

The programs involved in such a pipe take (except the first ) input data from its standard input and receive filters ( except the last ) output data to its standard output ready, see also Filter (Unix). Requires an application to specify file names for the input or output, so you can achieve as a file name often by specifying a minus sign that is written to standard output or read from the standard input. If they have implemented this convention, so writing to or reading from a pipe can also be realized with such applications.

Example:

Tar cf - / home / user / ogg / mycolouringbook | ssh-l user server " cd / var / ogg && tar xvf - " Here are the contents of a directory is packed with tar to an archive, sent over an SSH connection to another computer and decompressed there.

Named Pipe

A named pipe, also called a FIFO ( first-in of -first-out ) is a pipe that can be opened by two processes at runtime using a file name for reading or writing. In a named pipe, the process must have no common root, the processes must be authorized only for access to the pipe and to know the name of the pipe.

Example:

Mkfifo einefifo cat / var / log / messages > einefifo & grep sshd < einefifo FIFOs outlast them using processes, because they are part of the file system. However, a FIFO have no content, so long as it is open by a process. This means that the buffered content is lost if a writing process to an end of the tube closes without a reading process has opened the other end.

Using a one-way Unix pipe

This source code has the same effect as the shell command who | sort

# include # include # include # include   int main (void ) {   / / Pipe_verbindung for reading and for writing pipe_verbindung int pipe_verbindung;   / / Initialization by the function pipe pipe ( pipe_verbindung );   / / Create child process if ( fork () == 0) {          / / Dup2 connects the file descriptor of the pipe with the standard output file descriptor          dup2 ( pipe_verbindung, 1);            / / Read the output must be closed, as this process reads nothing          close ( pipe_verbindung );             / / Execute command, standard output of command is connected to the pipe          execlp ( "who", "who", NULL);          } / / Then create the second child process else if ( fork () == 0) {          dup2 ( pipe_verbindung, 0);          close ( pipe_verbindung );          execlp ( "sort", "sort", NULL);          } } The writing process ( process1 ) initially also have read access to the pipe (unidirectional). Therefore, he needs to lock for reading its file descriptor (0). Likewise, the reading process ( Process2 ) has first write access to the pipe. Therefore, he needs to lock for writing its file descriptor (1). If the unneeded descriptors not locked, there will be complications: for example, has Process1 If no data to send more, it terminates. However Process2 will not terminate, there's a file descriptor ( 1) writing ( his own ) is set for further entries on the pipe. He waits, but there are no data. Another possible scenario is that process1 can not be terminated, because he has read access to the pipe and forever waiting for data down an opponent, but never which will arrive because it is firstly long since terminated, and secondly never sent which has.

Windows

Windows knows anonymous and named pipes. Named pipes can be over the pipe API analogous to the SMB shares as \ \ ServerName \ pipe \ pipeName appeal.

Using an unnamed pipe ( anonymous pipe ) in C

The program reads a user input and then uses a pipe to communicate the data to a child process. This converts all input to uppercase ( toupper ) and outputs them.

# include # include # include # include # include # include # include   / / The maximum length of the input is set to 2048 bytes. # define MAX_ZEICHEN 2048   int main (void ) { int fd, n, i; pid_t pid; char line [ MAX_ZEICHEN ];   / / We create the pipe. If a malfunction occurs, gives the / / Function return -1, so that we have here Possible errors / / Intercept and can be treated. if ( pipe (fd) <0) fprintf ( stderr, " Failed to create pipe () ");   / / A child process is created. if (( pid = fork () )> 0) { / / In the parent process close ( fd); fprintf (stdout, "parent :"); fgets (line, MAX_ZEICHEN, stdin); write ( fd, line, strlen ( line ) );   if ( waitpid (pid, NULL, 0) <0) fprintf (stderr, "Failed to waitpid ()"); }   / / In the else branch gets only the child process else { / / In child process close ( fd); n = read ( fd, line, MAX_ZEICHEN );   for (i = 0; i < n; i ) row [ i] = toupper ( line [i ]); fprintf ( stderr, " Child :");   write ( STDOUT_FILENO, line, n); } exit ( 0); } history

Pipes were 1972/73 invented by Douglas McIlroy for the Unix operating system. As early as 1964 he had called at the beginning of the Multics project (the predecessor of Unix) in a memo:

"We shouldhave some ways of connecting programs like garden hose - screw in another segment When It Becomes Necessary to massage data in another way. "