Shell Programming and Scripting

Hi,

I have a little knowledge about mkfifo, first-in-first-out, a special file, a named pipe and it involves inter-process communication,
as multiple processes can write data into a single file.

Here, I would like to know how this is helpful in executing the below Oracle 'imp' command which reads from an export dump file.

mkfifo /home/tst_pipe
gunzip < /home/exp_dump.exp.gz > /home/tst_pipe &
imp / file=/home/tst_pipe log=/home/imp_dump.log fromuser=xyz touser=abc

At step #1, we create a named pipe file, step #2 unzip the exp dump into the pipe in the background and step #3 does the actual import job.

Step #2, extracts from a big zip file in the background, how at step #3, we are able to process the import command without waiting for the unzip
process to complete.

Does it mean, both gunzip and imp shares the same file parallely, whatever unzip has completed, it is processed by imp on a first-in-first-out basis.

Please give a brief overview about 'mkfifo' and how the above 3 statements work together well, thank you.

---------- Post updated at 01:59 PM ---------- Previous update was at 01:43 PM ----------

Hi,

I forgot to add one more point.

Here, we can import directly from the exp dump, after totally unzipping it. I have heard that, if the dump is too big, we have limited space in the mount point, we cannot extract it fully and import it.

So, in this situation, we can go for the above mentioned solution using mkfifo, it does the job well with very limited space restriction as well, please throw light on this aspect as well, thank you.

The trick for named pipes is that until one side has a server, the other side will not open. I forget which, but easy to try in the shell. The /sbin/mknod command makes named pipes on some systems. The serving process needs to open and serve enough times to satisfy all the clients that open.

I do not use named pipes but once in a blue moon, as in ksh and bash, you can make a named pipe that evaporates on its own with prameters of the form "<(commands that write to a pipe being read)" or ">( commands that read from the pipe being written". For instance: "comm -13 <(sort file1) <(sort file2)" makes sorted version of files on named pipes as comm input files to show lines that are new (comm -13 means output is minus (-) deleted (1) and common(3) ). (On systems without something like /dev/fd/0, bash makes named pipes in /var/tmp, and never cleans them up, unless the bash people acted on my bug report and you have that patch!)

You can also, on UNIX that have /dev/fd/ visible file descriptors, use /dev/stdin with a leading pipe, like this trivial example: "cat file1 file2 | sort /dev/stdin", and /dev/stdout with a trailing pipe, like this trivial example: "sort -o /dev/stdout file1 file2 | uniq -d". We had a 32 bit file program and a file over 4g, that ran fine with "old_prog -i /dev/stdin <big_file", because the shell opened the big file open64() and the old_prog opened the named pipe with open(). Technically, this is not a named pipe, but a cue to the kernel to dup() an inherited, already open, flat file fd as the open(). The /dev/fd/# or /proc/fd/# inodes are fake, honored by special kernel handling. This solution is simpler, runs faster and with less CPU, than "cat big_file | old_prog -i /dev/stdin", which is a real pipe. Of course, if you have a program that either reads stdin or writes stdout, either by default or when given a '-', you can use the unnamed pipe '|', like this trivial example: "gunzip <xxx.tgz | tar tf -".

So, between these shell/kernel helps, one wonders if you really need a named pipe, or just like it because if feels more file-like! Admittedly, some of my pipe processing shell trees did scare the kids! However, no file space was used, no space limits hit, no write or read disk overhead, no pages loaded with cached file data, and pipeline parallelism -- what's not to like!