Sponsored Content
Top Forums Shell Programming and Scripting Processing a file list via named pipe Post 302808773 by benalt on Friday 17th of May 2013 01:13:04 PM
Old 05-17-2013
Processing a file list via named pipe

I have a ksh93 script I use that processes a file list in the order that they exist in the list. I would like to speed up processing of the list by having multiple processes handle it at once. I was thinking that perhaps a good way to handle this would be to write the list to a named pipe and some number of processes that I can specify would be created and read from the named pipe the file it could work on and when finished with that file it would again read the named pipe until nothing is left. Each process would read the named pipe after finishing processing a file. The files then would be processed close to the order they existed in the file list which I need done.

My only consideration is regarding how to stop the processes from then hanging waiting for more input from the named pipe when there are no more files left to process. Would the best way to handle that be to write several times some kind of exit flag to the named pipe so that the processes as they read the next file from the list will instead encounter the exit flag and know to exit or is there a better way?
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

IPC using named pipe

Hi All, I am facing a vague issue while trying to make two process talk to each other using named pipe. read process ========= The process which reads, basically creates FIFO using mkfifo - ret_val = mkfifo(HALF_DUPLEX, 0666);) func. It then opens the pipe using open func - fd =... (1 Reply)
Discussion started by: sharanbr
1 Replies

2. Programming

IPC using named pipe

Hi All, I am facing a vague issue while trying to make two process talk to each other using named pipe. read process ========= The process which reads, basically creates FIFO using mkfifo - ret_val = mkfifo(HALF_DUPLEX, 0666) func. It then opens the pipe using open func - fd = open... (2 Replies)
Discussion started by: sharanbr
2 Replies

3. UNIX for Dummies Questions & Answers

Named PIPE

Gurus, I've a File Transaction Server, which communicates with other servers and performs some processing.It uses many Named PIPE's. By mistake i copied a named PIPE into a text file. I heard that PIPE files shouldn't be copied.Isn't it? Since it's a production box, i'm afraid on... (2 Replies)
Discussion started by: Tamil
2 Replies

4. UNIX for Dummies Questions & Answers

Named Pipe contents to a file

I want to copy the contents of a named pipe to a file. I have tried using: cat pipe.p >> transcript.log but I have been unsuccessful, any ideas? (4 Replies)
Discussion started by: carl_vieyra
4 Replies

5. UNIX for Dummies Questions & Answers

Filtering mail into a named pipe

Hello, On my machine, all mail is stored in my /var/spool/mail. IS there a way to direct all mail that goes there into a namep pipe? Thank you, Dado (4 Replies)
Discussion started by: dadoprso
4 Replies

6. Shell Programming and Scripting

pipe to file named with date

I would like to pipe (redirect ? - what is the right term?) the output of my script to a file named with the current date. If I run this at a command prompt: date +'%Y%m%d" ...it returns "20110429" OK, that's good... so I try: ./script.sh > "'date +%Y%m%d'.csv" I get a file... (1 Reply)
Discussion started by: landog
1 Replies

7. Shell Programming and Scripting

Using Named pipe in shell script

Hi, I want to use a Named pipe to get input from a growing file for further processing. When I prototype this scenario using a while loop, the data is not written to the named pipe. This the script I use to get data into the Named pipe: #!/bin/ksh mkfifo pipe while (( n <= 10 )) do echo... (2 Replies)
Discussion started by: sudvishw
2 Replies

8. Shell Programming and Scripting

Named pipe performance

Hi, I am getting data into a Named pipe. Does Named pipe have any size restriction; I know it does not have any storage and it just passes on the data to the next process. I want to know, if there will be a difference in the Named pipe performance if the data input is more. (I am using DB2... (1 Reply)
Discussion started by: sudvishw
1 Replies

9. UNIX for Dummies Questions & Answers

Named pipe hanging?

Ok, I can't seem to figure this out or find anything on the web about this. I'm on Sun Solaris, UNIX. I have the following test script: #!/bin/ksh touch test.file LOG=./tmp.log rm -f ${LOG} PIPE=./tmp.pipe mkfifo ${PIPE} trap "rm -f ${PIPE}" EXIT tee -a ${LOG} < ${PIPE} & ... (17 Replies)
Discussion started by: Ditto
17 Replies

10. Shell Programming and Scripting

How to test named pipe file?

Hi ALL, How can I test a given file name exists and if it is a named pipe file in shell script ? Thanks............ (2 Replies)
Discussion started by: mycode.in
2 Replies
FIFO(4) 						     Linux Programmer's Manual							   FIFO(4)

NAME
fifo - first-in first-out special file, named pipe DESCRIPTION
A FIFO special file (a named pipe) is similar to a pipe, except that it is accessed as part of the file system. It can be opened by multi- ple processes for reading or writing. When processes are exchanging data via the FIFO, the kernel passes all data internally without writ- ing it to the file system. Thus, the FIFO special file has no contents on the file system, the file system entry merely serves as a refer- ence point so that processes can access the pipe using a name in the file system. The kernel maintains exactly one pipe object for each FIFO special file that is opened by at least one process. The FIFO must be opened on both ends (reading and writing) before data can be passed. Normally, opening the FIFO blocks until the other end is opened also. A process can open a FIFO in non-blocking mode. In this case, opening for read only will succeed even if noone has opened on the write side yet; opening for write only will fail with ENXIO (no such device or address) unless the other end has already been opened. Under Linux, opening a FIFO for read and write will succeed both in blocking and non-blocking mode. POSIX leaves this behaviour undefined. This can be used to open a FIFO for writing while there are no readers available. A process that uses both ends of the connection in order to communicate with itself should be very careful to avoid deadlocks. NOTES
When a process tries to write to a FIFO that is not opened for read on the other side, the process is sent a SIGPIPE signal. FIFO special files can be created by mkfifo(3), and are specially indicated in ls -l. SEE ALSO
mkfifo(3), mkfifo(1), pipe(2), socketpair(2), open(2), signal(2), sigaction(2) Linux Man Page 1999-06-20 FIFO(4)
All times are GMT -4. The time now is 05:51 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy