01-09-2007
named pipe with persistent buffer
Hey folks,
i need communicate between 2 processes in a reliable manner. The information delivery has to be guarenteed. I thought about proc 2 sending a signal to proc 1 when information has being written to disc and wirte() has been verified (sync/flush). The IPC method for the data is named pipes.
When proc1 simply writes the data to the named pipe, and the system crashes before proc2 could read it and persistently save it to disc the information is lost.
So my solution to this is:
proc1:
has opened a phys. file, tmpfile; and a named pipe
1. writes data to tmpfile on disc, verify write()
2. writes the data to pipe
3. waits for signal from proc 2
proc2:
has select or poll to the pipe
1. writes data to disc on pipe change, verifiy write()
2. sends signal to proc 1, meaning data is persistently saved.
prioc1:
4. emtpys tmpfile
what do you think about this idea?
Is the pipe necessary?
Could i as well read the physical tmpfile from proc2 instead of double write from proc1 to phys. file AND pipe. Performance?
If you have suggestions introducing completly different concepts, please consider that proc1 is an already written program and outputs the data to a physical file. I cant really change proc1, adding a sighandler and a sigsuspend along a tmpfile creation/ emptiyng and a double write are small changes, but already nearly to big for this process (testing). Regulatory reasons (risk).
My question is also, isn't there anything like a named pipe with persistent buffer on disc, and with fast buffer im mem for the usual work. the persistent buffer is copied to membuffer on open(). So data which was written to pipe but not read before crash is not lost (if the crash was after succesfull write to disc pipe internal). The write() call to this kind of pipe should return when data is assuredly written to disc, so the user program can assume once returned from the write() call, data is now delivered, guaranteed.
Sorry for the long story, but i hope you can help me along with this conceptual problem!
Thanks in advance!
Heck
10 More Discussions You Might Find Interesting
1. UNIX for Advanced & Expert Users
Hi All,
I am facing a vague issue while trying to make two process talk to each
other using named pipe.
read process
=========
The process which reads, basically creates FIFO using
mkfifo - ret_val = mkfifo(HALF_DUPLEX, 0666);) func.
It then opens the pipe using open func - fd =... (1 Reply)
Discussion started by: sharanbr
1 Replies
2. Programming
Hi All,
I am facing a vague issue while trying to make two process talk to each
other using named pipe.
read process
=========
The process which reads, basically creates FIFO using
mkfifo - ret_val = mkfifo(HALF_DUPLEX, 0666) func.
It then opens the pipe using open func - fd = open... (2 Replies)
Discussion started by: sharanbr
2 Replies
3. UNIX for Dummies Questions & Answers
Gurus,
I've a File Transaction Server, which communicates with other servers and performs some processing.It uses many Named PIPE's.
By mistake i copied a named PIPE into a text file.
I heard that PIPE files shouldn't be copied.Isn't it?
Since it's a production box, i'm afraid on... (2 Replies)
Discussion started by: Tamil
2 Replies
4. UNIX for Advanced & Expert Users
I did
cat < myFile >> myPipe
I was hoping that if I did ls -l, myPipe would now be holding the contents of myFile, and would be the same size. But it was 0.
Also strange was that when I did the command above, cat did not return control back to the shell. Why?
thanks (4 Replies)
Discussion started by: JamesByars
4 Replies
5. UNIX for Dummies Questions & Answers
Hello,
On my machine, all mail is stored in my /var/spool/mail.
IS there a way to direct all mail that goes there into a namep pipe?
Thank you,
Dado (4 Replies)
Discussion started by: dadoprso
4 Replies
6. Shell Programming and Scripting
I would like to pipe (redirect ? - what is the right term?) the output of my script to a file named with the current date.
If I run this at a command prompt:
date +'%Y%m%d"
...it returns "20110429"
OK, that's good... so I try:
./script.sh > "'date +%Y%m%d'.csv"
I get a file... (1 Reply)
Discussion started by: landog
1 Replies
7. Shell Programming and Scripting
Hi,
I want to use a Named pipe to get input from a growing file for further processing. When I prototype this scenario using a while loop, the data is not written to the named pipe.
This the script I use to get data into the Named pipe:
#!/bin/ksh
mkfifo pipe
while (( n <= 10 ))
do
echo... (2 Replies)
Discussion started by: sudvishw
2 Replies
8. Shell Programming and Scripting
Hi,
I am getting data into a Named pipe. Does Named pipe have any size restriction; I know it does not have any storage and it just passes on the data to the next process.
I want to know, if there will be a difference in the Named pipe performance if the data input is more. (I am using DB2... (1 Reply)
Discussion started by: sudvishw
1 Replies
9. Programming
Hi All !
I try to collect importent events from syslog and in my
syslog conf, there is something like this:
*.* |/logs/ipes/SLpipe1
I have a program, which opens this pipe and reads the messages from it.
But how this pipe works ? Where can I probably read something about the details,... (3 Replies)
Discussion started by: mabra
3 Replies
10. UNIX for Dummies Questions & Answers
Ok, I can't seem to figure this out or find anything on the web about this.
I'm on Sun Solaris, UNIX.
I have the following test script:
#!/bin/ksh
touch test.file
LOG=./tmp.log
rm -f ${LOG}
PIPE=./tmp.pipe
mkfifo ${PIPE}
trap "rm -f ${PIPE}" EXIT
tee -a ${LOG} < ${PIPE} &
... (17 Replies)
Discussion started by: Ditto
17 Replies
FIFO(4) Linux Programmer's Manual FIFO(4)
NAME
fifo - first-in first-out special file, named pipe
DESCRIPTION
A FIFO special file (a named pipe) is similar to a pipe, except that it is accessed as part of the file system. It can be opened by multi-
ple processes for reading or writing. When processes are exchanging data via the FIFO, the kernel passes all data internally without writ-
ing it to the file system. Thus, the FIFO special file has no contents on the file system, the file system entry merely serves as a refer-
ence point so that processes can access the pipe using a name in the file system.
The kernel maintains exactly one pipe object for each FIFO special file that is opened by at least one process. The FIFO must be opened on
both ends (reading and writing) before data can be passed. Normally, opening the FIFO blocks until the other end is opened also.
A process can open a FIFO in non-blocking mode. In this case, opening for read only will succeed even if noone has opened on the write side
yet; opening for write only will fail with ENXIO (no such device or address) unless the other end has already been opened.
Under Linux, opening a FIFO for read and write will succeed both in blocking and non-blocking mode. POSIX leaves this behaviour undefined.
This can be used to open a FIFO for writing while there are no readers available. A process that uses both ends of the connection in order
to communicate with itself should be very careful to avoid deadlocks.
NOTES
When a process tries to write to a FIFO that is not opened for read on the other side, the process is sent a SIGPIPE signal.
FIFO special files can be created by mkfifo(3), and are specially indicated in ls -l.
SEE ALSO
mkfifo(3), mkfifo(1), pipe(2), socketpair(2), open(2), signal(2), sigaction(2)
Linux Man Page 1999-06-20 FIFO(4)