UNIX fifo concurrent read from a named pipe Post: 303013524

Sponsored Content

Top Forums Shell Programming and Scripting UNIX fifo concurrent read from a named pipe Post 303013524 by Chubler_XL on Thursday 22nd of February 2018 04:41:46 PM

02-22-2018

Moderator

Probably best to use a mutex/semaphore to ensure only 1 process only reads from the pipe at a time.

I don't know what OS you are using so not sure if lockfile(1) is available to you.

This solution uses a lock directory as mkdir is supposed to be atomic:

readpipe.sh

Code:

lockdir=lock.dir

get_book() {
   local DONE=0
   while ! mkdir $lockdir 2> /dev/null
   do
      sleep 0.02
   done

   IFS="," read -u7 var || DONE=1

   rmdir $lockdir
   return $DONE
}

exec 7< namepipe
while get_book
do
   echo "$var" >> log$$.txt
done
exec 7<&-

Here is my test:

Code:

$ cat naveen 
mkfifo namepipe
./readpipe.sh &
./readpipe.sh &
exec 3<> namepipe
cat books.txt >&3
# Give time for readers to get established
sleep 1
# Close the write end
exec 3>&-
# wait for readers to complete
wait

$ ./naveen 

$ wc -l books.txt log*
 200 books.txt
 106 log2838.txt
  94 log2839.txt
 400 total

$ diff <(sort books.txt) <(cat log* | sort)

Update: below is an example using flock(1):

Code:

lockfile=lock.file

get_book() {
  var=$( (
      if flock -w .007 -n 200
      then
        IFS="," read -u7 var || exit 1
        echo "$var"
      fi) 200>$lockfile )
}

exec 7< namepipe

while get_book
do
  if [ -n "$var" ]
  then
          # Do any required work with the book record here...
          echo "$var" >> log$$.txt
  fi
done
exec 7<&-

Last edited by Chubler_XL; 02-25-2018 at 11:56 PM.. Reason: Added sleep to writer before close - avoid 1 process hogging the fifo

Chubler_XL

View Public Profile for Chubler_XL

Find all posts by Chubler_XL

10 More Discussions You Might Find Interesting

1. Programming

Pipe & fifo....

Could someone Help me with this code please? #include <stdio.h> #include <unistd.h> #include <sys/types.h> #include <sys/stat.h> #include <string.h> #include <fcntl.h> #define SIZE_B 256 /*buffer's size */ #define NUM_ARG 20 /* max number of args for any command */ int...

2. UNIX for Advanced & Expert Users

PIPE and FIFO buffer size

Hello! How I can increase (or decrease) the predefined pipe buffer size? Thanks!

3. Shell Programming and Scripting

FIFO named pipes

Hi...Can anyone please guide me on FIFO Pipes in UNIX.I have lerant things like creating fifo pipes,using them for reads and writes etc.I want to know what is the maximum amount of memory that such a pipe may have? Also can anyone guide me on where to get info on this topic from?

4. UNIX for Dummies Questions & Answers

Named PIPE

Gurus, I've a File Transaction Server, which communicates with other servers and performs some processing.It uses many Named PIPE's. By mistake i copied a named PIPE into a text file. I heard that PIPE files shouldn't be copied.Isn't it? Since it's a production box, i'm afraid on...

5. Shell Programming and Scripting

Reading from blocking fifo pipe in shell script

Hi!! I have a problem reading from a fifo pipe in shell script. The idea is simple, I have a C program with two pipe files: An input pipe I use to send commands in shell script to the C program (echo "command" > input.pipe) An output pipe that I read the result of the command also in...

6. UNIX for Dummies Questions & Answers

fifo or named pipe working?

Can someone explain to me the working of fifo() system call using simple C programs so that I can implement them in the UNIX environement?

7. UNIX for Advanced & Expert Users

Why not SIGPIPE for readers of pipe/FIFO?

Hi This is a exercise question from Unix network programming vol2. Why the SIGPIPE signal is generated only for writers when readers disappear. why not it is generated for readers when writer disappears. I guess, if the writer didn't get any response like the reader gets EOF, it will...

8. Programming

Pipe & fifo size limit

Hi guys. 1. how much is the size of pipe?(i mean the buffer size) 2. is this size different in various UNIX derivations? 3. what happens if we write to a full pipe? does it block until get some free space(the other side receive data) or returns an error? 3. FIFO s are physical files on the...

9. Shell Programming and Scripting

awk reading from named pipe (fifo)

I'm trying to read a fifo using awk and comming across some problems. I'm writing to the fifo from multiple processes invoked by GNU Parallel: mkfifo my_fifo awk '{ a = a + $2 } END { for (i in a) print i, a }' my_fifo | sort -nk1 > sorted_output grep -v '^@' massive_file | parallel...

10. UNIX for Beginners Questions & Answers

UNIX Pipe -Exit when there are no bytes to read

Hi All, I'm creating a program which reads millions of bytes from the PIPE and do some processing. As the data is more, the idea is to read the pipe parallely. Sun Solaris 8 See the code below: #!/bin/sh MAXTHREAD=30 awk '{print $1}' metadata.csv > nvpipe & while do ...

LEARN ABOUT DEBIAN

x2sys_merge

X2SYS_MERGE(1gmt)					       Generic Mapping Tools						 X2SYS_MERGE(1gmt)

NAME

       x2sys_merge - Merge an updated COEs tables

SYNOPSIS

       x2sys_merge -Amain_COElist.d -Mnew_COElist.d

DESCRIPTION

       x2sys_merge  will read two crossovers data base and output the contents of the main one updated with the COEs in the second one. The second
       file should only contain updated COEs relatively to the first one.  That is, it MUST NOT contain any new  two  tracks  intersections  (This
       point  is  NOT checked in the code). This program is useful when, for any good reason like file editing NAV correction or whatever, one had
       to recompute only the COEs between the edited files and the rest of the database.

       -A     Specify the file main_COElist.d with the main crossover error data base.

       -M     Specify the file new_COElist.d with the newly computed crossover error data base.

OPTIONS

       No space between the option flag and the associated arguments.

       EXAMPLES
	      To update the main COE_data.txt with the new COEs estimations saved in the smaller COE_fresh.txt, try

	      x2sys_merge -ACOE_data.txt -MCOE_fresh.txt > COE_updated.txt

SEE ALSO

       x2sys_binlist(1), x2sys_cross(1), x2sys_datalist(1), x2sys_get(1), x2sys_init(1), x2sys_list(1), x2sys_put(1), x2sys_report(1)

GMT 4.5.7							    15 Jul 2011 						 X2SYS_MERGE(1gmt)

10 More Discussions You Might Find Interesting

1. Programming

Pipe & fifo....

Discussion started by: M3xican

2. UNIX for Advanced & Expert Users

PIPE and FIFO buffer size

Discussion started by: Jus

3. Shell Programming and Scripting

FIFO named pipes

Discussion started by: tej.buch

4. UNIX for Dummies Questions & Answers

Named PIPE

Discussion started by: Tamil

5. Shell Programming and Scripting

Reading from blocking fifo pipe in shell script

Discussion started by: victorin

6. UNIX for Dummies Questions & Answers

fifo or named pipe working?

Discussion started by: lvkchaitanya

7. UNIX for Advanced & Expert Users

Why not SIGPIPE for readers of pipe/FIFO?

Discussion started by: kumaran_5555

8. Programming

Pipe & fifo size limit

Discussion started by: majid.merkava

9. Shell Programming and Scripting

awk reading from named pipe (fifo)

Discussion started by: nathanhaigh

10. UNIX for Beginners Questions & Answers

UNIX Pipe -Exit when there are no bytes to read

Discussion started by: mr_manii

LEARN ABOUT DEBIAN

x2sys_merge