UNIX Pipe -Exit when there are no bytes to read Post: 303015542

Sponsored Content

Top Forums UNIX for Beginners Questions & Answers UNIX Pipe -Exit when there are no bytes to read Post 303015542 by rbatte1 on Friday 6th of April 2018 12:54:05 PM

04-06-2018

Moderator

I will assume that you already made a pipe file with something like mknod /$PIPEDIR/nvpipe p however I would be concerned that you have no idea which thread is reading the (now) input at any time.

You might find that the first reading process locks up the pipe, I'm not sure. It might be more sensible to ignore the pipe altogether and do something more like this:-

Split metadata.csv into 30 roughly equal files
Fire off 30 processes that read a separate input file each to do whatever processing you need

If you can get the number of lines in your file, you should be able to get the require line-count like this:-

Code:

#!/bin/bash
threads=30                                            # How many threads you want to work with

all_lines=$(wc -l < metadata.csv)                     # Count all the lines in your full input file
req_lines=$(printf $all_lines / $threads +1 | bc      # Get the lines required in each split file
                                                      # Rules of BIDMAS apply so the 1 is added after the divide

split -ld $req_lines metadata.csv metadata.           # Note the trailing dot. This will generate $threads files
                                                      # of up to $req_lines each in the format metadata.nn
                                                      # so up to 100 threads if you choose
for split_file in metadata.??
do
   ( while IFS=',' read DIR1 DIR2
   do
     printf "Start at $(date)\n"
     ${COMPUSETBIN}/prg1.sh prg2.sh $DIR1 $DIR2
     printf "Ending at $(date)\n"
     printf "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n"
   done < $split_file
done ) &

wait               #  All threads must complete before this script will exit

It's untested, but does it get you started a bit better?

Probably there are better ways to do this in a single awk. What does your prg1.sh & prg2.sh actually do?

Kind regards,
Robin

rbatte1

View Public Profile for rbatte1

Visit rbatte1's homepage!

Find all posts by rbatte1

10 More Discussions You Might Find Interesting

1. Programming

client socket read returns 0 bytes

Hi I have apeculiar problem with sockets. I have a shared object for my client program. when I send a request to the server, it is suppose to process and sends back the result string to the client. For the first request, it is working fine i.e. client sends the req. and gets the...

2. UNIX for Dummies Questions & Answers

exit status of command in a pipe line

Hi, I am trying to test the exit status of the cleartool lsvtree statement below, but it doesn't seem to be working due to the tail pipe, which it is testing instead. Is there a way around this without adding a tonne of new code? cleartool lsvtree $testlocation/$exe_name | tail -15 ...

3. Shell Programming and Scripting

How to get exit code in a pipe-lined command?

I have a question about how to get the exit code of the first command when it appears in a pipe-lined command. For example, I have the following script: grep abc dddd | tee -a log if ] then echo "ERROR!" fi In the above script, ] is supposed to test the exit code of "grep abc...

4. Shell Programming and Scripting

Remove first N bytes and last N bytes from a binary file on AIX.

Hi all, Does anybody know or guide me on how to remove the first N bytes and the last N bytes from a binary file? Is there any AWK or SED or any command that I can use to achieve this? Your help is greatly appreciated!! Best Regards, Naveen.

5. Programming

Copying 1024 bytes data in 3-bytes chunk

Hi, If I want to copy a 1024 byte data stream in to the target location in 3-bytes chunk, I guess I can use the following script. dd bs=1024 count=3 if=/src of=/dest But, I would like to know, how to do it via a C program. I have tried this with memcpy(), that did not help.

6. Shell Programming and Scripting

Check the exit status in a pipe call

Guys, I have a problem :confused: and I need some help: I've to process many huge zip files. I'd code an application that receive the data from a pipe, so I can simple unzip the data and send it (via pipe) to my app. Something like that: gzip -dc <file> | app The problem is: How can I...

7. UNIX for Dummies Questions & Answers

X bytes of 0, Y bytes of random data, Z bytes of 5, T bytes of 1. ??

Hello guys. I really hope someone will help me with this one.. So, I have to write this script who: - creates a file home/student/vmdisk of 10 mb - formats that file to ext3 - mounts that partition to /mnt/partition - creates a file /mnt/partition/data. In this file, there will...

8. Shell Programming and Scripting

Shell script - entered input(1-40 bytes) needs to be converted exactly 40 bytes

hello, suppose, entered input is of 1-40 bytes, i need it to be converted to 40 bytes exactly. example: if i have entered my name anywhere between 1-40 i want it to be stored with 40 bytes exactly. enter your name: donald duck (this is of 11 bytes) expected is as below - display 11...

9. Shell Programming and Scripting

Exit to while read

Hi expert, How do i exit to while read, below is the script. I need to exit after execute echo or command. or any scripts that can search two patterns and if they found any patterns execute the command and exit. Thanks a lot.. tail -fn0 /tmp/test.log | \ while read line ; do ...

10. Shell Programming and Scripting

UNIX fifo concurrent read from a named pipe

I have created a fifo named pipe in solaris, which writes the content of a file, line by line, into pipe as below: $ mkfifo namepipe $ cat books.txt "how to write unix code" "how to write oracle code" $ cat books.txt >> namepipe & I have a readpipe.sh script which reads the named...

LEARN ABOUT PLAN9

pipe

PIPE(3) 						     Library Functions Manual							   PIPE(3)

NAME

       pipe - two-way interprocess communication

SYNOPSIS

       bind #| dir

       dir/data
       dir/ctl
       dir/data1
       dir/ctl1

DESCRIPTION

       An  attach(5) of this device allocates two new streams joined at the device end.  X/data and x/ctl are the data and control channels of one
       stream and x/data1 and x/ctl1 are the data and control channels of the other stream.

       Data written to one channel becomes available for reading at the other.	Write boundaries are preserved: each read terminates when the read
       buffer is full or after reading the last byte of a write, whichever comes first.

       Written data is buffered in kernel stream blocks.  The writer will block once the stream is full, typically after 32768 bytes or 16 writes.
       The writer will resume once the stream is less than half full.

       If there are multiple writers, each write is guaranteed to be available in a contiguous piece at the other end of the pipe.  If	there  are
       multiple readers, each read will return data from only one write.

       The  pipe(2)  system call performs an attach of this device and returns file descriptors to the new pipe's data and data1 files.  The files
       are open with mode ORDWR.

SEE ALSO

       pipe(2)

SOURCE

       /sys/src/9/port/devpipe.c

																	   PIPE(3)

10 More Discussions You Might Find Interesting

1. Programming

client socket read returns 0 bytes

Discussion started by: axes

2. UNIX for Dummies Questions & Answers

exit status of command in a pipe line

Discussion started by: topcat8

3. Shell Programming and Scripting

How to get exit code in a pipe-lined command?

Discussion started by: pankai

4. Shell Programming and Scripting

Remove first N bytes and last N bytes from a binary file on AIX.

Discussion started by: naveendronavall