Sponsored Content
Top Forums UNIX for Beginners Questions & Answers UNIX Pipe -Exit when there are no bytes to read Post 303015515 by mr_manii on Friday 6th of April 2018 06:40:45 AM
Old 04-06-2018
Display UNIX Pipe -Exit when there are no bytes to read

Hi All,

I'm creating a program which reads millions of bytes from the PIPE and do some processing. As the data is more, the idea is to read the pipe parallely.

Sun Solaris 8
See the code below:
Code:
#!/bin/sh
MAXTHREAD=30
awk '{print $1}' metadata.csv > nvpipe &
while [ $c -le $MAXTHREAD ]
do
   ${BIN}/parallel_wot.sh &
   PID=$!
   sleep 3
   c=`expr $c + 1`
STRING=$STRING","$PID
done

parallel_wot.sh
Code:
cd $PIPEDIR/
while IFS=',' read DIR1 DIR2
do
 echo starting at `date`
  ${COMPUSETBIN}/prg1.sh prg2.sh $DIR1 $DIR2
  echo ending at `date`
  echo ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
done < nvpipe

Now the problem is, as the pipe is read in parallel, once the pipe is emptied all other processes are waiting to read from the pipe except 1. The questions I now have is , how to exit from reading the pipe if there are no records.

Last edited by Scrutinizer; 04-06-2018 at 08:11 AM.. Reason: Extra code tags
 

10 More Discussions You Might Find Interesting

1. Programming

client socket read returns 0 bytes

Hi I have apeculiar problem with sockets. I have a shared object for my client program. when I send a request to the server, it is suppose to process and sends back the result string to the client. For the first request, it is working fine i.e. client sends the req. and gets the... (1 Reply)
Discussion started by: axes
1 Replies

2. UNIX for Dummies Questions & Answers

exit status of command in a pipe line

Hi, I am trying to test the exit status of the cleartool lsvtree statement below, but it doesn't seem to be working due to the tail pipe, which it is testing instead. Is there a way around this without adding a tonne of new code? cleartool lsvtree $testlocation/$exe_name | tail -15 ... (10 Replies)
Discussion started by: topcat8
10 Replies

3. Shell Programming and Scripting

How to get exit code in a pipe-lined command?

I have a question about how to get the exit code of the first command when it appears in a pipe-lined command. For example, I have the following script: grep abc dddd | tee -a log if ] then echo "ERROR!" fi In the above script, ] is supposed to test the exit code of "grep abc... (3 Replies)
Discussion started by: pankai
3 Replies

4. Shell Programming and Scripting

Remove first N bytes and last N bytes from a binary file on AIX.

Hi all, Does anybody know or guide me on how to remove the first N bytes and the last N bytes from a binary file? Is there any AWK or SED or any command that I can use to achieve this? Your help is greatly appreciated!! Best Regards, Naveen. (1 Reply)
Discussion started by: naveendronavall
1 Replies

5. Programming

Copying 1024 bytes data in 3-bytes chunk

Hi, If I want to copy a 1024 byte data stream in to the target location in 3-bytes chunk, I guess I can use the following script. dd bs=1024 count=3 if=/src of=/dest But, I would like to know, how to do it via a C program. I have tried this with memcpy(), that did not help. (3 Replies)
Discussion started by: royalibrahim
3 Replies

6. Shell Programming and Scripting

Check the exit status in a pipe call

Guys, I have a problem :confused: and I need some help: I've to process many huge zip files. I'd code an application that receive the data from a pipe, so I can simple unzip the data and send it (via pipe) to my app. Something like that: gzip -dc <file> | app The problem is: How can I... (7 Replies)
Discussion started by: Rkolbe
7 Replies

7. UNIX for Dummies Questions & Answers

X bytes of 0, Y bytes of random data, Z bytes of 5, T bytes of 1. ??

Hello guys. I really hope someone will help me with this one.. So, I have to write this script who: - creates a file home/student/vmdisk of 10 mb - formats that file to ext3 - mounts that partition to /mnt/partition - creates a file /mnt/partition/data. In this file, there will... (1 Reply)
Discussion started by: razolo13
1 Replies

8. Shell Programming and Scripting

Shell script - entered input(1-40 bytes) needs to be converted exactly 40 bytes

hello, suppose, entered input is of 1-40 bytes, i need it to be converted to 40 bytes exactly. example: if i have entered my name anywhere between 1-40 i want it to be stored with 40 bytes exactly. enter your name: donald duck (this is of 11 bytes) expected is as below - display 11... (3 Replies)
Discussion started by: shravan.300
3 Replies

9. Shell Programming and Scripting

Exit to while read

Hi expert, How do i exit to while read, below is the script. I need to exit after execute echo or command. or any scripts that can search two patterns and if they found any patterns execute the command and exit. Thanks a lot.. tail -fn0 /tmp/test.log | \ while read line ; do ... (12 Replies)
Discussion started by: lxdorney
12 Replies

10. Shell Programming and Scripting

UNIX fifo concurrent read from a named pipe

I have created a fifo named pipe in solaris, which writes the content of a file, line by line, into pipe as below: $ mkfifo namepipe $ cat books.txt "how to write unix code" "how to write oracle code" $ cat books.txt >> namepipe & I have a readpipe.sh script which reads the named... (2 Replies)
Discussion started by: naveen mani
2 Replies
PIPE(2) 							System Calls Manual							   PIPE(2)

NAME
pipe - create an interprocess communication channel SYNOPSIS
#include <unistd.h> int pipe(int fildes[2]) DESCRIPTION
The pipe system call creates an I/O mechanism called a pipe. The file descriptors returned can be used in read and write operations. When the pipe is written using the descriptor fildes[1] up to PIPE_MAX bytes of data are buffered before the writing process is suspended. A read using the descriptor fildes[0] will pick up the data. PIPE_MAX equals 7168 under Minix, but note that most systems use 4096. It is assumed that after the pipe has been set up, two (or more) cooperating processes (created by subsequent fork calls) will pass data through the pipe with read and write calls. The shell has a syntax to set up a linear array of processes connected by pipes. Read calls on an empty pipe (no buffered data) with only one end (all write file descriptors closed) returns an end-of-file. The signal SIGPIPE is generated if a write on a pipe with only one end is attempted. RETURN VALUE
The function value zero is returned if the pipe was created; -1 if an error occurred. ERRORS
The pipe call will fail if: [EMFILE] Too many descriptors are active. [ENFILE] The system file table is full. [ENOSPC] The pipe file system (usually the root file system) has no free inodes. [EFAULT] The fildes buffer is in an invalid area of the process's address space. SEE ALSO
sh(1), read(2), write(2), fork(2). NOTES
Writes may return ENOSPC errors if no pipe data can be buffered, because the pipe file system is full. BUGS
Should more than PIPE_MAX bytes be necessary in any pipe among a loop of processes, deadlock will occur. 4th Berkeley Distribution August 26, 1985 PIPE(2)
All times are GMT -4. The time now is 03:14 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy