Background processes in a pipeline don't run asynchronously?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Background processes in a pipeline don't run asynchronously?
# 1  
Old 09-29-2011
[Bash 4.2] Background processes in a pipeline don't run asynchronously? (very strange)

I'm wondering about what causes this behavior. Mainly, given the first case below, why the second changes the behavior of ''&'' to execute sequentially merely because the command group containing calls to f() is part of a pipeline. The 4th example bizarrely once again works as the first, even with the piped output. The behavior of the 5th example I suspect has a different cause, but is interesting nevertheless.

Each f() reports it's identity ($1) on stderr each time through it's read loop so we know whether they're being executed concurrently, then prints the totals after it's stdin is exhausted. For each example, each f shares a stream of 20 zeros.

Bash 4.2:
Code:
#!/usr/bin/env bash

set +m -f
shopt -s lastpipe

f() {
    local -i x y                   # Store each character in $x. $y is just a counter.
    while read -rN1 "x[y++]"; do
        printf '%d ' "${1}" >&2    # keep track of which job this is.
    done
    printf "${#x[@]} "             # Print the total number of reads by each job.
}

g() {
    f 1 <${1} &
    f 2 <${1}
}

declare -i ex=1

echo "example $((ex++)):"
# This works as I expect, f is backgrounded and two readers of one pipe each get about half the input:

read -ra x < <({ f 1 & f 2; } < <(printf '%.s0' {0..20}))
printf '%b\n' "\n${x[@]}\n"

echo "example $((ex++)):"
# In this equivalent version, f is not backgrounded, and one reader consumes all of the input, I can only assume for some reason because it's part of a pipeline:

{ f 1 & f 2; } < <(printf "%.s0" {0..20}) | {
    read -ra x
    printf '%b\n' "\n${x[@]}\n"
}

echo "example $((ex++)):"
# Same as above. Unsafe wordsplitting for brevity.

printf '%s\n' $'\n'$(printf '%.s0' {0..20} | { f 1 & f 2; })

printf '\n%s\n' "example $((ex++)):"
# Identical to the above two examples, except rather than using the command group's stdin, save the FD to $x and redirect individually. Now it behaves as the first example again. (WTF???)

{ f 1 <&${x} & f 2 <&${x}; } {x}< <(printf "%.s0" {0..20}) | {
    read -ra x
    printf '%b\n' "\n${x[@]}\n"
}

echo "example $((ex++)):"
# In this version, the name of the pipe is passed to g and then individually redirected to f, but it doesn't work. No matter how much data it gets, the pipe is closed before the second call to f sees it:

read -ra x < <(g <(printf "%.s0" {0..20}))
printf '%b\n' "\n${x[@]}\n"
wait

# vim: set fenc=utf-8 ff=unix ts=4 sts=4 sw=4 ft=sh nowrap et:

output:
Code:
 $ ./pipefork 
example 1:
2 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 2 1 2 2 1 
13
10

example 2:
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 
1
22

example 3:
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
1
22

example 4:
2 1 2 1 2 1 2 1 2 1 1 2 1 1 2 1 1 2 1 2 1 
10
13

example 5:
1 1 ./pipefork: line 16: /dev/fd/63: No such file or directory
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 
22


Last edited by ormaaj; 09-30-2011 at 03:08 AM.. Reason: Add more examples. Better comments.
# 2  
Old 10-03-2011
I think , in g() function , when ${1} fd goes to background ,f() function process at once,after then pipe is missing for that created 63fd..
therefore `/dev/fd/63` fd can not found by the g function.so g() function can not call f() function via same fd at the same time..
maybe you can try like this Smilie
Code:
..............
g() {
exec 4<${1}
f 1 <${1} &
f 2 <&4
}
...............
.........

regards
ygemici
# 3  
Old 10-06-2011
Quote:
Originally Posted by ygemici
I think , in g() function , when ${1} fd goes to background ,f() function process at once,after then pipe is missing for that created 63fd..
therefore `/dev/fd/63` fd can not found by the g function.so g() function can not call f() function via same fd at the same time..
maybe you can try like this Smilie
Code:
..............
g() {
exec 4<${1}
f 1 <${1} &
f 2 <&4
}
...............
.........

regards
ygemici
You wouldn't want exec there. Better would be to put the redirect after the function definition. Yours is essentially equivalent to the example before it.

Anyway my question was mostly answered on the Bash mailing list.

lists.gnu.org/archive/html/bug-bash/2011-10/msg00019.html

POSIX says that background processes are always supposed to implicitly have their stdin redirected to /dev/null. Bash for some reason only sometimes follows this, and the pipe somehow affects whether that happens. The background processes shouldn't ever be getting input unless stdin is explicitly redirected. The 5th example from an updated version of the above shows another strange behavior where stdin of a list is determined by any redirects in the list.

gist.github.com/1255778
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

List all background processes

How do I list the process in a Unix based system which are running in background? The following are options that I'm aware of, but they may not be appropiate. a. using ps -ef , and getting records of processes for which STATUS='S'(uninterruptible sleep) b. using jobs -l, and filtering... (5 Replies)
Discussion started by: kumarjt
5 Replies

2. Shell Programming and Scripting

Need help on background processes

Hi, I have a schell script parent.ksh from which I am calling three background processes a.ksh,b.ksh and c.ksh. Once these three processes completes the next step in parent.ksh should execute. How to achieve this? Please help me.... Thanks... (1 Reply)
Discussion started by: ravinunna
1 Replies

3. Shell Programming and Scripting

Background Processes

Ok guys so I have my first dummy shell almost done except for one tiny part: I do not know how to run a process in the background, from the code! I already know how to do that in a normal shell: $ program & However, no clue when it comes to how to program that thing. :eek: A very... (2 Replies)
Discussion started by: Across
2 Replies

4. Solaris

About running processes in background

Hi, I need to establish a procedure that will start an application in background each time my remote Solaris server is (re)started. This would be a kind of daemon. I am no sysadmin expert, so I am looking for pointers. How should I proceed? What are the main steps? Thanks, JVerstry (9 Replies)
Discussion started by: JVerstry
9 Replies

5. UNIX for Dummies Questions & Answers

Disadvantage of background processes

Hi, Inorder to improve the performance, I am trying to execute my command as a background process.. For eg: To zip large numbers of files present in a directory instead of using a single process, i do follow the below method: gunzip -c > / &... (3 Replies)
Discussion started by: unni.raj
3 Replies

6. SuSE

oracle background processes

I have installed oracle 10g on suse sles9. I do not see oracle background processes. ps -ef|grep ora_ gives me environment variables junk. ps -ef|grep smon does not show anything however database is up and running. Any idea how to tweak that? (1 Reply)
Discussion started by: vijayasawant
1 Replies

7. Linux

Question about background processes

Hi! First of all, let me warn you I'm quite new to the world of LINUX and Operating Systems understanding, so that's why I pose these newbie and stupid qustions... Anyway, I'm trying to build my own simple shell in C and I'm getting some problems in implementing the background process ('&')... (10 Replies)
Discussion started by: neimaD
10 Replies

8. Programming

Background processes in a dummy shell...

Hey guys, I am writing a very simple dummy shell in C++ and I am having trouble getting a process to run in the background. First of all, the shell has to recognize when I input a "&" at the end of the command, then it has to stick it in the background of the shell. I understand that if I want... (6 Replies)
Discussion started by: icer
6 Replies

9. Shell Programming and Scripting

Running two processes in background

hi there, here's what i need in my korn-shell: ... begin korn-shell script ... nohup process_A.ksh ; nohup process_B.ksh & ... "other stuff" ... end lorn-shell script in plain english i want process A and process B to run in the background so that the script can continue doing... (6 Replies)
Discussion started by: jacob_gs
6 Replies

10. UNIX for Advanced & Expert Users

Background processes

How do you capture the return code from a background process? I am dumping data to a fifo and then processing it in a c program. I need to know that the sql finished successfully to ensure no missing data. Thanks. ex. sqlplus user/password < get_data.sql > data_fifo.txt & bin/process_data... (2 Replies)
Discussion started by: korndog
2 Replies
Login or Register to Ask a Question