Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Weird redirection behaviour

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Weird redirection behaviour
# 1  
Old 08-29-2012
Question Weird redirection behaviour
Linux. Bash 4.2.10.

Let's say that i want the stderr of a command redirected to one file (err), and both stdout and stderr redirected to another file (outanderr).

Easy. Many ways to do it. But may times the file outanderr will be messed up (with the output and the error mixed toghether).

See:
Code:
lem@biggy:/tmp$ touch imafile
lem@biggy:/tmp$ (ls imafile idontexist 2>&1 >&3 | tee err >&3) 3>outanderr
lem@biggy:/tmp$ cat outanderr
ls: impossibile accedere a idontexist: File o directory non esistente
imafile

It looks nice (sorry for the Italian, but this is so like English that I'm sure it's easily understandable).
But let's retry. After a few times:

Code:
lem@biggy:/tmp$ (ls imafile idontexist 2>&1 >&3 | tee err >&3) 3>outanderr
lem@biggy:/tmp$ cat outanderr
ls: impossibile accedere a idontexistimafile
: File o directory non esistente

Here above the output (imafile) has gone in the middle of the error!

Another solution:
Code:
lem@biggy:/tmp$ ((ls imafile idontexist >&3) 2> >(tee err >&4)) 3>outanderr 4>&3
lem@biggy:/tmp$ cat outanderr
ls: impossibile accedere a idontexist: File o directory non esistente
imafile

Nice. Let's retry. After a few times:
Code:
lem@biggy:/tmp$ ((ls imafile idontexist >&3) 2> >(tee err >&4)) 3>outanderr 4>&3
lem@biggy:/tmp$ cat outanderr
ls: impossibile accedere a idontexistimafile
 directory non esistente

Again: output in the middle of the (semi-cutted) error.

The same with things like:
Code:
$ ls imafile idontexist >outanderr 2> >(tee err >>outanderr)

The only one I've found to be safe in my shell seems to be:
Code:
$ ls imafile idontexist 2> >(tee err >outanderr) | tee -a outanderr >/dev/null

So, my questions are:
a) what's going on?
b) since all redirections are set before executing the commands, where's the difference with a simple thing as $ ls imafile idontexist >outanderr 2>&1, which never ever fails?
c) the last solution is really safe? If so, why?

Thanks for everything.
--
Bye
# 2  
Old 08-29-2012
Bash - 3.2.25

I am getting only one error ...Smilie

Code:
ls: idontexist: No such file or directory

# 3  
Old 08-30-2012
I've been told that also
Code:
$ ls imafile idontexist 2> >(tee err >outanderr) | tee -a outanderr >/dev/null

has created problems to someone, so it isn't safe.

Thinking it were safe confused me.

Of course when there is only one command, as in
Code:
command >file 2>&1

the writes are sequential, so there cannot be problems.

When there are many (sub)processes, created by pipes or command substitution or whatever else, all concurrently writing on the same file, I cannot see a clear way to be sure they won't overlap - using only the shell tools: /var/log/syslog works flawlessly, but there's a daemon in control, Which I think has its own importance. Smilie
Depending on the scheduler, on the number of CPUs and maybe on many other things, some solutions may seem to work here for a while, but don't work there.

Beside some awful workarounds with a very tiny field of application, like something that:
Code:
( ls imafile idontexist 2>&1 >&3 | { sleep 1; tee err >&3; } ) 3>outanderr

I can't think of any real solution. But since I know very little at all about these black boxes we call "computers", I wouldn't be surprised to be totally wrong.

Thanks to anyone reading all this awkward stuff.
--
Bye
# 4  
Old 08-30-2012
Quote:
Originally Posted by Lem
So, my questions are:
a) what's going on?
Multiple processes are writing to the same file. Some of the streams are unbuffered (stderr) and some are buffered (stdout). You are at the mercies of the c library's stream buffering and the system scheduler.

(ls imafile idontexist 2>&1 >&3 | tee err >&3) 3>outanderr:
1. ls writes 'imafile' to its stdout stream (outanderr file). This stream is fully-buffered so the word just sits in a buffer.
2. ls writes the "idontexist" error message to the stderr stream (the pipe). This is an unbuffered stream, so the c library executes the system call to write the error message to the pipe.
3. ls is done, so all of its buffers are flushed and it exits. This prints the word "imafile" which had been buffered.

Initially, tee is sleeping, since there's nothing in the pipe. Once ls writes that error message, the system knows that it can wake tee. In your examples, tee is always woken before ls can flush "imafile" (between steps 2 and 3 above). The mixing of the streams occurs when tee isn't able to finish writing the entire message to its stdout before ls resumes at step 3.


Quote:
Originally Posted by Lem
b) since all redirections are set before executing the commands, where's the difference with a simple thing as $ ls imafile idontexist >outanderr 2>&1, which never ever fails?
There is only one process involved.


Quote:
Originally Posted by Lem
c) the last solution is really safe? If so, why?
No. ls imafile idontexist 2> >(tee err >outanderr) | tee -a outanderr >/dev/null is not immune.

Regards,
Alister
These 2 Users Gave Thanks to alister For This Post:
elixir_sinari Lem
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Programming

Sort behaviour

I see strange results when sorting with -n options and I wander if somebody can explain it. Input file and two results: $ cat aa 14 -1 11 -1 0 -1 0 $ sort -u aa -1 0 (1 Reply)
Discussion started by: migurus
1 Replies

2. Programming

different behaviour in fg and bg

fg = foreground bg = background I have a cobol program that I start with a very simple script. The script is not at fault as it has not changed and the program worked in fg and bg before. I have altered the logging in the program and moved my cursor declare to working storage. The program runs... (6 Replies)
Discussion started by: Bruble
6 Replies

3. Ubuntu

Weird rm behaviour

I am little bit confused by the behaviour of rm in Ubuntu. It seems that as a regular user I can delete files owned by another user even when the permissions are set to 644. Here is an example: cjohnson@carbon:~/test$ sudo touch testfile cjohnson@carbon:~/test$ ls -al total 8 drwxr-xr-x... (2 Replies)
Discussion started by: ccj4467
2 Replies

4. Shell Programming and Scripting

find: "weird" regex behaviour

I have these two files in current dir: oos.txt oos_(copy).txt I execute this find command:find . -regex './oos*.txt'And this outputs only the first file (oos.txt)! :confused: Only if I add another asterisk to the find find . -regex './oos*.*txt' do I also get the second file... (7 Replies)
Discussion started by: courteous
7 Replies

5. Shell Programming and Scripting

Weird script behaviour !

Hello, I am getting an infinite loop from a script in Linux. Here is the last version of the script in question. As you can see I tried to define everything properly: #!/bin/ksh # Script to loop over a series of dates set -ex typeset -i start_date=20090701 typeset -i... (2 Replies)
Discussion started by: stavros
2 Replies

6. Shell Programming and Scripting

cp -R behaviour

i 've noticed the following difference between freebsd cp and gnu cp from the freebsd cp man page: -R ... If the source_file ends in a /, the contents of the directory are copied rather than the directory itself. ... on gnu cp from the man pagewhile on gnu cp manpage: ‘-r'... (2 Replies)
Discussion started by: aegis
2 Replies

7. Shell Programming and Scripting

Weird sed behaviour in script

I've written a small script to replace certain words in all the the files in a directory. #!/bin/sh #Get list of files to be edited file_list=`ls -p` for i in $file_list do echo "Processing $i" alteredi=`echo "$i" | sed -e 's/\//d/'` if then if then #actual altering (2 Replies)
Discussion started by: Peetrus
2 Replies

8. UNIX for Advanced & Expert Users

Weird sudo behaviour

Hi gurus. I implemented sudo and have the following in my sudo config file *************** # User alias specification User_Alias VENDOR = user1 # User privilege specification VENDOR ALL = NOPASSWD: /bin/, /sbin/, /usr/local/bin/, \ !/bin/su,... (1 Reply)
Discussion started by: geomonap
1 Replies

9. Programming

Can some 1 explain why this behaviour

#include <iostream> using namespace std; int main() { const int l_test = 999999999; int *l_ptr = (int*) &l_test; cout<<"Constant Addr:"<<&l_test<<" Value:"<<l_test<<endl; cout<<"Pointer Addr:"<<l_ptr<<" Value:"<<*l_ptr<<endl; *l_ptr = 888888888; // Manipulating... (2 Replies)
Discussion started by: helpmenow
2 Replies
Login or Register to Ask a Question