Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Problems understanding pipes
Top Forums Programming Problems understanding pipes Post 302562707 by Corona688 on Friday 7th of October 2011 06:52:23 PM
Old 10-07-2011
Quote:
Originally Posted by ab_tall
What I meant to say is I am having a hard time debugging the child process once it execs, as gdb is attached to the parent.
(I thought as there is no explicit concept of threads in Linux, it would be OK to call the child process a thread, but i guess that lead to confusion.)
What system are you on? On linux, you can use strace, which lists all system calls your program and its children are making. (-f means 'follow children') Just system calls, only system calls, and nothing but system calls -- not line numbers or source code. But it's useful for clearing up mysteries like "why is my program freezing" -- it's stuck up on write(). I've also used it to track down where some silly programs were looking for config files in -- just hunt for open calls to see what files they're trying to open...

It ends up as an awful big list, but it's easy to cut down with grep.

Code:
$ gcc multipipe.c
$ strace ./a.out 2> log
<process runs and finishes>
$ egrep "(fork|execve|open|close|pipe|dup|read|write)\(" log
execve("./a.out", ["./a.out"], [/* 28 vars */]) = 0
open("/etc/ld.so.cache", O_RDONLY)      = 3
close(3)                                = 0
open("/lib/libc.so.6", O_RDONLY)        = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20m\1\0004\0\0\0"..., 512) = 512
close(3)                                = 0
pipe([3, 4])                            = 0
[pid  9380] close(4 <unfinished ...>
[pid  9381] close(4 <unfinished ...>
[pid  9380] pipe( <unfinished ...>
[pid  9381] close(3 <unfinished ...>
[pid  9381] execve("/usr/local/bin/ls", ["ls"], [/* 28 vars */] <unfinished ...>
[pid  9380] close(5 <unfinished ...>
[pid  9381] execve("/usr/bin/ls", ["ls"], [/* 28 vars */] <unfinished ...>
[pid  9380] close(3 <unfinished ...>
[pid  9381] execve("/bin/ls", ["ls"], [/* 28 vars */] <unfinished ...>
[pid  9382] close(3Process 9383 attached
[pid  9380] close(4 <unfinished ...>
[pid  9382] close(5 <unfinished ...>
[pid  9382] close(4 <unfinished ...>
[pid  9382] execve("/usr/local/bin/tac", ["tac"], [/* 28 vars */] <unfinished ...>
[pid  9381] open("/etc/ld.so.cache", O_RDONLY <unfinished ...>
[pid  9382] execve("/usr/bin/tac", ["tac"], [/* 28 vars */] <unfinished ...>
[pid  9383] close(4 <unfinished ...>
[pid  9381] close(3 <unfinished ...>
[pid  9381] open("/lib/librt.so.1", O_RDONLY <unfinished ...>
[pid  9381] read(3,  <unfinished ...>
[pid  9382] open("/etc/ld.so.cache", O_RDONLY <unfinished ...>
[pid  9382] close(3 <unfinished ...>
[pid  9381] close(3 <unfinished ...>
[pid  9382] open("/lib/libc.so.6", O_RDONLY <unfinished ...>
[pid  9383] execve("/usr/local/bin/less", ["less"], [/* 28 vars */] <unfinished ...>
[pid  9382] read(3,  <unfinished ...>
[pid  9383] execve("/usr/bin/less", ["less"], [/* 28 vars */] <unfinished ...>
[pid  9381] open("/lib/libacl.so.1", O_RDONLY) = 3
[pid  9381] read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0000\32\0\0004\0\0\0"..., 512) = 512
[pid  9382] close(3 <unfinished ...>
[pid  9383] open("/etc/ld.so.cache", O_RDONLY <unfinished ...>
[pid  9383] close(3 <unfinished ...>
[pid  9383] open("/lib/libncurses.so.5", O_RDONLY <unfinished ...>
[pid  9383] read(3,  <unfinished ...>
[pid  9381] close(3)                    = 0
[pid  9381] open("/lib/libc.so.6", O_RDONLY) = 3
[pid  9381] read(3,  <unfinished ...>
[pid  9383] close(3 <unfinished ...>
[pid  9383] open("/lib/libc.so.6", O_RDONLY <unfinished ...>
[pid  9383] read(3,  <unfinished ...>
[pid  9382] open("/tmp/tacpBNVeU", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600 <unfinished ...>
[pid  9383] close(3 <unfinished ...>
[pid  9383] open("/lib/libdl.so.2", O_RDONLY <unfinished ...>
[pid  9383] read(3,  <unfinished ...>
[pid  9382] read(0,  <unfinished ...>
[pid  9383] close(3 <unfinished ...>
[pid  9381] close(3 <unfinished ...>
[pid  9381] open("/lib/libpthread.so.0", O_RDONLY <unfinished ...>
[pid  9381] read(3,  <unfinished ...>
[pid  9381] close(3 <unfinished ...>
[pid  9381] open("/lib/libattr.so.1", O_RDONLY <unfinished ...>
[pid  9381] read(3,  <unfinished ...>
[pid  9381] close(3 <unfinished ...>
[pid  9383] open("/etc/terminfo/x/xterm", O_RDONLY|O_LARGEFILE <unfinished ...>
[pid  9383] read(3,  <unfinished ...>
[pid  9383] close(3)                    = 0
[pid  9383] open("/usr/bin/.sysless", O_RDONLY|O_LARGEFILE <unfinished ...>
[pid  9383] open("/etc/sysless", O_RDONLY|O_LARGEFILE <unfinished ...>
[pid  9383] open("/home/username/.less", O_RDONLY|O_LARGEFILE <unfinished ...>
[pid  9383] open("/home/username/.lesshst", O_RDONLY|O_LARGEFILE <unfinished ...>
[pid  9383] read(3,  <unfinished ...>
[pid  9383] read(3,  <unfinished ...>
[pid  9383] close(3 <unfinished ...>
[pid  9383] open("/dev/tty", O_RDONLY|O_LARGEFILE <unfinished ...>
[pid  9381] open(".", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC <unfinished ...>
[pid  9381] close(3 <unfinished ...>
[pid  9383] write(1, "\33[?1049h\33[?1h\33=", 15 <unfinished ...>
[pid  9383] read(0,  <unfinished ...>
[pid  9381] write(1, "a.out\nlog\nmultipipe.c\n", 22 <unfinished ...>
[pid  9382] read(0,  <unfinished ...>
[pid  9381] close(1 <unfinished ...>
[pid  9382] write(3, "a.out\nlog\nmultipipe.c\n", 22 <unfinished ...>
[pid  9381] close(2 <unfinished ...>
[pid  9382] read(3, "a.out\nlog\nmultipipe.c\n", 22) = 22
[pid  9382] close(0)                    = 0
[pid  9382] write(1, "multipipe.c\nlog\na.out\n", 22 <unfinished ...>
[pid  9383] write(1, "multipipe.c\33[m\nlog\33[m\na.out\33[m\n", 31 <unfinished ...>
[pid  9382] close(1 <unfinished ...>
[pid  9383] read(0,  <unfinished ...>
[pid  9382] close(2 <unfinished ...>
[pid  9383] write(1, "\33[7mlines 1-3/3 (END) \33[27m\33[K", 30 <unfinished ...>
[pid  9383] read(3,  <unfinished ...>
[pid  9383] write(1, "\r\33[K\33[?1l\33>\33[?1049l", 19) = 19

# what about just process 9383?  what was it doing?
$ egrep "(fork|execve|open|close|pipe|dup|read|write)\(" log | grep 9383

[pid  9382] close(3Process 9383 attached
[pid  9383] close(4 <unfinished ...>
[pid  9383] execve("/usr/local/bin/less", ["less"], [/* 28 vars */] <unfinished ...>
[pid  9383] execve("/usr/bin/less", ["less"], [/* 28 vars */] <unfinished ...>
[pid  9383] open("/etc/ld.so.cache", O_RDONLY <unfinished ...>
[pid  9383] close(3 <unfinished ...>
[pid  9383] open("/lib/libncurses.so.5", O_RDONLY <unfinished ...>
[pid  9383] read(3,  <unfinished ...>
[pid  9383] close(3 <unfinished ...>
[pid  9383] open("/lib/libc.so.6", O_RDONLY <unfinished ...>
[pid  9383] read(3,  <unfinished ...>
[pid  9383] close(3 <unfinished ...>
[pid  9383] open("/lib/libdl.so.2", O_RDONLY <unfinished ...>
[pid  9383] read(3,  <unfinished ...>
[pid  9383] close(3 <unfinished ...>
[pid  9383] open("/etc/terminfo/x/xterm", O_RDONLY|O_LARGEFILE <unfinished ...>
[pid  9383] read(3,  <unfinished ...>
[pid  9383] close(3)                    = 0
[pid  9383] open("/usr/bin/.sysless", O_RDONLY|O_LARGEFILE <unfinished ...>
[pid  9383] open("/etc/sysless", O_RDONLY|O_LARGEFILE <unfinished ...>
[pid  9383] open("/home/username/.less", O_RDONLY|O_LARGEFILE <unfinished ...>
[pid  9383] open("/home/username/.lesshst", O_RDONLY|O_LARGEFILE <unfinished ...>
[pid  9383] read(3,  <unfinished ...>
[pid  9383] read(3,  <unfinished ...>
[pid  9383] close(3 <unfinished ...>
[pid  9383] open("/dev/tty", O_RDONLY|O_LARGEFILE <unfinished ...>
[pid  9383] write(1, "\33[?1049h\33[?1h\33=", 15 <unfinished ...>
[pid  9383] read(0,  <unfinished ...>
[pid  9383] write(1, "multipipe.c\33[m\nlog\33[m\na.out\33[m\n", 31 <unfinished ...>
[pid  9383] read(0,  <unfinished ...>
[pid  9383] write(1, "\33[7mlines 1-3/3 (END) \33[27m\33[K", 30 <unfinished ...>
[pid  9383] read(3,  <unfinished ...>
[pid  9383] write(1, "\r\33[K\33[?1l\33>\33[?1049l", 19) = 19

$

lots of options for strace too, see the manpage.

---------- Post updated at 04:52 PM ---------- Previous update was at 04:36 PM ----------

Quote:
Originally Posted by ab_tall
how is the pipe underlying structure implemented?
If it is just another file, is it possible to see the contents of the pipe {so as to know what's being passed on between the read and write ends}.
It's not a file on disk. It's a memory buffer inside the kernel itself. The kernel keeps track of how many processes have which ends open and which processes need to be stopped or started when data becomes available or room in the buffer becomes available. When everything using the pipe finally closes it or exits, the kernel sees that nothing needs the buffer anymore and deletes it.

The buffer can vary in size on different systems. In most Linux I think it's 64 kilobytes.
Quote:
Finally,
Is there a way to find out which FDs point to the same underlying description?
If it's hidden somewhere in lsof, i'll dig deeper, but if not do let me know.
In Linux and a few UNIXes, you can see that under /proc/. Try this:

Code:
$ echo | ls -l /proc/self/fd
lr-x------ 1 username users 64 Oct  7 16:47 0 -> pipe:[618262]
lrwx------ 1 username users 64 Oct  7 16:47 1 -> /dev/pts/0
lrwx------ 1 username users 64 Oct  7 16:47 2 -> /dev/pts/0
lr-x------ 1 username users 64 Oct  7 16:47 200 -> /home/username/.ssh-agent
lr-x------ 1 username users 64 Oct  7 16:47 3 -> /proc/10073/fd

'self' is just a special folder meaning 'my own process number', so ls is listing its own open files. You could 'ls /proc/1234' to list process 1234's files.

0 is stdin, the pipe attaching it to 'echo'. I think 618262 is a unique number specific to that particular pipe. It's not a valid link, you can't open it -- it's just informational.

1 and 2 are stdout and stderr, both attached to the same terminal here. They're actually valid symlinks, try
Code:
echo asdf > /proc/fd/self/1

200 is a file my terminal opens on login, just a little script I set up to keep my SSH keys straight.

3 is the directory /proc/self/fd, which ls opened so it could list its own open files. The kernel decided 'self' meant 10073.

Sockets also show up in this list, if you have any open, being that sockets are FD's too...
Last edited by Corona688; 10-07-2011 at 07:41 PM..
This User Gave Thanks to Corona688 For This Post:
ab_tall
 

10 More Discussions You Might Find Interesting

1. Filesystems, Disks and Memory

PIPEs and Named PIPEs (FIFO) Buffer size

Hello! How I can increase or decrease predefined pipe buffer size? System FreeBSD 4.9 and RedHat Linux 9.0 Thanks! (1 Reply)
Discussion started by: Jus
1 Replies

2. Shell Programming and Scripting

cd using pipes

Hi, Can the cd command be invoked using pipes??? My actual question is slightly different. I am trying to run an executable from different folders and the path of these folders are obtained dynamically from the front end. Is there a way in which i can actually run the executable... (2 Replies)
Discussion started by: Sinbad
2 Replies

3. UNIX for Advanced & Expert Users

FIFO Pipes

Hi...Can anyone please guide me on FIFO Pipes in UNIX.I have lerant things like creating fifo pipes,using them for reads and writes etc.I want to know what is the maximum amount of memory that such a pipe may have? Also can anyone guide me on where to get info on this topic from? (4 Replies)
Discussion started by: tej.buch
4 Replies

4. UNIX for Advanced & Expert Users

Consolidating Pipes

This is something I've given a lot of thought to and come up with no answer. Say you have a data stream passing from a file, through process A, into process B. Process A only modifies a few bytes of the stream, then prints the rest of the stream unmodified. Is there any way to stream the file... (4 Replies)
Discussion started by: Corona688
4 Replies

5. Shell Programming and Scripting

named pipes

How to have a conversation between 2 processes using named pipes? (5 Replies)
Discussion started by: kanchan_agr
5 Replies

6. UNIX for Dummies Questions & Answers

learning about pipes!

im trying to figure out how to do the following: using pipes to combine grep and find commands to print all lines in files that start with the letter f in the current directory that contain the word "test" for example? again using pipes to combine grep and find command, how can I print all... (1 Reply)
Discussion started by: ez45
1 Replies

7. Shell Programming and Scripting

Problems understanding example code

I am really new to UNIX and programming in general and so apologies if this thread is a bit simple. I have searched and found a piece of sample code for a training program I am currently undertaking, but seeing as I am relatively new, I dont completely understand how it works. Here is the... (1 Reply)
Discussion started by: Makaer
1 Replies

8. UNIX for Dummies Questions & Answers

Problems understanding example code

I am really new to UNIX and programming in general and so apologies if this thread is a bit simple. I have searched and found a piece of sample code for a training program I am currently undertaking, but seeing as I am relatively new, I dont completely understand how it works. Here is the... (6 Replies)
Discussion started by: Makaer
6 Replies

9. Programming

Pipes in C

Hello all, I am trying to learn more about programming Unix pipes in C. I have created a pipe that does od -bc < myfile | head Now, I am trying to create od -bc < myfile | head | wc Here is my code, and I know I might be off, thats why I am here so I can get some clarification. #include... (1 Reply)
Discussion started by: petrca
1 Replies

10. Homework & Coursework Questions

Using Pipes and Redirection

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: Create a pipe to show the number of people who are logged into the system right now. Create a pipe to show... (2 Replies)
Discussion started by: lakers34kb
2 Replies
All times are GMT -4. The time now is 10:53 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy