UNIX Pipe


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers UNIX Pipe
# 1  
Old 10-12-2005
UNIX Pipe

Hi ,
I want to understand how the PIPE works in unix . Precisely what I am doing is this .

1. Creating a Named PIPE with the command

mknod sqlldr.dat p

2. Directing a file output to the PIPE file in the background

cat abc > sqlldr.dat
3.SQL Loader in oracle is reading the data from this .

Now I want to undertsnad how it's really working ?

My concern is file abc is going to be a big one . Does this cause any memory problem

Links to how pipe works will be helfful

Does any body has a idea of reading a PIPE file from utl_file in oracle ?
# 2  
Old 10-12-2005
a pipe is an intermediate file that allows to separate process to communicate.
The file uses FIFO I/O - first-in first-out. The last thing written to the file is the first thing read from the file.

I don't see any advantage to invoking sqlldr with a pipe. In fact, it may be a problem because pipes have a limit on "record" size - see /usr/include/limits.h _POSIX_PIPE_BUF and PIPE_BUF - and tables with several long columns will cause a problem with overflow.

One way pipes are really useful is to have two separate Oracle sessions comminuicating to one another using DBMS_PIPE
# 3  
Old 10-12-2005
I had an issue like this .

I have to delete few records from the file in the unix box and then load the data into the database .Precisely the records whose size is less then 250 . I think there is nothing in the sql loader which offers this . Checked when cluase etc .

What I am doing now is this : Read file and copy all the records with 250 chracters to a temp file . Then load the data from the temp file

I thougt the pipe will be useful in this case .

Thanks
Ashok
# 4  
Old 10-12-2005
Pipes and fifos cannot overflow. A fifo, aka named pipe, is more of a interprocess communication mechanism than a file. Unlimited writes to fifos are required by Posix. If you exceed PIPE_BUF, you lose the guarantee of atomicity. This only has an effect in the case of multiple writing processes. Some people wanted a PIPE_MAX and such a constant is available but it is the same as the max value in a ssize_t field. Setting O_NONBLOCK and writing more than PIPE_BUF will result in partial writes and a poorly designed application could lose data under those circumstances, but that is not exactly an overflow.

With one process reading from a pipe and another writing to a pipe, they will take turns running. In theory, a pipeline like:
yes | cat > /dev/null
can run for all eternity and that continues to be true if a fifo is used instead.
# 5  
Old 10-12-2005
Quote:
Pipes and fifos cannot overflow.
...
With one process reading from a pipe and another writing to a pipe, they will take turns running.
In fact, this sort of mechanism works very well when having to export Oracle data when the data exceeds 2 GB and the Oracle exp utility doesn't support large filesystems.
Code:
mknod pipefile p
compress < pipefile > exp.dmp.Z &
exp file=pipefile

# 6  
Old 10-12-2005
FWIW -
Under HPUX 11.0 Oracle 9.2.0.4.0 - 64bit Production - sqlldr does exactly that - when
the length of the logical record exceeds PIPE_BUF, data is sometimes lost and log errors about incomplete or lost data show up.

Overflow is not the right technical word - but I wasn't sure what the OP's level of understanding was. pipes really can't overflow, just lose track of EOR.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

UNIX Pipe -Exit when there are no bytes to read

Hi All, I'm creating a program which reads millions of bytes from the PIPE and do some processing. As the data is more, the idea is to read the pipe parallely. Sun Solaris 8 See the code below: #!/bin/sh MAXTHREAD=30 awk '{print $1}' metadata.csv > nvpipe & while do ... (3 Replies)
Discussion started by: mr_manii
3 Replies

2. Shell Programming and Scripting

UNIX fifo concurrent read from a named pipe

I have created a fifo named pipe in solaris, which writes the content of a file, line by line, into pipe as below: $ mkfifo namepipe $ cat books.txt "how to write unix code" "how to write oracle code" $ cat books.txt >> namepipe & I have a readpipe.sh script which reads the named... (2 Replies)
Discussion started by: naveen mani
2 Replies

3. Shell Programming and Scripting

Replacing UNIX 1 byte delimiters to the pipe separator

Hi I am new to Unix Shell scripting have a requirement where I have to replace the "unix 1 byte delimiter" with the "pipe" separator and also remove any carriage returns and line feeds if any The Source File 4 QFH Jungle Hill 32-34 City Road London SE23 3UX the output should be ... (3 Replies)
Discussion started by: dJHa
3 Replies

4. UNIX for Dummies Questions & Answers

Passing struct through unix pipe -solved

EDIT: Nevermind, called a friend who is good at this stuff and he figured it out :D Hi all, So I'm trying to teach myself to write programs for unix in c. I am currently creating a program, and I need to pass a struct through a pipe, but I can't figure out how. The struct I want to pass... (0 Replies)
Discussion started by: twnsfn34
0 Replies

5. Shell Programming and Scripting

Converting hex value 7C (for pipe) to CRLF in Unix

I am trying to convert a txt file that includes one long string of data. The lines are separated with hex value 7C (for pipe). I am trying to process this file using SQR (Peoplesoft) so I thought the easiest thing to do would be to replace the eol char with a CRLF in unix so I can just... (4 Replies)
Discussion started by: sfedak
4 Replies

6. UNIX for Advanced & Expert Users

unix command pipe

I am pretty new to UNIX. My client has a requirement where in a directory we have some files with somewhat similar name like test_XX.txt, test_XY.txt, test_XZ.txt, test_ZZ.txt, test_ZY.txt, test_ZX.txt, test_YY.txt......Out of these files few files have 0 bytes. Is there a way where we can go... (7 Replies)
Discussion started by: RubinPat
7 Replies

7. UNIX for Dummies Questions & Answers

Definition for the UNIX term Pipe

Looking for examples/definition of what the term Pipe means in UNIX. Please provide answers and illustrations if possible or direction. Thanks!:) (5 Replies)
Discussion started by: dfrost126
5 Replies

8. Shell Programming and Scripting

splitting a pipe delimited file in unix

Could one of you shad some light on this: I need to split the file by determining the record count and than splitting it up into 4 files. Please note, this is not a fixed record length but rather a "|" delimited file. I am not sure as how to handle reminder/offset for the 4th file. For... (4 Replies)
Discussion started by: ddedic
4 Replies

9. Programming

unix pipe in C

I currently stuck on a simple program that requires unix pipe. I'm have never programmed with unix pipe before, so if anyone can point me to the right different will be greatly appreciated! I'm suppose to write a program that the parent spawns many child processes and each of the child process... (1 Reply)
Discussion started by: meh
1 Replies

10. HP-UX

Compress while wrting to a file using a unix pipe

I would like to know if the following can be done. route output from an sql select directly to a pipe and compress it at the same time. regards Albert (2 Replies)
Discussion started by: booyena1
2 Replies
Login or Register to Ask a Question