UNIX Pipe | Unix Linux Forums | UNIX for Dummies Questions & Answers

  Go Back    


UNIX for Dummies Questions & Answers If you're not sure where to post a UNIX or Linux question, post it here. All UNIX and Linux newbies welcome !!

UNIX Pipe

UNIX for Dummies Questions & Answers


Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 10-12-2005
akrathi akrathi is offline
Registered User
 
Join Date: Sep 2005
Last Activity: 23 October 2009, 3:20 PM EDT
Posts: 41
Thanks: 0
Thanked 0 Times in 0 Posts
UNIX Pipe

Hi ,
I want to understand how the PIPE works in unix . Precisely what I am doing is this .

1. Creating a Named PIPE with the command

mknod sqlldr.dat p

2. Directing a file output to the PIPE file in the background

cat abc > sqlldr.dat
3.SQL Loader in oracle is reading the data from this .

Now I want to undertsnad how it's really working ?

My concern is file abc is going to be a big one . Does this cause any memory problem

Links to how pipe works will be helfful

Does any body has a idea of reading a PIPE file from utl_file in oracle ?
Sponsored Links
    #2  
Old 10-12-2005
jim mcnamara jim mcnamara is offline Forum Staff  
...@...
 
Join Date: Feb 2004
Last Activity: 27 November 2014, 8:24 AM EST
Location: NM
Posts: 10,283
Thanks: 288
Thanked 815 Times in 757 Posts
a pipe is an intermediate file that allows to separate process to communicate.
The file uses FIFO I/O - first-in first-out. The last thing written to the file is the first thing read from the file.

I don't see any advantage to invoking sqlldr with a pipe. In fact, it may be a problem because pipes have a limit on "record" size - see /usr/include/limits.h _POSIX_PIPE_BUF and PIPE_BUF - and tables with several long columns will cause a problem with overflow.

One way pipes are really useful is to have two separate Oracle sessions comminuicating to one another using DBMS_PIPE
Sponsored Links
    #3  
Old 10-12-2005
akrathi akrathi is offline
Registered User
 
Join Date: Sep 2005
Last Activity: 23 October 2009, 3:20 PM EDT
Posts: 41
Thanks: 0
Thanked 0 Times in 0 Posts
I had an issue like this .

I have to delete few records from the file in the unix box and then load the data into the database .Precisely the records whose size is less then 250 . I think there is nothing in the sql loader which offers this . Checked when cluase etc .

What I am doing now is this : Read file and copy all the records with 250 chracters to a temp file . Then load the data from the temp file

I thougt the pipe will be useful in this case .

Thanks
Ashok
    #4  
Old 10-12-2005
Perderabo's Avatar
Perderabo Perderabo is offline Forum Staff  
Unix Daemon (Administrator Emeritus)
 
Join Date: Aug 2001
Last Activity: 24 November 2014, 10:48 PM EST
Location: Ashburn, Virginia
Posts: 9,930
Thanks: 64
Thanked 438 Times in 261 Posts
Pipes and fifos cannot overflow. A fifo, aka named pipe, is more of a interprocess communication mechanism than a file. Unlimited writes to fifos are required by Posix. If you exceed PIPE_BUF, you lose the guarantee of atomicity. This only has an effect in the case of multiple writing processes. Some people wanted a PIPE_MAX and such a constant is available but it is the same as the max value in a ssize_t field. Setting O_NONBLOCK and writing more than PIPE_BUF will result in partial writes and a poorly designed application could lose data under those circumstances, but that is not exactly an overflow.

With one process reading from a pipe and another writing to a pipe, they will take turns running. In theory, a pipeline like:
yes | cat > /dev/null
can run for all eternity and that continues to be true if a fifo is used instead.
Sponsored Links
    #5  
Old 10-12-2005
tmarikle tmarikle is offline Forum Advisor  
Registered User
 
Join Date: Jan 2005
Last Activity: 27 September 2011, 12:36 PM EDT
Posts: 683
Thanks: 0
Thanked 4 Times in 4 Posts
Quote:
Pipes and fifos cannot overflow.
...
With one process reading from a pipe and another writing to a pipe, they will take turns running.
In fact, this sort of mechanism works very well when having to export Oracle data when the data exceeds 2 GB and the Oracle exp utility doesn't support large filesystems.

Code:
mknod pipefile p
compress < pipefile > exp.dmp.Z &
exp file=pipefile

Sponsored Links
    #6  
Old 10-12-2005
jim mcnamara jim mcnamara is offline Forum Staff  
...@...
 
Join Date: Feb 2004
Last Activity: 27 November 2014, 8:24 AM EST
Location: NM
Posts: 10,283
Thanks: 288
Thanked 815 Times in 757 Posts
FWIW -
Under HPUX 11.0 Oracle 9.2.0.4.0 - 64bit Production - sqlldr does exactly that - when
the length of the logical record exceeds PIPE_BUF, data is sometimes lost and log errors about incomplete or lost data show up.

Overflow is not the right technical word - but I wasn't sure what the OP's level of understanding was. pipes really can't overflow, just lose track of EOR.
Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Passing struct through unix pipe -solved twnsfn34 UNIX for Dummies Questions & Answers 0 03-06-2010 10:40 PM
Converting hex value 7C (for pipe) to CRLF in Unix sfedak Shell Programming and Scripting 4 01-21-2009 03:49 PM
unix command pipe RubinPat UNIX for Advanced & Expert Users 7 01-07-2009 05:44 PM
Definition for the UNIX term Pipe dfrost126 UNIX for Dummies Questions & Answers 5 04-14-2008 08:17 PM
unix pipe in C meh Programming 1 10-16-2006 08:34 PM



All times are GMT -4. The time now is 07:02 PM.