UNIX File Processsing...Urgent Please


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers UNIX File Processsing...Urgent Please
# 1  
Old 04-16-2008
UNIX File Processsing...Urgent Please

Gurus,
I am faced with a situation where a source system FTPs files to a directory in UNIX and we invoke SQLLDR process to process the files..The problem is we are continoulsy (3 minute)receiving files and we are processing data every 5 minutes.. Is there a way to avoid processing files that are curently being copied ?? I use a find command to process specific files...is there an extension to find command to check if the file is not completely copied?
Thanks
# 2  
Old 04-16-2008
There's not a very easy way to check if the file is complete or not but there's a few different tricks you can use:
  • Before sending the file, perform a checksum of it. Then either send that first in a seperate file, or include it in the file name. Then check this checksum matches before processing the file.
  • Have your FTP process at the other end, send the file as something like filename.tmp, then rename it once the transfer is finished
  • Use an app designed for the purpose like xcom (costs money though)

A more fiddly solution could be found using lsof to look at what files the ftpd has open but I think it's not the easiest way to go here Smilie
# 3  
Old 04-17-2008
lsof actually is not a difficult way to do so and it proved to be very helpful to me.

Another way could be -
is there any trailer record in each and every file that is being transferred, if so its going to be even more simple
say there is a trailer record like
TRL<blah><blah>

then before copying or processing the file check whether TRL record is in place, then you are done. Smilie
# 4  
Old 04-17-2008
Thanks for the answers...I think i have to go with lsof...but when i run it in my production box i get the following error?> any idea?
lsof: can't open /dev/mem: Permission denied
lsof: can't open /dev/kmem: Permission denied
# 5  
Old 04-17-2008
I would guess you are trying to run it as the user that processes the files?

lsof must be run by the process owner - in your case root I'd wager. That's why I was saying it'll be difficult Smilie
# 6  
Old 04-19-2008
one try please

try running it as sudo

Code:
sudo lsof arguments

 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

URGENT Reading a file and assessing the syntax shell script URGENT

I am trying to write a shell script which takes an input file as an arguement in the terminal e.g. bash shellscriptname.sh input.txt. I would like for the file to be read line by line each time checking if the .txt file contains certain words or letters(validating the syntax). If the line being... (1 Reply)
Discussion started by: Gurdza32
1 Replies

2. Shell Programming and Scripting

Need Urgent Help from UNIX gurus to get specific data from a file

Hi, I have a file monitor.txt as below... # Times are converted to local time from GMT. # Local Timezone: EST (GMT -05:00) PARAM1 { TIME 30; CC1 "xxxxx"; CC2 "xxxxx"; CC3 "xxxxx"; CC4 "xxxxx"; } PARAM2 { 4061 :... (3 Replies)
Discussion started by: zaq1xsw2
3 Replies

3. AIX

Processsing Units and Virtual Processors.

Hi, on redhat linux 5.5 (IBM PPC) os, whenever I am running the command top, it is showing 8 cpu. Processing Units Property Current Pending Minimum 0.1 Assigned 0.8 Maximum 1 Virtual Processors Property Current Pending Minimum 1 Assigned 4 ... (1 Reply)
Discussion started by: manoj.solaris
1 Replies

4. UNIX for Dummies Questions & Answers

Unix Command Urgent Help!

Hi All, i am new on this forum, can someone please answer the following questions as i am also new for UNIX, you help would be much appreciated: 1. How do you delete the full directory hierarchy? 2. How do you find out all process run by user Kamran ? 3. how would you extract something... (1 Reply)
Discussion started by: kami151
1 Replies

5. UNIX for Advanced & Expert Users

Need a unix script - very urgent

Hi All, I need a script that invokes the sql query every 14 days and send email Very Very urgent this one to me.... (6 Replies)
Discussion started by: bujjisveeru
6 Replies

6. Shell Programming and Scripting

(Urgent):Creating flat file using sql script and sqlplus from UNIX Shell Script

Hi, I need help urgently for following issue. Pls help me to resolve this issue. I am calling sql script file(file1.sql) from UNIX Shell Script(script1.ksh) using sql plus and trying to create flat file that contains all records returned from SQL query in SQL script(file1.sql) I given... (6 Replies)
Discussion started by: praka
6 Replies

7. UNIX for Dummies Questions & Answers

Urgent help needed to delete some text without opening the file in unix

Hi To delete some text in 2 files in line1 ( not complete line) in unix without opening the files. For example: source file is like this <?xml version="1.0"... (5 Replies)
Discussion started by: pyaranoid
5 Replies

8. HP-UX

HP Unix | Urgent Need Help

Hi, I'm new to unix and i'm using HP Unix I tried the following and getting error like 1. $sam sh:sam not found 2.Also i want to monitor/trace other user's command what they are using when am in to the same server Appreciate if any one help on this (4 Replies)
Discussion started by: msgobinathan
4 Replies

9. Shell Programming and Scripting

Urgent Need your help --New to Unix

I am getting data like this in a flat file seg No Test A4 0000002595, A2, test ,Datatest Test, A1 0000002595, A1,SELF , Employed,1,2,3,400.50 Test, A2 0000002595, A2, test ,Data,eurp Test, A3 0000002595, A2, test ,Data I need to produce a flat file so that output file... (0 Replies)
Discussion started by: nikeng
0 Replies

10. UNIX for Dummies Questions & Answers

I am in urgent need of all UNIX-derivates...

Hi, could anybody name all the different UNIX versions, producer, historie, etc? need a short briefing... THX a lot in advance... Greetings, PatrickXXO (2 Replies)
Discussion started by: PatrickXXO
2 Replies
Login or Register to Ask a Question