How to process only new files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to process only new files
# 1  
Old 07-17-2007
How to process only new files

Dear Experts,

I desperately need help in some scripting that im totally stuck at.

I have some files like this :-

-rw-rw---- 1 rtp99 ticgroup 2603099 Jul 17 13:24 cft.CO0102.20070717051933.20070717052451
-rw-rw---- 1 rtp99 ticgroup 3040772 Jul 17 13:24 cft.CO0101.20070717051133.20070717052453
-rw-rw---- 1 rtp99 ticgroup 3102533 Jul 17 13:36 cft.CO0102.20070717052451.20070717053611
-rw-rw---- 1 rtp99 ticgroup 3069764 Jul 17 13:37 cft.CO0101.20070717052453.20070717053741
-rw-rw---- 1 rtp99 ticgroup 3159779 Jul 17 13:48 cft.CO0102.20070717053611.20070717054803
-rw-rw---- 1 rtp99 ticgroup 3167636 Jul 17 13:48 cft.CO0101.20070717053741.20070717054844
-rw-rw---- 1 rtp99 ticgroup 3074107 Jul 17 13:59 cft.CO0102.20070717054803.20070717055925
-rw-rw---- 1 rtp99 ticgroup 3098716 Jul 17 14:00 cft.CO0101.20070717054844.20070717060030
-rw-rw---- 1 rtp99 ticgroup 3182712 Jul 17 14:10 cft.CO0102.20070717055925.20070717061033
-rw-rw---- 1 rtp99 ticgroup 3195226 Jul 17 14:11 cft.CO0101.20070717060030.20070717061148


As you can see the files come in every 2-10 mins. I need to process some info inside these files and dump the info elsewhere. The problem is, i dont know how to only pick up the next new file (latest file), and process it. I tried dumping the output of "ls" onto a file, and then using "comm" to find the new file, but this method seems to be slowing down my server a lot.

Appreciate any help. Many thanks.

Sara
# 2  
Old 07-17-2007
To find the latest file in the path

Code:
ls -t1 cft.C* | head -1 | read filename

So now the variable filename will have the latest file name.
# 3  
Old 07-17-2007
you can try this logic

Code:
while :
do
ls -lrt | tail -1  
echo # process them what ever you want
sleep 10
done

it takes last file only since i used "tail -1" command


some friends may come with better logic
# 4  
Old 07-17-2007
Sorry guys, I forgot to mention that I am only allowed to let the cron run every every 15 mins, so "ls -ltr | tail -1" is might miss out a file. Any way i will be able to pick up only the new files that were not processed the last time cron ran?
# 5  
Old 07-17-2007
Method 1: use find -newer and a reference file.
Code:
find . -type f -newer ref_file -print |\
while read filename
do
    :  something with $filename
done
touch ref_file

Method 2: check that you haven't processed the file before.
Code:
set -- *
for filename
do
    if ! grep -q $filename ref_file
    then
        :  something with $filename
        echo $filename >> ref_file
    fi
done

Combine methods if necessary.
# 6  
Old 07-17-2007
Thanks Ygor. I found that helpful. Thanks everyone.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Red Hat

Process to use open files

I would like to test open files usage in my system, if I want to create a process ( or script ) that can use a certain amount ( eg. 1000 ) of open files , is it possible to create such process ( or script ) ? (3 Replies)
Discussion started by: ust
3 Replies

2. Shell Programming and Scripting

What is the correct way to process files

Is there a general way I can use so as to process paths/files from your terminal without the need to backslash things? Till recently I used to use touch "file_here" but it does not work if file_here contains any of the characters ` and ", I still need to backslash them if I want to make the... (1 Reply)
Discussion started by: hakermania
1 Replies

3. Shell Programming and Scripting

Loop to process 2 files with same name in different path

Hello forum members, I hope you can help me with this I don't know hot to reach. I have a list of files in "/home/MyPath1/" and in "/home/MyPath2/". The files have the same name in both folders. (but different content, the content doesn't matter here I think) /home/MyPath1/ filename1.txt... (4 Replies)
Discussion started by: Ophiuchus
4 Replies

4. UNIX for Dummies Questions & Answers

Find the process and their files

Hi all, when i run a program in a Linux, it may call bunch of other files. How do i find those files? thanks in adanvance Sajith (1 Reply)
Discussion started by: email2sajith
1 Replies

5. OS X (Apple)

Open Files for a process

I am having a client/server application which will download files from server. If server has directories, it will create directories/sub directories and then download files. In this process, I observed that number of open files are more than 400 (which is approxmately same as number of dir/subdir... (1 Reply)
Discussion started by: satyam90
1 Replies

6. Shell Programming and Scripting

not able to process files

hi i have a dir contains files i am reading ls $path | while read FILE do grep "a GFT buffer" $path/$FILE ...... ..... done it is able to process the fist file, from secomd onwards it is giving the following error, 2nd file name is "sd_gfts_GFT1_TCPGFT.trc.4" grep:... (1 Reply)
Discussion started by: Satyak
1 Replies

7. Solaris

files asscciated with a particular process

if i am finding out that a process has a more cpu usage before killing the process what are the commands to find out the files which are used by the process. can anyone help this thanks (1 Reply)
Discussion started by: sureshlee_83
1 Replies

8. UNIX for Dummies Questions & Answers

How do I Process Multiple Files

Hi, First post here. In Mac OSX terminal I need to run a program against multiple files in a directory and append the output to tab separated variable file. I currently type the following to process just one file MBP:/Users/dc1743/Desktop/SFT root# ./myprogram myfile1.plist >>... (1 Reply)
Discussion started by: dc1743
1 Replies

9. Shell Programming and Scripting

How to process the files using .tar.gz files in script

Hi, I have some file in /users/xyz directoty with .tar.gz extension. i need to find these files and if found in need to run other commands. I now the command for finding files,but how to put if condition ?please help me Thanks (3 Replies)
Discussion started by: bmkreddy
3 Replies

10. HP-UX

How to get the opened files by a process

Hi, i'd like to get the files which are opened by a process during his execution using the c language. Thanks for helping me. (2 Replies)
Discussion started by: benoit.gaillard
2 Replies
Login or Register to Ask a Question