Emergency UNIX and Linux Support

Hi

I have requirement to read only unread files from the directory and load into database.

Scenario: I do receive bunch of files into my unix directory for every 15 mins. My ETL Process every once in a day and reads the files and loads into db table. I cannot move these file into different location after extraction as source system will ftp for previous 15 days so i receive the files again if there is no file. So i will have to keep files for 15 days atleast.

Could you please advise how we can write the script to read only unread files?

There are two options.

1. Rename the file after it is loaded to the db with a suffix, say .processed. Use this suffix to identify already read file.

2. Create a list file which would contain names of files that are processed (i.e., loaded to db). When you load the file to db, make an entry in this list file with the name of the file loaded.
Now, use this list file to identify read and unread files.

Thanks Krish

1st option may not be possible because if i rename the file, I will receive the same file again from sourece as they do ftp for 15 days of files.

2nd Option: i don't understand how do we identify processed and not processed files(code difficulty as i am not very good at Unix).

I've got one approach:
First i create file(FILE.ALL) with all the files
Second i find the difference between FILE.ALL and FILE.BACKUP and write into FILE.LIST
After processing all the files in FILE.LIST, i will append the list from FILE.LIST into FILE.BACKUP file.
seems so far so good, Now i need to remove the filenames from FILE.BACKUP which are older than 15 days.

Filename has date in it Eg:Filename_093013_xxxx.csv.

Could some one advise is this the good approach? and how do we remove the file name from FILE.BACKUP file by comparing dates?

Empty the processed files (maybe backup somewhere else before), so the names persist and keep ftp from copying them again.

Thanks Rudi,

I don't want to receive the files again. that is the reason i do keep all the processed files in the same directory location. I do house keeping activity on which received before 15 days.

If I understand you correctly you have a Unix box to which another node is ftp'ing files regularly. You have a process on this Unix box which needs to read these files but not the one's already processed. I assume that you can process these files in the chronological order that they are received in????? If so, here's another option......

At the end of your processing job you put the command:

Code:

 
date > timestamp

to create a file at the time of the process run called "timestamp".

At the start of the job you put:

Code:

 
find * -newer timestamp <other switches, whatever>

to only select files created (ftp'd onto the box) since the last run finished.

That way, all the historical files can be left in the directory and not be selected for processing.

The above assumes that I have completely understood you but, if not, do post back the issues.

Each file has three time stamps :

[a] access (read the file's contents) -atime
[b] change the status (modify the file or its attributes) -ctime
[c] modify (change the file's contents) -mtime
If your file are read-only, you can comapre the atime and the mtime