Visit Our UNIX and Linux User Community


Find Unread Files

 
Thread Tools Search this Thread
Homework and Emergencies Emergency UNIX and Linux Support Find Unread Files
# 1  
Old 09-30-2013
Find Unread Files

Hi

I have requirement to read only unread files from the directory and load into database.

Scenario: I do receive bunch of files into my unix directory for every 15 mins. My ETL Process every once in a day and reads the files and loads into db table. I cannot move these file into different location after extraction as source system will ftp for previous 15 days so i receive the files again if there is no file. So i will have to keep files for 15 days atleast.

Could you please advise how we can write the script to read only unread files?
# 2  
Old 09-30-2013
There are two options.

1. Rename the file after it is loaded to the db with a suffix, say .processed. Use this suffix to identify already read file.

2. Create a list file which would contain names of files that are processed (i.e., loaded to db). When you load the file to db, make an entry in this list file with the name of the file loaded.
Now, use this list file to identify read and unread files.
# 3  
Old 09-30-2013
Thanks Krish

1st option may not be possible because if i rename the file, I will receive the same file again from sourece as they do ftp for 15 days of files.

2nd Option: i don't understand how do we identify processed and not processed files(code difficulty as i am not very good at Unix).

I've got one approach:
First i create file(FILE.ALL) with all the files
Second i find the difference between FILE.ALL and FILE.BACKUP and write into FILE.LIST
After processing all the files in FILE.LIST, i will append the list from FILE.LIST into FILE.BACKUP file.
seems so far so good, Now i need to remove the filenames from FILE.BACKUP which are older than 15 days.

Filename has date in it Eg:Filename_093013_xxxx.csv.

Could some one advise is this the good approach? and how do we remove the file name from FILE.BACKUP file by comparing dates?
# 4  
Old 09-30-2013
Empty the processed files (maybe backup somewhere else before), so the names persist and keep ftp from copying them again.
# 5  
Old 09-30-2013
Thanks Rudi,

I don't want to receive the files again. that is the reason i do keep all the processed files in the same directory location. I do house keeping activity on which received before 15 days.
# 6  
Old 10-01-2013
If I understand you correctly you have a Unix box to which another node is ftp'ing files regularly. You have a process on this Unix box which needs to read these files but not the one's already processed. I assume that you can process these files in the chronological order that they are received in????? If so, here's another option......

At the end of your processing job you put the command:

Code:
 
date > timestamp

to create a file at the time of the process run called "timestamp".

At the start of the job you put:

Code:
 
find * -newer timestamp <other switches, whatever>

to only select files created (ftp'd onto the box) since the last run finished.

That way, all the historical files can be left in the directory and not be selected for processing.

The above assumes that I have completely understood you but, if not, do post back the issues.

Last edited by hicksd8; 10-01-2013 at 08:27 AM..
This User Gave Thanks to hicksd8 For This Post:
# 7  
Old 01-07-2014
Each file has three time stamps :

[a] access (read the file's contents) -atime
[b] change the status (modify the file or its attributes) -ctime
[c] modify (change the file's contents) -mtime
If your file are read-only, you can comapre the atime and the mtime

Previous Thread | Next Thread
Test Your Knowledge in Computers #979
Difficulty: Medium
macOS is based on the Unix operating system and on technologies developed between 1985 and 1997 at NeXT.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

find -ctime -1 cannot find files without extention

The problem is this one. I tar and gzip files on remote server find . -ctime -1 | tar -cvf transfer_dmz_start_daily.tar *${Today}*.*; Command find . -ctime -1 Doesn't find files without extension .csv .txt I have to collect all files for current day, when the program... (1 Reply)
Discussion started by: digioleg54
1 Replies

2. Shell Programming and Scripting

Find command to find a word from list of files

I need to find a word '% Retail by State' in the folder /usr/sas/reports/RetailSalesTaxallocation. When I tried like below, -bash-4.1$ cd /usr/sas/reports/RetailSalesTaxallocation -bash-4.1$ find ./ -name % Retail by State find: paths must precede expression: Retail Usage: find ... (10 Replies)
Discussion started by: Ram Kumar_BE
10 Replies

3. UNIX for Dummies Questions & Answers

find Search - Find files not matching a pattern

Hello all, this is my first and probably not my last question around here. I do hope you can help or at least point me in the right direction. My question is as follows, I need to find files and possible folders which are not owner = AAA group = BBB with a said location and all sub folders ... (7 Replies)
Discussion started by: kilobyter
7 Replies

4. UNIX for Dummies Questions & Answers

How to see unread e-mails only in Alpine?

Does anyone have any idea how to see only unread (new) e-mails in the Alpine client when using IMAP? I finally have a fast IMAP client, but don't want to go over all the e-mails I've already read through other clients... Thanks in advance for any hints. ---------- Post updated at 01:21 PM... (0 Replies)
Discussion started by: JamesR404
0 Replies

5. UNIX for Dummies Questions & Answers

Pine continuously marks old messages as unread

Hi, I have been having problem with pine for the past few weeks. I use email clinet Thunderbird to view my emails. Every time I open the thunderbird, all my emails were marked as unread. So, I logged into our email server to see what's wrong. even when I opened pine, all messages are labeled as... (0 Replies)
Discussion started by: veepine
0 Replies

6. Shell Programming and Scripting

what is the find to command to find the files created last 30 days

what is the find to command to find the files created last 30 days (5 Replies)
Discussion started by: rajkumar_g
5 Replies

7. Shell Programming and Scripting

perl (conky) and gmail/IMAP unread message count

Hi all, I use Conky monitor (Conky - Home) for my laptop and I needed a script to see the count of new messages on gmail/IMAP, then I made this small perl script (I hope they can be useful to someone :)) gimap.pl #!/usr/bin/perl # gimap.pl by gxmsgx # description: get the count of unread... (0 Replies)
Discussion started by: gxmsgx
0 Replies

8. Shell Programming and Scripting

Little bit weired : Find files in UNIX w/o using find or where command

Yes , I have to find a file in unix without using any find or where commands.Any pointers for the same would be very helpful as i am beginner in shell scritping and need a solution for the same. Thanks in advance. Regards Jatin Jain (10 Replies)
Discussion started by: jatin.jain
10 Replies

9. Shell Programming and Scripting

Find files older than 20 days & not use find

I need to find files that have the ending of .out and that are older than 20 days. However, I cannot use find as I do not want to search in the directories that are underneath the directory that I am searching in. How can this be done?? Find returns files that I do not want. (2 Replies)
Discussion started by: halo98
2 Replies

10. UNIX for Dummies Questions & Answers

Mark messages as unread

Hi Does anyone know how to mark messages as unread either in Pine or from a Terminal or some such ? Thanks, James (1 Reply)
Discussion started by: Rylann
1 Replies

Featured Tech Videos