Finding & Moving Oldest File by Parsing/Sorting Date Info in File Names


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Finding & Moving Oldest File by Parsing/Sorting Date Info in File Names
# 1  
Old 07-30-2008
Finding & Moving Oldest File by Parsing/Sorting Date Info in File Names

I'm trying to write a script that will look in an /exports folder for the oldest export file and move it to a /staging folder. "Oldest" in this case is actually determined by date information embedded in the file names themselves.

Also, the script should only move a file from /exports to /staging if staging is empty; if there's already a file in /staging then the script would exit without doing anything.

Here's an example of what the folders and files might look like:

/exports
job1_Friday_05302008.xml
job1_Monday_05262008.xml
job1_Monday_06022008.xml
job1_Thursday_05292008.xml
job1_Tuesday_06032008.xml
job1_Wednesday_05282008.xml

/staging

In this case, because /staging is empty, the script should move /exports/job1_Monday_05262008.xml to /staging. The next time the script runs it should exit if /exports still contains job1_Monday_05262008.xml, otherwise it should move job1_Wednesday_05282008.

Also, the "job1" prefix is static.

I've got a script that uses ls and cat to loop through file names and parse out the date information but i'm getting bogged down with how to handle the results and i'm starting to think i might be going down the wrong path...

Any suggestions would be appreciated
# 2  
Old 07-30-2008
Hammer & Screwdriver Here is a piece of it - I think anyway...

Assuming the file list is in a file already. (You could pipe the ls directly to the rest of the commands, if you were so inclined.)
Code:
> cat exp_contents 
job1_Friday_05302008.xml
job1_Monday_05262008.xml
job1_Monday_06022008.xml
job1_Thursday_05292008.xml
job1_Tuesday_06032008.xml
job1_Wednesday_05282008.xml

Now, try out the following command:
Code:
> cat exp_contents | tr "_" " " | awk '{print substr($3,5,4)"."substr($3,1,4)"|"$1"_"$2"_"$3}' | sort
2008.0526|job1_Monday_05262008.xml
2008.0528|job1_Wednesday_05282008.xml
2008.0529|job1_Thursday_05292008.xml
2008.0530|job1_Friday_05302008.xml
2008.0602|job1_Monday_06022008.xml
2008.0603|job1_Tuesday_06032008.xml

You can append to this a head -1 to get the top line.
You can also do a cut -d"|" -f2 to get the filename back for the oldest entry.

If this works, then the rest of the logic will flow much smoother.
# 3  
Old 07-30-2008
Look like a homework.
Please read https://www.unix.com/unix-dummies-que...om-forums.html before posting, especially 5 and 6.
# 4  
Old 07-30-2008
Clarification...

No, it's a real-world problem Smilie

It's just written out like a homework assignment because i've done enough technical work to know that when you ask for help, clarity is king.

I'm working on an interface that gets daily xml exports from an ftp site and synchronizes them with Siebel CRM. I've got Oracle Fusion Middleware to:

A) transfer files from the FTP site to an /exports folder

C) route files from a /staging folder to an Siebel inbound web service.

But I need a Step B) to meter out the appropriate files from /exports to /staging in the right order (first in, first out). I'm trying to do this with shell scripting but this is a bit outside my expertise, as you can tell... Smilie
# 5  
Old 07-30-2008
Hammer & Screwdriver Can you get thru what I posted previously?

From my earlier post, you should be able to determine the oldest file. Are you there yet?

You can set a variable such like:

Code:
old_file=$(cat exp_contents | tr "_" " " | awk '{print substr($3,5,4)"."substr($3,1,4)"|"$1"_"$2"_"$3}' | sort | head -1 | cut -d"|" -f2)
echo $old_file

If you are that far, then onto the 'easier' part of the requirements.
# 6  
Old 07-30-2008
Working Script

Yes, it's working perfectly!

Here's the full script i'm using in case it's helpful for anyone else.

It has an extra line at the top to purge files in an /archive folder that are > 7 days old.

And the folder names and locations are changed a bit from what they were in my original post.

But it seems to be working nicely.

Thanks again for the help!


Code:
#!/bin/ksh
find ../archive -name *.xml -type f -mtime +7 -exec rm {} \;
PROCESSING=$(ls ../processing | head -1)
if [[ -z $PROCESSING ]];then
   INCOMING=$(ls ../incoming/*.xml | cat | tr "_" " " | awk '{print substr($3,5,4)"."substr ($3,1,4)"|"$1"_"$2"_"$3}' | sort | head -1 | cut -d"|" -f2)
   if [[ -n $INCOMING ]];then
      mv $INCOMING ../processing
   fi
fi

# 7  
Old 07-30-2008
You can use the file time, if staging is empty move the oldest xml file from exports to staging.
Code:
[ "$(ls -A staging)" ] || mv /exports/$(ls -t /exports/*.xml | tail -1) /staging/

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

File Move & Sort by Name - Kick out Bad File Names & More

I have a dilemma, we have users who are copying files to "directory 1." These images have file names which include the year it was taken. I need to put together a script to do the following: Examine the file naming convention, ensuring it's the proper format (e.g. test-1983_filename-123.tif)... (8 Replies)
Discussion started by: Nvizn
8 Replies

2. Shell Programming and Scripting

Moving files only by oldest file one at a time

Hi I am want to create a script where the file gets moved from the current folder to a folder transfer based on the oldest first. This script should run one file at a time using a loop. I want it as a loop because I want to do some processing while I have one file. Can anyone guide me on this? (2 Replies)
Discussion started by: chamajid
2 Replies

3. Shell Programming and Scripting

sorting file names with hyphen and underscore

Hi, I have two types of file names filename1_12345 or filename1-12345 at the same time I have second type filename2-12345 in a txt file. I am trying to write awk script that will sort these names with below pattern ALL file names like filename1_12345 and filename1-12345 will go to... (1 Reply)
Discussion started by: rider29
1 Replies

4. Shell Programming and Scripting

Sorting file based on names

Hi I have some files in directory and the names of files are like jnhld_15233_2010-11-23 jnhld_15233_2007-10-01 jnhld_15233_2001-05-04 jnhld_15233_2011-11-11 jnhld_15233_2005-06-07 jnhld_15233_2000-04-01 ..etc How can i sort these files based on the date in the file name so that ... (4 Replies)
Discussion started by: morbid_angel
4 Replies

5. Shell Programming and Scripting

Sorting and moving file sequence with gaps

Hello, I have lots of sequentially numbered files which make up an image sequence. I'm trying to do two things with it: #1: Find gaps in the sequence and move each range of sequencial files into their own subfolder. #2: Designate a starting point (file) and move every 24th file into... (4 Replies)
Discussion started by: ex_H
4 Replies

6. Shell Programming and Scripting

Parsing Log File Based on Date & Error

I'm still up trying to figure this out and it is driving me nuts. I have a log file which has a basic format of this... 2010-10-10 22:25:42 Init block 'UA Deployment Date': Dynamic refresh of repository scope variables has failed. The ODBC function has returned an error. The database... (4 Replies)
Discussion started by: k1ko
4 Replies

7. Shell Programming and Scripting

Need help in finding filesize , moving file , gzipping file

Hi , Please help with the following questions 1) how can i find size of a file ? i have written du -k $flname > s1 . Is this right ? Any other better suggeastions ? 2) how do I use mv command for moving the file ? I need the syntax with some examples 3) Command for printing the total... (1 Reply)
Discussion started by: Learning!
1 Replies

8. UNIX and Linux Applications

Finding the oldest file in a directory without ls

I am trying to determine the oldest and most recent files in a huge directory. I am using an ls -tr statement outside my find statement. The directory is too big and I am getting an "arg list too long" error. Is there something I can put in my find statement that doesn't create a list to... (2 Replies)
Discussion started by: hiyofjord
2 Replies

9. Shell Programming and Scripting

Finding the oldest file in a particular directory

Hi all, I am a newbie to scripting and I need your help regarding finding the oldest file in a particular directory. My intention is to remove that oldest file. Are there any options available with the "find" command to do this.. Thanks in advance for your help Pavan (4 Replies)
Discussion started by: pavan_movva
4 Replies

10. Shell Programming and Scripting

Finding the oldest file

Hi:- I need help with a script I need to modify: - what's the best/easiest way to find out the oldest file in a directory and then move this file to another directory? Thanks, (5 Replies)
Discussion started by: janet
5 Replies
Login or Register to Ask a Question