Help with Archiving multiple files based on name and date


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with Archiving multiple files based on name and date
# 8  
Old 08-03-2011
The code still not working. May be I am not getting it to work as expected. I will try tweaking the code.

If you can throw some more light with comments and any new code will be helpful.

Thanks
# 9  
Old 08-03-2011
I sense the requirements are still blurry. The files come into current, and then you want to move the prior same-prefix to archive, but if there is already that name in archive, what do you want to do?
# 10  
Old 08-03-2011
I provided a solution based on the original post and it does work as per original requirement.

Then the requirement was changed and it is still not clear.

For instance, are these the only files in the specified directories?

If so, then simply change any references from "File?_extract_*" to "*".

Or if there are other files there and they do not end with a date, then change the file name to "*[01][0-9][0-3][0-9]20[1-9][0-9]".
# 11  
Old 08-03-2011
Dear GURUS,

First of all I am thankful for being patient with me. Let me explain in more detail:

Current will always have the filename with the latest "MMDDYYYY".
Archive will be having the last 6 MMDDYYYY for each of the filenames.

Whenever a file comes with a new MMDDYYYY then only you move the old file from CURRENT to ARCHIVE else you leave the old file in the CURRENT itself.

Say on August 1 is the first day. Both Current and Archive will be empty.
You are getting three files:
File1_extract_08012011.csv
BAXT_INV_08012011.csv
HMIA_CLS_08012011.csv

We will copy all the 3 file to Current folder.

Then next day Aug2, you are getting the following 3 files:
File1_extract_08022011.csv
BAXT_INV_08022011.csv
HMIA_CLS_08022011.csv

Now all the three files with 08012011 will be moved to ARCHIVE and the CURRENT will the 08022011 files.

Now on Aug3, we get only two files:
File1_extract_08032011.csv
BAXT_INV_08032011.csv

Now we should move only File1_extract_08022011.csv and BAXT_INV_08022011.csv to ARCHIVE and keep the HMIA_CLS_08022011.csv in the CURRENt folder. The latest 08032011 files will also be copied to CURRENT.

As of Aug3, ARCHIVE folder will have:

File1_extract_08012011.csv
BAXT_INV_08012011.csv
HMIA_CLS_08012011.csv
File1_extract_08022011.csv
BAXT_INV_08022011.csv

CURRENT folder will have:
File1_extract_08032011.csv
BAXT_INV_08032011.csv
HMIA_CLS_08022011.csv


This continues and at any point of time in future the ARCHIVE should have only the last 6 MMDDYYYY's.

Hope this gives a clear picture of what I am trying to do.

Thanks
# 12  
Old 08-03-2011
So, a file to be moved to archive does not need to worry about overwrite.

Do you want to integrate the archive 6 file limit at the same time?
Code:
#!/usr/bin/ksh
 
cd $CURRENT_DIR
 
for file in *
do
 file_base=${file%_[01][0-9][0-3][0-9]20[0-3][0-9]}
 
 if [ $file_base = "$last_file_base" ]
 then
  mv last_file $ARCHIVE_DIR # one at a time for simplicity
  while (( 6 < $( ls $ARCHIVE_DIR/${file_base}_[01][0-9][0-3][0-9]20[0-3][0-9] 2>/dev/null | wc -l ) ))
  do
    ls -tr $ARCHIVE_DIR/${file_base}_[01][0-9][0-3][0-9]20[0-3][0-9] | read x
    rm -f $x
  done
 fi
 
 last_file=$file  last_file_base=$file_base
done

# 13  
Old 08-03-2011
Yes Please that will be great.

Also if possible if you an include comments that will be informative for a beginner like me.

For example if you can add some comments here:
Code:
while (( 6 < $( ls $ARCHIVE_DIR/${file_base}_[01][0-9][0-3][0-9]20[0-3][0-9] 2>/dev/null | wc -l ) ))


Rather than blindly copying the code, I would like to learn and use it.

Thanks

Last edited by Franklin52; 08-06-2011 at 04:45 PM.. Reason: Please use code tags for data and code samples, thank you
# 14  
Old 08-03-2011
Well,
  1. go to the current dir so names have no dir,
  2. Get the sorted list of visible files and go through them one at a time in order as file,
  3. Strip off the date portion at the end, with underscore, as file_base,
  4. If the stored last file base is the same (blank first time never equal), then the last file is younger. Actually, it's be nicer if the suffix was YYYY-MM-DD, as this fails at new year.
Try again:
Code:
#!/usr/bin/ksh
 
cd $CURRENT_DIR
 
ls *_[01][0-9][0-3][0-9]20[0-3][0-9] | sed '
  s/\(.*\)_\([01][0-9][0-3][0-9]\)20\([0-3][0-9]\)/\3\2 \1 &/
 ' | sort | while read xxkey file_base file
do
 if [ $file_base = "$last_file_base" ]
 then
  mv last_file $ARCHIVE_DIR # one at a time for simplicity
  while (( 6 < $(
                     ls $ARCHIVE_DIR/${file_base}_[01][0-9][0-3][0-9]20[0-3][0-9] 2>/dev/null | wc -l
                     ) ))
  do
    ls -tr $ARCHIVE_DIR/${file_base}_[01][0-9][0-3][0-9]20[0-3][0-9] | read x
    rm -f $x
  done
 fi
 
 last_file=$file  last_file_base=$file_base
done

  1. Go to the current dir so file names have no dir prefix,
  2. list just the well named files to pipe,
  3. prefix them with the key field YYMMDD on pipe, sticking the base in the stream using sed for simplicity,
  4. sort them by date and then prefix pipe to pipe,
  5. 'while read' puts the three fields into three variables for each line from stdin pipe until EOF.
  6. If same base as last file, last file must be moved.
  7. If file is moved, in a subshell that captures stdout as a string $(...), list the archive dir for that base and date wild card suffix and count the lines of the list,
  8. (( )) is ksh arithmetic mode, so you can say 6 < for testing the line count, # I put 6 first, > 6, as 6 is smaller tha $(...), gets lost at end.
  9. while 6 is less than that line count,
  10. list by file mod time oldest first (is mod time a safe test, or do we need a key rearrange and sort like above?),
  11. read the first name,
  12. remove that name,
  13. end while 6 loop with done,
  14. end if bases are same test with fi,
  15. save file name and prefix in last_* variables for next pass,
  16. end while read file loop with done

Last edited by DGPickett; 08-03-2011 at 02:34 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

UNIX script to replace old date with current date dynamically in multiple files present in a folder

I am trying to work on a script where it is a *(star) delimited file has a multiple lines starts with RTG and 3rd column=TD8 I want to substring the date part and I want to replace with currentdate minus 15 days. Here is an example. iam using AIX server $ cat temp.txt RTG*888*TD8*20180201~... (1 Reply)
Discussion started by: Shankar455
1 Replies

2. UNIX for Dummies Questions & Answers

Find the count of files by last created date based on the given date range

My unix version is IBM AIX Version 6.1 I tried google my requirement and found the below answer, find . -newermt 2012-06-15 08:13" ! -newermt 2012-06-15 18:20" But newer command is not working in AIX version 6.1 unix I have given my requirement below: Input: atr files: ... (1 Reply)
Discussion started by: yuvaa27
1 Replies

3. Shell Programming and Scripting

Divide an EBCDIC files into multiple files based on value at 45-46 bytes

Hi All, I do have an EBCDIC file sent from the z/os , this file has records with different record types in it, the type of record is identified by bytes 45-46 like value 12 has employee record value 14 has salaray record and etc.... we do now want to split the big ebcdic file into multiple... (3 Replies)
Discussion started by: okkadu
3 Replies

4. Shell Programming and Scripting

Archiving assistance needed date to month

We have year folder say in a path /opt/informat/Archive a folder 2012. And in the same folder /opt/informat/Archive we have different folders month based, like 201210, 201211, 201212, this have data for each month, ie files. Now time to time i need to move the monthly folders to the main folder... (1 Reply)
Discussion started by: raghavraok
1 Replies

5. UNIX for Dummies Questions & Answers

Remove files based on date

Hello team, I have a number of files in a folder which are dated yesterday and today.Can i remove all the files which i created today based on date?? is there any syntax for this ?? (1 Reply)
Discussion started by: kanakaraju
1 Replies

6. Shell Programming and Scripting

Copy files based on date

Hi all i am having so many files in my directory.Is there any option to copy files based on date. example i am having file this -rw-rw-r-- 1 ram user 1 Feb 2 17:12 abc -rw-rw-r-- 1 ram user 1 Feb 2 17:12 bnw -rwxrwxr-x 1 ram user 21122 Feb 4... (3 Replies)
Discussion started by: suryanarayana
3 Replies

7. Shell Programming and Scripting

Need script to select multiple files from archive directory based on the date range

hi all, here is the description to my problem. input parameters: $date1 & $date2 based on the range i need to select the archived files from the archived directory and moved them in to working directory. can u please help me in writing the code to select the multiple files based on the... (3 Replies)
Discussion started by: bbc17484
3 Replies

8. Shell Programming and Scripting

Get the newest files based on date

Hello friends, I'm learning to script, and I need help. How can I get the latest/newest files based on date? the format is as following: Feb 07 19:25 TESTPWD_file_1vk6pn40_19519_1 Feb 07 19:46 TESTPWD_file_1uk6pn40_19518_2 Feb 07 19:47 TESTPWD_file_20k6pn40_19520_2 Feb 07 19:56... (5 Replies)
Discussion started by: Beginer0705
5 Replies

9. Shell Programming and Scripting

Count of files based on date?

Hi Friends, Can anyone help me with this: To get the count of files that are existing in a directory created on a perticular date like in the example (01/08) .(having same pattern for the filename) ex: FileName Creted Date FILE001 01/08/2007 FILE005 ... (6 Replies)
Discussion started by: sbasetty
6 Replies

10. UNIX for Dummies Questions & Answers

Remove files based on date

I am trying to write a shell script that will remove files in a directory based on the date. For instance, remove all files older than yesterday. Any ideas? (4 Replies)
Discussion started by: hshapiro
4 Replies
Login or Register to Ask a Question