Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Help with Archiving multiple files based on name and date

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with Archiving multiple files based on name and date
# 22  
Old 08-03-2011
Do we worry about x_01012011.txt and x_01012011.csv both being present?

Now file_base contains the .ext as well as the prefix:
Code:
#!/usr/bin/ksh
 
cd $CURRENT_DIR
 
ls *_[01][0-9][0-3][0-9]20[0-3][0-9].* | sed '
  s/\(.*\)_\([01][0-9][0-3][0-9]\)20\([0-3][0-9]\)\(\..*\)/\3\2 \1\4 &/
 ' | sort | while read xxkey file_base iext file
do
 if [ $file_base = "$last_file_base" ]
 then
  mv last_file $ARCHIVE_DIR # one at a time for simplicity
  while (( 6 < $(
               ls $ARCHIVE_DIR/${file_base%.*}_[01][0-9][0-3][0-9]20[0-3][0-9].${file_base##*.} 2>/dev/null | wc -l
               ) ))
  do
    ls $ARCHIVE_DIR/${file_base%.*}_[01][0-9][0-3][0-9]20[0-3][0-9].${file_base##*.} | sed '
      s/.*_\([01][0-9][0-3][0-9]\)20\([0-3][0-9]\)\..*/\2\1 &/
     ' | sort | read y x
    rm -f $x
  done
 fi
 
 last_file=$file  last_file_base=$file_base
done

# 23  
Old 08-04-2011
DGPickett:

I am using the following code where I just added additional "echo statements".
There are two files inside the CURRENT directory - BAXT_INV_08032011.csv and BAXT_INV_08042011.csv. The ARCHIVE has BAXT_INV_08012011.csv.
Code:
#!/usr/bin/ksh

ARCHIVE_DIR=/opt/data/Archive
CURRENT_DIR=/opt/data/Current


cd $CURRENT_DIR
 
ls *_[01][0-9][0-3][0-9]20[0-3][0-9].* | sed '
  s/\(.*\)_\([01][0-9][0-3][0-9]\)20\([0-3][0-9]\)\(\..*\)/\3\2 \1\4 &/
 ' | sort | while read xxkey file_base iext file
do
 if [ $file_base = "$last_file_base" ]
 echo file_base: $file_base
echo last_file_base: $last_file_base
 then
  mv last_file $ARCHIVE_DIR # one at a time for simplicity
  while (( 6 < $(
               ls $ARCHIVE_DIR/${file_base%.*}_[01][0-9][0-3][0-9]20[0-3][0-9].${file_base##*.} 2>/dev/null | wc -l
               ) ))
  do
    ls $ARCHIVE_DIR/${file_base%.*}_[01][0-9][0-3][0-9]20[0-3][0-9].${file_base##*.} | sed '
      s/.*_\([01][0-9][0-3][0-9]\)20\([0-3][0-9]\)\..*/\2\1 &/
     ' | sort | read y x
    rm -f $x
  done
 fi
 
 last_file=$file  last_file_base=$file_base
 echo last_file: $last_file
 echo file: $file
 echo last_file: $file_base
done

I am getting the following when I execute the script:
Code:
file_base: BAXT_INV.csv
last_file_base:
mv: cannot access last_file
last_file:
file:
last_file: BAXT_INV.csv
file_base: BAXT_INV.csv
last_file_base: BAXT_INV.csv
mv: cannot access last_file
last_file:
file:
last_file: BAXT_INV.csv

I script should check keep BAXT_INV_08042011.csv in CURRENT and move the BAXT_INV_08032011.csv to ARCHIVE.
Let me know if I am missing anything in the script.
Also will it be a good idea to always keep the new files in a separate directory( say temp feed) outside the current and archive folder. Then compare whether we have a matching file with old MMDDYYYY in Current - if yes then move the CURRENT file to ARCHIVE and then move the new file from temp feed to Current.
Please advice.

Thanks
Last edited by Franklin52; 08-06-2011 at 04:46 PM.. Reason: Please use code tags for data and code samples, thank you
# 24  
Old 08-04-2011
We call that requirements creep, but yes, it simplifes things tremendously for the first phase. Just remove ${file%%_[0-9]*}*${file##*.} !
# 25  
Old 08-05-2011
Thanks a bunch DGPickett!!!!!.

Everything works fine.
# 26  
Old 08-09-2011
One tool we all wish for is something to mv two files locking the directories in the middle, so there is never two or no files of a type, just one.

You can build the directory somewhere else on the drive, and mv it so it replaces and destroys the old direcory in one mv. This only works for lead directories, with no sub-directories, as you cannot hard link directories.

You can make a clone tree, hard link across all the non-dir inodes in the subtree, and then mv the clone tree to destroy the original and all its children.

Some sites install new file sets in clone trees, and just put a sym-link in to say which clone tree is the current active version. If space gets tight, just delete an old clone tree and get back some of the space.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

UNIX script to replace old date with current date dynamically in multiple files present in a folder

I am trying to work on a script where it is a *(star) delimited file has a multiple lines starts with RTG and 3rd column=TD8 I want to substring the date part and I want to replace with currentdate minus 15 days. Here is an example. iam using AIX server $ cat temp.txt RTG*888*TD8*20180201~... (1 Reply)
Discussion started by: Shankar455
1 Replies

2. UNIX for Dummies Questions & Answers

Find the count of files by last created date based on the given date range

My unix version is IBM AIX Version 6.1 I tried google my requirement and found the below answer, find . -newermt “2012-06-15 08:13" ! -newermt “2012-06-15 18:20" But newer command is not working in AIX version 6.1 unix I have given my requirement below: Input: atr files: ... (1 Reply)
Discussion started by: yuvaa27
1 Replies

3. Shell Programming and Scripting

Divide an EBCDIC files into multiple files based on value at 45-46 bytes

Hi All, I do have an EBCDIC file sent from the z/os , this file has records with different record types in it, the type of record is identified by bytes 45-46 like value 12 has employee record value 14 has salaray record and etc.... we do now want to split the big ebcdic file into multiple... (3 Replies)
Discussion started by: okkadu
3 Replies

4. Shell Programming and Scripting

Archiving assistance needed date to month

We have year folder say in a path /opt/informat/Archive a folder 2012. And in the same folder /opt/informat/Archive we have different folders month based, like 201210, 201211, 201212, this have data for each month, ie files. Now time to time i need to move the monthly folders to the main folder... (1 Reply)
Discussion started by: raghavraok
1 Replies

5. UNIX for Dummies Questions & Answers

Remove files based on date

Hello team, I have a number of files in a folder which are dated yesterday and today.Can i remove all the files which i created today based on date?? is there any syntax for this ?? (1 Reply)
Discussion started by: kanakaraju
1 Replies

6. Shell Programming and Scripting

Copy files based on date

Hi all i am having so many files in my directory.Is there any option to copy files based on date. example i am having file this -rw-rw-r-- 1 ram user 1 Feb 2 17:12 abc -rw-rw-r-- 1 ram user 1 Feb 2 17:12 bnw -rwxrwxr-x 1 ram user 21122 Feb 4... (3 Replies)
Discussion started by: suryanarayana
3 Replies

7. Shell Programming and Scripting

Need script to select multiple files from archive directory based on the date range

hi all, here is the description to my problem. input parameters: $date1 & $date2 based on the range i need to select the archived files from the archived directory and moved them in to working directory. can u please help me in writing the code to select the multiple files based on the... (3 Replies)
Discussion started by: bbc17484
3 Replies

8. Shell Programming and Scripting

Get the newest files based on date

Hello friends, I'm learning to script, and I need help. How can I get the latest/newest files based on date? the format is as following: Feb 07 19:25 TESTPWD_file_1vk6pn40_19519_1 Feb 07 19:46 TESTPWD_file_1uk6pn40_19518_2 Feb 07 19:47 TESTPWD_file_20k6pn40_19520_2 Feb 07 19:56... (5 Replies)
Discussion started by: Beginer0705
5 Replies

9. Shell Programming and Scripting

Count of files based on date?

Hi Friends, Can anyone help me with this: To get the count of files that are existing in a directory created on a perticular date like in the example (01/08) .(having same pattern for the filename) ex: FileName Creted Date FILE001 01/08/2007 FILE005 ... (6 Replies)
Discussion started by: sbasetty
6 Replies

10. UNIX for Dummies Questions & Answers

Remove files based on date

I am trying to write a shell script that will remove files in a directory based on the date. For instance, remove all files older than yesterday. Any ideas? (4 Replies)
Discussion started by: hshapiro
4 Replies
Login or Register to Ask a Question