Help with Archiving multiple files based on name and date


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with Archiving multiple files based on name and date
# 1  
Old 08-01-2011
Help with Archiving multiple files based on name and date

Dear Gurus,

I am a novice in shell scripts. I have a requirement where I need to move files every day from Current Folder to Archive folder.

Daily I will be receiving 5 files in the folder - /opt/data/feeds/.
The feeds folder has two sub-folders - Current and Archive.

For example the first day, the I receive file and names will be like:
Code:
File1_extract_08012011
File2_extract_08012011
File3_extract_08012011
File4_extract_08012011
File5_extract_08012011

The last 8 characters are the mmddyyyy.

Very first time Current folder will be empty so I just move the files from
/opt/data/feeds/ to /opt/data/feeds/Current.

Again the second day, I will be receiving the following files:
Code:
File1_extract_08022011
File2_extract_08022011
File3_extract_08022011
File4_extract_08022011
File5_extract_08022011

When I receive the files, I should the move the existing files from Current to Archive folder and then place the new files in the "Current" folder.

Also there is another condition where say for example on the third day, I receive only 3 out of 5.
Code:
File1_extract_08032011
File3_extract_08032011
File5_extract_08032011

In this case, the current folder should keep the file2 and file4 (which is from 08022011) and file1,3,5 (which is from 08032011). The file1,3,5(of 08022011) should be moved to Archive. The current folder should always have the most recent date for each file.

Also the archive folder should have only the last 6 days for each file.

So every day the current folder will have 5 files and Archive folder will have 30 files.

Please help me on this.

Thanks in advance.
Shankar

Last edited by radoulov; 08-01-2011 at 05:54 PM.. Reason: Code tags.
# 2  
Old 08-01-2011
Usually, the trick is to put the files all in the archive, and as each comes in, link it to current without a date in the name, after removing any link from before.

How do you know the file is fully written and can be used? Discovery seems silly, when the producer could just install it.
# 3  
Old 08-01-2011
These files are actually report outputs with .CSV extension. They are moved to this folder via application.

I am using something like below:


Code:
CURR_DIRECTORY=/opt/data/feeds/Current
ARCH_DIRECTORY=/opt/data/feeds/Archive

ls -1 *.xlsx > all_archive_files.txt
archive_list=${CURR_DIRECTORY}/all_archive_files.txt
echo archive_list: $archive_list
for archive_file in `cat $archive_list`
do
echo archive_file:$archive_file
echo 
cd ${ARCH_DIRECTORY}

if [ -f $archive_file ]; then
echo "This filename [$archive_file] exists"
echo "Move Unsuccessful :-("
else
echo "The filename [$archive_file] does not exist"
mv -f ${CURR_DIRECTORY}/$archive_file "${ARCH_DIRECTORY}"
echo "Move Successful :-)"
fi 
done


The above script just checks whether the file in Current directory is different from Archive directory and then does the move.
Daily you get files and the last 8characters of the file name will be current system date.
I am struggling to check the file names with the file saved date and then archiving for 6 days.

Any help with the script is highly appreciated.

Thanks

Last edited by radoulov; 08-01-2011 at 05:54 PM.. Reason: Code tags!
# 4  
Old 08-01-2011
Well, if you list file names stripped of date and detect duplicates, you know who needs moving down. Don't overwork making file lists, env var and pipes are fine:
Code:
cd ${CURR_DIRECTORY}
to_move=$(
 ls | sed 's/[0-9]*$//' | sort | uniq -d | while read p
  do
    ls -tr $p* | sed '$d'
  done
 )
if [ "$to_move" != "" ]
then
 mv $to_move ${ARCHIVE_DIR}
fi

# 5  
Old 08-01-2011
See if this works for you:
Code:
#!/usr/bin/ksh

mNew='/opt/data/feeds/'
mCurrent='/opt/data/feeds/Current/'
mArchive='/opt/data/feeds/Archive/'

#
# Removing 5 files from 5 days ago in Archive:
#
typeset -i mCnt=31
ls -1at ${mArchive}File?_extract_* | while read mFName; do
  mCnt=${mCnt}-1
  if [[ ${mCnt} -le 5 ]]; then
    echo "Now removing <$mFName>."
    rm -f ${mFName}
    if [[ ${mCnt} -eq 1 ]]; then
      break
    fi
  fi
done

#
# Moving existing Current files to Archive:
#
mv ${mCurrent}File?_extract_* ${mArchive}

#
# Moving new files to Current and
# make sure all 5 files are there:
#
mMMDDYYYY=$(date +"%m%d%Y")
mCnt=1
while [[ ${mCnt} -le 5 ]]; do
  mFName='File'${mCnt}'_extract_'${mMMDDYYYY}
  touch ${mCurrent}${mFName}
  mv ${mNew}${mFName} ${mCurrent}${mFName}
  mCnt=${mCnt}+1
done

# 6  
Old 08-02-2011
Hi Shell_life,

I am still not getting the script to work. The issue I notice is the file name.
The names I gave was an example.
If the file name format changes then the script won't work.
The only non changing criteria in the file name is that it always have the date(mmddyyyy) at the end. It can have any name/length as prefix.
For example: The files can be
File1_extract_08012011
Open_inv_08012011
BAXT_CONV_08012011

Please help.
# 7  
Old 08-02-2011
You can do it going through a sorted list with history in variables. If you hit a later file, you move the last file to archive. Here, * is a sorted list, and the ksh does it all but the mv internally, until 2040:
Code:
#!/usr/bin/ksh
 
cd $CURRENT_DIR
 
for file in *
do
 file_base=${file%_[01][0-9][0-3][0-9]20[0-3][0-9]}
 
 if [ $file_base = "$last_file_base" ]
 then
  mv last_file $ARCHIVE_DIR # one at a time for simplicity
 fi
 
 last_file=$file  last_file_base=$file_base
done

Really, polling sucks! The creator app should do this, too, after a good file create.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

UNIX script to replace old date with current date dynamically in multiple files present in a folder

I am trying to work on a script where it is a *(star) delimited file has a multiple lines starts with RTG and 3rd column=TD8 I want to substring the date part and I want to replace with currentdate minus 15 days. Here is an example. iam using AIX server $ cat temp.txt RTG*888*TD8*20180201~... (1 Reply)
Discussion started by: Shankar455
1 Replies

2. UNIX for Dummies Questions & Answers

Find the count of files by last created date based on the given date range

My unix version is IBM AIX Version 6.1 I tried google my requirement and found the below answer, find . -newermt 2012-06-15 08:13" ! -newermt 2012-06-15 18:20" But newer command is not working in AIX version 6.1 unix I have given my requirement below: Input: atr files: ... (1 Reply)
Discussion started by: yuvaa27
1 Replies

3. Shell Programming and Scripting

Divide an EBCDIC files into multiple files based on value at 45-46 bytes

Hi All, I do have an EBCDIC file sent from the z/os , this file has records with different record types in it, the type of record is identified by bytes 45-46 like value 12 has employee record value 14 has salaray record and etc.... we do now want to split the big ebcdic file into multiple... (3 Replies)
Discussion started by: okkadu
3 Replies

4. Shell Programming and Scripting

Archiving assistance needed date to month

We have year folder say in a path /opt/informat/Archive a folder 2012. And in the same folder /opt/informat/Archive we have different folders month based, like 201210, 201211, 201212, this have data for each month, ie files. Now time to time i need to move the monthly folders to the main folder... (1 Reply)
Discussion started by: raghavraok
1 Replies

5. UNIX for Dummies Questions & Answers

Remove files based on date

Hello team, I have a number of files in a folder which are dated yesterday and today.Can i remove all the files which i created today based on date?? is there any syntax for this ?? (1 Reply)
Discussion started by: kanakaraju
1 Replies

6. Shell Programming and Scripting

Copy files based on date

Hi all i am having so many files in my directory.Is there any option to copy files based on date. example i am having file this -rw-rw-r-- 1 ram user 1 Feb 2 17:12 abc -rw-rw-r-- 1 ram user 1 Feb 2 17:12 bnw -rwxrwxr-x 1 ram user 21122 Feb 4... (3 Replies)
Discussion started by: suryanarayana
3 Replies

7. Shell Programming and Scripting

Need script to select multiple files from archive directory based on the date range

hi all, here is the description to my problem. input parameters: $date1 & $date2 based on the range i need to select the archived files from the archived directory and moved them in to working directory. can u please help me in writing the code to select the multiple files based on the... (3 Replies)
Discussion started by: bbc17484
3 Replies

8. Shell Programming and Scripting

Get the newest files based on date

Hello friends, I'm learning to script, and I need help. How can I get the latest/newest files based on date? the format is as following: Feb 07 19:25 TESTPWD_file_1vk6pn40_19519_1 Feb 07 19:46 TESTPWD_file_1uk6pn40_19518_2 Feb 07 19:47 TESTPWD_file_20k6pn40_19520_2 Feb 07 19:56... (5 Replies)
Discussion started by: Beginer0705
5 Replies

9. Shell Programming and Scripting

Count of files based on date?

Hi Friends, Can anyone help me with this: To get the count of files that are existing in a directory created on a perticular date like in the example (01/08) .(having same pattern for the filename) ex: FileName Creted Date FILE001 01/08/2007 FILE005 ... (6 Replies)
Discussion started by: sbasetty
6 Replies

10. UNIX for Dummies Questions & Answers

Remove files based on date

I am trying to write a shell script that will remove files in a directory based on the date. For instance, remove all files older than yesterday. Any ideas? (4 Replies)
Discussion started by: hshapiro
4 Replies
Login or Register to Ask a Question