shell script preserving last 2 versions of files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting shell script preserving last 2 versions of files
# 1  
Old 10-28-2008
shell script preserving last 2 versions of files

I need some help with the logic and syntax for a shell script (ksh) that will search a directory and look for similar files and save only the last two versions. The version number is in the file name. However, the files are of varying name lengths and may have 1 or many files, with no limit to the number of files. I am not sure that using the find command for date/timestamp is a good idea because these are adhoc files that get created.

For example:
Directory may have files like below

apps_V01.xml
betarelease_V01.xml
betarelease_V02.xml
betarelease_V03.xml
test_V01.xml
test_V02.xml
test_V03.xml
test_V04.xml
testing_V01.xml
testing_V(..).xml Representing all numbers 2 -99
testing_V100.xml

The result should be:
apps_V01.xml
betarelease_V02.xml
betarelease_V03.xml
test_V03.xml
test_V04.xml
testing_V99.xml
testing_V100.xml

I thought about putting the listing into a text file and then substringing the names using awk, but don't know how I would handle the number of similar files. My thought is to output the listing to a file, read the file until it reaches a new file creating an array of files and then save the last two in the array. Then read for the next set of files. But again, not sure how to do that. A problem also occurs when I only have 1 version of a file. I welcome any sed, awk or ksh commands. I don't know enough about Perl or any other language in order to do this. Some help would be greatly appreciated. I have searched more than 300 postings and not coming up with anything fairly close to what I need to accomplish.

Last edited by synergy_texas; 10-28-2008 at 07:56 PM..
# 2  
Old 10-28-2008
Question

Do you want to delete then the extra versions?
For example, somehow you want to execute
>rm betarelease_V01.xml
as a file out of condition?
# 3  
Old 10-28-2008
That is correct. I want to keep only the last two versions of each file and delete all others. My sample output shows the results I would get if the script works correctly. That part would not be hard. If I could get all the other files in a text file, then I could run a "for loop" that would delete all the files from the directory that exist in the text. That I can do.

for i in `cat removefiles.txt`
do
rm temp/$i
done
# 4  
Old 10-28-2008
Hi, I thought about it, and even though I may do it the hard way, I think it gets the job done... and to be on the safe side, make working copies, this one moves the wanted files into their own directory.

I tried it and it works on my machine Smilie

Code:
#!/bin/bash
#Assuming Your current working directory is ok to work in and the files
#are located in the directory verfiles (create working copies there)

#First sort, if the naming is consistent, that should be a good start
ls -1 verfiles | sort > verfiles.srt

#split the filenames on each "firstname" so they will have their own file
prefix=""
while read filename; do
#Determine the "prefix"
curprefix=$(expr match "$filename" '\(^[a-Z]*_\)')
#Has it been read before?
if [ x$curprefix != x$prefix ] ; then
	#We have a new "firstname"
	prefix=$curprefix
fi
echo $filename >> $prefix.lst
done < verfiles.srt
#cleanup
rm verfiles.srt

#Where to keep the "last two" of each...
mkdir saved-verfiles

#Now for each .lst file, manipulate it a little for easier numerical sorting
for x in *.lst; do
#And this next bit is VERY lazy of me, but as I said, 
#I ASSUME that the naming is consistent ;)
myarr=($(tr V " " < $x | tr . " " |sort -n -k2 | tail -2))

mv verfiles/${myarr[0]}V${myarr[1]}.${myarr[2]} saved-verfiles
mv verfiles/${myarr[3]}V${myarr[4]}.${myarr[5]} saved-verfiles
done

Hope it helps...

/Lakris
# 5  
Old 10-28-2008
Hammer & Screwdriver

The following are the files I worked with:
Code:
> cat file100
apps_V01.xml
betarelease_V01.xml
betarelease_V02.xml
betarelease_V03.xml
test_V01.xml
test_V02.xml
test_V03.xml
test_V04.xml
testing_V01.xml
testing_V96.xml
testing_V97.xml
testing_V98.xml
testing_V99.xml
testing_V100.xml

And here is the script:
(could probably be better, but it is getting late!)
Code:
> cat keep_two
#! /usr/bin/bash

# to force/create necessary test files
while read filename; do touch $filename; done<file100

cut -d"_" -f1 file100 | sort | uniq -c >file101
cat file100 | cut -d"." -f1 | awk -F"_" '{print $2" "$1}' | cut -c2- >file102
rm file103

while read filecnt filename
   do
   if [ $filecnt -gt 2 ]
     then
     filekill=$((filecnt-2))
     grep "${filename}" file102 | head -"$filekill" >>file103
   fi
done <file101

while read filenum filename
   do
   rm ${filename}_V${filenum}.*
done <file103

The script creates the files in that early step based on the contents of file100. That way, I could test the delete capability.
# 6  
Old 10-29-2008
Hi,

Blow code will print out the file name which should not be deleted, in other words the last two versions for each category.

I am stop here to find any easy way, you may use your smart to polish [echo $file] to address your issue.

Code:
for file in `ls *.xml | sort -r | nawk -F"[V|.]" '{if(_[$1]<2){print $0;_[$1]++}}' `
do
	echo $file
done

# 7  
Old 10-29-2008
Computer

Smilie Thanks for the responses. All were great ideas. Summer_cherry had the most compact code and easiest to manipulate for an inexperienced shell programmer like me. However, I did like all your responses and will try to learn from each of your approaches. Thank you all very much.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help with korn shell script to get the latest file versions in a directory

I want to write a korn shell script to get the latest three versions for a file in the directory having lot of files with various versions (files with prefix as same but time stamp as suffix) and compress it and at the same time have to remove the remaining versions of the file (other than latest... (4 Replies)
Discussion started by: maheshbabu
4 Replies

2. Shell Programming and Scripting

Preserving variable in script run with sudo

In a nutshell, I want $USER to reflect my user ID in a script run with sudo. I'm working with OSX in case that makes a difference. If I 'sudo echo $USER', I get my user ID. But if I 'sudo myscript.sh' and myscript.sh has a line to echo $USER, I get 'root' I'm hoping there's a switch I can add... (1 Reply)
Discussion started by: jnojr
1 Replies

3. Shell Programming and Scripting

2 versions, 1 script (A tale of madness.)

So, I have a machine running solaris (x86) and it has two different versions of Math::BigInt installed on it. The older version is 1.77. The newer version is 1.87 (via activeperl). When I run my code, one of the modules I use needs at least version 1.78. Somehow, it defaults to looking at the... (5 Replies)
Discussion started by: mrwatkin
5 Replies

4. Shell Programming and Scripting

Copying a directory structure with the latest versions of files

Hello I have three directory structures for code releases. Each directory structure looks like this: bash-3.00$ ls -R | more .: Test_Release_1 Test_Release_2 Test_Release_3 ./Test_Release_1/dbcode: rp_online_import_srdp.pkb-1 srdp_ar_validation.pkb-1... (1 Reply)
Discussion started by: Glyn_Mo
1 Replies

5. Shell Programming and Scripting

Script to delete older versions of unique files

I have directory where new sub directories and files being created every few minutes. The directories are like abc_date, def_date, ghi_date. I am looking to keep the latest 2 unique directories and delete everything else. Here is what I have so far This gives me unique names excluding the... (5 Replies)
Discussion started by: zzstore
5 Replies

6. Shell Programming and Scripting

Need a script to delete previous versions of files

Hi. I need a script (either bash or perl) that can delete previous versions of files. For instance, from our continuous build process I get directories such as build5_dev_1.21 build5_dev_1.22 build5_dev_1.23 build5_dev_1.24 I need a script that I can run every night (using "at"... (6 Replies)
Discussion started by: jbsimon000
6 Replies

7. Shell Programming and Scripting

Shell versions affecting find?

Hi all, I just spent 30hours writing a script (my first) that makes extensive use of the find command on my mac os x terminal and which was meant to be run as a cron job on my server. The script is an automated backup system for a bunch of source control repositories. On the mac it works... (6 Replies)
Discussion started by: cpower
6 Replies

8. Shell Programming and Scripting

How to manage multiple versions of a set of shell and SQL script utilities

Hi all --- I have the need to manage multiple versions of a set of utility scripts -- both shell and SQL and other .dat files. I am wondering if anyone out there knows of a good way to "PATH" to SQL and text files in a way similar to how PATH facilitates finding executables in a pre-specified... (2 Replies)
Discussion started by: DennisB
2 Replies

9. Shell Programming and Scripting

Purge files, keep 3 versions of last 4 days

Hello, I currently generate a file every 15 minutes for 12 hours a day. I would like to clean the directory on a daily basis. I only want to keep the latest 3 versions for the last 4 days in the directory. Any suggestions? Thanks, Barbara (7 Replies)
Discussion started by: blt123
7 Replies

10. UNIX for Dummies Questions & Answers

Moving .html files while preserving hyperlink integrity?

Hi, I use a Windows-based program called <a href="http://www.coast.com">Coast Webmaster</a> for moving large numbers of HTML files in one directory to another directory. As you drag and drop each file or entire directory of files to new locations in the web root directory tree, this utility... (1 Reply)
Discussion started by: Buddy123
1 Replies
Login or Register to Ask a Question