One final thing, at the moment the find is in a FOR loop and within that loop, prior to the move, it does a couple of checks for things like if the destination directory it is moving it to exists, and if not it creates it before doing the move.
Presumably I can do code like this in the new fastermove script, which could receive a parameter which is the path to the new destination?
the destination is dependant on the path of each file being moved, I need to get that information from each incoming file, and set the destination accordingly. Do you think this is easily possible or is this making things complicated for this case?
Sorry, I have probably confused things along the way.
Basically we have a file system which has numerous folders and sub-folders, and is populated each day by incoming systems. Each day we archive a day's worth of data by using mtime +7 on the find command.
The day's worth of data must end up in its own compressed folder, keeping the sub-directory hierarchy, so at the moment we do the following -
make a directory of format YYYYMMDD, so today would be 20060109
then run the find command and for each file loop round, moving it to the same directory path in the dated directory made above. This bit is the bit that takes the time, we have approx 100,000 files each day. The code we do this with is shown below -
Code:
for File in `find . -type f -mtime +7`
do
Archive="archive/20060109/"`dirname "$File"`
if [ ! -d "$Archive" ]
then
echo `date +'%d/%m/%Y %T'`" [$BaseName] Creating archive directory $Archive ..."
mkdir -p "$Archive"
status=$?
if [ "$status" != "0" ]
then
echo `date +'%d/%m/%Y %T'`" [$BaseName] ERROR! Failed to create directory $Archive. " \
"Return code=$status." 1>&2
statuses=1
continue
fi
fi
mv "$File" "archive/20060109/$File"
status=$?
if [ "$status" != "0" ]
then
echo `date +'%d/%m/%Y %T'`" [$BaseName] ERROR! Failed to move data file $File. " \
"Return code=$status." 1>&2
statuses=1
continue
fi
done
....
....
tar -cf "20060109.tar" "20060109"
....
....
compress "20060109.tar"
Thanks in advance for any suggestions on how this could be speeded up, it currently takes an average of 15 hours to run!
It is interesting that the script runs at all. The backtick syntax asks ksh to run the internal command and build a command line with the results. Command lines tend to have smaller limits than you would need. Or so I thought. I just got:
/usr/bin/echo `find /usr -type f -print` to work! I guess the limited got upped when we switched to 64 bits.
You have a rough problem here. We must not run even one program per file. It takes too much time to launch programs. So believe it or not, I think the faster approach will involve two stages: One to copy the the files, and another to delete them. And you need to use the date command to name your archive directory. Do you really edit this script every day?
cd /big/file/collection
Archive="archive/$(date "+%Y%m%d")
mkdir -p $Archive
find . -type f -mtime +7 | cpio -pd $Archive
(cd $Archive ; find . -type f )| xargs rm
Yes, that's the whole thing...I think. But it's untested.
It is interesting that the script runs at all. The backtick syntax asks ksh to run the internal command and build a command line with the results. Command lines tend to have smaller limits than you would need. Or so I thought. I just got:
/usr/bin/echo `find /usr -type f -print` to work! I guess the limited got upped when we switched to 64 bits.
You have a rough problem here. We must not run even one program per file. It takes too much time to launch programs. So believe it or not, I think the faster approach will involve two stages: One to copy the the files, and another to delete them. And you need to use the date command to name your archive directory. Do you really edit this script every day?
cd /big/file/collection
Archive="archive/$(date "+%Y%m%d")
mkdir -p $Archive
find . -type f -mtime +7 | cpio -pd $Archive
(cd $Archive ; find . -type f )| xargs rm
Yes, that's the whole thing...I think. But it's untested.
Maybe my eyes deceive me here, but isnt your code deleting the very files you just moved into $Archive?
Hi,
Not sure whether this is the right place to post it. I decided to post it here 'coz Advanced and Expert users will most likely have the answer to what I am looking for.
I want to backup scripts that I have access to to a tar file file and zip it. At the moment I am creating a directory... (4 Replies)
This will be covered elsewhere im sure but i just cant seem to find my exact issue.
I want to backup my systems using tar, command is:
tar -cjpf /backup /bin /etc /home /opt /root /sbin /usr /var /bootWhen i include the / directory it also tar's the /lib /sys /proc /dev filesystems too (and... (8 Replies)
I am backing up some data to an NTFS formatted backup drive. I have to preserve the Unix permissions of the data being backed up and therfore use backup into a tar file.
I would like to backup the differnential data in the tar file similiar to how Rsync works so as to save on backup time as it... (1 Reply)
Hi all
i'm looking for good advice regarding backup plan becuase its first time to me handle large scale database expected to grow up 10000 - 20000 record per year with daily operations on it
I'm working as sysAdmin in educational organization ( junior level ) with mixed OSes environment... (3 Replies)
Hi friends,
I am planning to backup my Solaris Servers to SAN storage using tar.
Also palnning to automate the job using Crontab.
Can anyone advise how to make the date change automatically everyday for backup.
Pls correct me if I am wrong. Thanks (7 Replies)
Hi all,
i need to backup files on network from RHEL 4 machine
tape drive is installed on solaris 10 machine and want ot use this
using
# tar cv /myfiles |ssh -l myuser myhost 'buffer -o /dev/rmt/0 "
to backup these file but getting getting error " sh buffer not found '
even "buffer-1.19-1"... (2 Replies)
Hello everyone!
I'm trying to make incremental tar archives of a folder for an example. On the box I use is UNIX AIX installed. I tried some sample codes I found on several web pages but with no success. Don't know what I'm doing wrong. Please write some sample code to make incremental tar... (0 Replies)
I am trying to do a full system backup using tar. It then after maybe 12 or so hours comes up with tar: write error: unexpected EOF. I have thoroughly cleaned the drive and tried to use a different drive but it still gives me this error. Can someone help. I am on solaris 8. (1 Reply)
Hi all & anyone.
I'm trying to selectively backup up some old Apache log files before they are removed from the system (Slackware box).
Have created a file listing of what I want backed up ...Below is a portion of the file ./selectedbkup... (2 Replies)
Hi all,
I would like to append list of files to already taken tar backup
file. can anybody help?
last month backup :
cd /accounts/11
tar -cvf monthback.tar *
Now I want to add /accounts/12 to monthback.tar
is it possible?
Krishna (1 Reply)
01-06-2006
