Sponsored Content
Full Discussion: Backup help/advice using TAR
Top Forums UNIX for Dummies Questions & Answers Backup help/advice using TAR Post 94720 by tonysab on Tuesday 3rd of January 2006 09:57:11 AM
Old 01-03-2006
Backup help/advice using TAR

Every day we back up all files on our system that are older than 7 days, so effectively we do a day's worth at a time.
The way we do this is to issue a find command using mtime +7 - we then loop round and for each result we issue a MV to move the file to a newly created directory. We then TAR the directory and finally compress it to give us a .Z file.

On average we process about 100,000 files per day and this job to do the backup typically takes around 15 hours! Sometimes it can take over 24 hours which causes us problems.

By By adding simple echo statements to the code we have determined that it is the MV commands that are taking the time.

The question therefore is how to improve this.

After a bit of research I tried to pipe the results of the find straight into the TAR using xargs, to avoid doing all the MVs. However in testing this appears to take even longer than before, presumably because TAR is being passed 100,000 files instead of 1 directory containing 100,000 files?

Anyway here is the command -

find . -type f -mtime +7 | xargs -i ksh -c "tar -rf archive.tar" {}; rm {}"

the rm is in there because we need to delete them too.

I am a bit of a newbie so can anyone advise on this or suggest an alternative way?

Thanks in advance,
Tony
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

tar backup

Hi all, I would like to append list of files to already taken tar backup file. can anybody help? last month backup : cd /accounts/11 tar -cvf monthback.tar * Now I want to add /accounts/12 to monthback.tar is it possible? Krishna (1 Reply)
Discussion started by: krishna
1 Replies

2. Filesystems, Disks and Memory

Using `tar` for a selective backup.

Hi all & anyone. I'm trying to selectively backup up some old Apache log files before they are removed from the system (Slackware box). Have created a file listing of what I want backed up ...Below is a portion of the file ./selectedbkup... (2 Replies)
Discussion started by: Cameron
2 Replies

3. UNIX for Dummies Questions & Answers

Tar backup

I am trying to do a full system backup using tar. It then after maybe 12 or so hours comes up with tar: write error: unexpected EOF. I have thoroughly cleaned the drive and tried to use a different drive but it still gives me this error. Can someone help. I am on solaris 8. (1 Reply)
Discussion started by: TMashie
1 Replies

4. Shell Programming and Scripting

tar - incremental backup

Hello everyone! I'm trying to make incremental tar archives of a folder for an example. On the box I use is UNIX AIX installed. I tried some sample codes I found on several web pages but with no success. Don't know what I'm doing wrong. Please write some sample code to make incremental tar... (0 Replies)
Discussion started by: Funky_ass
0 Replies

5. Red Hat

tar backup on network

Hi all, i need to backup files on network from RHEL 4 machine tape drive is installed on solaris 10 machine and want ot use this using # tar cv /myfiles |ssh -l myuser myhost 'buffer -o /dev/rmt/0 " to backup these file but getting getting error " sh buffer not found ' even "buffer-1.19-1"... (2 Replies)
Discussion started by: ajays
2 Replies

6. UNIX for Dummies Questions & Answers

Backup with tar

Hi friends, I am planning to backup my Solaris Servers to SAN storage using tar. Also palnning to automate the job using Crontab. Can anyone advise how to make the date change automatically everyday for backup. Pls correct me if I am wrong. Thanks (7 Replies)
Discussion started by: solaris5.10
7 Replies

7. Filesystems, Disks and Memory

an advice regarding backup plan

Hi all i'm looking for good advice regarding backup plan becuase its first time to me handle large scale database expected to grow up 10000 - 20000 record per year with daily operations on it I'm working as sysAdmin in educational organization ( junior level ) with mixed OSes environment... (3 Replies)
Discussion started by: h@foorsa.biz
3 Replies

8. UNIX for Dummies Questions & Answers

Tar differential backup

I am backing up some data to an NTFS formatted backup drive. I have to preserve the Unix permissions of the data being backed up and therfore use backup into a tar file. I would like to backup the differnential data in the tar file similiar to how Rsync works so as to save on backup time as it... (1 Reply)
Discussion started by: jelloir
1 Replies

9. Red Hat

Backup and restore using tar

This will be covered elsewhere im sure but i just cant seem to find my exact issue. I want to backup my systems using tar, command is: tar -cjpf /backup /bin /etc /home /opt /root /sbin /usr /var /bootWhen i include the / directory it also tar's the /lib /sys /proc /dev filesystems too (and... (8 Replies)
Discussion started by: Tommyk
8 Replies

10. UNIX for Advanced & Expert Users

Advice on a backup script, maybe one is out there already?

Hi, Not sure whether this is the right place to post it. I decided to post it here 'coz Advanced and Expert users will most likely have the answer to what I am looking for. I want to backup scripts that I have access to to a tar file file and zip it. At the moment I am creating a directory... (4 Replies)
Discussion started by: newbie_01
4 Replies
th_get_pathname(3)						  C Library Calls						th_get_pathname(3)

NAME
th_get_pathname, th_get_uid, th_get_gid, th_get_mode, th_get_crc, th_get_size, th_get_mtime, th_get_devmajor, th_get_devminor, th_get_linkname - extract individual fields of a tar header TH_ISREG, TH_ISLNK, TH_ISSYM, TH_ISCHR, TH_ISBLK, TH_ISDIR, TH_ISFIFO - determine what kind of file a tar header refers to TH_ISLONGNAME, TH_ISLONGLINK - determine whether the GNU extensions are in use SYNOPSIS
#include <libtar.h> char *th_get_linkname(TAR *t); char *th_get_pathname(TAR *t); mode_t th_get_mode(TAR *t); uid_t th_get_uid(TAR *t); gid_t th_get_gid(TAR *t); int th_get_crc(TAR *t); off_t th_get_size(TAR *t); time_t th_get_mtime(TAR *t); major_t th_get_devmajor(TAR *t); minor_t th_get_devminor(TAR *t); int TH_ISREG(TAR *t); int TH_ISLNK(TAR *t); int TH_ISSYM(TAR *t); int TH_ISCHR(TAR *t); int TH_ISBLK(TAR *t); int TH_ISDIR(TAR *t); int TH_ISFIFO(TAR *t); int TH_ISLONGNAME(TAR *t); int TH_ISLONGLINK(TAR *t); VERSION
This man page documents version 1.2 of libtar. DESCRIPTION
The th_get_*() functions extract individual fields from the current tar header associated with the TAR handle t. The TH_IS*() macros are used to evaluate what kind of file is pointed to by the current tar header associated with the TAR handle t. The TH_ISLONGNAME() and TH_ISLONGLINK() macros evaluate whether or not the GNU extensions are used by the current tar header associated with the TAR handle t. This is only relevant if the TAR_GNU option was used when tar_open() was called. SEE ALSO
tar_open(3) University of Illinois Jan 2001 th_get_pathname(3)
All times are GMT -4. The time now is 07:22 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy