Advice on a backup script, maybe one is out there already?


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Advice on a backup script, maybe one is out there already?
# 1  
Old 12-10-2016
Advice on a backup script, maybe one is out there already?

Hi,

Not sure whether this is the right place to post it. I decided to post it here 'coz Advanced and Expert users will most likely have the answer to what I am looking for.

I want to backup scripts that I have access to to a tar file file and zip it. At the moment I am creating a directory that sort of mimic the directory structure that contains the file that I want to backup.

For example, the source directory make look like this:

Code:
/u01/db/mysql
/u01/db/mysql/scripts
/u01/db/mysql/scripts/user01
/u01/db/mysql/scripts/user02
/u01/db/mysql/sql

And at the moment, I am creating a backup directory script that look like below:

Code:
/u02/bak/db/mysql
/u02/bak/db/mysql/scripts
/u02/bak/db/mysql/scripts/user01
/u02/bak/db/mysql/scripts/user02
/u02/bak/db/mysql/sql

I then manually copy the file from each directory, the .sh, .ksh, .sql, .log etc. to each directory, then I do below.

Code:
cd /u02
tar -cvf bak.tar ./bak
gzip bak,tar

As you may have guess, this takes a loooong time Smilie So I am wanting to know if someone can recommend a short process that will more or less give the same result.

In its simplest form, I want to find files of .sh, .ksh, .sql and other file types tar them to a file and gzip the tar file. I only want to find files that I have permission to. Yeah, some of the files are root owned and I do get permission denied when I am doing the cp manually

Kinda hoping that maybe someone had done a similar thing of what I am trying to do, preferably a script. Any advice will be very much appreciated. Thanks in advance.
# 2  
Old 12-10-2016
There seems to be a misunderstanding. A backup means an exact copy, not a parallel copy. Plus if the data is important, consider getting the backup onto separate media as well.

Why: 'restore a backup' means extracting the backup file into many correctly named files. Correctly named includes the path name. Your backups all have files with the wrong name, unless - to restore - you want to unpack the tarball, then manually copy each file back where it came from.

Not that this will not work. It is duplicated effort and wastes disk space - you actually have 3 copies of each file.

Code:
tar cvfz /path/to/backup/backup_`date +%Y%m%d`.tar.gz /u01/db/mysql

This creates a backup file with the backup date as part of the name, already compressed, on whatever device you choose to place it. Suppose someone then accidentally trashes a file, /u01/db/mysql/scripts/foo.ksh. So. You want to restore it to the way it was before it got munged:
choose the date you want to restore back to, say 20161201. - Dec 1, 2106
Code:
cd /u01/db/mysql/scripts
tar xvfz /path/to/backup/backup_20161201.tar.gz /u01/db/mysql/scripts/foo.shl .

# 3  
Old 12-12-2016
If the content is a database, then backing up in this way may also not give you a consistent restore in any case, especially if you are unable to read some of them!

You might be better getting the database tools to make a backup. Another alternate (but I'm guaranteeing it) might be to use LVM snapshots if you OS supports it. This way, you can be sure that the files are idle and although the database might go into crash-recovery (as though you've had a power loss) it should give you a point-in-time restore from when you split off the copy.

What OS are you using?

One worries that this is not a good plan though. If you cannot read all the data, how would you plan to restore it?


On your point about getting it to run faster, it will always depend on a number of factors, such as volume of data, if there is I/O contention etc. I/O contention can occur for various reasons, such as sharing physical media/access path (shared controllers, fibre etc.) or other users even.

Is the device SAN based? There may be a way to use SAN tools to copy a LUN that would give a suitable restore point. Of course it would revert everything on that LUN, so that might not be what you want, but perhaps you could mount the copy LUN as another filesystem and back that up away from the live database.


Can you explain a bit more about your server setup?


Kind regards,
Robin
# 4  
Old 12-12-2016
Are users locked out during process?

One other concern is relative to file access. Do you lock out users during the process? Otherwise (and especially since you say this takes time) by the time the last file is processed and first could have changed. And your backup set of files will no longer be in synch with each other.
# 5  
Old 12-13-2016
I forgot to mention that some databases allow you to freeze the data files allowing you to backup the files. It keeps a separate log of all updates pending and refers to it in normal processing. When you remove the freeze (would that be a thaw?) the updates are written to the files.

They have to be set up to allow this functionality.



Robin
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help with Backup Shell Script for Network Device Configuration backup

HI all, im new to shell scripting. need your guidence for my script. i wrote one script and is attached here Im explaining the requirement of script. AIM: Shell script to run automatically as per scheduled and backup few network devices configurations. Script will contain a set of commands... (4 Replies)
Discussion started by: saichand1985
4 Replies

2. Shell Programming and Scripting

Script advice

All I have 2 parent directories - input and output. Each parent has multiple sub-directories...each sub-directory has multiple files. Each parent directory structure is a mirror image of itself I need to poll the imput directory and if a new file is found, encrypt the file, move the file to... (2 Replies)
Discussion started by: davidra
2 Replies

3. Shell Programming and Scripting

Advice on script

Hi folks, I use following script:- #!/bin/sh # cd Linbread TODAY=`date +"%m%d"` DATA=`grep $TODAY linbread.dat` HOUR=`date +"%H"` if then TOD="Morning" elif then TOD="Afternoon" else TOD="Evening" fi echo $DATA | gawk -F"|" '{printf("%s\n\n%s",$2,$3)}' > $$tmp fold -s -w60... (0 Replies)
Discussion started by: satimis
0 Replies

4. Filesystems, Disks and Memory

an advice regarding backup plan

Hi all i'm looking for good advice regarding backup plan becuase its first time to me handle large scale database expected to grow up 10000 - 20000 record per year with daily operations on it I'm working as sysAdmin in educational organization ( junior level ) with mixed OSes environment... (3 Replies)
Discussion started by: h@foorsa.biz
3 Replies

5. Shell Programming and Scripting

Script Help/Advice

Alright, I feel like I have a pretty good basic knowledge of shell scripting, but this one is throwing me for a loop. I know I've seen something similar done with awk, but I couldn't find it with the search function. I've grepped through my log file and get results like this: --... (14 Replies)
Discussion started by: earnstaf
14 Replies

6. Shell Programming and Scripting

Script Advice please?

Ok. I want to parse a log file and search only for denied traffic for the previous hour. The log looks like this: Jun 18 17:47:56 routername 36806: Jun 18 17:53:01.088: %SEC-6-IPACCESSLOG: list ingress-filter denied tcp 1.2.3.4(1234) -> 6.7.8.9(53), 4 packets I only really care about the... (12 Replies)
Discussion started by: earnstaf
12 Replies

7. Shell Programming and Scripting

Advice on Script

I would like some advice on how to logically put together a script to handle a daily task of data gathering for the following problem. I have two files, file1 has 125,000 records that I cut and remove unwanted fields through scripts and cron. In file2, I have 25000 records that has the same... (4 Replies)
Discussion started by: greengrass
4 Replies

8. Shell Programming and Scripting

first script. need help and advice.

Hello everyone, This is my first post here and this is the first time I am using UNIX OS (Slackware). I find it really useful and powerful and would like to master it but as you may guess I am expreicing quite a few problems. I've been reading a few documentations about it and bash this week... (17 Replies)
Discussion started by: sanchopansa
17 Replies

9. UNIX for Dummies Questions & Answers

Backup help/advice using TAR

Every day we back up all files on our system that are older than 7 days, so effectively we do a day's worth at a time. The way we do this is to issue a find command using mtime +7 - we then loop round and for each result we issue a MV to move the file to a newly created directory. We then TAR the... (20 Replies)
Discussion started by: tonysab
20 Replies

10. Shell Programming and Scripting

c-shell script advice please.

Hi, I have the following script running in my cron. -------------------------------------------------------------------- #!/bin/csh bnstat -p GPD_VSLinux | grep pg | grep varcon | awk '{print $1, $2, $3, $4, $5, $6, $7, $8, $9, $10}' > /tmp/LX_xbatch.log bnstat -p GPD_VSLinux_test | grep pg... (2 Replies)
Discussion started by: killerserv
2 Replies
Login or Register to Ask a Question