Considered basic but advanced outcome (Custom Backup Solution)


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Considered basic but advanced outcome (Custom Backup Solution)
# 1  
Old 03-02-2013
Considered basic but advanced outcome (Custom Backup Solution)

Ive a problem that I'm reaching out for help.

Ive written (With bits and pieces) of script that is not running as expected or is having an issue causing processes to spiral out of control.

The script does this:

Unloads a UV database server

Tars up a few folders

Transfers the file to another server

Reloads the UV database

Simple actually but I'm lost in its dysfunction.

Code:
#!/bin/sh
# Unload the UV Database
# Suspend the database
uv -admin -L
# Validate the database
uv -admin -R
#Sync any cached data back to disk
sync
# What is the script backing up exactly
backup_files="/home /etc /u2 /usr/spool/uv /boot /opt"
# Where is it backing up to
dest="/backup"
# Create a daily variable for the archive filename and add the hostname to it
day=$(date +%A)
hostname=$(hostname -s)
# Find which week of the month 1-4 it is.
day_num=$(date +%d)
if (( $day_num <= 7 )); then
        week_file="$hostname-week1.tgz"
elif (( $day_num > 7 && $day_num <= 14 )); then
        week_file="$hostname-week2.tgz"
elif (( $day_num > 14 && $day_num <= 21 )); then
        week_file="$hostname-week3.tgz"
elif (( $day_num > 21 && $day_num < 32 )); then
        week_file="$hostname-week4.tgz"
fi
# Find if the Month is odd or even.
month_num=$(date +%m)
month=$(expr $month_num % 2)
if [ $month -eq 0 ]; then
        month_file="$hostname-month2.tgz"
else
        month_file="$hostname-month1.tgz"
fi
# Create archive filename.
if [ $day_num == 1 ]; then
        archive_file=$month_file
elif [ $day != "Saturday" ]; then
        archive_file="$hostname-$day.tgz"
else
        archive_file=$week_file
fi
# Print start status message.
echo
echo "Backing up $backup_files to $dest/$archive_file"
date
# Backup the files using tar.
tar czf $dest/$archive_file $backup_files
# Print end status message.
echo
echo "Backup finished"
date
# Long listing of files in $dest to check file sizes.
ls -lh $dest/
# Release the Database
uv -admin -U

The result is that it will write to a local server (CIFS) but its performance is so bad it never completes or takes down the server.

//location /backup cifs rw,user=domain/user,password=PASS,uid=1000,gid=1000 0 0

I end up with multiple uvsh processes running at 100 percent, or 50 percent. Yes I'm stumped here, I expect this to be simple but its not acting right.
# 2  
Old 03-02-2013
You show us a script, but you don't say what its name is. (If it is uvsh and you have multiple copies of this script running at the same time, it is no wonder that you're having problems.)

If uvsh is something else and it is accessing your database while you suspend, validate, backup, and release it by running this script; that probably isn't good either.

What type of system are you using? What type of shell is /bin/sh on your system? (Is it a Bourne shell, a Korn shell, a Bash shell, or what?)

When you watch this script run, what messages do you see (and does that tell you what portion of the script is running slowly)?
# 3  
Old 03-02-2013
You show us a script, but you don't say what its name is. its a script called manually: backup.sh (If it is uvsh and you have multiple copies of this script running at the same time The multiple instances of uvsh appear from running the script, but I'm unsure of its relation nor should it happen? I am unsure , it is no wonder that you're having problems.)

If uvsh is something else and it is accessing your database while you suspend, validate, backup, and release it by running this script; that probably isn't good either. I unload the database which should close it to be tar'd based on eclipse's recommendation

What type of system are you using? What type of shell is /bin/sh on your system? (Is it a Bourne shell, a Korn shell, a Bash shell, or what?)
Redhat, bash, is that an ok answer?

When you watch this script run, what messages do you see (and does that tell you what portion of the script is running slowly)?
The script messages seem to function normally and in the past created a 15gb archive of the required back'd up folders, now it is not and I'm stumped

I have very limited basic skills in bash scripting.

I appreciate your reply and hope I've been able to follow up here with answers to your questions.

---------- Post updated at 05:55 PM ---------- Previous update was at 05:52 PM ----------

What I'm confused about is the concept here is to:

Unload the universe JBOSS database so all tables are closed etc.

Then tar up and transfer the archive based on the day/week/month (A rotation).

The script causes a process (uvsh) to run multiple times and the saving of the archive goes to an absolute crawl. The main part of this script is a simple log rotation example.

---------- Post updated at 06:49 PM ---------- Previous update was at 05:55 PM ----------

You can see the process is a gzip but I'm not understanding why uvsh is being called multiple times when the script runs?

Last edited by Scrutinizer; 03-03-2013 at 03:44 AM.. Reason: Removed non-responsive image
# 4  
Old 03-03-2013
I'm afraid that I can't help much more with the information given.

I'm not familiar with uv, uvsh, or eclipse. The gzip you're seeing could be a side effect of using the "z" option letter in the 1st argument to your invocation of tar:
[CODE]tar czf ...[ICODE]

Having three separate calls to date to set day, day_num, and month_num is dangerous if they could be called close to midnight, but that won't affect the problem you're seeing.

It sounds like you have tried running this script several times and restarted another copy before an earlier script finished. I'm guessing that this has left you with database commands hanging while you have another script that is trying to make a copy of the database. Before trying to run it again, you probably need to kill off every instance of uvsh and backup.sh, but that may leave your database in an inconsistent state. Hopefully, after killing off all of the uvsh instances, you can manually use:
Code:
uv -admin -R

to put the database in a consistent state and then run your backup script to make a backup copy and restart the database server.

Good luck.
# 5  
Old 03-03-2013
That script has three uv commands (of which the DB validation may take its time), and then just some shell cmds to compose the backup names (which could be optimized) and the tar to create the backups. It seems /backup is an smbmounted share - why don't you replace it for debugging purposes? /dev/null would be a splendid candidate...
Why don't you separate the script into independent blocks and try to find out which one is the culprit dragging its feet?
# 6  
Old 03-03-2013
Great suggestions! I woke up this morning with another fix which I'll try (Moving the script from the cifs to the local file system) although RudiC might be onto something to troubleshoot.

This is an outside managed system which doesn't cover backup and the local VM of this entire server runs here with the very same results so its definitely something in the script.

Will report back shortly with results of suggestions.

---------- Post updated at 08:32 AM ---------- Previous update was at 08:12 AM ----------

# Suspend the database
uv -admin -L
# Validate the database
uv -admin -R
#Sync any cached data back to disk
sync


I ran each of these separately to see when the uvsh processes would go nuts (Best info I can come up with to describe what's happening).

uv -admin -L (File suspension happens, no issue)
uv -admin -R (Validation happens, although quickly, Id expect it to take longer than a blip then prompt)

sync, now is when the multiple uvsh processes pop up and what I'm suspecting is the major performance drag, incomplete process etc.

Not wanting to think ahead on this just want to report what Ive found by breaking each part of the script down to manual cli entry and report back.

I'll try the /dev/null and see what happens now.

Decided to watch and see what happens, its definitely the sync'ing of the database command to the server that's causing it (This is not the case actually). I'm now curious why this is happening as it either is something that must be done, fully no matter how long it takes, or is a flag of an issue we're not aware of in the system (Which I doubt.. I know its a reach).

Last edited by coastdweller; 03-03-2013 at 04:40 PM..
# 7  
Old 03-03-2013
Unless you have replaced the standard UNIX sync command with something else, it will not start any database operations. The only thing the standard sync command does is schedule flushing of file system buffers to disk. On a large server with a huge amount of memory, flushing those buffers to disk may literally take an hour depending on the disk speeds and I/O data paths. While scheduled flushes are being processed, the system may indeed be sluggish.

Many database systems perform synchronous writes to be sure that data for completed transactions is safely stored on the underlying files rather than just sitting in buffers to be flushed later. It doesn't sound like your database software does this. If transactions are flushed when they are completed, there will be a small performance penalty at the end of each transaction (and you data would be safe if a power failure or other calamity occurs later), but you wouldn't suffer the big hits you're seeing now. Note that the sync command does not wait for data to be flushed before it returns; it just schedules the flush and returns letting the OS perform the flush to disk while it continues to process other requests.
Login or Register to Ask a Question

Previous Thread | Next Thread

7 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Backup solution using rsync

Hello All, I am looking at a fast way to script some backups. I am looking at using rsync to do the leg work. I am having a hard time conceiving a script though. I have a tree with subfolders within subfolders. I was looking at the /xd option to parse the tree. Directory of k:\ ... (4 Replies)
Discussion started by: jvamos
4 Replies

2. Linux

Backup solution

Hi all, i am not sure whether i selected the right topic or not, so excuse me, well i am mainly a Windows admin, but i do *NIX administration from time to time, for now i need to use an open source solution for backup windows environment mainly, i spent last days playing with bacula and backupPC,... (0 Replies)
Discussion started by: XP_2600
0 Replies

3. Red Hat

RHEL 5 basic vs advanced

Hello there to the community Question 1 : how can you tell if the RHEL 5 version you have is BASIC or ADVANCED. ? Question 2 : except for virtualization limits on BASIC RHEL which is 2 physical CPUs and 4 virtual guest (redhat virtualization , not VMWARE) and Advanced allows unlimited CPU and... (4 Replies)
Discussion started by: dplinux
4 Replies

4. AIX

Offsite Backup Solution

I'm looking for a program or some way to backup my server to another location onto another server. Does anyone have any ideas? What utility, server O/S(I would like to use Linux RH or CentOS)? Whats the best way to do this? I am fairly new with Unix. Thanks in advance (6 Replies)
Discussion started by: ITAdmin08
6 Replies

5. AIX

Backup solution

Hello, I'm looking for a backup solution for my system. I have 3 AIX virtual partitions running on a IBM p5 server. Each partition has it's data on a DS4700 storage server. Also, I have a RedHat running on an IBM p720. This server has the tape hardware. I would like to know if I can backup from... (4 Replies)
Discussion started by: enzote
4 Replies

6. Solaris

Sun and backup solution

My company is in the process of building a new datacenter and I am responsible for setting up the backup environment (everything from racking to implementation). I guess it is pretty basic but I don't have much experience with the initial setup. What we have so far is a SunV890 (backup server), Sun... (1 Reply)
Discussion started by: Jshwon
1 Replies

7. UNIX for Advanced & Expert Users

enterprise backup solution

We are using sun_sparc solaris 2.5.1 with oracle database 8.0.5 We are considering to buy for buying backup software for this purpose . Our systems integrator says veritas could not be installed due to some technical reasons (they are veritas authorised dealer) . And is suggesting Netvault... (3 Replies)
Discussion started by: Hitesh Shah
3 Replies
Login or Register to Ask a Question