No alarm triggered when backup tape is full

 
Thread Tools Search this Thread
Special Forums UNIX and Linux Applications No alarm triggered when backup tape is full
# 1  
Old 04-02-2012
No alarm triggered when backup tape is full

Hi folks, i encountered an issue, no alarm is triggered when the tape is full on the Solaris server.
The tape is 72GB.
There is an alarm generating shell script in the server, which will trigger an alarm when it detects the tape is full.
Hence, I will receive SMS and email alerts.
This has been working for quite sometimes, however the issue happens only a few months ago.
I have checked that no one has modified the shell script.
The alerting system which sends out SMS and email alerts are working fine for other alarms, except for the tape full alarm.
Any other things which i can check on the server in order to troubleshoot this issue?
Could it be some detections have failed somewhere?
# 2  
Old 04-02-2012
how the script is executing ? through cron ?
if yes, did u check the cron logs ?
# 3  
Old 04-02-2012
Are you saying that this used to work? Or have your backups just got bigger over time and now tape is filling up?

What backup utility is being used? tar, cpio, ufsdump, or what?

Some utilities require you to tell it how big the tape is on the command line.
# 4  
Old 04-03-2012
Yes, the script is executing thru cron as follows:
# Backup /data directories into tape device
#==========================================
0 5 * * * /export/home/provadm/script/tape_backup.sh >/dev/null 2>&1
#


I was unable to find any cron logs from /var/log.
Can you advise me which directory can I look for cron logs?
Thanks Smilie


---------- Post updated at 12:15 PM ---------- Previous update was at 12:05 PM ----------

Quote:
Originally Posted by hicksd8
Are you saying that this used to work? Or have your backups just got bigger over time and now tape is filling up?

What backup utility is being used? tar, cpio, ufsdump, or what?

Some utilities require you to tell it how big the tape is on the command line.


Yes, this used to be working fine a year ago.
I've tested filling up a tape with some dummy files which i created in the directory, till it reaches the capacity of the tape(72GB). However, there is no alarm triggered.

I use tar as the backup utility.

---------- Post updated at 12:43 PM ---------- Previous update was at 12:15 PM ----------

Below is the shell script which was created by the previous engineer:


#!/bin/sh
#set -x

VERSION="Version: 1.06"
######################
# Choose the HOME dir
#####################
#HOME=/export/home/oracle
HOME=/export/home/provadm

BINDIR=$HOME/bin
LOGDIR=$HOME/log
SCRDIR=$HOME/script
CONFDIR=$HOME/config
TEMP=$HOME/temp

#CONFIG_FILE="$CONFDIR/logInfo.cfg"
#######################
# Choose the BACKUP dir
#######################
#LSMS dir is /data1
BACKUPDIR=/data
#PGW,RPT,LNPDB dir is /data
#BACKUPDIR=/data

#BACKUPDIR2=/data/data1
#LOGS=`cat $CONFIG_FILE | grep -v ";" | /bin/awk '{print $2}'`
#LOGS=`echo $LOGS`
newhost=`hostname |sed 's/LIV_//'| sed 's/_01//'`
TAPE_DEVICE=/dev/rmt/0
TAPE_DEVICE_APPEND=/dev/rmt/0n

datestr=`date +%d-%m-%Y`
timestr=`date +%H:%M:%S`
event_day=`/bin/date +%Y%m%d`
currdate=`$HOME/bin/lastdate +1 | awk '{printf "%s",substr($0,1,6)}'`
LOG=$LOGDIR/TAPE/$currdate/tape_bk_$event_day.log
TMP_LOG=$TEMP/tape_backup.tmp

# Format yyyymmdd
yest01=`$BINDIR/lastdate 1 | /bin/awk '{print $1}'`
# Format MMM
mm=`$BINDIR/lastdate 1 | /bin/awk '{print $2}' | cut -c1-3`
# Format DD
dd=`$BINDIR/lastdate 1 | /bin/awk '{print $1}' | cut -c7-8`
# Format YYYY
yyyy=`$BINDIR/lastdate 1 | /bin/awk '{print $1}' | cut -c1-4`
# Format yyyymm
yestmonth01=`$BINDIR/lastdate 1 | /bin/awk '{print $1}' | cut -c1-6`
# ==============
# Date and Month
# ==============
REP_MONTH=`date "+%Y%m"`
REP_DATE=`date "+%Y%m%d"`
FIRST_DAY=`$BINDIR/lastdate | /bin/awk '{print $1}' | cut -c7-8`

chktape()
{
status=`/bin/mt -f $TAPE_DEVICE_APPEND status | grep 'sense' | awk '{print substr($1,1,5)}'`
}

tarcmd()
{
# Check the $TMP_LOG file if exists, if yes remove the file
if [ -f $TMP_LOG ]; then
rm $TMP_LOG
fi

#/data1/ACCESS
#All Servers
/bin/tar cvf $TAPE_DEVICE_APPEND $BACKUPDIR/ACCESS/$yestmonth01/*$yest01*.tar.Z >> $LOG 2>>$TMP_LOG

if [ $FIRST_DAY = "01" ]; then
#/data1/BACKUP
# All Servers
/bin/tar cvf $TAPE_DEVICE_APPEND $BACKUPDIR/BACKUP/$yestmonth01/*_$yestmonth01.* >> $LOG 2>>$TMP_LOG
fi

#/data/BACKUP
# LNPDB, RPT
#/bin/tar cvf $TAPE_DEVICE_APPEND $BACKUPDIR/BACKUP/$yestmonth01/*$yest01*.tar.gz >> $LOG 2>>$TMP_LOG

#/data1/SYS
# All Servers
/bin/tar cvf $TAPE_DEVICE_APPEND $BACKUPDIR/SYS/$yestmonth01/$yest01*.Z >> $LOG 2>>$TMP_LOG

#/data1/TDR1
# LSMS, RPT
#/bin/tar cvf $TAPE_DEVICE_APPEND $BACKUPDIR/TDR1/$yestmonth01/$yest01*.Z >> $LOG 2>>$TMP_LOG

#/data1/TDR1_M
# LSMS
#/bin/tar cvf $TAPE_DEVICE_APPEND $BACKUPDIR/TDR1_M/$yestmonth01/$yest01*.TDR1_M.Z >> $LOG 2>>$TMP_LOG

#/data/TDR2
# PGW, RPT
/bin/tar cvf $TAPE_DEVICE_APPEND $BACKUPDIR/TDR2/$yestmonth01/$yest01*.*.TDR2.Z >> $LOG 2>>$TMP_LOG

# /data/TDR2_M
# PGW
/bin/tar cvf $TAPE_DEVICE_APPEND $BACKUPDIR/TDR2_M/$yestmonth01/$yest01*.TDR2_M.Z >> $LOG 2>>$TMP_LOG

# /data/IPMP
# LSMS, PGW, LNPDB, RPT
/bin/tar cvf $TAPE_DEVICE_APPEND $BACKUPDIR/IPMP/$yestmonth01/IPMP.log.$yest01.*.Z >> $LOG 2>>$TMP_LOG
}

# Main
status=""
chktape

if [ "$status" = "sense" ]
then
# Got tape inside
# Clear alarm
$SCRDIR/gen_alarm.sh 21 0 "$newhost No Tape detected"

# Start doing backup
echo "" | /bin/tee -a $LOG
echo "Tape Backup Start ($datestr $timestr) ..." | /bin/tee -a $LOG
echo "" | /bin/tee -a $LOG

#echo "Logs folder: [$LOGS]"
#for logfolder in $LOGS
#do
tarcmd >> $LOG
#done

if [ $? -ne 0 ]
then
# Put the stderr message generated from tarcmd() into the $LOG file
cat $TMP_LOG >> $LOG
echo "" | /bin/tee -a $LOG
echo "Tape Backup was failed" >> $LOG
#cd $BINDIR
$SCRDIR/gen_alarm.sh 23 1 "$newhost Tape backup error"
else
# Put the stderr message generated from tarcmd() into the $LOG file
cat $TMP_LOG >> $LOG
echo "" | /bin/tee -a $LOG
echo "Tape Backup Succeeded" >> $LOG
#cd $BINDIR
$SCRDIR/gen_alarm.sh 23 0 "$newhost Tape backup error"
fi

EOT=`cat $TMP_LOG | grep 'tar: write error: unexpected EOF' | wc -l`
if [ $EOT -ge 1 ]; then
# Tape Full
# Raise alarm
$SCRDIR/gen_alarm.sh 22 1 "$newhost Tape full"

echo "" | /bin/tee -a $LOG
echo "Tape Full ($datestr $timestr) ..." | /bin/tee -a $LOG
echo "" | /bin/tee -a $LOG
else
# Tape NOT Full
# Clear alarm
$SCRDIR/gen_alarm.sh 22 0 "$newhost Tape full"
fi

echo "" | /bin/tee -a $LOG
datestr=`date +%d-%m-%Y`
timestr=`date +%H:%M:%S`
echo "Tape Backup End ($datestr $timestr) ..." | /bin/tee -a $LOG
else
# No tape inside
# Raise alarm
$SCRDIR/gen_alarm.sh 21 1 "$newhost No Tape detected"

echo "" | /bin/tee -a $LOG
echo "Tape Backup Fail ($datestr $timestr) ..., unable to find tape" | /bin/tee -a $LOG
echo "" | /bin/tee -a $LOG
fi

# Housekeep the tape_bk_yyyymmdd.log
hk_month=`$BINDIR/lastdate 121 | /bin/awk '{print $1}' | /bin/cut -c1-6`
\rm $LOGDIR/TAPE/$currdate/tape_bk_$hk_month*.log
find $TEMP -name 'tape_backup.tmp' -exec rm {} \;
# 5  
Old 04-03-2012
The script appears to append up to 6 "tar" archives each run to a tape. All commands use the "no rewind" tape device, so if the tapes are cycled one would expect the tape to eventually become full. Maybe you recycle tapes with another process?

If you always use new backup tapes straight out of the wrapper, please mention this. Also please mention how big you think the total of the 6 "tar" archives is.


I'm not so sure that just testing of "status" (with "mt") and then looking for a single error message from "tar" is sufficient to determine if the tape is already full or where "tar" has failed.

For a problem tape, try running the command:
Code:
mt -f /dev/rmt/0n status

and then trying to see whether there is any clue that the tape is already full.

Also, the messages from "tar" (if it ran at all) are probably in the main backup log pointed to by $LOG.

Note: The cron command line redirection to /dev/null hides any output which is not trapped by the script. This would include any script syntax errors or untrapped output from commands.

The whole process would need very good record keeping if you wanted to restore a particular file from a particular named archive in a particular tape partition.

If the intention is only to ever have one backup set on one tape, then the script may need to start by rewinding the tape. Depends on what your backup strategy is and whether you change the tapes on a wide backup cycle like most sites.

Last edited by methyl; 04-03-2012 at 02:43 PM..
# 6  
Old 04-03-2012
I couldn't reformat the original post of the script because it contains too much extraneous HTML.

Here is the original script minus any surplus commented-out lines and indented for readability. This is not a corrected script, just a diagnostic so we can read it!

Code:
#!/bin/sh
#set -x

VERSION="Version: 1.06"
######################
# Choose the HOME dir
######################
HOME=/export/home/provadm
BINDIR=$HOME/bin
LOGDIR=$HOME/log
SCRDIR=$HOME/script
CONFDIR=$HOME/config
TEMP=$HOME/temp
#######################
# Choose the BACKUP dir
#######################
BACKUPDIR=/data
newhost=`hostname |sed 's/LIV_//'| sed 's/_01//'`
TAPE_DEVICE=/dev/rmt/0
TAPE_DEVICE_APPEND=/dev/rmt/0n
datestr=`date +%d-%m-%Y`
timestr=`date +%H:%M:%S`
event_day=`/bin/date +%Y%m%d`
currdate=`$HOME/bin/lastdate +1 | awk '{printf "%s",substr($0,1,6)}'`
LOG=$LOGDIR/TAPE/$currdate/tape_bk_$event_day.log
TMP_LOG=$TEMP/tape_backup.tmp
# Format yyyymmdd
yest01=`$BINDIR/lastdate 1 | /bin/awk '{print $1}'`
# Format MMM
mm=`$BINDIR/lastdate 1 | /bin/awk '{print $2}' | cut -c1-3`
# Format DD
dd=`$BINDIR/lastdate 1 | /bin/awk '{print $1}' | cut -c7-8`
# Format YYYY
yyyy=`$BINDIR/lastdate 1 | /bin/awk '{print $1}' | cut -c1-4`
# Format yyyymm
yestmonth01=`$BINDIR/lastdate 1 | /bin/awk '{print $1}' | cut -c1-6`
# ==============
# Date and Month
# ==============
REP_MONTH=`date "+%Y%m"`
REP_DATE=`date "+%Y%m%d"`
FIRST_DAY=`$BINDIR/lastdate | /bin/awk '{print $1}' | cut -c7-8`

chktape()
{
status=`/bin/mt -f $TAPE_DEVICE_APPEND status | grep 'sense' | awk '{print substr($1,1,5)}'`
}

tarcmd()
{
# Check the $TMP_LOG file if exists, if yes remove the file
if [ -f $TMP_LOG ]; then
	rm $TMP_LOG
fi
#/data1/ACCESS
#All Servers
/bin/tar cvf $TAPE_DEVICE_APPEND $BACKUPDIR/ACCESS/$yestmonth01/*$yest01*.tar.Z >> $LOG 2>>$TMP_LOG

if [ $FIRST_DAY = "01" ]; then
	#/data1/BACKUP
	# All Servers 
	/bin/tar cvf $TAPE_DEVICE_APPEND $BACKUPDIR/BACKUP/$yestmonth01/*_$yestmonth01.* >> $LOG 2>>$TMP_LOG
fi

#/data/BACKUP
# LNPDB, RPT
#/bin/tar cvf $TAPE_DEVICE_APPEND $BACKUPDIR/BACKUP/$yestmonth01/*$yest01*.tar.gz >> $LOG 2>>$TMP_LOG

#/data1/SYS
# All Servers
/bin/tar cvf $TAPE_DEVICE_APPEND $BACKUPDIR/SYS/$yestmonth01/$yest01*.Z >> $LOG 2>>$TMP_LOG

#/data/TDR2
# PGW, RPT
/bin/tar cvf $TAPE_DEVICE_APPEND $BACKUPDIR/TDR2/$yestmonth01/$yest01*.*.TDR2.Z >> $LOG 2>>$TMP_LOG

# /data/TDR2_M
# PGW
/bin/tar cvf $TAPE_DEVICE_APPEND $BACKUPDIR/TDR2_M/$yestmonth01/$yest01*.TDR2_M.Z >> $LOG 2>>$TMP_LOG

# /data/IPMP
# LSMS, PGW, LNPDB, RPT
/bin/tar cvf $TAPE_DEVICE_APPEND $BACKUPDIR/IPMP/$yestmonth01/IPMP.log.$yest01.*.Z >> $LOG 2>>$TMP_LOG
}

# Main
status=""
chktape
if [ "$status" = "sense" ]
then
	# Got tape inside
	# Clear alarm
	$SCRDIR/gen_alarm.sh 21 0 "$newhost No Tape detected"
	# Start doing backup
	echo "" | /bin/tee -a $LOG
	echo "Tape Backup Start ($datestr $timestr) ..." | /bin/tee -a $LOG
	echo "" | /bin/tee -a $LOG
	tarcmd >> $LOG
	if [ $? -ne 0 ]
	then
		# Put the stderr message generated from tarcmd() into the $LOG file
		cat $TMP_LOG >> $LOG
		echo "" | /bin/tee -a $LOG
		echo "Tape Backup was failed" >> $LOG
		$SCRDIR/gen_alarm.sh 23 1 "$newhost Tape backup error"
	else
		# Put the stderr message generated from tarcmd() into the $LOG file
		cat $TMP_LOG >> $LOG
		echo "" | /bin/tee -a $LOG
		echo "Tape Backup Succeeded" >> $LOG
		$SCRDIR/gen_alarm.sh 23 0 "$newhost Tape backup error"
	fi

	EOT=`cat $TMP_LOG | grep 'tar: write error: unexpected EOF' | wc -l`
	if [ $EOT -ge 1 ]; then
		# Tape Full
		# Raise alarm
		$SCRDIR/gen_alarm.sh 22 1 "$newhost Tape full"
		echo "" | /bin/tee -a $LOG
		echo "Tape Full ($datestr $timestr) ..." | /bin/tee -a $LOG
		echo "" | /bin/tee -a $LOG
	else
		# Tape NOT Full
		# Clear alarm
		$SCRDIR/gen_alarm.sh 22 0 "$newhost Tape full"
	fi

	echo "" | /bin/tee -a $LOG
	datestr=`date +%d-%m-%Y`
	timestr=`date +%H:%M:%S`
	echo "Tape Backup End ($datestr $timestr) ..." | /bin/tee -a $LOG
else
	# No tape inside
	# Raise alarm
	$SCRDIR/gen_alarm.sh 21 1 "$newhost No Tape detected"
	echo "" | /bin/tee -a $LOG
	echo "Tape Backup Fail ($datestr $timestr) ..., unable to find tape" | /bin/tee -a $LOG
	echo "" | /bin/tee -a $LOG
fi

# Housekeep the tape_bk_yyyymmdd.log
hk_month=`$BINDIR/lastdate 121 | /bin/awk '{print $1}' | /bin/cut -c1-6`
\rm $LOGDIR/TAPE/$currdate/tape_bk_$hk_month*.log
find $TEMP -name 'tape_backup.tmp' -exec rm {} \;


Last edited by methyl; 04-03-2012 at 02:46 PM..
# 7  
Old 04-03-2012
Code:
	tarcmd >> $LOG
	if [ $? -ne 0 ]

This bit looks very dodgy. I can't see anything in the function tarcmd which would cause it to exit with a non-zero status. If you want to test $? you need to detect the error inside the function immediately it happens, then issue a return 1.

Conclusion. The error handling for tar failures in the function tarcmd is not present at all.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. AIX

Nimol: full backup,is possible?

I have a NIMOL(nim on linux) server working. It do a full backup of my rootvg with those commands nimol_config -r -L old-2018 nimol_config -m /usr/bin/ssh -C -L new-2018 nimol_backup -c ibmaix -L new-2018 My question is: is possible like hp-ux do with "make_tape_recovery" a FULL... (1 Reply)
Discussion started by: Linusolaradm1
1 Replies

2. Homework & Coursework Questions

incremental or full backup ???

Hi. Can someone tell me if the following script that i have made is a script for INCREMENTAL BACKUP or FULL BACKUP. My teacher told me that is doing an FULL BACKUP. • find /etc /var /home -newer /backups/.backup_reference > /backups/.files_to_archive • touch /backups/.backup_reference • tar... (1 Reply)
Discussion started by: bender-alex
1 Replies

3. UNIX for Dummies Questions & Answers

incremental and full backup.. please help me

Hi, i'm new here(and a newbie) and i need some help with a project. I need to write a script for an incremental backup (this must be executed every day at 24:00) and a full backup (executed once a month) for etc/var/home directories. Can someone please help me with this? And a small explanation of... (9 Replies)
Discussion started by: bender-alex
9 Replies

4. Red Hat

Full OS-Backup

Hi, our redhat farm is growing (rhel 3 4 5) and the only backup we do is an incremental filesystembackup, which makes it very hard to restore the complete os is there a solution for a complete os-backup to a network server, like nim in aix, an the possibility to restore the whole red hat... (1 Reply)
Discussion started by: funksen
1 Replies

5. Solaris

full system backup

I have unix server with OS 5.8 ,,, I tried ufsdump 0ua -f /dev/rmt/0 / to perform full system backup on tape but I failed could any one give a procedure for full system backup on solaris machine using tapes??? (1 Reply)
Discussion started by: mm00123
1 Replies

6. Solaris

Full backup and Restore

Dear All ; first how are you every body I'm just subscribed in your forum and i hope i found what i searched for along time . I'm not a Solaris specialist but i read more to build a Network Management Station depends on Solaris as OS and it is working good now . my problem is how to perform... (16 Replies)
Discussion started by: Basha
16 Replies

7. UNIX for Dummies Questions & Answers

System full backup to tape

hi, Recently, I had receive one system. it's run on the unix tru64 server. I looking some log files when i know don't work few months age system backup to tape. Below error: INFO:Tape backups to tape tape0 starting for backup list: slot2:/disk4 Backup Command Variable... (0 Replies)
Discussion started by: Tlg13team
0 Replies

8. SCO

Backup to SCSI Tape Backup aborts

I am trying to make a full backup of my system using the cpio command. The Tape Unit is a SCSI DDS. The process started fine but after about 30 minutes, it just stopped and showed the following message: 1755 Signal 31 - Core dumped Any idea of what is causing this and how to fix it? ... (4 Replies)
Discussion started by: zionpc
4 Replies

9. UNIX for Dummies Questions & Answers

Check backup file size on backup tape

Hi, I performed backup on tape and I want to append more files to my previous backup on the same backup tape. But before I do that I need to know the backup file size of the first backup I performed so that I know the available size on the backup tape. Can someone help me what command I will use... (0 Replies)
Discussion started by: ayhanne
0 Replies

10. UNIX for Dummies Questions & Answers

Mounting DLT tape and to backup file to tape

Hi there: I'm new here Can anyone help me: I have DS15 Alpha server : Unix 5.1B Now i need to connect a DLVT VS80 1U Rackmount Tape Drive unit. What is the exact comman to mount the DLTape IV?? How do i make backuo @ copy file to the tape? Thanx to all (0 Replies)
Discussion started by: ayzeayem
0 Replies
Login or Register to Ask a Question