How to recover from failure with sconlinebackup in HP-UX 11.11

rpraveen1982 · #1 02-27-2012

Hi,

Can someone help me to come out of this error?
We have a HP-UX 11.11 system. I started a scbackup (online) as mirroring was enabled and the backup went on successfully. But from the scbackup logs I see that after the splitting of the logical volumes for online backup usually a MERGE-DELETE process gets activated on splitted Logical Volumes which doesn't seem to have started. As a result the backup logical volumes still exists

/dev/vg00/lvol3backup
262144 67696 182295 27% /sconlinebackup
/dev/vg00/lvol10backup
4849664 3203 4543558 0% /sconlinebackup/alcatel
/dev/vg00/lvol11backup
12451840 8729879 3489339 71% /sconlinebackup/alcatel/7.1
/dev/vg00/lvol4backup
9830400 54438 9164965 1% /sconlinebackup/home
/dev/vg00/lvol12backup
60227584 4112 59752984 0% /sconlinebackup/ldaphome
/dev/vg00/lvol5backup
15925248 5738955 9549650 38% /sconlinebackup/opt
/dev/vg00/lvol13backup
131072 63624 63233 50% /sconlinebackup/opt/JacORB2.1.3.2
/dev/vg00/lvol1backup
127573 41450 79744 34% /sconlinebackup/stand
/dev/vg00/lvol6backup
4915200 25505 4584090 1% /sconlinebackup/tmp
/dev/vg00/lvol7backup
11599872 1605092 9370107 15% /sconlinebackup/usr
/dev/vg00/lvol14backup
98369536 56545616 41497176 58% /sconlinebackup/usr/Systems/CMC_1_2.1_Master
/dev/vg00/lvol15backup
262144 35917 212088 14% /sconlinebackup/usr/Systems/Global_Instance_7.1.2_Master
/dev/vg00/lvol8backup
17563648 789674 15725601 5% /sconlinebackup/var
/dev/vg00/lvol9backup
2097152 1682 1964504 0% /sconlinebackup/var/adm/crash
laoaps,sys,root #

I paste below the scbackup logs

=====>>>>> 03-Nov-2011 23:46:52 Starting SCBACKUP procedure
( Backup-Restore Ver. 3.0.1 )
NOTE: > scbackup -t 0
=====>>>>> 03-Nov-2011 23:46:53 Ignite procedure activated
( Backup-Restore Ver. 3.0.1 )
03-Nov-2011 23:47:05 Start asking for the Ignite tape
03-Nov-2011 23:47:08 The request of the Ignite tape has been fulfilled
* Creating local directories for configuration files and archive.
======= 11/03/11 23:47:08 CTT Started make_tape_recovery. (Thu Nov 03
23:47:08 CTT 2011)
@(#)Ignite-UX Revision C.6.10.97
@(#)ignite/net_recovery (opt) $Revision: 10.834 $ $Date: 2006/06/14
13:47:40 $
* Testing for necessary pax patch.
* Passed pax tests.
* Checking Versions of Ignite-UX filesets
* Creating System Configuration.
* /opt/ignite/bin/save_config -f
/var/opt/ignite/recovery/2011-11-03,23:47/system_cfg vg00
* Backing Up Volume Group /dev/vg00
* /usr/sbin/vgcfgbackup /dev/vg00
* Creating Map Files for Volume Group /dev/vg00
* /usr/sbin/vgexport -s -p -m /etc/lvmconf/vg00.mapfile /dev/vg00
* Creating Control Configuration.
* Creating Archive File List
* Creating Archive Configuration
* /opt/ignite/lbin/make_arch_config -c
/var/opt/ignite/recovery/2011-11-03,23:47/archive_cfg -g
/var/opt/ignite/recovery/2011-11-03,23:47/flist -n 2011-11-03,23:47 -r
pa -b 64 -d Recovery\ Archive -t 1 -i 1 -m t
* Saving the information about archive to
/var/opt/ignite/recovery/previews
*

* **** This is just Preview Only. ****
* **** No archive has been created yet. ****
* **** You may modify the configure files ****
* **** and use -r option for resuming to create archive. ****

======= 11/03/11 23:47:28 CTT make_tape_recovery completed successfully!
* Archive created and rewind the tape to beginning of tape.

NOTE: ******* RESUMING *******
NOTE: archive size: 1557500 kB
NOTE: last preview is on 2011-11-03,23:47
NOTE: The archive name is 2011-11-03,23:47
NOTE: The archive description Recovery Archive
NOTE: The archive is on local tape drive /dev/rmt/0mn
NOTE: The pax option is tar
NOTE: You want to resume archive creation from the configuration files
locally on this system.
NOTE: In Resume Mode. Do you wish to continue?
Continue?(y/n) * Creating The Tape Archive
50266+0 records in
50266+0 records out
* /opt/ignite/data/scripts/make_sys_image -c n -d /dev/rmt/0mn -t n -s
local -n 2011-11-03,23:47 -m t -w
/var/opt/ignite/recovery/latest/recovery.log -u -R -g
/var/opt/ignite/recovery/latest/flist -a 1557500

* Preparing to create a system archive.
* Archiving contents of laoaps via tar image to local device/dev/rmt/0mn.
* Creation of system archive complete.
　
======= 11/03/11 23:53:21 CTT make_tape_recovery completed successfully!
03-Nov-2011 23:53:49 Please, label the Ignite Recovery Tape as:
laoaps - Ignite Bootable - 03-Nov-2011 23:47
=====>>>>> 03-Nov-2011 23:53:49 Ignite Procedure terminated
=====>>>>> 04-Nov-2011 01:53:58 On-line backup procedure activated
04-Nov-2011 01:54:20 Starting Logical Volumes SPLIT operation ...
Processing ROOT LOGICAL VOLUME /dev/vg00/lvol3
Logical volume "/dev/vg00/lvol3backup" has been successfully created with
character device "/dev/vg00/rlvol3backup".
Logical volume "/dev/vg00/lvol3" has been successfully split.
Volume Group configuration for /dev/vg00 has been saved in /etc/lvmconf/vg00.conf
Checking FS on splitted Logical Volume
log replay in progress
replay complete - marking super-block as CLEAN
Processing LOGICAL VOLUME /dev/vg00/lvol10
Logical volume "/dev/vg00/lvol10backup" has been successfully created with
character device "/dev/vg00/rlvol10backup".
Logical volume "/dev/vg00/lvol10" has been successfully split.
Volume Group configuration for /dev/vg00 has been saved in /etc/lvmconf/vg00.conf
Checking FS on splitted Logical Volume
log replay in progress
replay complete - marking super-block as CLEAN
Processing LOGICAL VOLUME /dev/vg00/lvol11
Logical volume "/dev/vg00/lvol11backup" has been successfully created with
character device "/dev/vg00/rlvol11backup".
Logical volume "/dev/vg00/lvol11" has been successfully split.
Volume Group configuration for /dev/vg00 has been saved in /etc/lvmconf/vg00.conf
Checking FS on splitted Logical Volume
log replay in progress
replay complete - marking super-block as CLEAN
Processing LOGICAL VOLUME /dev/vg00/lvol4
Logical volume "/dev/vg00/lvol4backup" has been successfully created with
character device "/dev/vg00/rlvol4backup".
Logical volume "/dev/vg00/lvol4" has been successfully split.
Volume Group configuration for /dev/vg00 has been saved in /etc/lvmconf/vg00.conf
Checking FS on splitted Logical Volume
log replay in progress
replay complete - marking super-block as CLEAN
Processing LOGICAL VOLUME /dev/vg00/lvol12
Logical volume "/dev/vg00/lvol12backup" has been successfully created with
character device "/dev/vg00/rlvol12backup".
Logical volume "/dev/vg00/lvol12" has been successfully split.
Volume Group configuration for /dev/vg00 has been saved in /etc/lvmconf/vg00.conf
Checking FS on splitted Logical Volume
log replay in progress
replay complete - marking super-block as CLEAN
Processing LOGICAL VOLUME /dev/vg00/lvol5
Logical volume "/dev/vg00/lvol5backup" has been successfully created with
character device "/dev/vg00/rlvol5backup".
Logical volume "/dev/vg00/lvol5" has been successfully split.
Volume Group configuration for /dev/vg00 has been saved in /etc/lvmconf/vg00.conf
Checking FS on splitted Logical Volume
log replay in progress
replay complete - marking super-block as CLEAN
Processing LOGICAL VOLUME /dev/vg00/lvol13
Logical volume "/dev/vg00/lvol13backup" has been successfully created with
character device "/dev/vg00/rlvol13backup".
Logical volume "/dev/vg00/lvol13" has been successfully split.
Volume Group configuration for /dev/vg00 has been saved in /etc/lvmconf/vg00.conf
Checking FS on splitted Logical Volume
log replay in progress
replay complete - marking super-block as CLEAN
Processing LOGICAL VOLUME /dev/vg00/lvol1
Logical volume "/dev/vg00/lvol1backup" has been successfully created with
character device "/dev/vg00/rlvol1backup".
Logical volume "/dev/vg00/lvol1" has been successfully split.
Volume Group configuration for /dev/vg00 has been saved in /etc/lvmconf/vg00.conf
Checking FS on splitted Logical Volume
** /dev/vg00/lvol1backup
** Last Mounted on /stand
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
119 files, 0 icont, 41450 used, 86123 free (291 frags, 10729 blocks)
***** MARKING FILE SYSTEM CLEAN *****
***** FILE SYSTEM WAS MODIFIED *****
Processing LOGICAL VOLUME /dev/vg00/lvol6
Logical volume "/dev/vg00/lvol6backup" has been successfully created with
character device "/dev/vg00/rlvol6backup".
Logical volume "/dev/vg00/lvol6" has been successfully split.
Volume Group configuration for /dev/vg00 has been saved in /etc/lvmconf/vg00.conf
Checking FS on splitted Logical Volume
log replay in progress
replay complete - marking super-block as CLEAN
Processing LOGICAL VOLUME /dev/vg00/lvol7
Logical volume "/dev/vg00/lvol7backup" has been successfully created with
character device "/dev/vg00/rlvol7backup".
Logical volume "/dev/vg00/lvol7" has been successfully split.
Volume Group configuration for /dev/vg00 has been saved in /etc/lvmconf/vg00.conf
Checking FS on splitted Logical Volume
log replay in progress
replay complete - marking super-block as CLEAN
Processing LOGICAL VOLUME /dev/vg00/lvol14
Logical volume "/dev/vg00/lvol14backup" has been successfully created with
character device "/dev/vg00/rlvol14backup".
Logical volume "/dev/vg00/lvol14" has been successfully split.
Volume Group configuration for /dev/vg00 has been saved in /etc/lvmconf/vg00.conf
Checking FS on splitted Logical Volume
log replay in progress
replay complete - marking super-block as CLEAN
Processing LOGICAL VOLUME /dev/vg00/lvol15
Logical volume "/dev/vg00/lvol15backup" has been successfully created with
character device "/dev/vg00/rlvol15backup".
Logical volume "/dev/vg00/lvol15" has been successfully split.
Volume Group configuration for /dev/vg00 has been saved in /etc/lvmconf/vg00.conf
Checking FS on splitted Logical Volume
log replay in progress
replay complete - marking super-block as CLEAN
Processing LOGICAL VOLUME /dev/vg00/lvol8
Logical volume "/dev/vg00/lvol8backup" has been successfully created with
character device "/dev/vg00/rlvol8backup".
Logical volume "/dev/vg00/lvol8" has been successfully split.
Volume Group configuration for /dev/vg00 has been saved in /etc/lvmconf/vg00.conf
Checking FS on splitted Logical Volume
log replay in progress
replay complete - marking super-block as CLEAN
Processing LOGICAL VOLUME /dev/vg00/lvol9
Logical volume "/dev/vg00/lvol9backup" has been successfully created with
character device "/dev/vg00/rlvol9backup".
Logical volume "/dev/vg00/lvol9" has been successfully split.
Volume Group configuration for /dev/vg00 has been saved in /etc/lvmconf/vg00.conf
Checking FS on splitted Logical Volume
log replay in progress
replay complete - marking super-block as CLEAN
04-Nov-2011 01:54:52 ... Logical Volumes SPLIT operation ended successfully
04-Nov-2011 01:55:11 Start asking for the tape Volume #1
04-Nov-2011 01:55:11 The request for Volume #1 has been fulfilled

Can someone help me how to come out of this as I'm afraid the next time I start the backup it will fail.

methyl · #2 02-27-2012

Did this problem happen on 03-04 Nov 2011 , or have you posted the wrong log?

methyl · #3 02-27-2012

Quote:

======= 11/03/11 23:53:21 CTT make_tape_recovery completed successfully!
03-Nov-2011 23:53:49 Please, label the Ignite Recovery Tape as:
laoaps - Ignite Bootable - 03-Nov-2011 23:47
=====>>>>> 03-Nov-2011 23:53:49 Ignite Procedure terminated

=====>>>>> 04-Nov-2011 01:53:58 On-line backup procedure activated
04-Nov-2011 01:54:20 Starting Logical Volumes SPLIT operation ...

There is a full day between these events. Did a second backup start and not finish?

First check every one of the mounted "backup" partitions with "fuser" on the mountpoint and make sure that no process whatsover has any of these partitions open. This is in case your previous backup is still running somehow and perhaps waiting for a tape?
Your vg00 is the largest I have ever seen (it would normally be just Operating System). Does it actually fit on a tape?

The outline plan is to work up your list of "backup" mountpoints from the bottom up ensuring that you "umount" in the reverse order of the original "mount".

Quote:

dev/vg00/lvol9backup
2097152 1682 1964504 0% /sconlinebackup/var/adm/crash

Outline of what do do, but please DYOR and get a colleague to double check every command before typing.

Code:

Using the least volatile partition on your list as an example:

# Look at "Mirror copies" and check that it is definitely one less than is normal.
/usr/sbin/lvdisplay /dev/vg00/lvol9
#
# Unmount the filesystem
/usr/sbin/umount /sconlinebackup/var/adm/crash
#
# Merge the split mirror back:
/usr/sbin/lvmerge /dev/vg00/lvol9backup /dev/vg00/lvol9
#
# Look at "Mirror copies" and check that is it now normal
/usr/sbin/lvdisplay /dev/vg00/lvol9

Beware that "lvmerge" is very quick for a filesystem which has not changed but could
 take an hour or more for a Logical Volume which is now quite different.
In my experience you can only run one "lvmerge" at any one time.

rpraveen1982 · #4 02-28-2012

Thank you methyl for your immediate reply. I shall do the steps suggested by you and give you a feedback.

---------- Post updated 02-28-12 at 09:41 AM ---------- Previous update was 02-27-12 at 10:27 PM ----------

Hi Methyl,

Thanks your proposal did work. Also I saw there is another option in scbackup i.e. scbackup -R that also does the MERGE-DELETE function.

Thread Tools	Search this Thread
Show Printable Version Email this Page Subscribe to this Thread	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Recover solaris 9	newsol	Solaris	0	10-11-2011 09:33 AM
Solaris 10 MPXIO doesn't recover path failure	UKPhil	Solaris	3	12-07-2010 03:30 PM
using the recover command	brian112	UNIX for Dummies Questions & Answers	2	07-07-2010 10:58 AM
recover after delete using rm -rf	jenovaux	Solaris	5	07-08-2009 07:29 AM
how to recover with awk	mips	Shell Programming and Scripting	1	04-08-2004 10:18 AM

The Following User Says Thank You to rpraveen1982 For This Useful Post:
methyl (02-28-2012)