zfs raidz2 - insufficient replicas Post: 302584000

Sponsored Content

Operating Systems Solaris zfs raidz2 - insufficient replicas Post 302584000 by skk on Thursday 22nd of December 2011 07:36:27 AM

12-22-2011

Registered User

I thought I was OK, but zpool scrub hangs forever at 20% across multiple cold boots, and importing from oracle solaris 11 live hangs forever... I what do I do now, older versions of open solaris will not import the pool anymore...

---------- Post updated at 07:36 AM ---------- Previous update was at 12:48 AM ----------

Well it's 4AM and now I am getting mad. I think this whole bloody mess is caused by Oracle's 'great new solaris 11' package. My advice, don't touch this piece.

To recap, I wanted to try the best newest Solaris release for my super critical file server, so I downloaded from Oracle what I thought was the right iso. I booted, and just as the boot menu came up I got a phone call. When I came back the thing had automatically installed a new version of solaris over one of the drives in the zpool. (BAD #1 ).

Since I have a raidz2 I wasn't too worried ( at first ) and I booted into my original 2009 opensolaris. However, I got the errors shown in the above posts. I exported and could not re-import my pool. Oracale had done something to that drive breaking my entire pool even though it is a radz2 and should be tolerant of 2 drive failures at the same time. ( BAD #2 ).

Since I could do nothing with my pool with my old OS, I tried on the new Oracle solaris, and could indeed import my pool in degraded state because of the one overwritten drive. Fine, I wanted to scrub everything first ( I don't know if this was wise or not ) so I did zpool scrub, which eventually hung forever at 11%. All access to the drive was similarly hung. Rebooting the machine did not change this situation which seemed increasingly dire ( BAD #3 )

I finally got out of this problem by unplugging drives to fault the pool and rebooting in single user mode. Eventually I was able to stop the scrub with the "zpool scrub -s" command, in single user mode. And when I rebooted, I could access my pool again. My first priority at this point was to back up all data immediately. I began to copy off my most important stuff, but unfortunately before I could copy even a fraction off the file system hung again ( BAD #4 ).

Googling around I found most causes for hanging zpool commands are related to hardware failure, so going on a hunch, I figured Oracle phased out or screwed up the drivers for my disks. I still could not import my pool in the old opensolaris OS, because of whatever the Oracle install wrote on that drive. So I booted in Oracle solaris, in single user mode, and did "zpool offline <pool> <drive>" and it worked! Then I rebooted into good old opensolaris, and imported my pool. It worked!!!!

So at this point it is now more like 5AM and I have backed up most of my critical data, way more than I could before anyway. It appears at this point that I was correct and the drivers for either my motherboard or my hard drive controller card were broken by the Oracle release in a way that let it silently trash my zpool. I have a SIIG 2 drive sata and a MSI n1996 motherboard, not sure which the problem is with, but whichever, it works fine in opensolaris 2009 and previous versions.

I just want to warn people that are not real Solaris experts from even trying this Oracle package. Personally I am migrating to fbsd as soon as I can...

skk

View Public Profile for skk

Find all posts by skk

9 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

SDS and replicas

Hello, We are using Solstice Disk Suite on Solaris 2.7. We want to add two striped volume with six disks. On each disk, we take a slice and we create the stripe. That I want to know : Is it necessary to add two replicas on the same slice on the new disks, as made before on the others...

2. HP-UX

Insufficient permissions on ftp'ed files from WIN2K to HP-UX

We have an application running on Win2K and this application ftp files to HP-UX using ftpdc as user id. The files are created in HP-UX with following permissions: -rw-r----- 1 ftpdc users 968321 Apr 12 22:57 aaaa.txt There is a job that runs on HP-UX trying to modify this file using the...

3. Solaris

insufficient metadevice database replicas ERROR

First I would like to thank this forum for assisting me in setting up my 1st sunbox. Could not have done it if it had not been for you guys and google :D I have mirrored my box and have SUCCESSFULLY tested booting from the rootdisk and rootmirror successfully. I am now looking at configuring...

4. Solaris

Problem with DiskSuite replicas

Good morning, I have Solstice disk suite installed on my server. One disk broke so I sostitute it. A replica was present on this disk. I had delete and then recreate it with commands metadb -d and metadb -a. Now when I inquire for the status of replicas I see this: stp11# metadb -i ...

5. UNIX for Advanced & Expert Users

insufficient available memory

Hi, when navigating on application webpages (peoplesoft applications) the users receive : INSUFFICIENT AVAILABLE MEMORY. I issued vmstat on UNIX server ( where is hosted Web server and application server and DB). Here is the result : :vmstat 2 10 System configuration: lcpu=4 mem=30720MB...

6. UNIX for Advanced & Expert Users

Issue with insufficient swap or memory space

Hi, When I execute one of my shellscript I am getting the below mentioned error message .This application takes 2input files which have the records counts 26463 and 1178046 exec(2): insufficient swap or memory available. exec(2): insufficient swap or memory available. exec(2): insufficient swap...

7. Solaris

metadevices: how to test metadb (how to corrupt replicas/understanding replicas)

Hi all, I recently started exploring Solaris 10. I am testing metadevices now. I have been reading about the state databases here: 6.State Database (Overview) (Solaris Volume Manager Administration Guide) - Sun Microsystems So I created 3 metadbs on 2 slices (6 in total; c1t1d0s3...

8. Solaris

Solaris Volume Manger - Database Replicas Question - Benefits of Increasing Default Size?

Hey all! I was hoping someone knew anything about this one... I know with Solaris Volume Manager the default Database Replica size is 8192 blocks (4MB approximately) Now I know you can increase this amount but is there any point? The reason I am asking this is that I've setup mirroring on...

9. Solaris

13 disk raidz2 pool lost

Hi guys, I appreciate any help in this regard, we have lost sensitive data in the company. One box with 2 disk mirrored and a 3ware controller handling 13 disks in a raidz2 pool. Suddenly the box restart and keeps "Reading ZFS config" for hours. Unplugging disk by disk we isolate the disk...

LEARN ABOUT CENTOS

amzfs-snapshot

AMZFS-SNAPSHOT(8)					  System Administration Commands					 AMZFS-SNAPSHOT(8)

NAME

       amzfs-snapshot - Amanda script to create zfs snapshot

DESCRIPTION

       amzfs-snapshot is an Amanda script implementing the Script API. It should not be run by users directly. It create a zfs snapshot of the
       filesystem where the path specified is mounted.

       PRE-DLE-* create a snapshot and the POST-DLE-* destroy the snapshot, *-DLE-AMCHECK, *-DLE-ESTIMATE and *-DLE-BACKUP must be set to be
       executed on the client:

	   execute-on  pre-dle-amcheck, post-dle-amcheck, pre-dle-estimate, post-dle-estimate, pre-dle-backup, post-dle-backup
	   execute-where client

       The PRE_DLE_* script output a DIRECTORY property telling where the directory is located in the snapshot. The application must be able to
       use the DIRECTORY property, amgtar can do it.

       The script is run as the amanda user, it must have the priviledge to create and destroy snapshot:

	   zfs allow -ldu AMANDA_USER mount,snapshot,destroy FILESYSTEM

       Some system doesn't have "zfs allow", but you can give the Amanda backup user the rights to manipulate ZFS filesystems by using the
       following command:

       usermod -P "ZFS File System Management,ZFS Storage Management" AMANDA_USER
       This will require that your run zfs under pfexec, set the PFEXEC property to YES.

       The format of the DLE must be one of:

       Desciption	       Example
       ----------	       -------
       Mountpoint	       /data
       Arbitrary mounted dir   /data/interesting_dir
       ZFS pool name	       datapool
       ZFS filesystem	       datapool/database
       ZFS logical volume      datapool/dbvol

       The filesystem must be mounted.

PROPERTIES

       This section lists the properties that control amzfs-snapshot's functionality. See amanda-scripts(7) for information on the Script API,
       script configuration.

       DF-PATH

	   Path to the 'df' binary, search in $PATH by default.

       ZFS-PATH

	   Path to the 'zfs' binary, search in $PATH by default.

       PFEXEC-PATH

	   Path to the 'pfexec' binary, search in $PATH by default.

       PFEXEC

	   If "NO" (the default), pfexec is not used, if set to "YES" then pfexec is used.

EXAMPLE

       In this example, a dumptype is defined to use amzfs-snapshot script to create a snapshot and use amgtar to backup the snapshot.
	define script-tool amzfs_snapshot {
	  comment "backup of zfs snapshot"
	  plugin  "amzfs-snapshot"
	  execute-on  pre-dle-amcheck, post-dle-amcheck, pre-dle-estimate, post-dle-estimate, pre-dle-backup, post-dle-backup
	  execute-where client
	  #property "DF-PATH" "/usr/sbin/df"
	  #property "ZFS-PATH" "/usr/sbin/zfs"
	  #property "PFEXEC-PATH" "/usr/sbin/pfexec"
	  #property "PFEXEC" "NO"
	}

	define dumptype user-zfs-amgtar {
	  dt_amgtar
	  script "amzfs_snapshot"
	}

SEE ALSO

       amanda(8), amanda.conf(5), amanda-client.conf(5), amanda-scripts(7)

       The Amanda Wiki: : http://wiki.zmanda.com/

AUTHORS

       Jean-Louis Martineau <martineau@zmanda.com>
	   Zmanda, Inc. (http://www.zmanda.com)

       Dustin J. Mitchell <dustin@zmanda.com>
	   Zmanda, Inc. (http://www.zmanda.com)

Amanda 3.3.3							    01/10/2013							 AMZFS-SNAPSHOT(8)