13 disk raidz2 pool lost Post: 302707289

Sponsored Content

Operating Systems Solaris 13 disk raidz2 pool lost Post 302707289 by tatxo on Friday 28th of September 2012 10:05:05 AM

09-28-2012

Registered User

13 disk raidz2 pool lost

Hi guys, I appreciate any help in this regard, we have lost sensitive data in the company.

One box with 2 disk mirrored and a 3ware controller handling 13 disks in a raidz2 pool. Suddenly the box restart and keeps "Reading ZFS config" for hours.

Unplugging disk by disk we isolate the disk was causing the system not to be able to restar and we execute 'zpool clear -F' as suggested by 'zpool status' command. During hours of proccess we get a console error from the controller, and the system hangs, so we decide to change such disk, getting the pool from DEGRADED to FAULTED. After one 'zpool clear' we get the pool again DEGRADED, but no access to data, so we try to roll back with previous disks. (we didn't commit any 'zpool replace').

The box keeps restarting, freezing and unable to boot, so we decide to plug the original 13 disks in another box with same hardware.

Now we are trying to import the pool here, after hours of proccess and huge disk activity, the box hangs and the import doesn't succeed. This is the result of 'zpool import' command:

Code:

state: DEGRADED
status: The pool was last accessed by another system.
action: The pool can be imported despite missing or damaged devices.  The
        fault tolerance of the pool may be compromised if imported.
   see: http://www.sun.com/msg/ZFS-8000-EY
config:

        zsan08rz2     DEGRADED
          raidz2-0    DEGRADED
            c10t2d0   FAULTED  corrupted data
            c10t2d0   ONLINE
            c10t5d0   ONLINE
            c10t9d0   ONLINE
            c10t0d0   ONLINE
            c10t1d0   ONLINE
            c10t4d0   ONLINE
            c10t8d0   ONLINE
            c10t12d0  ONLINE
            c10t11d0  ONLINE
            c10t3d0   ONLINE
            c10t7d0   ONLINE
            c10t6d0   ONLINE

Any ideas? Note that c10t2d0 is duplicated, and note that during las import process we got this error from the controller in the console:

Code:

zsan08 tw: WARNING: tw0: tw_aen_task AEN 0x000a Drive error detected unit=7 port=13

This drive seems to be different than the drive c10t2d0.

Suggestions? Thanks!

tatxo

View Public Profile for tatxo

Find all posts by tatxo

7 More Discussions You Might Find Interesting

1. Infrastructure Monitoring

zfs - migrate from pool to pool

Here are the details. cnjr-opennms>root$ zfs list NAME USED AVAIL REFER MOUNTPOINT openpool 20.6G 46.3G 35.5K /openpool openpool/ROOT 15.4G 46.3G 18K legacy openpool/ROOT/rds 15.4G 46.3G 15.3G / openpool/ROOT/rds/var 102M ...

2. Ubuntu

Disk Space lost mysteriously upon breaking a process.

Hi All, Today when I was working on a script to generate custom wordlist. So I ran a script and the output was directed to /tmp. The disk space was around 19 gb. While the script was running, I decided to direct the o/p file to my 1TB drive. So I broke the run using Ctrl + C. Now when I...

3. Boot Loaders

Lost MBR on disk

trying to recover a lost partition table, where the signature (0x55AA) has been lost, though attempting to restore using a number of tools (fdisk, testdisk et al) the write fails. also the os is unable to read the disk geometry correctly, after attempting to correct the geometry, the updated...

4. Solaris

zfs raidz2 - insufficient replicas

I lost my system volume in a power outage, but fortunately I had a dual boot and I could boot into an older opensolaris version and my raidz2 7 drive pool was still fine. I even scrubbed it, no errors. However, the older os has some smb problems so I wanted to upgrade to opensolaris11. I...

5. Solaris

Lost Root Password on VXVM Encapsulated Root Disk

Hi All Hope it's okay to post on this sub-forum, couldn't find a better place I've got a 480R running solaris 8 with veritas volume manager managing all filesystems, including an encapsulated root disk (I believe the root disk is encapsulated as one of the root mirror disks has an entry under...

6. Solaris

Need to remove a disk from zfs pool

I accidently added a disk in different zpool instead of pool, where I want. root@prtdrd21:/# zpool status cvfdb2_app_pool pool: cvfdb2_app_pool state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM cvfdb2_app_pool ONLINE 0 0 0...

7. Solaris

How to clear a removed single-disk pool from being listed by zpool import?

On an OmniOS server, I removed a single-disk pool I was using for testing. Now, when I run zpool import it will show it as FAULTED, since that single disk not available anymore. # zpool import pool: fido id: 7452075738474086658 state: FAULTED status: The pool was last...

LEARN ABOUT DEBIAN

zdb

zdb(8)							  System Administration Commands						    zdb(8)

NAME

       zdb - ZFS debugger

SYNOPSIS

       zdb pool

DESCRIPTION

       The  zdb command is used by support engineers to diagnose failures and gather statistics. Since the ZFS file system is always consistent on
       disk and is self-repairing, zdb should only be run under the direction by a support engineer.

       If no arguments are specified, zdb, performs basic consistency checks on  the  pool  and  associated  datasets,	and  report  any  problems
       detected.

       Any options supported by this command are internal to Sun and subject to change at any time.

EXIT STATUS

       The following exit values are returned:

       0    The pool is consistent.

       1    An error was detected.

       2    Invalid command line options were specified.

ATTRIBUTES

       See attributes(5) for descriptions of the following attributes:

       +-----------------------------+-----------------------------+
       |      ATTRIBUTE TYPE	     |	    ATTRIBUTE VALUE	   |
       +-----------------------------+-----------------------------+
       |Availability		     |SUNWzfsu			   |
       +-----------------------------+-----------------------------+
       |Interface Stability	     |Unstable			   |
       +-----------------------------+-----------------------------+

SEE ALSO

       zfs(8), zpool(8), attributes(5)

SunOS 5.11							    31 Oct 2005 							    zdb(8)

7 More Discussions You Might Find Interesting

1. Infrastructure Monitoring

zfs - migrate from pool to pool

Discussion started by: pupp

2. Ubuntu

Disk Space lost mysteriously upon breaking a process.

Discussion started by: morningSunshine

3. Boot Loaders

Lost MBR on disk

Discussion started by: xaphan

4. Solaris

zfs raidz2 - insufficient replicas

Discussion started by: skk