13 disk raidz2 pool lost Post: 302707289

Sponsored Content

Operating Systems Solaris 13 disk raidz2 pool lost Post 302707289 by tatxo on Friday 28th of September 2012 10:05:05 AM

09-28-2012

Registered User

13 disk raidz2 pool lost

Hi guys, I appreciate any help in this regard, we have lost sensitive data in the company.

One box with 2 disk mirrored and a 3ware controller handling 13 disks in a raidz2 pool. Suddenly the box restart and keeps "Reading ZFS config" for hours.

Unplugging disk by disk we isolate the disk was causing the system not to be able to restar and we execute 'zpool clear -F' as suggested by 'zpool status' command. During hours of proccess we get a console error from the controller, and the system hangs, so we decide to change such disk, getting the pool from DEGRADED to FAULTED. After one 'zpool clear' we get the pool again DEGRADED, but no access to data, so we try to roll back with previous disks. (we didn't commit any 'zpool replace').

The box keeps restarting, freezing and unable to boot, so we decide to plug the original 13 disks in another box with same hardware.

Now we are trying to import the pool here, after hours of proccess and huge disk activity, the box hangs and the import doesn't succeed. This is the result of 'zpool import' command:

Code:

state: DEGRADED
status: The pool was last accessed by another system.
action: The pool can be imported despite missing or damaged devices.  The
        fault tolerance of the pool may be compromised if imported.
   see: http://www.sun.com/msg/ZFS-8000-EY
config:

        zsan08rz2     DEGRADED
          raidz2-0    DEGRADED
            c10t2d0   FAULTED  corrupted data
            c10t2d0   ONLINE
            c10t5d0   ONLINE
            c10t9d0   ONLINE
            c10t0d0   ONLINE
            c10t1d0   ONLINE
            c10t4d0   ONLINE
            c10t8d0   ONLINE
            c10t12d0  ONLINE
            c10t11d0  ONLINE
            c10t3d0   ONLINE
            c10t7d0   ONLINE
            c10t6d0   ONLINE

Any ideas? Note that c10t2d0 is duplicated, and note that during las import process we got this error from the controller in the console:

Code:

zsan08 tw: WARNING: tw0: tw_aen_task AEN 0x000a Drive error detected unit=7 port=13

This drive seems to be different than the drive c10t2d0.

Suggestions? Thanks!

tatxo

View Public Profile for tatxo

Find all posts by tatxo

7 More Discussions You Might Find Interesting

1. Infrastructure Monitoring

zfs - migrate from pool to pool

Here are the details. cnjr-opennms>root$ zfs list NAME USED AVAIL REFER MOUNTPOINT openpool 20.6G 46.3G 35.5K /openpool openpool/ROOT 15.4G 46.3G 18K legacy openpool/ROOT/rds 15.4G 46.3G 15.3G / openpool/ROOT/rds/var 102M ...

2. Ubuntu

Disk Space lost mysteriously upon breaking a process.

Hi All, Today when I was working on a script to generate custom wordlist. So I ran a script and the output was directed to /tmp. The disk space was around 19 gb. While the script was running, I decided to direct the o/p file to my 1TB drive. So I broke the run using Ctrl + C. Now when I...

3. Boot Loaders

Lost MBR on disk

trying to recover a lost partition table, where the signature (0x55AA) has been lost, though attempting to restore using a number of tools (fdisk, testdisk et al) the write fails. also the os is unable to read the disk geometry correctly, after attempting to correct the geometry, the updated...

4. Solaris

zfs raidz2 - insufficient replicas

I lost my system volume in a power outage, but fortunately I had a dual boot and I could boot into an older opensolaris version and my raidz2 7 drive pool was still fine. I even scrubbed it, no errors. However, the older os has some smb problems so I wanted to upgrade to opensolaris11. I...

5. Solaris

Lost Root Password on VXVM Encapsulated Root Disk

Hi All Hope it's okay to post on this sub-forum, couldn't find a better place I've got a 480R running solaris 8 with veritas volume manager managing all filesystems, including an encapsulated root disk (I believe the root disk is encapsulated as one of the root mirror disks has an entry under...

6. Solaris

Need to remove a disk from zfs pool

I accidently added a disk in different zpool instead of pool, where I want. root@prtdrd21:/# zpool status cvfdb2_app_pool pool: cvfdb2_app_pool state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM cvfdb2_app_pool ONLINE 0 0 0...

7. Solaris

How to clear a removed single-disk pool from being listed by zpool import?

On an OmniOS server, I removed a single-disk pool I was using for testing. Now, when I run zpool import it will show it as FAULTED, since that single disk not available anymore. # zpool import pool: fido id: 7452075738474086658 state: FAULTED status: The pool was last...

LEARN ABOUT XFREE86

systemd-networkd-wait-online.service

SYSTEMD-NETWORKD-WAIT-ONLINE.SERVICE(8) 		     systemd-networkd.service			   SYSTEMD-NETWORKD-WAIT-ONLINE.SERVICE(8)

NAME

       systemd-networkd-wait-online.service, systemd-networkd-wait-online - Wait for network to come online

SYNOPSIS

       systemd-networkd-wait-online.service

       /lib/systemd/systemd-networkd-wait-online

DESCRIPTION

       systemd-networkd-wait-online is a one-shot system service that waits for the network to be configured. By default, it will wait for all
       links it is aware of and which are managed by systemd-networkd.service(8) to be fully configured or failed, and for at least one link to
       gain a carrier.

OPTIONS

       The following options are understood:

       -i, --interface=
	   Network interface to wait for before deciding if the system is online. This is useful when a system has several interfaces which will
	   be configured, but a particular one is necessary to access some network resources. This option may be used more than once to wait for
	   multiple network interfaces. When used, all other interfaces are ignored.

       --ignore=
	   Network interfaces to be ignored when deciding if the system is online. By default, only the loopback interface is ignored. This option
	   may be used more than once to ignore multiple network interfaces.

       --timeout=
	   Fail the service if the network is not online by the time the timeout elapses. A timeout of 0 disables the timeout. Defaults to 120
	   seconds.

SEE ALSO

       systemd(1), systemd-networkd.service(8)

systemd 237												   SYSTEMD-NETWORKD-WAIT-ONLINE.SERVICE(8)