Serial Split Brain detected in solaris10


 
Thread Tools Search this Thread
Operating Systems Solaris Serial Split Brain detected in solaris10
# 1  
Old 04-02-2010
Serial Split Brain detected in solaris10

Hello,

I have connected storedge d2 to V440 running solaris 10. I had a raid 5 volume created using VXVM software. This morning when I rebooted the system, I get below error,

Code:
Apr  2 14:21:10 servername vxvm:vxconfigd: [ID 702911 daemon.notice] V-5-1-9576 Split Brain. da id is 0.1, while dm id is 0.0 for dm myapp-dg01
Apr  2 14:21:10 servername last message repeated 1 time
Apr  2 14:21:10 servername vxvm:vxconfigd: [ID 702911 daemon.notice] V-5-1-9576 Split Brain. da id is 0.1, while dm id is 0.0 for dm myapp-dg02
Apr  2 14:21:10 servername vxvm:vxconfigd: [ID 702911 daemon.error] V-5-1-569 Disk group myapp-dg, Disk c2t2d0s2: Cannot auto-import group:
Apr  2 14:21:10 servername vxvm:vxconfigd: [ID 702911 daemon.error]        Serial Split Brain detected. Run vxsplitlines to import the diskgroup


/usr/lib/vxvm/bin/vxsplitlines -g myapp-dg


Code:
VxVM vxdg listssbinfo NOTICE V-0-0-0 There are 2 pools

All the disks in the first pool have the same config copies 
All the disks in the second pool may not have the same config copies 

Number of disks in the first pool: 6 
Number of disks in the second pool: 2 

To import the diskgroup with config copy from the first pool issue the command
/usr/sbin/vxdg (-s) -o selectcp=1267475633.69.servername import myapp-dg

To import the diskgroup with config copy from the second pool issue the command
/usr/sbin/vxdg (-s) -o selectcp=1267475616.65.servername import myapp-dg



Verbose : /usr/lib/vxvm/bin/vxsplitlines -g myapp-dg -v

Code:
VxVM vxdg listssbinfo NOTICE V-0-0-0. There are 2 pools

All the disks in the first pool have the same config copies 
All the disks in the second pool may not have the same config copies 

To see the configuration copy from a disk issue the command
/etc/vx/diag.d/vxprivutil dumpconfig <private path>

To import the diskgroup with config copy from a disk issue the command
/usr/sbin/vxdg (-s) -o selectcp=<diskid> import myapp-dg

                                Pool 0
DEVICE          DISK         DISK ID                      DISK PRIVATE PATH        
myapp-dg08     c2t10d0s2    1267475633.69.servername        /dev/vx/rdmp/c2t10d0s2   
myapp-dg07     c2t9d0s2     1267475633.69.servername        /dev/vx/rdmp/c2t9d0s2    
myapp-dg06     c2t8d0s2     1267475633.69.servername        /dev/vx/rdmp/c2t8d0s2    
myapp-dg05     c2t2d0s2     1267475633.69.servername        /dev/vx/rdmp/c2t2d0s2    
myapp-dg04     c2t1d0s2     1267475633.69.servername        /dev/vx/rdmp/c2t1d0s2    
myapp-dg03     c2t0d0s2     1267475633.69.servername        /dev/vx/rdmp/c2t0d0s2    


                                Pool 1
DEVICE          DISK         DISK ID                      DISK PRIVATE PATH        
myapp-dg02     c1t3d0s2     1267475616.65.servername        /dev/vx/rdmp/c1t3d0s2    
myapp-dg01     c1t2d0s2     1267475608.63.servername        /dev/vx/rdmp/c1t2d0s2


/usr/sbin/vxdg -s -f -o selectcp=1267475624.67.servername import myapp-dg
Code:
VxVM vxdg ERROR V-5-1-10978 Disk group myapp-dg: import failed: 
Operation must be executed on master


vxdg import myapp-dg
Code:
VxVM vxconfigd NOTICE V-5-1-9576 Split Brain. da id is 0.1, while dm id is 0.0 for dm myapp-dg01
VxVM vxconfigd NOTICE V-5-1-9576 Split Brain. da id is 0.1, while dm id is 0.0 for dm myapp-dg01
VxVM vxconfigd NOTICE V-5-1-9576 Split Brain. da id is 0.1, while dm id is 0.0 for dm myapp-dg02
VxVM vxdg ERROR V-5-1-10978 Disk group myapp-dg: import failed: 
Serial Split Brain detected. Run vxsplitlines to import the diskgroup

vxdisk list
Code:
DEVICE       TYPE            DISK         GROUP        STATUS
c1t0d0s2     auto:none       -            -            online invalid
c1t1d0s2     auto:cdsdisk    -            -            online
c1t2d0s2     auto:cdsdisk    -            -            online
c1t3d0s2     auto:cdsdisk    -            -            online
c2t0d0s2     auto:cdsdisk    -            -            online
c2t1d0s2     auto:cdsdisk    -            -            online
c2t2d0s2     auto:cdsdisk    -            -            online
c2t8d0s2     auto:cdsdisk    -            -            online
c2t9d0s2     auto:cdsdisk    -            -            online
c2t10d0s2    auto:cdsdisk    -            -            online


Except the first 2 disks, every disk was part of raid 5 volume and 1 disk as spare and 1 for log.


Any idea how I can recover this?

Thanks in advance.Smilie

Edit 1
More Info :

vxconfigd -k
Code:
VxVM vxconfigd NOTICE V-5-1-9576 Split Brain. da id is 0.1, while dm id is 0.0 for dm myapp-dg01
VxVM vxconfigd NOTICE V-5-1-9576 Split Brain. da id is 0.1, while dm id is 0.0 for dm myapp-dg01
VxVM vxconfigd NOTICE V-5-1-9576 Split Brain. da id is 0.1, while dm id is 0.0 for dm myapp-dg02
VxVM vxconfigd ERROR V-5-1-569 Disk group myapp-dg, Disk c2t2d0s2: Cannot auto-import group:
        Serial Split Brain detected. Run vxsplitlines to import the diskgroup

Code:
/etc/vx/diag.d/vxprivutil dumpconfig /dev/vx/rdmp/c2t2d0s2 
VxVM vxprivutil ERROR V-5-1-595 Disk has no enabled config copies


Last edited by upengan78; 04-02-2010 at 05:14 PM..
# 2  
Old 04-02-2010
What kind of setup is this? A cluster or have you somehow connected two non-clustered nodes to the same storage?

---------- Post updated 2010-04-03 at 12:01 AM ---------- Previous update was 2010-04-02 at 11:58 PM ----------

Also can you post the output of:

Code:
vxdg -o alldgs list

# 3  
Old 04-02-2010
Smilie

Quote:
Originally Posted by reborg
What kind of setup is this? A cluster or have you somehow connected two non-clustered nodes to the same storage?

---------- Post updated 2010-04-03 at 12:01 AM ---------- Previous update was 2010-04-02 at 11:58 PM ----------

Also can you post the output of:

Code:
vxdg -o alldgs list

Hello,

Thanks for reply.

This is a single machine (v440) connected via scsi cable to only ESM on storedge D2. Other scsi port is terminated using LVD/SE terminator. No cluster software on the v440 as well as no such intention.

Code:
 vxdg -o alldgs list
VxVM vxdg ERROR V-5-1-776 Option alldgs not recognized, recognized options are:
        failfast - set the failfast timeout for the disk (msec)
        verify - verifies that the specified disk group task can be performed
                without violating any storage-specific guidelines,
                but does not perform the task
        override - performs the  disk  group  task  and  ignores  any
                storage-specific   guidelines
        remove - removes the objects involved in a move/split/join from
                thisdiskgroup
        complete - perform recovery by retrying original operation
        clean - clear any move/split/join-related internal flags
        expand  - select additional objects that reside on same disks
        dgtype - select dg type (private, shared, san)
        clearreserve - clear reservations before import
        groupreserve - import with group reserve
        listreserve - show group reserve information with list command
        ejectkey - eject host membership
        dgid - need to use dgid for the dg to be initialized
        selectcp - Select cp on this disk
        site - Site based import
        noreonline - Don't reonline disks
        coordinator - dg disks reserved as coordinator disks




        site - disk group site name
        tag -  disk group tag name[=value]
        source - disk group tag source
        useclonedev - import clone disks
        updateid - update ids when importing clone disks
        addmirror - Add mirror on the new site
        rmmirror - Remove mirror on the site being removed
        overridessb - Override SSB on reattaching a disk/site
        alldisks - Reattach a site only if all disks are online
        autotagging - Autotaging for newly added disks

# 4  
Old 04-02-2010
sorry, a lapse in concentration, the command I wanted was:

Code:
vxdisk -o alldgs list

# 5  
Old 04-02-2010
No Problem,

vxdisk -o alldgs list
Code:
DEVICE       TYPE            DISK         GROUP        STATUS
c1t0d0s2     auto:none       -            -            online invalid
c1t1d0s2     auto:cdsdisk    -            -            online
c1t2d0s2     auto:cdsdisk    -            (myapp-dg)  online
c1t3d0s2     auto:cdsdisk    -            (myapp-dg)  online
c2t0d0s2     auto:cdsdisk    -            (myapp-dg)  online
c2t1d0s2     auto:cdsdisk    -            (myapp-dg)  online
c2t2d0s2     auto:cdsdisk    -            (myapp-dg)  online
c2t8d0s2     auto:cdsdisk    -            (myapp-dg)  online
c2t9d0s2     auto:cdsdisk    -            (myapp-dg)  online
c2t10d0s2    auto:cdsdisk    -            (myapp-dg)  online

# 6  
Old 04-02-2010
Ok, it doesn't look as bad as it might have been.

Can you also post a dxdisk list for each disk eg:
Code:
vxdg list c2t2d0s2

# 7  
Old 04-02-2010
MySQL

Quote:
Originally Posted by reborg
Ok, it doesn't look as bad as it might have been.

Can you also post a dxdisk list for each disk eg:
Code:
vxdg list c2t2d0s2

okay, vxdisk list {}

Code:
Device:    c1t2d0s2
devicetag: c1t2d0
type:      auto
hostid:    servername
disk:      name= id=1267475608.63.servername
group:     name=myapp-dg id=1267475668.79.servername
info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags:     online ready private autoconfig autoimport
pubpaths:  block=/dev/vx/dmp/c1t2d0s2 char=/dev/vx/rdmp/c1t2d0s2
guid:      {b132cae8-1dd1-11b2-9c85-0003ba3376dd}
udid:      SEAGATE%5FST373207LSUN72G%5FDISKS%5F303630393334335850480000
site:      -
version:   3.1
iosize:    min=512 (bytes) max=2048 (blocks)
public:    slice=2 offset=65792 len=143283520 disk_offset=0
private:   slice=2 offset=256 len=65536 disk_offset=0
update:    time=1270238982 seqno=0.33
ssb:       actual_seqno=0.1
headers:   0 240
configs:   count=1 len=48144
logs:      count=1 len=7296
Defined regions:
 config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
 config   priv 000256-048207[047952]: copy=01 offset=000192 enabled
 log      priv 048208-055503[007296]: copy=01 offset=000000 enabled
 lockrgn  priv 055504-055647[000144]: part=00 offset=000000
Multipathing information:
numpaths:   1
c1t2d0s2        state=enabled

Device:    c1t3d0s2
devicetag: c1t3d0
type:      auto
hostid:    servername
disk:      name= id=1267475616.65.servername
group:     name=myapp-dg id=1267475668.79.servername
info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags:     online ready private autoconfig autoimport
pubpaths:  block=/dev/vx/dmp/c1t3d0s2 char=/dev/vx/rdmp/c1t3d0s2
guid:      {b5d6e6b0-1dd1-11b2-9c85-0003ba3376dd}
udid:      SEAGATE%5FST373207LSUN72G%5FDISKS%5F303630393333585242540000
site:      -
version:   3.1
iosize:    min=512 (bytes) max=2048 (blocks)
public:    slice=2 offset=65792 len=143283520 disk_offset=0
private:   slice=2 offset=256 len=65536 disk_offset=0
update:    time=1270238982 seqno=0.32
ssb:       actual_seqno=0.1
headers:   0 240
configs:   count=1 len=48144
logs:      count=1 len=7296
Defined regions:
 config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
 config   priv 000256-048207[047952]: copy=01 offset=000192 enabled
 log      priv 048208-055503[007296]: copy=01 offset=000000 enabled
 lockrgn  priv 055504-055647[000144]: part=00 offset=000000
Multipathing information:
numpaths:   1
c1t3d0s2        state=enabled

Device:    c2t0d0s2
devicetag: c2t0d0
type:      auto
hostid:    servername
disk:      name= id=1267475624.67.servername
group:     name=myapp-dg id=1267475668.79.servername
info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags:     online ready private autoconfig autoimport
pubpaths:  block=/dev/vx/dmp/c2t0d0s2 char=/dev/vx/rdmp/c2t0d0s2
guid:      {bacffc74-1dd1-11b2-9c85-0003ba3376dd}
udid:      FUJITSU%5FMAP3367N%20SUN36G%5FDISKS%5F00N05W5D%20%20%20%20
site:      -
version:   3.1
iosize:    min=512 (bytes) max=2048 (blocks)
public:    slice=2 offset=65792 len=71061376 disk_offset=0
private:   slice=2 offset=256 len=65536 disk_offset=0
update:    time=1270238982 seqno=0.31
ssb:       actual_seqno=0.0
headers:   0 240
configs:   count=1 len=48144
logs:      count=1 len=7296
Defined regions:
 config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
 config   priv 000256-048207[047952]: copy=01 offset=000192 enabled
 log      priv 048208-055503[007296]: copy=01 offset=000000 enabled
 lockrgn  priv 055504-055647[000144]: part=00 offset=000000
Multipathing information:
numpaths:   1
c2t0d0s2        state=enabled

Device:    c2t1d0s2
devicetag: c2t1d0
type:      auto
hostid:    servername
disk:      name= id=1267475633.69.servername
group:     name=myapp-dg id=1267475668.79.servername
info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags:     online ready private autoconfig autoimport
pubpaths:  block=/dev/vx/dmp/c2t1d0s2 char=/dev/vx/rdmp/c2t1d0s2
guid:      {c003b6f4-1dd1-11b2-9c85-0003ba3376dd}
udid:      FUJITSU%5FMAP3367N%20SUN36G%5FDISKS%5F00N05W67%20%20%20%20
site:      -
version:   3.1
iosize:    min=512 (bytes) max=2048 (blocks)
public:    slice=2 offset=65792 len=71061376 disk_offset=0
private:   slice=2 offset=256 len=65536 disk_offset=0
update:    time=1270238982 seqno=0.36
ssb:       actual_seqno=0.0
headers:   0 240
configs:   count=1 len=48144
logs:      count=1 len=7296
Defined regions:
 config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
 config   priv 000256-048207[047952]: copy=01 offset=000192 enabled
 log      priv 048208-055503[007296]: copy=01 offset=000000 disabled
 lockrgn  priv 055504-055647[000144]: part=00 offset=000000
Multipathing information:
numpaths:   1
c2t1d0s2        state=enabled

Device:    c2t2d0s2
devicetag: c2t2d0
type:      auto
hostid:    servername
disk:      name= id=1267475642.71.servername
group:     name=myapp-dg id=1267475668.79.servername
info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags:     online ready private autoconfig autoimport
pubpaths:  block=/dev/vx/dmp/c2t2d0s2 char=/dev/vx/rdmp/c2t2d0s2
guid:      {c531f974-1dd1-11b2-9c85-0003ba3376dd}
udid:      FUJITSU%5FMAP3367N%20SUN36G%5FDISKS%5F00N05W0R%20%20%20%20
site:      -
version:   3.1
iosize:    min=512 (bytes) max=2048 (blocks)
public:    slice=2 offset=65792 len=71061376 disk_offset=0
private:   slice=2 offset=256 len=65536 disk_offset=0
update:    time=1270238982 seqno=0.36
ssb:       actual_seqno=0.0
headers:   0 240
configs:   count=1 len=48144
logs:      count=1 len=7296
Defined regions:
 config   priv 000048-000239[000192]: copy=01 offset=000000 disabled
 config   priv 000256-048207[047952]: copy=01 offset=000192 disabled
 log      priv 048208-055503[007296]: copy=01 offset=000000 enabled
 lockrgn  priv 055504-055647[000144]: part=00 offset=000000
Multipathing information:
numpaths:   1
c2t2d0s2        state=enabled

Device:    c2t8d0s2
devicetag: c2t8d0
type:      auto
hostid:    servername
disk:      name= id=1267475649.73.servername
group:     name=myapp-dg id=1267475668.79.servername
info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags:     online ready private autoconfig autoimport
pubpaths:  block=/dev/vx/dmp/c2t8d0s2 char=/dev/vx/rdmp/c2t8d0s2
guid:      {ca0db7f8-1dd1-11b2-9c85-0003ba3376dd}
udid:      HITACHI%5FDK32EJ36NSUN36G%5FDISKS%5FHITACHI%20DK32EJ-36NC%20%20%20%20%20435J6553
site:      -
version:   3.1
iosize:    min=512 (bytes) max=2048 (blocks)
public:    slice=2 offset=65792 len=71061376 disk_offset=0
private:   slice=2 offset=256 len=65536 disk_offset=0
update:    time=1270238982 seqno=0.22
ssb:       actual_seqno=0.0
headers:   0 240
configs:   count=1 len=48144
logs:      count=1 len=7296
Defined regions:
 config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
 config   priv 000256-048207[047952]: copy=01 offset=000192 enabled
 log      priv 048208-055503[007296]: copy=01 offset=000000 enabled
 lockrgn  priv 055504-055647[000144]: part=00 offset=000000
Multipathing information:
numpaths:   1
c2t8d0s2        state=enabled

Device:    c2t9d0s2
devicetag: c2t9d0
type:      auto
hostid:    servername
disk:      name= id=1267475658.75.servername
group:     name=myapp-dg id=1267475668.79.servername
info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags:     online ready private autoconfig autoimport
pubpaths:  block=/dev/vx/dmp/c2t9d0s2 char=/dev/vx/rdmp/c2t9d0s2
guid:      {ced888c6-1dd1-11b2-9c85-0003ba3376dd}
udid:      HITACHI%5FDK32EJ36NSUN36G%5FDISKS%5FHITACHI%20DK32EJ-36NC%20%20%20%20%20435B6497
site:      -
version:   3.1
iosize:    min=512 (bytes) max=2048 (blocks)
public:    slice=2 offset=65792 len=71061376 disk_offset=0
private:   slice=2 offset=256 len=65536 disk_offset=0
update:    time=1270238982 seqno=0.21
ssb:       actual_seqno=0.0
headers:   0 240
configs:   count=1 len=48144
logs:      count=1 len=7296
Defined regions:
 config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
 config   priv 000256-048207[047952]: copy=01 offset=000192 enabled
 log      priv 048208-055503[007296]: copy=01 offset=000000 enabled
 lockrgn  priv 055504-055647[000144]: part=00 offset=000000
Multipathing information:
numpaths:   1
c2t9d0s2        state=enabled

Device:    c2t10d0s2
devicetag: c2t10d0
type:      auto
hostid:    servername
disk:      name= id=1267475665.77.servername
group:     name=myapp-dg id=1267475668.79.servername
info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags:     online ready private autoconfig autoimport
pubpaths:  block=/dev/vx/dmp/c2t10d0s2 char=/dev/vx/rdmp/c2t10d0s2
guid:      {d394af98-1dd1-11b2-9c85-0003ba3376dd}
udid:      HITACHI%5FDK32EJ36NSUN36G%5FDISKS%5FHITACHI%20DK32EJ-36NC%20%20%20%20%2043536259
site:      -
version:   3.1
iosize:    min=512 (bytes) max=2048 (blocks)
public:    slice=2 offset=65792 len=71061376 disk_offset=0
private:   slice=2 offset=256 len=65536 disk_offset=0
update:    time=1270238982 seqno=0.26
ssb:       actual_seqno=0.0
headers:   0 240
configs:   count=1 len=48144
logs:      count=1 len=7296
Defined regions:
 config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
 config   priv 000256-048207[047952]: copy=01 offset=000192 enabled
 log      priv 048208-055503[007296]: copy=01 offset=000000 enabled
 lockrgn  priv 055504-055647[000144]: part=00 offset=000000
Multipathing information:
numpaths:   1
c2t10d0s2       state=enabled

Login or Register to Ask a Question

Previous Thread | Next Thread

5 More Discussions You Might Find Interesting

1. What is on Your Mind?

The Human Brain project

A global group of scientists are spending the next ten years and a billion dollars to try and develop a computer simulation of the brain: https://www.humanbrainproject.eu/ I always found it fascinating that the brain can understand itself. This almost sounds like in a few years the computer... (0 Replies)
Discussion started by: figaro
0 Replies

2. Ubuntu

Ubuntu 9.04 Serial application to telnet to serial device

Hello! I am working on an application which reads environmental instruments which have serial ports. The application requires a serial port to be present to talk to the device (i.e. /dev/ttyS0 ). In some instances the environmental devices will be 100's of yards away from the computer, so a... (5 Replies)
Discussion started by: mvona
5 Replies

3. Programming

Brain Teaser Extended

Hi Gurus, To the Brain Teaser, if I add another condition, say the executable should not be altered, how the program should be altered? (no perl please, purely C). I forgot to mention this condition my staff had mentioned. ( forgot then and got now :D ) The program executed the first time... (4 Replies)
Discussion started by: vrk1219
4 Replies

4. Programming

C Brain Teaser

Dear Gurus, I have encountered a C question, which I thought of sharing with you. This question was asked by one of my technical training staff...Though my training was over I'm still thinking of a solution for this.. Write a C program to do a small task(lets say just simply printing a "Hello... (34 Replies)
Discussion started by: vrk1219
34 Replies

5. Shell Programming and Scripting

Brain Bench Certification

Hi, Can anybody provide me Pointers to Practice tests or any Material to prepare for Brainbench certification in Unix Shell Scripting? Also how good is this Certification for UNIX programmers. Is it worth it? I'm planning to take this certification in 2 weeks. Kindly let me know all the pros... (0 Replies)
Discussion started by: pavan_emani
0 Replies
Login or Register to Ask a Question