Sun Fire[TM] 12K/15K/E20K/E25K: Recovering from a System Controller disk failure

Asset ID:	1-72-1004598.1
Update Date:	2010-06-10
Keywords:

Solution Type Problem Resolution Sure

Solution 1004598.1 : Sun Fire[TM] 12K/15K/E20K/E25K: Recovering from a System Controller disk failure

Applies to:

Sun Fire 12K Server
Sun Fire 15K Server
Sun Fire E20K Server
Sun Fire E25K Server
All Platforms

Symptoms

OS version: Solaris[TM] 8 10/01
SC software version: SMS 1.1 or higher
SDS version: SDS 4.2.1

Prerequisites:
Output of 'metadb -i' command (included in explorer).
Output of 'metastat -p' command (included in explorer)
Output for the 'prtvtoc' command for all disks (included in explorer)
Recent backup tape for filesystems (if both disks lost)

Changes

{CHANGE}

Cause

{CAUSE}

Solution

Scenario #1: Loss of 1 of 2 disks on SC
1. As we have to shutdown the SC to replace the defective disk, we need to ensure that this SC is the SPARE before shutting it down. As user sms-svc, use the 'showfailover' command to determine status of the Scs:

     
% showfailover -v
     SC Failover Status:     ACTIVE
      Clock Phase Locked:           .....................................Yes
      HASRAM Status (by location):  
           HASRAM (CSB at CS1):      ....................................Good
           HASRAM (CSB at CS0):      ....................................Good
      Status of sf15k-sc1:          
      Role:                         ....................................MAIN
      …
      Status of sf15k-sc0:          
      Role:                         ...................................SPARE

...

2. Use the 'metastat' command to determine the failed submirrors:

d10: Mirror
     Submirror 0: d11
       State: Okay    
     Submirror 1: d12
       State: Needs maintenance          
     …
d20: Mirror
     Submirror 0: d21
       State: Okay         
     Submirror 1: d22
       State: Needs maintenance         
     …
d30: Mirror
     Submirror 0: d31
       State: Okay        
     Submirror 1: d32
       State: Needs maintenance

3. Use the 'metadb' command to determine unavailable/unreadable state databases replicas:

      flags           first blk       block count
      a m  p  luo        16              1034            /dev/dsk/c0t2d0s4
      a    p  luo        1050            1034            /dev/dsk/c0t2d0s4
      a    p  luo        2084            1034            /dev/dsk/c0t2d0s4
      a    p  luo        16              1034            /dev/dsk/c0t2d0s5
      a    p  luo        1050            1034            /dev/dsk/c0t2d0s5
      a    p  luo        2084            1034            /dev/dsk/c0t2d0s5
      M    p     Unknown         Unknown   /dev/dsk/c0t3d0s4
      M    p     Unknown         Unknown   /dev/dsk/c0t3d0s4
      M    p     Unknown         Unknown   /dev/dsk/c0t3d0s4
      M    p     Unknown         Unknown   /dev/dsk/c0t3d0s5
      M    p     Unknown         Unknown   /dev/dsk/c0t3d0s5
      M    p     Unknown         Unknown   /dev/dsk/c0t3d0s5

4. Use the 'metadb' command to delete the state databases replicas on the bad disk:

# metadb -d -f c0t3d0s4
      # metadb -d -f c0t3d0s5

Depending on how the disk has failed, this step may not succeed. If this is the case, we will delete the state database replicas during reboot (step 9).

5. As we have to shutdown the SC to replace the defective disk, we need to ensure that the SC will boot using the correct OBP alias. Prevent the SC rebooting after shutdown to the ok prompt by setting "auto-boot " to false using the 'eeprom' command as superuser on the SC:

# eeprom 'auto-boot =false'

6. If a disk failure occurs on the MAIN SC, loss of a disk is not a failover condition so a failover will need to be forced to the other SC by using the 'setfailover' command as user sms-svc:

% setfailover force

This action will force the former MAIN SC to reset and reboot as SPARE and transfer the role of MAIN SC to the opposite SC. If a disk failure occurs on the SPARE SC, disable failover on the MAIN SC using the 'setfailover' command as user sms-svc. The SPARE SC can then be shut down:

On the MAIN SC:
% setfailover off

On the SPARE SC
# init O

7. Replace the defective disk in the SCPER board (see the Sun Fire 15K System Service Manual 806-3512-xx).

      Disk c0t2d0 is J3 which is the lower hard drive
      Disk c0t3d0 is J2 which is the upper hard drive
To power off the SCPER, you must run poweroff SC#.

8. Boot using the correct OBP alias:
      ok devalias
      disk2 /pci@1f,0/pci@1,1/scsi@2/sd@2,0:a
      disk3 /pci@1f,0/pci@1,1/scsi@2/sd@3,0:a
If faulty disk was c0t2d0 (disk2), boot from disk3
If faulty disk was c0t3d0 (disk3), boot from disk2

      ok boot disk2
      or
      ok boot disk3

9. If step 4 above failed (the metadb -d -f command was unsuccessful due to the nature of the disk failure) OR If a reboot occurs before replacing the disk, the current boot will fail and stop in single-user mode as 51% readable state database replicas are needed. If this is the case, in single-user mode, use the 'metadb' command to delete the state databases replicas (ignore any "Read-only file system" error messages),

# metadb -d -f c0t3d0s4
      # metadb -d -f c0t3d0s5

then proceed with normal startup.

10. Partition the new disk in the same manner as it was before using the 'format' command.

11. Recreate state databases with the 'metadb' command using previous configuration

      # metadb -a -c3 -f c0t3d0s4
      # metadb -a -c3 -f c0t3d0s5

This configuration can be checked using 'metadb -i'.

12. Use the 'metareplace' command to re-enable the sub-mirrors:

     # metareplace -e d10 c0t3d0s0
      # metareplace -e d20 c0t3d0s1
      # metareplace -e d30 c0t3d0s7

This operation will take about 20 minutes per every gigabyte of filesystem. This configuration can be checked using 'metastat'.

Note : In case the resync process does not complete properly (reporting needs maintenance ), the sub-mirrors had to be metadetached, metacleared and then metattached again. This will automatically start the resync process and the metadevices will then go to the "OK" state.

13. Set auto-boot to true using the 'eeprom' command as superuser:

      # eeprom 'auto-boot=true'

14. Failover must be enabled using the 'setfailover' command as user sms-svc user on the MAIN SC:

     % setfailover on

15. Synchronize data from MAIN SC to SPARE SC using the 'setdatasync' command as user sms-svc on the MAIN SC:

      % setdatasync backup

Scenario #2: Loss of both disks on SC

If the SDS-mirrored root disk for SC has been completely destroyed, here are steps to resolve issues with the mirrored boot configuration. If this problem occurs on the MAIN SC, failover to the opposite SC using the 'setfailover' command as user sms-svc:

% setfailover force

We are now working on a SPARE SC which has the defective disks.

1. If disk failures have occurred, replace the defective disks in SCPER board (see Sun Fire 15K System Service Manual 806-3512-xx). If disks have not failed but have been corrupted, continue with step 2 below.

2. Boot cdrom using Solaris 8 10/01 CD, partition BOTH disks in the same manner as they were before (use format, prtvtoc or other SC as example), newfs each of the partitions and mount the root filesystem to /a:

      ok boot cdrom -s
      # format
      # newfs /dev/rdsk/c0t2d0s0
      # newfs /dev/rdsk/c0t2d0s7
      # mount /dev/dsk/c0t2d0s0 /a

3. Restore the root filesystem from backup tape into /a and initialize the root block using:

# installboot /usr/platform/sun4u/lib/fs/ufs/bootblk /dev/rdsk/c0t2d0s0

4. Restore the /export/install filesystem.

5. Modify /etc/system file: remove all lines between the "MDD root info" lines and between the "MDD database info" lines:

      Begin MDD root info (do not edit)
      forceload: misc/md_trans
      forceload: misc/md_raid
      forceload: misc/md_hotspares
      forceload: misc/md_sp
      forceload: misc/md_stripe
      forceload: misc/md_mirror
      forceload: drv/pcipsy
      forceload: drv/simba
      forceload: drv/glm
      forceload: drv/sd
      rootdev:/pseudo/md@0:0,10,blk
      * End MDD root info (do not edit)
      * Begin MDD database info (do not edit)
      set md:mddb_bootlist1="sd:20:16 sd:20:1050 sd:20:2084 sd:21:16 sd:21:1050"
      set md:mddb_bootlist2="sd:21:2084 sd:28:16 sd:28:1050 sd:28:2084 sd:29:16"
      set md:mddb_bootlist3="sd:29:1050 
sd:29:2084"
      * End MDD database info (do not edit)

6. Modify /etc/vfstab file by changing all metadevices for the root filesystem back to regular slices. Comment out all other metadevices:

Before

            #device          device           mount           FS      fsck  mount   mount
            #to mount        to fsck          point           type    pass  at boot options
            fd       -        /dev/fd fd      -       no    -
            /proc    -        /proc    proc    -     no      -
            /dev/md/dsk/d20 -        -        swap    -     no      -
            /dev/md/dsk/d10 /dev/md/rdsk/d10        /        ufs     1     no      logging
            /dev/md/dsk/d30 /dev/md/rdsk/d30      /export/install ufs     2     yes     logging
            swap     -        /tmp     tmpfs   -     yes     -

After

           #device          device           mount           FS      fsck  mount   mount
            #to mount        to fsck          point           type    pass  at boot options
            fd       -        /dev/fd fd      -     no      -
            /proc    -        /proc    proc    -     no      -
            /dev/dsk/c0t2d0s1       -        -        swap    -     no      -
            /dev/dsk/c0t2d0s0 /dev/rdsk/c0t2d0s0      /        ufs     1     no      logging
            #/dev/md/dsk/d30 /dev/md/rdsk/d30      /export/install ufs     2     yes     logging
            swap     -        /tmp     tmpfs   -     yes     -

7. Remove all lines (except comment lines) from /etc/lvm/mddb.cf file.

8. Boot the system from the freshly restored boot disk:

      ok boot disk2

At this time, this SC is defined as the SPARE SC.

For reference, disk aliases are:

ok devalias
      
disk2 /pci@1f,0/pci@1,1/scsi@2/sd@2,0:a

      disk3 /pci@1f,0/pci@1,1/scsi@2/sd@3,0:a

9. Re-create state databases with the 'metadb' command using previous configuration:

      # metadb -a -c3 -f c0t2d0s4
      # metadb -a -c3 -f c0t2d0s5
      # metadb -a -c3 -f c0t3d0s4
      # metadb -a -c3 -f c0t3d0s5

10. Modify the /etc/lvm/md.tab, make sure that all mirrors are one-way mirrors, make sure that the one-way mirrors refer to the restored side:

Before

           d10     -m              d11 d12
            d11             1 1     /dev/dsk/c0t2d0s0
            d12             1 1     /dev/dsk/c0t3d0s0
            d20     -m              d21 d22
            d21             1 1     /dev/dsk/c0t2d0s1
            d22             1 1     /dev/dsk/c0t3d0s1
            d30     -m              d31 d32
            d31             1 1     /dev/dsk/c0t2d0s7
            d32             1 1     /dev/dsk/c0t3d0s7

After

            d10     -m              d11
            d11             1 1     /dev/dsk/c0t2d0s0
            d12             1 1     /dev/dsk/c0t3d0s0
            d20     -m              d21
            d21             1 1     /dev/dsk/c0t2d0s1
            d22             1 1     /dev/dsk/c0t3d0s1
            d30     -m              d31
            d31             1 1     /dev/dsk/c0t2d0s7
            d32             1 1     /dev/dsk/c0t3d0s7

11. Create the metadevices:

# metainit -f -a

12. Set the metadevice as a root device:

     # metaroot d10

13. Restore metadevice entries in /etc/vfstab file:
      #device          device           mount           FS      fsck  mount   mount
      #to mount        to fsck          point           type    pass  at boot options
      fd       -        /dev/fd fd      -       no    -
      /proc    -        /proc    proc    -     no      -
      /dev/md/dsk/d20 -        -        swap    -     no      -
      /dev/md/dsk/d10 /dev/md/rdsk/d10        /        ufs     1     no      logging
      /dev/md/dsk/d30 /dev/md/rdsk/d30      /export/install ufs     2     yes     logging
      swap     -        /tmp     tmpfs   -     yes     -

14. Reboot

15. Second way mirrors can now be attached to mirrored metadevices using the 'metattach' command:

      # metattach d10 d12
      # metattach d20 d22
      # metattach d30 d32

16. Failover must be enabled using the 'setfailover' command as user sms-svc on the MAIN SC:

      % setfailover on

17. Synchronize data from MAIN SC to SPARE SC using the 'setdatasync' command as user sms-svc on the MAIN SC:

      % setdatasync backup

Note: Scripts are available on the EIS-CD to set up the SC disks:

      /sun/tools/SF15K/SF15k-sc-bootdisks-start.sh
      /sun/tools/SF15K/SF15k-sc-bootdisks-finish.sh

After running the scripts:

      # df -k
      Filesystem            kbytes    used   avail capacity  Mounted on
      /dev/md/dsk/d10      8261393 1948634 6230146    24%    /
      /proc                      0       0       0     0%    /proc
      fd                         0       0       0     0%    /dev/fd
      mnttab                     0       0       0     0%    /etc/mnttab
      swap                 2185528       8 2185520     1%    /var/run
      swap                 2187656    2136 2185520     1%    /tmp
      /dev/md/dsk/d30      7061557 1370547 5620395    20%    /export/install
      # metastat -p
      d10 -m d11 d12 1
      d11 1 1 c0t2d0s0
      d12 1 1 c0t3d0s0
      d20 -m d21 d22 1
      d21 1 1 c0t2d0s1
      d22 1 1 c0t3d0s1
      d30 -m d31 d32 1
      d31 1 1 c0t2d0s7
      d32 1 1 c0t3d0s7
      # metadb -i
       flags           first blk       block count
           a m  p  luo        16              1034            /dev/dsk/c0t2d0s4
           a    p  luo        1050            1034            /dev/dsk/c0t2d0s4
           a    p  luo        2084            1034            /dev/dsk/c0t2d0s4
           a    p  luo        16              1034            /dev/dsk/c0t2d0s5
           a    p  luo        1050            1034            /dev/dsk/c0t2d0s5
           a    p  luo        2084            1034            /dev/dsk/c0t2d0s5
           a    p  luo        16              1034            /dev/dsk/c0t3d0s4
           a    p  luo        1050            1034            /dev/dsk/c0t3d0s4
           a    p  luo        2084            1034            /dev/dsk/c0t3d0s4
           a    p  luo        16              1034            /dev/dsk/c0t3d0s5
           a    p  luo        1050            1034            /dev/dsk/c0t3d0s5
           a    p  luo        2084            1034            /dev/dsk/c0t3d0s5
      # format / partition / print
      c0t2d0
      Part      Tag    Flag     Cylinders        Size            Blocks
        0       root    wm       0 - 3560        8.00GB    (3561/0/0) 16779432
        1       swap    wu    3561 - 4451        2.00GB    (891/0/0)   4198392
        2     backup    wm       0 - 7505       16.86GB    (7506/0/0) 35368272
        3 unassigned    wm       0               0         (0/0/0)           0
        4 unassigned    wm    4452 - 4456       11.50MB    (5/0/0)       23560
        5 unassigned    wm    4457 - 4461       11.50MB    (5/0/0)       23560
        6 unassigned    wm       0               0         (0/0/0)           0
        7 unassigned    wm    4462 - 7505        6.84GB    (3044/0/0) 14343328
      c0t3d0
        Part      Tag    Flag     Cylinders        Size            Blocks
        0       root    wm       0 - 3560        8.00GB    (3561/0/0) 16779432
        1       swap    wu    3561 - 4451        2.00GB    (891/0/0)   4198392
        2     backup    wm       0 - 7505       16.86GB    (7506/0/0) 35368272
        3 unassigned    wu       0               0         (0/0/0)           0
        4 unassigned    wm    4452 - 4456       11.50MB    (5/0/0)       23560
        5 unassigned    wm    4457 - 4461       11.50MB    (5/0/0)       23560
        6 unassigned    wu       0               0         (0/0/0)           0
        7 unassigned    wm    4462 - 7505        6.84GB    (3044/0/0) 14343328
      
      ok printenv boot-device
      boot-device=disk2 disk3

      ok devalias
      disk2 /pci@1f,0/pci@1,1/scsi@2/sd@2,0:a
      disk3 /pci@1f,0/pci@1,1/scsi@2/sd@3,0:a

Scenario #3 Errors on Drive but no failure of disk on SC

1. As we have to shutdown the SC to replace the defective disk, we need to ensure that this SC is the SPARE before shutting it down. As user sms-svc, use the 'showfailover' command to determine status of the SCs:

% showfailover -v
      SC Failover Status:     ACTIVE
      Clock Phase Locked: .....................................Yes
      HASRAM Status (by location):  
           HASRAM (CSB at CS1): ....................................Good
           HASRAM (CSB at CS0): ....................................Good
      Status of sf15k-sc1:          
      Role: ....................................MAIN
      …
      Status of sf15k-sc0:          
      Role: ...................................SPARE

2. Use the 'metastat' command to determine the failed submirrors:

      d10: Mirror
           Submirror 0: d11
             State: Okay    
           Submirror 1: d12
             State: Needs maintenance          
           ...
      d20: Mirror
           Submirror 0: d21
             State: Okay         
           Submirror 1: d22
             State: Needs maintenance         
           ...
      d30: Mirror
           Submirror 0: d31
             State: Okay        
           Submirror 1: d32
             State: Needs maintenance         
           ...

3. Use the 'metadb' command to determine unavailable/unreadable state databases replicas:

              flags           first blk       block count
            a m  p  luo        16              1034        /dev/dsk/c0t2d0s4
            a    p  luo        1050            1034        /dev/dsk/c0t2d0s4
            a    p  luo        2084            1034        /dev/dsk/c0t2d0s4
            a    p  luo        16              1034        /dev/dsk/c0t2d0s5
            a    p  luo        1050            1034        /dev/dsk/c0t2d0s5
            a    p  luo        2084            1034        /dev/dsk/c0t2d0s5
            M    p     Unknown         Unknown /dev/dsk/c0t3d0s4
            M    p     Unknown         Unknown /dev/dsk/c0t3d0s4
            M    p     Unknown         Unknown /dev/dsk/c0t3d0s4
            M    p     Unknown         Unknown /dev/dsk/c0t3d0s5
            M    p     Unknown         Unknown /dev/dsk/c0t3d0s5
            M    p     Unknown         Unknown /dev/dsk/c0t3d0s5
      # prtvtoc /dev/rdsk/c0t3d0s2 > /var/tmp/prtvtoc.orig
      # metadb  d  f c0t3d0s4
      # metadb  d  f c0t3d0s4
      # metadetach d10 d12 d20 d22 d30 d32

4. As we have to shutdown the SC to replace the defective disk, we need to ensure that the SC will boot using the correct OBP alias. Prevent the SC rebooting after shutdown to the ok prompt by setting "auto-boot " to false using the 'eeprom' command as superuser on the SC:

# eeprom 'auto-boot=false'

5. If a disk failure occurs on the MAIN SC, loss of a disk is not a failover condition so a failover will need to be forced to the other SC by using the 'setfailover' command as user sms-svc:

% setfailover force

6. This action will force the former MAIN SC to reset and reboot as SPARE and transfer the role of MAIN SC to the opposite SC. If a disk failure occurs on the SPARE SC, disable failover on the MAIN SC using the setfailover' command as user sms-svc. The SPARE SC can then be shut down:

On the MAIN SC

            
% setfailover off

On the SPARE SC

# init 0

7. Replace the defective disk in the SCPER board (see the Sun Fire 15K System Service Manual 806-3512-xx).

Disk c0t2d0 is J3 which is the lower hard drive

c0t3d0 is J2 which is the upper hard drive

To power off the SCPER, you must run poweroff SC#.

8. Boot using the correct OBP alias:

     ok devalias
      disk2 /pci@1f,0/pci@1,1/scsi@2/sd@2,0:a
      disk3 /pci@1f,0/pci@1,1/scsi@2/sd@3,0:a

If faulty disk was c0t2d0 (disk2), boot from disk3
If faulty disk was c0t3d0 (disk3), boot from disk2

ok boot disk2
      or
      ok boot disk3

9. If step 4 above failed (the metadb -d -f command was unsuccessful due to the nature of the disk failure)

OR
If a reboot occurs before replacing the disk, the current boot will fail and stop in single-user mode as 51% readable state database replicas are needed. If this is the case, in single-user mode, use the 'metadb' command to delete the state databases replicas (ignore any "Read-only file system" error messages)

# metadb -d -f c0t3d0s4
      # metadb -d -f c0t3d0s5

then proceed with normal startup.

10. Partition the new disk in the same manner as it was before using:

fmthard  s /var/tmp/prtvtoc.orig /dev/rdsk/c0t3d0s2

11. Re-create state databases with the 'metadb' command using previous configuration

# metadb -a -c3 -f c0t3d0s4
      # metadb -a -c3 -f c0t3d0s5

This configuration can be checked using 'metadb -i'.

12. metattach d10 d12 d20 d22 d30 d32

13. To verify that the metattach completed properly, use metastat i

14. Set auto-boot to true using the 'eeprom' command as superuser:
# eeprom 'auto-boot =true'

15. Failover must be enabled using the 'setfailover' command as user sms-svc user on the MAIN SC:

% setfailover on

16. Synchronize data from MAIN SC to SPARE SC using the 'setdatasync' command as user sms-svc on the MAIN SC:

% setdatasync backup

Product
Sun Fire 15K Server
Sun Fire E25K Server
Sun Fire E20K Server
Sun Fire 12K Server
High-End Servers

Keywords: disk, failure, system, controller, disksuite, sds, sms

Change History:
Date: 2010-04-28
User Name: Cootware
Action: Amended format performed review of material.
Comment: If article is set to HTML then it doesn't fit in statement.
Note: On 6/10, document was completely reformatted because migration basically destroyed it. NIV

Attachments

This solution has no attachment