| Asset ID: |
1-71-1005025.1 |
| Update Date: | 2011-03-02 |
| Keywords: | |
Solution Type
Technical Instruction Sure
Solution
1005025.1
:
IPMP with Dynamic Reconfiguration (DR)
| Related Items |
- Sun Fire 12K Server
- Solaris SPARC Operating System
- Sun Fire 15K Server
|
| Related Categories |
- GCS>Sun Microsystems>Servers>High-End Servers
- GCS>Sun Microsystems>Operating Systems>Solaris Operating System
|
PreviouslyPublishedAs
207052
Applies to:
Sun Fire 12K Server Sun Fire 15K Server Solaris SPARC Operating System - Version: 8.0 and later [Release: 8.0 and later] All Platforms
Goal
This doc assumes familiarity with IPMP (IP Multipathing) and it does not contain description on how to configure IPMP. It only describes the procedures to remove a network interface from an IPMP multipathing group and subsequently add it back using Dynamic Reconfiguration (DR).
Dynamic Reconfiguration (IPMP RCM Module) for IPMP, works with IPMP in.mpathd(1M) daemon to failover or failback network interfaces that are part of a DR operation so that IPMP is aware of the current DR operation.
Restricted by IPMP, a network interface cannot be shared between network groups. It can not combine IPv4 and IPv6 protocols in the same multipathing group although DR with IPMP works for both IPv4 and IPv6.
The DR procedures described here is for IPMP with IPv4.
It is important to note that each network interface in the domain should have a unique MAC address. Please ensure setting local-mac-address? to true in the OpenBoot PROM. Also any network interface that has been certified to work with IPMP would be able to work under DR.
Solaris[TM] releases 8 9 and 10 have been verified to work with IPMP and DR. Below is the software and patch matrix required to run IPMP with DR properly:
Solaris 8
108528 Solaris Kernel Patch (in.mpathd, ip, icmp, ifconfig)
109898 ARP Patch
109900 /etc/init.d/network and /sbin/ifparse Patch
109902 in.ndpd multipathing offlining IPMP capability (Update 3)
110378 mipagent patch needed for offlining capability
111802 Introduce RCM suspend/resume (Update 6)
110839 introduce RCM IPMP module (Update 6)
Solaris 9
112233 Solaris Kernel Patch (ip)
112911 ifconfig, in.mpathd and in.routed Patch
112912 libinetcfg Patch
112854 icmp Patch
113464 IPMP Headers Patch
Solaris 10
118833 Solaris kernel patch
120011 Solaris kernel patch
141532 in.mpathd patch
Solution
Here are the steps to DR attach and detach a Cassini Enternet (CE) network interface in a IPMP multiple active paths without standby configuration (i.e active-active setup). It works for active-standy configuration as well:
1. Verify the first network interface IPMP settings. # cat /etc/hostname.ce0
129.158.142.253 netmask + broadcast + group mytestgroup up \
addif 129.158.142.61 deprecated -failover netmask + broadcast + up 2. Verify the second network interface IPMP settings. # cat /etc/hostname.ce2
129.158.142.59 netmask + broadcast + group mytestgroup up \
addif 129.158.142.58 deprecated -failover netmask + broadcast + up 3. Verify domain IPMP failback is enabled. # cat /etc/default/mpathd | grep FAILBACK
FAILBACK=yes
4. Verfiy the domain IPMP settings. # ifconfig -a
lo0: flags=1000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ce0: flags=1000843 mtu 1500 index 2
inet 129.158.142.253 netmask ffffff00 broadcast 129.158.142.255
groupname mytestgroup
ether 0:3:ba:e:91:d3
ce0:1: flags=9040843 mtu 1500 index 2
inet 129.158.142.61 netmask ffffff00 broadcast 129.158.142.255
ce2: flags=1000843 mtu 1500 index 3
inet 129.158.142.59 netmask ffffff00 broadcast 129.158.142.255
groupname mytestgroup
ether 0:3:ba:e:91:77
ce2:1: flags=9040843 mtu 1500 index 3
inet 129.158.142.58 netmask ffffff00 broadcast 129.158.142.255
dman0: flags=1008843 mtu 1500 index 4
inet 192.168.102.13 netmask ffffffe0 broadcast 192.168.102.31
ether 0:0:be:ab:3e:c1
5. Obtain the Attachment Point(Ap_Id) of the network interface to be detached. # cfgadm -l
Ap_Id Type Receptacle Occupant Condition
IO7 MCPU connected configured ok
IO8 HPCI connected configured ok
SB7 CPU connected configured ok
SB8 CPU connected configured ok
c0 scsi-bus connected configured unknown
c1 scsi-bus connected unconfigured unknown
c2 scsi-bus connected unconfigured unknown
c3 scsi-bus connected unconfigured unknown
pcisch0:e08b1slot1 pci-pci/hp connected configured ok
pcisch1:e08b1slot0 unknown connected unconfigured unknown
pcisch2:e08b1slot3 pci-pci/hp connected configured ok
pcisch3:e08b1slot2 unknown connected unconfigured unknown
#
6. Execute cfgadm command to unconfigure the network interface, ce2 in
this case (pcisch2:e08b1slot3). # cfgadm -c unconfigure pcisch2:e08b1slot3
May 6 15:03:43 v4u-15kb-l in.mpathd[92]: Successfully failed over from NIC ce2 to NIC ce0 7. Review the domain /var/adm/messages file and you should be seeing
the following output. May 10 14:08:21 v4u-15kb-l in.mpathd[92]: [ID 832587 daemon.error] Successfully failed over from NIC ce2 to NIC ce0
May 10 14:08:21 v4u-15kb-l in.routed[111]: [ID 970160 daemon.notice] unable to get interface flags for ce2: No such device or address
May 10 14:08:21 v4u-15kb-l in.routed[111]: [ID 970160 daemon.notice] unable to get interface flags for ce2:1: No such device or address
May 10 14:08:21 v4u-15kb-l in.routed[111]: [ID 472501 daemon.notice] ce2:1 has no ifIndex: No such device or address
May 10 14:08:22 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1/network@0 (ce2) offline
May 10 14:08:22 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1/network@1 (ce3) offline
May 10 14:08:22 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1/scsi@2,1 (glm3) offline
May 10 14:08:22 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1/network@0 (ce2) offline
May 10 14:08:22 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1/network@1 (ce3) offline
May 10 14:08:22 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1/scsi@2 (glm2) offline
May 10 14:08:22 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1/scsi@2,1 (glm3) offline
May 10 14:08:22 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1 (pci_pci1) offline
8. Confirm the network IP has been failed over successfully. # ifconfig -a
lo0: flags=1000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ce0: flags=1000843 mtu 1500 index 2
inet 129.158.142.253 netmask ffffff00 broadcast 129.158.142.255
groupname mytestgroup
ether 0:3:ba:e:91:d3
ce0:1: flags=9040843 mtu 1500 index 2
inet 129.158.142.61 netmask ffffff00 broadcast 129.158.142.255
ce0:2: flags=1000843 mtu 1500 index 2
inet 129.158.142.59 netmask ffffff00 broadcast 129.158.142.255
dman0: flags=1008843 mtu 1500 index 4
inet 192.168.102.13 netmask ffffffe0 broadcast 192.168.102.31
ether 0:0:be:ab:3e:c1 9. Verify the Ap_Id Occupant is in "unconfigured" state. # cfgadm -l | grep pcisch2:e08b1slot3
pcisch2:e08b1slot3 unknown connected unconfigured unknown
10. Proceed to disconnect the network interface. # cfgadm -c disconnect pcisch2:e08b1slot3 11. Verify the Ap_Id Receptable is now in "disconnected" state. # cfgadm -l | grep pcisch2:e08b1slot3
pcisch2:e08b1slot3 unknown disconnected unconfigured unknown
12. Step 6 and 10 above can be combined onto a single command line. # cfgadm -c disconnect pcisch2:e08b1slot3 13. Physically remove and replace the network interface.
14. After the network interface has inserted, perform DR connect to connect the network interface. # cfgadm -c connect pcisch2:e08b1slot3 15. Review the domain /var/adm/messages file you should be seeing the following messages. May 10 14:35:06 v4u-15kb-l pcisch: [ID 370704 kern.info] PCI-device: pci@1,pci_pci1
May 10 14:35:06 v4u-15kb-l genunix: [ID 936769 kern.info] pci_pci1 is /pci@11d,70000/pci@1
May 10 14:35:06 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1 (pci_pci1) online
May 10 14:35:07 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1 (pci_pci1) offline
May 10 14:35:07 v4u-15kb-l pcisch: [ID 370704 kern.info] PCI-device: pci@1,pci_pci1
May 10 14:35:07 v4u-15kb-l genunix: [ID 936769 kern.info] pci_pci1 is /pci@11d,70000/pci@1
May 10 14:35:07 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1 (pci_pci1) online
May 10 14:35:07 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1/network@0 (ce2) online
May 10 14:35:08 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1/network@1 (ce3) online
May 10 14:35:08 v4u-15kb-l scsi: [ID 365881 kern.info] /pci@11d,700000/pci@1/scsi@2 (glm2):
May 10 14:35:08 v4u-15kb-l glm2 supports power management.
May 10 14:35:08 v4u-15kb-l scsi: [ID 365881 kern.info] /pci@11d,700000/pci@1/scsi@2 (glm2):
May 10 14:35:08 v4u-15kb-l Rev. 7 Symbios 53c896 found.
May 10 14:35:08 v4u-15kb-l pci_pci: [ID 370704 kern.info] PCI-device: scsi@2, glm2
May 10 14:35:08 v4u-15kb-l genunix: [ID 936769 kern.info] glm2 is /pci@11d,70000/pci@/scsi@2
May 10 14:35:08 v4u-15kb-l pci_pci: [ID 370704 kern.info] PCI-device: scsi@2, glm2
May 10 14:35:08 v4u-15kb-l genunix: [ID 936769 kern.info] glm2 is /pci@11d,700000/pci@1/scsi@2
May 10 14:35:08 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1/scsi@2 (glm2) online
May 10 14:35:09 v4u-15kb-l genunix: [ID 408822 kern.info] NOTICE: ce2: no fault external to device; service available
May 10 14:35:09 v4u-15kb-l genunix: [ID 611667 kern.info] NOTICE: ce2: xcvr addr:0x01 - link up 100 Mbps full duplex
May 10 14:35:19 v4u-15kb-l scsi: [ID 365881 kern.info] /pci@11d,700000/pci@1/scsi@2,1 (glm3):
May 10 14:35:19 v4u-15kb-l glm3 supports power management.
May 10 14:35:19 v4u-15kb-l scsi: [ID 365881 kern.info] /pci@11d,700000/pci@1/scsi@2,1 (glm3):
May 10 14:35:19 v4u-15kb-l Rev. 7 Symbios 53c896 found.
May 10 14:35:19 v4u-15kb-l pci_pci: [ID 370704 kern.info] PCI-device: scsi@2,1, glm3
May 10 14:35:19 v4u-15kb-l genunix: [ID 936769 kern.info] glm3 is /pci@11d,700000/pci@1/scsi@2,1
May 10 14:35:19 v4u-15kb-l pci_pci: [ID 370704 kern.info] PCI-device: scsi@2,1, glm3
May 10 14:35:19 v4u-15kb-l genunix: [ID 936769 kern.info] glm3 is /pci@11d,700000/pci@1/scsi@2,1
May 10 14:35:19 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1/scsi@2,1 (glm3) online 16. Verify the Ap_Id Receptable is now in "connected" state. # cfgadm -l | grep pcisch2:e08b1slot3
pcisch2:e08b1slot3 unknown connected unconfigured unknown 17. Proceed to DR configure the network interface. # cfgadm -c configure pcisch2:e08b1slot3 18. Verify the Ap_Id Occupant is now in "configured" state. # cfgadm -l | grep pcisch2:e08b1slot3
pcisch2:e08b1slot3 pci-pci/hp connected configured ok 19. Step 14 and 17 above can be combined onto a single command line.
20. Verify the network IP has been failed back. # ifconfig -a
lo0: flags=1000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ce0: flags=1000843 mtu 1500 index 2
inet 129.158.142.253 netmask ffffff00 broadcast 129.158.142.255
groupname mytestgroup
ether 0:3:ba:e:91:d3
ce0:1: flags=9040843 mtu 1500 index 2
inet 129.158.142.61 netmask ffffff00 broadcast 129.158.142.255
dman0: flags=1008843 mtu 1500 index 4
inet 192.168.102.13 netmask ffffffe0 broadcast 192.168.102.31
ether 0:0:be:ab:3e:c1
ce2: flags=1000843 mtu 1500 index 5
inet 129.158.142.59 netmask ffffff00 broadcast 129.158.142.255
groupname mytestgroup
ether 0:3:ba:e:91:77
ce2:1: flags=9040843 mtu 1500 index 5
inet 129.158.142.58 netmask ffffff00 broadcast 129.158.142.255
Product
Sun Fire 15K Server
Sun Fire 12K Server
Solaris 9 8/03 Operating System
Solaris 8 Operating System
Solaris 10
IPMP, Internet Protocol Multi-Pathing, DR, Dynamic Reconfiguration, cfgadm
Previously Published As 76037
Attachments
This solution has no attachment
|
