Asset ID:	1-71-1019113.1
Update Date:	2011-05-25
Keywords:

Solution Type Technical Instruction Sure

Solution 1019113.1 : Sun[TM] Sparc Enterprise Mx000 (OPL) Servers Memory Configuration Rules

Introduction

Introduction

In the Mx000 (OPL) Servers resources are located on 1 to 16 Physical System Boards (PSB). A Physical System Board (PSB) consists of up to four (4) CPU's, up to 32 DIMM's and one IOU resource (optional).
I/O resources on an IOU are tied to a specific CPU/Memory resource, and that CPU, Memory & I/O resource are assigned to a XSB (eXtended System Board). The XSB can then be assigned to a domain. To be usable for a domain, a PSB must first be configured in either UNI-XSB or QUAD-XSB mode.

A M3000 PSB contains a fixed number of pieces of hardware:
- A single CPU
- Up to 4 or 8 DIMM's
- IO

A M4000/M5000 - PSB contains of up to seven pieces of hardware:
- Up to two CPUM (each containing two CPU's for a total of four physical CPU's).
- Up to four MEMB's (each containing 4 or 8 DIMM's ; up to 32 total physical DIMM's)
- One IOU ( IOU's are needed to access internal devices – DVD, Tape, Disk, Network)
The M4000 has 1 PSB and the M5000 has 2.

A M8000/M9000 PSB contains up to two pieces of hardware:
- One CMU (each CMU has 2 or 4 CPU's and 16 or 32 DIMM's)
- One IOU (Optional, at least one required per domain)

In UNI-XSB mode, we get 1 XSB consisting of the whole PSB.
In QUAD-XSB mode, we get up to 4 XSB's, consisting each of one quarter of a PSB.
The M3000 (Ikkaku) can be viewed as a single QUAD-XSB (1 CPU, 8 DIMMs).

The Mx000 (OPL) Servers use DDR2 DIMMs. 1, 2, 4, or 8 GB types can be used.
OPL uses Memory Groups. XSBs can access 2 groups, Group A and Group B.
Group A must always be populated, Group B can optionally be populated to increase memory capacity.

OPL automatically configures memory interleaving to obtain the best performance.
Interleaving takes place within a XSB (UNI or QUAD).
All DIMMs within a XSB are included in a single interleaving scheme, there is only one interleave factor defined inside a XSB. This has the consequence that the number of DIMMS in a Group can only be 4, 8 or 16 and that the number of DIMMs in Group B must be either 0 or the same as the number of DIMMs in Group A
Populating Group B does not affect interleaving or mirroring, it only increases the memory capacity.

Interleave factor = 2	4 DIMMS in Group A	optionally 4 DIMMs in Group B	UNI-XSB or QUAD-XSB
Interleave factor = 4	8 DIMMS in Group A	optionally 8 DIMMs in Group B	UNI-XSB
Interleave factor = 8	16 DIMMS in Group A	optionally 16 DIMMs in Group B	UNI-XSB

Note: Interleave factor of 3 is not possible, therefore XSBs with 12 DIMMS (or 24 DIMMs ) are not permitted.
Note: By definition, a QUAD-XSB can only hold 4 DIMMs in Group A and optionally 4 DIMMS in Group B.

Memory can be configured in mirror mode for higher reliability. (except M3000)
In mirrored mode, both the the memory capacity and the the interleave factor are halved.

The minimal hardware required for a functioning XSB (UNI or QUAD) is to have at least 1 CPU and at least 4 DIMMs in Group A.

DIMM naming scheme and physical location

The DIMM names as output by the showhardconf command and as named on the FRUs is as follows:

		Physical System Board nn (PSB)
UNI-XSB mode	QUAD-XSB mode	M3000 / M4000 / M5000				M8000 / M9000
XSB#	XSB#	Group A		Group B		Group A	Group B
nn-0	nn-0 M3000: only this row	MEMB#0 or MEMB#4	MEM#0A MEM#1A MEM#2A MEM#3A	MEMB#0 or MEMB#4	MEM#0B MEM#1B MEM#2B MEM#3B	MEM#00A MEM#01A MEM#02A MEM#03A	MEM#00B MEM#01B MEM#02B MEM#03B
	nn-0 M3000: only this row	On M3000: MBU_A	MEM#0A MEM#1A MEM#2A MEM#3A	On M3000: MBU_A	MEM#0B MEM#1B MEM#2B MEM#3B	MEM#00A MEM#01A MEM#02A MEM#03A	MEM#00B MEM#01B MEM#02B MEM#03B
	nn-1	MEMB#1 or MEMB#5	MEM#0A MEM#1A MEM#2A MEM#3A	MEMB#1 or MEMB#5	MEM#0B MEM#1B MEM#2B MEM#3B	MEM#10A MEM#11A MEM#12A MEM#13A	MEM#10B MEM#11B MEM#12B MEM#13B
	nn-2	MEMB#2 or MEMB#6	MEM#0A MEM#1A MEM#2A MEM#3A	MEMB#2 or MEMB#6	MEM#0B MEM#1B MEM#2B MEM#3B	MEM#20A MEM#21A MEM#22A MEM#23A	MEM#20B MEM#21B MEM#22B MEM#23B
	nn-3	MEMB#3 or MEMB#7	MEM#0A MEM#1A MEM#2A MEM#3A	MEMB#3 or MEMB#7	MEM#0B MEM#1B MEM#2B MEM#3B	MEM#30A MEM#31A MEM#32A MEM#33A	MEM#30B MEM#31B MEM#32B MEM#33B

Applicable memory population rules

The rules are summarized in the 5 tables below and the additional Group rules.
The gray areas depict the hardware that can be installed.
It is important to understand that the rules apply per XSB.

M3000
PSB #	CPUM # installed	Resulting XSB #	MEMB # required	# of DIMMs		Resulting interleave Factor
				Group A	Group B
						Normal	Mirrored
0	N/A	00-0	N/A	4	0 or 4	2	N/A

UNI-XSB mode M4000 / M5000
PSB #	CPUM # installed	Resulting XSB #	MEMB # required	# of DIMMs on each MEMB		Resulting interleave Factor
				Group A	Group B
						Normal	Mirrored
0	0	00-0	0	4	0 or 4	2	1
	0 and 1		0 and 1			4	2
			0,1,2 and 3			8	4
1	2	01-0	4	4	0 or 4	2	1
	2 and 3		4 and 5			4	2
			4,5,6 and 7			8	4

Note: PSB1 only exists in M5000.

QUAD-XSB mode M4000 / M5000

PSB #

CPUM #

installed

Resulting

XSB #

MEMB #

required

# of DIMMs on each MEMB

Resulting interleave factor

Group A

Group B

Normal

Mirrored

0 *

00-0

00-1

00-2

00-3

2 *

01-0

01-1

01-2

01-3

*Note: Install these first.
Note: PSB1 only exists in M5000.
Note: Each PSB can individually be set to UNI-XSB or QUAD -XSB mode.

UNI-XSB M8000 / M9000 (2 and 4 CPU CMUs)
PSB nn	XSB #	DIMM quantity on each CMU		Resulting interleave factor
		Group A	Group B	Normal	Mirrored
nn= 0..16	nn-0	16	0	8	4
			16

QUAD-XSB M8000 / M9000 (4 CPU CMUs only)
PSB nn	XSB #	DIMM quantity on each CMU		Resulting interleave factor
		Group A	Group B	Normal
nn= 0..16	nn-0 nn-1 nn-2 nn-3	16	0	2
			16

Note: Each PSB can individually be set to UNI-XSB or QUAD -XSB mode.
Note: For architectural reasons, M8000 / M9000 do not allow memory mirroring in QUAD-XSB mode.

Additional Group Rules

within a XSB (UNI or QUAD), the following rules apply in addition to the above 4 tables:

All DIMMs within a Group (A or B) must be of the same size / rank
The size of the DIMMs in Group B must be smaller than or equal the size of the DIMMs in Group A.
The number of DIMMs in Group B must be either 0 or the same as the number of DIMMs in Group A

Additional considerations

Implication of the above rules for M4000 / M5000

Half populated MEMB, with only Group A populated are supported.
In UNI-XSB mode, MEMBs must be installed in powers of 2 (1,2 or 4).
3 MEMBs is not possible, this is due to the single interleaving scheme used on an XSB.
It is permitted to have 3 MEMBs on a PSB, as long as the PSB is configured in QUAD-XSB mode.
The minimum requirement for UNI-XSB mode is CPUM#0 / MEMB#0 (and CPUM#2 / MEMB#4)
A CPU on a CPUM without its associated MEMB and vice-versa cannot be used in QUAD-XSB mode.
The “setupfru” command will report “Operation has completed. However, a configuration error was detected”. If you configure a PSB in QUAD-XSB mode and one of the QUAD-XSB is made of a CPU and no corresponding MEMB or vice-versa.
In this case, "showboards" will report the QUAD-XSB as "Unmount".
XSBs can have different memory configurations.
Only identically configured QUAD-XSBs can be reconfigured into one UNI-XSB
Filling group B does not increase the interleaving factor, thus does not increase memory performance.
Only the memory capacity is increased.

Memory upgrade

The rule requiring that Group A contains the DIMMs with the higher capacity may dictate that in case of a memory upgrade, DIMMs from Group A must be moved to Group B and the new higher capacity DIMMs be installed in Group A.

Performance considerations

UNI-XSB mode allows higher memory performance because it can use higher interleave factors. If all resources of a PSB are needed in one domain, then it is advisable to configure the PSB in UNI-XSB mode.

RAS considerations

Memory mirroring is an effective way to protect against memory failures. Configuring an XSB for memory mirroring halves the available memory capacity. It also halves the interleave factor. Memory mirroring should therefore best be used in UNI-XSB mode.

Best Practice

Try to avoid mixed configurations. It is advisable to have all identical MEMBs in a PSB of a M4000 / M5000. This allows to change between UNI-XSB and QUAD-XSB mode without loosing maximum performance or having systems containing unusable hardware.

If the capability to switch between UNI-XSB and QUAD-XSB mode is desired, then both the rules for UNI-XSB and those for QUAD_XSB must be followed. The supported M8000 / M9000 configurations enforce this.

How to identify DIMMs inside a system from XSCF

The showhardconf command run on the XSCF display the type of DIMMs

Type 1A = 1GB / rank1
Type 1B = 1GB / rank2
Type 2A = 2GB / rank1
Type 2B = 2GB / rank2
Type 4A = 4GB / rank1
Type 4B = 4GB / rank2
Type 8B = 8GB / rank2

Steps to Follow
Follow these steps to configure OPL memory.
A supported memory configuration must be built according to the above "Applicable memory population rules" section.

For every PSB, use the appropriate table:

M3000
UNI-XSB mode M4000 / M5000
QUAD-XSB mode M4000 / M5000
UNI-XSB mode M8000 / M9000
QUAD-XSB mode M8000 / M9000

For every PSB, follow the additional group rules.

Product
Sun SPARC Enterprise M3000 Server
Sun SPARC Enterprise M4000 Server
Sun SPARC Enterprise M5000 Server
Sun SPARC Enterprise M8000 Server
Sun SPARC Enterprise M9000 Server
Sun SPARC Enterprise M9000-32 Server
Sun SPARC Enterprise M9000-64 Server

Attachments

This solution has no attachment