Asset ID:	1-71-1009222.1
Update Date:	2011-05-30
Keywords:

---

Solution Type  Technical Instruction Sure

Solution  1009222.1 :   Sun Fire[TM] 15K/12K Servers: setkeyswitch ops report "[5358] Transmission or pcd(1M) handling of domain-down event failed: ecode=1711"  


PreviouslyPublishedAs
212762

Applies to:

Sun Fire 15K Server
All Platforms

Goal

The SMS (Short Message Service) CLI (command-line interface), setkeyswitch(1M) changes the position of the virtual keyswitch to the specified value. setkeyswitch is responsible for powering on or powering off boards and bringing up a domain.

This short article documents the trouble-shooting process generally undertaken to isolate the root-cause behind failed setkeyswitch operations that report the error msg "[5358] Transmission or pcd(1M) handling of domain-down event failed: ecode=1711"

Solution

The earliest symptoms manifested via this error condition vary; for example, a Domain OS initiated Solaris[TM] reboot/init 6 operations appears to be in a hung condition. Under such circumstances, the SMS domain logs would document the following log messages:


Upon further investigations into domain Rs' POST log dir (/var/opt/SUNWSMS/SMS1.4.1/adm/R/post) would report the following anomaly:

--> empty POST logs captured off the above domain reboot event & the domain's status would remain hung at "In Recovery":


In addition, all attempts at recovering the domain's operations via the setkeyswitch CLI would yield the ecode=1711 error message:


Looking through the SMS platform logs through the same time period yielded the following log extracts :


As observed from the above SMS platform log extracts, the following conditions are recognized:

1. The DXS daemon from SMS, which is responsible for providing virtual console functionality, dynamic reconfiguration mailbox support, and PCI mailbox software support to its resident domains had actually shutdown shortly after the domain OS reboot was initiated
2. The SMS PCD service was not able to successfully acquire the current board list from its database & setup checkpointing information. In addition, it had also flagged a ENOSPC (errno 28) against its attempt to access parts of the PCD database (domain_info & sysboard_info)

Given the above findings, one can reinforce the conclusions reached off the above observations via looking through the current contents of the PCD's database repository:


As observed from the above, the 2 temporary files that PCD have setup to initiate checkpointing ops are actually non-populated (empty).
Hence, given the above data presented and the fact that PCD had flagged a ENOSPC (errno 28) against its attempt to access parts of the PCD database, we can reasonably assume that the root-cause surrounds the issue of making available sufficient disk space to accommodate the two critical elements of SMS facilitating the Solaris reboot event:

1. Sufficient disk space to accommodate the POST log (hpost -Q ops) generated as a result of the Solaris reboot event;
2. Sufficient disk space to accommodate PCD's write checkpointing information (of the Domain Down event sequence) to its PCD database repository.

The error condition was finally isolated to the following disk full condition at the root file system:


Final redress will entail free'ing up sufficient disk space to accommodate normal SMS operations managing & monitoring its resident domains.

Product
System Management Services 1.4.1 Software and above
Sun Fire 12K/15K/20K/25K

Internal section

Keywords: starcat, hpost, pcd 1711, sysboard_info.tmp domain_info.tmp, checkpoint, chkpt, hang, reboot, hpost ecode=44, DSMD_EVENT_DOMAIN_STOP

Previously Published As 80097



Attachments

This solution has no attachment