Sun Fire[TM] 12K - E25K / Sparc Enterprise Mx000: WARNING: fcode_timer: Timeout waiting for interpreter

Asset ID:	1-72-1003550.1
Update Date:	2011-01-04
Keywords:

Solution Type Problem Resolution Sure

Solution 1003550.1 : Sun Fire[TM] 12K - E25K / Sparc Enterprise Mx000: WARNING: fcode_timer: Timeout waiting for interpreter - Interpreter did not pick up request

Applies to:

Sun Fire 12K Server
Sun Fire 15K Server
Sun Fire E20K Server
Sun Fire E25K Server
Sun SPARC Enterprise M8000 Server - Version: Not Applicable and later [Release: N/A and later]
All Platforms

Symptoms

When trying to DR (Dynamic Reconfiguration) a component into Sun Fire[TM] 12K-25K domain, it fails to add the board with the following message:

    # /usr/sbin/cfgadm -v -c configure IO5
    assign IO5 
    assign IO5 done
    poweron IO5
    poweron IO5 done
    test IO5 
    test IO5 done 
    connect IO5
    cfgadm: Hardware specific failure: connect IO5: Solaris[TM] failed to probe: IO5

The Starcat domain's /var/adm/messages file, or the domain console log file on the System Controller shows the following errors:

    Mar 20 11:45:42 Domain-A fcodem: [ID 696114 kern.warning] WARNING: fcode_timer: Timeout waiting for interpreter - Interpreter did not pick up request
    Mar 20 11:45:42 Domain-A gptwo_pci: [ID 687272 kern.warning] WARNING: gptwo_pci: Unable to probe pci leaf bc
    Mar 20 11:46:27 Domain-A fcodem: [ID 696114 kern.warning] WARNING: fcode_timer: Timeout waiting for interpreter -Interpreter did not pick up request
    Mar 20 11:46:27 Domain-A gptwo_pci: [ID 687272 kern.warning] WARNING: gptwo_pci: Unable to probe pci leaf bc
    Mar 20 11:47:12 Domain-A fcodem: [ID 696114 kern.warning] WARNING: fcode_timer: Timeout waiting for interpreter -Interpreter did not pick up request
    Mar 20 11:47:12 Domain-A gptwo_pci: [ID 687272 kern.warning] WARNING: gptwo_pci: Unable to probe pci leaf bd
    Mar 20 11:47:57 Domain-A fcodem: [ID 696114 kern.warning] WARNING: fcode_timer: Timeout waiting for interpreter -Interpreter did not pick up request
    Mar 20 11:47:57 Domain-A gptwo_pci: [ID 687272 kern.warning] WARNING: gptwo_pci: Unable to probe pci leaf bd
    Mar 20 11:47:57 Domain-A genunix: [ID 408114 kern.info] /address-extender-queue@bf,0 (axq9) offline
    Mar 20 11:48:00 Domain-A dcs: [ID 397954 daemon.error] <21638> config_change_state: Hardware specific failure:
    connect IO5: Solaris failed to probe: IO5

Performing an addboard on a Mx000 system will produce the following output:

    XSCF> addboard -c configure -d 0 01-0
    XSB#01-0 will be configured into DomainID 0. Continue?[y|n] :y
    Initial diagnosis started. [1800sec]
      0..... 30..... 60..... 90.....120.....150.....180.....210..end
    Initial diagnosis has completed.
    Start configuring XSB to domain.

    connect SB4: Firmware probe failed
    XSB#01-0 could not be configured into DomainID 0 due to operating system error.

The Mx000 domain /var/adm/messages will have messages:

    Dec 30 03:20:40 domain fcodem: [ID 696114 kern.warning] WARNING: fcode_timer: Timeout waiting for interpreter - Interpreter did not pick up request
    Dec 30 03:20:40 domain opl_cfg: [ID 695635 kern.warning] WARNING: IKP: Unable to probe PCI leaf (1-4-0)
    Dec 30 03:20:40 domain drmach: [ID 702911 kern.warning] WARNING: Firmware probe failed
    Dec 30 03:20:40 domain dcs: [ID 223655 daemon.error] <17045> config_change_state: Hardware specific failure: connect SB1: Firmware probe failed

Cause

If efdaemon is not running, the errors on a DR operation as shown above will occur.

The efdaemon can die due to resource problems on the domain or it may have been disabled in error.

The svcs command will show the status of efdaemon in Solaris[TM] 10.
# svcs efdaemon
disabled Dec_21 svc:/platform/sun4v/efdaemon:default

Solution

A fix for efdaemon dying and not being automatically restarted is provided by Patch ID: 112988-01 for Solaris[TM] 8 and Patch ID: 113224-01 for Solaris[TM] 9.

If the service was disabled in error then use the svcadm command to re-enable the service.
# svcadm enable /platform/sun4u/efdaemon

Relief/Workaround

# ps -ef | grep efd
root  1563 14146  0 13:29:00 console  0:00 grep efd

The efdaemon is confirmed not to be running, so restart it:

# /usr/lib/efcode/sparcv9/efdaemon

# ps -ef | grep efd

root  1533     1  0 13:34:38          0:00 /usr/lib/efcode/sparcv9/efdaemon

root  1563 14146  0 13:35:00 console  0:00 grep efd

Once restarted, DR operations should function normally:

# cfgadm -v -c configure IO5

    assign IO5
assign IO5 done
poweron IO5
poweron IO5 done
test IO5
test IO5 done
connect IO5
connect IO5 done
configure IO5

The domain's /var/adm/messages file and the domain console log file on the System Controller will show the successful DR operation:

Mar 20 13:36:16 Domain-A dr: OS configure dr@0:IO5::pci0
Mar 20 13:36:16 Domain-A dr: OS configure dr@0:IO5::pci1
Mar 20 13:36:16 Domain-A dr: OS configure dr@0:IO5::pci2
Mar 20 13:36:16 Domain-A dr: OS configure dr@0:IO5::pci3

@ Internal Comments
Reference Radiance Case ID 64006151, Bug ID: 4710558 and Bug ID: 4633009.
case 3-2651632893 for opl
12K, 15K, 20K, 25K, Starcat, fcode_timer, DR, dr, cfgadm, Hardware specific failure
Previously Published As 75023

Attachments

This solution has no attachment