Commands such as drvconfig can hang on fully populated servers : Can't open /dev/ksyms

Asset ID:	1-72-1011677.1
Update Date:	2009-09-30
Keywords:

Solution Type Problem Resolution Sure

Solution 1011677.1 : Commands such as drvconfig can hang on fully populated servers : Can't open /dev/ksyms

Related Items


Sun Enterprise 4500 Server
 Sun Enterprise 5500 Server
 Sun Enterprise 3500 Server
 Sun Enterprise 6500 Server
 Solaris SPARC Operating System

Related Categories


GCS>Sun Microsystems>Servers>Midrange Servers

PreviouslyPublishedAs
216006

Symptoms

Various Solaris[TM] commands can hang on fully configured Ultra[TM] Enterprise
Servers running Solaris 2.5.1 and 2.6.
Some commands which can hang include:
drvconfig
modinfo
prtconf
netstat
dmesg
crash
adb

The problem is a direct result of the OS's inability to "open" /dev/ksyms.
The root cause is the exhaustion of kobj symbol space as outlined in
Internal BUG ID 4100378 "kobj symbol space should grow dynamically."

As more and more drivers are added to a server, this problem becomes more
evident, especially in the case of large arrays being added to a server
(often seen with EMC arrays) and causing a drvconfig hang situation.
An excellent way to confirm the problem is to compare truss output for
the failed command against truss output for the same (successful) command
on another system that is running the same OS.
For example, the truss output for a failed dmesg command on a 2.5.1 server
looks something like:
dilbert#truss -o /tmp/dmesg dmesg
execve("/usr/sbin/dmesg", 0xEFFFFE68, 0xEFFFFE70)   argc = 1
*** SGID: rgid/egid/sgid = 1 / 3 / 3   ***
open("/dev/zero", O_RDONLY)         = 3
mmap(0x00000000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xEF7C0000
open("/usr/lib/libkvm.so.1", O_RDONLY)      = 4
fstat(4, 0xEFFFFA80)                = 0
mmap(0x00000000, 8192, PROT_READ|PROT_EXEC, MAP_SHARED, 4, 0) = 0xEF7B0000
mmap(0x00000000, 90112, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF790000
munmap(0xEF796000, 57344)           = 0
mmap(0xEF7A4000, 4687, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4, 16384) = 0xEF7A4000
close(4)                    = 0
open("/usr/lib/libelf.so.1", O_RDONLY)      = 4
fstat(4, 0xEFFFFA80)                = 0
mmap(0xEF7B0000, 8192, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_FIXED, 4, 0) = 0xEF7B0000
mmap(0x00000000, 131072, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF760000
munmap(0xEF770000, 57344)           = 0
mmap(0xEF77E000, 5328, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4, 57344) = 0xEF77E000
close(4)                    = 0
open("/usr/lib/libc.so.1", O_RDONLY)        = 4
fstat(4, 0xEFFFFA80)                = 0
mmap(0xEF7B0000, 8192, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_FIXED, 4, 0) = 0xEF7B0000
mmap(0x00000000, 622592, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF680000
munmap(0xEF700000, 57344)           = 0
mmap(0xEF70E000, 29304, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4, 516096) = 0xEF70E000
mmap(0xEF716000, 5320, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xEF716000
close(4)                    = 0
open("/usr/lib/libdl.so.1", O_RDONLY)       = 4
fstat(4, 0xEFFFFA80)                = 0
mmap(0xEF7B0000, 8192, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_FIXED, 4, 0) = 0xEF7B0000
close(4)                    = 0
open("/usr/platform/SUNW,Ultra-Enterprise/lib/libc_psr.so.1", O_RDONLY) = 4
fstat(4, 0xEFFFF8A8)                = 0
mmap(0x00000000, 8192, PROT_READ|PROT_EXEC, MAP_SHARED, 4, 0) = 0xEF750000
mmap(0x00000000, 81920, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF730000
munmap(0xEF734000, 57344)           = 0
mmap(0xEF742000, 5464, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4, 8192) = 0xEF742000
close(4)                    = 0
close(3)                    = 0
munmap(0xEF750000, 8192)            = 0
open("/usr/platform/SUNW,Ultra-Enterprise/lib/libkvm_psr.so.1", O_RDONLY) = 3
fstat(3, 0xEFFFFAC8)                = 0
mmap(0x00000000, 8192, PROT_READ|PROT_EXEC, MAP_SHARED, 3, 0) = 0xEF750000
mmap(0x00000000, 90112, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xEF660000
munmap(0xEF666000, 57344)           = 0
mmap(0xEF674000, 4687, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 16384) = 0xEF674000
close(3)                    = 0
munmap(0xEF750000, 8192)            = 0
brk(0x00021320)                 = 0
Running truss against the same command ( that runs successfully ) on
a box with an identical OS ( in this case 2.5.1 ) looks something
like:
dogbert# truss -o /tmp/dmesg dmesg
execve("/usr/sbin/dmesg", 0xEFFFFE90, 0xEFFFFE98)   argc = 1
*** SGID: rgid/egid/sgid = 1 / 3 / 3   ***
open("/dev/zero", O_RDONLY)         = 3
mmap(0x00000000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xEF7C0000
open("/usr/lib/libkvm.so.1", O_RDONLY)      = 4
fstat(4, 0xEFFFFB44)                = 0
mmap(0x00000000, 8192, PROT_READ|PROT_EXEC, MAP_SHARED, 4, 0) = 0xEF7B0000
mmap(0x00000000, 90112, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF790000
munmap(0xEF796000, 57344)           = 0
mmap(0xEF7A4000, 4687, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4, 16384) = 0xEF7A4000
close(4)                    = 0
open("/usr/lib/libelf.so.1", O_RDONLY)      = 4
fstat(4, 0xEFFFFB44)                = 0
mmap(0xEF7B0000, 8192, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_FIXED, 4, 0) = 0xEF7B0000
mmap(0x00000000, 122880, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF760000
munmap(0xEF76E000, 57344)           = 0
mmap(0xEF77C000, 4460, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4, 49152) = 0xEF77C000
close(4)                    = 0
open("/usr/lib/libc.so.1", O_RDONLY)        = 4
fstat(4, 0xEFFFFB44)                = 0
mmap(0xEF7B0000, 8192, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_FIXED, 4, 0) = 0xEF7B0000
mmap(0x00000000, 622592, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF680000
munmap(0xEF700000, 57344)           = 0
mmap(0xEF70E000, 26688, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4, 516096) = 0xEF70E000
mmap(0xEF716000, 2696, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xEF716000
close(4)                    = 0
open("/usr/lib/libdl.so.1", O_RDONLY)       = 4
fstat(4, 0xEFFFFB44)                = 0
mmap(0xEF7B0000, 8192, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_FIXED, 4, 0) = 0xEF7B0000
close(4)                    = 0
open("/usr/platform/SUNW,Ultra-1/lib/libc_psr.so.1", O_RDONLY) = 4
fstat(4, 0xEFFFF9A4)                = 0
mmap(0x00000000, 8192, PROT_READ|PROT_EXEC, MAP_SHARED, 4, 0) = 0xEF750000
mmap(0x00000000, 81920, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF730000
munmap(0xEF734000, 57344)           = 0
mmap(0xEF742000, 5440, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4, 8192) = 0xEF742000
close(4)                    = 0
close(3)                    = 0
munmap(0xEF750000, 8192)            = 0

open("/usr/platform/SUNW,Ultra-1/lib/libkvm_psr.so.1", O_RDONLY) Err#2 ENOENT
brk(0x00021320)                 = 0
brk(0x00023320)                 = 0
open("/dev/ksyms", O_RDONLY)            = 3
...
...
...
By comparing the truss output, it becomes apparent that dmesg is
hanging when trying to "open" /dev/ksyms.

Resolution

To resolve, either upgrade to Solaris[TM]
7, or just manually define a
kobj symbol space that is large enough to accommodate all required
devices and modules on the afflicted server.
To check the current kobj_map_space_len setting, run command..
# echo kobj_map_space_len/X|adb -k
physmem f832
kobj_map_space_len:
kobj_map_space_len:                         100000
The example above displays a default setting of 1MB ( 0x100000)
To double the size of kobj_map_space_len from the example above, add
the following entry into /etc/system then reboot..
set kobj_map_space_len=0x200000
The minimum required size of kobj_map_space_len will is function of the
number of drivers and modules that the server requires.   Hence, required
minimum values will vary.
Please note that any defined value *MUST* FALL UPON AN EVEN PAGE BOUNDARY!!
Typically, given these constraints, the following definition should be
more than adequate for most large fully populated servers:

set kobj_map_space_len=0x300000

If there is any reason to suspect that 3MB is not large enough
to accommodate the server's needs, then try:
set kobj_map_space_len=0x400000
A reboot is required for the new table size to take effect.

Product
Solaris 7 Operating System
Sun Enterprise 6500 Server
Sun Enterprise 5500 Server
Sun Enterprise 4500 Server
Sun Enterprise 3500 Server

Internal Comments



 Solaris 7 resolves this problem by altering the method by which the kobj symbol 

 space is allocated ( In Solaris 7 kobj symbol space is allocated dynamically,   

 whereas, in 2.5.1 and 2.6 kobj symbol space is statically defined when the 

 system first boots ).   Since this bug primarily affects large servers, it 

 is very doubtful that this solution will be well received by Customers.   

 Hence, the best method of addressing this issue is to provide method 1 as an

 intermediate solution.

Previously Published As
20267

Change History
Date: 2006-01-18
User Name: 95826
Action: Update Canceled
Comment: *** Restored Published Content *** SSH AUDIT
Version: 0
Date: 2006-01-18
User Name: 95826
Action: Update Started
Comment: SSH AUDIT
Version: 0
Date: 2004-10-14
User Name: 98836
Action: Update Canceled
Comment: *** Restored Published Content *** no time for knowledge because of workflow
Version: 0
Date: 2004-02-19
User Name: 98836
Action: Updated
Comment: Document Accepted Q304 EMEA OS Cleanup
Version: 0
Date: 2003-11-14
User Name: 81292
Action: Approved
Comment: KCCP: OK to publish
Version: 0
Date: 2003-11-14
User Name: 90101
Action: Approved
Comment: Reviewed as part of KM&O KCCP project; content is valid.
Version: 0
Date: 2003-09-15
User Name: 90101
Action: Updated
Comment:
Version: 0
Date: 2003-05-20
User Name: Administrator
Action: Migration from KMSCreator
Comment: updated by : Alan Pereira
comment : Reviewed against Validation Checklist.
Minor editorial changes.
date : Jul 30, 2001

updated by : Edward Carpenter
comment : Evaluated
date : Jul 26, 2001

updated by : Edward Carpenter
comment : No comment
date : Jul 11, 2001

updated by : Edward Carpenter
comment : Added drvconfig scenario, etc.
date : Jul 11, 2001

updated by : Jan Parcel
comment : This is needed by customers, and so I'm
issuing it.
date : Aug 26, 1999
Version: 0
Product_uuid
a723cbe0-2bd5-11d6-8430-f07db3f28928|Solaris 7 Operating System
29ccbffa-0a18-11d6-944e-9f2643ef1cd7|Sun Enterprise 6500 Server
29c606ce-0a18-11d6-98c8-de563f7f1f97|Sun Enterprise 5500 Server
29bdb91a-0a18-11d6-9d2c-c8c4d893e741|Sun Enterprise 4500 Server
29b720d2-0a18-11d6-9865-e60ab6457390|Sun Enterprise 3500 Server

Attachments

This solution has no attachment