Asset ID:	1-72-1003483.1
Update Date:	2010-08-03
Keywords:

---

Solution Type  Problem Resolution Sure

Solution  1003483.1 :   Oracle database restart takes longer on high end systems running Solaris[TM] 8  


PreviouslyPublishedAs
204885

Symptoms
Two main issues reported on the 15k/25k running Solaris[TM] 8 are listed below:

• Oracle restart/startup takes several times longer on a system that is up and running for several days. The problem is not reproducible on a freshly booted system.
• During the Oracle restart phase, the system becomes unresponsive. Login to the system or database hangs or slows down. Starting a new process takes a long time. The problem goes away when database startup is completed.

Resolution
Solaris[TM] maintains a per page size freelist. Supported page sizes are: 8K, 64K, 512K and 4M. At boot time all the system memory is promoted to a 4M size freelist. Smaller page size freelists are populated by demoting a page from the bigger page size freelist as the application requests memory. Having a large pool of 4M pages is the main contributing factor for the fast startup of an oracle database as experienced during boot time.
Databases requesting an ISM segment are the main consumer of 4M pages, whereas file system cache and other user applications allocate memory from the 8K page freelist. File systems use main memory, called page cache, to cache file system pages. Copying large files to/from the file system or reading large Database tables into the memory generates huge demand for 8K pages and thus fragment the larger pages to keep up with the demand. As demonstrated in the lab tests, sustaining file system paging activity can potentially exhaust all the 4M pages in the system.

An application requesting large pages when few 4M pages are left in the system may cause it to wait longer to allocate the page. This is because the system will attempt to coalesce smaller sized pages into 4M page to satisfy the application's request. Building a single 4M page, in some cases, may involve a more complex task of relocating smaller pages out of 4M chunks of memory.

System core dump taken while the system was exhibiting the slow oracle startup symptoms shows that the Oracle process was waiting on a shmat() call to return. The kernel thread for shmat() call was running on the processor on which the page_freelist_coalesce() routine was active. The page_freelist_coalesce() routine returns after coalescing all the 4M pages possible from the current smaller sized pages in freelists. Since the routine holds the lock on the freelist, list of free pages, during coalescing, it causes processes/threads needing memory to wait longer to allocate memory and thus negatively effects the system response time.

Recommendation:

• Upgrade Sun Fire[TM] 15K/25K systems to Solaris[TM] 8-117350-23 and Solaris[TM] 9- 117171-17
• Instead of using file system cache for caching data base records use raw, quickio, or directio.

Relief/Workaround
Using a file system cache, called page cache, for caching database records was a good idea on a 32-bit operating system, where the size of the SGA was limited by the process address space (< 4GB). With a 64-bit operating system and database, there is no such limitation and thus this is no longer necessary. Using 64bit Oracle with a properly-sized SGA in relation to the database working set size while bypassing the page cache is the preferred route for deployment on larger Sun systems. The most common technique to bypass the file system cache is to use UFS DirectIO. When VERITAS is installed on the system, use VERITAS QuickIO(ODM). There are situations, such as with read-heavy DBs, where disabling the File System cache could have some negative performance impact. The primary reason, however, for the performance penalty is usually attributed to the loss in filesystem pre-fetching.

There are several advantages of using DirectIO, QuickIO(ODM) or RawIO:

• Low memory demand for file system cache and thus less fragmentation of large pages.
• Removal of the POSIX single writer lock, therefore giving the application the ability to have true parallel IO to a file. This fact is extremely important with a workload that issues high rate of concurrent IO to individual files.
• Reduction in kernel code path execution, that helps in reducing the cpu cycles per IO request.

Product
Operating Environments

Internal Comments
The following is strictly for the use of Sun employees:

Solaris 8-117350-23 and Solaris 9-117171-17 contain following critical performance fixes. It is recommended that customers upgrade to latest kernel patches and if possible to Solaris 9 for optimum performance.

5095432 - Oracle startup takes too long due to memory fragmentation

4802594 - Idle loop degrades IO performance on large psets.

5059920 - Idle loop is not scalable on large systems.

5054052 - disp_getwork() is greedy and negatively impacts dispatch latency

5050686 - Solaris mutexes should be made more efficient under contention

5046939 - kcage_freemem grows too large when large ISM segments are assigned on SF15k

oracle, database, slow, performance, starcat, restart, memory fragmentation, scalability, high sys time
Previously Published As
76796

Change History
The Product "Operating Environments" is too broad. In order to publish this
document, all applicable versions of Solaris must be entered in the product
statement (i.e. Solaris 9 operating system, Solaris 8 Operating System...etc).
You can look up the possibilities using the Swordfish Lookup tool (gives proper
product Nomenclature). http://krep.emea.sun.com/stats/swordfish/
Search Solaris and after clicking on applicable versions click Result button and
copy and paste into product statement.

"MIGRATION CLEANUP"

Attachments

This solution has no attachment