Tunable Base Page Size

Tunable Base Page Size Table of Contents Executive summary... 1 What is Tunable Base Page Size?... 1 How Base Page Size Affects the System... 1 Integrity Virtual Machines Platform Manager... 2 Working with Tunable Base Page Size... 2 Tuning the base page size... 2 base_pagesize kernel tunable parameter... 2 Static tuning only... 3 Installing Tunable Base Page Size enhancement... 3 Updating the HP-UX core... 3 Updating system utilities... 3 Updating applications... 4 Known limitations... 4 Java Virtual Machine... 4 HFS Filesystem... 4 Performance considerations... 5 Summary... 5 Appendices... 6 Application programming for tunable base pages... 6 Obsolete programming constructs... 6 Proper programming practices... 6 Application compatibility... 7 Oracle Database Manager... 7 SAP ERP product... 7 SAS... 7 Tuxedo... 7 Veritas Enterprise Services Administrator... 7 WebLogic 9.1... 7 WebSphere 6.1... 7 For more information... 8

Executive summary Tunable Base Page Size is a new feature introduced in update 3 to HP-UX 11i v3. The system administrator can now adjust the size of a base page used in managing system memory, by invoking the kctune command to change the tunable base_pagesize and then rebooting the system. The larger base page sizes allow memory to be managed more efficiently, resulting in a decrease in kernel memory usage. This can also result in an increase in system performance for some workloads. This white paper describes this new feature, how the size of a base page affects the system, and how to use the new feature, including guidance on choosing a base page size for your configuration. What is Tunable Base Page Size? A base page is the unit of physical memory management in the HP-UX kernel. It is the smallest block of physical memory that can be allocated for storing data and code by the kernel. Base pages are also the smallest unit of memory protection. The size of a base page has been made a tunable parameter for Integrity platforms in Update 3 to HP-UX 11i v3. Previously, the base page size was fixed by the constant NBPG, whose value was 4 kb. Virtual memory, as seen by applications, is managed by the kernel in units of base pages. Applications can also use "large pages", which are logical entities whose size is a multiple of the system base page size. The getconf(1) command can be used to obtain the base page size of the system. Typical usage to obtain the system page size is: getconf PAGE_SIZE On HP-UX Integrity systems without the Tunable Base Page Size enhancement, and on all HP 9000 (PA-RISC) platforms, the base page size is always 4 kb. How Base Page Size Affects the System The HP-UX kernel maintains several tables of per-page data for each base page in the system. As systems support larger amounts of physical memory, the overhead of managing the physical memory of the system becomes significant. This overhead is both a space issue (amount of memory consumed by this data) and a performance issue (time that the kernel spends managing the system memory) The amount of memory consumed by these tables is sensitive to the size of a base page. The larger the base page size, the smaller the amount of memory consumed by the tables. Reducing the size of the tables makes more of the system s physical memory available to applications. Many of the kernel s data structures and algorithms that manage system memory are directly impacted by the number of pages that the kernel must manage. Increasing the base page size from 4kB to 16kB, for example, reduces the overhead of managing memory by a factor of 4. Figure 1 below shows how the base page size impacts kernel memory consumption for static tables overall, and for the largest of those tables, the pfdat, or Page Frame Data, table. 1

Figure 1: Kernel memory overhead as a function of base page size Integrity Virtual Machines Platform Manager The Integrity Virtual Machines Host, also called the Platform Manager, derives great benefit from using 64 kb base pages. The Host automatically tunes the base page size appropriately. Under no circumstances should the base page size for the Integrity Virtual Machines Host be changed to any value other than 64 kb. Of course, the virtual machine instances, also called the guests, can use whatever base page size is appropriate for the guest workload. The primary benefit of large base pages to Integrity Virtual Machines configurations is the reduction in kernel memory consumption in the Host. Using 64kB base pages in the Host frees up physical memory that can then be assigned to Guest Virtual Machines. Working with Tunable Base Page Size Tuning the base page size The base page size is tuned by invoking the kctune(1m)command to change the kernel tunable parameter base_pagesize. The parameter exists on Integrity platforms only; the base page size cannot be changed from the fixed value of 4 kb on HP 9000 (PA-RISC) platforms. The base page size is expressed as a value in kilobytes. The value 4 represents a base page size of 4 kb (4096 bytes). base_pagesize kernel tunable parameter The default value of the base_pagesize parameter is 4. The Tunable Base Page Size enhancement has absolutely no effect as long as base_pagesize remains at its default value. The minimum value to which base_pagesize can be tuned is also 4. It is not possible to adjust the parameter to a value smaller than the previous fixed default. The maximum value to which base_pagesize can be tuned is 64. The other permitted values are 8 and 16. Please refer to the base_pagesize(5) manpage for complete details 2

Static tuning only The base_pagesize tunable is not dynamic: any change requires a reboot to take effect. Installing Tunable Base Page Size enhancement The code to enable the Tunable Base Page Size enhancement is available as a set of patches in Update 3 to HP-UX 11i v3. Updating the HP-UX core All of the files needed to update the HP-UX 11i v3 core for the Tunable Base Page Size enhancement are installed as a consequence of installing the patch PHKL_38038. A reboot following installation is required. Installing the enhancement does not change system behavior in any way. It is required to in increase the base_pagesize tunable above its default value of 4 to change the system behavior. Updating system utilities A number of system utilities are sensitive to the base page size, and so must be updated before they are used on a system with the base page size tuned to a value larger than the default. Since these utilities are not part of the core, they are not installed along with PHKL_38038. Table 1 shows the utilities that must be updated if they are used with large base pages: Table 1. Utilities requiring update before use with large base pages Utility Minimum required version Included in Update 3? Java Virtual Machine, version 4 1.4.2.19 YES, as T1457AA version 1.4.2.19 Java Virtual Machine, version 5 Java Virtual Machine, version 6 5.0.13 YES, as Java15JRE version 1.5.0.13.00 6.0.1 YES, as Java60JRE version 1.6.0.01.00 ONC+ B.11.31.01 YES, as ONCplus version B.11.31.04 FiberChannel td driver B.11.31.0809 YES, as FibrChanl-00 version B.11.31.0809 HP SIM 5.2 Update 2 YES, as HPSIM-HP-UX version C.05.02.02.00 3

Updating applications While most applications are not sensitive to the base page size, some applications will not operate properly when the base page size is tuned to a value larger than the default. Some applications implicitly assume that the size of a base page is 4 kb. This assumption may lead to inappropriate sizes for data objects created by the application. For example, an application under an object size constraint of 1 MB, may incorrectly assume that an object whose size is 200 pages is smaller than the 1 MB limit. Instructions for application programmers to make their code independent of base page size are given in the appendix Application programming for tunable base pages. It is also possible for applications that don't depend directly on the base page size to have problems when used with large base pages. For example, the increase in object size due to rounding to a page boundary can cause stack overflow. This is likely only for applications whose stack size is adjusted to a value smaller than the default. HP has not validated that all third-party applications and drivers work properly with large base pages. For that reason, the kctune command displays a warning message when base_pagesize is tuned to a value larger than 4. System administrators are advised to conduct controlled experiments with their entire application stack before tuning the base page size to a value larger than the default. Known limitations Java Virtual Machine Earlier versions of the Java Virtual Machine do not operate properly with large base pages. The versions delivered with Update 3, and the latest versions available via Web release, work correctly at all base page sizes. For the 1.4 release stream, version 1.4.2.19 is the minimum version required for large base pages. For the version 5 release stream, version 5.0.13 (also called 1.5.0.13) is required. The version 6 release stream, starting with version 6.0.1, works correctly at all base page sizes. Some applications bundle the Java Virtual Machine with their application distribution. That is, even though the system may have the most current Java version, the application points to a Java directory that was installed along with the application. If the bundled Java version is older than the minimum required version, the application will not operate properly with large base pages. List of such applications identified by HP is given in the appendix Application Compatibility. It is possible that additional applications also bundle older Java versions. HFS Filesystem The HFS filesystem does not support configurations where the HFS block size is smaller than the system base page size. Misconfigured HFS filesystems will cause the mount command to fail. There are two ways to avoid this problem: 1. Use only VxFS filesystems (the preferred solution). 2. Make your HFS filesystems with blksize=64k, fragsize=8k. It is strongly recommended that systems configured with non-default settings of the base_pagesize parameter use only the VxFS filesystem to avoid problems with HFS. Information on how to convert existing HFS filesystems to VxFS can be found in the HP-UX 11i v2 manual, HP Servers and Workstations: Managing Systems and Workgroups, A Guide for HP-UX System Administrators, at http://docs.hp.com. 4

Note 1 The default block size for the HFS filesystem is 8kB. HFS filesystems created with a filesystem block size at least equal to the base page size function correctly. HFS filesystems configured with a 64k block size (and an 8k fragment size) will work with any supported base page size. Please refer to the mkfs_hfs(1m) manpage for complete details. Performance considerations The kernel operations that manage virtual memory on behalf of applications are more efficient when the base page size is larger. Starting processes with large code segments and managing the virtual memory when large data objects are created can be significantly faster with large base pages. Since physical memory is managed in units of base pages, some memory allocations must be rounded up to a page boundary. This can cause inefficiencies if the object size is significantly smaller than a base page. For large objects, those a megabyte in size or larger, the rounding effect is of no consequence, even for the largest base page size. For objects a few hundred bytes in size, the rounding impact can be substantial at the larger base page sizes. Filesystem reads and writes through the Unified File Cache are done at page granularity. If even one byte is written, the entire base page must be transferred. Depending on the application data reference pattern, the larger base page sizes can cause more disk traffic than the default size. This does not hold true if the application is using Direct I/O. Direct I/O bypasses the File Cache and goes directly to the disk. Some aspects of large base pages are an unqualified performance benefit. Other aspects depend on the behavior of applications. On balance, HP has found that the performance "sweet spot" is a 16 kb base page size. Many application workloads will show a slight performance improvement with 16 kb base pages as compared to the default. Some workloads, particularly those that manipulate a large number of small objects, are better off with 8 kb or 4 kb base pages. It would be unusual for an application workload to benefit from base pages as large as 64 kb, and therefore we do not recommend that configuration. Finally, as customer applications grow to use larger and larger amounts of memory, they would benefit from an increase in the system base page size. Summary The Tunable Base Page Size enhancement to HP-UX 11i v3 Update 3 changes the base page size managed by the HP-UX kernel on Integrity platforms to be a tunable parameter. This allows customers to select the base page size that is best suited to their application workload. 5

Appendices Application programming for tunable base pages The standards-based interfaces needed to obtain the base page size of the underlying system have long existed in HP-UX. Most portable application code already uses these interfaces. Some applications, however, may still rely on hard-coded notions of the system base page size. This document provides example code that can replace many legacy idioms in a page size independent manner. The getconf(1) command provides this capability for use in scripting. The sysconf(2) and getpagesize(2) system calls provide this ability for compiled code. Using these interfaces enables application developers write page size independent code in user space. In the kernel, the macros NBPG and PGSHIFT are the basis for a whole set of macros that do pageoriented operations. In HP-UX 11i v3, these macros were modified in a way that is mostly sourcecompatible, but there are some subtle (and not so subtle) syntactic and semantic differences. For example, it is no longer be possible to dimension an array at compile time using NBPG, because the value is not known until run time. These changes have been present in 11i v3 since first customer shipment, so existing 11i v3 kernel-intrusive code will not see a change when the Tunable Base Page Size enhancement is installed. Obsolete programming constructs HP-UX releases prior to 11i v3 included a manifest constant called NBPG in some kernel header files. This constant corresponded to the number of bytes in a base page. While not intended for application use, it was nonetheless visible in the application-visible namespace under some circumstances. The constant is obsolete, was removed from 11i v3 header files, and should not be used by applications. In 11i v3, the NBPG macro was retracted into the _KERNEL namespace within the HP-UX header files, so it is now no longer visible to user space. On Integrity systems, NBPG is no longer a constant, but now returns the current base page size of the running kernel. Proper programming practices HP-UX provides two system calls that are capable of returning the base page size of the running kernel: sysconf(2) and getpagesize(2). These calls can be used in any user space code that need to have knowledge of the underlying base page size. A brief summary of each is provided below. Note that the principal difference between them is that sysconf(2) returns a signed long, while getpagesize(2) returns a signed int. getpagesize(2) is in fact a wrapper routine for sysconf(_sc_page_size). See the sysconf(2) and getpagesize(2) man pages on any HP-UX 11i release for full details. System calls that create and operate on memory-mapped objects generally require that addresses and file offsets used for these objects be aligned and sized according to the value returned by sysconf() when passed _SC_PAGESIZE or _SC_PAGE_SIZE. These calls include, but are not necessarily limited to mmap(2), munmap(2), mprotect(2), msync(2), and madvise(2). The system configurable variable PAGESIZE mentioned in the pthread(3t) manual pages is affected by the setting of the base_pagesize tunable. A call to sysconf(_sc_page_size) should be made at program startup time to obtain the page size currently in effect. 6

Application compatibility HP performs Customer Usage Testing on each update. The results of that testing for HP-UX 11i v3 Update 3 are summarized below. The material in this section will be updated as the details change. The most current version of this document is available online at http://docs.hp.com/en/en/14670/enw-tbps-tw.pdf. Oracle Database Manager Versions of the Oracle Database Manager, both single instance and RAC, up to and including Oracle 11gR1 bundle older versions of Java that do not operate properly with non-default base page sizes. These products should not be used with non-default base page sizes. There were no problems with the Oracle client at any base page size. SAP ERP product The installation of SAP uses a Java Virtual Machine bundled in with the distribution. The version of Java that is bundled does not work properly on a system with a base page size of 64 kb. The workaround is to complete the installation with a system base page size of 4 kb, 8 kb, or 16 kb, and then adjust the base page size to 64 kb, if desired. SAS SAS version 9.1.3sp4 works properly only when the system base page size is 4 kb or 8 kb. The SAS will installation process will hang in the base page size is 16 kb or 64 kb. Starting with version 9.2, SAS works properly at all base page sizes. Tuxedo The installation of Tuxedo uses a Java Virtual Machine bundled in with the distribution. The version of Java that is bundled, 1.3.0.7, causes a core dump on a system with a base page size of 64 kb. The workaround is to complete the installation with a system base page size of 4 kb, 8 kb, or 16 kb, and then adjust the base page size to 64 kb, if desired. Veritas Enterprise Services Administrator The Veritas Enterprise Services Administrator, part of the Symantec VxVM product, as delivered into Update 3, bundles in a version of the Java Virtual Machine that does not work with values of the system base page size greater than the default. This situation is remedied by a patch to the VxVM product that was released in October 2008. The patch, whose identifier is PHCO_37694, is now available on the HP patch hub. WebLogic 9.1 An installation will not install all directories when performed on a system with a base page size of 64 kb. A workaround is to complete the installation with a system base page size of 4 kb, 8 kb, or 16 kb, and then adjust the base page size to 64 kb, if desired. Alternatively, the Java stack size can be increased, for example, java -Xss512k -jar server910_generic.jar WebSphere 6.1 The installation of WebSphere uses a Java Virtual Machine bundled in with the distribution. The version of Java that is bundled does not work properly on a system with a base page size of 64 kb. The workaround is to complete the installation with a system base page size of 4 kb, 8 kb, or 16 kb, and then adjust the base page size to 64 kb, if desired. 7

For more information See the HP-UX manpage base_pagesize(5) for details about the base_pagesize tunable. See the manpage kctune(1m) for information about tuning kernel parameters. See the manpage getconf(1) for details of obtaining system configuration information, such as base page size. To help us improve our documents, please provide feedback at www.hp.com/solutions/feedback Technology for better business outcomes Copyright 2008 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. UNIX is a registered trademark of The Open Group. 14670-ENW-TBPS-TW, October 2008 8