Download OpenEdge RDBMS Shared Memory

OpenEdge Release 10.1B and higher
RDBMS Shared Memory
February 13, 2008
By Keith Sanborn and Gus Bjorklund
Overview
A set of shared memory enhancements were introduced in OpenEdge Release 10.1B that enable
increased shared memory addressability and better management of the shared memory allocation
process. Increased shared memory addressability allows larger pools of shared memory for
databases, which can reduce operating system resource requirements. The ability to customize
shared memory allocation allows user specified shared memory segment sizing to meet the needs
of specific environments. This paper describes the extent of the changes concentrating on 32-bit
platforms, and describes how the changes will affect OpenEdge in your particular environment.
OpenEdge Shared Memory Changes
The shared-memory architecture for the OpenEdge RDBMS has changed in two important ways:

First, the method used by the OpenEdge RDBMS to address shared memory is enhanced so
that OpenEdge can now address larger amounts of allocated shared memory on both 32 and
64-bit operating systems.

Second, you can now specify the maximum shared-memory segment size that the OpenEdge
RDBMS will allocate. Previously, the maximum segment size was fixed at 128 megabytes
for 32-bit systems, and 1 gigabyte for 64-bit systems. These enhancements enable the
OpenEdge RDBMS to allocate large shared memory pools using the segment sizes you
specify. However, each operating system imposes additional limitations that the OpenEdge
RDBMS must follow.
OpenEdge RDBMS Release 10 can now address larger amounts of allocated shared memory
than previous releases. On 32-bit systems, the addressable limit is 4 gigabytes and on 64-bit
systems, the addressable limit is 8 terabytes. These limits are architectural limits and are often
reduced in practice by limitations imposed by the underlying operating system and other factors.
This is particularly true on 32-bit systems because the entire process address space is limited to 4
gigabytes and you cannot use all of it for shared memory. In practice, most 32-bit OpenEdge
releases are limited to a maximum of approximately 2 gigabytes of shared memory.
In addition to providing for increased shared memory addressability, OpenEdge Release 10 also
introduces the database configuration parameter, –shmsegsize, or Shared Memory Segment Size.
This parameter allows the setting the maximum size of shared memory segments for each
database at database startup time. If –shmsegsize is omitted during database startup, OpenEdge
will size shared memory segments automatically based on the shared memory requirements of
the database.
No discussion of shared memory can begin without first considering some operating system
principles, and one of the most important to understand is the concept of the Process Address
Space.
The 32-bit Process Address Space
Many factors affect the total amount of shared memory that the OpenEdge RDBMS can allocate,
including program size, stack space required, shared libraries, memory-mapped files, and
memory allocated from the heap. The address space and memory allocation policies used by
different operating systems are complex and vary considerably from one operating system to
another. Explaining the details of each memory model for each operating system is beyond the
scope of this paper, but the following diagram provides a sample layout for a generic 32-bit
process address space.
Generic 32-bit Operating System
Process Address Space
Kernel
Program &
Program Data
Most 32-bit operating systems divide the 4 GB address
space into different address ranges or sections. Each
section has explicit purposes. For example, the
operating system must provide a section of the address
space for the program or the code that is running, and
additional sections for the program’s stack and static
data.
Stack
Heap
Global Data
Including,
Shared Memory &
Shared Libraries
or DLL’s
4 GB
Process
Address
Space
Consider the diagram to the left and notice that the
entire 4 GB process address space is divided into
multiple sections including, the kernel, the program
and its data, stack space, heap space used for dynamic
memory allocation, and global data that includes
shared memory and shared libraries or DLL’s.
The size of each section varies depending on operating
system and does affect the size of allocated shared
memory segments.
2
The following table summarizes some of the key details of the process address space for Solaris,
Linux, Windows, AIX, and HP-UX. The table shows the total memory allocation per process,
the memory required for the kernel, and the memory available in the user address space.
Operating
System
Total Memory
per Process
Solaris
4 GB
Linux
Reserved
Kernel
Memory
User Address Space
Available
~4 GB shared between program, its data & stack, and
shared memory
4 GB
Kernel and user
space are separated
into their own 4
GB address spaces
1 GB
Windows
4 GB
2 GB
2 GB shared between program, its data & stack, and
shared memory
AIX
4 GB divided into
sixteen 256 MB
segments
One 256 MB
segment
One 256 MB segment for the program
One 256 MB segment for the program’s data & stack
Twelve 256 MB segments available for shared memory
HP-UX
4 GB divided into
four 1 GB
quadrants
3 GB shared between program, its data & stack, and
shared memory
The four quadrants are organized as follows:
1. 1 GB Program
2. 1 GB program data
3. 1 GB global space
4. .75 GB global space
A 32-bit program can allocate and manage a maximum
1.75 GB of shared memory.
Testing the Limits
Progress ran a series of tests to determine the maximum amount of shared memory that
OpenEdge Release 10.1B and Release 10.1C are able to allocate for a single database. Testing
took place on Solaris, Linux, Windows, AIX, HP-UX, and Unixware using the appropriate 32-bit
versions of OpenEdge. A test was considered successful after the database started, a simple
ABL query ran and returned the correct records, PROMON ran and reported correct metrics, and
the database shutdown cleanly.
The table below describes the conditions that define the tests and the resulting maximum-shared
memory size that the 32-bit OpenEdge RDBMS is able to allocate. The test variables include:




The platform including its architecture and operating system release
The OpenEdge release information
The database block size
The startup configuration parameters used when starting the database
The only two database configuration parameters used during testing were –B (Database Buffer
Pool) and –shmsegsize (Shared Memory Segment Size). Defaults were accepted for all other
parameters. –B provides a quick and efficient method to increase the shared memory
3
requirements for the database and the buffer pool is initialized so all of shared memory is
“touched”. The –shmsegsize parameter provides a more efficient way for the operating system
to fit smaller shared memory segments into the address space. By manipulating these two
parameters, the amount of allocated shared memory was adjusted up and down. The maximum
allocated shared memory achieved is shown in the following table.
Operating System
and Hardware
OpenEdge
Release
Solaris 5.10
Sun Sparc - 64-bit
32 GB Ram
Shared Memory Allocated
(as reported by PROMON R&D/Status:
Shared Resources)
8K database block size
32-bit OE 10.1C
for Solaris
–B 460000
–shmsegsize 512
Total shared memory = 3,884 MB
32-bit OE 10.1B
for Solaris
–B 480000
–shmsegsize 512
Total shared memory = 3,892 MB
Linux RH EL 4 U4
Intel x86 - 32-bit
1,010 MB Ram
Database
Information &
Startup Arguments
8K database block size
32-bit OE 10.1C
for Linux
–B 346000
–shmsegsize 512
Total shared memory = 2,807 MB
32-bit OE 10.1B
for Linux
–B 347000
–shmsegsize 512
Total shared memory = 2,810 MB
Windows 2003 Server
4K database block size
Intel x86 - 32-bit
32 GB Ram
32-bit OE 10.1C
for Windows(1,2)
–B 406000
–shmsegsize 512
Total shared memory = 1,711 MB
Client/server ABL
connection
32-bit OE 10.1B
for Windows
–B 420000
–shmsegsize 512
Total shared memory = 1,746 MB
Windows 2003 Server
4K database block size
Intel x86 - 32-bit
32 GB Ram
32-bit OE 10.1C
for Windows
–B 350000
–shmsegsize 256
Total shared memory 1,477 MB
Self-Service ABL
Connection3
32-bit OE 10.1B
for Windows
–B 350000
–shmsegsize 256
Total shared memory 1,474 MB
1
OpenEdge for Windows does not support the Windows /3Gb parameter.
On 32-bit versions of Windows, the /3GB parameter enables 4 GT RAM Tuning, a feature that enlarges the usermode virtual address space to 3 GB and restricts the kernel-mode components to the remaining 1 GB.
( http://msdn2.microsoft.com/en-us/library/ms791558.aspx )
3
The OpenEdge ABL client is a larger program, loads multiple DLLs and has larger data and heap requirements
than the database broker or server. The additional resource requirements reduce the amount of address space
available for shared memory mapping.
2
4
Operating System
and Hardware
OpenEdge
Release
AIX 5.2
Database
Information &
Startup Arguments
Shared Memory Allocated
(as reported by PROMON R&D/Status:
Shared Resources)
8K database block size
IBM Power PC - 64-bit &
IBM Power PC – 32-bit
32 GB Ram
32-bit OE 10.1C
for AIX
–B 340000
–shmsegsize 512
Total shared memory = 2,763 MB
32-bit OE 10.1B
for AIX
–B 345000
–shmsegsize 512
Total shared memory = 2,801 MB
HP-UX 11.11
HP hp9000 – 64-bit
PA_RISC
4 GB Ram
8K database block size
32-bit OE 10.1C
for HP-UX
–B 200000
–shmsegsize 256
Total shared memory = 1,669 MB
32-bit OE 10.1B
for HP-UX
–B 202000
–shmsegsize 256
Total shared memory = 1,684 MB
Unixware 7.1.4
Intel x86 Pentium XEON
2 GB Ram
8K database block size
32-bit OE 10.1C
for Unixware
–B 350000
–shmsegsize 512
Total shared memory = 2,839 MB
32-bit OE 10.1B
for Unixware
–B 360000
–shmsegsize 512
Total shared memory = 2,915 MB
Shared Memory Segment Size (–shmsegsize) Configuration Parameter
The new database configuration parameter, –shmsegsize, or Shared Memory Segment Size, was
introduced to allow tuning of the shared memory segment sizes. The valid values for the –
shmsegsize parameter are:

32-bit systems
128, 256, 512, 1024, 2048, 4096 megabytes or
1g, 2g, 4g where g is gigabytes
The 4096 and 4g values are allowed on 32-bits platform, but in reality
a 4 GB, shared memory segment cannot be allocated on 32-bit
platforms. Specifying a –shmsegsize of 4096 or 4g will allow the
OpenEdge creation of shared memory segments greater than 2 GB on
systems that support segments greater than 2 GB.

64-bit systems
1024, 2048, 4096, 8192, 16383, 32767 megabytes or
1g, 2g, 4g, 8g, 16g, 32g where g is gigabytes
5
The –shmsegsize configuration parameter allows control over the maximum segment size used
by the OpenEdge RDBMS. The actual segment sizes created may be smaller than this due to a
variety of factors such as:




The actual amount of shared memory needed may be less than the requested –shmsegsize.
The operating system's maximum segment size may less than the requested –shmsegsize.
The amount of available unused address space may be less than the requested –shmsegsize.
The value of SHMMAX and other kernel tunables may impose lower limits.
The initial maximum-shared segment size is calculated based on your shared memory
requirements and the support provided by your operating system. If –shmsegsize is omitted,
OpenEdge will attempt to fit all the shared memory required into the fewest shared memory
segments possible. Previously the fixed maximum was 128 megabytes for 32-bit systems and 1
gigabyte for 64-bit systems. When supported by the operating system OpenEdge creates fewer
but larger segments. For some operating systems, this enables use of slightly larger database
buffer pools with 32-bit versions of the OpenEdge RDBMS.
The Impact on Your Database
The database data structures placed into shared memory are larger in Release 101B and higher
than they were in prior releases. In particular, shared pointers are now 64 bits wide for both 64bit and 32-bit versions of the OpenEdge RDBMS. Previously they were 32 bits wide on 32-bit
versions. As a result, some data structures that are quite numerous, such as database buffer-pool
headers and lock table entries, consume noticeably more memory than before.
While the changes in release 10.1B and later have definite advantages, they are also not without
some drawbacks. Thus, if you have a 32-bit system and you upgrade from an earlier release of
OpenEdge, you might encounter errors when you start the database server or when clients try to
connect to it in self-serving mode because the larger size shared segments cannot be allocated
within the available address space. The following table identifies the potential problems that
could occur and presents their solutions:
Problem
You cannot start the server because data
structures do not fit and they did before.
OpenEdge Message(s) in the database log
file or standard output:
Solution
Lower the value of –B by 10 percent to overcome
this problem if you are close to the maximum
amount of shared memory for the system and the
value of –B is large. Continue to reduce the
shared memory requirements for the database
until it starts successfully.
The data space of the process
is not enough for the shm
segment (1176)
6
Problem
Solution
The server starts, but clients cannot connect
in self-serving mode. This may be caused by
lack of sufficient contiguous free address
space in combination with the larger default
segment size used by the server.
Lower the value of –shmsegsize. When the
address space has several smaller holes, the
operating system may be able to locate the
segments in them. This is likely to happen in
Windows, where the system has mapped .dll's in a
manner such that there are no large chunks of free
address space. The problem is similar to filling a
box with variable size blocks of wood. Small
blocks are easier to fit into the box than larger
ones.
Or
A client that connects to several databases
in self-serving mode may no longer be able
to do so. This can happen due to larger
segments.
OpenEdge Message(s) in the database log
file or standard output:
Unable to attach shared
memory “segment name” error
##. (1720)
There may be a conflict between large
shared ABL procedure libraries (that are
mapped in a single contiguous memory
segment) and large shared memory
segments created by the server.
Lower the value of –shmsegsize.
7
Summary
The shared memory addressing enhancements made in OpenEdge Release 10.1B and higher
increases the shared memory addressability on 32-bit and 64-bit platforms and increases the
flexibility of shared memory allocation when the –shmsegsize configuration parameter is used.
On 32-bit platforms, the OpenEdge RDBMS will allocate more than 2 GB of shared memory if
the operating systems and machine supports the increased shared memory allocations. Even
though the OpenEdge RDBMS can address larger shared memory pools than it did in prior
releases, it can still only allocate and utilize as much shared memory as the operating system
allows. Understanding the basic principles used by your operating system to “build” a process
address space will help you understand how to set your configuration parameters when you start
your databases.
References
AIX 5L Version 5.3 General Programming Concepts: Writing and Debugging Programs
SC23-4896-04, Fifth Edition (November 2007)
Solaris Internals, Jim Mauro and Richard McDougall, Sun Microsystems, Inc. 2001
Inside Microsoft Windows Internals, Covering Windows 2000, Windows XP, Windows Server
2003, Mark E. Russinovich, David A. Solomon, Jim Allchin, - Microsoft Press, 2005
8