Download Improving high availability in WebSphere Commerce using DB2 HADR

Improving high availability in WebSphere Commerce
using DB2 HADR
Xiao Qing (Shawn) Wang
Software Engineer
IBM WebSphere Commerce
Ramiah Tin
Software Architect
IBM WebSphere Commerce
March 28, 2007
© Copyright International Business Machines Corporation 2007. All rights reserved.
Introduction.......................................................................................................................................2
Overview...........................................................................................................................................3
Prerequisites and configuration information .....................................................................................3
Knowledge requirement................................................................................................................3
Software configuration..................................................................................................................4
Hardware configuration ................................................................................................................4
Recommendation ..........................................................................................................................5
Introduction to topology with HADR and TSA ................................................................................5
High Availability Disaster Recovery.............................................................................................5
Tivoli System Automation ............................................................................................................6
Reliable Scalable Cluster Technology...........................................................................................7
Automatic Client Reroute .............................................................................................................7
Connection pool purge policy in WebSphere Application Server .................................................8
Typical WebSphere Commerce environment with HADR and TSA ............................................9
Setting up the topology ...................................................................................................................10
Installing WebSphere Commerce................................................................................................11
Configuring HADR on a primary/standby database ...................................................................11
Enabling and configuring client reroute in a HADR environment..............................................14
Installing Tivoli System Automation ..........................................................................................15
Defining and administering a TSA cluster ..................................................................................17
Configuring and registering instance and HADR with TSA for automatic management ...........18
Running WebSphere Commerce stress tests with HADR and TSA................................................21
Normal operations with HADR enabled .....................................................................................21
Testing process failure: Standby Instance Failure.......................................................................21
Testing physical failure: Standby system failure.........................................................................22
Testing process failure: Primary instance failure ........................................................................22
1
Testing physical failure: Primary system failure.........................................................................22
Switching the roles of primary database and standby database: Graceful takeover....................23
Performance overhead of HADR ................................................................................................24
Impact of HADR with NEARSYNC mode.............................................................................24
Failure duration in different disaster situations.......................................................................25
Restrictions and recommendations .................................................................................................30
Restrictions in this topology .......................................................................................................30
Recommendations.......................................................................................................................30
Appropriately set your synchronization modes for HADR.....................................................30
Use the LOGINDEXBUILD database parameter ...................................................................30
Set appropriate BUFFPAGE size to eliminate enough database physical read.......................31
Update C++ library version in AIX.........................................................................................31
Keep the database logs in different devices ............................................................................31
Conclusion ......................................................................................................................................31
Resources ........................................................................................................................................31
About the authors............................................................................................................................32
Introduction
As Internet commerce becomes more popular and important to corporations, WebSphere®
Commerce customers are investigating high availability and disaster recovery strategies to
ensure their commerce site is available to users even in cases of disaster, and to ensure valuable
data can be recovered with minimal disruption to their site. WebSphere Commerce runs
seamlessly on top of WebSphere Application Server (Network Deployment Edition), which
provides application level clustering capability through horizontal and vertical clustering
topologies. With redundant Web servers and application servers, and with smart routing, a
WebSphere Commerce application already provides high availability. To complete the high
availability and disaster recovery picture, you need to consider the database tier l. This article
describes a feature provided by IBM® DB2® Universal Database (UDB) for Linux, Unix, and
Windows® v8.2 to achieve that goal.
DB2 UDB v8.2 provides a new feature called High Availability Disaster Recovery (HADR),
which supports a configuration where an active primary database processes all user database
requests. A standby database server, possibly in a remote location, is kept "in synch" with the
changes occur in the primary database. If the primary database server fails, the standby server
takes over as the active primary database server. Cluster management tools are used to monitor
the health of the primary database server and to automatically coordinate the takeover process if
the primary database server fails.
2
Overview
This article focuses on an environment that includes:
•
•
•
•
WebSphere Commerce
WebSphere Application Server (Network Deployment Edition)
Tivoli System Automation
DB2 UDB v8.2 with the HADR feature
Setting up the topology describes the steps for configuring DB2 UDB v8.2 for HADR and
Automatic Client Reroute (ACR) with WebSphere Commerce. Setting up the topology also
describes the configuration required to use Tivoli System Automation (TSA) to monitor the health
of the environment, and to integrate the automatic scripts provided by DB2 v9.1 to coordinate and
automate disaster recovery. Running stress tests with HADR and TSA describes performance
tests that simulate a heavy load environment, illustrates the implications of using an HADR
environment, and measures the duration of failover and error. It also lists recommendations and
suggestions for implementing a solution that uses DB2 UDB v8.2 with the HADR feature and
ACR capabilities together with WebSphere Commerce.
Throughout this article, there are detailed procedures, sample scripts, and output examples so that
you can apply similar techniques in your environment.
This article is intended for:
z
Technical architects and software developers deploying WebSphere Commerce for high
availability.
z
Database administrators managing the database.
Prerequisites and configuration information
The environment described in this article includes multiple products as follows:
•
•
•
•
WebSphere Commerce v6.0.0.1
DB2 UDB v8.2.3
Tivoli System Automation v2.1
AIX v5.2
Knowledge requirement
This section covers required background knowledge, as well as hardware and software. Read this
section before installing and configuring the software described in this article.
Basic knowledge of WebSphere Commerce:
http://publib.boulder.ibm.com/infocenter/wchelp/v6r0m0/index.jsp
3
Basic knowledge of DB2 UDB Version 8.2, HADR and ACR
Information about DB2 UDB Version 8.2:
http://publib.boulder.ibm.com/infocenter/db2luw/v8/index.jsp
Information about HADR:
http://publib.boulder.ibm.com/infocenter/db2luw/v8//index.jsp?topic=/com.ibm.db2.udb.doc/core/
c0011585.htm
Basic knowledge of TSA cluster manager software
Information about TSA V2.1:
http://publib.boulder.ibm.com/tividd/td/IBMTivoliSystemAutomationforMultiplatforms2.1.html
Information about RSCT:
http://publib.boulder.ibm.com/epubs/pdf/bl5adm11.pdf
Basic knowledge of AIX operating system:
http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp
Software configuration
The specific versions of the software described in this paper are:
z
Operating system: AIX v5.2, maintenance level 5 (5200-05)
z
WebSphere Commerce: WebSphere Commerce v6.0.0.1
z
WebSphere Application Server: Network Deployment Edition v6.0.2.5
z
DB2 UDB: DB2 UDB v8.2.3 (64 bit)
z
Tivoli Product: Tivoli System Automation v2.1
z
RSCT: RSCT v2.3.7.1
Hardware configuration
The 2-tier hardware configuration described in this article is two machines for WebSphere
Commerce database nodes (primary and standby), each with the following configuration:
z
z
z
Processors: IBM Power PC® 4 CPU 1.40 GHz
Memory: 16 GB
Hard disk: 140GB*1 + 70GB*1
One machine for WebSphere Commerce node and one machine for TSA node, each with the
following configuration:
z
z
z
Processors: IBM Power PC® 4 CPU 1.40 GHz
Memory: 16 GB
Hard disk: 140GB*1
4
Recommendation
To achieve optimal performance with HADR, ensure that your system meets the following
requirements for hardware, operating systems, and DB2 UDB:
•
Use the same hardware and software for the system where the primary database resides and
for the system where the standby database resides. This helps to maintain the workload when
the standby system takes over as primary.
•
The operating system on the primary and standby databases should be the same version,
including patches.
•
A TCP/IP interface must be available between the HADR host machines, and a high-speed,
high-capacity network is recommended.
•
The database versions used by the primary and standby databases must be identical. During
rolling upgrades, the database version of the standby database may be later than the primary
database for a short time. The DB2 UDB version of the primary database can never be later
than the standby database version.
Introduction to topology with HADR and TSA
This section describes roles and responsibilities of each software products that are involved in this
topology, and how they work together when combined together.
High Availability Disaster Recovery
High availability (HA) strategies enable database solutions to remain available to process client
application requests despite hardware or software failure, which can ensure:
•
•
•
Transactions are processed efficiently, without appreciable performance degradation.
Systems recover quickly when hardware or software failures occur, or when disaster strikes.
Software that powers the enterprise databases is continuously running and available for
transaction processing.
HADR is a database replication feature that protects against data loss in the event of a partial or
complete site failure by replicating changes from a source database, called the primary database,
to a target database, called the standby database. Figure 1 illustrates the architecture of HADR.
5
Figure 1. HADR architecture
Figure 1 illustrates the following key aspects of a HADR environment:
•
Enabling HADR on a primary server starts up a process called db2hadrp that communicates
with the standby server. At the same time, on the standby server, a process called db2hadrs is
started, which receives log records from the primary server, writes them to the log file on the
standby server, and applies those transactions to the data and index pages.
•
When an application is running on the primary server, normal insert/update/delete activity
results in log records being written to the log buffer. When the log buffer is full, or whenever a
transaction commits, the log buffer is flushed to disk (to the log files) prior to the application
receiving a successful return code to its commit request.
•
When the primary database is in peer state, log pages are shipped to the standby database
whenever the primary database flushes a log page to disk. The log pages are written to the
local log files on the standby database to ensure that the primary and standby databases have
identical log file sequences. The log pages can then be replayed on the standby database to
keep the standby database synchronized with the primary database.
•
Since HADR ensures the primary and standby databases have identical log file sequences. If a
disaster happens at the primary site, the standby database server can take over as the new
primary database server to handle client application requests.
Tivoli System Automation
HADR does not automatically monitor the health of a primary database node (for example,
psvt01). A database administrator (or some other mechanism) must monitor the HADR pair
manually (for example, psvt01 and psvt02), and issue appropriate takeover commands in the event
of a primary database failure - this is where TSA automation comes in. TSA manages the
availability of applications running in Linux systems or clusters on xSeries®, zSeries®, iSeries®,
pSeries®, and AIX systems or clusters. It consists of the following features:
•
High availability and resource monitoring: TSA provides a high availability environment. It
offers mainframe-like high availability by using fast detection of outages and sophisticated
knowledge about application components and their relationships.
6
•
Policy based automation:
TSA configures high availability systems through the use of policies that define the
relationships among the various components.
•
Automatic recovery:
TSA quickly and consistently performs an automatic restart of failed resources or whole
applications either in place or on another system of a Linux or AIX cluster. This greatly
reduces system outages.
•
Automatic movement of applications:
TSA manages the cluster-wide relationships among resources for which it is responsible.
•
Resource grouping:
You can group resources together in TSA. Once grouped, all relationships among the
members of the group are established, such as location relationships, start and stop
relationships, and so on.
In summary, TSA is a product that provides high availability by automating resources, such as
processes, applications, IP addresses, and others in Linux-based clusters. To automate an IT
resource (for example, a DB2 database instance), you define the resource to TSA. Furthermore,
these resources must all be contained in at least one resource group. If these resources are always
required to be hosted on the same machine, they are placed in the same resource group. For more
information about TSA, see the Tivoli Software Information Center.
Reliable Scalable Cluster Technology
Reliable Scalable Cluster Technology (RSCT) is a product that is fully integrated into TSA. RSCT
is a set of software products that provides a comprehensive clustering environment for AIX and
Linux. RSCT provides clusters with improved system availability, scalability, and ease of use.
RSCT provides three basic components, or layers, of functionality:
1.
2.
3.
Resource Monitoring and Control (RMC) provides global access for configuring, monitoring,
and controlling resources in a peer domain.
High Availability Group Services (HAGS) is a distributed coordination, messaging, and
synchronization services.
High Availability Topology Services (HATS) provides a scalable heartbeat for adapter and
node failure detection, and a reliable messaging service in a peer domain.
For more information about RSCT, see the IBM Cluster Information Center.
Automatic Client Reroute
Automatic Client Reroute (ACR) is another feature that was first introduced in DB2 UDB v8.2. If
a database application loses communication with a DB2 database server, ACR reroutes that client
application to an alternate database server so that the application can continue its work with
minimal interruption. Rerouting is only possible when an alternate database location has been
specified at the primary database server. ACR is only supported with the TCP/IP protocol.
7
ACR is not tied to HADR. You can use it with HADR, clustering software, in a partitioned
database environment, replication, and so on. ACR automatically and transparently reconnects
DB2 database client applications to an alternate server without the application or end user being
exposed to a communications error. The alternate server information is stored on the primary
database server, and loaded into the client’s cache upon a successful connection to the primary
database server. This means that for a client application to know the alternate database server, it
must first successfully connect to the primary database server.
When ACR is configured, the built-in retry logic alternates between the original primary server
and the alternate server for 10 minutes, or until a database connection is re-established to the
primary database server.
Acquiescently, the retry logic built into ACR will:
•
Try to re-establish a connection to the original primary server to ensure there is no
"accidental" failure.
•
Alternate connection attempts between both the original primary database server and the
alternate database server every 2 seconds for 30-60 seconds.
•
Alternate connection attempts between both the original primary database server and the
alternate database server every 5 seconds for 1-2 minutes.
•
Alternate connection attempts between both the original primary database server and the
alternate database server every 10 seconds for 2-5 minutes.
•
Alternate connection attempts between both the original primary database server and the
alternate database server every 30 seconds for 5-10 minutes.
•
If no connection to the original primary database server is made after all of these attempts, the
SQL30081N error code is returned to the client application.
At the same time, there are two DB2 database registry variables called
DB2_MAX_CLIENT_CONNRETRIES and DB2_CONNRETRIES_INTERVAL that helps to
accurately configure the retry logic of ACR:
z
z
DB2_MAX_CLIENT_CONNRETRIES defines the maximal number for a DB2 client retry to
connect to the database server.
DB2_CONNRETRIES_INTERVAL defines the interval between two different reconnection
attempts.
Note: The tests in this article do not cover these two DB2 database registry variables.
For more information about ACR, see The IBM DB2 Version 8.2 Automatic Client Reroute
Facility.
Connection pool purge policy in WebSphere Application Server
WebSphere Application Server (hereafter called Application Sever) recognizes and handles DB2
database client reroute error codes and cleans up the connections in the connection pool according
8
to the database server change. There are two types of policies you can set in the Application
Server connection pool that specifies how to purge connections when a stale connection or a fatal
connection error is detected by the connection pool manager:
1.
EntirePool
All connections in the pool are marked stale. Any connection not in use is immediately
closed. Any connections in use are closed, and a stale connection exception is returned upon
the next operation on that connection. Subsequent getConnection() requests from the
application result in new connections to the database being opened. When using this purge
policy, there is a slight possibility that some connections in the pool are closed unnecessarily
when they are not stale. However, this is a rare occurrence. In most cases, a purge policy of
EntirePool is the best choice.
2.
FailingConnectionOnly
Only the connection that caused the stale connection exception is closed. Although this
setting eliminates the possibility that valid connections are closed unnecessarily, it
complicates application recovery. Because only the current failing connection is closed, there
is a good possibility that the next getConnection() request from the application returns a
connection from the pool that is also stale, resulting in more stale connection exceptions.
In the configuration described in this article, when the primary server goes down, pooled
connections that are not valid might exist in the free pool. For this reason, the Application Server
purge pool policy should be set to “EntirePool” with the configuration described in this article.
This is true when the purge policy is failingConnectionOnly. In this case, the failing connection is
removed from the pool, but the remaining connections in the pool might not be valid. This is
highly possible because the continued request from end users will use the invalid connections and
result in exceptions. For more details about Application Server connection pool, see the
WebSphere Application Information Center.
Typical WebSphere Commerce environment with HADR and TSA
If the primary database server fails, a DBA can use TSA to monitor a HADR primary database
server and automatically failover to a standby database server. Figure 2 illustrates a common
configuration for this case.
9
Figure 2. Typical WebSphere Commerce environment with HADR and TSA enabled
This is a four-node topology:
•
•
•
•
psvt01 hosts the primary database.
psvt05 hosts the standby database.
psvt07 stands for the WebSphere Commerce instance deployed in Application Server.
psvt03 provides quorum between the primary and standby nodes.
In this configuration, TSA can help to:
•
Monitor the HADR pair for primary database failure (both process failure and system
failure), and issue appropriate takeover commands on the standby database if the primary
database fails.
•
Monitor the status of the standby database. For both process failure and system failure on
the standby server, TSA issues corresponding commands to restart the standby server
•
Provide quorum to avoid a “split” brain. psvt03 acts as a heartbeat node and a tie breaker
in the event of communication failure between psvt01 and psvt05.
Setting up the topology
Once you understand the complete topology of the WebSphere Commerce environment with
10
HADR and TSA enabled, and prepared all of the hardware and software, you can start the set up
procedures. The following five steps sets up the whole environment depicted in Figure 2 above.
Note: In this topology, all of these machines are co-located side-by-side, which means network
delay will not play a significant part of the results.
(P): Primary database server - psvt01
(S): Standby database server - psvt05
(H): TSA heartbeat node - psvt03
(C): WebSphere Commerce node - psvt07
Installing WebSphere Commerce
Follow the installation procedure described in the WebSphere Commerce Information Center.
Review the logs carefully to make sure the following components are installed and configured
successfully and correctly:
•
•
•
•
•
DB2 UDB v8.2.3
WebSphere Application Server v6.0.2.5
IBM Http Server v6.0.2.5
WebSphere Commerce v6.0.0.1
Instance for WebSphere Commerce has been successfully created
After the installation, modify the Application Server connection pool’s configuration, if different:
1.
2.
3.
4.
5.
Logon to the Application Server administration console: http://hostname:9061/ibm/console
Go to the connection pool configuration page by following nested drop-down choices:
Resources Æ JDBC providers Æ demo - WebSphere Commerce JDBC Provider ÆData
Source Æ WebSphere Commerce DB2 DataSource demo Æ Connection pools.
Set the Purge policy to “Entire Pool”.
Set the Maximum connections configuration parameter to an appropriate number.
Maximum connections are larger than the number of maximum concurrent users. For
example, in this article, “Maximum connections” is set to 50 because the tests described
simulate a workload of 50 concurrent users.
Save the configuration and exit.
Configuring HADR on a primary/standby database
You can configure HADR using the command line processor (CLP), the Set Up High Availability
Disaster Recovery (HADR) wizard in the Control Center, or by the corresponding application
programming interfaces (APIs). Use the following procedure to configure the primary and standby
databases for HADR using the CLP:
11
Steps:
1. Enable log archiving, configure other DB2 parameters on the primary database:
(P) $ db2 update db configuration for rmall using LOGRETAIN
RECOVERY
(P) $ db2 update db configuration for rmall using LOGINDEXBUILD
ON
(P) $ db2 update db configuration for rmall using INDEXREC
RESTART
(P) $ db2 update db configuration for rmall using NEWLOGPATH
”/db2log/”
Note: It is highly recommended that you use the NEWLOGPATH configuration parameter to put
database logs on a separate device from the database once the database is created. This protects
your database from media failure where the logs are stored, and improves the overall performance
of database system.
2. Make an offline backup:
(P) $ db2 deactivate db rmall
(P) $ db2 backup db rmall
3. Move the backup image to the standby database host and restore the database to the roll forward
pending state:
(S) $ db2 restore db rmall replace history file
4. Configure HADR and client reroute on the primary database. As primary instance owner, set
HADR related parameters for the primary database:
(P) $ db2 update db configuration for rmall using HADR_LOCAL_HOST
psvt01
(P) $ db2 update db configuration for rmall using HADR_LOCAL_SVC
18819
(P) $ db2 update db configuration for rmall using
HADR_REMOTE_HOST psvt05
(P) $ db2 update db configuration for rmall using HADR_REMOTE_SVC
18820
(P) $ db2 update db configuration for rmall using
HADR_REMOTE_INST db2inst1
(P) $ db2 update db configuration for rmall using HADR_SYNCMODE
NEARSYNC
Note: The configuration for this article uses the NEARSYNC synchronization mode. Transaction
response time is shorter with NEARSYNC than with the other synchronous modes. However,
protection against data loss is greater with other synchronization modes.
5. Configure HADR and client reroute on the standby database. As the standby instance owner, set
HADR related parameters for the standby database:
12
(S) $ db2 update
psvt05
(S) $ db2 update
18820
(S) $ db2 update
HADR_REMOTE_HOST
(S) $ db2 update
18819
(S) $ db2 update
HADR_REMOTE_INST
(S) $ db2 update
NEARSYNC
db configuration for rmall using HADR_LOCAL_HOST
db configuration for rmall using HADR_LOCAL_SVC
db configuration for rmall using
psvt01
db configuration for rmall using HADR_REMOTE_SVC
db configuration for rmall using
db2inst1
db configuration for rmall using HADR_SYNCMODE
Start HADR on the standby database:
(S) $ db2 start hadr on db rmall as standby
Note: It is recommended to start HADR on the standby server before you start HADR on a
primary database server. When you start HADR on a primary database server, the database server
waits up to HADR_TIMEOUT seconds for a standby database server to connect to it. If there is still
no standby database server connected after HADR_TIMEOUT seconds have passed, the HADR
start on the primary database server fails.
6. Start HADR on the primary database:
(P) $ db2 start hadr on db rmall as primary
7. Verify that HADR has been successfully configured:
1. Review the db2diag.log on both the primary and the standby database server to see
whether HADR is configured correctly.
2. Examine the HADR status from a snapshot of both the primary database and standby
database. Figure 3 and Figure 4 show the HADR section returned by the GET
SNAPSHOT command.
Figure 3. Snapshot for the primary database server
13
Figure 4. Snapshot for the standby database server
Recommendation:
By default, the log receive buffer size on the standby database is two times the value specified
for the LOGBUFSZ configuration parameter on the primary database. There might be times
when this size is not sufficient. When the primary and standby databases are in HADR peer
state, and the primary database is experiencing a high transaction load, the log receive buffer on
the standby database might fill to capacity and the log shipping operation from the primary
database might stall. To manage these temporary peaks, increase the size of the log receive
buffer on the standby database by modifying the DB2_HADR_BUF_SIZE registry variable.
Enabling and configuring client reroute in a HADR environment
For a client application to be transparently redirected to an alternate standby database server when
there is a loss of communication with the primary database server, specify that alternate server's
location on the primary database server. To do this, use the UPDATE ALTERNATE SERVER
FOR DATABASE command in the primary database server. Here is an example of the steps to
specify an alternate database server:
(P) $ db2 update alternate server for database rmall using
hostname psvt05 port 50000
(S) $ db2 update alternate server for database rmall using
hostname psvt01 port 50000
Note: Port 50000 is the TCP/IP port used by the DB2 database client to communicate with the
alternate DB2 database server.
Figure 5. Directory information in DB2 client
14
The alternate database server information is stored on the primary database server, and loaded into
the client's cache upon a successful connection to the primary database server (Figure 5). This
means that for a client application to know the standby server, it must first successfully connect to
the primary server.
As well as identifying an alternate database server on the primary database server, you must
configure DB2 database clients to detect when the primary database server has failed (stopped
responding) and redirect application connections to the alternate database server. The amount of
time the DB2 database client takes to realize that the primary has stopped responding depends on
configuration parameters. There are three ways to configure a DB2 database client to wait for a
timeout period before concluding that a database server has stopped responding:
•
The DB2 database registry variable called DB2TCP_CLIENT_RCVTIMEOUT specifies
the number of seconds the DB2 database client will wait for a response from the primary
database server before deciding the database server has stopped responding. You can use
the following command to set this registry variable:
(C) $ db2set DB2TCP_CLIENT_RCVTIMEOUT = 60
•
DB2TCP_CLIENT_CONTIMEOUT specifies how long the DB2 database client will wait
on a connection before deciding the database server has stopped responding. You can use
following command to set this registry variable:
(C) $ db2set DB2TCP_CLIENT_CONTIMEOUT = 60
•
WebSphere Commerce uses the DB2 CLI-based JDBC driver to communicate with the
DB2 database server. CLI has its own receive timeout configuration, which might
interfere with the DB2TCP_CLIENT_RCVTIMEOUT setting. To avoid this situation, add
an entry “ReceiveTimeout” to the DB2 CLI configuration file ~/sqllib/cfg
/db2cli.ini and set it to 60. By default, this value is 0, which means no timeout.
Notes:
•
If you set these variables too low, you cause unnecessary timeouts. Set these values based
on an estimate of the time the longest running query takes.
•
Using DB2TCP_CLIENT_CONTIMEOUT and DB2TCP_CLIENT_RCVTIMEOUT to
configure a DB2 database client to wait for a timeout period before identifying a primary
database server has failed might only result in a delay in how long the outage gets
detected at the client. This affects the timeliness of your failover mechanism.
•
Setting DB2TCP_CLIENT_CONTIMEOUT limits the number of retries to 1, just like
setting DB2TCP_CLIENT_RCVTIMEOUT.
•
Setting either DB2_CONNRETRIES_INTERVAL or
DB2_MAX_CLIENT_CONNRETRIES overrides the retry limit imposed by either
DB2TCP_CLIENT_CONTIMEOUT or DB2TCP_CLIENT_RCVTIMEOUT.
Installing Tivoli System Automation
The following steps give you a brief introduction on implementing the Tivoli System Automation
for multi-platforms policy based self-healing capability running on AIX:
15
1.
2.
3.
4.
5.
6.
7.
After you have purchased Tivoli System Automation v2.1, you can download a tar file for the
AIX operating system. The name for the archive for AIX platforms is C85W5ML.tar.
Download the installation archive into your local directory, and use the tar xvf command
to extract the archive. When you have extracted the files, you find the installation wizard in
the following directory:
SAM2100Base/installSAM
TSA is contained in several packages that must be installed on every node in the cluster to be
automated. TSA for Multiplatforms requires a certain RSCT level to be installed on that
system prior to the installation. RSCT is part of AIX, although not all of the RSCT related
filesets are installed by default within operating system. For TSA pre-installation checking, a
more recent level of RSCT may be required. In this case, for TSA v2.1 installation on AIX
v5.2, the required RSCT version is 2.3.7.1. After you install the base filesets, you can
download the specific updated filesets from IBM Support Fix Central.
Note: The latest level of some RSCT filesets are updated to 2.3.7.3 (such as rsct.core.utils),
which are also supported by AIX maintenance level (ML--5.2.0.0). It is recommended to
download the latest filesets.
Install the product including the automation adapter with the installSAM script. The
installation process is finished automatically.
Copy the automatic scripts shipped by DB2 v9 from the DB2 installation package to the local
directory in all of the cluster nodes. In this case, they are copied into
/software/Auto_Script.
Note: IBM has created scripts that enable TSA to work seamlessly with the DB2 database.
You can download the latest TSA automatic scripts from the DB2 for Linux site.
Change the environment variable. For example:
export CT_MANAGEMENT_SCOPE=2
export PATH =
$PATH:/usr/sbin/rsct/bin:/usr/opt/IBM/db2_08_01/instance/
:/software/Auto_Script/
Confirm that TSA has been installed successfully (Figure 6). If TSA has been installed, and
if the level of RSCT is correct, then you can start a TSA domain. Use following commands to
verify that TSA has been installed successfully and that the level of RSCT is correct:
1. lsrpdomain - should show the domain as Online with RSCT level of 2.3.7.1.
2. lsrpnode - should show all nodes in that domain with RSCT level 2.3.7.1.
3. lssrc –ls IBM.RecoveryRM - should show an IVN and AVN of 2.1.0.0.
16
Figure 6. TSA post installation check
Defining and administering a TSA cluster
Before configuring your TSA cluster, verify that all of the installations of TSA in your topology
know about one another and can communicate with one another in what is referred to as a TSA
cluster domain. Many of the TSA commands that you will use in the following steps require
Remote Shell (RSH) to be supported by your cluster nodes. RSH allows a user who is on one node
to run commands on another node in the cluster environment. To configure RSH, perform
following two steps:
1.
To configure RSH to allow the root user to issue remote commands on each nodes, add the
following lines to the file /root/.rhosts:
On (P),(S),(H):
psvt01 root
psvt03 root
psvt05 root
2. Verify that RSH is working by issuing the following commands. You will see the directories
listed:
(P)(S)(H) # rsh psvt01 ls
(P)(S)(H) # rsh psvt03 ls
(P)(S)(H) # rsh psvt05 ls
This is essential for managing HADR by TSA. The following scenarios show how you can create
a cluster and add nodes to the cluster, and how you can check the status of the Tivoli System
Automation daemon (IBM.RecoveryRM).
Configuration steps:
1.
Run the following command as root in each host to prepare the proper security environment
between the TSA nodes, so it is allowed to communicate between the cluster nodes:
(P)(S)(H) # preprpnode psvt01 psvt03 psvt05
2. Issue the following command to create the cluster domain:
17
(P) # mkrpdomain HADR_Commerce psvt01 psvt03 psvt05
3.
Now issue command to bring the cluster online. Note: all future TSA commands will run
relative to this active domain.
(P) # startrpdomain HADR_Commerce
4.
In seconds, the cluster starts , and you can look up the status of HADR_Commerce,
(P) # lsrpdomain
You will get an output similar to Figure 7:
Figure 7. TSA domain status
5.
Ensure all nodes are online in the domain as follows:
(P) # lsrpnode
You will get an output similar to Figure 8:
Figure 8. TSA Cluster nodes status
6.
On each online node in the cluster, an IBM Tivoli System Automation daemon
(IBM.RecoveryRM) is running. You can check the status and process id of the daemon with
the lssrc command:
lssrc –s IBM.RecoveryRM
You will get an output similar to Figure 9:
Figure 9. TSA recovery daemon status
Configuring and registering instance and HADR with TSA for
automatic management
As the base for automation, the components involved must first be described in a set of RSCT
defined resources. Due to diverse characteristics of resources, there are various RSCT resource
classes to accommodate the differences. In a TSA cluster, a resource is any piece of hardware or
software that has been defined to IBM Resource Monitoring and Control (RMC). So in this case,
the DB2 database instance and the HADR pair of database are both resources in the cluster, which
are configured and registered with TSA for automation management. As explained above, every
application needs to be defined as a resource to be managed and automated with TSA. Application
resources are usually defined in the generic resource class IBM.Application. In this resource class,
there are several attributes that define a resource, but at least three of them are application-
18
specific:
•
•
•
StartCommand
StopCommand
MonitorCommand
These commands may be scripts or binary executables. You must ensure that the scripts are well
tested, and produce the desired effects within a reasonable period of time. This is necessary
because these commands are the only interface between TSA and the application.
The automatic package shipped with DB2 v9 includes several scripts that can control the behavior
of the DB2 resources defined in TSA cluster environment. Here is a description of the scripts.
For the DB2 database instance:
1. regdb2salin: This script registers the DB2 instance into TSA cluster environment as a
resource.
2. db2_start.ksh, db2_stop.ksh, db2_monitor: These three scripts are registered
as part of TSA resource automation policy. TSA refers to this policy when monitoring DB2
database instances and when responding to predefined events, such as restarting a DB2
database instance when TSA detects that the DB2 database instance has terminated.
For the HADR database pair:
1. reghadrsalin: This script registers the DB2 HADR pair with the TSA environment.
2. hadr_start.ksh, hadr_stop.ksh, hadr_monitor.ksh: These three scripts
are registered as part of the TSA automation policy for TSA to monitor and control the
behavior of HADR database pair.
Notes:
z
A resource is any piece of hardware or software that has been defined to RMC.
z
A resource class is a set of resources of the same type. For example, while a resource might
be a particular file system or particular host machine, a resource class is a set of file systems,
or a set of host machines.
z
A resource attribute describes some characteristics of a resource.
Steps:
First, register DB2 instances with TSA for management:
1. Register the DB2 instances:
(P) # regdb2salin –a db2inst1 –r –l psvt01
(S) # regdb2salin –a db2inst1 –r –l psvt05
2. Verify that the resource groups (for example, db2_db2inst1_psvt01_0-rg and
db2_db2inst1_psvt01_0-rg) are registered and are online by issuing the following
command:
(P) # lsrg –g db2_db2inst1_psvt01_0-rg
You will get an output similar to Figure 10:
19
Figure 10. Status of instance resource group
Second, register HADR with TSA for management:
1. Start HADR on both primary and standby database.
2. As instance owner, ensure that the HADR pair is in “peer” state as follows:
(P)(S) % db2 get snapshot for db on rmall
3. As root on the primary node, create a resource group for the HADR pair as follows:
(P) # reghadrsalin –a db2inst1 –b db2inst1 –d rmall
4. Check the status of all TSA resource groups by issuing the following command:
(P) # lsrg –g db2hadr_rmall-rg
You will get an output similar to Figure 11:
Figure 11. Status of HADR resource group
Third, check the status of the whole cluster (Figure 12):
(H) # getstatus
20
Figure 12. Status of the whole cluster
Running WebSphere Commerce stress tests with HADR and
TSA
In this article, three types of stress tests are covered to evaluate the behavior of this integrated
environment. All of the tests are based on the same test configuration (browse/buy = 0.55/0.45,
concurrent users = 50, no think time), while with the same business model B2C (WebSphere
Commerce).
1. Run a normal stress test case without HADR, and get the result to setup the baseline.
2. Run a normal stress test case with DB2 HADR NEARSYNC mode, and then get the result to
assess the performance overhead, if any, due to HADR.
3. Run the test case again with a NEARSYNC mode in different failure scenarios (Abnormal
Termination) to compare the performance impact with baseline result.
Normal operations with HADR enabled
After the first successful connection to the primary database server, you can transfer the alternate
server location information (host name or IP address, and service name or port number) to the
client. When your environment is enabled for Automatic Client Reroute, the DB2 client
automatically retries the original database server location, and then an alternate database server
location that you specify on the DB2 database server itself.
Testing process failure: Standby Instance Failure
Steps:
1. On the standby server, as the standby database instance owner, issue the command to
simulate an DB2 instance failure:
(S) $ db2_kill
Note: Record the time you issued the command.
21
2. On the heartbeat node, issue the command “getstatus” repeatedly, until the instance comes
back.
Check the duration for standby instance failover in db2diag.log on the standby database server.
Testing physical failure: Standby system failure
Steps:
1. On the standby server, as the root user, issue the command to simulate a system failure in
standby system:
(S) # shutdown
Note: Record the time you issued the command.
After the original standby system restart, check the duration for the standby database server to
restart and rejoin as the standby database server again.
Testing process failure: Primary instance failure
Steps:
1. On the primary server, as the primary database instance owner, issue the command to
simulate a DB2 instance failure:
(P) $ db2_kill
Note: Record the time you issued the command.
Figure 13. Status of cluster after issuing db2_kill
2. On the heartbeat node, issue the command of “getstatus” repeatedly (Figure 13), until the
instance comes back.
3. Check the duration for instance restarting in db2diag.log on the primary database server.
TSA will detect the DB2 instance failure, and restart the instance according to script shipped in
DB2 v9.
Testing physical failure: Primary system failure
Steps:
1. On the primary server, as the root user, issue the command to simulate a system failure in
primary system:
22
(P) # shutdown
Note: Record the time you issued the command.
2. On the heartbeat node, issue the command “getstatus” repeatedly, until the status of standby
instance becomes “online” from “offline”:
(H) # getstatus
3. Check the duration when the standby database finishes taking over as a new primary server in
db2diag.log on the standby database server.
4. After the original primary system restart, check the duration for the original database
server
to restart and rejoin as a new standby database server, which means the HADR pair comes back.
Switching the roles of primary database and standby database:
Graceful takeover
To perform regular software and/or hardware maintenance on the primary node, it is required to
take the primary node down and reroute the traffic to the standby server. In HADR terms, this is
known as a “graceful” takeover. You can plan this planned activity during the setup whether TSA
is enabled or not.
In the environment where TSA is not enabled, you can issue the command “db2 takeover
hadr on db rmall” on the standby database server.
In the environment where TSA is enabled, such as in the current setup, it is highly recommended
to use the command supported by the TSA cluster infrastructure to achieve it as the HADR pair
and DB2 instance as resource groups (Figure 14):
(H) # rgreq –o move –n psvt05 db2hadr_rmall-rg
Figure 14. Status of cluster after initiating graceful takeover
23
Performance overhead of HADR
One of the key considerations before you plan to implement HADR in a WebSphere Commerce
production environment is performance impact. This section discusses some analysis and
methodology about how to evaluate the duration of a failover. Since these systems are not in a
controlled environment, the performance data can vary from time to time. Note that the results
given below are the average from multiple repeatable executions.
Impact of HADR with NEARSYNC mode
With HADR, you can specify one of three synchronization modes to choose your preferred level
of protection from potential loss of data. The synchronization mode indicates how log writing is
managed between the primary and standby databases. These modes apply only when the primary
and standby databases are in peer state.
1.
2.
3.
SYNC (synchronous)
NEARSYNC (near synchronous)
ASYNC (asynchronous)
This article only covers the NEARSYNC mode. While this mode has a shorter transaction
response time than synchronous mode, it also provides slightly less protection against transaction
loss. In this mode, log writes are considered successful only when the log records have been
written to the log files on the primary database, and when the primary database has received
acknowledgement from the standby system that the logs have also been written to main memory
on the standby system.
The first measurements about the performance impact are the throughput, number of errors, CPU
utilization on the database node comparing between HADR enabled and HADR disabled. Based
on the result listed in Table 1, the performance impact due to HADR NEARSYNC mode is fairly
minimal with no significant performance degradation.
Notes:
•
The throughput and overall response time are not absolute, but instead with a normalized
value using 1000 and 100 respectively for comparison.
•
Since automation test tool is used in this case to simulate the appropriate workload to the
database server, from the end users’ (test tool) perspective, the errors in this measurement
stand for the Receive/Send timeout or the mismatched return that cannot be parsed by the
test tool.
Table 1. Performance comparison table for baseline test
HADR setup
Throughput
Error Number
None (baseline test)
1000 /h
8
CPU
Utilization
63%
Overall
response time
100.0 time unit
24
NEARSYNC mode
Performance
degradation
979.4 /h
2.06%
12
+ 4 errors
63.5%
0.8%
102.3 time unit
2.3%
For more information about HADR synchronization modes, see IBM DB2 UDB Data Recovery
and High Availability Guide and Reference, V8.2.
Failure duration in different disaster situations
The second measurement for performance impact is the duration of unavailability in both
simulated system failure and DB2 instance failure.
If a system disaster happens in a primary database server, the overall time taken for failover can
be roughly broken down to the following phases (Figure 15):
1. After issuing the command “shutdown” in primary system (T0), it takes 5-10 seconds for
TSA to detect the disaster. The policy is defined in the TSA automation scripts. Then TSA
initiates a takeover process on the standby server (T1) by issuing the HADR command to the
standby server.
2. The standby server finishes taking over and becomes the new primary database server (T2).
3. The DB2 client detects communication failure with the primary server and initiates an ACR
process (T3). This phase could start before item 2 as shown in Figure 15. Note that this phase
depends on the timeout values setting described earlier.
4. ACR fails to reconnect to the original primary database server (T4).
5. ACR connects to standby database server successfully and returns the SQL error code
(30108) to Application Server. Then Application Server initiates a purge process to refresh
any stale connections and reestablishes the connections to the new primary database server.
Note that the duration for ACR to reroute successfully to the new primary database server
depends on how long the HADR takeover process takes, and when the new primary database
server is ready to handle transactions (T5).
25
Figure 15. Time line for primary system failure scenario
In the event that an instance failure happens on the primary database server, the overall time taken
for restart is composed of the following phases (Figure 16):
1. After issuing the command “db2_kill” in primary system (T0), it takes 5-10 seconds for TSA
to detect the disaster. The policy is defined in the TSA automation scripts. Then TSA tries to
restart the failed primary instance (T1).
2. The failed instance finishes restarting and recovery (T2).
3. The DB2 client detects communication outage with the database server and initiates an ACR
process (T3). This phase could start before item 2 as shown in Figure 16. Note that this phase
depends on the timeout values setting described earlier.
4. ACR fails to connect to the standby database server (T4).
5. ACR finishes reconnecting to the original primary database server successfully, and ACR
returns the SQL error code (30108) to Application Server. Application Server initiates a purge
process to refresh any stale connections and reestablishes the connections to the restored
database server. Note that the duration for ACR to reconnect successfully to the original
primary database server depends on how long the instance restart process takes, and when the
original primary database server is ready to handle transactions (T5).
26
Figure 16. Time line for primary instance failure scenario
The ACR is initiated when either a primary instance failure or a primary system failure happens. It
first attempts to detect an available primary database server and then tries a connection. No matter
the database server is the original one or a new one, the mechanism for ACR to handle the
abnormal termination is fairly similar. In this article, the same measurements are used to figure out
the overall duration for unavailability:
1.
2.
From the database’s perspective:
The duration of unavailability is the time between when the primary disaster happens and the
time when the database server comes back (no matter if it is the standby database server
takeover or the failed instance restart, T1—T2 in the picture).
From end users’ perspective:
The measurement is the average duration of unavailability between when the primary disaster
happens, the time when the Application Sever connection manager purges all the inactive
connections, and when the new connection is reestablished (T1 – T5 in Figure 16 can stand
for a typical duration).
27
The results are listed in Table 2 below:
Table 2. Result table for primary system/instance failures
Scenario
Failover time from database’s
perspective
Primary system failure
22.33 seconds
Primary instance failure
Failover time from end
users’ perspective
90.97 seconds
79.39 seconds
82.53 seconds
Notes:
•
In the scenario of a primary system failure, the standby takes over as the new primary
server. In a scenario of a primary instance failure, it is designed to restart the failed
instance instead of the standby taking over.
•
In these two scenarios shown above, from the database’s perspective, even the duration
for a takeover is quicker than to restart the instance. The same values (60 seconds) are
used to set the timeout settings, which is the limiting factor. This explains why the
unavailability from the end users’ perspective is very close to each other.
Now take a look at the results when the standby database fails. Since all the transactions are
handled by the primary server, and no real fail over occurs, the recovery logic is relatively simpler
than others:
z
z
After simulating the disaster in standby system, it takes 5-10 seconds for TSA to detect the
disaster. The policy is defined the TSA automation scripts. TSA detects a failure (no matter if
it’s a system failure or instance failure) on a standby server, and initiates a restore process on
standby server.
Duration for the standby node finishes restoring and rejoins as a peer standby server again.
The results are listed in Table 3 below. Note that the duration represents the window where HADR
is not available to protect the primary database server. Data loss can occur if the primary database
server fails during this period of time. As a result, the shorter the duration, the better it is.
Table 3. Result table for standby system/instance failure
Scenario
Duration for standby server comes back and rejoins as
standby node again
Standby system failure
47.552 seconds
Standby instance failure
24.664 seconds
The whole process is similar to the primary system failure and instance failure, which you can
split into the following steps (Figure 17):
1) TSA initiates a graceful takeover process on the standby server or the DBA manually starts
the process (T1).
2) The standby server finishes taking over and becomes the new primary server (T2).
3) The DB2 client detects communication outage with the database server and initiates the ACR
process (T3). This phase can start before item 2 as shown in Figure 17.
28
4)
5)
ACR fails to connect to the original primary server (T4).
ACR connects successfully to the standby server and returns the SQL error code (30108) to
Application Server. Then Application Sever initiates a purge process to refresh any stale
connections and reestablishes the connections to the new primary server. Note that the
duration for ACR to reroute to the new primary server depends on how long the HADR
takeover process will take and when the new primary server is ready to handle transactions
(T5).
Figure 17. Time line for a graceful takeover scenario
The results are listed in Table 4 below:
Table 4. Result table for a graceful takeover scenario
Scenario
Failover time from database’s
perspective
Graceful takeover
11.83 seconds
Failover time from end users’
perspective
21.16 seconds
Note:
In the scenario of a graceful takeover, the DB2 client tries to reconnect to the original primary
database server first as it is defined in the ACR default rule, but the primary database server can
return an error code to the DB2 client quickly without waiting for a connection or receive timeout
(60 seconds). This explains why the failover duration is much shorter than the “forced” scenarios.
29
Restrictions and recommendations
This section provides restrictions and recommendations.
Restrictions in this topology
Many products work together to form this complex topology, which leads to some limitations or
restrictions in this topology. In this case, the following are some key restrictions for HADR, ACR
and TSA:
•
The primary and standby databases must have the same operating system version and the
same DB2 UDB version. During rolling upgrades, the database version of the standby
database may be later than the primary database for a short time.
•
Reads operations on the standby server are not supported, which means clients cannot
connect to the standby database.
•
•
•
Log archiving is performed by the current primary database.
Normal backup operations are not supported on the standby database.
ACR is only supported when the communications protocol used for database is TCP/IP.
Recommendations
Following are recommendations to improve system performance or to ensure the whole topology
works more reliably and efficiently.
Appropriately set your synchronization modes for HADR
SYNC (synchronous) provides the greatest protection against transaction loss, while ASYNC
(asynchronous) has the shortest transaction response time among the three modes, and causes
potential data loss. For WebSphere Commerce, it is recommended not to use ASYNC mode due to
its potential data loss, which defeats the purpose for full disaster recovery.
NEARSYNC (near synchronous mode) seems to be a compromise between performance impact
and data protection. However, performance impact has many other factors that are not covered by
this article, such as network delay if the primary and secondary servers are far apart. Readers are
encouraged to evaluate their own environment and perform some loading testing to decide if
SYNC (synchronous) mode is acceptable. The general rule of thumb is to use the highest data
protection mode if acceptable.
Use the LOGINDEXBUILD database parameter
For HADR databases, set the LOGINDEXBUILD database configuration parameter to ON to
ensure that complete information is logged for index creation, recreation, and reorganization. The
indexes are rebuilt on the standby system during HADR log replay, and are available when a
30
failover takes place. This shortens the response time for the standby server to takeover.
Set appropriate BUFFPAGE size to eliminate enough database physical read
Since it is faster to fetch data directly from memory than from hard disk. DB2 uses buffer page to
cache temporary read and write in memory, which can significantly improve the performance of
the whole database system. Customization in the buffer pool is one of the most important aspects
in database optimization. In this case, enlarge the size of buffer pool size and cache most of the
read operation, which will improve the performance of database system.
Update C++ library version in AIX
For the C++ library, the xlC.rteshould be at version 7.0.0.1 or higher. If you are below that level,
the software will also run and you can go on. However, the RSCT has the xlC.rte level at the prerequest level, so upgrade the package later if time permits to avoid any potential issues.
Keep the database logs in different devices
Again, it is highly recommended that you use the newlogpath configuration parameter to put
database logs on a separate device once the database is created. This prevents media failure from
destroying a database and improves the overall performance of database system.
Conclusion
By using HADR and TSA, WebSphere Commerce supports an efficient high available solution in
the tier of databases, and data protection with minimal performance impact to basic and normal
operations from the end users’ perspective. This article showed an entire solution on how to setup
the topology, as well as the measurement used and result analysis based on this solution. From this
article, you also learned how to apply the same approach in your own setup to achieve high
availability.
Resources
•
•
•
•
IBM DB2 UDB Data Recovery and High Availability Guide and Reference, V8.2
IBM DB2 Universal Database Administration Guide: Implementation, V8
IBM DB2 Universal Database Administration Guide: Performance, V8
IBM Tivoli System Automation for Multiplatforms Base Component User’s Guide, Version
2.1
•
•
IBM Cluster Information Center
WebSphere Commerce Information Center
31
About the authors
Xiao Qing (Shawn) Wang is still a freshman in IBM. He joined IBM China Lab about a year ago,
and now works in the WepSphere Commerce System Verification Test team. His areas of interest
include automation test, test tools, and unified process investigation. In his spare time, he enjoys
sports, collects stamps, and travels. You can reach him at [email protected].
Ramiah Tin is the Business Continuity Architect for WebSphere Commerce in the IBM Software
Group at the IBM Toronto Lab. He has a Master of Engineering degree from the University of
Toronto, Canada. You can reach him at [email protected].
32