Download IBM Content Manager V8.4.2

IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
IBM Content Manager V8.4.2
High Availability configuration using
IBM DB2 V9.5
with IBM Tivoli System Automation
and
IBM WebSphere Application Server V6.1 with
IBM WebSphere Application Server Network Deployment
on
IBM AIX V5.3
Bin Kong
Lead HA/DR Tester, System Verification Test
IBM China Development Lab, Beijing, China
Steve Soria
Lead Architect for IBM Content Manager HA/DR solutions
IBM Silicon Valley Lab, San Jose, California, USA
Copyright IBM (2004, 2009). This information contained within this document is the property of IBM.
The above notice and this paragraph must be included on all copies of this document that are made.
Copyright © 2004, 2009 IBM
Page 1 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Table of Contents
Special Notices....................................................................................................................................5
About this Document ..........................................................................................................................6
Preface ................................................................................................................................................7
IBM’s Caveats Regarding Highly Available Environments .......................................................8
Audience and Prerequisite Skills ................................................................................................9
What We Hope You will Learn ..................................................................................................9
Comments, Critiques, and Criticism .........................................................................................10
Design Considerations for a Highly Available topology ..................................................................11
Planning an HA environment for Content Manager .................................................................14
Content Manager product prerequisites for HA environments .................................................17
Scenario Description.........................................................................................................................18
Two Nodes Scenario .................................................................................................................18
Hardware Setup.................................................................................................................................20
Servers ......................................................................................................................................20
Storage ......................................................................................................................................20
Internal Disk (SAS)...........................................................................................................20
Fiber Channel Attached Storage (IBM DS4200) ..............................................................20
Network Attached Storage (IBM N3700) .........................................................................22
Network.....................................................................................................................................23
Operating System Preparation ..........................................................................................................25
Virtual I/O Server .....................................................................................................................25
Base AIX Installation................................................................................................................25
Prerequisite Software ................................................................................................................25
Storage Configuration ...............................................................................................................26
Logical Volume and Mount Points ...........................................................................................27
File System Configuration on Node 1...............................................................................27
File System Configuration on Node 2...............................................................................28
Preparation for DB2 ..........................................................................................................29
Users and Groups......................................................................................................................29
NFS Configuration....................................................................................................................30
DB2 Database with Tivoli System Automation................................................................................32
Pre-Install Validation ................................................................................................................32
Software Installation .................................................................................................................32
License Validation ....................................................................................................................40
Setup Instance: db2inst1 ...........................................................................................................40
Setup Instance: db2inst2 ...........................................................................................................46
DB2 High Availability Instance Configuration Utility.....................................................................49
Cluster Preparation....................................................................................................................49
Time Synchronization...............................................................................................................49
The db2haicu Interactive Setup on Node1 ................................................................................50
Creating a Cluster Domain................................................................................................50
Copyright © 2004, 2009 IBM
Page 2 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Quorum Configuration......................................................................................................51
Network Setup ..................................................................................................................51
Cluster Manager Selection ................................................................................................53
Failover Policy ..................................................................................................................54
Virtual IP Address Setup...................................................................................................55
The db2haicu Interactive Setup on Node2 ................................................................................55
Cluster Manager Selection ................................................................................................56
Failover Policy ..................................................................................................................56
Virtual IP Address Setup...................................................................................................57
DB2 and Tivoli System Automation environment acceptance testing .....................................58
Verify with db2pd Utility..................................................................................................59
Verify environment during runtime ..................................................................................62
Verify with db2haicu utility in Maintenance Mode..........................................................63
Verifying the Tivoli System Automation environment ....................................................64
IBM WebSphere Application Server ................................................................................................65
Install WebSphere Application Server Network Deployment ..................................................65
IBM HTTP Server and IBM HTTP Server Plug-in Installation ...............................................67
Applying Fixpacks ....................................................................................................................70
WebSphere Application Server Profile Management Tool ......................................................71
Create WebSphere Application Server Cluster.........................................................................75
Create WebSphere Application Server JMS data source..........................................................80
Add Web Server Definition ......................................................................................................87
SSL configuration for http server (Optional) ............................................................................90
IP Sprayer Definition ................................................................................................................91
WebSphere Application Server and WebSphere Application Server Network Deployment
environment acceptance testing ................................................................................................91
Content Manager...............................................................................................................................93
Pre-Install Validation ................................................................................................................93
Installation.................................................................................................................................93
Initial Configuration for Library Server Database ....................................................................94
Initial Configuration for Resource Manager ...........................................................................105
WebSphere Data source pretest configuration........................................................................115
Client Reroute .........................................................................................................................116
Failover Time Tuning .............................................................................................................117
Validation and Test .........................................................................................................................118
Validating Active Components...............................................................................................118
Node Application and Database (Library Server, Resource Manager Database) ...........118
Application Server (Resource Manager).........................................................................119
HTTP Server (Resource Manager) .................................................................................120
Content Manager Component Failures ...................................................................................121
Database (Library Server, Resource Manager Database) ...............................................121
Application Server (Resource Manager).........................................................................121
HTTP Server (Resource Manager) .................................................................................122
Appendix.........................................................................................................................................123
List of Tables ..........................................................................................................................123
List of Figures .........................................................................................................................123
Copyright © 2004, 2009 IBM
Page 3 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
Copyright © 2004, 2009 IBM
2009.12.28
Page 4 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Special Notices
The information contained in this publication was derived under specific operating and environmental
conditions and is distributed on an "as is" basis without any warranty either expressed or implied. IBM
specifically disclaims any implied warranties of merchantability or fitness for any particular purpose.
IBM assumes no responsibility for any errors that may appear in this document. The use of this
information or the implementation of any of these techniques are the responsibility of the user and
depends on the user's ability to evaluate and integrate them into the user's operational environment. While
each item might have been reviewed by IBM for accuracy in a specific situation, there is no guarantee
that the same or similar results will be obtained elsewhere. Users attempting to adapt these techniques to
their own environments do so at their own risk.
The information contained in this document is subject to change without any notice. IBM reserves the
right to make any such changes without obligation to notify any person of such revision or changes. IBM
makes no commitment to keep the information contained herein up to date.
References in this publication to IBM products, programs, or services do not imply that IBM intends to
make these available in all countries in which IBM operates. Any reference to an IBM licensed program
in this publication is not intended to state or imply that only IBM's program may be used. Any
functionally equivalent program may be used instead. Equivalent product, program, or services that do
not infringe any of the intellectual property rights of IBM may be used instead of the IBM product,
program, or service.
The evaluation and verification of operation in conjunction with other products, except those expressly
designated by IBM, are the responsibility of the user.
IBM may have patents or pending patent application covering the subject matter in this document. The
furnishing of this document does not give you license to these patents. Any information about non-IBM
("vendor") products in this document has been supplied by the vendor and IBM assumes no responsibility
for its accuracy or completeness.
IBM, the IBM logo, ibm.com, AIX, DB2, Tivoli, and WebSphere are trademarks of International
Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names
might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the
Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml.
Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United
States, other countries, or both.
Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or
both.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Other company, product, or service names may be trademarks or service marks of others.
© Copyright International Business Machines Corporation 2004, 2009.
Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA
ADP Schedule Contract with IBM Corp.
Copyright © 2004, 2009 IBM
Page 5 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
About this Document
This white paper provides hands-on guidance for how to install and set up an IBM® DB2® Content
Manager V8.4.2 system on IBM® AIX® using IBM® DB2® and IBM® Tivoli® Storage Automation
and deploying an active-passive high availability environment. For the Content Manager resource
manager (“RM”) web application, the document describes how to configure a managed cell with two
nodes using IBM® WebSphere® Application Server Network Deployment and one cluster running two
RM application server instances (cluster members) using IBM® WebSphere® Application Server.
The topics covered in this white paper include the following:
In-depth installation and configuration recommendations for the Content Manager library server
(“LS”) and RM databases
In-depth installation and configuration recommendations for the RM application running in a
cluster with WebSphere Application Server , including how to use WebSphere Application
Server Network Deployment clustering capabilities.
Version
Description
Author(s)
Date
0.1
Initial Draft
Bin Kong
17.08.2009
0.2
First complete draft
Steven Soria
16.11.2009
0.3
Review revisions
Bin Kong
28.11.2009
0.4
Second complete draft
Steven Soria
03.12.2009
0.5
Final Draft
Bin Kong
Steven Soria
28.12.2009
Special thanks to Arya Adeli, Khalid Alodhaibi, Holger Koenig, Konstantin Konson, Cataldo Mega,
Stefan Momma, and Steve Raspudic for technical advice, comments, reviews etc. in many areas of this
document.
Copyright © 2004, 2009 IBM
Page 6 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Preface
This publication is intended to help content management solution designers understand how to enable
highly available content management solutions on officially supported platforms using the IBM® DB2®
Content Manager Version 8 Release 4 Modification Release 2 product components. It serves as a
reference example on how to set up the IBM® DB2® Content Manager stack in a highly available
configuration and outlines the relationships between the components in the stack.
Because Content Manager components are a hybrid of databases, database applications, Web applications,
and thick client applications, the reference configuration presented makes strong use of the
platform/operating system-specific cluster support and the high availability (“HA”) capabilities built into
the database and Web application server products. HA environments are designed to eliminate single
points of failure (“POF”) throughout the application stack. In this white paper, we accomplish this by
leveraging the HA capabilities and services in DB2, WebSphere Application Server, and the clustering
software support provided by the underlying operating systems.
Although the Content Manager high availability concepts discussed here may be applicable to many
different operating systems, the actual Content Manager high availability environment that is
implemented depends heavily on the individual hardware and software components used, thus making a
direct comparison of the different operational scenarios impossible. Implementations will vary and some
concepts or details of how we configured our environment may not apply to another specific environment
or configuration. We strongly encourage anyone who intends to set up a high availability configuration
with these or any other products to employ experienced, professionals who understand the technical
complexities inherent in such environments.
Setting up and maintaining any type of clustering environment is expensive, time consuming, complex,
and very prone to error. With that said, it should be obvious that it is imperative to thoroughly test the
overall setup. Setup of an HA environment requires a deep understanding of the various HA, storage, and
networking technologies involved in addition to familiarity of the underlying applications. Clustering
solutions cannot be considered an out-of-box solution. In fact, building and maintaining clustering
solutions requires a range of skills, tasks, resources, and extensive testing in order to failover resources
properly to alternate nodes. We strongly encourage anyone who intends to set up a high availability
configuration with these or any other products to employ experienced professionals who understand the
technical complexities inherent in such environments. Malfunctions of the HA environment can often be
attributed to insufficient testing of the various resources that might become unavailable at some point in
time.
We recognize that there are many possible different configuration variants and that setting up such an
environment correctly is complex. For existing Content Manager customers who wish to modify an
existing Content Manager architecture to take advantage of the architecture described in this document, or
for customers with requirements that are incompatible with the architecture described in this document,
such services are not available as part of IBM standard product support. It is, therefore, highly
recommended to use consulting services, such as IBM® Global Business Services®, for high availability
planning, testing, implementation, maintenance, and support for your Content Manager product. IBM
standard product support for HA configurations is strictly limited to the HA reference configuration and
Content Manager product version described in this document, as well as the method of installing and
configuring Content Manager on the HA reference configuration for the stated Content Manager product
version. Although beyond the scope of this document, IBM standard product support also includes the
application of maintenance releases (“fixpacks”) to the HA reference configuration and Content Manager
product version described herein. The remainder of this document assumes you will be performing a clean
Copyright © 2004, 2009 IBM
Page 7 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
installation of Content Manager, in an environment that has no existing Content Manager installations or
databases, whether partially installed, installed but not running, or running.
Please be aware that IBM standard support does not include troubleshooting or advising on physical
storage, network topologies, or other non-IBM high availability infrastructure components. For problems
with IBM-branded products used in your highly available infrastructure which are not part of Content
Manager, you must contact IBM support specific to those IBM products.
IBM’s Caveats Regarding Highly Available Environments
Clustering should be left to experts.
This document is not intended to be a generic high availability support statement of system-specific
cluster services. Content Manager is an enterprise-class application that customers might want to install
and set up in a high availability configuration. Installing this kind of application in a clustered
environment is not an out-of-the-box solution.
Customers have the choice of installing Content Manager in a great number of configurations and, as
such, IBM has not tested all potential configurations on all of the various hardware platforms given the
considerable number of possible configurations of Content Manager in a high availability environment.
IBM has published a number of white papers as reference material for those interested in high availability
configurations. Each of these white papers targeted a different operating system and environment, with
the main objective of each to describe a tested high availability implementation on these operating
systems with detailed configuration procedures. These configurations might require a separate agreement
with Global Business Services (or another IBM certified vendor certified to provide high availability
solutions in a Content Manager context) for support.
In general, proper design, installation, configuration, fail-over testing of Content Manager, and support
for running Content Manager in an HA environment needs to be considered as a complex endeavor
potentially prone to errors. Therefore, again, IBM advises the end-user and organization implementing an
HA system to contract with an authorized consulting organization certified to provide support of high
availability environments. IBM standard product support is strictly limited to the reference configuration
and Content Manager product version described in this document, as well as the method of installing and
configuring the reference configuration for the stated Content Manager product version. Although beyond
the scope of this document, IBM standard product support also includes the application of maintenance
releases (“fixpacks”) to the reference configuration and Content Manager product version described
herein. For existing Content Manager customers who wish to modify an existing Content Manager
architecture to take advantage of the architecture described in this document, or for customers with
requirements that are incompatible with the architecture described in this document, such services are not
available as part of IBM standard product support. It is, therefore, highly recommended to use
consulting services, such as Global Business Services, for high availability planning, testing,
implementation, maintenance, and support for your Content Manager product.
Setting up any type of clustering environment is expensive, time consuming, complex, and very prone to
error. With that said, it is imperative to thoroughly test the overall setup. We consider it ill advised for any
entity to set up a clustering environment unless they already have a competent staff solely dedicated to
testing and maintaining such an environment. HA environments require considerable investment in
additional hardware, skilled people, and time to create a proper test environment that is separate from a
production environment.
Our experience has been that many entities attempt to set up a clustered environment, but fail to budget
correctly for such projects. In other words, these companies try to implement HA environments without
allocating a reasonable amount of resources.
Copyright © 2004, 2009 IBM
Page 8 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
We believe that, among the keys to success, are:
Allocating appropriate amounts of resources when implementing a clustered environment can
make for a more successful outcome.
Working closely with an HA qualified vendor.
The most common sources of failure are:
Not testing the various potential scenarios before going into production: Production is the wrong
time to find out your cluster does not work as expected.
Failure to properly budget sufficient funds for an HA project, both capital & expense.
Poor scheduling and rushed delivery time frame that prevents proper testing.
Poor Change Control procedures within the organization.
Failure of having skilled personnel competent with clustering technologies
Not having a Test Lab.
As a reminder, the information in this white paper is intended to complement, not to substitute the
Content Manager product documentation provided by IBM. See the publications section of the IBM
Programming Announcement for Content Manager for more information about what publications are
considered to be product documentation.
Audience and Prerequisite Skills
Throughout this paper, it is presumed that the reader has a good understanding of the business
requirements for high availability in an Content Manager environment. If available, operating systemspecific knowledge of all software components used for this solution will also help to better understand
the technical details described.
Given the complexity of the Content Manager product itself and the many prerequisite applications, it is
almost impossible to document all variations of possible Content Manager high availability scenarios.
Therefore, while we will refer at times to alternative configurations or choices we could have made in
setting up our HA environment, we will primarily focus on details of the specific HA reference
configuration we set up. We will also not try to fully describe how to configure the related and required
co-products, but rather focus on details relevant to the configuration of the chosen Content Manager high
availability environment.
What We Hope You will Learn
After you read this paper, we hope that you, through the example of how we set up our environment,
understand:
the basics of a design approach for implementing high availability using Content Manager and its
co-products on AIX and DB2 in the specific environment we chose
components in production
how high availability is achieved using the Content Manager product components
general concepts for planning for an HA environment
general concepts for installing an HA environment
Copyright © 2004, 2009 IBM
Page 9 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Comments, Critiques, and Criticism
Reader comments and critiques regarding this report are welcome. Please send your thoughts, comments,
critique, or other feedback to [email protected].
Copyright © 2004, 2009 IBM
Page 10 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Design Considerations for a Highly Available topology
At the IBM Information On Demand 2009 conference, statistics were presented highlighting the
tremendous explosion of content: 15 petabytes of data generated per day, 200 billion emails per day, and
80% of that from unstructured content. Significantly, the leading cause of this data explosion is no longer
due to human-generated content, but rather machine-generated content. This has naturally led to huge
investments in content management systems, transforming ever-increasing volumes of content into
genuine assets. With the increasing need of businesses to leverage this information efficiently and
effectively, one of the key challenges of an information-led transformation has been to implement costeffective measures that ensure this new asset, this content, remains available.
But what does it mean to be “highly available”, and what does it mean in the context of an enterprise
application such as Content Manager? The answers to these questions are highly dependent on the goals
of the organization requiring a highly-available solution. Companies must have a clear understanding of
what they expect to achieve from such configurations before embarking upon the implementation of a
solution. For example, it may be acceptable for some to limit the definition of availability to end-users
simply being able to access mission-critical systems, that the servers are “up” and responding to requests,
even if the responses generate errors. For others, availability may additionally require that end-users be
able to retrieve business data from mission-critical systems, even if the data is not necessarily the most
recent version of that data. Or, perhaps availability means that end-users can not only access systems and
retrieve data, but that that data must be accurate, current, and reliable. Availability may also mean that
recovery is transparent to the end-user, perceived as “continuous availability”, or that the end-user may
experience some down time but that recovery capability does exist, with the full capabilities of the system
and its data restorable. The graph below illustrates the tradeoffs between cost to implement versus time to
recover, representing the sophistication of the solution with respect to the evaluated criticality of business
data and applications. The benefits of a highly available solution must be weighed against the expense,
time to implementation, and personnel expertise in designing the solution. Certainly, these are not trivial
considerations.
Copyright © 2004, 2009 IBM
Page 11 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Broadly, there are three approaches toward solving the availability problem. It is common to combine
multiple methods to achieve higher levels of availability.
Coordinated backup and recovery
The general objective of a coordinated backup and recovery strategy is to ensure the system can
holistically be restored such that all normal operations are resumed with no data loss, a state of
consistency, within a recovery time that fulfills established mean-time-to-recovery (“MTTR”) objectives,
and to a point-in-time closest to the time of failure. Complete backups must be taken for all file systems
relevant to the applications and data that are mission-critical. If databases or offline storage, such as
IBM® Tivoli® Storage Manager or shared storage, are utilized, they too will require a backup strategy.
There are often many strategies for backup policies, with considerations including the use of operating
system, vendor-provided, or third-party backup tools, the frequency and scheduling of backups that seek
to balance end-user usage patterns with MTTR requirements, and the appropriate ratio of incremental to
full backup versus storage costs required to archive the backups and compliance requirements dictating
how long the backups must be kept. In cases where mission-critical business data resides on multiple
machines and/or storage devices, coordination of backups is required to ensure consistency and reliability
of systems and data. Duplicates of the backups may be stored in a different geography to ensure their
survival in the event of some disaster at the primary site. A backup and recovery strategy is generally
considered the “poor man’s high availability” due to its manual-intensive nature and lack of flexibility.
This means it is likely to take a day or longer to fully recover from an outage. However, it provides low
total-cost-of-ownership (TCO) because it can generally be implemented using existing hardware,
operating system tools, and simple scripting.
Disaster Recovery (“DR”)
This availability strategy is intended to protect against failure at a site-wide or data center level. The idea
is that a secondary site is standing by, ready to take over from the primary site in the event a natural or
catastrophic disaster, such as an earthquake, fire, or flood, renders the primary inaccessible. Because such
a disaster is assumed to occur over a large area, primary and standby sites are typically geographically
disparate from one another. The secondary site is typically an exact duplicate of the primary, even in
terms of hardware, number of servers, hardware capability, storage capacity, and so on. Storage-level
Copyright © 2004, 2009 IBM
Page 12 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
replication, cluster server replication, database replication, or other technologies provide data duplication
at the secondary site. To ensure the secondary site is as closely matched as possible to the primary,
hardware mirroring or software-based synchronous replication is generally preferred such that a write is
not complete until the data is written to both the primary and secondary site. The drawback, of course, is
that this extends response times due to the synchronicity of every operation. Thus, MTTR in a DR
scenario often has an inverse relationship to performance. Observe also that DR focuses on mirroring or
data duplication from one site to another; it does not generally deal with processes. In classical DR, when
a failure occurs at the primary site, the secondary site must be manually transitioned and an integrity/data
verification process completed. Some cluster service providers can perform the failover automatically via
a heartbeat server that is remote from both the primary and standby site. Even in this scenario, however,
there is often a manual verification process since the data on the standby site is not guaranteed to be insync with the primary. An individual business must assess at what point it is appropriate to restore
services to end-users. Recovery time for a DR strategy is typically in terms of hours and is considered a
high availability solution. Although typically an inter-site recovery strategy, it is common for DR
strategies to be implemented within a single site as well, and thus it becomes more of a high availability
solution than a true DR solution.
High Availability (“HA”)
Whereas DR focuses on site-wide disaster, an HA strategy deals with the reality that machines fail, even
within a single site or data center. It deals with two primary requirements: that hardware failures are
recovered in terms of minutes or even seconds, not hours or days, and that recovery is automatic and
nearly transparent to end-users. In an HA configuration, multiple servers are ready to takeover in case
another fails in what is known as a server cluster. Depending on the underlying technology, cluster
members can be all active (“active-active” configurations), or one member can be active while all others
are passive (“active-passive” configurations). Cluster members are often referred to as nodes and are
monitored via heartbeat mechanisms between each node in the cluster. Although the exact logic varies,
the general idea is that when a heartbeat is “missed”, that node is marked as unavailable and incoming
requests are automatically re-routed to other available nodes. This is accomplished through floater IP
addresses (“virtual IP’s”) than the clustering software automatically remaps to the physical IP’s of other
nodes. Additionally, HA configurations make use of shared storage, with identical mount points on each
node. Applications are installed at identical locations on each node along with operating system user
equivalence. Mission-critical data is placed on the shared storage device to allow all nodes to access the
same copy of data. This allows for very fast failover time because everything is “ready to go” on each
node: there is no data integrity issue between nodes because it’s on shared storage and there is no need to
reconfigure applications because common mount points, virtual IP’s, and operating system user
equivalence make this transparent. Typically, in-flight operations (those transactions which were
executing during a node failure) return an error and need to be retried. Depending on the underlying
technology and if all nodes are active, a load balancing can strategy can be put in place to improve
scalability and hardware utilization. Observe that cluster nodes can be physically separate machines,
logical partitions (“LPAR’s”), or virtualized servers.
Combining strategies
As stated, it is very common to combine multiple strategies. “Cluster farms” combine HA and DR
technologies, benefiting from the transparency of a single machine failure, while also providing a solution
for data center or site-wide failure. Shops that use an HA+DR strategy may sometimes choose not to
implement a backup strategy since multiple DR sites generally suffice. However, others consider backups
to be a fail-safe method in the event all other strategies fail. All of these strategies can also be employed
as the underlying infrastructure of a cloud computing environment.
Copyright © 2004, 2009 IBM
Page 13 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Planning an HA environment for Content Manager
For this white paper, we have chosen to focus on installing and configuring Content Manager in an HA
environment. In doing so, we are prioritizing the need for content stored in a Content Manager repository
to be available to end-users, as part of a critical business process that utilizes content as an information
asset. The most common, and therefore disruptive, source of downtime is at the machine-level. Various
types of failure at the machine-level are common, whether it be from hard drive failure, network
controller failure, or motherboard and CPU failure. The need to guard against such common single points
of failure (“SPOF”) and maintain availability warrants the investment and skills development in an HA
solution. While DR is a viable alternative to HA, the costs involved in duplication of both server and
storage components better serve a site-based recovery strategy as opposed to a single-server recovery
strategy. Benefitting from the use of shared storage and reduced failover times allows the HA
configuration implemented here to result in a robust Content Manager server environment, minimizing
the likelihood that any SPOF prevents critical access to content.
Content Manager is an enterprise-class product which utilizes two or more Content Manager server
components, one or more database servers, one or more WebSphere servers, and potentially many Cbased or Web-based client applications. Providing sufficient redundancies to mitigate the impact of
SPOF’s is not a trivial task and requires significant up-front planning, design, and testing to ensure
success. Therefore, you should consider this white paper a guide that describes one possible HA solution.
Ultimately, the HA capabilities of Content Manager rest primarily on the HA infrastructure itself. The
intent of this white paper is not to impose a single HA solution, but merely to present one that is fullytested and supported, and represents what we believe to be a ‘best practices’ HA architecture for Content
Manager. Remember that only the HA solution described in this document, with the specific method of
installation and configuration performed, is supported through IBM standard support for HA
configurations. Otherwise, IBM standard support covers only runtime behavior that is re-producible in a
non-HA environment.
Although Content Manager client applications are the end-user interface to the Content Manager servers,
clients typically do not contain mission-critical business data or server applications. Clients are typically
deployed to many workstations, thereby implicitly redundant, or distributed via web application servers,
which generally also have a method of providing high availability. For client-side configuration or other
files, a simple backup policy is generally sufficient. On the other hand, without Content Manager server
availability, there can be no access to content and client applications are useless. Thus, this white paper
does not address client-side applications but rather focuses on the mission-critical pieces, the Content
Manager servers. This white paper will also not address technologies external to Content Manager servers,
particularly with respect to the content stores themselves. Because there are many options for content
storage, such as local file system or Tivoli Storage Manager, Network Attached Storage (NAS) or IBM®
General Parallel File System® (GPFS), Tivoli Storage Manager replication or Direct Access Storage
Device (DASD) mirroring, customers are advised to consult the documentation or manufacturer of their
respective storage devices to assess appropriate availability solutions. In this white paper, we make no
assumption as to the location or availability infrastructure of content, so long as it is available and
servable by the Content Manager resource manager.
Copyright © 2004, 2009 IBM
Page 14 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
In the diagram above, we identify the major Content Manager components and illustrate various potential
SPOF’s. As stated, this white paper will not address Content Manager client applications although typical
strategies are shown in the box at far left: clustering technology, mirroring or replication, virtualization,
backup and recovery, etc. The box at far right shows the Content Manager resource manager application
server. Content Manager has long supported WebSphere Application Server Network Deployment and
this continues to be the strategy recommended in this white paper as well for the Content Manager
resource manager web application. In this white paper, you will configure the Content Manager resource
manager web application in a WebSphere Application Server Network Deployment environment with
redundant IBM HTTP Servers. Although beyond the scope of this white paper, customers may choose to
implement a similar strategy for their web-based Content Manager client applications. Also beyond the
scope of this document, the Tivoli Storage Manager cluster shown above is optional but can be used to
provide content replication and Tivoli Storage Manager server redundancy in the event of failure of a
single Tivoli Storage Manager server. In the center are the Content Manager library server and Content
Manager resource manager database servers. For the databases, you will configure DB2 and Tivoli
System Automation to provide a highly available, mutual takeover scenario.
In choosing to use DB2 and Tivoli System Automation for database failover, our view is that the database
server is to the Content Manager server what the Content Manager server is to client applications. In other
words, just as client applications are useless if the Content Manager servers are not up and running, so too
are the Content Manager servers useless if the database server is not up and running. Thus, at the most
basic level, the database is the critical availability component for a Content Manager server environment.
While it is true that cluster server providers can be configured to monitor a process, they cannot monitor
the integrity of the database itself. Some cluster servers monitor only the operating system, storage
subsystems, and network communication, which won’t account for dead database server processes at all.
The results can be imprecise and unreliable. Database vendor-provided HA technology, therefore,
provides the lowest level of granularity for determining whether a database and all its intrinsic
components represent an available state. For this reason, we have chosen to utilize DB2 and Tivoli
Copyright © 2004, 2009 IBM
Page 15 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
System Automation. Although outside the scope of this document, another advantage of Tivoli System
Automation is that it can also be used to manage user-defined resources, with separate failover semantics.
Refer to Tivoli System Automation documentation for details.
To implement a solution that provides availability for the Content Manager library server database,
Content Manager resource manager database, and Content Manager resource manager application, known
collectively as the Content Manager server components, while also minimizing total-cost-of-ownership,
this white paper uses software technologies available in the pre-requisite products already required by
Content Manager today. Specifically, we will use Tivoli System Automation, shipped as part of DB2, to
provide DB2 database instance failover. We’ll use Storage Area Network (SAN) storage with Fibre
Channel as the shared storage for database files and NAS storage as the shared storage for Content
Manager resource manager Local Area Network-Based Object Server (LBOS) data, or content. Lastly,
we’ll use WebSphere Application Server Network Deployment to manage a cluster of WebSphere
Application Server servers for Content Manager resource manager application availability. We’ll use
multiple HTTP servers and an IBM® WebSphere® Edge Dispatcher to provide IP spraying and load
balancing. Although beyond the scope of this white paper, we recommend that the SAN and NAS each
employ RAID mirroring and controller redundancy, as well as multiple Fibre Channel adapters in each
database server node, to eliminate them as SPOF’s.
The diagram above focuses and illustrates the basic premise of our database HA configuration. Since DB2
and Tivoli System Automation implement an active-passive configuration for HA configurations, we’ll
employ what is known as a mutual-takeover scenario. In this scenario, the Content Manager library server
database will be active on node 1 (blue nodes) and passive on node 2 (purple nodes), while the Content
Manager resource manager database will be active on node 2 and passive on node 1. The diagram does
not show redundancies for the Fibre Channel or Ethernet controllers, but we will implement redundancy
for the Ethernet controller. Notice that Content Manager library server runtime binaries will be copied to
each member in the cluster. While it is possible to place the LS binaries on shared storage, we recommend
against that for optimal performance and redundancy. Also notice that the LS and RM databases are each
in their own DB2 instance. This is due to the fact that Tivoli System Automation monitors DB2 databases
Copyright © 2004, 2009 IBM
Page 16 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
at the instance level. Thus, in order to implement the mutual takeover scenario described in this document,
we need to place them in separate DB2 instances. Note that it is not required to implement the mutual
takeover scenario. If your Content Manager library server and Content Manager resource manager
databases will be on separate DB2 server clusters (or if you have multiple RM’s), then you will have four
database server boxes (or LPAR’s), two of which will be idle since they will be passive. Such
configurations are beyond the scope of this document to describe but are only trivial variations of what’s
described in this document. In such configurations, there is no need to create a second DB2 instance; you
would simply create the RM database in the default db2inst1 DB2 instance and configure the RM
database for HA as described later in this document for the LS.
As Tivoli System Automation managed resources, Tivoli System Automation will poll the IBM®
Reliable Scalable Cluster Technology (RSCT) subcomponent of the AIX operating system, which
performs heartbeats with other cluster members based on a communication group of Ethernet controllers.
There are various configuration strategies for managing the frequency of the heartbeat and Tivoli System
Automation polling to balance between incoming user requests versus failed-node detection and splitbrain strategies. We recommend settings in this document based on minimization of failover times. You
will need to test and evaluate what is appropriate for your environment. Refer to AIX, DB2, and Tivoli
System Automation documentation for more details on configuration parameters for these subsystems.
When a heartbeat is missed, Tivoli System Automation will failover to the passive node, mount file
systems, perform file system checks, remap the virtual IP to the actual physical IP of the newly active
node, and start the DB2 instance. Note that failover time is directly impacted by the speed of these
operations and availability of subcomponents. Tivoli System Automation fails over the DB2 instance but
not other, custom processes, such as the Content Manager library server monitor for DB2. This C-based
process runs asynchronously with the Content Manager library server and can only be active on one node
at a time. This means that you must manually start the Content Manager library server monitor after a
failover has occurred. Multiple, active Content Manager library server monitor behavior is unknown and
will fail since there is no active DB2 instance on the passive node for the monitor to connect to. Similarly,
IBM® DB2® Net Search Extender also runs asynchronously to the Content Manager library server and
DB2 instance and must be addressed separately. Please see DB2 documentation for more details.
Content Manager product prerequisites for HA environments
Note that this document assumes that your environment meets all normal Content Manager product prerequisite hardware, software, version, and patch-level requirements as stated in Content Manager v8.4.2
product documentation. In addition, the HA environment described in this document, using DB2 and
Tivoli System Automation to failover Content Manager server databases, requires DB2 v9.5 fixpack 5 or
later. Non-HA Content Manager installations do not need to upgrade to this level.
Also, be aware that the DB2 limited-use license provided as part of the Content Manager product does not
include a license for Tivoli System Automation. As such, you must either upgrade your DB2 license to a
full DB2 license, or purchase a Tivoli System Automation license. Please contact your IBM sales
representative for details.
Unless otherwise stated, all other Content Manager hardware and software pre-requisites are the same as
those listed in Content Manager v8.4.2 product documentation.
Copyright © 2004, 2009 IBM
Page 17 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Scenario Description
Content Manager is an enterprise class application, which consists of several specialized components. The
core components of every Content Manager implementation are the library server and one or many
resource managers.
The library server itself is mostly a database application, all its logic is implemented within the database
(currently as stored procedures) and its data is stored within database tables and views. The library server
implements functions such as authentication and authorization, data modeling, document routing, search
and the storage of descriptive information for each object (so called meta data). The library server can
also maintain relationships between different objects and can provide referential integrity.
The resource manager is responsible for storing the actual content (files). Currently the resource manager
logic is implemented as a web application, running in a web application server. The content is typically
stored to local disk, but can be stored also to storage devices managed by IBM Tivoli Storage Manager or
as blobs in the database. The resource manager also maintains internal information, which is again stored
in a relational database.
Two Nodes Scenario
We will setup two Active/Standby DB2 instances. For db2inst1, it will be active on Node 1 (cmaix25).
Once Node 1 meet with malfunction, Node 2 (cmaix26) will get all resources for db2inst1 and start it up.
Also, a float IP (9.123.112.251) is used with db2inst1, so that client can just use this IP to connect to any
database on db2inst1. Same strategy is used by db2inst2, although it will be active on Node 2 (cmaix26),
and cmaix25 is a standby node for this instance.
For example, LS DB is running on db2inst1 on cmaix25, and other machines can connect to DB via IP
9.123.112.251. Also, RM DB is running on db2inst2 on cmaix26 with IP 9.123.112.252. When cmaix25
meet hardware/software error, cmaix26 will take db2inst1 and IP 9.123.112.251. This means cmaix26
have two IPs and two instances running on it when cmaix25 is outage.
Copyright © 2004, 2009 IBM
Page 18 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
CM Clients
cmaix25.cn.ibm.com
(Active LS DB)
AIX 5.3 ML9
cmaix26.cn.ibm.com
(Active RM DB)
Public Network
9.123.112.199
DB2 UDB 9.5
with TSA
9.123.112.200
INSTANCE
db2inst1 (Standby )
ICMNLSDB Database
(Virtual IP: 9.123 .112 .251 )
DATABASE
icmnlsdb (Active)
INSTANCE
db2inst2 (Standby )
DATABASE
icmnlsdb (Standby )
INSTANCE
db2inst2 (Active)
RMDB Database
(Virtual IP: 9.123 .112 .252 )
DATABASE
rmdb (Standby )
DATABASE
rmdb (Active)
FC
FC
FC
FC
IBM WebSphere
Application Server 6.1
Network Deployment
DB2 UDB 9.5
with TSA
IBM N3700 (NAS)
Object Storage - LBOSDATA
INSTANCE
db2inst1 (Active)
AIX 5.3 ML9
IBM TotalStorage DS 4200
IBM HTTP Server
WAS Plugin
IBM WebSphere
Application Server 6.1
Network Deployment
IBM HTTP Server
WAS Plugin
WAS Node CMAIX25
icmrm
WAS Node CMAIX26
icmrm
WAS Cluster
ICMRM
Deployment Manager
192.168.17.19
192.168.17.20
Private Network
Figure 1: Overall Scenario (2 Node Configuration)
Copyright © 2004, 2009 IBM
Page 19 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Hardware Setup
This chapter summarizes the hardware we have used to setup Content Manager, including the servers,
network and storage. We understand, that there is a wide range of options available, and there are many
different ways to implement such a solution. We have chosen to use a configuration with just 2 servers,
each running all components of Content Manager.
Servers
We have used two identical IBM® POWER6® Server 520. Here are the most important characteristics of
these servers:
Processors: 4x POWER6 CPU @4.204 GHz
Hard Disk: 146 GB SAS DASD (“Serial Attached SCSI” and “Direct Access Storage Device”), IBM
ST3146855SS
Memory: 8 GB
Fiber Channel Adapter: QLE2462
Network Cards: Dual 1 GB Ethernet card (10N9042)
The IBM® pSeries® POWER6 server is connected via Gigabit Ethernet to the public network, and two
Ethernet switches handle the network traffic. Also included into the pSeries server is a Fiber Channel
Switch, which handles the connection to the IBM® TotalStorage® DS4200.
Storage
Each server has access to three different types of hard disk storage devices: internal disk, fiber channel
attached storage (TotalStorage DS4200) and a NFS server (TotalStorage N3700). We used different
storage types for different components of the Content Manager software stack.
Internal Disk (SAS)
We used one internal disk to store the operating system as well as installed software. This information is
NOT shared between the nodes in the cluster. Also on this internal disk, we placed the AIX paging space.
For consideration of redundancy, another internal disk can be added to create mirrorvg with original disk.
Fiber Channel Attached Storage (IBM DS4200)
Attached to all nodes in our cluster is an IBM DS4200, a storage subsystem, which is connect via
redundant fiber channel connections to the pSeries server 520. In our scenario we will use the DS4200 to
store DB2 instance and DB2 database files. The benefit of using fiber attached storage is its resilience,
which in most configurations includes redundant communication paths and redundant disks (RAID).
The configuration of your storage subsystem is a very important factor for the performance of Content
Management solution. The number of operations per second and the throughput of your storage
subsystem directly influence the performance of your database. Frequently, individual table spaces,
Copyright © 2004, 2009 IBM
Page 20 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
database log files, recovery files etc. are separated onto individual disk arrays, to achieve high
performance and no blocking access.
In our scenario, we kept the configuration of the DS4200 simple. Our DS4200 was filled with 16x1TB
hard disks, which have been configured into a single RAID-5 array of 15 disks and one hot spare. This is
certainly not the perfect configuration for performance but easy to setup and maintain.
On this array we created 2 logical drives, one is used by db2inst1, and another is used by db2inst2.
Machine 1 (cmaix25) will be active node for db2inst1, and standby node for db2inst2. Machine 2
(cmaix26) will be active node for db2inst2, and standby node for db2inst1. This can easily be done by
using the IBM® System Storage™ Manager.
root@cmaix25 # lscfg -vl fcs0|grep Network
Network Address.............10000000C97E64B5
root@cmaix26 # lscfg -vl fcs0|grep Network
Network Address.............10000000C973F7A6
This command will list the WWPN available on your AIX servers. Depending on your hardware
configuration, you might get several WWPNs listed. In our case, the WWPNs are:
cmaix25: 10000000C97E64B5
cmaix26: 10000000C973F7A6
With this information, define your logical drives on your storage server and map them to the WWPNs.
Table 1: Logical Drive definitions on DS4200 contain our definitions:
Name
Size
Host OS
Mapping
DB2INST1VG
40GB
AIX
10000000C97E64B5
10000000C973F7A6
DB2INST2VG
40GB
AIX
10000000C97E64B5
10000000C973F7A6
Table 1: Logical Drive definitions on DS4200
Figure 2: DS4200 Configuration shows the definitions in the System Storage Manager Client. You can
see the 2 defined logical drives and its mapping to the host group CMAIX25_26. Note that in this
screenshot, the mapping has been defined to two nodes: cmaix25 and cmaix26 as mentioned above.
Copyright © 2004, 2009 IBM
Page 21 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 2: DS4200 Configuration
When you do your storage system configuration, the most important questions are:
How many logical drives will I need?
How large should the logical drives be?
You will need at least two logical drives, one for db2inst1 (used by LS database ICMNLSDB), and one
for db2inst2 (used by RM database RMDB). Both of them will contain the database files (tablespace etc.)
which will grow. Please see your IBM sales representative for sizing information for a given number of
documents and workload to better size your database files.
Network Attached Storage (IBM N3700)
To store the actual documents, Content Manager can use several storage options, for example file systems
or Tivoli Storage Manager. The setup for the storage to file system is simple. It requires the administrator
to create the mount point and define it to Content Manager. In our sample configuration, we have used a
Network File System (NFS) file system hosted on an IBM N3700 as storage volume.
On your NAS Server, create the volume to be exported. Figure 3: IBM N3700 Volume Definition shows
the values we have used. We created a volume of 50GB and then defined the NFS exports.
Copyright © 2004, 2009 IBM
Page 22 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 3: IBM N3700 Volume Definition
When you define the NFS exports, you basically have to define the following settings:
Path: /vol/Lbosdata
Read-Write access for the following hosts:
cmaix25.cn.ibm.com
cmaix26.cn.ibm.com
Root access for the following hosts:
cmaix25.cn.ibm.com
cmaix26.cn.ibm.com
Security: UNIX System (sys)
Network
In our sample configuration, each server requires two physical network interfaces (NIC) and three IP
addresses. For HA consideration, we also recommend to use four NICs on each server. Then combine
every two NICs to a VIP. Of course, this also means two IP per server. The servers are accessible via a
public network, and are interconnected via a private network. The public network is used by client
applications to access the servers services. The private network should be used exclusively for heart beat
monitor. The third IP is used as a float virtual IP, it can be used on any active node.
On the public network, one IP address is required per server. Also, there will be a virtual IP for each
server on public network. On the private network, one IP address is required per server, and is typically in
a non-routed subnet such as 192.168.x.x for example. Table 2: IP Configuration show the network
settings we have used.
Copyright © 2004, 2009 IBM
Page 23 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
Our Name
IP Address
Used for
cmaix25
9.123.112.199
Host name node 1
cmaix25-priv
192.168.17.19
Private connect node 1
cmlsdb
9.123.112.251
Virtual IP for LS DB
cmaix26
9.123.112.200
Host name node 2
cmaix26-priv
192.168.17.20
Private connect node 2
cmrmdb
9.123.112.252
Virtual IP for RM DB
cmrmapp
9.123.112.254
Virtual IP for RM APP
Your Name
2009.12.28
Your IP Address
Table 2: IP Configuration
It is important, that all host names can be resolved. For the physical IP on the public subnet, you should
add them into your DNS. For the private interconnect, you can optionally add them to your /etc/hosts files.
Samples below shows what should be added into /etc/hosts file.
9.123.112.199
192.168.17.19
9.123.112.251
9.123.112.200
192.168.17.20
9.123.112.252
9.123.112.254
cmaix25
cmaix25-priv
cmlsdb
cmaix26
cmaix26-priv
cmrmdb
cmrmapp
Copyright © 2004, 2009 IBM
Page 24 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Operating System Preparation
This chapter contains information on how to prepare your AIX operating system for the user of Content
Manager. It describes necessary changes to your AIX configuration to run Content Manager and DB2
software. As we will prepare two servers with same configuration, all the steps described in this chapter
must be completed on ALL nodes participating in the cluster!
Virtual I/O Server
The Virtual I/O Server is part of the IBM System p5 Advanced Power Virtualization hardware feature.
Virtual I/O Server allows sharing of physical resources between LPARs including virtual SCSI and
virtual networking. This allows more efficient utilization of physical resources through sharing between
LPARs and facilitates server consolidation.
Before using VIO servers, please see your IBM AIX representative to eliminate single points of failure
(SPOF).
Base AIX Installation
For our setup, we installed AIX 5.3 TL9 SP2 with 64bit kernel. During the installation we’ve selected
default options, including the package selection. DON’T install OS update package before you finish
prerequisite software installation.
After installation, make sure that you assign sufficient paging space. A rule of thumb is to create a Paging
Space of at least the size of your physical memory.
root@cmaix25 # lsps -a
Page Space
Physical Volume
hd6
hdisk0
Volume Group
rootvg
Size %Used Active Auto Type
4096MB
1 yes
yes
lv
Then, check system disk and make sure we have enough space for future software installation.
Recommended configuration is shown below:
root@cmaix25 # df -m
Filesystem
MB blocks
/dev/hd4
2048.00
/dev/hd2
6144.00
/dev/hd9var
2048.00
/dev/hd3
2048.00
/dev/hd1
4096.00
/proc
/dev/hd10opt
4096.00
/dev/lv00
256.00
Free %Used
1986.82
3%
4396.60
29%
1976.23
4%
2041.96
1%
4095.04
1%
3942.52
4%
247.92
4%
Iused %Iused Mounted on
4731
2% /
41918
5% /usr
4633
2% /var
31
1% /tmp
5
1% /home
- /proc
1950
1% /opt
18
1% /var/adm/csd
Prerequisite Software
On AIX, high availability is implemented with RSCT components. Then following packages must be
installed before we setup HA environment:
Copyright © 2004, 2009 IBM
Page 25 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
rsct.basic
rsct.compat.basic
rsct.compat.clients
Also, another package is prerequisite for Tivoli System Automation:
rsct.opt.storagerm
These filesets can be found in AIX installation DVD, you can just insert Installation DVD into pSeries
server and then run “smitty install” command to finish installation.
Finally we’ve applied the latest patches to our AIX installation. Our version is:
root@cmaix26 # oslevel –s
5300-09-02-0849
Additional, on the AIX version later than 5.3 TL9, this is a known problem described at:
http://www-01.ibm.com/support/docview.wss?uid=isg1IZ47850
We must close http4websm service and leave port 80 for our future http usage.
root@cmaix26 # vi /etc/inittab
: webserverstart:2:once:startsrc -s http4websm >/dev/null 2>&1
root@cmaix26 # stopsrc -s http4websm
0513-044 The http4websm Subsystem was requested to stop.
Storage Configuration
The first external storage we want to attach is an IBM DS4200, which is a fiber channel attached storage
server. Previously we have already defined two volumes on the DS4200 and mapped them to the WWPN
of our servers. In our case, we did have redundant fiber channel connections between the pSeries server
and the DS4200 storage server.
With AIX, driver is integrated, and multipath support is build into the fiber channel adapter driver. After
fresh installation, disk status will look similar to:
root@cmaix26 # lspv
hdisk0
000a5db26d77585c
rootvg
active
Then DS4200 can be assigned to each server with steps in previous chapter, and “cfgmgr” command is
executed on each server to discover LUNs of DS4200. We can find two disks on each server:
root@cmaix26 # lspv
hdisk0
000a5db26d77585c
hdisk2
none
hdisk3
none
rootvg
None
None
active
For future usage, each disk should have PVID, as “000925926d77bb8e” of hdisk0. But other disks from
DS4200 may not get PVID automatically, we need update their information.
root@cmaix26 # chdev -l hdisk2 -a pv=yes
hdisk2 changed
root@cmaix26 # chdev -l hdisk3 -a pv=yes
hdisk3 changed
Copyright © 2004, 2009 IBM
Page 26 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
root@cmaix26 # lspv
hdisk0
000a5db26d77585c
hdisk2
000a5d72aa712308
hdisk3
000a5d72baaf227e
rootvg
None
None
2009.12.28
active
After running on each server, configuration is finished. The first disk (hdisk0) is our internal disk, and
they have different PVID on each server. The next two disks (hdisk2 and hdisk3) are our two volumes
over the fiber channel connections, which show same PVID on each server.
Logical Volume and Mount Points
Next, we want to prepare the directories and mount points for DB2 software installation. In our case, we
want to install the DB2 software into the default directory /opt/IBM/db2/V9.5.
Further we create two mount point (/DB2INST1 and /DB2INST2 on both cmaix25 and cmaix26), where
we will later mount DB2 instance volume to store DB2 instance file and DB file. Also, we can place
home directory of db2inst1, db2inst2, db2fenc1, db2fenc2 into it.
Finally we will create the mount point for the resource manager storage volume, typically called
“lbosdata”. Here we will later mount an NFS volume.
File System Configuration on Node 1
Create Volume Group
We can use “smitty” or using two commands below to create two VGs on two different disks. Notice that
VG Major Number (ID of Volume Group) of cmaix25 and cmaix26 should be SAME for every hdisk.
root@cmaix25 # mkvg -S -y DB2VG1 -f -n -V 45 hdisk2
DB2VG1
root@cmaix25 # mkvg -S -y DB2VG2 -f -n -V 46 hdisk3
DB2VG2
root@cmaix25 # lspv
hdisk0
000925926d77bb8e
hdisk2
000a5d72aa712308
hdisk3
000a5d72baaf227e
rootvg
DB2VG1
DB2VG2
active
active
active
Create File System
Now we can create file systems and mount points on DB2VG1 and DB2VG2, scripts are used to create
20G JFS2 file system for /DB2INST1, and same size for /DB2INST2. The commands below will create
the necessary directories. This will also define proper ownership and permissions for the directories.
root@cmaix25 # crfs -v jfs2 -g DB2VG1 -a size=20G -m /DB2INST1 -A no -p rw -a
agblksize=4096
File system created successfully.
20970676 kilobytes total disk space.
New File System size is 41943040
root@cmaix25 # crfs -v jfs2 -g DB2VG2 -a size=20G -m /DB2INST2 -A no -p rw -a
agblksize=4096
File system created successfully.
20970676 kilobytes total disk space.
New File System size is 41943040
Copyright © 2004, 2009 IBM
Page 27 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
root@cmaix25 # mount /DB2INST1
root@cmaix25 # mount /DB2INST2
root@cmaix25 # chown -R bin:bin /DB2INST1
root@cmaix25 # chown -R bin:bin /DB2INST2
root@cmaix25 # chmod -R 755 /DB2INST1
root@cmaix25 # chmod -R 755 /DB2INST2
Create Users and Groups
As we will create four user’s home directory under /DB2INST1 and /DB2INST2, we should run scripts to
create users and groups with both two directory mounted on same machine. Scripts are provided on next
paragraph, just modify user name and password, in our case, we ran them here on node1 (maix25) with
/DB2INST1 and /DB2INST2 mounted.
root@cmaix25 # <Create Groups and Users with /DB2INST1 and /DB2INST2 mounted>
File System Configuration on Node 2
Then we can setup totally same environment on cmaix26, active RMDB node. We can just import
information from cmaix25 to cmaix26, without creating new File Systems and Volume Groups.
Close DB2VG1 and DB2VG2 from cmaix25.
root@cmaix25 # umount /DB2INST1
root@cmaix25 # umount /DB2INST2
root@cmaix25 # varyoffvg DB2VG1
root@cmaix25 # varyoffvg DB2VG2
Check VG number for DB2VG1 and DB2VG2 on cmaix25, this is important to keep this information
same on both two nodes.
root@cmaix25 # ls -al /dev/DB2VG1
crw-rw---1 root
system
45,
0 Aug 06 15:23 /dev/DB2VG1
root@cmaix25 # ls -al /dev/DB2VG2
crw-rw---1 root
system
46,
0 Aug 06 15:27 /dev/DB2VG2
Turn back to cmaix26, we can use same VG Major number to import same hdisk:
root@cmaix26 # importvg -y DB2VG1 -V 45 hdisk2
root@cmaix26 # importvg -y DB2VG2 -V 46 hdisk3
root@cmaix26 # mount /DB2INST1
root@cmaix26 # mount /DB2INST2
Create Users and Groups
Node2 (cmaix26) now have same File System information with Node1 (cmaix25), but user information is
not synchronized. We should run scripts to create users and groups on Node2, which are exactly same
with previous scripts on Node1.
root@cmaix26 # <Create Groups and Users with /DB2INST1 and /DB2INST2 mounted>
Copyright © 2004, 2009 IBM
Page 28 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Preparation for DB2
Finally, we can mount /DB2INST1 Node1 (cmaix25) and mount /DB2INST2 on Node2 (cmaix26).
On Node2 (cmaix26):
root@cmaix26 # umount /DB2INST1
root@cmaix26 # varyoffvg DB2VG1
On Node1 (cmaix25):
root@cmaix25 # varyonvg DB2VG1
root@cmaix25 # mount /DB2INST1
Users and Groups
Both DB2 UDB and Content Manager, require certain operating system users and groups to exist. For
DB2 UDB v9.5, typically three groups (dasadm1, db2iadm1 and db2fadm1), and three users (dasusr1,
db2inst1, db2fenc1) are required. Content Manager requires the group ibmcmgrp and ther user
ibmcmadm. It is important to create the users and groups in an identical way, sharing the same numerical
ids and names, on all nodes in the cluster. Table 3: Operating system groups and users show the settings
we have used.
Group or User
Role
Numeric ID:
Group: dasadm1
DB2 Admin Server Group
101
Group: db2iadm1
DB2 Instance Group 1
102
Group: db2fadm1
DB2 Fenced Group 1
103
Group: db2iadm2
DB2 Instance Group 2
104
Group: db2fadm2
DB2 Fenced Group 2
105
Group: ibmcmgrp
All Content Manager users
207
User: dasusr1
DB2 Admin Server User
203
User: db2inst1
DB2 instance 1 owner
204
User: db2fenc1
DB2 Fenced user
205
User: db2inst2
DB2 instance 2 owner
206
User: db2fenc2
DB2 fenced user
207
User: ibmcmadm
Content Manager
configuration owner
208
User: icmadmin
Content Manager admin for
LS DB
209
Copyright © 2004, 2009 IBM
Name in your
Environment
Page 29 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
LS DB
User: icmconct
Content Manager connect
user
210
User: rmadmin
Content Manager admin for
RM DB
211
Table 3: Operating system groups and users
The following commands will create these definitions. In our case, we used group ids 101 through 105
and 207 for the groups and users ids 203 through 211. Run them on each node with the steps in “Mount
Points” chapter, and /DB2INST1 and /DB2INST2 must be mounted on same machine before running this
scripts. Also, change password to replace “YOUR_PASSWORD” area.
# mkgroup -A id=101 dasadm1
# mkgroup -A id=102 db2iadm1
# mkgroup -A id=103 db2fadm1
# mkgroup -A id=104 db2iadm2
# mkgroup -A id=105 db2fadm2
# mkgroup -A id=207 ibmcmgrp
#
# mkuser id=203 pgrp=dasadm1 groups=dasadm1,staff home='/home/dasusr1' dasusr1
# mkuser id=204 pgrp=db2iadm1 groups=db2iadm1,staff,dasadm1,db2fadm1
home='/DB2INST1/db2inst1' db2inst1
# mkuser id=205 pgrp=db2fadm1 groups=db2fadm1,staff,db2iadm1 home='/DB2INST1/db2fenc1'
db2fenc1
# mkuser id=206 pgrp=db2iadm2 groups=db2iadm2,staff,dasadm1,db2fadm2
home='/DB2INST2/db2inst2' db2inst2
# mkuser id=207 pgrp=db2fadm2 groups=db2fadm2,staff,db2iadm2 home='/DB2INST2/db2fenc2'
db2fenc2
#
# mkuser id=208 pgrp=ibmcmgrp groups=ibmcmgrp home='/home/ibmcmadm' ibmcmadm
# mkuser id=209 pgrp=staff groups=staff,db2iadm1,ibmcmgrp home='/home/icmadmin' icmadmin
# mkuser id=210 pgrp=staff groups=staff,ibmcmgrp home='/home/icmconct' icmconct
# mkuser id=211 pgrp=staff groups=staff,db2iadm2,ibmcmgrp home='/home/rmadmin' rmadmin
#
# for i in dasusr1 db2inst1 db2fenc1 db2inst2 db2fenc2 ibmcmadm icmadmin icmconct rmadmin ;
do echo echo $i:[YOUR_PASSWORD] \| chpasswd ;done | ksh
# for i in dasusr1 db2inst1 db2fenc1 db2inst2 db2fenc2 ibmcmadm icmadmin icmconct rmadmin ;
do pwdadm -c $i ;done
NFS Configuration
The last piece of external storage we need is the default storage volume for the resource manager. Earlier
we have defined both, an NFS export on the IBM N3700 and a directory (/lbosdata) to become the mount
point. All that is missing is the mount itself. To mount the NFS volume, add to your /etc/filesystems the
following line:
/lbosdata:
Copyright © 2004, 2009 IBM
Page 30 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
dev
vfs
nodename
mount
options
account
2009.12.28
= /Vol/Lbosdata
= nfs
= cmaix27
= true
= rw
= false
In this sample, cmaix27.cn.ibm.com is the hostname of the NFS server, /Vol/Lbosdata the name of the
exported volume, /lbosdata the mount point on the client side, nfs the file system type and the remaining
parts are options for the mount command. The volume will be automatically mounted after a reboot. To
mount the volume immediately, run:
# mount /Lbosdata
Repeat the above steps on all nodes participating in the cluster. Now verify, that all required file systems
are mounted. Below is the output for one of our servers:
root@cmaix25 # df -m
Filesystem
MB blocks
Free %Used
……
cmaix27:/Vol/Lbosdata 40960.00 40952.47
Copyright © 2004, 2009 IBM
Iused %Iused Mounted on
1%
3
1% /lbosdata
Page 31 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
DB2 with Tivoli System Automation
Now we are ready to start with the installation of DB2. We have installed the prerequisite software and
configured our network and storage. Tivoli System Automation is provided with DB2 installation
package. The steps outlined in this document should be performed on both nodes, however, different
nodes will setup different instance.
Pre-Install Validation
Check both of two servers, and make sure /DB2INST1 is mounted on host cmaix25, and /DB2INST2 is
mounted on cmaix26.
# df –m (on cmaix25)
/dev/fslv00
20480.00 20178.25
2%
283
1% /DB2INST1
# df –m (on cmaix26)
/dev/fslv01
20480.00 20178.25
2%
283
1% /DB2INST2
Software Installation
We need to install DB2 software on both servers, with the same configuration; however, we will
not setup DB2 Instance at this step. Insert DB2 installation media, and run “db2setup”. Then
Figure 4: DB2 setup wizard will be shown.
NOTE: Although the following screen shots are from fixpack 3, remember that HA support for
Content Manager on DB2 requires DB2 v9.5 fixpack 5 or higher.
Figure 4: DB2 setup wizard
Copyright © 2004, 2009 IBM
Page 32 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Choose “Install a Product” and select “Install New” in Figure 5: Install New DB2 product.
Figure 5: Install New DB2 product
Click “Next” in Figure 6: Welcome window.
Figure 6: Welcome window
Accept software license and “Next” when you see Figure 7: Accept License.
Copyright © 2004, 2009 IBM
Page 33 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 7: Accept License
For installation type, we select “Custom” in Figure 8: Installation Type.
Figure 8: Installation Type
In step 4, we can use default option for Figure 9: Response file.
Copyright © 2004, 2009 IBM
Page 34 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 9: Response file
Check the features that you need to install in Figure 10: Select the features to Install.
Figure 10: Select the features to Install
For Figure 11: Select the langurages to Install, select language and installation directory.
Copyright © 2004, 2009 IBM
Page 35 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 11: Select the langurages to Install
We can ignore installing DB2 information center for Figure 12: Specify the location of DB2
Information Center.
Figure 12: Specify the location of DB2 Information Center
Install IBM® Tivoli® System Automation for Multiplatforms Base Component (SA MP Base
Component) at the step of Figure 13: Install the IBM SA MP Component. For DB2 v9.5, SA MP
version 2.2 is provided. Recall that you must have purchased a full DB2 or Tivoli System
Copyright © 2004, 2009 IBM
Page 36 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Automation license to use Tivoli System Automation with DB2 and Content Manager. The
limited-use DB2 license provided with your Content Manager license does not include Tivoli
System Automation. You must check if license is installed correctly after installation, as
introduced later.
Figure 13: Install the IBM SA MP Component
Use existing user dasusr1 which is already created on both of two DAS servers for Figure 14: Set
user information for the DB2 Administration Server.
Copyright © 2004, 2009 IBM
Page 37 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 14: Set user information for the DB2 Administration Server
We will not create DB2 instance in Figure 15: Set up a DB2 instance.
Figure 15: Set up a DB2 instance
Customer can select if notification is needed in Figure 16: Set up notification.
Copyright © 2004, 2009 IBM
Page 38 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 16: Set up notification
Here is the summary file in Figure 17: Summary, click “Finish” after checking settings.
Figure 17: Summary
Repeat the same steps in another node, NODE2. Then, we should check license on both two
nodes.
Copyright © 2004, 2009 IBM
Page 39 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Notice: If Tivoli System Automation is not installed during DB2 software installation, we can run
installSAM under /db2/aix/tsamp directory on DB2 installation media.
root@cmaix25 # cd db2/aix/tsamp
root@cmaix25 # ./installSAM
prereqSAM: All prerequisites for the ITSAMP installation are met on operating system AIX
5300-09
installSAM: Packages will be installed from directory ./AIX
installSAM: The following package is not installed yet and needs to be installed ./AIX/sam.core
installSAM: A general License Agreement and License Information specifically for System
Automation will be shown. Scroll down using the Enter key (line by line) or Space bar (page by
page). At the end you will be asked to accept the terms to be allowed to install the product. Select
Enter to continue
... ...
installSAM: To accept all terms of the preceding License Agreement and License Information type
'y', anything else to decline
License Validation
Now, check your DB2 and Tivoli System Automation license on both nodes:
# cd /opt/IBM/db2/V9.5/adm
# ./db2licm -l
#
# samlicm -s
Make sure you have correct license, please be aware of following APAR before registering
Tivoli System Automation license:
IZ47958: SA MP LICENSE CANNOT BE INSTALLED AS NON-ROOT USER USING
'SAMLICM' (http://www01.ibm.com/support/docview.wss?rs=820&context=SSRM2X&dc=DB550&uid=swg1IZ47958
&loc=en_US&cs=UTF-8&lang=en&rss=ct820tivoli)
Setup Instance: db2inst1
On node1 (cmaix25), we will setup a DB2 instance named db2inst1. Related files will be placed
at db2inst1 user’s home directory.
root@cmaix25 # cd /opt/IBM/db2/V9.5/instance
root@cmaix25 # ./db2isetup
DB2 instance setup wizard as Figure 18: DB2 instance Setup wizard will be shown, click “Next”.
Copyright © 2004, 2009 IBM
Page 40 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 18: DB2 instance Setup wizard
Create a DB2 instance in Figure 19: Set up a DB2 instance.
Figure 19: Set up a DB2 instance
As Content Manager supports single partition instance only, we check that option in Figure 20:
Set up partitioning options for the DB2 instance.
Copyright © 2004, 2009 IBM
Page 41 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 20: Set up partitioning options for the DB2 instance
In Figure 21: Set user information for the DB2 instance owner, we use existing user “db2inst1” for
DB2 instance owner, which is already created in previous steps.
Figure 21: Set user information for the DB2 instance owner
For fenced user in Figure 22: Set user Information for the fenced user, we use existing user db2fenc1.
Copyright © 2004, 2009 IBM
Page 42 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 22: Set user Information for the fenced user
In Figure 23: Configure instance communication and startup, we use 50001 port for db2inst1, and
port for db2inst2 should be different in following chapter. Uncheck “Autostart the instance at
system startup”, as DB2 instance will be monitored and managed by SA MP.
Figure 23: Configure instance communication and startup
DB2 tools catalog is optional for Figure 24: Prepare the DB2 tools catalog.
Copyright © 2004, 2009 IBM
Page 43 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 24: Prepare the DB2 tools catalog
Users can specify a contact for health monitor notification in Figure 25: Specify a contact for health
monitor notification.
Figure 25: Specify a contact for health monitor notification
Content Manager doesn’t use DB2 Text Search server. For text search feature, please refer to
DB2 Net Search Extender component. We can select “Do not configure at this time” in Figure 26:
Configure the DB2 Text Search server
Copyright © 2004, 2009 IBM
Page 44 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 26: Configure the DB2 Text Search server
Check summary in Figure 27: Summary and click “Finish”.
Figure 27: Summary
After db2inst1 setup, check /etc/services, we can find following content. As db2inst1 will be run
on cmaix26 when cmaix25 is not available, we should add same content on /etc/services of
cmaix26 too.
DB2_db2inst1
60000/tcp
Copyright © 2004, 2009 IBM
Page 45 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
DB2_db2inst1_1 60001/tcp
DB2_db2inst1_2 60002/tcp
DB2_db2inst1_END
60003/tcp
db2c_db2inst1
50001/tcp
Setup Instance: db2inst2
On cmaix26, we will setup another DB2 instance named db2inst2. Related files will be put at
db2inst2 user’s home directory.
root@cmaix26 # cd /opt/IBM/db2/V9.5/instance
root@cmaix26 # ./db2isetup
Difference is that we will assign existing user db2inst2 in Figure 28: Set user information for the
DB2 instance owner.
Figure 28: Set user information for the DB2 instance owner
And add db2fenc2 for fenced user in Figure 29: Set user information for the fenced user.
Copyright © 2004, 2009 IBM
Page 46 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 29: Set user information for the fenced user
As port 50001 is kept for db2inst1, db2inst2 will automatically get port 50002. Uncheck
“Autostart the instance at system startup” in Figure 30: Configure instance communication and
startup.
Figure 30: Configure instance communication and startup
In Figure 31: Summary, view the summary and click “Finish”.
Copyright © 2004, 2009 IBM
Page 47 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 31: Summary
Also, new content will be added at /etc/services of cmaix26:
DB2_db2inst2
60004/tcp
DB2_db2inst2_1 60005/tcp
DB2_db2inst2_2 60006/tcp
DB2_db2inst2_END
60007/tcp
db2c_db2inst2
50002/tcp
After modification, check /etc/services on both two servers. Both of them must have the same
configuration:
DB2_db2inst1
60000/tcp
DB2_db2inst1_1 60001/tcp
DB2_db2inst1_2 60002/tcp
DB2_db2inst1_END
60003/tcp
db2c_db2inst1
50001/tcp
DB2_db2inst2
60004/tcp
DB2_db2inst2_1 60005/tcp
DB2_db2inst2_2 60006/tcp
DB2_db2inst2_END
60007/tcp
db2c_db2inst2
50002/tcp
Copyright © 2004, 2009 IBM
Page 48 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
DB2 High Availability Instance Configuration Utility
Now we can start DB2 HA configuration. We will setup two instances on cmaix25 and cmaix26, for
db2inst1, its original location is /DB2INST1/db2inst1 directory on cmaix25, and db2inst2 will be put at
/DB2INST2/db2inst2 directory on cmaix26. Also, db2inst1 uses a virtual IP 9.123.112.251, and db2inst2
uses another virtual IP 9.123.112.252. When one of the servers is down, the other server will acquire and
use the resources, including virtual IP.
Tivoli System Automation is working with DB2 to provide high availability, and it’s configured using a
utility named db2haicu, which is a tool available with DB2 9.5 and stands for the “DB2 High
Availability Instance Configuration Utility”. This utility takes in user input regarding the software and
hardware environment of a DB2 instance, and configures the instance for High Availability using the
Tivoli System Automation cluster manager. During this configuration process, all necessary resources,
dependencies, and equivalencies are automatically defined to Tivoli System Automation. Note: Tivoli
System Automation does not need to be manually installed on your system as it is pre-packaged with DB2
9.5.
Two input methods can be used to provide the necessary data to db2haicu. The first method is the
interactive mode, where you are prompted for input at the command line. The second input method is the
XML mode, where db2haicu can parse the necessary data from a user defined XML file.
The db2haicu tool can be used to configure such an HA system. During the db2haicu configuration
process, the necessary HA resources and their relationships are defined to the cluster manager. Failure
events in the HA system can then be detected automatically and takeover operations can be run without
manual intervention.
Cluster Preparation
Before using the db2haicu tool, the primary and the standby nodes must be prepared with the proper
security environment. With root authority, issue the following command on all cluster nodes:
root@cmaix25 # preprpnode cmaix25 cmaix26
root@cmaix26 # preprpnode cmaix25 cmaix26
This command only needs to be run once per node and not for every DB2 instance that is made highly
available.
Time Synchronization
It is recommended (but not mandatory) that the time and dates on cluster nodes be synchronized.
Synchronized clocks can make problem determination more straightforward because time ordering of
events that appear in various log files can be performed without applying a delta time correction to
account for time differences between machines. Note that the xntpd subsystem can be used for this
purpose. Refer to your operating system documentation for information on how to configure xntpd for
your system.
Copyright © 2004, 2009 IBM
Page 49 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
The db2haicu Interactive Setup on Node1
After the preceding preliminary configuration steps are completed, the db2haicu tool can be used to
automate HA failover.
The db2haicu must be run on the node hosting the DB2 instance. The details involving the process are
outlined in the following section.
Creating a Cluster Domain
Log on to the machine hosting the DB2 instance (for our case, cmaix25 is owner of db2inst1), and issue
the ‘db2haicu’ command:
The following welcome message will be displayed on the screen:
Welcome to the DB2 High Availability Instance Configuration Utility (db2haicu).
You can find detailed diagnostic information in the DB2 server diagnostic log file called
db2diag.log. Also, you can use the utility called db2pd to query the status of the cluster domains
you create.
For more information about configuring your clustered environment using db2haicu, see the topic
called 'DB2 High Availability Instance Configuration Utility (db2haicu)' in the DB2 Information
Center.
db2haicu determined the current DB2 database manager instance is db2inst1. The cluster
configuration that follows will apply to this instance.
db2haicu is collecting information on your current setup. This step may take some time as
db2haicu will need to activate all databases for the instance to discover all paths ...
When you use db2haicu to configure your clustered environment, you create cluster domains. For
more information, see the topic 'Creating a cluster domain with db2haicu' in the DB2 Information
Center. db2haicu is searching the current machine for an existing active cluster domain ...
db2haicu did not find a cluster domain on this machine. db2haicu will now query the system for
information about cluster nodes to create a new cluster domain ...
db2haicu did not find a cluster domain on this machine. To continue configuring your clustered
environment for high availability, you must create a cluster domain; otherwise, db2haicu will exit.
Create a domain and continue? [1]
1. Yes
2. No
1
Create a unique name for the new domain:
DB2Domain
Nodes must now be added to the new domain.
How many cluster nodes will the domain DB2Domain contain?
2
Enter the host name of a machine to add to the domain:
cmaix25
Copyright © 2004, 2009 IBM
Page 50 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Enter the host name of a machine to add to the domain:
cmaix26
db2haicu can now create a new domain containing the 2 machines that you specified. If you
choose not to create a domain now, db2haicu will exit.
Create the domain now? [1]
1. Yes
2. No
1
Creating domain DB2Domain in the cluster ...
Creating domain DB2Domain in the cluster was successful.
Quorum Configuration
After the domain creation has completed, a quorum must be configured for the cluster domain. The
supported quorum type for this solution is a ‘network quorum’. A network quorum (or network tiebreaker)
is a ping-able IP address that is used to decide which node in the cluster will serve as the ‘active’ node
during a site failure, and which nodes will be offline. Note that the machine hosting this IP address does
not need any particular software or operating system level installed; its primary requirement is that it can
be pinged from all nodes in the cluster, and must remain pingable in the case of cluster node failures.
You will be prompted by db2haicu to enter quorum configuration values:
You can now configure a quorum device for the domain. For more information, see the topic
"Quorum devices" in the DB2 Information Center. If you do not configure a quorum device for the
domain, then a human operator will have to manually intervene if subsets of machines in the
cluster lose connectivity.
Configure a quorum device for the domain called DB2Domain? [1]
1. Yes
2. No
1
The following is a list of supported quorum device types:
1. Network Quorum
Enter the number corresponding to the quorum device type to be used: [1]
1
Specify the network address of the quorum device:
9.123.112.1
Configuring quorum device for domain DB2Domain ...
Configuring quorum device for domain DB2Domain was successful.
Network Setup
After the quorum configuration, you may define the public network of your system to db2haicu. If
network failure detection is important to your configuration, you must follow the prompts and add the
networks to the cluster at this point. The db2haicu tool automatically discovers all network interfaces.
Copyright © 2004, 2009 IBM
Page 51 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
The cluster manager found 4 network interface cards on the machines in the domain. You can use
db2haicu to create networks for these network interface cards. For more information, see the topic
'Creating networks with db2haicu' in the DB2 Information Center.
Create networks for these network interface cards? [1]
1. Yes
2. No
1
Enter the name of the network for the network interface card: en0 on cluster node:
cmaix25.cn.ibm.com
1. Create a new public network for this network interface card.
2. Create a new private network for this network interface card.
1
Enter selection:
Are you sure you want to add the network interface card en0 on cluster node cmaix25.cn.ibm.com
to the network db2_public_network_0? [1]
1. Yes
2. No
1
Adding network interface card en0 on cluster node cmaix25.cn.ibm.com to the network
db2_public_network_0 ...
Adding network interface card en0 on cluster node cmaix25.cn.ibm.com to the network
db2_public_network_0 was successful.
Enter the name of the network for the network interface card: en1 on cluster node:
cmaix26.cn.ibm.com
1. db2_public_network_0
2. Create a new public network for this network interface card.
3. Create a new private network for this network interface card.
3
Enter selection:
Are you sure you want to add the network interface card en1 on cluster node cmaix26.cn.ibm.com
to the network db2_private_network_0? [1]
1. Yes
2. No
1
Adding network interface card en1 on cluster node cmaix26.cn.ibm.com to the network
db2_private_network_0 ...
Adding network interface card en1 on cluster node cmaix26.cn.ibm.com to the network
db2_private_network_0 was successful.
Enter the name of the network for the network interface card: en0 on cluster node:
cmaix26.cn.ibm.com
1. db2_private_network_0
2. db2_public_network_0
3. Create a new public network for this network interface card.
4. Create a new private network for this network interface card.
Enter selection:
Copyright © 2004, 2009 IBM
Page 52 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
2
Are you sure you want to add the network interface card en0 on cluster node cmaix26.cn.ibm.com
to the network db2_public_network_0? [1]
1. Yes
2. No
1
Adding network interface card en0 on cluster node cmaix26.cn.ibm.com to the network
db2_public_network_0 ...
Adding network interface card en0 on cluster node cmaix26.cn.ibm.com to the network
db2_public_network_0 was successful.
Enter the name of the network for the network interface card: en1 on cluster node:
cmaix25.cn.ibm.com
1. db2_private_network_0
2. db2_public_network_0
3. Create a new public network for this network interface card.
4. Create a new private network for this network interface card.
Enter selection:
1
Are you sure you want to add the network interface card en1 on cluster node cmaix25.cn.ibm.com
to the network db2_private_network_0? [1]
1. Yes
2. No
1
Adding network interface card en1 on cluster node cmaix25.cn.ibm.com to the network
db2_private_network_0 ...
Adding network interface card en1 on cluster node cmaix25.cn.ibm.com to the network
db2_private_network_0 was successful.
Cluster Manager Selection
After the network definitions, db2haicu prompts you for the cluster manager software that you are using
for the current HA setup. For our purpose, we select Tivoli System Automation:
Retrieving high availability configuration parameter for instance db2inst1 ...
The cluster manager name configuration parameter (high availability configuration parameter) is
not set. For more information, see the topic "cluster_mgr - Cluster manager name configuration
parameter" in the DB2 Information Center. Do you want to set the high availability configuration
parameter?
The following are valid settings for the high availability configuration parameter:
1.TSA
2.Vendor
Enter a value for the high availability configuration parameter: [1]
1
Setting a high availability configuration parameter for instance db2inst1 to TSA.
Copyright © 2004, 2009 IBM
Page 53 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Failover Policy
Now you need to configure the failover policy for the instance db2inst1. The failover policy determines
the machines on which the cluster manager will restart the database manager if the database manager goes
offline unexpectedly.
For our purpose, we select option 3. Note that the failover policy is a powerful concept for larger clusters
(with more nodes and more partitions), but for a our refer configuration, two-node single partition setup
(such as this one) should select option 3.
Now you need to configure the failover policy for the instance db2inst1. The failover policy
determines the machines on which the cluster manager will restart the database manager if the
database manager is brought offline unexpectedly.
The following are the available failover policies:
1. Local Restart -- during failover, the database manager will restart in place on the local
machine
2. Round Robin -- during failover, the database manager will restart on any machine in the
cluster domain
3. Active/Passive -- during failover, the database manager will restart on a specific machine
4. M+N -- during failover, the database partitions on one machine will failover to any other
machine in the cluster domain (used with DPF instances)
5. Custom -- during failover, the database manager will restart on a machine from a userspecified list
Enter your selection:
3
Then db2haicu will prompt you to designate any noncritical mount points. For this case, we chose to
designate only one noncritical mount point. Note that you should add any mount points to the noncritical
path list that you are sure that you never want to failover. This list should include any mount points
specified in /etc/filesystems that are local mount points and will never be failed over.
You can identify mount points that are noncritical for failover. For more information, see the topic
'Identifying mount points that are noncritical for failover' in the DB2 Information Center. Are
there any mount points that you want to designate as noncritical? [2]
1. Yes
2. No
2
The db2haicu tool will now automatically add the DB2 partition instance to the specified cluster manager
at this point.
Active/Passive failover policy was chosen. You need to specify the host names of an
active/passive pair.
Enter the host name for the active cluster node:
cmaix25
Enter the host name for the passive cluster node:
cmaix26
Adding DB2 database partition 0 to the cluster ...
Adding DB2 database partition 0 to the cluster was successful.
Copyright © 2004, 2009 IBM
Page 54 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Virtual IP Address Setup
Once the database partition has been added to the cluster, db2haicu will prompt you to create a virtual IP
address:
Do you want to configure a virtual IP address for the DB2 partition: 0? [2]
1. Yes
2. No
1
Enter the virtual IP address:
9.123.112.251
Enter the subnet mask for the virtual IP address 9.123.112.251: [255.255.255.0]
255.255.255.0
Select the network for the virtual IP 9.123.112.251:
1. db2_private_network_0
2. db2_public_network_0
Enter selection:
2
Adding virtual IP address 9.123.112.251 to the domain ...
Adding virtual IP address 9.123.112.251 to the domain was successful.
You must make sure that your IP address and subnet mask values are well formed and correspond with
the subnet mask of the network you chose. All invalid inputs will be rejected. In such a case, you are
advised to examine the IP addresses and netmasks of the NIC components of the network (using the
‘ifconfig’ command) and verify that the IP address and netmask specified are compatible with each of the
NICs in the network. In addition, make sure that the IP address that you want to add is not already present
on the network.
All cluster configurations have been completed successfully. db2haicu exiting ...
After the virtual IP address configuration, the Automated Cluster-controlled HA configuration is
complete. Run command ‘lssam’ and Figure 32: db2inst1 Tivoli System Automation status is shown.
Figure 32: db2inst1 Tivoli System Automation status
The db2haicu Interactive Setup on Node2
As Node2 will share same Cluster, Quorum device and Network with Node1, issue “db2haicu” will go to
Cluster Manager Selection directly.
Copyright © 2004, 2009 IBM
Page 55 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Cluster Manager Selection
After the network definitions, db2haicu prompts you for the cluster manager software that you are using
for the current HA setup. For our purpose, we select Tivoli System Automation:
Welcome to the DB2 High Availability Instance Configuration Utility (db2haicu).
You can find detailed diagnostic information in the DB2 server diagnostic log file called
db2diag.log. Also, you can use the utility called db2pd to query the status of the cluster domains
you create.
For more information about configuring your clustered environment using db2haicu, see the topic
called 'DB2 High Availability Instance Configuration Utility (db2haicu)' in the DB2 Information
Center.
db2haicu determined the current DB2 database manager instance is db2inst2. The cluster
configuration that follows will apply to this instance.
db2haicu is collecting information on your current setup. This step may take some time as
db2haicu will need to activate all databases for the instance to discover all paths ...
When you use db2haicu to configure your clustered environment, you create cluster domains. For
more information, see the topic 'Creating a cluster domain with db2haicu' in the DB2 Information
Center. db2haicu is searching the current machine for an existing active cluster domain ...
db2haicu found a cluster domain called DB2Domain on this machine. The cluster configuration
that follows will apply to this domain.
Retrieving high availability configuration parameter for instance db2inst2 ...
The cluster manager name configuration parameter (high availability configuration parameter) is
not set. For more information, see the topic "cluster_mgr - Cluster manager name configuration
parameter" in the DB2 Information Center. Do you want to set the high availability configuration
parameter?
The following are valid settings for the high availability configuration parameter:
1.TSA
2.Vendor
Enter a value for the high availability configuration parameter: [1]
1
Setting a high availability configuration parameter for instance db2inst2 to TSA.
Now you need to configure the failover policy for the instance db2inst2. The failover policy
determines the machines on which the cluster manager will restart the database manager if the
database manager is brought offline unexpectedly.
Failover Policy
Now you need to configure the failover policy for the instance db2inst1. The failover policy determines
the machines on which the cluster manager will restart the database manager if the database manager goes
offline unexpectedly.
For our purpose, we select option 3. Note that the failover policy is a powerful concept for larger clusters
(with more nodes and more partitions), but for a simple two-node single partition setup (such as this one),
it is generally best to select option 3.
The following are the available failover policies:
Copyright © 2004, 2009 IBM
Page 56 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
1. Local Restart -- during failover, the database manager will restart in place on the local
machine
2. Round Robin -- during failover, the database manager will restart on any machine in the
cluster domain
3. Active/Passive -- during failover, the database manager will restart on a specific machine
4. M+N -- during failover, the database partitions on one machine will failover to any other
machine in the cluster domain (used with DPF instances)
5. Custom -- during failover, the database manager will restart on a machine from a userspecified list
Enter your selection:
3
Then db2haicu will prompt you to designate any noncritical mount points. For this case, we chose to
designate only one noncritical mount point.
You can identify mount points that are noncritical for failover. For more information, see the topic
'Identifying mount points that are noncritical for failover' in the DB2 Information Center. Are
there any mount points that you want to designate as noncritical? [2]
1. Yes
2. No
2
Active/Passive failover policy was chosen. You need to specify the host names of an
active/passive pair.
Enter the host name for the active cluster node:
cmaix26
Enter the host name for the passive cluster node:
cmaix25
Note that you should add any mount points to the noncritical path list that you are sure that you never
want to failover. This list should include any mount points specified in /etc/fstab that are local mount
points and will never be failed over.
Adding DB2 database partition 0 to the cluster ...
Adding DB2 database partition 0 to the cluster was successful.
The db2haicu tool will now automatically add the DB2 partition instance to the specified cluster manager
at this point.
Virtual IP Address Setup
Once the database partition has been added to the cluster, db2haicu will prompt you to create a virtual IP
address:
Do you want to configure a virtual IP address for the DB2 partition: 0? [2]
1. Yes
2. No
1
Enter the virtual IP address:
9.123.112.252
Enter the subnet mask for the virtual IP address 9.123.112.252: [255.255.255.0]
Copyright © 2004, 2009 IBM
Page 57 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
255.255.255.0
Select the network for the virtual IP 9.123.112.252:
1. db2_private_network_0
2. db2_public_network_0
Enter selection:
2
Adding virtual IP address 9.123.112.252 to the domain ...
Adding virtual IP address 9.123.112.252 to the domain was successful.
You must make sure that your IP address and subnet mask values are well formed and correspond with
the subnet mask of the network you chose. All invalid inputs will be rejected. In such a case, you are
advised to examine the IP addresses and netmasks of the NIC components of the network (using the
‘ifconfig’ command) and verify that the IP address and netmask specified are compatible with each of the
NICs in the network. In addition, make sure that the IP address that you want to add is not already present
on the network.
After the virtual IP address configuration, the Automated Cluster-controlled HA configuration is
complete.
All cluster configurations have been completed successfully. db2haicu exiting ...
DB2 and Tivoli System Automation environment acceptance testing
At this point, you should take some time to validate your DB2 and Tivoli System Automation operating
environment before proceeding any further. Devise tests which will mimic the way you expect to utilize
DB2 and Tivoli System Automation in the context of Content Manager. For example, if you follow the
methodology of this white paper, you may wish to create a sample database in each of your two DB2
instances, testing various failover scenarios. Does the failover occur as you expect it to, with MTTR
consistent with your SLA’s or internal user requirements? Can you failback as you expect to? What
considerations might be noteworthy as a result of a failover or failback? Are there manual steps that will
need to be performed before users can regain access to the DB2 database server?
Develop test scenarios for situations you think might be problematic. Develop as thorough a procedure as
possible and be sure you understand all ramifications. Ultimately, the capability of Content Manager to
remain highly available depends on the underlying robustness of the infrastructure implemented. If DB2
and Tivoli System Automation are not configured in a way that meets your requirements, it is unlikely
that Content Manager sitting on top of this highly available environment, will be able to either. Do not
underestimate the importance of testing at this stage.
Ensure your DB2 and Tivoli System Automation environment are configured properly before proceeding
further. IBM standard support does not include troubleshooting or advising on physical storage, network
topologies, or other non-IBM high availability infrastructure components. For questions or problems with
IBM-branded products used in your highly available infrastructure which are not part of Content Manager,
you must contact IBM support specific to those IBM products. For example, if you suspect problems with
or have questions about DB2 or Tivoli System Automation failover performance or capabilities, you
should direct questions to IBM support for DB2 or Tivoli System Automation.
The information below may suggest considerations in devising your DB2 and Tivoli System Automation
acceptance tests. Additionally, you should refer to both DB2 and Tivoli System Automation product
documentation and white papers for additional testing methodologies. Acceptance testing is highly
correlated to the specific requirements domain of an organization and cannot be generalized. As such
Copyright © 2004, 2009 IBM
Page 58 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
testing relates to the underlying infrastructure and are not Content Manager-related, it lies outside the
scope of IBM standard support. If you need additional consultancy services, please contact IBM Global
Business Services or your IBM sales representative.
Verify with db2pd Utility
The db2pd tool is used for troubleshooting because it can return quick and immediate information from
the DB2 memory sets. With “-ha” parameter, it dumps out the overall cluster state, and in doing so
collects information for all cluster objects which should exist and returns the current state. For more
information on the db2pd utility, please see the DB2 Information Center topic, Monitoring and
troubleshooting using db2pd and db2pd - Monitor and troubleshoot DB2 database command.
Here is sample after running db2pd with db2inst1:
$ db2pd -ha
DB2 HA Status
Instance Information:
Instance Name
Number Of Domains
Number Of RGs for instance
= db2inst1
=1
=1
Domain Information:
Domain Name
Cluster Version
Cluster State
Number of nodes
= DB2Domain
= 2.4.10.0
= Online
=2
Node Information:
Node Name
--------------------cmaix25
cmaix26
State
------------------Online
Online
Resource Group Information:
Resource Group Name
= db2_db2inst1_0-rg
Resource Group LockState
= Unlocked
Resource Group OpState
= Online
Resource Group Nominal OpState = Online
Number of Group Resources
=3
Number of Allowed Nodes
=2
Allowed Nodes
------------cmaix25
cmaix26
Member Resource Information:
Resource Name
= db2mnt-DB2INST1-rs
Resource State
= Online
Resource Type
= Mount
Mount Resource Path
= /DB2INST1
Number of Allowed Nodes
=2
Allowed Nodes
-------------
Copyright © 2004, 2009 IBM
Page 59 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
cmaix25
cmaix26
Resource Name
Resource State
Resource Type
DB2 Partition Number
Number of Allowed Nodes
Allowed Nodes
------------cmaix25
cmaix26
= db2_db2inst1_0-rs
= Online
= DB2 Partition
=0
=2
Resource Name
Resource State
Resource Type
= db2ip_9_123_112_251-rs
= Online
= IP
Network Information:
Network Name
----------------------db2_public_network_0
Number of Adapters
-----------------2
Node Name
----------------------cmaix25.cn.ibm.com
cmaix26.cn.ibm.com
Network Name
----------------------db2_private_network_0
Adapter Name
-----------------en0
en0
Number of Adapters
-----------------2
Node Name
----------------------cmaix25.cn.ibm.com
cmaix26.cn.ibm.com
Adapter Name
-----------------en1
en1
Quorum Information:
Quorum Name
Quorum State
------------------------------------------------------db2_Quorum_Network_9_123_112_1:19_9_50
Online
Fail
Offline
Operator
Offline
And for db2inst2 user, exported result will be:
$ db2pd -ha
DB2 HA Status
Instance Information:
Instance Name
Number Of Domains
Number Of RGs for instance
Copyright © 2004, 2009 IBM
= db2inst2
=1
=1
Page 60 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
Domain Information:
Domain Name
Cluster Version
Cluster State
Number of nodes
Node Information:
Node Name
--------------------cmaix26
cmaix25
2009.12.28
= DB2Domain
= 2.4.10.0
= Online
=2
State
------------------Online
Online
Resource Group Information:
Resource Group Name
= db2_db2inst2_0-rg
Resource Group LockState
= Unlocked
Resource Group OpState
= Online
Resource Group Nominal OpState = Online
Number of Group Resources
=3
Number of Allowed Nodes
=2
Allowed Nodes
------------cmaix26
cmaix25
Member Resource Information:
Resource Name
= db2mnt-DB2INST2-rs
Resource State
= Online
Resource Type
= Mount
Mount Resource Path
= /DB2INST2
Number of Allowed Nodes
=2
Allowed Nodes
------------cmaix26
cmaix25
Resource Name
Resource State
Resource Type
DB2 Partition Number
Number of Allowed Nodes
Allowed Nodes
------------cmaix26
cmaix25
= db2_db2inst2_0-rs
= Online
= DB2 Partition
=0
=2
Resource Name
Resource State
Resource Type
= db2ip_9_123_112_252-rs
= Online
= IP
Network Information:
Network Name
-----------------------
Copyright © 2004, 2009 IBM
Number of Adapters
------------------
Page 61 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
db2_public_network_0
Node Name
----------------------cmaix25.cn.ibm.com
cmaix26.cn.ibm.com
Network Name
----------------------db2_private_network_0
Node Name
----------------------cmaix25.cn.ibm.com
cmaix26.cn.ibm.com
2009.12.28
2
Adapter Name
-----------------en0
en0
Number of Adapters
-----------------2
Adapter Name
-----------------en1
en1
Quorum Information:
Quorum Name
Quorum State
------------------------------------------------------Operator
Offline
Fail
Offline
db2_Quorum_Network_9_123_112_1:19_9_50
Online
Verify environment during runtime
Throughout the runtime of the db2 engine, any time a call is required to the backend HA layer, an implicit
validation of the object model takes place. For example, any time a new disk mount is added (e.g. new
tablespace) the HA mount resource is created in the cluster and validated.
In a clustered environment, some database manager instance configuration and administration operations
require related cluster configuration changes. The DB2 High Availability (HA) Feature enables the
database manager to automatically request cluster manager configuration changes whenever you perform
certain database manager instance configuration and administration operations.
When you perform the following database manager instance configuration and administration operations,
the database manager automatically performs related cluster manager configuration for you:
Starting a database using START DATABASE or db2start
Stopping a database using STOP DATABASE or db2stop
Creating a database using CREATE DATABASE
Adding storage using CREATE TABLESPACE
Removing storage using ALTER TABLESPACE DROP or DROP TABLESPACE
Dropping a database using DROP TABLESPACE
Restoring a database using the RESTORE DATABASE or db2Restore
Specifying the table space containers for redirected restore using SET TABLESPACE
CONTAINERS
Rolling a database forward using ROLLFORWARD DATABASE or db2Rollforward
Recovering a database using RECOVER DATABASE or db2Recover
Creating event monitors using CREATE EVENT MONITOR
Dropping event monitors using DROP EVENT MONITOR
Creating and altering external routines using:
CREATE PROCEDURE
CREATE FUNCTION
Copyright © 2004, 2009 IBM
Page 62 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
CREATE FUNCTION
CREATE METHOD
ALTER PROCEDURE
ALTER FUNCTION
ALTER METHOD
Dropping external routines using:
DROP PROCEDURE
DROP FUNCTION
DROP METHOD
Start DB2 High Availability Disaster Recovery (HADR) operations for a database using START
HADR
Stop HADR operations for a database using STOP HADR
Cause a HADR standby database to take over as a HADR primary database using TAKEOVER
HADR
Setting the database manager configuration parameter DIAGPATH or SPM_LOG_PATH
Setting the database configuration parameter NEWLOGPATH, OVERFLOWLOGPATH,
MIRRORLOGPATH, or FAILARCHPATH
Dropping a database manager instance using db2idrop
Verify with db2haicu utility in Maintenance Mode
When you run DB2 High Availability Instance Configuration Utility (db2haicu) and there is already a
cluster domain created for the current database manager instance, db2haicu operates in maintenance mode.
When db2haicu is running in maintenance mode, db2haicu presents you with a list of configuration and
administration tasks that you can perform. This may be helpful if your acceptance testing uncovers errors
in your DB2 and Tivoli System Automation high availability infrastructure configuration.
db2haicu maintenance tasks include adding cluster elements such as databases or cluster nodes to the
cluster domain, and removing elements from the cluster domain. db2haicu maintenance tasks also include
modifying the details of cluster domain elements such as the failover policy for the database manager
instance.
For more information about the db2haicu utility, please refer to the DB2 Information Center topic,
Maintaining a cluster domain using DB2 High Availability Instance Configuration Utility (db2haicu).
$ db2haicu
Welcome to the DB2 High Availability Instance Configuration Utility (db2haicu).
You can find detailed diagnostic information in the DB2 server diagnostic log file called
db2diag.log. Also, you can use the utility called db2pd to query the status of the cluster domains
you create.
For more information about configuring your clustered environment using db2haicu, see the topic
called 'DB2 High Availability Instance Configuration Utility (db2haicu)' in the DB2 Information
Center.
db2haicu determined the current DB2 database manager instance is db2inst1. The cluster
configuration that follows will apply to this instance.
db2haicu is collecting information on your current setup. This step may take some time as
db2haicu will need to activate all databases for the instance to discover all paths ...
When you use db2haicu to configure your clustered environment, you create cluster domains. For
Copyright © 2004, 2009 IBM
Page 63 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
more information, see the topic 'Creating a cluster domain with db2haicu' in the DB2 Information
Center. db2haicu is searching the current machine for an existing active cluster domain ...
db2haicu found a cluster domain called DB2Domain on this machine. The cluster configuration
that follows will apply to this domain.
Select an administrative task by number from the list below:
1. Add or remove cluster nodes.
2. Add or remove a network interface.
3. Add or remove a highly available database.
4. Add or remove a mount point.
5. Add or remove an IP address.
6. Add or remove a non-critical path.
7. Move DB2 database partitions and HADR databases for scheduled maintenance.
8. Change failover policy for this instance.
9. Create a new quorum device for the domain.
10. Destroy the domain.
11. Exit.
Enter your selection:
11
All cluster configurations have been completed successfully. db2haicu exiting ...
Verifying the Tivoli System Automation environment
Tivoli System Automation monitoring tools are available via command-line or GUI interfaces such as the
Integrated Solutions Console. Refer to the Tivoli software Information Center topic, End-to-End
Automation Management Administrator's and User's Guide.
Copyright © 2004, 2009 IBM
Page 64 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
IBM WebSphere Application Server
In this chapter, we will install IBM WebSphere Application Server Network Deployment 6.1. WebSphere
Application Server Network Deployment is the edition of WebSphere Application Server designed to
provide high availability and load balancing features. We chose to implement a WebSphere Application
Server Network Deployment environment consisting of:
2x IBM HTTP Server, each with WebSphere Application Server Network Deployment plug-in
2x Application Server, running as a WebSphere Application Server Network Deployment cluster
1x WebSphere Application Server Network Deployment Deployment Manager
The redundant components (IBM HTTP Server, WebSphere Application Server plug-in, Application
Server) will be distributed across both nodes. The Deployment Manager node will not be a highly
available configuration because it is only needed for administration.
Install WebSphere Application Server Network Deployment
In this first step, we will install the application server component of IBM WebSphere Application Server
Network Deployment Version 6.1. You have to complete these steps on all nodes of your planned
WebSphere Application Server Network Deployment cluster.
Start the WebSphere Application Server Network Deployment 6.1.0.0 installation:
# cd /nfs/software/was/WAS
# ./install
● Welcome Screen → Next
● License Agreement → Review the license agreement. If you agree, select “I agree...” → Next
● System Prerequisite Check: This check fails on our servers, as shown in Figure 33: WebSphere
Application Server Network Deployment Prerequisite Check. We will ignore this error.
Figure 33: WebSphere Application Server Network Deployment Prerequisite Check
Copyright © 2004, 2009 IBM
Page 65 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Click Next.
● Install Sample Applications: We do not want to use the sample applications, so keep the selection
disabled. Click Next.
● Installation Directory: Confirm the default installation directory of /usr/IBM/WebSphere/AppServer
→ Next
● WebSphere Application Server Environments: Select None as shown in Figure 34: WebSphere
Application Server Network Deployment Environments. We will create the profiles later using the
profile management tool.
Figure 34: WebSphere Application Server Network Deployment Environments
Click Next.
● A warning as shown in Figure 35: WebSphere Application Server Profile Warning appears. Confirm
the message by clicking on Yes.
Figure 35: WebSphere Application Server Profile Warning
● The summary is displayed. → Next
● Installation Result: Uncheck the option to create a new profile. We will run this script later manually.
Click Finish to close the installation wizard.
Copyright © 2004, 2009 IBM
Page 66 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
IBM HTTP Server and IBM HTTP Server Plug-in Installation
In this step, we will install the IBM HTTP Server component and the IBM HTTP Server Plug-in of IBM
WebSphere Application Server Version 6.1. You have to complete these steps on all nodes of your
planned WebSphere Application Server Network Deployment cluster.
Start the IBM HTTP Server 6.1.0.0 installation:
# cd /nfs/software/was/IHS
# ./install
● On the welcome screen, click Next.
● Review the license Agreement. If you agree, select Accept. → Next
● A system prerequisite check will be completed. The result will be failed as shown in Figure 36: IBM
HTTP Server System Prerequisites Check.
Figure 36: IBM HTTP Server System Prerequisites Check
We will ignore this error. → Next
● Installation Directory: Confirm the default installation directory of /usr/IBM/HTTPServer
→ Next
● Keep the default port value assignments of 80 and 8008 as shown in Figure 37: IBM HTTP Server
Port Values Assignment.
Copyright © 2004, 2009 IBM
Page 67 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 37: IBM HTTP Server Port Values Assignment
Click Next.
● On the next panel, HTTP Administration Server Authentication: Provide a user ID and a password if
necessary, for example wasadmin and its password. Click Next.
● On the panel “Setup HTTP Administration Server” uncheck both options as shown in Figure 38: IBM
HTTP Server Setup HTTP Administration Server. We will not use the administration interface.
Figure 38: IBM HTTP Server Setup HTTP Administration Server
Click Next.
Copyright © 2004, 2009 IBM
Page 68 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
● The next panel asks for details about the IBM HTTP Server Plug-in for WebSphere Application
Server. On the first node, keep the default values as shown in Figure 39: IBM HTTP Server Plug-In
for WebSphere Application Server
Figure 39: IBM HTTP Server Plug-In for WebSphere Application Server
On the second node cmaix26, change the web server definition to “webserver2”. Click Next.
● The summary is displayed, as shown in Figure 40: IBM HTTP Server Installation Summary.
Figure 40: IBM HTTP Server Installation Summary
Click Next.
● Installation Result → Finish
Copyright © 2004, 2009 IBM
Page 69 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Applying Fixpacks
Next, we will update the WebSphere Application Server and Edge components to a higher fixpack level.
We used fixpack 13 for our tests. You will need to download 4 components for WebSphere fixpack
installation:
● IBM Update Installer
● Fix pack for WebSphere Application Server
● Fix pack for IBM HTTP Server
● Fix pack for IBM HTTP Server Plug-In
First, install the update installer (please notice that update installer should be the version delivered with
the fixpack, in our tests, fixpack 13):
● Welcome → Next
● License Agreement → Review the agreement. If you agree, select Accept. → Next
● System Prerequisite Check: Completed Failed → Next
● Installation Directory: Confirm /usr/IBM/WebSphere/UpdateInstaller → Next
● Installation Summary → Next
● Installation Complete: Keep checkbox selected as shown in Figure 41: Update Installer Installation
Result. → Finish
Figure 41: Update Installer Installation Result
The IBM Update Installer is started and will appear on your screen after a few seconds.
● Welcome → Next
Copyright © 2004, 2009 IBM
Page 70 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
● Product Selection: In the drop down list, you will find 3 components:
HTTP Server
HTTP Plug-in
WebSphere Application Server
Select one of the components. Click Next.
● Maintenance Option Selection: Install maintenance package → Next
● Specify Directory → Select the directory where you did download the patches. Click Next.
● Select packages to install as shown in Figure 42: Update Available Maintenance Packages → Next
Figure 42: Update Available Maintenance Packages
● Summary → Next
● Installation Complete: Relaunch the wizard for the other 2 components. If you did update all 3
components, click Finish.
WebSphere Application Server Profile Management Tool
Now we will create the necessary profiles in WebSphere Application Server. On one node, we will create
the deployment manager to serve as the administrative interface for all our WebSphere Application Server
Network Deployment cell clusters. In our scenario, the deployment manager will be created on cmaix26.
On all nodes we will create a custom node, which we will federate into our deployment manager
environment. A custom node does not contain any administrative interface.
Start the profile management tool on the node, where you want to create the deployment manager.
Complete these steps on one node only!
# cd /usr/IBM/WebSphere/AppServer/bin/ProfileManagement
# ./pmt.sh
Copyright © 2004, 2009 IBM
Page 71 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
● Welcome → Next
● Select the type of environment to create: Select Deployment Manager as shown in Figure 43: Profile
Management Tool - Environments.
Figure 43: Profile Management Tool - Environments
● Profile Creation Options: Keep typical. → Next
● Administrative Security (shown in Figure 44: Profile Management Tool - Administrative Security).
Specify a user name and its password to access any WebSphere Application Server Network
Deployment administrative function. This user is maintained within WebSphere and is by default not
an operating system user.
Copyright © 2004, 2009 IBM
Page 72 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 44: Profile Management Tool - Administrative Security
● Profile Creation Summary: Review the summary as shown in Figure 45: Profile Management Tool Deployment Manager Summary.
Figure 45: Profile Management Tool - Deployment Manager Summary
Copyright © 2004, 2009 IBM
Page 73 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Click Create.
● Profile created successful. Uncheck Launch the First Step Console. Click Finish to close the profile
management tool.
Now, start the Deployment Manager from a terminal:
# cd /usr/IBM/WebSphere/AppServer/bin
# ./startManager.sh
Start the profile management tool one node.
Complete these steps on all nodes you want to participate in the WebSphere Application Server Network
Deployment cluster!
# cd /opt/IBM/WebSphere/AppServer/bin/ProfileManagement
# ./pmt.sh
● Welcome → Next
● Select the type of environment to create: Select “Custom profile” → Next
● Profile Creation Options: Keep typical. → Next
● Federation: On this panel, you define if the custom node should become managed by a deployment
manager. Provide the deployment manager host name. Keep the default port number. If you have
enabled administrative security for the deployment manager, specify the user name and its password
as shown in Figure 46: Profile Management Tool - Federation.
Figure 46: Profile Management Tool - Federation
→ Next
Copyright © 2004, 2009 IBM
Page 74 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
● Profile Creation Summary is displayed. Review the settings, as shown in Figure 47: Profile
Management Tool - Custom Profile Summary.
Figure 47: Profile Management Tool - Custom Profile Summary
→ Create
● Profile created successful. Uncheck Launch the First Step Console. Finish.
Repeat the custom profile creation on all nodes you want to participate in your WebSphere Application
Server Network Deployment cluster.
Create WebSphere Application Server Cluster
In this step, we will create a cluster between both of our application server nodes. This configuration can
be accomplished in the WebSphere administration interface, Integrated Solutions Console.
● Expand Servers in the left navigation panel.
● Click on Clusters.
● In the right panel, “Server Clusters”, click New. This will start the Create a new cluster wizard,
which you will complete in the next steps.
● First, you need to provide a name for your cluster. You need to remember this name later on for the
deployment of the resource manager application of Content Manager. Type in the name of your
cluster (for example, RMCluster). Keep the other settings on default, as shown in Figure 48:
WebSphere Application Server Create a new cluster (1). Click Next.
Copyright © 2004, 2009 IBM
Page 75 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 48: WebSphere Application Server Create a new cluster (1)
● On the next panel, create first cluster member, you are required to provide information about your
first member and on a template. You can choose the member name freely. In our case, as shown in
Figure 49: WebSphere Application Server Create a new cluster (2), we've chosen a member name as
cmaix25, indicating that this application server is assigned to the node cmaix25Node01. In the drop
down list below, select the appropriate node. Keep the default weight of 2. With the weight parameter,
you can influence the load balancing behavior. Having an equal weight across all nodes will provide
an equal balancing.
Copyright © 2004, 2009 IBM
Page 76 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 49: WebSphere Application Server Create a new cluster (2)
For “basis for first cluster” member, keep the default selection of “Create the member using an
application server template” - default. Click Next.
● On this panel, as shown in Figure 50: WebSphere Application Server - Create a new cluster (3), you
can add more nodes to the cluster. Type in the member name (for example cmaix26), select the node
and weight, keep “generate unique HTTP ports” checked and click Add member. On the lower
section of this panel, a list of all nodes will be populated. When you are done defining all nodes, click
Next.
Copyright © 2004, 2009 IBM
Page 77 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 50: WebSphere Application Server - Create a new cluster (3)
● The next panel, as shown in Figure 51: WebSphere Application Server - Create a new cluster
(Summary), is a summary of your settings. Click Finish to start the creation of the new cluster.
Copyright © 2004, 2009 IBM
Page 78 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 51: WebSphere Application Server - Create a new cluster (Summary)
● Finally you have to save your settings. A warning like shown in Figure 52: WebSphere Application
Server - Save Changes is displayed. Click on Save to store the configuration and to populate the
information to the nodes across your WebSphere Application Server Network Deployment cluster.
Copyright © 2004, 2009 IBM
Page 79 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 52: WebSphere Application Server - Save Changes
Create WebSphere Application Server JMS data source
The WebSphere Administration API uses Java Messaging Service (JMS) for application event
notification when certain device and job events occur. By default, the Device Manager server uses the
Service Integration Bus (“SIB”) feature, which was introduced in WebSphere Application Server, version
6, along with the default messaging JMS provider. Refer to the WebSphere Application Server
Information Center for information on the Service Integration Bus and messaging resources.
To prevent the SIB database from being a SPOF in an HA environment, this database can be created in
the HA-enabled DB2 instance of either db2inst1 or db2inst2. Since the Content Manager resource
manager database typically has a lighter runtime work load than the Content Manager library server
database, we suggest creating the SIB database on db2inst2.
● Create an empty database on db2inst2 for JMS datasource. This database will be configured
automatically during Content Manager resource manager application configuration.
db2inst2@cmaix26 # db2 create database SIBDB
● In Security → Security administration, applications, and infrastructure, click Java Authentication
and Authorization Services → J2C authentication data as Figure 53: Secure administration,
application, and infrastructure, then select ‘New’ to create a new J2C Authentication.
Copyright © 2004, 2009 IBM
Page 80 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 53: Secure administration, application, and infrastructure
● For improved security, we suggest creating another DB2 user for this database. In our tests, for
simplicity, we re-use db2inst2 to manage this database as in Figure 54: New J2C user. Then click
“OK” and “Save”.
Figure 54: New J2C user
● In Resources → JDBC →JDBC Provides, select Scope to be “Cluster=RMCluster”, and click
“New”.
Copyright © 2004, 2009 IBM
Page 81 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
● For Figure 55: Create new JDBC provider, select Database type to be “DB2”, Provider type to be
“DB2 Universal JDBC Driver Provider” and Implementation type “Connection pool data source”.
Click “Next”.
Figure 55: Create new JDBC provider
● Use $IBMCMROOT/lib as the database class path as Figure 56: Database class path information, and
click “Next”.
Copyright © 2004, 2009 IBM
Page 82 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 56: Database class path information
● Check summary and click “Finish” and then “Save”.
● In Resources → JDBC →Data sources, select Scope to be “Cluster=RMCluster”, and click “New”.
The WebSphere Administration API uses Java Messaging Service (JMS) for application event
notification when certain device and job events occur. By default, the Device Manager server uses the
Service Integration Bus (“SIB”) feature, which was introduced in WebSphere Application Server, version
6, along with the default messaging JMS provider. Refer to the WebSphere Application Server
Information Center for information on the Service Integration Bus and messaging resources.
Content Manager requires a messaging database in place, to allow the resource manager to exchange
information between both nodes for the resource manager services workload: replicator, migrator, purger,
and stager. To prevent the SIB database from being a SPOF in an HA environment, we want to place
this information in our DB2 database environment.
● In Figure 57: Create basic data source information, input SIBDB_datasource as “Data source name”,
jdbc/sibdb as ‘JNDI name”, and select SIBDB_User for “Component-managed authentication alias
and XA recovery authentication alias”. Please remember this JNDI name; we will use it later during
Content Manager resource manager application configuration. Click “Next”.
Copyright © 2004, 2009 IBM
Page 83 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 57: Create basic data source information
● For Figure 58: Select JDBC provider, choose “Select an existing JDBC provider”, and choose “DB2
Universal JDBC driver Provider” that we created in previous step. Click “Next”.
Figure 58: Select JDBC provider
Copyright © 2004, 2009 IBM
Page 84 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
● In Figure 59: Create a data source, input SIBDB for “Database name”, and select Driver type to be 4.
Use Virtual IP and port of db2inst2 for server name and port number. Remove “Use this data source
in container managed persistence (CMP)”, then “Next”.
Figure 59: Create a data source
● View the summary and click “Finish”, then “Save”.
● In Environment →WebSphere Variables as Figure 60: WebSphere Variables, input
$IBMCMROOT/lib as DB2UNIVERSAL_JDBC_DRIVER_PATH for all cluster and nodes.
Copyright © 2004, 2009 IBM
Page 85 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 60: WebSphere Variables
● System administration →Node agents, select both node agent and restart them as Figure 61: Restart
Node Agents.
Figure 61: Restart Node Agents
● Same with Figure 62: Nodes Synchronize, In System administration →Nodes, select
cmaix25Node01 and cmaix26Node02, select “Synchronize”.
Copyright © 2004, 2009 IBM
Page 86 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 62: Nodes Synchronize
● Now we can test connection for SIBDB that we created in datasource as Figure 63: Test connection
for SIBDB datasource. In Resource → JDBC → Data sources, select SIBDB_datasource, and click
“Test connection”. It should display successful in messages.
Figure 63: Test connection for SIBDB datasource
Add Web Server Definition
In this step, we will add our web server definition into our deployment manager. While still in the
Integrated Solutions Console, navigate in the left panel by expanding Servers → Web Servers. Click on
new. A panel as shown in Figure 64: New Web Server Definition (1) appears.
Select your first node from the drop down list and type in the web server name. The first node is
cmaix25Node01 and the web server name is webserver1 (In our case, it’s webserver25). Second node will
be cmaix26Node01 with a web server name of webserver2 (In our case, it’s webserver26).
Copyright © 2004, 2009 IBM
Page 87 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 64: New Web Server Definition (1)
On the next panel, shown in Figure 65: New Web Server Definition (2), you specify the web server
template. We use the IBM HTTP Server (IHS).
Figure 65: New Web Server Definition (2)
The next panel, shown in Figure 66: New Web Server Definition (3), queries information about the web
server installation location and its plug-in. In our case, all settings have been correct.
Copyright © 2004, 2009 IBM
Page 88 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 66: New Web Server Definition (3)
In the last step, shown in Figure 67: New Web Server Definition (4), a summary is displayed. Review the
settings and click finish to create the definitions. Now review the same steps for the second IBM HTTP
Server you have installed.
Figure 67: New Web Server Definition (4)
Finally, you must save your changes. On the upper part of you screen, you should see the information
shown in Figure 68: New Web Server Definition – Save Changes. Click on the Save link embedded in
this message.
Copyright © 2004, 2009 IBM
Page 89 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 68: New Web Server Definition – Save Changes
Finally, you can start the IBM HTTP Server from the WebSphere Application Server Integrated Solution
Console. After repeating previous steps to set another webserver definition, select both web servers
(check the two check boxes) and then click the Start button. As a result, both servers should be started and
a panel similar to Figure 69: Web Server - Start will be displayed. Notice, that the status should now be
started (as indicated by a green icon, instead of the red one as shown in the figure below) for both servers
and a success message will be displayed above the table.
Figure 69: Web Server - Start
SSL configuration for http server (Optional)
By default, the web servers that we installed only support http with port 80. Content Manager needs https
support in some cases. As in Content Manager 8.4.2, https is only used for System Administration
management by default. This will be optional and will not be described in this document. You can use
default https port on any node or define IP sprayer to map to default https port directly.
For the purposes of testing only, we chose to bypass the use of https. Instead, we will change the port
number for https communication to bypass the IBM HTTP Server and talk directly to WebSphere
Application Server. Note that we do not recommend this for production.
Copyright © 2004, 2009 IBM
Page 90 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Open the Content Manager System Administration Client and expand Resource Manager. Right click on
our resource manager “icmrm” and select properties. Change the port number for the https protocol to the
one used by your WebSphere Application Server, most likely 9443. See Figure 70: Update Resource
Manager- Resource Manager Properties.
Figure 70: Update Resource Manager- Resource Manager Properties
The drawback from this implementation is that, for administration, no load balancing will take place and
no high availability is configured. Since Content Manager administration is typically less frequent, this
might be acceptable.
IP Sprayer Definition
As we have two web servers, we must define an IP sprayer to use both of them. Please see your IBM
WebSphere Application Server representative for the IP sprayer solution. We’ll not talk technology here,
and just assume that IP address of cmrmapp is already assigned to this IP sprayer.
WebSphere Application Server and WebSphere Application Server
Network Deployment environment acceptance testing
As before with the databases, you should take some time to validate your WebSphere Application Server
and WebSphere Application Server Network Deployment cluster environment. Devise tests which will
Copyright © 2004, 2009 IBM
Page 91 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
mimic the way you expect to utilize the WebSphere cluster in the context of Content Manager. For
example, if you follow the methodology of this white paper, you may wish to deploy a sample application
to your WebSphere Application Server Network Deployment cell, testing various failover scenarios. Does
the failover occur as you expect it to, with MTTR consistent with your SLA’s or internal user
requirements? Can you failback as you expect to? What considerations might be noteworthy as a result of
a failover or failback? Are there manual steps that will need to be performed before users can regain
access to the application? What about the IBM HTTP Server? If it is configured as a SPOF, what
additional recovery procedures do you need to develop? If it isn’t, what redundancies, and checks on
those redundancies, have you put into place to mitigate outages?
Develop test scenarios for situations you think might be problematic. Develop as thorough a procedure as
possible and be sure you understand all ramifications. As with the databases, the capability of Content
Manager to remain highly available depends on the underlying robustness of the infrastructure
implemented. If WebSphere is not configured in a way that meets your requirements, it is unlikely that
the Content Manager RM application sitting on top of the WebSphere environment, will be able to either.
Do not underestimate the importance of testing at this stage.
Ensure your WebSphere Application Server and WebSphere Application Server Network Deployment
environment are configured properly before proceeding further. IBM standard support does not include
troubleshooting or advising on physical storage, network topologies, or other non-IBM high availability
infrastructure components. For questions or problems with IBM-branded products used in your highly
available infrastructure which are not part of Content Manager, you must contact IBM support specific to
those IBM products. For example, if you suspect problems with or have questions about WebSphere
Application Server Network Deployment failover performance or capabilities, you should direct questions
to IBM support for WebSphere Application Server Network Deployment.
Refer also to the WebSphere Application Server Information Center for more details on WebSphere
Application Server and WebSphere Application Server Network Deployment. In particular, the topics,
Monitoring overall system health and Using the installation verification tools may prove helpful in
diagnosing problems.
Acceptance testing is highly correlated to the specific requirements domain of an organization
and cannot be generalized. As such testing relates to the underlying infrastructure and are not
Content Manager-related, it lies outside the scope of IBM standard support. If you need
additional consultancy services, please contact IBM Global Business Services or your IBM sales
representative.
Copyright © 2004, 2009 IBM
Page 92 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Content Manager
In this chapter we will install and configure Content Manager. Steps to complete are:
● installation of Content Manager on both nodes
● Configure Content Manager library server database on Node A
● Configure Content Manager resource manager database on Node B
● Deploy Resource Manager Application on WebSphere Application Server Network Deployment
Cluster
Further we will configure a remote client.
Pre-Install Validation
Before continuing further, ensure that you have performed thorough acceptance testing for your
infrastructure at this time. This should include DB2, Tivoli System Automation, and WebSphere
Application Server, and the rest of your HA infrastructure. As mentioned in previous sections, it is the
infrastructure itself that provides HA capabilities. If failover at the infrastructure level is not behaving
as you expect, Content Manager will not be able to either. The risk of inadequate acceptance testing is
that Content Manager servers are not highly available, do not conform to your MTTR requirements,
and/or result in unanticipated dissynchronization points within your infrastructure that require additional
manual recovery.
Installation
The installation of Content Manager is goes as follows. Unzip the installation source and run the
command install_CM as root. You will complete these steps on both of two nodes. Then you can
configure different component on different node, as described in the architecture.
After executing install_CM, the installation wizard will appear. Read the license agreement and select
“Accept” if you agree to it. Click Next.
The installation destination is displayed. Note, that on UNIX environments, the installation target is
predefined as /opt/IBM/db2cmv8. Click Next.
The product component list is displayed. Select the components you want to install. We kept the default
selection of all components. Mandatory components are the library server, resource manager database the
resource manager application. Optional components are the system administration client and the
information center. Click Next.
On the next panel, you need to select the server type. Select DB2 and click Next.
The summary panel is displayed, similar to our as shown in Figure 71: Content Manager Installation
Summary. Verify the information displayed in the summary and click Next to start the installation.
Copyright © 2004, 2009 IBM
Page 93 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 71: Content Manager Installation Summary
After a short time, the installation should complete. You are asked, if you want to run the configuration
wizard. De-select this option, as we are not ready to complete this step yet.
Initial Configuration for Library Server Database
In this step, we will use the Content Manager Configuration wizard to complete our setup on Library
Server database. Run config_CM in /opt/IBM/db2cmv8/bin directory.
After a few seconds, the Content Manager configuration wizard appears. On the first panel, as shown
in Figure 72: Configuration Options, select “Add configuration”.
Copyright © 2004, 2009 IBM
Page 94 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 72: Configuration Options
Note that you can start the configuration wizard several times. You can also use the configuration wizard
to remove existing configuration.
On the next panel, like the one shown in Figure 73: Component Selection, you can select the
components you want to configure. You may select components individually or you may just select
all components at once.
Figure 73: Component Selection
Copyright © 2004, 2009 IBM
Page 95 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Since we want to configure the library server and the system administration client, we kept only Library
server and System administration client selected.
Now you need to select the database type in Figure 74: Database Type. We want to use DB2 as the
database.
Figure 74: Database Type
Next, you are required to provide information about an administrative user in Figure 75:
Administrative User. This user is required to store the log and configuration files of Content
Manager and is also used during the installation process.
Copyright © 2004, 2009 IBM
Page 96 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 75: Administrative User
We have already created this user and the associated group, so we provide this information as shown
above.
In the next panel, as shown in Figure 76: System Information, you need to provide the fully qualified
host name. This value is normally detected automatically and prepopulated into this field. Also, we
can specify cmlsdb.cn.ibm.com for LS hostname.
Figure 76: System Information
Copyright © 2004, 2009 IBM
Page 97 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
The next panel, shown in Figure 77: DB2 Database Product Directory, ask about DB2 product
directory. For DB2 v9.5, default installation directory is /opt/IBM/db2/V9.5.
Figure 77: DB2 Database Product Directory
On the next panel, as shown in Figure 78: Library Server Database, you need to select Create new
database as we only have DB2 instance configured.
Figure 78: Library Server Database
The information you provide on the next panel, shown in Figure 79: Library Server Database
Information, is used to connect to the library server database at install time.
Copyright © 2004, 2009 IBM
Page 98 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 79: Library Server Database Information
Note, icmadmin user is already created, and database should use port 50001 as we defined.
The next panel, shown in Figure 80: DB2 Database instance, will detect the instance name to use. As
db2inst1 is created on cmaix25 and HA configuration is completed, we will just use it.
Figure 80: DB2 Database instance
The next screen, shown is Figure 81: Library Server Database, you need to provide database
connection user ID and password. This user, unlike the administrative user defined before, does not
have any special privileges on the database.
Copyright © 2004, 2009 IBM
Page 99 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 81: Library Server Database
We can use predefined database user icmconct user to finish this step.
The next screen, shown in Figure 82: Library Server Database Settings, allows you to define
additional settings for the library server. Keep the library server ID as 1. You may change the
transaction duration, which is by default 3 minutes. At the moment, we will keep default selections,
all features can be enabled after installation, if necessary.
Figure 82: Library Server Database Settings
Copyright © 2004, 2009 IBM
Page 100 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
The next panel, Figure 83: Library Server Connection, prompts you for the resource manager
database information, we use cmrmdb.cn.ibm.com as hostname, which is mapped to RM database
virtual IP.
Figure 83: Library Server Connection
The next panel, in Figure 84: Library Server Connection (cont.), collects the information for
Resource Manager application in Websphere Application Server. We use cmrmapp.cn.ibm.com for
the hostname, and use 80 for Resource manager Web application port, 443 for Resource manager
secure Web application port. You should use IP sprayer hostname (‘cmrmapp’ in our sample) in
order to get connection to both two web servers and two Websphere Application Servers.
Copyright © 2004, 2009 IBM
Page 101 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 84: Library Server Connection (cont.)
The next few panels, starting with Figure 85: System Administration Client, collect information for
client connection, as for example used by the Content Manager System Administration Client. On
this first panel, keep the default selection of local. This select stored configuration information
locally to /home/ibmcmadm/cmgmt/connectors. Alternatively you could place this information
remotely on an HTTP or LDAP server.
Figure 85: System Administration Client
Copyright © 2004, 2009 IBM
Page 102 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 86: System Administration Client Connection prompts for information of library server. We
use icmnlsdb for database name and cmlsdb.cn.ibm.com as LS hostname.
Figure 86: System Administration Client Connection
Now we have provided all necessary information for cmaix25 and Figure 87: Start Configuration is
displayed. When you click on Next, the configuration settings are applied, the LS database will be
created.
Figure 87: Start Configuration
Copyright © 2004, 2009 IBM
Page 103 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
The configuration will run for several minutes. First, it will create necessary table for the library
server, and next create the access modules. All actions are tracked into the configuration log file.
Figure 88: Content Manager Configuration Complete shows the expected end of the configuration
phase. Click finish to close the wizard.
Figure 88: Content Manager Configuration Complete
After configuration finish on first node cmaix25, we need to deploy the necessary code of Content
Manager onto the second node cmaix26, we need copy mandatory files after installation. Make sure that
you preserve ownership and access permissions during the copy operation! You need to copy the
following directories to the node cmaix26:
/opt/IBM/db2cmv8/lib/lib32/ICMNLSSP
/opt/IBM/db2cmv8/lib/lib64/ICMNLSSP
/opt/IBM/db2cmv8/lib/ICMNLSSP
/opt/IBM/db2cmv8/lib/ICMNLSUF
/opt/IBM/db2cmv8/lib/ICMNWFSP
/etc/rc.cmlsproc
Note: Any time you apply a fixpack to the primary node, you MUST copy these files again to all other
nodes in your cluster! Content Manager is NOT aware of this manual copy!
On the second node cmaix26, you must also manually create the directories used by Content Manager for
its log files as ibmcmadm user. If this directory does not exists, no log file will be written on your
secondary node.
In the commands below, replace ICMNLSDB with the name of your library server database.
# su - ibmcmadm
$ mkdir -p log/ls/ICMNLSDB
$ chmod -R 777 log
Copyright © 2004, 2009 IBM
Page 104 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Alternatively you may copy the whole directory tree structure, starting at /home/ibmcmadm/log, from
your primary node to your secondary node.
Initial Configuration for Resource Manager
On the other node, cmaix26, we will setup Resource Manager Component only. Now we run config_CM
on /opt/IBM/db2cmv8/bin to start Content Manager Configuration wizard as shown in Figure 89:
Configure .
Figure 89: Configure Content Manager
We select Resource manager database and Resource manager application in Figure 90: Product
components.
Copyright © 2004, 2009 IBM
Page 105 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 90: Product components
Same with steps on Library Server database configuration, we select DB2 in Figure 91: Server
Database Type.
Figure 91: Server Database Type
And also use existing user ibmcmadm in Figure 92: Administrative user.
Copyright © 2004, 2009 IBM
Page 106 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 92: Administrative user
For hostname in Figure 93: System Information, we use cmrmdb.cn.ibm.com or Resource Manager
Database virtual IP.
Figure 93: System Information
Also, use default DB2 product directory in Figure 94: DB2 Database Product Directory.
Copyright © 2004, 2009 IBM
Page 107 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 94: DB2 Database Product Directory
In Figure 95: Resource Manager Database, select Create new database.
Figure 95: Resource Manager Database
Also, input Resource manager database name in Figure 96: Resource Manager Database Information.
We used predefined user rmadmin and port 50002 for this step.
Copyright © 2004, 2009 IBM
Page 108 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 96: Resource Manager Database Information
DB2 database instance db2inst2 should be used in Figure 97: DB2 Database instance.
Figure 97: DB2 Database instance
Next, you are prompted for information about your WebSphere Application Server environment.
You have to specify the installation directory and the profile home directory. Since we are using a
network deployment configuration, the profile home directory is the directory of the deployment
manager. Typical values are shown in Figure 98: Resource Manager Application. If you’ve enabled
Copyright © 2004, 2009 IBM
Page 109 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
application server security during the installation of WebSphere Application Server, also check the
check box.
Figure 98: Resource Manager Application
Next, you are prompted to complete information about the resource manager application, as shown in
Figure 99: Resource Manager Application (cont). Most of the time you should leave the application
name and application context as its default values. Changes are only required if you want to deploy
multiple resource manager applications on the same server. Also keep the “Specify the JDBC path
for the WebSphere Application Server variable” checked.
Copyright © 2004, 2009 IBM
Page 110 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 99: Resource Manager Application (cont)
The next panel will prompt you for this variable value. Specify the directory to the Content Manager
JAVA directory. The value shown in Figure 100: WebSphere Variables fits our Content Manager
installation. You can change this value after the configuration completes in the WebSphere
Application Server administrative console.
Figure 100: WebSphere Variables
Copyright © 2004, 2009 IBM
Page 111 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Next, the Content Manager configuration wizard will connect to you WebSphere installation to
detect application server and web server information. This might take several seconds to complete
while it validates a large number of the parameters you have specified on the previous panels.
The Content Manager configuration wizard detects, that you want to have a WebSphere Application
Server Network Deployment environment and prompts if we want to deploy the resource manager
application ingot a cluster or a stand-alone application server. Select cluster, as shown in Figure 101:
Resource Manager Application Deployment Target.
Figure 101: Resource Manager Application Deployment Target
The next panel, shown in Figure 102: Cluster Information for the Resource Manager Application,
lists all detected WebSphere Application Server clusters. Select to cluster you’ve created for the
resource manager application. The second field queries for the messaging database. We can input the
value as we defined in WebSphere Application Server for jdbc/sibdb.
Copyright © 2004, 2009 IBM
Page 112 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 102: Cluster Information for the Resource Manager Application
Next, you are prompted to complete connection information for the Resource Manager application.
Here you have several options. For the host name, we suggest you use the virtual alias of your
IP/HTTP sprayer. For the HTTP port, keep 80. This will use the IBM HTTP server and its load
balancing features embedded in the WebSphere Application Server plug-in. The same applies for the
HTTPS port: for now, keep the default port of 443, or change it to other port that we used to route to
9043 port. Figure 103: HTTP Connection Information for the Resource Manager Application shows
typical values.
Figure 103: HTTP Connection Information for the Resource Manager Application
Copyright © 2004, 2009 IBM
Page 113 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 104: Resource Manager Application Connections need information of Library server. Give LS
virtual hostname and port 50001.
Figure 104: Resource Manager Application Connections
The input Library Server administration ID and password in Figure 105: Resource Manager
Application Connection (cont)
Figure 105: Resource Manager Application Connection (cont)
Now we can wait until Configuration finish as in Figure 106: Content Manager Configuration
Complete.
Copyright © 2004, 2009 IBM
Page 114 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 106: Content Manager Configuration Complete
WebSphere Data source pretest configuration
This is additional Configuration for HA consideration. After Content Manager resource manager
application configuration, We can see two additional data sources named icmrm_LS_database and
icmrm_database. Because Resource Manager will get connection from WebSphere Connection pool, and
won’t check connection validation for performance consideration. User may meet many failed
transactions during Database fail over. To eliminate this issue, we can update Data source properties.
In Resources → JDBC → Data sources as Figure 107: Data source properties, select
icmrm_LSdatabase as in . Select “WebSphere Application Server data source properties”.
Figure 107: Data source properties
Copyright © 2004, 2009 IBM
Page 115 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
In Figure 108: Pretest Connection Properties, enable “Pretest existing pooled connections”, and give a
non-zero value for “Retry interval”. Also enable “Pretest new connections”.
Figure 108: Pretest Connection Properties
Also, repeat these steps for another data source: icmrm_database.
Important: These options have performance impact, and we recommend setting them only when you have
to reduce transaction error during fail over.
Client Reroute
The client reroute feature allows a DB2 client application to recover from a lost database connection in
case of a network failure. In the HA configuration discussed in this paper, we use a virtual IP address that
allows clients to connect to the active node. Identify the IP address that will be created and failed over as
part of the HA failover. In addition, identify the port number for the instance TCP/IP listener by checking
the value of the SVCENAME DBM CFG parameter.
Issue the following command on the node hosting the instance directory to configure the virtual IP
address for client reroute:
root@cmaix25 # su - db2inst1
$ db2 update alternate server for database \
icmnlsdb using hostname 9.123.112.251 port 50001
In this example, 9.26.124.22 is the virtual IP address and port 55445 is the value used in the DBM CFG
parameter ‘SVCENAME’.
Copyright © 2004, 2009 IBM
Page 116 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Failover Time Tuning
On AIX, DB2 uses the RSCT sub-component for heartbeat detection. You may choose to modify RSCT
communication group parameters to improve DB2 failover time. Be aware, however, that shorter DB2
failover times may result in longer application response times. You should modify these settings only
after careful testing that balances end-user response requirements with MTTR requirements.
List all the Communication Groups in the Cluster:
root@cmaix25 # lscomg
Name Sensitivity Period Priority Broadcast SourceRouting NIMPathName NIMParameters Grace
CG2 4
1
1
Yes
Yes
-1 (Default)
CG1 4
1
1
Yes
Yes
-1 (Default)
Get detailed information about each Communications Group:
root@cmaix25 # lscomg -i CG1
Name NodeName
IPAddress
Subnet
SubnetMask
en0 cmaix26.cn.ibm.com 9.123.112.200 9.123.112.0
255.255.255.0
en0 cmaix25.cn.ibm.com 9.123.112.199 9.123.112.0
255.255.255.0
Check which nodes belong to which Communication Group:
root@cmaix25 # lsrsrc IBM.NetworkInterface Name CommGroup NodeNameList
Resource Persistent Attributes for IBM.NetworkInterface
resource 1:
Name
= "en1"
CommGroup
= "CG2"
NodeNameList = {"cmaix25.cn.ibm.com"}
resource 2:
Name
= "en0"
CommGroup
= "CG1"
NodeNameList = {"cmaix25.cn.ibm.com"}
NOTES:
A NIC/IP can belong to ONLY one Communications Group at a time per single Node - i.e. if there are
multiple NICs on a NIC in a Cluster, they will be placed in different Communication Groups.
Sensitivity: The number of missed heartbeats that constitute a failure
Period: The number of milliseconds between heartbeats
Ping Grace Period: The maximum number of milliseconds to wait for a heartbeat response. If no
response is received after the specific number of milliseconds, the count of missed heartbeats is
increased.
Change the Sensitivity of the Communication Group CG1
root@cmaix25 # chrsrc -s "Name like 'CG1'" IBM.CommunicationGroup Sensitivity=4
Change the Heartbeat Period of the Communication Group CG1
root@cmaix25 # chrsrc -s "Name like 'CG1'" IBM.CommunicationGroup PeriodMilliSec=200
Change the Ping Grace period of the Communication Group CG1
root@cmaix25 # chrsrc -s "Name like 'CG1'" IBM.CommunicationGroup
PingGracePeriodMilliSec=30000
Copyright © 2004, 2009 IBM
Page 117 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Modify the time to aquire quorum.
If you are using the IP quorum (the default), you can reduce the amount of time which Tivoli System
Automation waits to verify quorum acquisition via (9.123.112.1 is the IP address of the quorum that you
want to use. It can be the same as the existing one you already created with db2haicu).
root@cmaix25 # mkrsrc IBM.TieBreaker Type="EXEC" Name="mynetworktb1"
DeviceInfo="PATHNAME=/usr/sbin/rsct/bin/samtb_net Address=9.123.112.1 Log=1
PostReserveWaitTime=1 HeartbeatPeriod=2
root@cmaix25 # chrsrc -c IBM.PeerNode OpQuorumTieBreaker="mynetworktb1"
In using the specific values shown for the parameters discussed, we are adhering to the design choices we
made earlier, in Planning an HA environment for Content Manager, where we adopted the philosophy
that Content Manager server availability was critical, even over other considerations, since end users
cannot access content without access to the Content Manager servers. Similarly, critical to Content
Manager server availability is that Content Manager server databases must themselves be available. This
was our guiding principle in designing our specific test environment. Your design and planning may
dictate other choices for availability parameters such as those above.
In our testing, the default settings caused failover times to increase as our overall test workload increased
against Content Manager servers. The specific settings for the parameters discussed above allowed us to
retain failover times of approximately one minute for the Content Manager server databases. Again, you
should understand the benefits versus tradeoffs before making any changes to your RSCT settings.
Validation and Test
In this chapter, we describe how you can validate that all components in the configuration are working
and that load is actually balanced across the components. Further we discuss how the different Content
Manager components will react to outages of other components and how to recover from such situations.
Validating Active Components
In this chapter, we will review how to identify active components in our Content Manager environment.
It's a collection of tools, scripts and log files which are frequently used for analysis.
Node Application and Database (Library Server, Resource Manager
Database)
The Node application and database is in our case a combination of:
db2inst1 Resource Group (Active on cmaix25)
db2isnt2 Resource Group (Active on cmaix26)
Note that to display the state of the cluster, you can issue the Tivoli System Automation command
‘lssam’ or the ‘db2pd -ha’ command. Issue the ‘lssam’ command to see the resources created during this
process as Figure 109: Tivoli System Automation status for db2inst1 and db2inst2.
root@cmaix25 # su – db2inst1
$ lssam
Copyright © 2004, 2009 IBM
Page 118 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 109: Tivoli System Automation status for db2inst1 and db2inst2
Application Server (Resource Manager)
In the described configuration, a clustered IBM WebSphere Application Server configuration is used. It
consists of:
One deployment manager (only installed on node cmaix26)
Two node agents (one on cmaix25 and one on cmaix26), which act as helpers for the deployment
managers
Two application servers
The deployment manager is only used for administrative tasks and provides the Integrated Solutions
Console to administrate all components of WebSphere Application Server, including the HTTP server. To
start the deployment manager, run following command on node cmaix26:
# /usr/IBM/WebSphere/AppServer/bin/startManager.sh
ADMU0116I: Tool information is being logged in file
/ usr /IBM/WebSphere/AppServer/profiles/Dmgr01/logs/dmgr/startServer.log
ADMU0128I: Starting tool with the Dmgr01 profile
ADMU3100I: Reading configuration for server: dmgr
ADMU3200I: Server launched. Waiting for initialization status.
ADMU3000I: Server dmgr open for e-business; process id is 40576
The deployment manager is not required at runtime for the Content Manager resource manager
component.
The same applies for the node agents, which are helper applications for the deployment managers and
perform functions such as configuration updates and remote start and stop operations. To start one of the
node agents on both nodes:
# / usr /IBM/WebSphere/AppServer/profiles/Custom01/bin/startNode.sh
ADMU0116I: Tool information is being logged in file
/ usr /IBM/WebSphere/AppServer/profiles/Custom01/logs/nodeagent/startServer.log
ADMU0128I: Starting tool with the Custom01 profile
Copyright © 2004, 2009 IBM
Page 119 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
ADMU3100I: Reading configuration for server: nodeagent
ADMU3200I: Server launched. Waiting for initialization status.
ADMU3000I: Server nodeagent open for e-business; process id is 40533
The only mandatory components for Content Manager at runtime are the two application servers. In
earlier chapters, these servers have been started from the WebSphere Application Server Integrated
Solutions Console. These servers can also be started on a command line, here an example for the server
on cmaix25:
# / usr /IBM/WebSphere/AppServer/profiles/Custom01/bin/startServer.sh cmaix25
ADMU0116I: Tool information is being logged in file
/ usr /IBM/WebSphere/AppServer/profiles/Custom01/logs/ cmaix25/startServer.log
ADMU0128I: Starting tool with the Custom01 profile
ADMU3100I: Reading configuration for server: cmaix25
ADMU3200I: Server launched. Waiting for initialization status.
ADMU3000I: Server cmaix25 open for e-business; process id is 41765
To verify the distribution of incoming requests, access logging can be used. To enable the access log,
open the WebSphere Application Server Integrated Solutions Console. Navigate to Servers →
Application Servers → RMCMAIX25/RMCMAIX26 → Web Container Settings → Web container
transport chains → WCInboundDefault → HTTP Inbound Channel (HTTP 2). Enable the checkbox
“Enable access and error logging” and click “apply”.
Then click on “HTTP error and NCSA access logging” and select the check box “Enable service at server
startup”. Click “apply” and save your changes. Apply the same changes for the other application server,
and then restart the cluster. Every incoming request is now added to the
$SERVER_LOG_ROOT/http_access.log.
HTTP Server (Resource Manager)
An IBM HTTP Server is used to route and load-balance incoming client requests via HTTP to the
WebSphere Application Server. These two components are connected with the IBM HTTP Server PlugIn. Normally, the load between the two application servers is load-balanced in a round-robin fashion. If
one of the application servers becomes unavailable, all load is redirected to the available one.
To start the IBM HTTP Server, run the following command on both nodes:
# /usr/IBM/HTTPServer/bin/apachectl start
The HTTP Server logs all requests into an activity log file, by default into
/usr/IBM/HTTPServer/logs/access_log. This will provide an overview about all incoming requests. The
details which are captured can be configured in the httpd.conf file.
Note: The configuration discussed in this white paper installs and configures two active HTTP Servers.
However, only one is actually used in our test environment. To eliminate this single point of failure, we
recommend the configuration of an IP sprayer in front of both HTTP servers. Content Manager library
server configuration data for the Content Manager resource manager can be set to reflect the IP sprayer
address instead of individual WebSphere Application Server servers. Depending on the IP sprayer chosen,
requests to the RM application may be load balanced across the HTTP servers and automatically redirected in the event an HTTP server fails.
Copyright © 2004, 2009 IBM
Page 120 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Content Manager Component Failures
This chapter summarizes what happens if a certain component of the Content Manager stack fails. Most
of the time, the recovery will be automatic and frequently unnoticeable for the end user. However,
operational procedures should be in place, making sure that corrective actions are taken.
Database (Library Server, Resource Manager Database)
The database is probably the most important components of Content Manager, as all other parts do have a
dependency on it. The database is configured in a DB2 with Tivoli System Automation environment,
using one active instance and one standby instance on two servers. If one of the servers becomes
unavailable, Tivoli System Automation will detect this error and start the DB2 instance on another node.
Also, client requests will be moved to the active DB2 instance. Complete failover generally takes from
seconds to minutes but depends on several factors including overall workload against the Content
Manager servers, db2haicu settings, and RSCT settings.
In case a connection is already established to the failing DB2 instance, this connection will be
disconnected. If a transaction is in progress on the failing instance, this transaction will be rolled back.
Other transactions will continue to report an error, or hang until another node starts-up.
The following is a list of components that access the database. The list also contains a description of the
behavior when the component loses an active connection to a failing DB2 instance. Common across all
components is an error condition (rollback) when a transaction is in progress on the failing instance.
Application using Content Manager Java API (eClient, Content Manager System Administration
Client, WEBi, custom applications): The connection to the server is lost. An exception will be
thrown. Reconnect can be implemented silently in the application but it will fail until another node
startup. If no automatic reconnect is implemented, the user must log-in again.
Application using Content Manager C++ API (custom applications): The connection to the server is
lost. An exception will be thrown. Reconnect can be implemented silently in the application but it
will fail until another node startup. If no automatic reconnect is implemented, the user must log-in
again.
IBM® DB2® Content Manager Client for Windows: The Content Manager Client for Windows
automatically drops a connection after 30 seconds of inactivity. A new connection will be seamlessly
established after this period. If the user initiates a request before the 30 seconds timeout, an error
message will be displayed.
Content Manager resource manager: An error message will be captured in the log of the resource
manager application server. Also, the first request to the resource manager may fail will a database
connection error and the transaction will be rolled back. Subsequent requests will work normally.
This behavior applies to the resource manager servlet as well as to the services such as migrator,
replicator etc.
Application Server (Resource Manager)
Request to the two application servers in the WebSphere Application Server Network Deployment cluster
are balanced by the IBM HTTP Server plug-in. If one application server becomes unavailable, subsequent
requests will be routed to the remaining active servers.
Copyright © 2004, 2009 IBM
Page 121 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
HTTP Server (Resource Manager)
The HTTP Server, if configured like described in this document, is a single point of failure. If the HTTP
Server becomes unavailable, access to the resource manager for client requests become impossible. All
transactions, which include the resource manager, will fail.
To mitigate this risk, we recommend configuring an IP sprayer in front of the HTTP Server. This
document already included all steps to setup a second HTTP Server, only the routing of the requests to the
second HTTP Server is missing.
Copyright © 2004, 2009 IBM
Page 122 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Appendix
List of Tables
Table 1: Logical Drive definitions on DS4200 ................................................................................... 21
Table 2: IP Configuration ................................................................................................................... 24
Table 3: Operating system groups and users....................................................................................... 30
List of Figures
Figure 1: Overall Scenario (2 Node Configuration) ........................................................................... 19
Figure 2: DS4200 Configuration......................................................................................................... 22
Figure 3: IBM N3700 Volume Definition .......................................................................................... 23
Figure 4: DB2 setup wizard ................................................................................................................ 32
Figure 5: Install New DB2 product..................................................................................................... 33
Figure 6: Welcome window................................................................................................................ 33
Figure 7: Accept License .................................................................................................................... 34
Figure 8: Installation Type.................................................................................................................. 34
Figure 9: Response file ....................................................................................................................... 35
Figure 10: Select the features to Install............................................................................................... 35
Figure 11: Select the langurages to Install .......................................................................................... 36
Figure 12: Specify the location of DB2 Information Center............................................................... 36
Figure 13: Install the IBM SA MP Component .................................................................................. 37
Figure 14: Set user information for the DB2 Administration Server.................................................. 38
Figure 15: Set up a DB2 instance........................................................................................................ 38
Figure 16: Set up notification.............................................................................................................. 39
Figure 17: Summary ........................................................................................................................... 39
Figure 18: DB2 instance Setup wizard ............................................................................................... 41
Figure 19: Set up a DB2 instance........................................................................................................ 41
Figure 20: Set up partitioning options for the DB2 instance .............................................................. 42
Figure 21: Set user information for the DB2 instance owner ............................................................. 42
Figure 22: Set user Information for the fenced user ........................................................................... 43
Figure 23: Configure instance communication and startup ................................................................ 43
Figure 24: Prepare the DB2 tools catalog ........................................................................................... 44
Figure 25: Specify a contact for health monitor notification .............................................................. 44
Figure 26: Configure the DB2 Text Search server.............................................................................. 45
Figure 27: Summary ........................................................................................................................... 45
Figure 28: Set user information for the DB2 instance owner ............................................................. 46
Figure 29: Set user information for the fenced user............................................................................ 47
Figure 30: Configure instance communication and startup ................................................................ 47
Figure 31: Summary ........................................................................................................................... 48
Figure 32: db2inst1 Tivoli System Automation status........................................................................ 55
Figure 33: WebSphere Application Server Network Deployment Prerequisite Check ...................... 65
Figure 34: WebSphere Application Server Network Deployment Environments .............................. 66
Figure 35: WebSphere Application Server Profile Warning .............................................................. 66
Figure 36: IBM HTTP Server System Prerequisites Check................................................................ 67
Figure 37: IBM HTTP Server Port Values Assignment ..................................................................... 68
Figure 38: IBM HTTP Server Setup HTTP Administration Server.................................................... 68
Copyright © 2004, 2009 IBM
Page 123 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 39: IBM HTTP Server Plug-In for WebSphere Application Server........................................ 69
Figure 40: IBM HTTP Server Installation Summary.......................................................................... 69
Figure 41: Update Installer Installation Result ................................................................................... 70
Figure 42: Update Available Maintenance Packages.......................................................................... 71
Figure 43: Profile Management Tool - Environments ........................................................................ 72
Figure 44: Profile Management Tool - Administrative Security ........................................................ 73
Figure 45: Profile Management Tool - Deployment Manager Summary ........................................... 73
Figure 46: Profile Management Tool - Federation ............................................................................. 74
Figure 47: Profile Management Tool - Custom Profile Summary...................................................... 75
Figure 48: WebSphere Application Server Create a new cluster (1) .................................................. 76
Figure 49: WebSphere Application Server Create a new cluster (2) .................................................. 77
Figure 50: WebSphere Application Server - Create a new cluster (3)................................................ 78
Figure 51: WebSphere Application Server - Create a new cluster (Summary) .................................. 79
Figure 52: WebSphere Application Server - Save Changes ............................................................... 80
Figure 53: Secure administration, application, and infrastructure ...................................................... 81
Figure 54: New J2C user..................................................................................................................... 81
Figure 55: Create new JDBC provider................................................................................................ 82
Figure 56: Database class path information ........................................................................................ 83
Figure 57: Create basic data source information................................................................................. 84
Figure 58: Select JDBC provider ........................................................................................................ 84
Figure 59: Create a data source........................................................................................................... 85
Figure 60: WebSphere Variables ........................................................................................................ 86
Figure 61: Restart Node Agents.......................................................................................................... 86
Figure 62: Nodes Synchronize............................................................................................................ 87
Figure 63: Test connection for SIBDB datasource ............................................................................. 87
Figure 64: New Web Server Definition (1) ........................................................................................ 88
Figure 65: New Web Server Definition (2) ........................................................................................ 88
Figure 66: New Web Server Definition (3) ........................................................................................ 89
Figure 67: New Web Server Definition (4) ........................................................................................ 89
Figure 68: New Web Server Definition – Save Changes.................................................................... 90
Figure 69: Web Server - Start ............................................................................................................. 90
Figure 70: Update Resource Manager- Resource Manager Properties ............................................... 91
Figure 71: Content Manager Installation Summary............................................................................ 94
Figure 72: Configuration Options ....................................................................................................... 95
Figure 73: Component Selection......................................................................................................... 95
Figure 74: Database Type ................................................................................................................... 96
Figure 75: Administrative User........................................................................................................... 97
Figure 76: System Information ........................................................................................................... 97
Figure 77: DB2 Database Product Directory ...................................................................................... 98
Figure 78: Library Server Database .................................................................................................... 98
Figure 79: Library Server Database Information ................................................................................ 99
Figure 80: DB2 Database instance...................................................................................................... 99
Figure 81: Library Server Database .................................................................................................. 100
Figure 82: Library Server Database Settings .................................................................................... 100
Figure 83: Library Server Connection .............................................................................................. 101
Figure 84: Library Server Connection (cont.)................................................................................... 102
Figure 85: System Administration Client ......................................................................................... 102
Figure 86: System Administration Client Connection ...................................................................... 103
Figure 87: Start Configuration .......................................................................................................... 103
Figure 88: Content Manager Configuration Complete ..................................................................... 104
Figure 89: Configure Content Manager ............................................................................................ 105
Copyright © 2004, 2009 IBM
Page 124 of 125
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX
2009.12.28
Figure 90: Product components ........................................................................................................ 106
Figure 91: Server Database Type...................................................................................................... 106
Figure 92: Administrative user ......................................................................................................... 107
Figure 93: System Information ......................................................................................................... 107
Figure 94: DB2 Database Product Directory .................................................................................... 108
Figure 95: Resource Manager Database ........................................................................................... 108
Figure 96: Resource Manager Database Information ....................................................................... 109
Figure 97: DB2 Database instance.................................................................................................... 109
Figure 98: Resource Manager Application ....................................................................................... 110
Figure 99: Resource Manager Application (cont)............................................................................. 111
Figure 100: WebSphere Variables .................................................................................................... 111
Figure 101: Resource Manager Application Deployment Target ..................................................... 112
Figure 102: Cluster Information for the Resource Manager Application ......................................... 113
Figure 103: HTTP Connection Information for the Resource Manager Application ....................... 113
Figure 104: Resource Manager Application Connections ................................................................ 114
Figure 105: Resource Manager Application Connection (cont) ....................................................... 114
Figure 106: Content Manager Configuration Complete ................................................................... 115
Figure 107: Data source properties ................................................................................................... 115
Figure 108: Pretest Connection Properties........................................................................................ 116
Figure 109: Tivoli System Automation status for db2inst1 and db2inst2......................................... 119
Copyright © 2004, 2009 IBM
Page 125 of 125