Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 IBM Content Manager V8.4.2 High Availability configuration using IBM DB2 V9.5 with IBM Tivoli System Automation and IBM WebSphere Application Server V6.1 with IBM WebSphere Application Server Network Deployment on IBM AIX V5.3 Bin Kong Lead HA/DR Tester, System Verification Test IBM China Development Lab, Beijing, China Steve Soria Lead Architect for IBM Content Manager HA/DR solutions IBM Silicon Valley Lab, San Jose, California, USA Copyright IBM (2004, 2009). This information contained within this document is the property of IBM. The above notice and this paragraph must be included on all copies of this document that are made. Copyright © 2004, 2009 IBM Page 1 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Table of Contents Special Notices....................................................................................................................................5 About this Document ..........................................................................................................................6 Preface ................................................................................................................................................7 IBM’s Caveats Regarding Highly Available Environments .......................................................8 Audience and Prerequisite Skills ................................................................................................9 What We Hope You will Learn ..................................................................................................9 Comments, Critiques, and Criticism .........................................................................................10 Design Considerations for a Highly Available topology ..................................................................11 Planning an HA environment for Content Manager .................................................................14 Content Manager product prerequisites for HA environments .................................................17 Scenario Description.........................................................................................................................18 Two Nodes Scenario .................................................................................................................18 Hardware Setup.................................................................................................................................20 Servers ......................................................................................................................................20 Storage ......................................................................................................................................20 Internal Disk (SAS)...........................................................................................................20 Fiber Channel Attached Storage (IBM DS4200) ..............................................................20 Network Attached Storage (IBM N3700) .........................................................................22 Network.....................................................................................................................................23 Operating System Preparation ..........................................................................................................25 Virtual I/O Server .....................................................................................................................25 Base AIX Installation................................................................................................................25 Prerequisite Software ................................................................................................................25 Storage Configuration ...............................................................................................................26 Logical Volume and Mount Points ...........................................................................................27 File System Configuration on Node 1...............................................................................27 File System Configuration on Node 2...............................................................................28 Preparation for DB2 ..........................................................................................................29 Users and Groups......................................................................................................................29 NFS Configuration....................................................................................................................30 DB2 Database with Tivoli System Automation................................................................................32 Pre-Install Validation ................................................................................................................32 Software Installation .................................................................................................................32 License Validation ....................................................................................................................40 Setup Instance: db2inst1 ...........................................................................................................40 Setup Instance: db2inst2 ...........................................................................................................46 DB2 High Availability Instance Configuration Utility.....................................................................49 Cluster Preparation....................................................................................................................49 Time Synchronization...............................................................................................................49 The db2haicu Interactive Setup on Node1 ................................................................................50 Creating a Cluster Domain................................................................................................50 Copyright © 2004, 2009 IBM Page 2 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Quorum Configuration......................................................................................................51 Network Setup ..................................................................................................................51 Cluster Manager Selection ................................................................................................53 Failover Policy ..................................................................................................................54 Virtual IP Address Setup...................................................................................................55 The db2haicu Interactive Setup on Node2 ................................................................................55 Cluster Manager Selection ................................................................................................56 Failover Policy ..................................................................................................................56 Virtual IP Address Setup...................................................................................................57 DB2 and Tivoli System Automation environment acceptance testing .....................................58 Verify with db2pd Utility..................................................................................................59 Verify environment during runtime ..................................................................................62 Verify with db2haicu utility in Maintenance Mode..........................................................63 Verifying the Tivoli System Automation environment ....................................................64 IBM WebSphere Application Server ................................................................................................65 Install WebSphere Application Server Network Deployment ..................................................65 IBM HTTP Server and IBM HTTP Server Plug-in Installation ...............................................67 Applying Fixpacks ....................................................................................................................70 WebSphere Application Server Profile Management Tool ......................................................71 Create WebSphere Application Server Cluster.........................................................................75 Create WebSphere Application Server JMS data source..........................................................80 Add Web Server Definition ......................................................................................................87 SSL configuration for http server (Optional) ............................................................................90 IP Sprayer Definition ................................................................................................................91 WebSphere Application Server and WebSphere Application Server Network Deployment environment acceptance testing ................................................................................................91 Content Manager...............................................................................................................................93 Pre-Install Validation ................................................................................................................93 Installation.................................................................................................................................93 Initial Configuration for Library Server Database ....................................................................94 Initial Configuration for Resource Manager ...........................................................................105 WebSphere Data source pretest configuration........................................................................115 Client Reroute .........................................................................................................................116 Failover Time Tuning .............................................................................................................117 Validation and Test .........................................................................................................................118 Validating Active Components...............................................................................................118 Node Application and Database (Library Server, Resource Manager Database) ...........118 Application Server (Resource Manager).........................................................................119 HTTP Server (Resource Manager) .................................................................................120 Content Manager Component Failures ...................................................................................121 Database (Library Server, Resource Manager Database) ...............................................121 Application Server (Resource Manager).........................................................................121 HTTP Server (Resource Manager) .................................................................................122 Appendix.........................................................................................................................................123 List of Tables ..........................................................................................................................123 List of Figures .........................................................................................................................123 Copyright © 2004, 2009 IBM Page 3 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX Copyright © 2004, 2009 IBM 2009.12.28 Page 4 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Special Notices The information contained in this publication was derived under specific operating and environmental conditions and is distributed on an "as is" basis without any warranty either expressed or implied. IBM specifically disclaims any implied warranties of merchantability or fitness for any particular purpose. IBM assumes no responsibility for any errors that may appear in this document. The use of this information or the implementation of any of these techniques are the responsibility of the user and depends on the user's ability to evaluate and integrate them into the user's operational environment. While each item might have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Users attempting to adapt these techniques to their own environments do so at their own risk. The information contained in this document is subject to change without any notice. IBM reserves the right to make any such changes without obligation to notify any person of such revision or changes. IBM makes no commitment to keep the information contained herein up to date. References in this publication to IBM products, programs, or services do not imply that IBM intends to make these available in all countries in which IBM operates. Any reference to an IBM licensed program in this publication is not intended to state or imply that only IBM's program may be used. Any functionally equivalent program may be used instead. Equivalent product, program, or services that do not infringe any of the intellectual property rights of IBM may be used instead of the IBM product, program, or service. The evaluation and verification of operation in conjunction with other products, except those expressly designated by IBM, are the responsibility of the user. IBM may have patents or pending patent application covering the subject matter in this document. The furnishing of this document does not give you license to these patents. Any information about non-IBM ("vendor") products in this document has been supplied by the vendor and IBM assumes no responsibility for its accuracy or completeness. IBM, the IBM logo, ibm.com, AIX, DB2, Tivoli, and WebSphere are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml. Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Other company, product, or service names may be trademarks or service marks of others. © Copyright International Business Machines Corporation 2004, 2009. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Copyright © 2004, 2009 IBM Page 5 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 About this Document This white paper provides hands-on guidance for how to install and set up an IBM® DB2® Content Manager V8.4.2 system on IBM® AIX® using IBM® DB2® and IBM® Tivoli® Storage Automation and deploying an active-passive high availability environment. For the Content Manager resource manager (“RM”) web application, the document describes how to configure a managed cell with two nodes using IBM® WebSphere® Application Server Network Deployment and one cluster running two RM application server instances (cluster members) using IBM® WebSphere® Application Server. The topics covered in this white paper include the following: In-depth installation and configuration recommendations for the Content Manager library server (“LS”) and RM databases In-depth installation and configuration recommendations for the RM application running in a cluster with WebSphere Application Server , including how to use WebSphere Application Server Network Deployment clustering capabilities. Version Description Author(s) Date 0.1 Initial Draft Bin Kong 17.08.2009 0.2 First complete draft Steven Soria 16.11.2009 0.3 Review revisions Bin Kong 28.11.2009 0.4 Second complete draft Steven Soria 03.12.2009 0.5 Final Draft Bin Kong Steven Soria 28.12.2009 Special thanks to Arya Adeli, Khalid Alodhaibi, Holger Koenig, Konstantin Konson, Cataldo Mega, Stefan Momma, and Steve Raspudic for technical advice, comments, reviews etc. in many areas of this document. Copyright © 2004, 2009 IBM Page 6 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Preface This publication is intended to help content management solution designers understand how to enable highly available content management solutions on officially supported platforms using the IBM® DB2® Content Manager Version 8 Release 4 Modification Release 2 product components. It serves as a reference example on how to set up the IBM® DB2® Content Manager stack in a highly available configuration and outlines the relationships between the components in the stack. Because Content Manager components are a hybrid of databases, database applications, Web applications, and thick client applications, the reference configuration presented makes strong use of the platform/operating system-specific cluster support and the high availability (“HA”) capabilities built into the database and Web application server products. HA environments are designed to eliminate single points of failure (“POF”) throughout the application stack. In this white paper, we accomplish this by leveraging the HA capabilities and services in DB2, WebSphere Application Server, and the clustering software support provided by the underlying operating systems. Although the Content Manager high availability concepts discussed here may be applicable to many different operating systems, the actual Content Manager high availability environment that is implemented depends heavily on the individual hardware and software components used, thus making a direct comparison of the different operational scenarios impossible. Implementations will vary and some concepts or details of how we configured our environment may not apply to another specific environment or configuration. We strongly encourage anyone who intends to set up a high availability configuration with these or any other products to employ experienced, professionals who understand the technical complexities inherent in such environments. Setting up and maintaining any type of clustering environment is expensive, time consuming, complex, and very prone to error. With that said, it should be obvious that it is imperative to thoroughly test the overall setup. Setup of an HA environment requires a deep understanding of the various HA, storage, and networking technologies involved in addition to familiarity of the underlying applications. Clustering solutions cannot be considered an out-of-box solution. In fact, building and maintaining clustering solutions requires a range of skills, tasks, resources, and extensive testing in order to failover resources properly to alternate nodes. We strongly encourage anyone who intends to set up a high availability configuration with these or any other products to employ experienced professionals who understand the technical complexities inherent in such environments. Malfunctions of the HA environment can often be attributed to insufficient testing of the various resources that might become unavailable at some point in time. We recognize that there are many possible different configuration variants and that setting up such an environment correctly is complex. For existing Content Manager customers who wish to modify an existing Content Manager architecture to take advantage of the architecture described in this document, or for customers with requirements that are incompatible with the architecture described in this document, such services are not available as part of IBM standard product support. It is, therefore, highly recommended to use consulting services, such as IBM® Global Business Services®, for high availability planning, testing, implementation, maintenance, and support for your Content Manager product. IBM standard product support for HA configurations is strictly limited to the HA reference configuration and Content Manager product version described in this document, as well as the method of installing and configuring Content Manager on the HA reference configuration for the stated Content Manager product version. Although beyond the scope of this document, IBM standard product support also includes the application of maintenance releases (“fixpacks”) to the HA reference configuration and Content Manager product version described herein. The remainder of this document assumes you will be performing a clean Copyright © 2004, 2009 IBM Page 7 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 installation of Content Manager, in an environment that has no existing Content Manager installations or databases, whether partially installed, installed but not running, or running. Please be aware that IBM standard support does not include troubleshooting or advising on physical storage, network topologies, or other non-IBM high availability infrastructure components. For problems with IBM-branded products used in your highly available infrastructure which are not part of Content Manager, you must contact IBM support specific to those IBM products. IBM’s Caveats Regarding Highly Available Environments Clustering should be left to experts. This document is not intended to be a generic high availability support statement of system-specific cluster services. Content Manager is an enterprise-class application that customers might want to install and set up in a high availability configuration. Installing this kind of application in a clustered environment is not an out-of-the-box solution. Customers have the choice of installing Content Manager in a great number of configurations and, as such, IBM has not tested all potential configurations on all of the various hardware platforms given the considerable number of possible configurations of Content Manager in a high availability environment. IBM has published a number of white papers as reference material for those interested in high availability configurations. Each of these white papers targeted a different operating system and environment, with the main objective of each to describe a tested high availability implementation on these operating systems with detailed configuration procedures. These configurations might require a separate agreement with Global Business Services (or another IBM certified vendor certified to provide high availability solutions in a Content Manager context) for support. In general, proper design, installation, configuration, fail-over testing of Content Manager, and support for running Content Manager in an HA environment needs to be considered as a complex endeavor potentially prone to errors. Therefore, again, IBM advises the end-user and organization implementing an HA system to contract with an authorized consulting organization certified to provide support of high availability environments. IBM standard product support is strictly limited to the reference configuration and Content Manager product version described in this document, as well as the method of installing and configuring the reference configuration for the stated Content Manager product version. Although beyond the scope of this document, IBM standard product support also includes the application of maintenance releases (“fixpacks”) to the reference configuration and Content Manager product version described herein. For existing Content Manager customers who wish to modify an existing Content Manager architecture to take advantage of the architecture described in this document, or for customers with requirements that are incompatible with the architecture described in this document, such services are not available as part of IBM standard product support. It is, therefore, highly recommended to use consulting services, such as Global Business Services, for high availability planning, testing, implementation, maintenance, and support for your Content Manager product. Setting up any type of clustering environment is expensive, time consuming, complex, and very prone to error. With that said, it is imperative to thoroughly test the overall setup. We consider it ill advised for any entity to set up a clustering environment unless they already have a competent staff solely dedicated to testing and maintaining such an environment. HA environments require considerable investment in additional hardware, skilled people, and time to create a proper test environment that is separate from a production environment. Our experience has been that many entities attempt to set up a clustered environment, but fail to budget correctly for such projects. In other words, these companies try to implement HA environments without allocating a reasonable amount of resources. Copyright © 2004, 2009 IBM Page 8 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 We believe that, among the keys to success, are: Allocating appropriate amounts of resources when implementing a clustered environment can make for a more successful outcome. Working closely with an HA qualified vendor. The most common sources of failure are: Not testing the various potential scenarios before going into production: Production is the wrong time to find out your cluster does not work as expected. Failure to properly budget sufficient funds for an HA project, both capital & expense. Poor scheduling and rushed delivery time frame that prevents proper testing. Poor Change Control procedures within the organization. Failure of having skilled personnel competent with clustering technologies Not having a Test Lab. As a reminder, the information in this white paper is intended to complement, not to substitute the Content Manager product documentation provided by IBM. See the publications section of the IBM Programming Announcement for Content Manager for more information about what publications are considered to be product documentation. Audience and Prerequisite Skills Throughout this paper, it is presumed that the reader has a good understanding of the business requirements for high availability in an Content Manager environment. If available, operating systemspecific knowledge of all software components used for this solution will also help to better understand the technical details described. Given the complexity of the Content Manager product itself and the many prerequisite applications, it is almost impossible to document all variations of possible Content Manager high availability scenarios. Therefore, while we will refer at times to alternative configurations or choices we could have made in setting up our HA environment, we will primarily focus on details of the specific HA reference configuration we set up. We will also not try to fully describe how to configure the related and required co-products, but rather focus on details relevant to the configuration of the chosen Content Manager high availability environment. What We Hope You will Learn After you read this paper, we hope that you, through the example of how we set up our environment, understand: the basics of a design approach for implementing high availability using Content Manager and its co-products on AIX and DB2 in the specific environment we chose components in production how high availability is achieved using the Content Manager product components general concepts for planning for an HA environment general concepts for installing an HA environment Copyright © 2004, 2009 IBM Page 9 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Comments, Critiques, and Criticism Reader comments and critiques regarding this report are welcome. Please send your thoughts, comments, critique, or other feedback to [email protected]. Copyright © 2004, 2009 IBM Page 10 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Design Considerations for a Highly Available topology At the IBM Information On Demand 2009 conference, statistics were presented highlighting the tremendous explosion of content: 15 petabytes of data generated per day, 200 billion emails per day, and 80% of that from unstructured content. Significantly, the leading cause of this data explosion is no longer due to human-generated content, but rather machine-generated content. This has naturally led to huge investments in content management systems, transforming ever-increasing volumes of content into genuine assets. With the increasing need of businesses to leverage this information efficiently and effectively, one of the key challenges of an information-led transformation has been to implement costeffective measures that ensure this new asset, this content, remains available. But what does it mean to be “highly available”, and what does it mean in the context of an enterprise application such as Content Manager? The answers to these questions are highly dependent on the goals of the organization requiring a highly-available solution. Companies must have a clear understanding of what they expect to achieve from such configurations before embarking upon the implementation of a solution. For example, it may be acceptable for some to limit the definition of availability to end-users simply being able to access mission-critical systems, that the servers are “up” and responding to requests, even if the responses generate errors. For others, availability may additionally require that end-users be able to retrieve business data from mission-critical systems, even if the data is not necessarily the most recent version of that data. Or, perhaps availability means that end-users can not only access systems and retrieve data, but that that data must be accurate, current, and reliable. Availability may also mean that recovery is transparent to the end-user, perceived as “continuous availability”, or that the end-user may experience some down time but that recovery capability does exist, with the full capabilities of the system and its data restorable. The graph below illustrates the tradeoffs between cost to implement versus time to recover, representing the sophistication of the solution with respect to the evaluated criticality of business data and applications. The benefits of a highly available solution must be weighed against the expense, time to implementation, and personnel expertise in designing the solution. Certainly, these are not trivial considerations. Copyright © 2004, 2009 IBM Page 11 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Broadly, there are three approaches toward solving the availability problem. It is common to combine multiple methods to achieve higher levels of availability. Coordinated backup and recovery The general objective of a coordinated backup and recovery strategy is to ensure the system can holistically be restored such that all normal operations are resumed with no data loss, a state of consistency, within a recovery time that fulfills established mean-time-to-recovery (“MTTR”) objectives, and to a point-in-time closest to the time of failure. Complete backups must be taken for all file systems relevant to the applications and data that are mission-critical. If databases or offline storage, such as IBM® Tivoli® Storage Manager or shared storage, are utilized, they too will require a backup strategy. There are often many strategies for backup policies, with considerations including the use of operating system, vendor-provided, or third-party backup tools, the frequency and scheduling of backups that seek to balance end-user usage patterns with MTTR requirements, and the appropriate ratio of incremental to full backup versus storage costs required to archive the backups and compliance requirements dictating how long the backups must be kept. In cases where mission-critical business data resides on multiple machines and/or storage devices, coordination of backups is required to ensure consistency and reliability of systems and data. Duplicates of the backups may be stored in a different geography to ensure their survival in the event of some disaster at the primary site. A backup and recovery strategy is generally considered the “poor man’s high availability” due to its manual-intensive nature and lack of flexibility. This means it is likely to take a day or longer to fully recover from an outage. However, it provides low total-cost-of-ownership (TCO) because it can generally be implemented using existing hardware, operating system tools, and simple scripting. Disaster Recovery (“DR”) This availability strategy is intended to protect against failure at a site-wide or data center level. The idea is that a secondary site is standing by, ready to take over from the primary site in the event a natural or catastrophic disaster, such as an earthquake, fire, or flood, renders the primary inaccessible. Because such a disaster is assumed to occur over a large area, primary and standby sites are typically geographically disparate from one another. The secondary site is typically an exact duplicate of the primary, even in terms of hardware, number of servers, hardware capability, storage capacity, and so on. Storage-level Copyright © 2004, 2009 IBM Page 12 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 replication, cluster server replication, database replication, or other technologies provide data duplication at the secondary site. To ensure the secondary site is as closely matched as possible to the primary, hardware mirroring or software-based synchronous replication is generally preferred such that a write is not complete until the data is written to both the primary and secondary site. The drawback, of course, is that this extends response times due to the synchronicity of every operation. Thus, MTTR in a DR scenario often has an inverse relationship to performance. Observe also that DR focuses on mirroring or data duplication from one site to another; it does not generally deal with processes. In classical DR, when a failure occurs at the primary site, the secondary site must be manually transitioned and an integrity/data verification process completed. Some cluster service providers can perform the failover automatically via a heartbeat server that is remote from both the primary and standby site. Even in this scenario, however, there is often a manual verification process since the data on the standby site is not guaranteed to be insync with the primary. An individual business must assess at what point it is appropriate to restore services to end-users. Recovery time for a DR strategy is typically in terms of hours and is considered a high availability solution. Although typically an inter-site recovery strategy, it is common for DR strategies to be implemented within a single site as well, and thus it becomes more of a high availability solution than a true DR solution. High Availability (“HA”) Whereas DR focuses on site-wide disaster, an HA strategy deals with the reality that machines fail, even within a single site or data center. It deals with two primary requirements: that hardware failures are recovered in terms of minutes or even seconds, not hours or days, and that recovery is automatic and nearly transparent to end-users. In an HA configuration, multiple servers are ready to takeover in case another fails in what is known as a server cluster. Depending on the underlying technology, cluster members can be all active (“active-active” configurations), or one member can be active while all others are passive (“active-passive” configurations). Cluster members are often referred to as nodes and are monitored via heartbeat mechanisms between each node in the cluster. Although the exact logic varies, the general idea is that when a heartbeat is “missed”, that node is marked as unavailable and incoming requests are automatically re-routed to other available nodes. This is accomplished through floater IP addresses (“virtual IP’s”) than the clustering software automatically remaps to the physical IP’s of other nodes. Additionally, HA configurations make use of shared storage, with identical mount points on each node. Applications are installed at identical locations on each node along with operating system user equivalence. Mission-critical data is placed on the shared storage device to allow all nodes to access the same copy of data. This allows for very fast failover time because everything is “ready to go” on each node: there is no data integrity issue between nodes because it’s on shared storage and there is no need to reconfigure applications because common mount points, virtual IP’s, and operating system user equivalence make this transparent. Typically, in-flight operations (those transactions which were executing during a node failure) return an error and need to be retried. Depending on the underlying technology and if all nodes are active, a load balancing can strategy can be put in place to improve scalability and hardware utilization. Observe that cluster nodes can be physically separate machines, logical partitions (“LPAR’s”), or virtualized servers. Combining strategies As stated, it is very common to combine multiple strategies. “Cluster farms” combine HA and DR technologies, benefiting from the transparency of a single machine failure, while also providing a solution for data center or site-wide failure. Shops that use an HA+DR strategy may sometimes choose not to implement a backup strategy since multiple DR sites generally suffice. However, others consider backups to be a fail-safe method in the event all other strategies fail. All of these strategies can also be employed as the underlying infrastructure of a cloud computing environment. Copyright © 2004, 2009 IBM Page 13 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Planning an HA environment for Content Manager For this white paper, we have chosen to focus on installing and configuring Content Manager in an HA environment. In doing so, we are prioritizing the need for content stored in a Content Manager repository to be available to end-users, as part of a critical business process that utilizes content as an information asset. The most common, and therefore disruptive, source of downtime is at the machine-level. Various types of failure at the machine-level are common, whether it be from hard drive failure, network controller failure, or motherboard and CPU failure. The need to guard against such common single points of failure (“SPOF”) and maintain availability warrants the investment and skills development in an HA solution. While DR is a viable alternative to HA, the costs involved in duplication of both server and storage components better serve a site-based recovery strategy as opposed to a single-server recovery strategy. Benefitting from the use of shared storage and reduced failover times allows the HA configuration implemented here to result in a robust Content Manager server environment, minimizing the likelihood that any SPOF prevents critical access to content. Content Manager is an enterprise-class product which utilizes two or more Content Manager server components, one or more database servers, one or more WebSphere servers, and potentially many Cbased or Web-based client applications. Providing sufficient redundancies to mitigate the impact of SPOF’s is not a trivial task and requires significant up-front planning, design, and testing to ensure success. Therefore, you should consider this white paper a guide that describes one possible HA solution. Ultimately, the HA capabilities of Content Manager rest primarily on the HA infrastructure itself. The intent of this white paper is not to impose a single HA solution, but merely to present one that is fullytested and supported, and represents what we believe to be a ‘best practices’ HA architecture for Content Manager. Remember that only the HA solution described in this document, with the specific method of installation and configuration performed, is supported through IBM standard support for HA configurations. Otherwise, IBM standard support covers only runtime behavior that is re-producible in a non-HA environment. Although Content Manager client applications are the end-user interface to the Content Manager servers, clients typically do not contain mission-critical business data or server applications. Clients are typically deployed to many workstations, thereby implicitly redundant, or distributed via web application servers, which generally also have a method of providing high availability. For client-side configuration or other files, a simple backup policy is generally sufficient. On the other hand, without Content Manager server availability, there can be no access to content and client applications are useless. Thus, this white paper does not address client-side applications but rather focuses on the mission-critical pieces, the Content Manager servers. This white paper will also not address technologies external to Content Manager servers, particularly with respect to the content stores themselves. Because there are many options for content storage, such as local file system or Tivoli Storage Manager, Network Attached Storage (NAS) or IBM® General Parallel File System® (GPFS), Tivoli Storage Manager replication or Direct Access Storage Device (DASD) mirroring, customers are advised to consult the documentation or manufacturer of their respective storage devices to assess appropriate availability solutions. In this white paper, we make no assumption as to the location or availability infrastructure of content, so long as it is available and servable by the Content Manager resource manager. Copyright © 2004, 2009 IBM Page 14 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 In the diagram above, we identify the major Content Manager components and illustrate various potential SPOF’s. As stated, this white paper will not address Content Manager client applications although typical strategies are shown in the box at far left: clustering technology, mirroring or replication, virtualization, backup and recovery, etc. The box at far right shows the Content Manager resource manager application server. Content Manager has long supported WebSphere Application Server Network Deployment and this continues to be the strategy recommended in this white paper as well for the Content Manager resource manager web application. In this white paper, you will configure the Content Manager resource manager web application in a WebSphere Application Server Network Deployment environment with redundant IBM HTTP Servers. Although beyond the scope of this white paper, customers may choose to implement a similar strategy for their web-based Content Manager client applications. Also beyond the scope of this document, the Tivoli Storage Manager cluster shown above is optional but can be used to provide content replication and Tivoli Storage Manager server redundancy in the event of failure of a single Tivoli Storage Manager server. In the center are the Content Manager library server and Content Manager resource manager database servers. For the databases, you will configure DB2 and Tivoli System Automation to provide a highly available, mutual takeover scenario. In choosing to use DB2 and Tivoli System Automation for database failover, our view is that the database server is to the Content Manager server what the Content Manager server is to client applications. In other words, just as client applications are useless if the Content Manager servers are not up and running, so too are the Content Manager servers useless if the database server is not up and running. Thus, at the most basic level, the database is the critical availability component for a Content Manager server environment. While it is true that cluster server providers can be configured to monitor a process, they cannot monitor the integrity of the database itself. Some cluster servers monitor only the operating system, storage subsystems, and network communication, which won’t account for dead database server processes at all. The results can be imprecise and unreliable. Database vendor-provided HA technology, therefore, provides the lowest level of granularity for determining whether a database and all its intrinsic components represent an available state. For this reason, we have chosen to utilize DB2 and Tivoli Copyright © 2004, 2009 IBM Page 15 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 System Automation. Although outside the scope of this document, another advantage of Tivoli System Automation is that it can also be used to manage user-defined resources, with separate failover semantics. Refer to Tivoli System Automation documentation for details. To implement a solution that provides availability for the Content Manager library server database, Content Manager resource manager database, and Content Manager resource manager application, known collectively as the Content Manager server components, while also minimizing total-cost-of-ownership, this white paper uses software technologies available in the pre-requisite products already required by Content Manager today. Specifically, we will use Tivoli System Automation, shipped as part of DB2, to provide DB2 database instance failover. We’ll use Storage Area Network (SAN) storage with Fibre Channel as the shared storage for database files and NAS storage as the shared storage for Content Manager resource manager Local Area Network-Based Object Server (LBOS) data, or content. Lastly, we’ll use WebSphere Application Server Network Deployment to manage a cluster of WebSphere Application Server servers for Content Manager resource manager application availability. We’ll use multiple HTTP servers and an IBM® WebSphere® Edge Dispatcher to provide IP spraying and load balancing. Although beyond the scope of this white paper, we recommend that the SAN and NAS each employ RAID mirroring and controller redundancy, as well as multiple Fibre Channel adapters in each database server node, to eliminate them as SPOF’s. The diagram above focuses and illustrates the basic premise of our database HA configuration. Since DB2 and Tivoli System Automation implement an active-passive configuration for HA configurations, we’ll employ what is known as a mutual-takeover scenario. In this scenario, the Content Manager library server database will be active on node 1 (blue nodes) and passive on node 2 (purple nodes), while the Content Manager resource manager database will be active on node 2 and passive on node 1. The diagram does not show redundancies for the Fibre Channel or Ethernet controllers, but we will implement redundancy for the Ethernet controller. Notice that Content Manager library server runtime binaries will be copied to each member in the cluster. While it is possible to place the LS binaries on shared storage, we recommend against that for optimal performance and redundancy. Also notice that the LS and RM databases are each in their own DB2 instance. This is due to the fact that Tivoli System Automation monitors DB2 databases Copyright © 2004, 2009 IBM Page 16 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 at the instance level. Thus, in order to implement the mutual takeover scenario described in this document, we need to place them in separate DB2 instances. Note that it is not required to implement the mutual takeover scenario. If your Content Manager library server and Content Manager resource manager databases will be on separate DB2 server clusters (or if you have multiple RM’s), then you will have four database server boxes (or LPAR’s), two of which will be idle since they will be passive. Such configurations are beyond the scope of this document to describe but are only trivial variations of what’s described in this document. In such configurations, there is no need to create a second DB2 instance; you would simply create the RM database in the default db2inst1 DB2 instance and configure the RM database for HA as described later in this document for the LS. As Tivoli System Automation managed resources, Tivoli System Automation will poll the IBM® Reliable Scalable Cluster Technology (RSCT) subcomponent of the AIX operating system, which performs heartbeats with other cluster members based on a communication group of Ethernet controllers. There are various configuration strategies for managing the frequency of the heartbeat and Tivoli System Automation polling to balance between incoming user requests versus failed-node detection and splitbrain strategies. We recommend settings in this document based on minimization of failover times. You will need to test and evaluate what is appropriate for your environment. Refer to AIX, DB2, and Tivoli System Automation documentation for more details on configuration parameters for these subsystems. When a heartbeat is missed, Tivoli System Automation will failover to the passive node, mount file systems, perform file system checks, remap the virtual IP to the actual physical IP of the newly active node, and start the DB2 instance. Note that failover time is directly impacted by the speed of these operations and availability of subcomponents. Tivoli System Automation fails over the DB2 instance but not other, custom processes, such as the Content Manager library server monitor for DB2. This C-based process runs asynchronously with the Content Manager library server and can only be active on one node at a time. This means that you must manually start the Content Manager library server monitor after a failover has occurred. Multiple, active Content Manager library server monitor behavior is unknown and will fail since there is no active DB2 instance on the passive node for the monitor to connect to. Similarly, IBM® DB2® Net Search Extender also runs asynchronously to the Content Manager library server and DB2 instance and must be addressed separately. Please see DB2 documentation for more details. Content Manager product prerequisites for HA environments Note that this document assumes that your environment meets all normal Content Manager product prerequisite hardware, software, version, and patch-level requirements as stated in Content Manager v8.4.2 product documentation. In addition, the HA environment described in this document, using DB2 and Tivoli System Automation to failover Content Manager server databases, requires DB2 v9.5 fixpack 5 or later. Non-HA Content Manager installations do not need to upgrade to this level. Also, be aware that the DB2 limited-use license provided as part of the Content Manager product does not include a license for Tivoli System Automation. As such, you must either upgrade your DB2 license to a full DB2 license, or purchase a Tivoli System Automation license. Please contact your IBM sales representative for details. Unless otherwise stated, all other Content Manager hardware and software pre-requisites are the same as those listed in Content Manager v8.4.2 product documentation. Copyright © 2004, 2009 IBM Page 17 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Scenario Description Content Manager is an enterprise class application, which consists of several specialized components. The core components of every Content Manager implementation are the library server and one or many resource managers. The library server itself is mostly a database application, all its logic is implemented within the database (currently as stored procedures) and its data is stored within database tables and views. The library server implements functions such as authentication and authorization, data modeling, document routing, search and the storage of descriptive information for each object (so called meta data). The library server can also maintain relationships between different objects and can provide referential integrity. The resource manager is responsible for storing the actual content (files). Currently the resource manager logic is implemented as a web application, running in a web application server. The content is typically stored to local disk, but can be stored also to storage devices managed by IBM Tivoli Storage Manager or as blobs in the database. The resource manager also maintains internal information, which is again stored in a relational database. Two Nodes Scenario We will setup two Active/Standby DB2 instances. For db2inst1, it will be active on Node 1 (cmaix25). Once Node 1 meet with malfunction, Node 2 (cmaix26) will get all resources for db2inst1 and start it up. Also, a float IP (9.123.112.251) is used with db2inst1, so that client can just use this IP to connect to any database on db2inst1. Same strategy is used by db2inst2, although it will be active on Node 2 (cmaix26), and cmaix25 is a standby node for this instance. For example, LS DB is running on db2inst1 on cmaix25, and other machines can connect to DB via IP 9.123.112.251. Also, RM DB is running on db2inst2 on cmaix26 with IP 9.123.112.252. When cmaix25 meet hardware/software error, cmaix26 will take db2inst1 and IP 9.123.112.251. This means cmaix26 have two IPs and two instances running on it when cmaix25 is outage. Copyright © 2004, 2009 IBM Page 18 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 CM Clients cmaix25.cn.ibm.com (Active LS DB) AIX 5.3 ML9 cmaix26.cn.ibm.com (Active RM DB) Public Network 9.123.112.199 DB2 UDB 9.5 with TSA 9.123.112.200 INSTANCE db2inst1 (Standby ) ICMNLSDB Database (Virtual IP: 9.123 .112 .251 ) DATABASE icmnlsdb (Active) INSTANCE db2inst2 (Standby ) DATABASE icmnlsdb (Standby ) INSTANCE db2inst2 (Active) RMDB Database (Virtual IP: 9.123 .112 .252 ) DATABASE rmdb (Standby ) DATABASE rmdb (Active) FC FC FC FC IBM WebSphere Application Server 6.1 Network Deployment DB2 UDB 9.5 with TSA IBM N3700 (NAS) Object Storage - LBOSDATA INSTANCE db2inst1 (Active) AIX 5.3 ML9 IBM TotalStorage DS 4200 IBM HTTP Server WAS Plugin IBM WebSphere Application Server 6.1 Network Deployment IBM HTTP Server WAS Plugin WAS Node CMAIX25 icmrm WAS Node CMAIX26 icmrm WAS Cluster ICMRM Deployment Manager 192.168.17.19 192.168.17.20 Private Network Figure 1: Overall Scenario (2 Node Configuration) Copyright © 2004, 2009 IBM Page 19 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Hardware Setup This chapter summarizes the hardware we have used to setup Content Manager, including the servers, network and storage. We understand, that there is a wide range of options available, and there are many different ways to implement such a solution. We have chosen to use a configuration with just 2 servers, each running all components of Content Manager. Servers We have used two identical IBM® POWER6® Server 520. Here are the most important characteristics of these servers: Processors: 4x POWER6 CPU @4.204 GHz Hard Disk: 146 GB SAS DASD (“Serial Attached SCSI” and “Direct Access Storage Device”), IBM ST3146855SS Memory: 8 GB Fiber Channel Adapter: QLE2462 Network Cards: Dual 1 GB Ethernet card (10N9042) The IBM® pSeries® POWER6 server is connected via Gigabit Ethernet to the public network, and two Ethernet switches handle the network traffic. Also included into the pSeries server is a Fiber Channel Switch, which handles the connection to the IBM® TotalStorage® DS4200. Storage Each server has access to three different types of hard disk storage devices: internal disk, fiber channel attached storage (TotalStorage DS4200) and a NFS server (TotalStorage N3700). We used different storage types for different components of the Content Manager software stack. Internal Disk (SAS) We used one internal disk to store the operating system as well as installed software. This information is NOT shared between the nodes in the cluster. Also on this internal disk, we placed the AIX paging space. For consideration of redundancy, another internal disk can be added to create mirrorvg with original disk. Fiber Channel Attached Storage (IBM DS4200) Attached to all nodes in our cluster is an IBM DS4200, a storage subsystem, which is connect via redundant fiber channel connections to the pSeries server 520. In our scenario we will use the DS4200 to store DB2 instance and DB2 database files. The benefit of using fiber attached storage is its resilience, which in most configurations includes redundant communication paths and redundant disks (RAID). The configuration of your storage subsystem is a very important factor for the performance of Content Management solution. The number of operations per second and the throughput of your storage subsystem directly influence the performance of your database. Frequently, individual table spaces, Copyright © 2004, 2009 IBM Page 20 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 database log files, recovery files etc. are separated onto individual disk arrays, to achieve high performance and no blocking access. In our scenario, we kept the configuration of the DS4200 simple. Our DS4200 was filled with 16x1TB hard disks, which have been configured into a single RAID-5 array of 15 disks and one hot spare. This is certainly not the perfect configuration for performance but easy to setup and maintain. On this array we created 2 logical drives, one is used by db2inst1, and another is used by db2inst2. Machine 1 (cmaix25) will be active node for db2inst1, and standby node for db2inst2. Machine 2 (cmaix26) will be active node for db2inst2, and standby node for db2inst1. This can easily be done by using the IBM® System Storage™ Manager. root@cmaix25 # lscfg -vl fcs0|grep Network Network Address.............10000000C97E64B5 root@cmaix26 # lscfg -vl fcs0|grep Network Network Address.............10000000C973F7A6 This command will list the WWPN available on your AIX servers. Depending on your hardware configuration, you might get several WWPNs listed. In our case, the WWPNs are: cmaix25: 10000000C97E64B5 cmaix26: 10000000C973F7A6 With this information, define your logical drives on your storage server and map them to the WWPNs. Table 1: Logical Drive definitions on DS4200 contain our definitions: Name Size Host OS Mapping DB2INST1VG 40GB AIX 10000000C97E64B5 10000000C973F7A6 DB2INST2VG 40GB AIX 10000000C97E64B5 10000000C973F7A6 Table 1: Logical Drive definitions on DS4200 Figure 2: DS4200 Configuration shows the definitions in the System Storage Manager Client. You can see the 2 defined logical drives and its mapping to the host group CMAIX25_26. Note that in this screenshot, the mapping has been defined to two nodes: cmaix25 and cmaix26 as mentioned above. Copyright © 2004, 2009 IBM Page 21 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 2: DS4200 Configuration When you do your storage system configuration, the most important questions are: How many logical drives will I need? How large should the logical drives be? You will need at least two logical drives, one for db2inst1 (used by LS database ICMNLSDB), and one for db2inst2 (used by RM database RMDB). Both of them will contain the database files (tablespace etc.) which will grow. Please see your IBM sales representative for sizing information for a given number of documents and workload to better size your database files. Network Attached Storage (IBM N3700) To store the actual documents, Content Manager can use several storage options, for example file systems or Tivoli Storage Manager. The setup for the storage to file system is simple. It requires the administrator to create the mount point and define it to Content Manager. In our sample configuration, we have used a Network File System (NFS) file system hosted on an IBM N3700 as storage volume. On your NAS Server, create the volume to be exported. Figure 3: IBM N3700 Volume Definition shows the values we have used. We created a volume of 50GB and then defined the NFS exports. Copyright © 2004, 2009 IBM Page 22 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 3: IBM N3700 Volume Definition When you define the NFS exports, you basically have to define the following settings: Path: /vol/Lbosdata Read-Write access for the following hosts: cmaix25.cn.ibm.com cmaix26.cn.ibm.com Root access for the following hosts: cmaix25.cn.ibm.com cmaix26.cn.ibm.com Security: UNIX System (sys) Network In our sample configuration, each server requires two physical network interfaces (NIC) and three IP addresses. For HA consideration, we also recommend to use four NICs on each server. Then combine every two NICs to a VIP. Of course, this also means two IP per server. The servers are accessible via a public network, and are interconnected via a private network. The public network is used by client applications to access the servers services. The private network should be used exclusively for heart beat monitor. The third IP is used as a float virtual IP, it can be used on any active node. On the public network, one IP address is required per server. Also, there will be a virtual IP for each server on public network. On the private network, one IP address is required per server, and is typically in a non-routed subnet such as 192.168.x.x for example. Table 2: IP Configuration show the network settings we have used. Copyright © 2004, 2009 IBM Page 23 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX Our Name IP Address Used for cmaix25 9.123.112.199 Host name node 1 cmaix25-priv 192.168.17.19 Private connect node 1 cmlsdb 9.123.112.251 Virtual IP for LS DB cmaix26 9.123.112.200 Host name node 2 cmaix26-priv 192.168.17.20 Private connect node 2 cmrmdb 9.123.112.252 Virtual IP for RM DB cmrmapp 9.123.112.254 Virtual IP for RM APP Your Name 2009.12.28 Your IP Address Table 2: IP Configuration It is important, that all host names can be resolved. For the physical IP on the public subnet, you should add them into your DNS. For the private interconnect, you can optionally add them to your /etc/hosts files. Samples below shows what should be added into /etc/hosts file. 9.123.112.199 192.168.17.19 9.123.112.251 9.123.112.200 192.168.17.20 9.123.112.252 9.123.112.254 cmaix25 cmaix25-priv cmlsdb cmaix26 cmaix26-priv cmrmdb cmrmapp Copyright © 2004, 2009 IBM Page 24 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Operating System Preparation This chapter contains information on how to prepare your AIX operating system for the user of Content Manager. It describes necessary changes to your AIX configuration to run Content Manager and DB2 software. As we will prepare two servers with same configuration, all the steps described in this chapter must be completed on ALL nodes participating in the cluster! Virtual I/O Server The Virtual I/O Server is part of the IBM System p5 Advanced Power Virtualization hardware feature. Virtual I/O Server allows sharing of physical resources between LPARs including virtual SCSI and virtual networking. This allows more efficient utilization of physical resources through sharing between LPARs and facilitates server consolidation. Before using VIO servers, please see your IBM AIX representative to eliminate single points of failure (SPOF). Base AIX Installation For our setup, we installed AIX 5.3 TL9 SP2 with 64bit kernel. During the installation we’ve selected default options, including the package selection. DON’T install OS update package before you finish prerequisite software installation. After installation, make sure that you assign sufficient paging space. A rule of thumb is to create a Paging Space of at least the size of your physical memory. root@cmaix25 # lsps -a Page Space Physical Volume hd6 hdisk0 Volume Group rootvg Size %Used Active Auto Type 4096MB 1 yes yes lv Then, check system disk and make sure we have enough space for future software installation. Recommended configuration is shown below: root@cmaix25 # df -m Filesystem MB blocks /dev/hd4 2048.00 /dev/hd2 6144.00 /dev/hd9var 2048.00 /dev/hd3 2048.00 /dev/hd1 4096.00 /proc /dev/hd10opt 4096.00 /dev/lv00 256.00 Free %Used 1986.82 3% 4396.60 29% 1976.23 4% 2041.96 1% 4095.04 1% 3942.52 4% 247.92 4% Iused %Iused Mounted on 4731 2% / 41918 5% /usr 4633 2% /var 31 1% /tmp 5 1% /home - /proc 1950 1% /opt 18 1% /var/adm/csd Prerequisite Software On AIX, high availability is implemented with RSCT components. Then following packages must be installed before we setup HA environment: Copyright © 2004, 2009 IBM Page 25 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 rsct.basic rsct.compat.basic rsct.compat.clients Also, another package is prerequisite for Tivoli System Automation: rsct.opt.storagerm These filesets can be found in AIX installation DVD, you can just insert Installation DVD into pSeries server and then run “smitty install” command to finish installation. Finally we’ve applied the latest patches to our AIX installation. Our version is: root@cmaix26 # oslevel –s 5300-09-02-0849 Additional, on the AIX version later than 5.3 TL9, this is a known problem described at: http://www-01.ibm.com/support/docview.wss?uid=isg1IZ47850 We must close http4websm service and leave port 80 for our future http usage. root@cmaix26 # vi /etc/inittab : webserverstart:2:once:startsrc -s http4websm >/dev/null 2>&1 root@cmaix26 # stopsrc -s http4websm 0513-044 The http4websm Subsystem was requested to stop. Storage Configuration The first external storage we want to attach is an IBM DS4200, which is a fiber channel attached storage server. Previously we have already defined two volumes on the DS4200 and mapped them to the WWPN of our servers. In our case, we did have redundant fiber channel connections between the pSeries server and the DS4200 storage server. With AIX, driver is integrated, and multipath support is build into the fiber channel adapter driver. After fresh installation, disk status will look similar to: root@cmaix26 # lspv hdisk0 000a5db26d77585c rootvg active Then DS4200 can be assigned to each server with steps in previous chapter, and “cfgmgr” command is executed on each server to discover LUNs of DS4200. We can find two disks on each server: root@cmaix26 # lspv hdisk0 000a5db26d77585c hdisk2 none hdisk3 none rootvg None None active For future usage, each disk should have PVID, as “000925926d77bb8e” of hdisk0. But other disks from DS4200 may not get PVID automatically, we need update their information. root@cmaix26 # chdev -l hdisk2 -a pv=yes hdisk2 changed root@cmaix26 # chdev -l hdisk3 -a pv=yes hdisk3 changed Copyright © 2004, 2009 IBM Page 26 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX root@cmaix26 # lspv hdisk0 000a5db26d77585c hdisk2 000a5d72aa712308 hdisk3 000a5d72baaf227e rootvg None None 2009.12.28 active After running on each server, configuration is finished. The first disk (hdisk0) is our internal disk, and they have different PVID on each server. The next two disks (hdisk2 and hdisk3) are our two volumes over the fiber channel connections, which show same PVID on each server. Logical Volume and Mount Points Next, we want to prepare the directories and mount points for DB2 software installation. In our case, we want to install the DB2 software into the default directory /opt/IBM/db2/V9.5. Further we create two mount point (/DB2INST1 and /DB2INST2 on both cmaix25 and cmaix26), where we will later mount DB2 instance volume to store DB2 instance file and DB file. Also, we can place home directory of db2inst1, db2inst2, db2fenc1, db2fenc2 into it. Finally we will create the mount point for the resource manager storage volume, typically called “lbosdata”. Here we will later mount an NFS volume. File System Configuration on Node 1 Create Volume Group We can use “smitty” or using two commands below to create two VGs on two different disks. Notice that VG Major Number (ID of Volume Group) of cmaix25 and cmaix26 should be SAME for every hdisk. root@cmaix25 # mkvg -S -y DB2VG1 -f -n -V 45 hdisk2 DB2VG1 root@cmaix25 # mkvg -S -y DB2VG2 -f -n -V 46 hdisk3 DB2VG2 root@cmaix25 # lspv hdisk0 000925926d77bb8e hdisk2 000a5d72aa712308 hdisk3 000a5d72baaf227e rootvg DB2VG1 DB2VG2 active active active Create File System Now we can create file systems and mount points on DB2VG1 and DB2VG2, scripts are used to create 20G JFS2 file system for /DB2INST1, and same size for /DB2INST2. The commands below will create the necessary directories. This will also define proper ownership and permissions for the directories. root@cmaix25 # crfs -v jfs2 -g DB2VG1 -a size=20G -m /DB2INST1 -A no -p rw -a agblksize=4096 File system created successfully. 20970676 kilobytes total disk space. New File System size is 41943040 root@cmaix25 # crfs -v jfs2 -g DB2VG2 -a size=20G -m /DB2INST2 -A no -p rw -a agblksize=4096 File system created successfully. 20970676 kilobytes total disk space. New File System size is 41943040 Copyright © 2004, 2009 IBM Page 27 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 root@cmaix25 # mount /DB2INST1 root@cmaix25 # mount /DB2INST2 root@cmaix25 # chown -R bin:bin /DB2INST1 root@cmaix25 # chown -R bin:bin /DB2INST2 root@cmaix25 # chmod -R 755 /DB2INST1 root@cmaix25 # chmod -R 755 /DB2INST2 Create Users and Groups As we will create four user’s home directory under /DB2INST1 and /DB2INST2, we should run scripts to create users and groups with both two directory mounted on same machine. Scripts are provided on next paragraph, just modify user name and password, in our case, we ran them here on node1 (maix25) with /DB2INST1 and /DB2INST2 mounted. root@cmaix25 # <Create Groups and Users with /DB2INST1 and /DB2INST2 mounted> File System Configuration on Node 2 Then we can setup totally same environment on cmaix26, active RMDB node. We can just import information from cmaix25 to cmaix26, without creating new File Systems and Volume Groups. Close DB2VG1 and DB2VG2 from cmaix25. root@cmaix25 # umount /DB2INST1 root@cmaix25 # umount /DB2INST2 root@cmaix25 # varyoffvg DB2VG1 root@cmaix25 # varyoffvg DB2VG2 Check VG number for DB2VG1 and DB2VG2 on cmaix25, this is important to keep this information same on both two nodes. root@cmaix25 # ls -al /dev/DB2VG1 crw-rw---1 root system 45, 0 Aug 06 15:23 /dev/DB2VG1 root@cmaix25 # ls -al /dev/DB2VG2 crw-rw---1 root system 46, 0 Aug 06 15:27 /dev/DB2VG2 Turn back to cmaix26, we can use same VG Major number to import same hdisk: root@cmaix26 # importvg -y DB2VG1 -V 45 hdisk2 root@cmaix26 # importvg -y DB2VG2 -V 46 hdisk3 root@cmaix26 # mount /DB2INST1 root@cmaix26 # mount /DB2INST2 Create Users and Groups Node2 (cmaix26) now have same File System information with Node1 (cmaix25), but user information is not synchronized. We should run scripts to create users and groups on Node2, which are exactly same with previous scripts on Node1. root@cmaix26 # <Create Groups and Users with /DB2INST1 and /DB2INST2 mounted> Copyright © 2004, 2009 IBM Page 28 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Preparation for DB2 Finally, we can mount /DB2INST1 Node1 (cmaix25) and mount /DB2INST2 on Node2 (cmaix26). On Node2 (cmaix26): root@cmaix26 # umount /DB2INST1 root@cmaix26 # varyoffvg DB2VG1 On Node1 (cmaix25): root@cmaix25 # varyonvg DB2VG1 root@cmaix25 # mount /DB2INST1 Users and Groups Both DB2 UDB and Content Manager, require certain operating system users and groups to exist. For DB2 UDB v9.5, typically three groups (dasadm1, db2iadm1 and db2fadm1), and three users (dasusr1, db2inst1, db2fenc1) are required. Content Manager requires the group ibmcmgrp and ther user ibmcmadm. It is important to create the users and groups in an identical way, sharing the same numerical ids and names, on all nodes in the cluster. Table 3: Operating system groups and users show the settings we have used. Group or User Role Numeric ID: Group: dasadm1 DB2 Admin Server Group 101 Group: db2iadm1 DB2 Instance Group 1 102 Group: db2fadm1 DB2 Fenced Group 1 103 Group: db2iadm2 DB2 Instance Group 2 104 Group: db2fadm2 DB2 Fenced Group 2 105 Group: ibmcmgrp All Content Manager users 207 User: dasusr1 DB2 Admin Server User 203 User: db2inst1 DB2 instance 1 owner 204 User: db2fenc1 DB2 Fenced user 205 User: db2inst2 DB2 instance 2 owner 206 User: db2fenc2 DB2 fenced user 207 User: ibmcmadm Content Manager configuration owner 208 User: icmadmin Content Manager admin for LS DB 209 Copyright © 2004, 2009 IBM Name in your Environment Page 29 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 LS DB User: icmconct Content Manager connect user 210 User: rmadmin Content Manager admin for RM DB 211 Table 3: Operating system groups and users The following commands will create these definitions. In our case, we used group ids 101 through 105 and 207 for the groups and users ids 203 through 211. Run them on each node with the steps in “Mount Points” chapter, and /DB2INST1 and /DB2INST2 must be mounted on same machine before running this scripts. Also, change password to replace “YOUR_PASSWORD” area. # mkgroup -A id=101 dasadm1 # mkgroup -A id=102 db2iadm1 # mkgroup -A id=103 db2fadm1 # mkgroup -A id=104 db2iadm2 # mkgroup -A id=105 db2fadm2 # mkgroup -A id=207 ibmcmgrp # # mkuser id=203 pgrp=dasadm1 groups=dasadm1,staff home='/home/dasusr1' dasusr1 # mkuser id=204 pgrp=db2iadm1 groups=db2iadm1,staff,dasadm1,db2fadm1 home='/DB2INST1/db2inst1' db2inst1 # mkuser id=205 pgrp=db2fadm1 groups=db2fadm1,staff,db2iadm1 home='/DB2INST1/db2fenc1' db2fenc1 # mkuser id=206 pgrp=db2iadm2 groups=db2iadm2,staff,dasadm1,db2fadm2 home='/DB2INST2/db2inst2' db2inst2 # mkuser id=207 pgrp=db2fadm2 groups=db2fadm2,staff,db2iadm2 home='/DB2INST2/db2fenc2' db2fenc2 # # mkuser id=208 pgrp=ibmcmgrp groups=ibmcmgrp home='/home/ibmcmadm' ibmcmadm # mkuser id=209 pgrp=staff groups=staff,db2iadm1,ibmcmgrp home='/home/icmadmin' icmadmin # mkuser id=210 pgrp=staff groups=staff,ibmcmgrp home='/home/icmconct' icmconct # mkuser id=211 pgrp=staff groups=staff,db2iadm2,ibmcmgrp home='/home/rmadmin' rmadmin # # for i in dasusr1 db2inst1 db2fenc1 db2inst2 db2fenc2 ibmcmadm icmadmin icmconct rmadmin ; do echo echo $i:[YOUR_PASSWORD] \| chpasswd ;done | ksh # for i in dasusr1 db2inst1 db2fenc1 db2inst2 db2fenc2 ibmcmadm icmadmin icmconct rmadmin ; do pwdadm -c $i ;done NFS Configuration The last piece of external storage we need is the default storage volume for the resource manager. Earlier we have defined both, an NFS export on the IBM N3700 and a directory (/lbosdata) to become the mount point. All that is missing is the mount itself. To mount the NFS volume, add to your /etc/filesystems the following line: /lbosdata: Copyright © 2004, 2009 IBM Page 30 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX dev vfs nodename mount options account 2009.12.28 = /Vol/Lbosdata = nfs = cmaix27 = true = rw = false In this sample, cmaix27.cn.ibm.com is the hostname of the NFS server, /Vol/Lbosdata the name of the exported volume, /lbosdata the mount point on the client side, nfs the file system type and the remaining parts are options for the mount command. The volume will be automatically mounted after a reboot. To mount the volume immediately, run: # mount /Lbosdata Repeat the above steps on all nodes participating in the cluster. Now verify, that all required file systems are mounted. Below is the output for one of our servers: root@cmaix25 # df -m Filesystem MB blocks Free %Used …… cmaix27:/Vol/Lbosdata 40960.00 40952.47 Copyright © 2004, 2009 IBM Iused %Iused Mounted on 1% 3 1% /lbosdata Page 31 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 DB2 with Tivoli System Automation Now we are ready to start with the installation of DB2. We have installed the prerequisite software and configured our network and storage. Tivoli System Automation is provided with DB2 installation package. The steps outlined in this document should be performed on both nodes, however, different nodes will setup different instance. Pre-Install Validation Check both of two servers, and make sure /DB2INST1 is mounted on host cmaix25, and /DB2INST2 is mounted on cmaix26. # df –m (on cmaix25) /dev/fslv00 20480.00 20178.25 2% 283 1% /DB2INST1 # df –m (on cmaix26) /dev/fslv01 20480.00 20178.25 2% 283 1% /DB2INST2 Software Installation We need to install DB2 software on both servers, with the same configuration; however, we will not setup DB2 Instance at this step. Insert DB2 installation media, and run “db2setup”. Then Figure 4: DB2 setup wizard will be shown. NOTE: Although the following screen shots are from fixpack 3, remember that HA support for Content Manager on DB2 requires DB2 v9.5 fixpack 5 or higher. Figure 4: DB2 setup wizard Copyright © 2004, 2009 IBM Page 32 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Choose “Install a Product” and select “Install New” in Figure 5: Install New DB2 product. Figure 5: Install New DB2 product Click “Next” in Figure 6: Welcome window. Figure 6: Welcome window Accept software license and “Next” when you see Figure 7: Accept License. Copyright © 2004, 2009 IBM Page 33 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 7: Accept License For installation type, we select “Custom” in Figure 8: Installation Type. Figure 8: Installation Type In step 4, we can use default option for Figure 9: Response file. Copyright © 2004, 2009 IBM Page 34 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 9: Response file Check the features that you need to install in Figure 10: Select the features to Install. Figure 10: Select the features to Install For Figure 11: Select the langurages to Install, select language and installation directory. Copyright © 2004, 2009 IBM Page 35 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 11: Select the langurages to Install We can ignore installing DB2 information center for Figure 12: Specify the location of DB2 Information Center. Figure 12: Specify the location of DB2 Information Center Install IBM® Tivoli® System Automation for Multiplatforms Base Component (SA MP Base Component) at the step of Figure 13: Install the IBM SA MP Component. For DB2 v9.5, SA MP version 2.2 is provided. Recall that you must have purchased a full DB2 or Tivoli System Copyright © 2004, 2009 IBM Page 36 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Automation license to use Tivoli System Automation with DB2 and Content Manager. The limited-use DB2 license provided with your Content Manager license does not include Tivoli System Automation. You must check if license is installed correctly after installation, as introduced later. Figure 13: Install the IBM SA MP Component Use existing user dasusr1 which is already created on both of two DAS servers for Figure 14: Set user information for the DB2 Administration Server. Copyright © 2004, 2009 IBM Page 37 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 14: Set user information for the DB2 Administration Server We will not create DB2 instance in Figure 15: Set up a DB2 instance. Figure 15: Set up a DB2 instance Customer can select if notification is needed in Figure 16: Set up notification. Copyright © 2004, 2009 IBM Page 38 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 16: Set up notification Here is the summary file in Figure 17: Summary, click “Finish” after checking settings. Figure 17: Summary Repeat the same steps in another node, NODE2. Then, we should check license on both two nodes. Copyright © 2004, 2009 IBM Page 39 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Notice: If Tivoli System Automation is not installed during DB2 software installation, we can run installSAM under /db2/aix/tsamp directory on DB2 installation media. root@cmaix25 # cd db2/aix/tsamp root@cmaix25 # ./installSAM prereqSAM: All prerequisites for the ITSAMP installation are met on operating system AIX 5300-09 installSAM: Packages will be installed from directory ./AIX installSAM: The following package is not installed yet and needs to be installed ./AIX/sam.core installSAM: A general License Agreement and License Information specifically for System Automation will be shown. Scroll down using the Enter key (line by line) or Space bar (page by page). At the end you will be asked to accept the terms to be allowed to install the product. Select Enter to continue ... ... installSAM: To accept all terms of the preceding License Agreement and License Information type 'y', anything else to decline License Validation Now, check your DB2 and Tivoli System Automation license on both nodes: # cd /opt/IBM/db2/V9.5/adm # ./db2licm -l # # samlicm -s Make sure you have correct license, please be aware of following APAR before registering Tivoli System Automation license: IZ47958: SA MP LICENSE CANNOT BE INSTALLED AS NON-ROOT USER USING 'SAMLICM' (http://www01.ibm.com/support/docview.wss?rs=820&context=SSRM2X&dc=DB550&uid=swg1IZ47958 &loc=en_US&cs=UTF-8&lang=en&rss=ct820tivoli) Setup Instance: db2inst1 On node1 (cmaix25), we will setup a DB2 instance named db2inst1. Related files will be placed at db2inst1 user’s home directory. root@cmaix25 # cd /opt/IBM/db2/V9.5/instance root@cmaix25 # ./db2isetup DB2 instance setup wizard as Figure 18: DB2 instance Setup wizard will be shown, click “Next”. Copyright © 2004, 2009 IBM Page 40 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 18: DB2 instance Setup wizard Create a DB2 instance in Figure 19: Set up a DB2 instance. Figure 19: Set up a DB2 instance As Content Manager supports single partition instance only, we check that option in Figure 20: Set up partitioning options for the DB2 instance. Copyright © 2004, 2009 IBM Page 41 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 20: Set up partitioning options for the DB2 instance In Figure 21: Set user information for the DB2 instance owner, we use existing user “db2inst1” for DB2 instance owner, which is already created in previous steps. Figure 21: Set user information for the DB2 instance owner For fenced user in Figure 22: Set user Information for the fenced user, we use existing user db2fenc1. Copyright © 2004, 2009 IBM Page 42 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 22: Set user Information for the fenced user In Figure 23: Configure instance communication and startup, we use 50001 port for db2inst1, and port for db2inst2 should be different in following chapter. Uncheck “Autostart the instance at system startup”, as DB2 instance will be monitored and managed by SA MP. Figure 23: Configure instance communication and startup DB2 tools catalog is optional for Figure 24: Prepare the DB2 tools catalog. Copyright © 2004, 2009 IBM Page 43 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 24: Prepare the DB2 tools catalog Users can specify a contact for health monitor notification in Figure 25: Specify a contact for health monitor notification. Figure 25: Specify a contact for health monitor notification Content Manager doesn’t use DB2 Text Search server. For text search feature, please refer to DB2 Net Search Extender component. We can select “Do not configure at this time” in Figure 26: Configure the DB2 Text Search server Copyright © 2004, 2009 IBM Page 44 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 26: Configure the DB2 Text Search server Check summary in Figure 27: Summary and click “Finish”. Figure 27: Summary After db2inst1 setup, check /etc/services, we can find following content. As db2inst1 will be run on cmaix26 when cmaix25 is not available, we should add same content on /etc/services of cmaix26 too. DB2_db2inst1 60000/tcp Copyright © 2004, 2009 IBM Page 45 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 DB2_db2inst1_1 60001/tcp DB2_db2inst1_2 60002/tcp DB2_db2inst1_END 60003/tcp db2c_db2inst1 50001/tcp Setup Instance: db2inst2 On cmaix26, we will setup another DB2 instance named db2inst2. Related files will be put at db2inst2 user’s home directory. root@cmaix26 # cd /opt/IBM/db2/V9.5/instance root@cmaix26 # ./db2isetup Difference is that we will assign existing user db2inst2 in Figure 28: Set user information for the DB2 instance owner. Figure 28: Set user information for the DB2 instance owner And add db2fenc2 for fenced user in Figure 29: Set user information for the fenced user. Copyright © 2004, 2009 IBM Page 46 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 29: Set user information for the fenced user As port 50001 is kept for db2inst1, db2inst2 will automatically get port 50002. Uncheck “Autostart the instance at system startup” in Figure 30: Configure instance communication and startup. Figure 30: Configure instance communication and startup In Figure 31: Summary, view the summary and click “Finish”. Copyright © 2004, 2009 IBM Page 47 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 31: Summary Also, new content will be added at /etc/services of cmaix26: DB2_db2inst2 60004/tcp DB2_db2inst2_1 60005/tcp DB2_db2inst2_2 60006/tcp DB2_db2inst2_END 60007/tcp db2c_db2inst2 50002/tcp After modification, check /etc/services on both two servers. Both of them must have the same configuration: DB2_db2inst1 60000/tcp DB2_db2inst1_1 60001/tcp DB2_db2inst1_2 60002/tcp DB2_db2inst1_END 60003/tcp db2c_db2inst1 50001/tcp DB2_db2inst2 60004/tcp DB2_db2inst2_1 60005/tcp DB2_db2inst2_2 60006/tcp DB2_db2inst2_END 60007/tcp db2c_db2inst2 50002/tcp Copyright © 2004, 2009 IBM Page 48 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 DB2 High Availability Instance Configuration Utility Now we can start DB2 HA configuration. We will setup two instances on cmaix25 and cmaix26, for db2inst1, its original location is /DB2INST1/db2inst1 directory on cmaix25, and db2inst2 will be put at /DB2INST2/db2inst2 directory on cmaix26. Also, db2inst1 uses a virtual IP 9.123.112.251, and db2inst2 uses another virtual IP 9.123.112.252. When one of the servers is down, the other server will acquire and use the resources, including virtual IP. Tivoli System Automation is working with DB2 to provide high availability, and it’s configured using a utility named db2haicu, which is a tool available with DB2 9.5 and stands for the “DB2 High Availability Instance Configuration Utility”. This utility takes in user input regarding the software and hardware environment of a DB2 instance, and configures the instance for High Availability using the Tivoli System Automation cluster manager. During this configuration process, all necessary resources, dependencies, and equivalencies are automatically defined to Tivoli System Automation. Note: Tivoli System Automation does not need to be manually installed on your system as it is pre-packaged with DB2 9.5. Two input methods can be used to provide the necessary data to db2haicu. The first method is the interactive mode, where you are prompted for input at the command line. The second input method is the XML mode, where db2haicu can parse the necessary data from a user defined XML file. The db2haicu tool can be used to configure such an HA system. During the db2haicu configuration process, the necessary HA resources and their relationships are defined to the cluster manager. Failure events in the HA system can then be detected automatically and takeover operations can be run without manual intervention. Cluster Preparation Before using the db2haicu tool, the primary and the standby nodes must be prepared with the proper security environment. With root authority, issue the following command on all cluster nodes: root@cmaix25 # preprpnode cmaix25 cmaix26 root@cmaix26 # preprpnode cmaix25 cmaix26 This command only needs to be run once per node and not for every DB2 instance that is made highly available. Time Synchronization It is recommended (but not mandatory) that the time and dates on cluster nodes be synchronized. Synchronized clocks can make problem determination more straightforward because time ordering of events that appear in various log files can be performed without applying a delta time correction to account for time differences between machines. Note that the xntpd subsystem can be used for this purpose. Refer to your operating system documentation for information on how to configure xntpd for your system. Copyright © 2004, 2009 IBM Page 49 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 The db2haicu Interactive Setup on Node1 After the preceding preliminary configuration steps are completed, the db2haicu tool can be used to automate HA failover. The db2haicu must be run on the node hosting the DB2 instance. The details involving the process are outlined in the following section. Creating a Cluster Domain Log on to the machine hosting the DB2 instance (for our case, cmaix25 is owner of db2inst1), and issue the ‘db2haicu’ command: The following welcome message will be displayed on the screen: Welcome to the DB2 High Availability Instance Configuration Utility (db2haicu). You can find detailed diagnostic information in the DB2 server diagnostic log file called db2diag.log. Also, you can use the utility called db2pd to query the status of the cluster domains you create. For more information about configuring your clustered environment using db2haicu, see the topic called 'DB2 High Availability Instance Configuration Utility (db2haicu)' in the DB2 Information Center. db2haicu determined the current DB2 database manager instance is db2inst1. The cluster configuration that follows will apply to this instance. db2haicu is collecting information on your current setup. This step may take some time as db2haicu will need to activate all databases for the instance to discover all paths ... When you use db2haicu to configure your clustered environment, you create cluster domains. For more information, see the topic 'Creating a cluster domain with db2haicu' in the DB2 Information Center. db2haicu is searching the current machine for an existing active cluster domain ... db2haicu did not find a cluster domain on this machine. db2haicu will now query the system for information about cluster nodes to create a new cluster domain ... db2haicu did not find a cluster domain on this machine. To continue configuring your clustered environment for high availability, you must create a cluster domain; otherwise, db2haicu will exit. Create a domain and continue? [1] 1. Yes 2. No 1 Create a unique name for the new domain: DB2Domain Nodes must now be added to the new domain. How many cluster nodes will the domain DB2Domain contain? 2 Enter the host name of a machine to add to the domain: cmaix25 Copyright © 2004, 2009 IBM Page 50 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Enter the host name of a machine to add to the domain: cmaix26 db2haicu can now create a new domain containing the 2 machines that you specified. If you choose not to create a domain now, db2haicu will exit. Create the domain now? [1] 1. Yes 2. No 1 Creating domain DB2Domain in the cluster ... Creating domain DB2Domain in the cluster was successful. Quorum Configuration After the domain creation has completed, a quorum must be configured for the cluster domain. The supported quorum type for this solution is a ‘network quorum’. A network quorum (or network tiebreaker) is a ping-able IP address that is used to decide which node in the cluster will serve as the ‘active’ node during a site failure, and which nodes will be offline. Note that the machine hosting this IP address does not need any particular software or operating system level installed; its primary requirement is that it can be pinged from all nodes in the cluster, and must remain pingable in the case of cluster node failures. You will be prompted by db2haicu to enter quorum configuration values: You can now configure a quorum device for the domain. For more information, see the topic "Quorum devices" in the DB2 Information Center. If you do not configure a quorum device for the domain, then a human operator will have to manually intervene if subsets of machines in the cluster lose connectivity. Configure a quorum device for the domain called DB2Domain? [1] 1. Yes 2. No 1 The following is a list of supported quorum device types: 1. Network Quorum Enter the number corresponding to the quorum device type to be used: [1] 1 Specify the network address of the quorum device: 9.123.112.1 Configuring quorum device for domain DB2Domain ... Configuring quorum device for domain DB2Domain was successful. Network Setup After the quorum configuration, you may define the public network of your system to db2haicu. If network failure detection is important to your configuration, you must follow the prompts and add the networks to the cluster at this point. The db2haicu tool automatically discovers all network interfaces. Copyright © 2004, 2009 IBM Page 51 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 The cluster manager found 4 network interface cards on the machines in the domain. You can use db2haicu to create networks for these network interface cards. For more information, see the topic 'Creating networks with db2haicu' in the DB2 Information Center. Create networks for these network interface cards? [1] 1. Yes 2. No 1 Enter the name of the network for the network interface card: en0 on cluster node: cmaix25.cn.ibm.com 1. Create a new public network for this network interface card. 2. Create a new private network for this network interface card. 1 Enter selection: Are you sure you want to add the network interface card en0 on cluster node cmaix25.cn.ibm.com to the network db2_public_network_0? [1] 1. Yes 2. No 1 Adding network interface card en0 on cluster node cmaix25.cn.ibm.com to the network db2_public_network_0 ... Adding network interface card en0 on cluster node cmaix25.cn.ibm.com to the network db2_public_network_0 was successful. Enter the name of the network for the network interface card: en1 on cluster node: cmaix26.cn.ibm.com 1. db2_public_network_0 2. Create a new public network for this network interface card. 3. Create a new private network for this network interface card. 3 Enter selection: Are you sure you want to add the network interface card en1 on cluster node cmaix26.cn.ibm.com to the network db2_private_network_0? [1] 1. Yes 2. No 1 Adding network interface card en1 on cluster node cmaix26.cn.ibm.com to the network db2_private_network_0 ... Adding network interface card en1 on cluster node cmaix26.cn.ibm.com to the network db2_private_network_0 was successful. Enter the name of the network for the network interface card: en0 on cluster node: cmaix26.cn.ibm.com 1. db2_private_network_0 2. db2_public_network_0 3. Create a new public network for this network interface card. 4. Create a new private network for this network interface card. Enter selection: Copyright © 2004, 2009 IBM Page 52 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 2 Are you sure you want to add the network interface card en0 on cluster node cmaix26.cn.ibm.com to the network db2_public_network_0? [1] 1. Yes 2. No 1 Adding network interface card en0 on cluster node cmaix26.cn.ibm.com to the network db2_public_network_0 ... Adding network interface card en0 on cluster node cmaix26.cn.ibm.com to the network db2_public_network_0 was successful. Enter the name of the network for the network interface card: en1 on cluster node: cmaix25.cn.ibm.com 1. db2_private_network_0 2. db2_public_network_0 3. Create a new public network for this network interface card. 4. Create a new private network for this network interface card. Enter selection: 1 Are you sure you want to add the network interface card en1 on cluster node cmaix25.cn.ibm.com to the network db2_private_network_0? [1] 1. Yes 2. No 1 Adding network interface card en1 on cluster node cmaix25.cn.ibm.com to the network db2_private_network_0 ... Adding network interface card en1 on cluster node cmaix25.cn.ibm.com to the network db2_private_network_0 was successful. Cluster Manager Selection After the network definitions, db2haicu prompts you for the cluster manager software that you are using for the current HA setup. For our purpose, we select Tivoli System Automation: Retrieving high availability configuration parameter for instance db2inst1 ... The cluster manager name configuration parameter (high availability configuration parameter) is not set. For more information, see the topic "cluster_mgr - Cluster manager name configuration parameter" in the DB2 Information Center. Do you want to set the high availability configuration parameter? The following are valid settings for the high availability configuration parameter: 1.TSA 2.Vendor Enter a value for the high availability configuration parameter: [1] 1 Setting a high availability configuration parameter for instance db2inst1 to TSA. Copyright © 2004, 2009 IBM Page 53 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Failover Policy Now you need to configure the failover policy for the instance db2inst1. The failover policy determines the machines on which the cluster manager will restart the database manager if the database manager goes offline unexpectedly. For our purpose, we select option 3. Note that the failover policy is a powerful concept for larger clusters (with more nodes and more partitions), but for a our refer configuration, two-node single partition setup (such as this one) should select option 3. Now you need to configure the failover policy for the instance db2inst1. The failover policy determines the machines on which the cluster manager will restart the database manager if the database manager is brought offline unexpectedly. The following are the available failover policies: 1. Local Restart -- during failover, the database manager will restart in place on the local machine 2. Round Robin -- during failover, the database manager will restart on any machine in the cluster domain 3. Active/Passive -- during failover, the database manager will restart on a specific machine 4. M+N -- during failover, the database partitions on one machine will failover to any other machine in the cluster domain (used with DPF instances) 5. Custom -- during failover, the database manager will restart on a machine from a userspecified list Enter your selection: 3 Then db2haicu will prompt you to designate any noncritical mount points. For this case, we chose to designate only one noncritical mount point. Note that you should add any mount points to the noncritical path list that you are sure that you never want to failover. This list should include any mount points specified in /etc/filesystems that are local mount points and will never be failed over. You can identify mount points that are noncritical for failover. For more information, see the topic 'Identifying mount points that are noncritical for failover' in the DB2 Information Center. Are there any mount points that you want to designate as noncritical? [2] 1. Yes 2. No 2 The db2haicu tool will now automatically add the DB2 partition instance to the specified cluster manager at this point. Active/Passive failover policy was chosen. You need to specify the host names of an active/passive pair. Enter the host name for the active cluster node: cmaix25 Enter the host name for the passive cluster node: cmaix26 Adding DB2 database partition 0 to the cluster ... Adding DB2 database partition 0 to the cluster was successful. Copyright © 2004, 2009 IBM Page 54 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Virtual IP Address Setup Once the database partition has been added to the cluster, db2haicu will prompt you to create a virtual IP address: Do you want to configure a virtual IP address for the DB2 partition: 0? [2] 1. Yes 2. No 1 Enter the virtual IP address: 9.123.112.251 Enter the subnet mask for the virtual IP address 9.123.112.251: [255.255.255.0] 255.255.255.0 Select the network for the virtual IP 9.123.112.251: 1. db2_private_network_0 2. db2_public_network_0 Enter selection: 2 Adding virtual IP address 9.123.112.251 to the domain ... Adding virtual IP address 9.123.112.251 to the domain was successful. You must make sure that your IP address and subnet mask values are well formed and correspond with the subnet mask of the network you chose. All invalid inputs will be rejected. In such a case, you are advised to examine the IP addresses and netmasks of the NIC components of the network (using the ‘ifconfig’ command) and verify that the IP address and netmask specified are compatible with each of the NICs in the network. In addition, make sure that the IP address that you want to add is not already present on the network. All cluster configurations have been completed successfully. db2haicu exiting ... After the virtual IP address configuration, the Automated Cluster-controlled HA configuration is complete. Run command ‘lssam’ and Figure 32: db2inst1 Tivoli System Automation status is shown. Figure 32: db2inst1 Tivoli System Automation status The db2haicu Interactive Setup on Node2 As Node2 will share same Cluster, Quorum device and Network with Node1, issue “db2haicu” will go to Cluster Manager Selection directly. Copyright © 2004, 2009 IBM Page 55 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Cluster Manager Selection After the network definitions, db2haicu prompts you for the cluster manager software that you are using for the current HA setup. For our purpose, we select Tivoli System Automation: Welcome to the DB2 High Availability Instance Configuration Utility (db2haicu). You can find detailed diagnostic information in the DB2 server diagnostic log file called db2diag.log. Also, you can use the utility called db2pd to query the status of the cluster domains you create. For more information about configuring your clustered environment using db2haicu, see the topic called 'DB2 High Availability Instance Configuration Utility (db2haicu)' in the DB2 Information Center. db2haicu determined the current DB2 database manager instance is db2inst2. The cluster configuration that follows will apply to this instance. db2haicu is collecting information on your current setup. This step may take some time as db2haicu will need to activate all databases for the instance to discover all paths ... When you use db2haicu to configure your clustered environment, you create cluster domains. For more information, see the topic 'Creating a cluster domain with db2haicu' in the DB2 Information Center. db2haicu is searching the current machine for an existing active cluster domain ... db2haicu found a cluster domain called DB2Domain on this machine. The cluster configuration that follows will apply to this domain. Retrieving high availability configuration parameter for instance db2inst2 ... The cluster manager name configuration parameter (high availability configuration parameter) is not set. For more information, see the topic "cluster_mgr - Cluster manager name configuration parameter" in the DB2 Information Center. Do you want to set the high availability configuration parameter? The following are valid settings for the high availability configuration parameter: 1.TSA 2.Vendor Enter a value for the high availability configuration parameter: [1] 1 Setting a high availability configuration parameter for instance db2inst2 to TSA. Now you need to configure the failover policy for the instance db2inst2. The failover policy determines the machines on which the cluster manager will restart the database manager if the database manager is brought offline unexpectedly. Failover Policy Now you need to configure the failover policy for the instance db2inst1. The failover policy determines the machines on which the cluster manager will restart the database manager if the database manager goes offline unexpectedly. For our purpose, we select option 3. Note that the failover policy is a powerful concept for larger clusters (with more nodes and more partitions), but for a simple two-node single partition setup (such as this one), it is generally best to select option 3. The following are the available failover policies: Copyright © 2004, 2009 IBM Page 56 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 1. Local Restart -- during failover, the database manager will restart in place on the local machine 2. Round Robin -- during failover, the database manager will restart on any machine in the cluster domain 3. Active/Passive -- during failover, the database manager will restart on a specific machine 4. M+N -- during failover, the database partitions on one machine will failover to any other machine in the cluster domain (used with DPF instances) 5. Custom -- during failover, the database manager will restart on a machine from a userspecified list Enter your selection: 3 Then db2haicu will prompt you to designate any noncritical mount points. For this case, we chose to designate only one noncritical mount point. You can identify mount points that are noncritical for failover. For more information, see the topic 'Identifying mount points that are noncritical for failover' in the DB2 Information Center. Are there any mount points that you want to designate as noncritical? [2] 1. Yes 2. No 2 Active/Passive failover policy was chosen. You need to specify the host names of an active/passive pair. Enter the host name for the active cluster node: cmaix26 Enter the host name for the passive cluster node: cmaix25 Note that you should add any mount points to the noncritical path list that you are sure that you never want to failover. This list should include any mount points specified in /etc/fstab that are local mount points and will never be failed over. Adding DB2 database partition 0 to the cluster ... Adding DB2 database partition 0 to the cluster was successful. The db2haicu tool will now automatically add the DB2 partition instance to the specified cluster manager at this point. Virtual IP Address Setup Once the database partition has been added to the cluster, db2haicu will prompt you to create a virtual IP address: Do you want to configure a virtual IP address for the DB2 partition: 0? [2] 1. Yes 2. No 1 Enter the virtual IP address: 9.123.112.252 Enter the subnet mask for the virtual IP address 9.123.112.252: [255.255.255.0] Copyright © 2004, 2009 IBM Page 57 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 255.255.255.0 Select the network for the virtual IP 9.123.112.252: 1. db2_private_network_0 2. db2_public_network_0 Enter selection: 2 Adding virtual IP address 9.123.112.252 to the domain ... Adding virtual IP address 9.123.112.252 to the domain was successful. You must make sure that your IP address and subnet mask values are well formed and correspond with the subnet mask of the network you chose. All invalid inputs will be rejected. In such a case, you are advised to examine the IP addresses and netmasks of the NIC components of the network (using the ‘ifconfig’ command) and verify that the IP address and netmask specified are compatible with each of the NICs in the network. In addition, make sure that the IP address that you want to add is not already present on the network. After the virtual IP address configuration, the Automated Cluster-controlled HA configuration is complete. All cluster configurations have been completed successfully. db2haicu exiting ... DB2 and Tivoli System Automation environment acceptance testing At this point, you should take some time to validate your DB2 and Tivoli System Automation operating environment before proceeding any further. Devise tests which will mimic the way you expect to utilize DB2 and Tivoli System Automation in the context of Content Manager. For example, if you follow the methodology of this white paper, you may wish to create a sample database in each of your two DB2 instances, testing various failover scenarios. Does the failover occur as you expect it to, with MTTR consistent with your SLA’s or internal user requirements? Can you failback as you expect to? What considerations might be noteworthy as a result of a failover or failback? Are there manual steps that will need to be performed before users can regain access to the DB2 database server? Develop test scenarios for situations you think might be problematic. Develop as thorough a procedure as possible and be sure you understand all ramifications. Ultimately, the capability of Content Manager to remain highly available depends on the underlying robustness of the infrastructure implemented. If DB2 and Tivoli System Automation are not configured in a way that meets your requirements, it is unlikely that Content Manager sitting on top of this highly available environment, will be able to either. Do not underestimate the importance of testing at this stage. Ensure your DB2 and Tivoli System Automation environment are configured properly before proceeding further. IBM standard support does not include troubleshooting or advising on physical storage, network topologies, or other non-IBM high availability infrastructure components. For questions or problems with IBM-branded products used in your highly available infrastructure which are not part of Content Manager, you must contact IBM support specific to those IBM products. For example, if you suspect problems with or have questions about DB2 or Tivoli System Automation failover performance or capabilities, you should direct questions to IBM support for DB2 or Tivoli System Automation. The information below may suggest considerations in devising your DB2 and Tivoli System Automation acceptance tests. Additionally, you should refer to both DB2 and Tivoli System Automation product documentation and white papers for additional testing methodologies. Acceptance testing is highly correlated to the specific requirements domain of an organization and cannot be generalized. As such Copyright © 2004, 2009 IBM Page 58 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 testing relates to the underlying infrastructure and are not Content Manager-related, it lies outside the scope of IBM standard support. If you need additional consultancy services, please contact IBM Global Business Services or your IBM sales representative. Verify with db2pd Utility The db2pd tool is used for troubleshooting because it can return quick and immediate information from the DB2 memory sets. With “-ha” parameter, it dumps out the overall cluster state, and in doing so collects information for all cluster objects which should exist and returns the current state. For more information on the db2pd utility, please see the DB2 Information Center topic, Monitoring and troubleshooting using db2pd and db2pd - Monitor and troubleshoot DB2 database command. Here is sample after running db2pd with db2inst1: $ db2pd -ha DB2 HA Status Instance Information: Instance Name Number Of Domains Number Of RGs for instance = db2inst1 =1 =1 Domain Information: Domain Name Cluster Version Cluster State Number of nodes = DB2Domain = 2.4.10.0 = Online =2 Node Information: Node Name --------------------cmaix25 cmaix26 State ------------------Online Online Resource Group Information: Resource Group Name = db2_db2inst1_0-rg Resource Group LockState = Unlocked Resource Group OpState = Online Resource Group Nominal OpState = Online Number of Group Resources =3 Number of Allowed Nodes =2 Allowed Nodes ------------cmaix25 cmaix26 Member Resource Information: Resource Name = db2mnt-DB2INST1-rs Resource State = Online Resource Type = Mount Mount Resource Path = /DB2INST1 Number of Allowed Nodes =2 Allowed Nodes ------------- Copyright © 2004, 2009 IBM Page 59 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 cmaix25 cmaix26 Resource Name Resource State Resource Type DB2 Partition Number Number of Allowed Nodes Allowed Nodes ------------cmaix25 cmaix26 = db2_db2inst1_0-rs = Online = DB2 Partition =0 =2 Resource Name Resource State Resource Type = db2ip_9_123_112_251-rs = Online = IP Network Information: Network Name ----------------------db2_public_network_0 Number of Adapters -----------------2 Node Name ----------------------cmaix25.cn.ibm.com cmaix26.cn.ibm.com Network Name ----------------------db2_private_network_0 Adapter Name -----------------en0 en0 Number of Adapters -----------------2 Node Name ----------------------cmaix25.cn.ibm.com cmaix26.cn.ibm.com Adapter Name -----------------en1 en1 Quorum Information: Quorum Name Quorum State ------------------------------------------------------db2_Quorum_Network_9_123_112_1:19_9_50 Online Fail Offline Operator Offline And for db2inst2 user, exported result will be: $ db2pd -ha DB2 HA Status Instance Information: Instance Name Number Of Domains Number Of RGs for instance Copyright © 2004, 2009 IBM = db2inst2 =1 =1 Page 60 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX Domain Information: Domain Name Cluster Version Cluster State Number of nodes Node Information: Node Name --------------------cmaix26 cmaix25 2009.12.28 = DB2Domain = 2.4.10.0 = Online =2 State ------------------Online Online Resource Group Information: Resource Group Name = db2_db2inst2_0-rg Resource Group LockState = Unlocked Resource Group OpState = Online Resource Group Nominal OpState = Online Number of Group Resources =3 Number of Allowed Nodes =2 Allowed Nodes ------------cmaix26 cmaix25 Member Resource Information: Resource Name = db2mnt-DB2INST2-rs Resource State = Online Resource Type = Mount Mount Resource Path = /DB2INST2 Number of Allowed Nodes =2 Allowed Nodes ------------cmaix26 cmaix25 Resource Name Resource State Resource Type DB2 Partition Number Number of Allowed Nodes Allowed Nodes ------------cmaix26 cmaix25 = db2_db2inst2_0-rs = Online = DB2 Partition =0 =2 Resource Name Resource State Resource Type = db2ip_9_123_112_252-rs = Online = IP Network Information: Network Name ----------------------- Copyright © 2004, 2009 IBM Number of Adapters ------------------ Page 61 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX db2_public_network_0 Node Name ----------------------cmaix25.cn.ibm.com cmaix26.cn.ibm.com Network Name ----------------------db2_private_network_0 Node Name ----------------------cmaix25.cn.ibm.com cmaix26.cn.ibm.com 2009.12.28 2 Adapter Name -----------------en0 en0 Number of Adapters -----------------2 Adapter Name -----------------en1 en1 Quorum Information: Quorum Name Quorum State ------------------------------------------------------Operator Offline Fail Offline db2_Quorum_Network_9_123_112_1:19_9_50 Online Verify environment during runtime Throughout the runtime of the db2 engine, any time a call is required to the backend HA layer, an implicit validation of the object model takes place. For example, any time a new disk mount is added (e.g. new tablespace) the HA mount resource is created in the cluster and validated. In a clustered environment, some database manager instance configuration and administration operations require related cluster configuration changes. The DB2 High Availability (HA) Feature enables the database manager to automatically request cluster manager configuration changes whenever you perform certain database manager instance configuration and administration operations. When you perform the following database manager instance configuration and administration operations, the database manager automatically performs related cluster manager configuration for you: Starting a database using START DATABASE or db2start Stopping a database using STOP DATABASE or db2stop Creating a database using CREATE DATABASE Adding storage using CREATE TABLESPACE Removing storage using ALTER TABLESPACE DROP or DROP TABLESPACE Dropping a database using DROP TABLESPACE Restoring a database using the RESTORE DATABASE or db2Restore Specifying the table space containers for redirected restore using SET TABLESPACE CONTAINERS Rolling a database forward using ROLLFORWARD DATABASE or db2Rollforward Recovering a database using RECOVER DATABASE or db2Recover Creating event monitors using CREATE EVENT MONITOR Dropping event monitors using DROP EVENT MONITOR Creating and altering external routines using: CREATE PROCEDURE CREATE FUNCTION Copyright © 2004, 2009 IBM Page 62 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 CREATE FUNCTION CREATE METHOD ALTER PROCEDURE ALTER FUNCTION ALTER METHOD Dropping external routines using: DROP PROCEDURE DROP FUNCTION DROP METHOD Start DB2 High Availability Disaster Recovery (HADR) operations for a database using START HADR Stop HADR operations for a database using STOP HADR Cause a HADR standby database to take over as a HADR primary database using TAKEOVER HADR Setting the database manager configuration parameter DIAGPATH or SPM_LOG_PATH Setting the database configuration parameter NEWLOGPATH, OVERFLOWLOGPATH, MIRRORLOGPATH, or FAILARCHPATH Dropping a database manager instance using db2idrop Verify with db2haicu utility in Maintenance Mode When you run DB2 High Availability Instance Configuration Utility (db2haicu) and there is already a cluster domain created for the current database manager instance, db2haicu operates in maintenance mode. When db2haicu is running in maintenance mode, db2haicu presents you with a list of configuration and administration tasks that you can perform. This may be helpful if your acceptance testing uncovers errors in your DB2 and Tivoli System Automation high availability infrastructure configuration. db2haicu maintenance tasks include adding cluster elements such as databases or cluster nodes to the cluster domain, and removing elements from the cluster domain. db2haicu maintenance tasks also include modifying the details of cluster domain elements such as the failover policy for the database manager instance. For more information about the db2haicu utility, please refer to the DB2 Information Center topic, Maintaining a cluster domain using DB2 High Availability Instance Configuration Utility (db2haicu). $ db2haicu Welcome to the DB2 High Availability Instance Configuration Utility (db2haicu). You can find detailed diagnostic information in the DB2 server diagnostic log file called db2diag.log. Also, you can use the utility called db2pd to query the status of the cluster domains you create. For more information about configuring your clustered environment using db2haicu, see the topic called 'DB2 High Availability Instance Configuration Utility (db2haicu)' in the DB2 Information Center. db2haicu determined the current DB2 database manager instance is db2inst1. The cluster configuration that follows will apply to this instance. db2haicu is collecting information on your current setup. This step may take some time as db2haicu will need to activate all databases for the instance to discover all paths ... When you use db2haicu to configure your clustered environment, you create cluster domains. For Copyright © 2004, 2009 IBM Page 63 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 more information, see the topic 'Creating a cluster domain with db2haicu' in the DB2 Information Center. db2haicu is searching the current machine for an existing active cluster domain ... db2haicu found a cluster domain called DB2Domain on this machine. The cluster configuration that follows will apply to this domain. Select an administrative task by number from the list below: 1. Add or remove cluster nodes. 2. Add or remove a network interface. 3. Add or remove a highly available database. 4. Add or remove a mount point. 5. Add or remove an IP address. 6. Add or remove a non-critical path. 7. Move DB2 database partitions and HADR databases for scheduled maintenance. 8. Change failover policy for this instance. 9. Create a new quorum device for the domain. 10. Destroy the domain. 11. Exit. Enter your selection: 11 All cluster configurations have been completed successfully. db2haicu exiting ... Verifying the Tivoli System Automation environment Tivoli System Automation monitoring tools are available via command-line or GUI interfaces such as the Integrated Solutions Console. Refer to the Tivoli software Information Center topic, End-to-End Automation Management Administrator's and User's Guide. Copyright © 2004, 2009 IBM Page 64 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 IBM WebSphere Application Server In this chapter, we will install IBM WebSphere Application Server Network Deployment 6.1. WebSphere Application Server Network Deployment is the edition of WebSphere Application Server designed to provide high availability and load balancing features. We chose to implement a WebSphere Application Server Network Deployment environment consisting of: 2x IBM HTTP Server, each with WebSphere Application Server Network Deployment plug-in 2x Application Server, running as a WebSphere Application Server Network Deployment cluster 1x WebSphere Application Server Network Deployment Deployment Manager The redundant components (IBM HTTP Server, WebSphere Application Server plug-in, Application Server) will be distributed across both nodes. The Deployment Manager node will not be a highly available configuration because it is only needed for administration. Install WebSphere Application Server Network Deployment In this first step, we will install the application server component of IBM WebSphere Application Server Network Deployment Version 6.1. You have to complete these steps on all nodes of your planned WebSphere Application Server Network Deployment cluster. Start the WebSphere Application Server Network Deployment 6.1.0.0 installation: # cd /nfs/software/was/WAS # ./install ● Welcome Screen → Next ● License Agreement → Review the license agreement. If you agree, select “I agree...” → Next ● System Prerequisite Check: This check fails on our servers, as shown in Figure 33: WebSphere Application Server Network Deployment Prerequisite Check. We will ignore this error. Figure 33: WebSphere Application Server Network Deployment Prerequisite Check Copyright © 2004, 2009 IBM Page 65 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Click Next. ● Install Sample Applications: We do not want to use the sample applications, so keep the selection disabled. Click Next. ● Installation Directory: Confirm the default installation directory of /usr/IBM/WebSphere/AppServer → Next ● WebSphere Application Server Environments: Select None as shown in Figure 34: WebSphere Application Server Network Deployment Environments. We will create the profiles later using the profile management tool. Figure 34: WebSphere Application Server Network Deployment Environments Click Next. ● A warning as shown in Figure 35: WebSphere Application Server Profile Warning appears. Confirm the message by clicking on Yes. Figure 35: WebSphere Application Server Profile Warning ● The summary is displayed. → Next ● Installation Result: Uncheck the option to create a new profile. We will run this script later manually. Click Finish to close the installation wizard. Copyright © 2004, 2009 IBM Page 66 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 IBM HTTP Server and IBM HTTP Server Plug-in Installation In this step, we will install the IBM HTTP Server component and the IBM HTTP Server Plug-in of IBM WebSphere Application Server Version 6.1. You have to complete these steps on all nodes of your planned WebSphere Application Server Network Deployment cluster. Start the IBM HTTP Server 6.1.0.0 installation: # cd /nfs/software/was/IHS # ./install ● On the welcome screen, click Next. ● Review the license Agreement. If you agree, select Accept. → Next ● A system prerequisite check will be completed. The result will be failed as shown in Figure 36: IBM HTTP Server System Prerequisites Check. Figure 36: IBM HTTP Server System Prerequisites Check We will ignore this error. → Next ● Installation Directory: Confirm the default installation directory of /usr/IBM/HTTPServer → Next ● Keep the default port value assignments of 80 and 8008 as shown in Figure 37: IBM HTTP Server Port Values Assignment. Copyright © 2004, 2009 IBM Page 67 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 37: IBM HTTP Server Port Values Assignment Click Next. ● On the next panel, HTTP Administration Server Authentication: Provide a user ID and a password if necessary, for example wasadmin and its password. Click Next. ● On the panel “Setup HTTP Administration Server” uncheck both options as shown in Figure 38: IBM HTTP Server Setup HTTP Administration Server. We will not use the administration interface. Figure 38: IBM HTTP Server Setup HTTP Administration Server Click Next. Copyright © 2004, 2009 IBM Page 68 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 ● The next panel asks for details about the IBM HTTP Server Plug-in for WebSphere Application Server. On the first node, keep the default values as shown in Figure 39: IBM HTTP Server Plug-In for WebSphere Application Server Figure 39: IBM HTTP Server Plug-In for WebSphere Application Server On the second node cmaix26, change the web server definition to “webserver2”. Click Next. ● The summary is displayed, as shown in Figure 40: IBM HTTP Server Installation Summary. Figure 40: IBM HTTP Server Installation Summary Click Next. ● Installation Result → Finish Copyright © 2004, 2009 IBM Page 69 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Applying Fixpacks Next, we will update the WebSphere Application Server and Edge components to a higher fixpack level. We used fixpack 13 for our tests. You will need to download 4 components for WebSphere fixpack installation: ● IBM Update Installer ● Fix pack for WebSphere Application Server ● Fix pack for IBM HTTP Server ● Fix pack for IBM HTTP Server Plug-In First, install the update installer (please notice that update installer should be the version delivered with the fixpack, in our tests, fixpack 13): ● Welcome → Next ● License Agreement → Review the agreement. If you agree, select Accept. → Next ● System Prerequisite Check: Completed Failed → Next ● Installation Directory: Confirm /usr/IBM/WebSphere/UpdateInstaller → Next ● Installation Summary → Next ● Installation Complete: Keep checkbox selected as shown in Figure 41: Update Installer Installation Result. → Finish Figure 41: Update Installer Installation Result The IBM Update Installer is started and will appear on your screen after a few seconds. ● Welcome → Next Copyright © 2004, 2009 IBM Page 70 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 ● Product Selection: In the drop down list, you will find 3 components: HTTP Server HTTP Plug-in WebSphere Application Server Select one of the components. Click Next. ● Maintenance Option Selection: Install maintenance package → Next ● Specify Directory → Select the directory where you did download the patches. Click Next. ● Select packages to install as shown in Figure 42: Update Available Maintenance Packages → Next Figure 42: Update Available Maintenance Packages ● Summary → Next ● Installation Complete: Relaunch the wizard for the other 2 components. If you did update all 3 components, click Finish. WebSphere Application Server Profile Management Tool Now we will create the necessary profiles in WebSphere Application Server. On one node, we will create the deployment manager to serve as the administrative interface for all our WebSphere Application Server Network Deployment cell clusters. In our scenario, the deployment manager will be created on cmaix26. On all nodes we will create a custom node, which we will federate into our deployment manager environment. A custom node does not contain any administrative interface. Start the profile management tool on the node, where you want to create the deployment manager. Complete these steps on one node only! # cd /usr/IBM/WebSphere/AppServer/bin/ProfileManagement # ./pmt.sh Copyright © 2004, 2009 IBM Page 71 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 ● Welcome → Next ● Select the type of environment to create: Select Deployment Manager as shown in Figure 43: Profile Management Tool - Environments. Figure 43: Profile Management Tool - Environments ● Profile Creation Options: Keep typical. → Next ● Administrative Security (shown in Figure 44: Profile Management Tool - Administrative Security). Specify a user name and its password to access any WebSphere Application Server Network Deployment administrative function. This user is maintained within WebSphere and is by default not an operating system user. Copyright © 2004, 2009 IBM Page 72 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 44: Profile Management Tool - Administrative Security ● Profile Creation Summary: Review the summary as shown in Figure 45: Profile Management Tool Deployment Manager Summary. Figure 45: Profile Management Tool - Deployment Manager Summary Copyright © 2004, 2009 IBM Page 73 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Click Create. ● Profile created successful. Uncheck Launch the First Step Console. Click Finish to close the profile management tool. Now, start the Deployment Manager from a terminal: # cd /usr/IBM/WebSphere/AppServer/bin # ./startManager.sh Start the profile management tool one node. Complete these steps on all nodes you want to participate in the WebSphere Application Server Network Deployment cluster! # cd /opt/IBM/WebSphere/AppServer/bin/ProfileManagement # ./pmt.sh ● Welcome → Next ● Select the type of environment to create: Select “Custom profile” → Next ● Profile Creation Options: Keep typical. → Next ● Federation: On this panel, you define if the custom node should become managed by a deployment manager. Provide the deployment manager host name. Keep the default port number. If you have enabled administrative security for the deployment manager, specify the user name and its password as shown in Figure 46: Profile Management Tool - Federation. Figure 46: Profile Management Tool - Federation → Next Copyright © 2004, 2009 IBM Page 74 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 ● Profile Creation Summary is displayed. Review the settings, as shown in Figure 47: Profile Management Tool - Custom Profile Summary. Figure 47: Profile Management Tool - Custom Profile Summary → Create ● Profile created successful. Uncheck Launch the First Step Console. Finish. Repeat the custom profile creation on all nodes you want to participate in your WebSphere Application Server Network Deployment cluster. Create WebSphere Application Server Cluster In this step, we will create a cluster between both of our application server nodes. This configuration can be accomplished in the WebSphere administration interface, Integrated Solutions Console. ● Expand Servers in the left navigation panel. ● Click on Clusters. ● In the right panel, “Server Clusters”, click New. This will start the Create a new cluster wizard, which you will complete in the next steps. ● First, you need to provide a name for your cluster. You need to remember this name later on for the deployment of the resource manager application of Content Manager. Type in the name of your cluster (for example, RMCluster). Keep the other settings on default, as shown in Figure 48: WebSphere Application Server Create a new cluster (1). Click Next. Copyright © 2004, 2009 IBM Page 75 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 48: WebSphere Application Server Create a new cluster (1) ● On the next panel, create first cluster member, you are required to provide information about your first member and on a template. You can choose the member name freely. In our case, as shown in Figure 49: WebSphere Application Server Create a new cluster (2), we've chosen a member name as cmaix25, indicating that this application server is assigned to the node cmaix25Node01. In the drop down list below, select the appropriate node. Keep the default weight of 2. With the weight parameter, you can influence the load balancing behavior. Having an equal weight across all nodes will provide an equal balancing. Copyright © 2004, 2009 IBM Page 76 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 49: WebSphere Application Server Create a new cluster (2) For “basis for first cluster” member, keep the default selection of “Create the member using an application server template” - default. Click Next. ● On this panel, as shown in Figure 50: WebSphere Application Server - Create a new cluster (3), you can add more nodes to the cluster. Type in the member name (for example cmaix26), select the node and weight, keep “generate unique HTTP ports” checked and click Add member. On the lower section of this panel, a list of all nodes will be populated. When you are done defining all nodes, click Next. Copyright © 2004, 2009 IBM Page 77 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 50: WebSphere Application Server - Create a new cluster (3) ● The next panel, as shown in Figure 51: WebSphere Application Server - Create a new cluster (Summary), is a summary of your settings. Click Finish to start the creation of the new cluster. Copyright © 2004, 2009 IBM Page 78 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 51: WebSphere Application Server - Create a new cluster (Summary) ● Finally you have to save your settings. A warning like shown in Figure 52: WebSphere Application Server - Save Changes is displayed. Click on Save to store the configuration and to populate the information to the nodes across your WebSphere Application Server Network Deployment cluster. Copyright © 2004, 2009 IBM Page 79 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 52: WebSphere Application Server - Save Changes Create WebSphere Application Server JMS data source The WebSphere Administration API uses Java Messaging Service (JMS) for application event notification when certain device and job events occur. By default, the Device Manager server uses the Service Integration Bus (“SIB”) feature, which was introduced in WebSphere Application Server, version 6, along with the default messaging JMS provider. Refer to the WebSphere Application Server Information Center for information on the Service Integration Bus and messaging resources. To prevent the SIB database from being a SPOF in an HA environment, this database can be created in the HA-enabled DB2 instance of either db2inst1 or db2inst2. Since the Content Manager resource manager database typically has a lighter runtime work load than the Content Manager library server database, we suggest creating the SIB database on db2inst2. ● Create an empty database on db2inst2 for JMS datasource. This database will be configured automatically during Content Manager resource manager application configuration. db2inst2@cmaix26 # db2 create database SIBDB ● In Security → Security administration, applications, and infrastructure, click Java Authentication and Authorization Services → J2C authentication data as Figure 53: Secure administration, application, and infrastructure, then select ‘New’ to create a new J2C Authentication. Copyright © 2004, 2009 IBM Page 80 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 53: Secure administration, application, and infrastructure ● For improved security, we suggest creating another DB2 user for this database. In our tests, for simplicity, we re-use db2inst2 to manage this database as in Figure 54: New J2C user. Then click “OK” and “Save”. Figure 54: New J2C user ● In Resources → JDBC →JDBC Provides, select Scope to be “Cluster=RMCluster”, and click “New”. Copyright © 2004, 2009 IBM Page 81 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 ● For Figure 55: Create new JDBC provider, select Database type to be “DB2”, Provider type to be “DB2 Universal JDBC Driver Provider” and Implementation type “Connection pool data source”. Click “Next”. Figure 55: Create new JDBC provider ● Use $IBMCMROOT/lib as the database class path as Figure 56: Database class path information, and click “Next”. Copyright © 2004, 2009 IBM Page 82 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 56: Database class path information ● Check summary and click “Finish” and then “Save”. ● In Resources → JDBC →Data sources, select Scope to be “Cluster=RMCluster”, and click “New”. The WebSphere Administration API uses Java Messaging Service (JMS) for application event notification when certain device and job events occur. By default, the Device Manager server uses the Service Integration Bus (“SIB”) feature, which was introduced in WebSphere Application Server, version 6, along with the default messaging JMS provider. Refer to the WebSphere Application Server Information Center for information on the Service Integration Bus and messaging resources. Content Manager requires a messaging database in place, to allow the resource manager to exchange information between both nodes for the resource manager services workload: replicator, migrator, purger, and stager. To prevent the SIB database from being a SPOF in an HA environment, we want to place this information in our DB2 database environment. ● In Figure 57: Create basic data source information, input SIBDB_datasource as “Data source name”, jdbc/sibdb as ‘JNDI name”, and select SIBDB_User for “Component-managed authentication alias and XA recovery authentication alias”. Please remember this JNDI name; we will use it later during Content Manager resource manager application configuration. Click “Next”. Copyright © 2004, 2009 IBM Page 83 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 57: Create basic data source information ● For Figure 58: Select JDBC provider, choose “Select an existing JDBC provider”, and choose “DB2 Universal JDBC driver Provider” that we created in previous step. Click “Next”. Figure 58: Select JDBC provider Copyright © 2004, 2009 IBM Page 84 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 ● In Figure 59: Create a data source, input SIBDB for “Database name”, and select Driver type to be 4. Use Virtual IP and port of db2inst2 for server name and port number. Remove “Use this data source in container managed persistence (CMP)”, then “Next”. Figure 59: Create a data source ● View the summary and click “Finish”, then “Save”. ● In Environment →WebSphere Variables as Figure 60: WebSphere Variables, input $IBMCMROOT/lib as DB2UNIVERSAL_JDBC_DRIVER_PATH for all cluster and nodes. Copyright © 2004, 2009 IBM Page 85 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 60: WebSphere Variables ● System administration →Node agents, select both node agent and restart them as Figure 61: Restart Node Agents. Figure 61: Restart Node Agents ● Same with Figure 62: Nodes Synchronize, In System administration →Nodes, select cmaix25Node01 and cmaix26Node02, select “Synchronize”. Copyright © 2004, 2009 IBM Page 86 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 62: Nodes Synchronize ● Now we can test connection for SIBDB that we created in datasource as Figure 63: Test connection for SIBDB datasource. In Resource → JDBC → Data sources, select SIBDB_datasource, and click “Test connection”. It should display successful in messages. Figure 63: Test connection for SIBDB datasource Add Web Server Definition In this step, we will add our web server definition into our deployment manager. While still in the Integrated Solutions Console, navigate in the left panel by expanding Servers → Web Servers. Click on new. A panel as shown in Figure 64: New Web Server Definition (1) appears. Select your first node from the drop down list and type in the web server name. The first node is cmaix25Node01 and the web server name is webserver1 (In our case, it’s webserver25). Second node will be cmaix26Node01 with a web server name of webserver2 (In our case, it’s webserver26). Copyright © 2004, 2009 IBM Page 87 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 64: New Web Server Definition (1) On the next panel, shown in Figure 65: New Web Server Definition (2), you specify the web server template. We use the IBM HTTP Server (IHS). Figure 65: New Web Server Definition (2) The next panel, shown in Figure 66: New Web Server Definition (3), queries information about the web server installation location and its plug-in. In our case, all settings have been correct. Copyright © 2004, 2009 IBM Page 88 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 66: New Web Server Definition (3) In the last step, shown in Figure 67: New Web Server Definition (4), a summary is displayed. Review the settings and click finish to create the definitions. Now review the same steps for the second IBM HTTP Server you have installed. Figure 67: New Web Server Definition (4) Finally, you must save your changes. On the upper part of you screen, you should see the information shown in Figure 68: New Web Server Definition – Save Changes. Click on the Save link embedded in this message. Copyright © 2004, 2009 IBM Page 89 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 68: New Web Server Definition – Save Changes Finally, you can start the IBM HTTP Server from the WebSphere Application Server Integrated Solution Console. After repeating previous steps to set another webserver definition, select both web servers (check the two check boxes) and then click the Start button. As a result, both servers should be started and a panel similar to Figure 69: Web Server - Start will be displayed. Notice, that the status should now be started (as indicated by a green icon, instead of the red one as shown in the figure below) for both servers and a success message will be displayed above the table. Figure 69: Web Server - Start SSL configuration for http server (Optional) By default, the web servers that we installed only support http with port 80. Content Manager needs https support in some cases. As in Content Manager 8.4.2, https is only used for System Administration management by default. This will be optional and will not be described in this document. You can use default https port on any node or define IP sprayer to map to default https port directly. For the purposes of testing only, we chose to bypass the use of https. Instead, we will change the port number for https communication to bypass the IBM HTTP Server and talk directly to WebSphere Application Server. Note that we do not recommend this for production. Copyright © 2004, 2009 IBM Page 90 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Open the Content Manager System Administration Client and expand Resource Manager. Right click on our resource manager “icmrm” and select properties. Change the port number for the https protocol to the one used by your WebSphere Application Server, most likely 9443. See Figure 70: Update Resource Manager- Resource Manager Properties. Figure 70: Update Resource Manager- Resource Manager Properties The drawback from this implementation is that, for administration, no load balancing will take place and no high availability is configured. Since Content Manager administration is typically less frequent, this might be acceptable. IP Sprayer Definition As we have two web servers, we must define an IP sprayer to use both of them. Please see your IBM WebSphere Application Server representative for the IP sprayer solution. We’ll not talk technology here, and just assume that IP address of cmrmapp is already assigned to this IP sprayer. WebSphere Application Server and WebSphere Application Server Network Deployment environment acceptance testing As before with the databases, you should take some time to validate your WebSphere Application Server and WebSphere Application Server Network Deployment cluster environment. Devise tests which will Copyright © 2004, 2009 IBM Page 91 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 mimic the way you expect to utilize the WebSphere cluster in the context of Content Manager. For example, if you follow the methodology of this white paper, you may wish to deploy a sample application to your WebSphere Application Server Network Deployment cell, testing various failover scenarios. Does the failover occur as you expect it to, with MTTR consistent with your SLA’s or internal user requirements? Can you failback as you expect to? What considerations might be noteworthy as a result of a failover or failback? Are there manual steps that will need to be performed before users can regain access to the application? What about the IBM HTTP Server? If it is configured as a SPOF, what additional recovery procedures do you need to develop? If it isn’t, what redundancies, and checks on those redundancies, have you put into place to mitigate outages? Develop test scenarios for situations you think might be problematic. Develop as thorough a procedure as possible and be sure you understand all ramifications. As with the databases, the capability of Content Manager to remain highly available depends on the underlying robustness of the infrastructure implemented. If WebSphere is not configured in a way that meets your requirements, it is unlikely that the Content Manager RM application sitting on top of the WebSphere environment, will be able to either. Do not underestimate the importance of testing at this stage. Ensure your WebSphere Application Server and WebSphere Application Server Network Deployment environment are configured properly before proceeding further. IBM standard support does not include troubleshooting or advising on physical storage, network topologies, or other non-IBM high availability infrastructure components. For questions or problems with IBM-branded products used in your highly available infrastructure which are not part of Content Manager, you must contact IBM support specific to those IBM products. For example, if you suspect problems with or have questions about WebSphere Application Server Network Deployment failover performance or capabilities, you should direct questions to IBM support for WebSphere Application Server Network Deployment. Refer also to the WebSphere Application Server Information Center for more details on WebSphere Application Server and WebSphere Application Server Network Deployment. In particular, the topics, Monitoring overall system health and Using the installation verification tools may prove helpful in diagnosing problems. Acceptance testing is highly correlated to the specific requirements domain of an organization and cannot be generalized. As such testing relates to the underlying infrastructure and are not Content Manager-related, it lies outside the scope of IBM standard support. If you need additional consultancy services, please contact IBM Global Business Services or your IBM sales representative. Copyright © 2004, 2009 IBM Page 92 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Content Manager In this chapter we will install and configure Content Manager. Steps to complete are: ● installation of Content Manager on both nodes ● Configure Content Manager library server database on Node A ● Configure Content Manager resource manager database on Node B ● Deploy Resource Manager Application on WebSphere Application Server Network Deployment Cluster Further we will configure a remote client. Pre-Install Validation Before continuing further, ensure that you have performed thorough acceptance testing for your infrastructure at this time. This should include DB2, Tivoli System Automation, and WebSphere Application Server, and the rest of your HA infrastructure. As mentioned in previous sections, it is the infrastructure itself that provides HA capabilities. If failover at the infrastructure level is not behaving as you expect, Content Manager will not be able to either. The risk of inadequate acceptance testing is that Content Manager servers are not highly available, do not conform to your MTTR requirements, and/or result in unanticipated dissynchronization points within your infrastructure that require additional manual recovery. Installation The installation of Content Manager is goes as follows. Unzip the installation source and run the command install_CM as root. You will complete these steps on both of two nodes. Then you can configure different component on different node, as described in the architecture. After executing install_CM, the installation wizard will appear. Read the license agreement and select “Accept” if you agree to it. Click Next. The installation destination is displayed. Note, that on UNIX environments, the installation target is predefined as /opt/IBM/db2cmv8. Click Next. The product component list is displayed. Select the components you want to install. We kept the default selection of all components. Mandatory components are the library server, resource manager database the resource manager application. Optional components are the system administration client and the information center. Click Next. On the next panel, you need to select the server type. Select DB2 and click Next. The summary panel is displayed, similar to our as shown in Figure 71: Content Manager Installation Summary. Verify the information displayed in the summary and click Next to start the installation. Copyright © 2004, 2009 IBM Page 93 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 71: Content Manager Installation Summary After a short time, the installation should complete. You are asked, if you want to run the configuration wizard. De-select this option, as we are not ready to complete this step yet. Initial Configuration for Library Server Database In this step, we will use the Content Manager Configuration wizard to complete our setup on Library Server database. Run config_CM in /opt/IBM/db2cmv8/bin directory. After a few seconds, the Content Manager configuration wizard appears. On the first panel, as shown in Figure 72: Configuration Options, select “Add configuration”. Copyright © 2004, 2009 IBM Page 94 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 72: Configuration Options Note that you can start the configuration wizard several times. You can also use the configuration wizard to remove existing configuration. On the next panel, like the one shown in Figure 73: Component Selection, you can select the components you want to configure. You may select components individually or you may just select all components at once. Figure 73: Component Selection Copyright © 2004, 2009 IBM Page 95 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Since we want to configure the library server and the system administration client, we kept only Library server and System administration client selected. Now you need to select the database type in Figure 74: Database Type. We want to use DB2 as the database. Figure 74: Database Type Next, you are required to provide information about an administrative user in Figure 75: Administrative User. This user is required to store the log and configuration files of Content Manager and is also used during the installation process. Copyright © 2004, 2009 IBM Page 96 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 75: Administrative User We have already created this user and the associated group, so we provide this information as shown above. In the next panel, as shown in Figure 76: System Information, you need to provide the fully qualified host name. This value is normally detected automatically and prepopulated into this field. Also, we can specify cmlsdb.cn.ibm.com for LS hostname. Figure 76: System Information Copyright © 2004, 2009 IBM Page 97 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 The next panel, shown in Figure 77: DB2 Database Product Directory, ask about DB2 product directory. For DB2 v9.5, default installation directory is /opt/IBM/db2/V9.5. Figure 77: DB2 Database Product Directory On the next panel, as shown in Figure 78: Library Server Database, you need to select Create new database as we only have DB2 instance configured. Figure 78: Library Server Database The information you provide on the next panel, shown in Figure 79: Library Server Database Information, is used to connect to the library server database at install time. Copyright © 2004, 2009 IBM Page 98 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 79: Library Server Database Information Note, icmadmin user is already created, and database should use port 50001 as we defined. The next panel, shown in Figure 80: DB2 Database instance, will detect the instance name to use. As db2inst1 is created on cmaix25 and HA configuration is completed, we will just use it. Figure 80: DB2 Database instance The next screen, shown is Figure 81: Library Server Database, you need to provide database connection user ID and password. This user, unlike the administrative user defined before, does not have any special privileges on the database. Copyright © 2004, 2009 IBM Page 99 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 81: Library Server Database We can use predefined database user icmconct user to finish this step. The next screen, shown in Figure 82: Library Server Database Settings, allows you to define additional settings for the library server. Keep the library server ID as 1. You may change the transaction duration, which is by default 3 minutes. At the moment, we will keep default selections, all features can be enabled after installation, if necessary. Figure 82: Library Server Database Settings Copyright © 2004, 2009 IBM Page 100 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 The next panel, Figure 83: Library Server Connection, prompts you for the resource manager database information, we use cmrmdb.cn.ibm.com as hostname, which is mapped to RM database virtual IP. Figure 83: Library Server Connection The next panel, in Figure 84: Library Server Connection (cont.), collects the information for Resource Manager application in Websphere Application Server. We use cmrmapp.cn.ibm.com for the hostname, and use 80 for Resource manager Web application port, 443 for Resource manager secure Web application port. You should use IP sprayer hostname (‘cmrmapp’ in our sample) in order to get connection to both two web servers and two Websphere Application Servers. Copyright © 2004, 2009 IBM Page 101 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 84: Library Server Connection (cont.) The next few panels, starting with Figure 85: System Administration Client, collect information for client connection, as for example used by the Content Manager System Administration Client. On this first panel, keep the default selection of local. This select stored configuration information locally to /home/ibmcmadm/cmgmt/connectors. Alternatively you could place this information remotely on an HTTP or LDAP server. Figure 85: System Administration Client Copyright © 2004, 2009 IBM Page 102 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 86: System Administration Client Connection prompts for information of library server. We use icmnlsdb for database name and cmlsdb.cn.ibm.com as LS hostname. Figure 86: System Administration Client Connection Now we have provided all necessary information for cmaix25 and Figure 87: Start Configuration is displayed. When you click on Next, the configuration settings are applied, the LS database will be created. Figure 87: Start Configuration Copyright © 2004, 2009 IBM Page 103 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 The configuration will run for several minutes. First, it will create necessary table for the library server, and next create the access modules. All actions are tracked into the configuration log file. Figure 88: Content Manager Configuration Complete shows the expected end of the configuration phase. Click finish to close the wizard. Figure 88: Content Manager Configuration Complete After configuration finish on first node cmaix25, we need to deploy the necessary code of Content Manager onto the second node cmaix26, we need copy mandatory files after installation. Make sure that you preserve ownership and access permissions during the copy operation! You need to copy the following directories to the node cmaix26: /opt/IBM/db2cmv8/lib/lib32/ICMNLSSP /opt/IBM/db2cmv8/lib/lib64/ICMNLSSP /opt/IBM/db2cmv8/lib/ICMNLSSP /opt/IBM/db2cmv8/lib/ICMNLSUF /opt/IBM/db2cmv8/lib/ICMNWFSP /etc/rc.cmlsproc Note: Any time you apply a fixpack to the primary node, you MUST copy these files again to all other nodes in your cluster! Content Manager is NOT aware of this manual copy! On the second node cmaix26, you must also manually create the directories used by Content Manager for its log files as ibmcmadm user. If this directory does not exists, no log file will be written on your secondary node. In the commands below, replace ICMNLSDB with the name of your library server database. # su - ibmcmadm $ mkdir -p log/ls/ICMNLSDB $ chmod -R 777 log Copyright © 2004, 2009 IBM Page 104 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Alternatively you may copy the whole directory tree structure, starting at /home/ibmcmadm/log, from your primary node to your secondary node. Initial Configuration for Resource Manager On the other node, cmaix26, we will setup Resource Manager Component only. Now we run config_CM on /opt/IBM/db2cmv8/bin to start Content Manager Configuration wizard as shown in Figure 89: Configure . Figure 89: Configure Content Manager We select Resource manager database and Resource manager application in Figure 90: Product components. Copyright © 2004, 2009 IBM Page 105 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 90: Product components Same with steps on Library Server database configuration, we select DB2 in Figure 91: Server Database Type. Figure 91: Server Database Type And also use existing user ibmcmadm in Figure 92: Administrative user. Copyright © 2004, 2009 IBM Page 106 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 92: Administrative user For hostname in Figure 93: System Information, we use cmrmdb.cn.ibm.com or Resource Manager Database virtual IP. Figure 93: System Information Also, use default DB2 product directory in Figure 94: DB2 Database Product Directory. Copyright © 2004, 2009 IBM Page 107 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 94: DB2 Database Product Directory In Figure 95: Resource Manager Database, select Create new database. Figure 95: Resource Manager Database Also, input Resource manager database name in Figure 96: Resource Manager Database Information. We used predefined user rmadmin and port 50002 for this step. Copyright © 2004, 2009 IBM Page 108 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 96: Resource Manager Database Information DB2 database instance db2inst2 should be used in Figure 97: DB2 Database instance. Figure 97: DB2 Database instance Next, you are prompted for information about your WebSphere Application Server environment. You have to specify the installation directory and the profile home directory. Since we are using a network deployment configuration, the profile home directory is the directory of the deployment manager. Typical values are shown in Figure 98: Resource Manager Application. If you’ve enabled Copyright © 2004, 2009 IBM Page 109 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 application server security during the installation of WebSphere Application Server, also check the check box. Figure 98: Resource Manager Application Next, you are prompted to complete information about the resource manager application, as shown in Figure 99: Resource Manager Application (cont). Most of the time you should leave the application name and application context as its default values. Changes are only required if you want to deploy multiple resource manager applications on the same server. Also keep the “Specify the JDBC path for the WebSphere Application Server variable” checked. Copyright © 2004, 2009 IBM Page 110 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 99: Resource Manager Application (cont) The next panel will prompt you for this variable value. Specify the directory to the Content Manager JAVA directory. The value shown in Figure 100: WebSphere Variables fits our Content Manager installation. You can change this value after the configuration completes in the WebSphere Application Server administrative console. Figure 100: WebSphere Variables Copyright © 2004, 2009 IBM Page 111 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Next, the Content Manager configuration wizard will connect to you WebSphere installation to detect application server and web server information. This might take several seconds to complete while it validates a large number of the parameters you have specified on the previous panels. The Content Manager configuration wizard detects, that you want to have a WebSphere Application Server Network Deployment environment and prompts if we want to deploy the resource manager application ingot a cluster or a stand-alone application server. Select cluster, as shown in Figure 101: Resource Manager Application Deployment Target. Figure 101: Resource Manager Application Deployment Target The next panel, shown in Figure 102: Cluster Information for the Resource Manager Application, lists all detected WebSphere Application Server clusters. Select to cluster you’ve created for the resource manager application. The second field queries for the messaging database. We can input the value as we defined in WebSphere Application Server for jdbc/sibdb. Copyright © 2004, 2009 IBM Page 112 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 102: Cluster Information for the Resource Manager Application Next, you are prompted to complete connection information for the Resource Manager application. Here you have several options. For the host name, we suggest you use the virtual alias of your IP/HTTP sprayer. For the HTTP port, keep 80. This will use the IBM HTTP server and its load balancing features embedded in the WebSphere Application Server plug-in. The same applies for the HTTPS port: for now, keep the default port of 443, or change it to other port that we used to route to 9043 port. Figure 103: HTTP Connection Information for the Resource Manager Application shows typical values. Figure 103: HTTP Connection Information for the Resource Manager Application Copyright © 2004, 2009 IBM Page 113 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 104: Resource Manager Application Connections need information of Library server. Give LS virtual hostname and port 50001. Figure 104: Resource Manager Application Connections The input Library Server administration ID and password in Figure 105: Resource Manager Application Connection (cont) Figure 105: Resource Manager Application Connection (cont) Now we can wait until Configuration finish as in Figure 106: Content Manager Configuration Complete. Copyright © 2004, 2009 IBM Page 114 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 106: Content Manager Configuration Complete WebSphere Data source pretest configuration This is additional Configuration for HA consideration. After Content Manager resource manager application configuration, We can see two additional data sources named icmrm_LS_database and icmrm_database. Because Resource Manager will get connection from WebSphere Connection pool, and won’t check connection validation for performance consideration. User may meet many failed transactions during Database fail over. To eliminate this issue, we can update Data source properties. In Resources → JDBC → Data sources as Figure 107: Data source properties, select icmrm_LSdatabase as in . Select “WebSphere Application Server data source properties”. Figure 107: Data source properties Copyright © 2004, 2009 IBM Page 115 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 In Figure 108: Pretest Connection Properties, enable “Pretest existing pooled connections”, and give a non-zero value for “Retry interval”. Also enable “Pretest new connections”. Figure 108: Pretest Connection Properties Also, repeat these steps for another data source: icmrm_database. Important: These options have performance impact, and we recommend setting them only when you have to reduce transaction error during fail over. Client Reroute The client reroute feature allows a DB2 client application to recover from a lost database connection in case of a network failure. In the HA configuration discussed in this paper, we use a virtual IP address that allows clients to connect to the active node. Identify the IP address that will be created and failed over as part of the HA failover. In addition, identify the port number for the instance TCP/IP listener by checking the value of the SVCENAME DBM CFG parameter. Issue the following command on the node hosting the instance directory to configure the virtual IP address for client reroute: root@cmaix25 # su - db2inst1 $ db2 update alternate server for database \ icmnlsdb using hostname 9.123.112.251 port 50001 In this example, 9.26.124.22 is the virtual IP address and port 55445 is the value used in the DBM CFG parameter ‘SVCENAME’. Copyright © 2004, 2009 IBM Page 116 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Failover Time Tuning On AIX, DB2 uses the RSCT sub-component for heartbeat detection. You may choose to modify RSCT communication group parameters to improve DB2 failover time. Be aware, however, that shorter DB2 failover times may result in longer application response times. You should modify these settings only after careful testing that balances end-user response requirements with MTTR requirements. List all the Communication Groups in the Cluster: root@cmaix25 # lscomg Name Sensitivity Period Priority Broadcast SourceRouting NIMPathName NIMParameters Grace CG2 4 1 1 Yes Yes -1 (Default) CG1 4 1 1 Yes Yes -1 (Default) Get detailed information about each Communications Group: root@cmaix25 # lscomg -i CG1 Name NodeName IPAddress Subnet SubnetMask en0 cmaix26.cn.ibm.com 9.123.112.200 9.123.112.0 255.255.255.0 en0 cmaix25.cn.ibm.com 9.123.112.199 9.123.112.0 255.255.255.0 Check which nodes belong to which Communication Group: root@cmaix25 # lsrsrc IBM.NetworkInterface Name CommGroup NodeNameList Resource Persistent Attributes for IBM.NetworkInterface resource 1: Name = "en1" CommGroup = "CG2" NodeNameList = {"cmaix25.cn.ibm.com"} resource 2: Name = "en0" CommGroup = "CG1" NodeNameList = {"cmaix25.cn.ibm.com"} NOTES: A NIC/IP can belong to ONLY one Communications Group at a time per single Node - i.e. if there are multiple NICs on a NIC in a Cluster, they will be placed in different Communication Groups. Sensitivity: The number of missed heartbeats that constitute a failure Period: The number of milliseconds between heartbeats Ping Grace Period: The maximum number of milliseconds to wait for a heartbeat response. If no response is received after the specific number of milliseconds, the count of missed heartbeats is increased. Change the Sensitivity of the Communication Group CG1 root@cmaix25 # chrsrc -s "Name like 'CG1'" IBM.CommunicationGroup Sensitivity=4 Change the Heartbeat Period of the Communication Group CG1 root@cmaix25 # chrsrc -s "Name like 'CG1'" IBM.CommunicationGroup PeriodMilliSec=200 Change the Ping Grace period of the Communication Group CG1 root@cmaix25 # chrsrc -s "Name like 'CG1'" IBM.CommunicationGroup PingGracePeriodMilliSec=30000 Copyright © 2004, 2009 IBM Page 117 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Modify the time to aquire quorum. If you are using the IP quorum (the default), you can reduce the amount of time which Tivoli System Automation waits to verify quorum acquisition via (9.123.112.1 is the IP address of the quorum that you want to use. It can be the same as the existing one you already created with db2haicu). root@cmaix25 # mkrsrc IBM.TieBreaker Type="EXEC" Name="mynetworktb1" DeviceInfo="PATHNAME=/usr/sbin/rsct/bin/samtb_net Address=9.123.112.1 Log=1 PostReserveWaitTime=1 HeartbeatPeriod=2 root@cmaix25 # chrsrc -c IBM.PeerNode OpQuorumTieBreaker="mynetworktb1" In using the specific values shown for the parameters discussed, we are adhering to the design choices we made earlier, in Planning an HA environment for Content Manager, where we adopted the philosophy that Content Manager server availability was critical, even over other considerations, since end users cannot access content without access to the Content Manager servers. Similarly, critical to Content Manager server availability is that Content Manager server databases must themselves be available. This was our guiding principle in designing our specific test environment. Your design and planning may dictate other choices for availability parameters such as those above. In our testing, the default settings caused failover times to increase as our overall test workload increased against Content Manager servers. The specific settings for the parameters discussed above allowed us to retain failover times of approximately one minute for the Content Manager server databases. Again, you should understand the benefits versus tradeoffs before making any changes to your RSCT settings. Validation and Test In this chapter, we describe how you can validate that all components in the configuration are working and that load is actually balanced across the components. Further we discuss how the different Content Manager components will react to outages of other components and how to recover from such situations. Validating Active Components In this chapter, we will review how to identify active components in our Content Manager environment. It's a collection of tools, scripts and log files which are frequently used for analysis. Node Application and Database (Library Server, Resource Manager Database) The Node application and database is in our case a combination of: db2inst1 Resource Group (Active on cmaix25) db2isnt2 Resource Group (Active on cmaix26) Note that to display the state of the cluster, you can issue the Tivoli System Automation command ‘lssam’ or the ‘db2pd -ha’ command. Issue the ‘lssam’ command to see the resources created during this process as Figure 109: Tivoli System Automation status for db2inst1 and db2inst2. root@cmaix25 # su – db2inst1 $ lssam Copyright © 2004, 2009 IBM Page 118 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 109: Tivoli System Automation status for db2inst1 and db2inst2 Application Server (Resource Manager) In the described configuration, a clustered IBM WebSphere Application Server configuration is used. It consists of: One deployment manager (only installed on node cmaix26) Two node agents (one on cmaix25 and one on cmaix26), which act as helpers for the deployment managers Two application servers The deployment manager is only used for administrative tasks and provides the Integrated Solutions Console to administrate all components of WebSphere Application Server, including the HTTP server. To start the deployment manager, run following command on node cmaix26: # /usr/IBM/WebSphere/AppServer/bin/startManager.sh ADMU0116I: Tool information is being logged in file / usr /IBM/WebSphere/AppServer/profiles/Dmgr01/logs/dmgr/startServer.log ADMU0128I: Starting tool with the Dmgr01 profile ADMU3100I: Reading configuration for server: dmgr ADMU3200I: Server launched. Waiting for initialization status. ADMU3000I: Server dmgr open for e-business; process id is 40576 The deployment manager is not required at runtime for the Content Manager resource manager component. The same applies for the node agents, which are helper applications for the deployment managers and perform functions such as configuration updates and remote start and stop operations. To start one of the node agents on both nodes: # / usr /IBM/WebSphere/AppServer/profiles/Custom01/bin/startNode.sh ADMU0116I: Tool information is being logged in file / usr /IBM/WebSphere/AppServer/profiles/Custom01/logs/nodeagent/startServer.log ADMU0128I: Starting tool with the Custom01 profile Copyright © 2004, 2009 IBM Page 119 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 ADMU3100I: Reading configuration for server: nodeagent ADMU3200I: Server launched. Waiting for initialization status. ADMU3000I: Server nodeagent open for e-business; process id is 40533 The only mandatory components for Content Manager at runtime are the two application servers. In earlier chapters, these servers have been started from the WebSphere Application Server Integrated Solutions Console. These servers can also be started on a command line, here an example for the server on cmaix25: # / usr /IBM/WebSphere/AppServer/profiles/Custom01/bin/startServer.sh cmaix25 ADMU0116I: Tool information is being logged in file / usr /IBM/WebSphere/AppServer/profiles/Custom01/logs/ cmaix25/startServer.log ADMU0128I: Starting tool with the Custom01 profile ADMU3100I: Reading configuration for server: cmaix25 ADMU3200I: Server launched. Waiting for initialization status. ADMU3000I: Server cmaix25 open for e-business; process id is 41765 To verify the distribution of incoming requests, access logging can be used. To enable the access log, open the WebSphere Application Server Integrated Solutions Console. Navigate to Servers → Application Servers → RMCMAIX25/RMCMAIX26 → Web Container Settings → Web container transport chains → WCInboundDefault → HTTP Inbound Channel (HTTP 2). Enable the checkbox “Enable access and error logging” and click “apply”. Then click on “HTTP error and NCSA access logging” and select the check box “Enable service at server startup”. Click “apply” and save your changes. Apply the same changes for the other application server, and then restart the cluster. Every incoming request is now added to the $SERVER_LOG_ROOT/http_access.log. HTTP Server (Resource Manager) An IBM HTTP Server is used to route and load-balance incoming client requests via HTTP to the WebSphere Application Server. These two components are connected with the IBM HTTP Server PlugIn. Normally, the load between the two application servers is load-balanced in a round-robin fashion. If one of the application servers becomes unavailable, all load is redirected to the available one. To start the IBM HTTP Server, run the following command on both nodes: # /usr/IBM/HTTPServer/bin/apachectl start The HTTP Server logs all requests into an activity log file, by default into /usr/IBM/HTTPServer/logs/access_log. This will provide an overview about all incoming requests. The details which are captured can be configured in the httpd.conf file. Note: The configuration discussed in this white paper installs and configures two active HTTP Servers. However, only one is actually used in our test environment. To eliminate this single point of failure, we recommend the configuration of an IP sprayer in front of both HTTP servers. Content Manager library server configuration data for the Content Manager resource manager can be set to reflect the IP sprayer address instead of individual WebSphere Application Server servers. Depending on the IP sprayer chosen, requests to the RM application may be load balanced across the HTTP servers and automatically redirected in the event an HTTP server fails. Copyright © 2004, 2009 IBM Page 120 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Content Manager Component Failures This chapter summarizes what happens if a certain component of the Content Manager stack fails. Most of the time, the recovery will be automatic and frequently unnoticeable for the end user. However, operational procedures should be in place, making sure that corrective actions are taken. Database (Library Server, Resource Manager Database) The database is probably the most important components of Content Manager, as all other parts do have a dependency on it. The database is configured in a DB2 with Tivoli System Automation environment, using one active instance and one standby instance on two servers. If one of the servers becomes unavailable, Tivoli System Automation will detect this error and start the DB2 instance on another node. Also, client requests will be moved to the active DB2 instance. Complete failover generally takes from seconds to minutes but depends on several factors including overall workload against the Content Manager servers, db2haicu settings, and RSCT settings. In case a connection is already established to the failing DB2 instance, this connection will be disconnected. If a transaction is in progress on the failing instance, this transaction will be rolled back. Other transactions will continue to report an error, or hang until another node starts-up. The following is a list of components that access the database. The list also contains a description of the behavior when the component loses an active connection to a failing DB2 instance. Common across all components is an error condition (rollback) when a transaction is in progress on the failing instance. Application using Content Manager Java API (eClient, Content Manager System Administration Client, WEBi, custom applications): The connection to the server is lost. An exception will be thrown. Reconnect can be implemented silently in the application but it will fail until another node startup. If no automatic reconnect is implemented, the user must log-in again. Application using Content Manager C++ API (custom applications): The connection to the server is lost. An exception will be thrown. Reconnect can be implemented silently in the application but it will fail until another node startup. If no automatic reconnect is implemented, the user must log-in again. IBM® DB2® Content Manager Client for Windows: The Content Manager Client for Windows automatically drops a connection after 30 seconds of inactivity. A new connection will be seamlessly established after this period. If the user initiates a request before the 30 seconds timeout, an error message will be displayed. Content Manager resource manager: An error message will be captured in the log of the resource manager application server. Also, the first request to the resource manager may fail will a database connection error and the transaction will be rolled back. Subsequent requests will work normally. This behavior applies to the resource manager servlet as well as to the services such as migrator, replicator etc. Application Server (Resource Manager) Request to the two application servers in the WebSphere Application Server Network Deployment cluster are balanced by the IBM HTTP Server plug-in. If one application server becomes unavailable, subsequent requests will be routed to the remaining active servers. Copyright © 2004, 2009 IBM Page 121 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 HTTP Server (Resource Manager) The HTTP Server, if configured like described in this document, is a single point of failure. If the HTTP Server becomes unavailable, access to the resource manager for client requests become impossible. All transactions, which include the resource manager, will fail. To mitigate this risk, we recommend configuring an IP sprayer in front of the HTTP Server. This document already included all steps to setup a second HTTP Server, only the routing of the requests to the second HTTP Server is missing. Copyright © 2004, 2009 IBM Page 122 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Appendix List of Tables Table 1: Logical Drive definitions on DS4200 ................................................................................... 21 Table 2: IP Configuration ................................................................................................................... 24 Table 3: Operating system groups and users....................................................................................... 30 List of Figures Figure 1: Overall Scenario (2 Node Configuration) ........................................................................... 19 Figure 2: DS4200 Configuration......................................................................................................... 22 Figure 3: IBM N3700 Volume Definition .......................................................................................... 23 Figure 4: DB2 setup wizard ................................................................................................................ 32 Figure 5: Install New DB2 product..................................................................................................... 33 Figure 6: Welcome window................................................................................................................ 33 Figure 7: Accept License .................................................................................................................... 34 Figure 8: Installation Type.................................................................................................................. 34 Figure 9: Response file ....................................................................................................................... 35 Figure 10: Select the features to Install............................................................................................... 35 Figure 11: Select the langurages to Install .......................................................................................... 36 Figure 12: Specify the location of DB2 Information Center............................................................... 36 Figure 13: Install the IBM SA MP Component .................................................................................. 37 Figure 14: Set user information for the DB2 Administration Server.................................................. 38 Figure 15: Set up a DB2 instance........................................................................................................ 38 Figure 16: Set up notification.............................................................................................................. 39 Figure 17: Summary ........................................................................................................................... 39 Figure 18: DB2 instance Setup wizard ............................................................................................... 41 Figure 19: Set up a DB2 instance........................................................................................................ 41 Figure 20: Set up partitioning options for the DB2 instance .............................................................. 42 Figure 21: Set user information for the DB2 instance owner ............................................................. 42 Figure 22: Set user Information for the fenced user ........................................................................... 43 Figure 23: Configure instance communication and startup ................................................................ 43 Figure 24: Prepare the DB2 tools catalog ........................................................................................... 44 Figure 25: Specify a contact for health monitor notification .............................................................. 44 Figure 26: Configure the DB2 Text Search server.............................................................................. 45 Figure 27: Summary ........................................................................................................................... 45 Figure 28: Set user information for the DB2 instance owner ............................................................. 46 Figure 29: Set user information for the fenced user............................................................................ 47 Figure 30: Configure instance communication and startup ................................................................ 47 Figure 31: Summary ........................................................................................................................... 48 Figure 32: db2inst1 Tivoli System Automation status........................................................................ 55 Figure 33: WebSphere Application Server Network Deployment Prerequisite Check ...................... 65 Figure 34: WebSphere Application Server Network Deployment Environments .............................. 66 Figure 35: WebSphere Application Server Profile Warning .............................................................. 66 Figure 36: IBM HTTP Server System Prerequisites Check................................................................ 67 Figure 37: IBM HTTP Server Port Values Assignment ..................................................................... 68 Figure 38: IBM HTTP Server Setup HTTP Administration Server.................................................... 68 Copyright © 2004, 2009 IBM Page 123 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 39: IBM HTTP Server Plug-In for WebSphere Application Server........................................ 69 Figure 40: IBM HTTP Server Installation Summary.......................................................................... 69 Figure 41: Update Installer Installation Result ................................................................................... 70 Figure 42: Update Available Maintenance Packages.......................................................................... 71 Figure 43: Profile Management Tool - Environments ........................................................................ 72 Figure 44: Profile Management Tool - Administrative Security ........................................................ 73 Figure 45: Profile Management Tool - Deployment Manager Summary ........................................... 73 Figure 46: Profile Management Tool - Federation ............................................................................. 74 Figure 47: Profile Management Tool - Custom Profile Summary...................................................... 75 Figure 48: WebSphere Application Server Create a new cluster (1) .................................................. 76 Figure 49: WebSphere Application Server Create a new cluster (2) .................................................. 77 Figure 50: WebSphere Application Server - Create a new cluster (3)................................................ 78 Figure 51: WebSphere Application Server - Create a new cluster (Summary) .................................. 79 Figure 52: WebSphere Application Server - Save Changes ............................................................... 80 Figure 53: Secure administration, application, and infrastructure ...................................................... 81 Figure 54: New J2C user..................................................................................................................... 81 Figure 55: Create new JDBC provider................................................................................................ 82 Figure 56: Database class path information ........................................................................................ 83 Figure 57: Create basic data source information................................................................................. 84 Figure 58: Select JDBC provider ........................................................................................................ 84 Figure 59: Create a data source........................................................................................................... 85 Figure 60: WebSphere Variables ........................................................................................................ 86 Figure 61: Restart Node Agents.......................................................................................................... 86 Figure 62: Nodes Synchronize............................................................................................................ 87 Figure 63: Test connection for SIBDB datasource ............................................................................. 87 Figure 64: New Web Server Definition (1) ........................................................................................ 88 Figure 65: New Web Server Definition (2) ........................................................................................ 88 Figure 66: New Web Server Definition (3) ........................................................................................ 89 Figure 67: New Web Server Definition (4) ........................................................................................ 89 Figure 68: New Web Server Definition – Save Changes.................................................................... 90 Figure 69: Web Server - Start ............................................................................................................. 90 Figure 70: Update Resource Manager- Resource Manager Properties ............................................... 91 Figure 71: Content Manager Installation Summary............................................................................ 94 Figure 72: Configuration Options ....................................................................................................... 95 Figure 73: Component Selection......................................................................................................... 95 Figure 74: Database Type ................................................................................................................... 96 Figure 75: Administrative User........................................................................................................... 97 Figure 76: System Information ........................................................................................................... 97 Figure 77: DB2 Database Product Directory ...................................................................................... 98 Figure 78: Library Server Database .................................................................................................... 98 Figure 79: Library Server Database Information ................................................................................ 99 Figure 80: DB2 Database instance...................................................................................................... 99 Figure 81: Library Server Database .................................................................................................. 100 Figure 82: Library Server Database Settings .................................................................................... 100 Figure 83: Library Server Connection .............................................................................................. 101 Figure 84: Library Server Connection (cont.)................................................................................... 102 Figure 85: System Administration Client ......................................................................................... 102 Figure 86: System Administration Client Connection ...................................................................... 103 Figure 87: Start Configuration .......................................................................................................... 103 Figure 88: Content Manager Configuration Complete ..................................................................... 104 Figure 89: Configure Content Manager ............................................................................................ 105 Copyright © 2004, 2009 IBM Page 124 of 125 IBM Content Manager v8.4.2 using DB2 with Tivoli System Automation on AIX 2009.12.28 Figure 90: Product components ........................................................................................................ 106 Figure 91: Server Database Type...................................................................................................... 106 Figure 92: Administrative user ......................................................................................................... 107 Figure 93: System Information ......................................................................................................... 107 Figure 94: DB2 Database Product Directory .................................................................................... 108 Figure 95: Resource Manager Database ........................................................................................... 108 Figure 96: Resource Manager Database Information ....................................................................... 109 Figure 97: DB2 Database instance.................................................................................................... 109 Figure 98: Resource Manager Application ....................................................................................... 110 Figure 99: Resource Manager Application (cont)............................................................................. 111 Figure 100: WebSphere Variables .................................................................................................... 111 Figure 101: Resource Manager Application Deployment Target ..................................................... 112 Figure 102: Cluster Information for the Resource Manager Application ......................................... 113 Figure 103: HTTP Connection Information for the Resource Manager Application ....................... 113 Figure 104: Resource Manager Application Connections ................................................................ 114 Figure 105: Resource Manager Application Connection (cont) ....................................................... 114 Figure 106: Content Manager Configuration Complete ................................................................... 115 Figure 107: Data source properties ................................................................................................... 115 Figure 108: Pretest Connection Properties........................................................................................ 116 Figure 109: Tivoli System Automation status for db2inst1 and db2inst2......................................... 119 Copyright © 2004, 2009 IBM Page 125 of 125