Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
HADR Users Guide Adaptive Server® Enterprise 15.7 SP100 DOCUMENT ID: DC01979-01-1570100-01 LAST REVISED: July 2013 Copyright © 2013 by Sybase, Inc. All rights reserved. This publication pertains to Sybase software and to any subsequent release until otherwise indicated in new editions or technical notes. Information in this document is subject to change without notice. The software described herein is furnished under a license agreement, and it may be used or copied only in accordance with the terms of that agreement. Upgrades are provided only at regularly scheduled software release dates. No part of this publication may be reproduced, transmitted, or translated in any form or by any means, electronic, mechanical, manual, optical, or otherwise, without the prior written permission of Sybase, Inc. Sybase trademarks can be viewed at the Sybase trademarks page at http://www.sybase.com/detail?id=1011207. Sybase and the marks listed are trademarks of Sybase, Inc. ® indicates registration in the United States of America. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. Java and all Java-based marks are trademarks or registered trademarks of Oracle and/or its affiliates in the U.S. and other countries. Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. IBM and Tivoli are registered trademarks of International Business Machines Corporation in the United States, other countries, or both. All other company and product names mentioned may be trademarks of the respective companies with which they are associated. Use, duplication, or disclosure by the government is subject to the restrictions set forth in subparagraph (c)(1)(ii) of DFARS 52.227-7013 for the DOD and as set forth in FAR 52.227-19(a)-(d) for civilian agencies. Sybase, Inc., One Sybase Drive, Dublin, CA 94568. Contents CHAPTER 1: Overview of the HADR System ................1 Members in an HADR System ..............................................3 Determining the Member Address List ...........................3 The Split-Brain Check: Preventing Multiple Primary Servers .......................................................................4 The monHADRMembers Table .......................................4 Member Modes and States ...................................................5 Determining the Member Mode and State ......................7 Changing a Server Mode ................................................8 HADR Privileges and Roles ................................................10 Privileges Required for the Maintenance User .............11 CHAPTER 2: Safely Terminating User Transactions on the Primary Server ...............................................13 CHAPTER 3: Installing, Configuring, and Administering the HADR System Using isql ...........15 Before You Begin .................................................................15 Standby System Administration ....................................15 Technical Prerequisites and Limitations ........................16 Configuring Adaptive Server for Remote Connections . . .16 Validating the Adaptive Server Configuration ..................17 Adding Adaptive Server Users ...........................................18 Adding the HADR User Login .......................................19 Initializing the HADR Nodes ...............................................19 Configuring a Node as an HADR Member ....................20 Installing Replication Server ..............................................21 HADR Users Guide iii Contents Materializing the Databases and Starting Replication ..................................................................................21 Adding and Dropping Servers from the Member List ......22 CHAPTER 4: Configuring and Administering HADR with the DR Agent ......................................................23 Setting the HADR Mode and State .....................................23 Viewing the HADR sysserver Entries ................................24 Initializing the Setup Between Two Servers ......................26 Performing Materialization ..................................................26 Performing Failover .............................................................27 Disabling or Enabling Replication .....................................28 Performing Teardown ..........................................................29 Troubleshooting with the DR Agent ...................................29 CHAPTER 5: Failover ....................................................33 Draining the Transaction Log .............................................33 Performing a Planned Failover ...........................................34 Performing an Unplanned Failover ....................................35 CHAPTER 6: Upgrading Adaptive Server in an HADR System ........................................................................39 Upgrading the Standby Adaptive Server ...........................40 CHAPTER 7: SDK and Open Server Support for HADR ..........................................................................43 SDK Support for HADR .......................................................43 Open Client Services ....................................................43 DR Map Callback .................................................43 CS_CTLINFO_T Parameters ...............................44 DR Map Callback Example ..................................45 Open Client Capabilities ......................................46 iv Adaptive Server Enterprise Contents CS_PROP_READONLY Connection Property ....46 Connection Properties for HADR in jConnect ...............46 Retrieve HADR State Change Messages from jConnect ..........................................................47 Retrieve DR_LIST_MAP Connection Property from jConnect ..................................................48 HADR Reconnection in jConnect .........................50 Connection Properties for HADR in Adaptive Server ODBC Driver ............................................................ 51 HADR Information Messages from Adaptive Server ODBC Driver ........................................52 HADRList Connection Property from Adaptive Server ODBC Driver ........................................53 Messages About HADR State Change in jConnect and Adaptive Server ODBC Driver ...........................55 Open Server Support for HADR .........................................58 CHAPTER 8: sp_hadr_admin Syntax ..........................61 Index ........................................................................................... 69 HADR Users Guide v Contents vi Adaptive Server Enterprise CHAPTER 1 Overview of the HADR System A typical high availability disaster recovery (HADR) configuration consists of two or more servers: one designated as the primary, on which all transaction processing takes place; the others as standby, which act as warm standbys for the primary server and contain copies of designated databases from the primary server. Note: This release supports only a single standby server. Some high availability solutions (for example, the Adaptive Server® Cluster Edition) share or use common resources between nodes. However, HADR is a "shared nothing" configuration: each node has separate resources, including disks. In an HADR system, servers are separate entities, and data is replicated from the primary server to the standby server. If the primary server fails, a standby server is promoted to the role of primary server either manually or automatically. Once the promotion is complete, clients can reconnect to the new primary server, and see data as on the previous primary server. Note: This release supports only a manual promotion. Servers can be separated geographically, which makes an HADR system capable of withstanding the loss of an entire computing facility. For example, your system may include a primary server that is located in San Francisco and a standby server in North Dakota, ensuring that if the primary server is destroyed, the standby server is safe and ready to assume control. Additionally, an HADR system includes Replication Server, which synchronizes the databases between the primary and standby servers. Adaptive Server uses the Replication Agent to communicate with Replication Server, and Replication Server uses Open Client connectivity to communicate with the standby Adaptive Server. The Replication Agent detects any data changes that are made on the primary server, and sends them to the primary Replication Server. In the figure below, the unidirectional arrows indicate that, although both Replication Servers are configured, only one direction is enabled at a time: HADR Users Guide 1 CHAPTER 1: Overview of the HADR System The most fundamental service offered by Adaptive Server HADR is the planned failover from the primary to the standby server, which allows maintenance activity to occur on the old primary server, while applications continue on the new primary. HADR provides protection in the event of a disaster: if the primary server is lost, the standby server can be used as a replacement. Client applications can be switched to the standby server, and the standby server is quickly available for users, although there may be a data loss. You can configure applications running in an HADR system to retrieve and maintain a list of addresses of the HADR members, and to receive asynchronous updates concerning any changes to the configuration. Applications distinguish between the primary and the standby server, and refrain from connecting to the latter unless the primary is unreachable. Connection attempts to the standby server without the necessary privileges are silently redirected to the primary via the login redirection mechanism, which is supported by connectivity libraries SAP® ERP runs on Adaptive Server configured for HADR. Configure and administer HADR using either isql to administer HADR outside the Business Suite, or the SAP Disaster Recovery Agent (DR Agent) to configure HADR inside the Business Suite: • • in isql – to administer HADR outside the Business Suite. Use the sp_hadr_admin system procedure. sp_hadr_admin is created when you run installmaster, but you must also enable HADR before this system procedure is available for use. DR Agent – to administer HADR inside the Business Suite. Client applications connect to the DR Agent through its TDS interface (ODBC, JDBC, or Open Client). The DR Agent uses JDBC to communicate with the servers on either host, and to other DR Agents in the HADR system. For complete information , see the DR Agent Users Guide. Note: The commands described in this manual are for reference. SAP recommends that you use the SAP tools for administering and configuring the HADR sytem. SAP tools are wrappers around DR Admin tools, which, in turn, are wrappers for the sp_hadr_admin system procedure 2 Adaptive Server Enterprise CHAPTER 1: Overview of the HADR System Members in an HADR System All servers that participate in an HADR system must belong to an HADR group. Use the sp_hadr_admin init parameter to define: • • The HADR group to which a member belongs A member's unique HADR server name, if it is different from its Adaptive Server name Recommendations for Naming Servers Servers in an HADR system are known to other members by their HADR server name, which must be unique within the HADR group. The HADR server name also allows Adaptive Servers with the same name to be uniquely identified within the HADR system. The HADR server name can be the same as the Adaptive Server name (as reported by select @@servername), as long as each server in the group has a unique server name. If an HADR group includes servers that have duplicate names, you must supply an alternative name to use as the HADR server name. Sybase® recommends that you: • • Use a naming scheme, such as including a location as part of the server name, that makes it easy to identify each server (for example, paris_server_01). Do not use "primary" or "standby" in server names, as these roles are reversed during failover. The HADR system automatically appends the letters "DR" to server names to identify them as HADR participants. The maximum length for a HADR server name is 28 characters (not counting the "DR"). The HADR system uses the server net name to contact another HADR node to exchange status information. The server net name can be: • • A simple network name that exists in the local interfaces file or directory services. A host name and port number (for example, paris_machine:8455). In this case, the specified IP address or host name and port number are used directly for connection without requiring directory lookup. If you do not supply a server net name, the HADR system uses the HADR_server_name (without the "DR" suffix) to contact other HADR nodes. However, in this case, the interfaces file must include an entry that directs the HADR server name to the correct Adaptive Server. You must manually add this entry. Determining the Member Address List The member list, which contains an entry for each member of an HADR configuration, determines the HADR system configuration. It includes: HADR Users Guide 3 CHAPTER 1: Overview of the HADR System • • • • • Member names (the HADR_server_name) A network address for each member Each member's failover standby server targets (if any) The HADR group name A generation number, which identifies the most recent version of the member list Note: If Adaptive Server is configured for password encryption (that is, net password encryption reqd is set to a value other than "DEFAULT"), you must also set encryption as a server option on any remote HADR servers: sp_serveroption server_HADR_nameDR, 'net password encryption', true To view the current local member list from the primary or standby server, issue: sp_hadr_admin listserver This example output shows the current member list for the Sybase_HADR system: name -------------server1 server2 network_name ---------------------------server1 server2 The Split-Brain Check: Preventing Multiple Primary Servers Only one server in the HADR group should perform client transactions at any one time. If more than one server assumes the role of the primary server, the databases on the HADR servers can no longer be synchronized, and the system enters a "split-brain" situation. The HADR system provides a check against this, which is performed either at start-up if the Adaptive Server configuration file instructs the server to start as a primary, or when you use sp_hadr_admin primary to manually promote a standby server to the primary server. The check connects to and queries each configured HADR member. If a remote HADR member in the group is identified as an existing primary server, the check does not allow the local server to be promoted to the primary server. Generally, you cannot override this check. If the check fails to connect to one or more remote HADR member, it assumes that the unreachable member may be a primary server, and refuses to promote the local server to primary. In this situation, you can use the force parameter to override the split-brain check: sp_hadr_admin primary, force Before using the force parameter, verify that there is no other primary server present in the group. The monHADRMembers Table The monHADRMembers table stores monitoring information about the members of the HADR system. The columns of the monHADRMembers table are: 4 Adaptive Server Enterprise CHAPTER 1: Overview of the HADR System Name Datatype Description GroupName varchar Name of the group ServerName varchar Name of the HADR member Mode varchar Current mode of the HADR member State varchar Current state of the HADR member ServerMap varchar Member connetion information, including: • • • Type of connection Host name port number This example shows output from monHADRMembers for HADR group MYGRP: select * from monHADRMembers GroupName ServerName Mode State ServerMap ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------MYGRP LONDONDR Standby Inactive tcp asekernel1 17682 MYGRP Primary tcp asekernel1 17672 PARISDR Active Use admin sqm_process_time to monitor Replication Server. See the Replication Server Reference Manual. See the Performance and Tuning Series: Monitoring Tables for information about configuring and querying the Adaptive Server monitoring tables. Member Modes and States Each server in an HADR configuration has an external mode that is visible to and known by other HADR members, as well as an internal state that is known only by the member. External modes are: HADR Users Guide 5 CHAPTER 1: Overview of the HADR System • • • • • Primary – the member of an HADR configuration on which active transaction processing by user applications is allowed to take place. Standby – member of an HADR configuration which contains copies of specific databases that originate on the primary member, and is available to take over transaction processing in the event that the primary member fails. Replication Server replicates database changes on the primary that are marked for replication to standby members. Disabled – HADR is disabled on this member. Unreachable – the local member (the server from which you enter commands) cannot reach this remote HADR member. Starting – HADR member is starting. Internal states are: • • • Active – normal state of the primary server. Transaction activity by user applications occurs in this state without any of the restrictions imposed by other states. The primary server is always in the active state after a restart, provided the split-brain check is successful. Inactive – restricts all activity by unprivileged connections, but imposes no restrictions on activity by privileged connections (including starting new transactions). Deactivating a primary server places it into the inactive state. Standby members are always in the inactive state. Deactivating – intermediate state between active and inactive states. The HADR system gracefully terminates transactions by unprivileged connections during the deactivating state. Use the sp_hadr_admin timeout parameter to specify a time limit for this state. The HADR system waits for the time specified by the timeout parameter for the transactions to complete. When the timeout period ends, the internal state is either rolled back to active or is advanced to inactive, depending on whether you included the force parameter with the sp_hadr_admin command. Note: When you include the force parameter, the HADR system forcibly terminates all the transactions started by privileged and unprivileged connections. Unprivilged connections cannot start new transactions when the server is in the deactivating state. The internal state of a primary server is not preserved across restarts. 6 Adaptive Server Enterprise CHAPTER 1: Overview of the HADR System Determining the Member Mode and State Use the sp_configure HADR mode parameter and the @@hadr_mode global variable to determine the external mode of an HADR member. Use the @@hadr_state global variable to determine the internal state of an HADR member. HADR mode also enables and disables HADR on Adaptive Server. To determine the external mode of the server, view the @@hadr_mode global variable, which returns: HADR Mode Value HADR Mode Name Description -1 Disabled HADR is disabled. HADR Users Guide 7 CHAPTER 1: Overview of the HADR System HADR Mode Value HADR Mode Name Description 0 Standby HADR is enabled. This is a standby server. 1 Primary HADR is enabled. This is a primary server. 2 Unreachable HADR is enabled, but the server is unreachable. This value is not seen by the local server. 3 Starting HADR is enabled, and the server is ready for initialization. To determine the internal state of the server, view the @@hadr_state global variable, which returns: HADR State Value HADR State Name Description 0 Unknown HADR is in an unknown state. This value is typically returned when HADR is in disabled mode. 1 Active Primary server allows transaction processing from user applications, but may not be actively replicating data. 2 Inactive Server is inactive, and does not allow transaction processing from user applications. 3 Deactivating The server is changing from the active to the inactive state, and the log is being drained. Eventually, the mode should transition to the inactive state. If deactivation times out, the mode may switch back to the active state. Changing a Server Mode Users who have allow hadr login and manage hadr privileges can issue sp_hadr_admin to change the mode of a server. The syntax is: sp_hadr_admin primary [, “force”] sp_hadr_admin standby 8 Adaptive Server Enterprise CHAPTER 1: Overview of the HADR System Use the primary parameter to move a server from standby to primary mode. The primary parameter verifies that the system does not already contain an existing primary server by connecting to each member in the group and querying for its mode. If any server identifies itself as the primary server, or if the sp_hadr_admin command cannot connect to any member in the system, the command fails and leaves the mode of the standby server unchanged. Use the force parameter to promote a server to primary mode if some servers are unreachable. The new primary server remains in the inactive state until you explicitly activate it to allow user transaction processing. However, if a server is reachable and identified as the primary server, promoting the local server to a primary role fails, even if you include the force parameter. Use the standby parameter to change a server's mode from primary to standby. The server must already be in the inactive state. You can also set the mode of the servers (from primary to standby or vice versa) using the sp_configure HADR mode configuration parameter. Set HADR mode to 1 on the primary and 0 on a standby server (the value for HADR mode on a non-HADR server is -1). You must restart the server for the change to HADR mode to take effect. Note: You cannot use the HADR mode configuration parameter to start a server as the primary server if any standby servers are unreachable. You can designate a server as a primary either at start-up, or manually once the server is running: • • At start-up – if HADR mode is set to 1 (primary), the server checks the member list in sysservers and the value for @@hadr_mode for each server in the list. If the variable is set to 1 for any of the servers other than the one you are starting, or if any of those servers are unreachable, the server that is starting resets HADR mode and @@hadr_state to 0, writes a message to the error log, and completes the start-up process in standby mode. You can then manually promote the server you are starting to primary mode. However, if all the remaining members are running and in standby mode, the server that is starting promotes itself to primary mode without manual intervention. Manually – privileged users can issue sp_hadr_admin primary to promote a standby server to primary mode. sp_hadr_admin primary verifies the member list in sysservers and the value for @@hadr_mode for each member in the list. However, sp_hadr_admin primary does not promote the current server to primary mode if any of the remaining members are unreachable or already designated as the primary. If this occurs, use the force parameter to promote a standby server to primary mode. HADR Users Guide 9 CHAPTER 1: Overview of the HADR System HADR Privileges and Roles The allow hadr login and manage hadr privileges determine a user's permissions in the HADR system. A user who is granted the allow hadr login privilege can log in to and perform transaction processing on both primary and standby servers, regardless of the mode. A user with this permission is called a privileged user. A user without the allow hadr login privilege cannot log in to the standby server. Granting allow hadr login does not control user access to objects in a database— you must separately grant the appropriate permission for specific objects. You need not enable granular permissions to grant allow hadr login. The maintenance user is automatically granted this permission so that replicated activity can be applied to standby members. To grant the allow hadr login privilege to users, use the grant command. This example grants allow hadr login privilege to user joe: grant allow hadr login to joe The manage hadr privilege allows users to manage the HADR configuration using sp_hadr_admin: • • 10 If you have not enabled granular permissions, users who have the sa_role also automatically have manage hadr permissions. If you have enabled granular permissions, you must specifically grant users manage hadr permissions. Adaptive Server Enterprise CHAPTER 1: Overview of the HADR System Privileges Required for the Maintenance User Replication Server uses the maintenance user to replicate DML and DDL commands. Grant maintenance users set session authorization permission from Adaptive Server to enable them to impersonate the original user who executed the DDL commands. To ensure maintenance user security, enable the hide_maintuser_pwd Replication Server configuration parameter. See Granular Permissions in the Replication Server Administration Guide Volume 1. To execute DDL and DML commands, the maintenance user must have these permissions: Operation Required Permissions or Roles (Granular Permissions On) Required Permissions or Roles (Granular Permissions Off) Insert any table Insert any table sa_role or alias to dbo Update any table Update any table sa_role or alias to dbo Delete any table Delete any table sa_role or alias to dbo HADR Users Guide 11 CHAPTER 1: Overview of the HADR System Operation Required Permissions or Roles (Granular Permissions On) Required Permissions or Roles (Granular Permissions Off) Execute any procedure Execute any procedure sa_role or alias to dbo Execute any function Execute any function sa_role or alias to dbo Set identity_insert on or off Identity_insert any sa_role, alias to dbo, or replication_role table Set identity_update on or off Identity_update any sa_role, alias to dbo, or replication role table Truncate any table Truncate any table Impersonate original user to ex- set session authorization ecute DDL or system procedure Write to an inactive server that is in soft quiesce mode 12 sa_role, alias to dbo, or replication_role set session authorization allow HADR login priv- sa_role or allow HADR ilege login privilege Adaptive Server Enterprise CHAPTER 2 Safely Terminating User Transactions on the Primary Server The HADR system must halt database processing activity (called a soft quiesce) on the primary server before it can perform a seamless planned application failover to the standby server. During this process, the primary server is in the deactivating state, and Adaptive Server prevents most database transactions from starting. Only privileged users can perform transactions on servers that are in the inactive state. Internally, some transactions (for example, housekeeper) still may occur. During a soft quiesce, the HADR system halts transaction processing in an orderly manner; allowing ongoing transactions to proceed until the specified timeout, but disallowing new transactions by unprivileged connections. After the primary server reaches the inactive state, threads from the primary server's Replication Agent drain the HADR database transaction logs (unless you issued sp_hadr_admin deactivate with the nodrain parameter). Once the log is completely drained, the inactive primary is ready for failover. Before you can promote a standby server to a primary server, you must first demote the original primary server to standby. To move a server to the inactive state, issue sp_hadr_admin deactivate. For example: sp_hadr_admin deactivate, '10', 'label' (return status = 0) User connections statistics:: 0 in xact, 0 in chained mode, 1 in unchained mode, 0 holding server si de cursors. Server reached INACTIVE state. Initiating log drain mechanism. Command 'deactivate' successful. (1 row affected) label should be a meaningful string that is associated with the current deactivation operation, and should be distinct from previous labels you used (for example, "scheduled_offline_07092013"). The Replication Server subsystem uses the label when it replicates data from the primary server to the standby server. HADR Users Guide 13 CHAPTER 2: Safely Terminating User Transactions on the Primary Server 14 Adaptive Server Enterprise CHAPTER 3 Installing, Configuring, and Administering the HADR System Using isql Use the sp_hadr_admin system procedure from the isql command line to manually install, configure, and administer the HADR system. Before You Begin HADR does not explicitly identify which databases participate in the HADR system. Instead, it identifies the databases in the primary and standby servers that are participating in the system based on the Replication Agent mode. If a Replication Agent is enabled for a specific database, the HADR system assumes this database is participating. If Replication Agent is not enabled for a specific database, it is considered to be not participating. Note: Do not enable or disable a Replication Agent on an HADR server unless you are sure the database is, or is not, particpating in the HADR system. The same set of Replication Agents must be enabled or disabled on all servers in an HADR group. The steps provided by this document assume that granular permissions are not enabled. See the Adaptive Server Security Administration Guide. Standby System Administration The machine that is hosting the standby server must contain an instance of Adaptive Server and its related databases. This standby server requires the same level of protection and scheduled maintenance as the primary server, including: • • Periodically scheduled housekeeper tasks, such as generating fresh statistics and running the reorg commands for data-only-locked (DOL) table and index maintenance. For example, issuing reorg rebuild index to rebuild indexes, reorg forwarded_rows to combine row data, or reorg rebuild table to rebuild tables. Backup and recovery processes, including disk-based file backup and recovery, and creating and archiving database-level dumps. You cannot use dump files from the primary server to recover the standby server: once replication begins, the physical attributes of the primary and standby databases are no longer equivalent, and they cannot share the same dump and load files for recovery. HADR Users Guide 15 CHAPTER 3: Installing, Configuring, and Administering the HADR System Using isql Technical Prerequisites and Limitations Depending on the audience and type of environment (SAP or non-SAP), the HADR system has some technical limitations and prerequisites. • • • HADR must replicate the master database. The sa and sso logins and passwords must be identical on both primary and standby servers. Logins, users, and roles on the standby server must be a subset of the logins, users, and roles on the primary server. Any unique logins, users, or roles that are defined on the standby server are lost when you materialize login data from the primary server's master database to the standby server. Database names must be the same on the primary and standby servers. You cannot run the HADR system on a single machine. You must use the same hardware platform and operating system on both machines. However, the operating system versions can differ, as long as both versions support the Adaptive Server application. HADR replicates master database passwords using ciphertext values from syslogins, so the standby server must be aware of cipher text values. See the Encrypted Columns Users Guide. Only the maintenance user can perform replication; therefore, replication must be defined for all databases (for example, master, and the application database), and must be able to access all databases on both sites. Transaction-log volume increases: replication writes additional information to the Adaptive Server transaction log for each database, which may increase log volume by 40 to 50%. Consequently, you must allocate more transaction log space or schedule more frequent transaction log dumps to ensure sufficient log space is available for your applications. • • • • • • HADR supports: Software Version Adaptive Server 15.7 SP100 Replication Server 15.7.1 SP100 and later Configuring Adaptive Server for Remote Connections Servers in an HADR system communicate using RPCs. Enable communication between servers by: • Setting these configuration parameters: • enable cis to 1 (the default) 16 Adaptive Server Enterprise CHAPTER 3: Installing, Configuring, and Administering the HADR System Using isql • • cis rpc handling to 1 Creating a login account specifically for HADR use. You can assign any name to the account, but it must be granted the sa_role, and use the master database as its default database. Sybase recommends that you use this login when you configure the HADR system. However, if you prefer to use a different login, you must use sp_addexternlogin to configure it as an external login that is the designated HADR user on remote nodes. See the Adaptive Server Reference Manual: Procedures. Validating the Adaptive Server Configuration Before installing Replication Server, verify that Adaptive Server is properly configured. 1. Disable trunc log on chkpt on the master and all other databases participating in the HADR system. For example, for the D01 database, enter: sp_dboption D01, 'trunc log on chkpt', false 2. Enable FIPS password storage on the primary and standby servers: sp_configure 'FIPS login password encryption', 3 3. To require all connections to send encrypted passwords, issue the following command on both the primary and standby servers: sp_configure 'net password encryption reqd', 3 A value of 3 for net password encyption reqd configures Adaptive Server for strong encryption. See net password encyption reqd in Setting Configuration Parameters in the System Administration Guide: Volume 1. All subsequent connections to the primary and standby server must include the isql -X parameter, which initiates the login connection to the server with client-side password encryption. -X enables extended password-encrypted connections and passwordencrypted connections without plain text password reconnection. isql -X specifies to the server that you require password encryption. The server returns an encryption key, which isql uses to encrypt your password. The server uses the same key to authenticate your password when it arrives. If isql fails, the system creates a core file that contains your password. If you did not include the encryption option, the password appears as plain text in the file. If you used the encryption option, your password is not readable. 4. To allow Adaptive Server to mark large databases for replication, add the -T9149 trace flag to the primary and standby runserver files (See the Adaptive Server Configuration Guide for your platform). 5. Restart the primary and standby servers. HADR Users Guide 17 CHAPTER 3: Installing, Configuring, and Administering the HADR System Using isql Adding Adaptive Server Users To apply activity to the target system, you must create the Adaptive Server maintenance user. The maintenance user must have a unique Adaptive Server login. You cannot use an existing Adaptive Server user as the maintenance user. Note: Protect the Replication Server maintenance user. See the Replication Server Administration Guide. Replication Server applies changes to the standby database using the unique maintenance user login. Replication Server ignores any activity other than replication performed by the maintenance user to prevent data loss or inconsistency between the primary and standby databases. To add the maintenance user, perform these steps on both the primary and standby server: 1. Create the maintenance user. This example creates a maintenance user named D01_maint: use master create login D01_maint with password Sybase123 sp_role "grant", replication_role, D01_maint 2. Grant the replication_role to the maintenance user, enabling it to replicate the truncate table command. 3. Use sp_addalias to alias the maintenance user to the database owner on the master and user databases, allowing this user to update tables that use IDENTITY columns. 4. Grant the sa_role to the maintenance user so it can replicate insert, update, and delete operations on all tables: sp_role "grant", sa_role, D01_maint 1 5. Create a new role named sap_maint_user_role: create role sap_maint_user_role 6. Grant set session authorization permissions to the maintenance user, allowing it to become another user when applying DDL commands to the replicate database. The permission must be granted to a user or a role, not a login: grant set session authorization to sap_maint_user_role 7. Alias D01 _maint to the database owner in the master and user databases: sp_addalias D01_maint, dbo 8. Automatically activate the maintenance user role upon login: a. Grant the sap_maint_user_role to the maintenance user: sp_role "grant", sa_maint_user_role, D01_maint b. Automatically activate the maintenance user role at login: 18 Adaptive Server Enterprise CHAPTER 3: Installing, Configuring, and Administering the HADR System Using isql alter login D01_maint add auto activated roles sap_maint_user_role Adding the HADR User Login Create a dedicated user login to enable HADR nodes to communicate with each other. The login must: • • • Exist on all HADR nodes Have the sa_role Be added as an external login for each server 1. Add the user login and grant it the appropriate permissions: a) From the master database, issue: create login loginame with password password b) Grant allow hadr login permissions to the login: grant allow hadr login to loginame c) Grant manage hadr: grant role sa_role to loginame d) Add the server as a remote server: sp_hadr_admin 'addserver', 'remote HADR name' e) Add the dedicated user to the remote server: sp_addexternlogin 'remote HADR name', loginame, user_name, password 2. Add the HADR user login to the HADR system: sp_hadr_admin 'setlogin', 'loginame' Initializing the HADR Nodes To initialize HADR nodes (the hosts on which the HADR system runs), name the servers, then configure them as HADR members. The HADR_server_name must be unique within the HADR group. Typically, you use the Adaptive Server server name as the HADR_server_name, but if two or more Adaptive Servers share the same server name, you must choose unique HADR_server_names. For example, if the HADR group COMMON includes Adaptive Servers COM_01 and COM_02, because the Adaptive Server server names are unique within the group, you may use COM_01 and COM_02 as their HADR_server_name values. However, if another HADR group named SAP_ERP includes servers that are all named ERP_SERVICE, you cannot use the Adaptive Server name as the HADR_server_name. Instead, define the HADR members according to their location or some other scheme; for example, ERP_LONDON and ERP_PARIS. Use the sp_hadr_admin 'init' command to define HADR server names. HADR Users Guide 19 CHAPTER 3: Installing, Configuring, and Administering the HADR System Using isql Note: You must also set the local server name for all HADR nodes (that is, select @@servername cannot return NULL). See also • Chapter 6, Upgrading Adaptive Server in an HADR System on page 39 • Chapter 8, sp_hadr_admin Syntax on page 61 Configuring a Node as an HADR Member After you name nodes, you must configure them as primary or standby servers. Prerequisites You must add the HADR user as a remote login once for each remote HADR server. See "Adding the HADR User Login." Task Note: If the Adaptive Servers are using password encryption (that is, net password encryption reqd is set to a value other than DEFAULT), all servers in the HADR group must have encryption set as a server option: sp_serveroption server_HADR_name 'net password encryption', true On all servers: 1. Set the HADR mode configuration parameter to 0: sp_configure 'HADR mode', 0 If there are no initialized HADR groups, the HADR system launches, in starting mode. 2. Create the HADR user login: create login 'user_name' with password 'password' 3. Initialize the HADR group: sp_hadr_admin 'init','group_name','local_HADR_server_name' The HADR mode changes to standby. 4. Add the HADR members (including the local node itself) to the HADR group: sp_hadr_admin 'addserver','HADR_server_name','host_name:port_number' Note: No additional steps are necessary if you are configuring a standby server. 5. If you are configuring a primary server, promote it to primary mode: sp_hadr_admin 'primary','force' 20 Adaptive Server Enterprise CHAPTER 3: Installing, Configuring, and Administering the HADR System Using isql You must include the force parameter if any of the HADR nodes added in step 5 are not online. 6. Activate the primary server: sp_hadr_admin 'activate' Installing Replication Server See the Replication Server Installation Guide for your platform for instructions. Materializing the Databases and Starting Replication After you have installed Replication Server, materialize the databases (including master) before starting replication, making the required logins and users available with the same roles and permissions to other databases. Note: Permissions for the sa and sso users and roles are not copied from the primary to the standby database during master database materialization, This exclusion ensures that any failure during materialization does not result in an inaccessible standby database. Instead, the standby server retains its original, uncorrupted sa and sso permissions. Newly installed standby databases may have default permissions assigned to the sa and sso users and roles. You must duplicate any customizations or additional grants to the sa and sso users and roles at the primary database before materialization can successfully complete. Materializing a database creates an initial copy of data from one site to the other. Once completed, the replication software maintains the accuracy of the target site by continuously applying changes that occur after materialization completes. The tasks required to materialize a database depends on the type and size of the database. See your Replication Server Administration Guide for instructions about materializing databases. After you have materialized the master database, use bcp to copy these tables to the standby server: • • • • • • • • • • syslogins sysloginroles sysremotelogins syssrvroles sysusers sysroles sysalternates sysattributes sysprotects syservers HADR Users Guide 21 CHAPTER 3: Installing, Configuring, and Administering the HADR System Using isql Adding and Dropping Servers from the Member List To add or drop servers from the member list, use the sp_hadr_admin system procedure. To add a server to the member list: sp_hadr_admin 'addserver', 'server_HADR_name' [, 'server_net_name' [, 'nopropagate']] This example adds server1 to the HADR member list: sp_hadr_admin addserver,server1 To remove a server from the member list, use: sp_hadr_admin dropserver, server_name [, "nopropagate”] This example drops server1 from the member list:: sp_hadr_admin dropserver, server1 (return status = 0) Command 'dropserver' successful. (1 row affected) Before promoting a standby server to primary mode, use sp_hadr_admin propagate to ensure that the standby server fetches the latest member list from the current primary server when it starts. Member lists may get outdated if a standby server was unreachable when the primary published its latest member list. sp_hadr_admin propagate does not accept any arguments, but increments the generation number in the member list (even if nothing else has changed) and distributes the member list to member servers and clients that use the HADR capabilities. However, if you issue sp_hadr_admin propagate when the generation number is up to date, the command propagates the current configuration to the specified standby members without notifying client connections with the hadr_list_cap capability set. Use nopropagate to prevent the HADR system from propagating member list changes to the standby servers. Member list changes are updated for user connections when you use nopropagate. 22 Adaptive Server Enterprise CHAPTER 4 Configuring and Administering HADR with the DR Agent For an SAP Business Suite for Adaptive Server disaster recovery solution, use the Disaster Recovery (DR) Agent, which supports automated setup and configuration, monitoring, and administration capabilities. See the DR Agent Users Guide for a complete description of starting, stopping, and administering the HADR system with the DR Agent. Setting the HADR Mode and State The DR Agent automically sets and examines the HADR mode and state (as defined by as defined by @@hadr_mode and @@hadr_state, respectively) during setup and other operations. The HADR mode and state does not change until the ERP database has been materialized. At this point, the primary HADR server should be in primary mode and the active state. Standby HADR servers should be in standby mode and the inactive state. The expected modes are: DR Setup Phase Expected Value for HADR_mode Expected Internal Notes State for Server Pre-setup -1 0 While configuring HADR with the DR Agent, HADR_mode can have a value of 3, and @@hadr_state set to 2, but this is unexpected. 1. Post-setup -1 0 While configuring HADR with the DR Agent, HADR_mode can have a value of 3, and @@hadr_state set to 2, but this is unexpected. 1. Post master database materialization -1 0 Mode and state do not change. HADR Users Guide 23 CHAPTER 4: Configuring and Administering HADR with the DR Agent DR Setup Phase Expected Value for HADR_mode Expected Internal Notes State for Server Post saptools database materialization -1 0 Mode and state do not change. During ERP database materialization 0 2 The HADR configuration runs as a subtask of ERP materialization, but runs materialization externally. Typically, you see this mode only on a primary server that has not been correctly promoted. Post ERP database ma- 1 (primary) terialization 0 (standby) 1 (primary) Expected modes of primary and standby servers in a configured HADR environment. After teardown -1 2 HADR is disabled on all HADR servers. The mode on each server is inactive. After teardown restart -1 0 The HADR mode changes to 0 when you restart Adaptive Server. 2 (standby) 1The HADR server is ready to be initialized when it is in starting mode. The DR Agent does not enable HADR, and proceeds with the initialization. Viewing the HADR sysserver Entries The sysservers table includes entries for HADR groups and HADR servers. These entries change as the replication environment changes. To view the HADR group and HADR server entries, issue: select srvstatus,srvname,srvnetname,srvclass from sysservers srvclass=16 OR srvclass=17 where To view a list of HADR servers, issue: sp_hadr_admin listserver Once you configure the HADR system (that is, after the ERP database materialization), sysservers should have one entry for an HADR group, and two entries for HADR 24 Adaptive Server Enterprise CHAPTER 4: Configuring and Administering HADR with the DR Agent servers. These entries should change only when DR Agent issues sap_disable_replication or sap_teardown. sap_disable_replication removes the remote HADR server entry for both the primary and standby servers. sap_teardown removes all HADR entries. Note: After replication is reenabled, the remote HADR server is not added to the sysservers table until ERP database materialization completes. The expected HADR group and HADR server entries in the sysservers table, depending on the phase of DR setup, are: DR Setup Phase srvs srvname tatus Post ERP materialization (on primary or standby servers) 8 Post disable (on primary) Post disable (on standby) Post enable (on primary) Post enable (on standby) srvnetname srvcla ss SID_group SID_local_logical_ASE_hostDR 17 9 SID_primary_logical_ASE_hostDR primary_physical_ASE_host :ASE_port 16 0 SID_standby_logical_ASE_hostDR standby_physical_ASE_host :ASE_port 16 8 SID_group SID_local_logical_ASE_hostDR 17 9 SID_primary_logical_ASE_hostDR primary_physical_ASE_host :ASE_port 16 8 SID_group SID_local_logical_ASE_hostDR 17 9 SID_standby_logical_ASE_hostDR standby_physical_ASE_host :ASE_port 16 8 SID_group SID_local_logical_ASE_hostDR 17 9 SID_primary_logical_ASE_hostDR primary_physical_ASE_host :ASE_port 16 8 SID_group SID_local_logical_ASE_hostDR 17 9 SID_standby_logical_ASE_hostDR standby_physical_ASE_host :ASE_port 16 SID_group SID_local_logical_ASE_hostDR 17 Post ERP Mat (on 8 primary or standby) HADR Users Guide 25 CHAPTER 4: Configuring and Administering HADR with the DR Agent DR Setup Phase Post sap_tear- srvs srvname tatus srvnetname srvcla ss 9 SID_primary_logical_ASE_hostDR primary_physical_ASE_host :ASE_port 16 9 SID_standby_logical_ASE_hostDR standby_physical_ASE_host :ASE_port 16 N/A N/A N/A N/A down Initializing the Setup Between Two Servers The HADR system is not configured during the initial setup because replication is not complete until after the ERP database has been materialized. 1. Verify that: • The DR_admin user and the replication maintenance user have the allow hadr login privilege. • Both Adaptive Servers are set to disabled mode (if HADR is enabled, a server could belong to a different HADR environment). • The HADR state after processing is set to disabled mode. 2. Issue these commands from either the DR Agent or from isql: sap_set sap_set_host sap_pre_setup_check sap_setup_replication Performing Materialization Once the materialize command processes the ERP database (that is, the primary Business Suite database), the DR Agent automatically configures the Adaptive Server HADR system. If the materialize command includes the username and password parameters, the DR Agent verifies that the user on the standby Adaptive Server has the allow hadr login privilege. The DR Agent allows the materialize command to continue regardless of the HADR mode or state on either Adaptive Server, which supports reissuing the materialize command to finish the process when an error occurs, or after a disable or enable process. 1. The DR Agent configures the HADR system on both Adaptive Servers, and the HADR setup is performed after the dump and load process. For each Adaptive Server, the DR Agent: 26 Adaptive Server Enterprise CHAPTER 4: Configuring and Administering HADR with the DR Agent a) Enables cis rpc handling: sp_configure ‘cis rpc handing’, 1 b) Sets the HADR mode to standby: sp_configure ‘HADR mode’, 0 c) Initializes the HADR and creates the group: sp_hadr_admin init d) Adds the HADR servers: sp_hadr_admin addserver e) Adds the external logins: sp_addexternlogin f) Verifies that the HADR mode is set to standby mode: select @@hadr_mode 2. Promote the primary Adaptive Server by: a) Setting the Adaptive Server to the primary mode: sp_hadr_admin primary b) Activating the Adaptive Server: sp_hadr_admin activate 3. Issue sp_hadr_admin status to verify the HADR mode for each server is set: • Source Adaptive Server – is set to primary mode and an active status. • Target Adaptive Server – is set to standby mode and an inactive status. Performing Failover Failover switches the HADR primary and standby settings between the Adaptive Servers. If the primary site is not available, the failover process might need to promote the standby site without deactivating the primary site. The DR Agent verifies that the source and target hosts identified in the sap_failover and sap_host_available commands match the HADR settings. The source host, if available, must be set to the HADR primary site, and the target host must be set to the HADR standby site. sap_failover: 1. Deactivates the source Adaptive Server, if the source host (the machine on which Adaptive Server runs) is available: sp_hadr_admin ‘deactivate’, 'timeout', 'label' 2. Demotes the source Adaptive Server: sp_hadr_admin ‘standby’ 3. Verifies the label. 4. Promotes the target host: sp_hadr_admin 'primary' HADR Users Guide 27 CHAPTER 4: Configuring and Administering HADR with the DR Agent 5. Activates the target host: sp_hadr_admin 'activate' The DR Agent runs the sap_host_available command when the original primary Adaptive Server is back online and ready to assume the role of the standby Adaptive Server. The original primary Adaptive Server is demoted to standby mode when it starts, because the other Adaptive Server is now configured as the primary. 1. Verify the HADR state with the DR Agent: sap_status path sap_status should report these values for the standby server: • • Hostname – standby HADR Status – inactive (orginial source Adaptive Server is set to standby mode and the inactive state. sap_status should report these values for the primary server: • • Hostname – primary HADR Status – active (original target Adaptive Server is set to primary mode and the active state. 2. Verify replication with the DR Agent: a. Send a ticket from the new primary server to the new standby server: sap_send_trace standy_host_name b. Issue sap_status path to verify that all paths have a Latency and Commit Time. Disabling or Enabling Replication Use disable and enable to stop replication without tearing down the HADR environment. Use the sap_disable_replication and sap_enable_replication commands when a failover process takes too long or the Replication Server partitions are not large enough to capture transactions, or when a materialization process must be reexecuted because of a failure or need to resynchronize the databases. After you execute the sap_disable_replication command, the primary and standby databases are no longer synchronized and must be rematerialized. The sap_enable_replication command resets the replication environment to a post-setup state so it is ready to be materialized. This process temporarily disables the HADR feature, dropping the remote server from the HADR server list on both the primary and standby Adaptive Server. This prevents users from failing over to the standby Adaptive Server. Note: If the HADR system is already configured, the DR Agent verifies that the host identified by the user is the primary site. 28 Adaptive Server Enterprise CHAPTER 4: Configuring and Administering HADR with the DR Agent 1. On the HADR primary Adaptive Server, drop the standby Adaptive Server from the HADR server list: sp_hadr_admin dropserver 2. If the HADR standby Adaptive Server is available, drop the primary Adaptive Server from the HADR server list on the standby Adaptive Server: sp_hadr_admin dropserver 3. Issue select @@hadr_mode and select @@hadr_state to verify that: : • The source Adaptive Server is set to primary mode and an active state. • The target Adaptive Server is set to standby mode and an inactive state. Performing Teardown Use the teardown process to delete the replication environment, disable the Replication Agent threads, and disable the HADR system on Adaptive Server. sap_teardown automatically: 1. Demotes the source Adaptive Server, if the source host (the machine on which Adaptive Server runs) is available: sp_hadr_admin 'standby' 2. Drops all servers from the HADR server list on both Adaptive Servers: sp_hadr_admin dropserver 3. Drops the HADR group from both servers: sp_hadr_admin dropgroup 4. Disables HADR on both servers: sp_configure ‘HADR mode’, -1 5. Disables CIS RPC Handling: sp_configure ‘cis rpc handing’, 0 After issuing the sap_teardown command, verify that the HADR state is inactive on both Adaptive Servers. If you restart Adaptive Server server at this point, the HADR mode changes to unknown. Troubleshooting with the DR Agent Login to the DR Agent to perform troubleshooting tasks. Log into the DR Agent with the isql utility. The syntax is: isql -w999 -Uuser -Ppassword -Shost:port Issue: • sap_help to display commands and usage information. HADR Users Guide 29 CHAPTER 4: Configuring and Administering HADR with the DR Agent • • • sap_version all to display the versions of all Adaptive Server, Replication Servers, DR Agents running on the HADR system sap_set to display all the settings in the environment including ports, host names, and user names. sap_status path to display the path state, or status of replication along a path. View the log file to see commands sent to DR Agent and the servers in the replication environment. The log file is located at $SYBASE/SCC-3_2/plugins/DR/log. See the DR Agent Users Guide. Sybase Replication Server documentation can be found here http://infocenter.sybase.com/ help/index.jsp?topic=/com.sybase.infocenter.help.rs.15.7.1.sp100/doc/html/title.html Installation Issues Known installation issues are: • Verify the standby SAP system is stopped before installing the primary servers. Issue this command to stop the standby system: • If the SAP installer encounters an error when configuring the HADR environment: 1. View the SAP installation log for errors, which contains the error as seen from the SAP installer point of view. 2. If you cannot resolve the error, examine the DR Agent error log on the host running the primary Replication Server. The log is located in $SYBASE/SCC-3_2/plugins/ DR/log. 3. Examine the Adaptive Server and Replication Server error logs for any errors or installation information. 4. Once you resolve the error, retry setup by selecting the Retry button in the SAP Installer (see your SAP installation documentation). If the database you materializing is in use: 1. Log into the standby Adaptive Server and issue sp_who to determine which process or processes are using the database. 2. Each process in the sp_who output has an associated user (see the Adaptive Server Reference Manual: Procedures). 3. Determine which users are using the database being materialized. 4. Have these users log off the server to remove the process. If necessary, use the kill command to remove the process. For example, to kill process number 1378, issue: stopsap r3 • kill 1378 • 30 See the Adaptive Server Reference Manual: Commands 5. Select the Retry button in the SAP installer to rerun materialization (see your SAP installation documentation). Before retrying materialization, you must reset replication: Adaptive Server Enterprise CHAPTER 4: Configuring and Administering HADR with the DR Agent 1. Log into the DR Agent. 2. Issue: sap_disable source_logical_host_name, database_name sap_enable source_logical_host_name, database_name • 3. Select the Retry button in the SAP installer to rerun materialization (see your SAP installation documentation). If you determine the values for net password encryption reqd are out of sync during database materialization, on the primary and standby server: 1. Log into the primary server. 2. Issue sp_configure net passwod encryption reqd to verify its Run Value is set to 1. If the Run Value is set to 0 or 2, issue sp_configure 'net password encryption reqd', 1 See the Adaptive Server System Administration Guide, Volume 1. DR Agent Known Issues The DR Agent includes these known issues: • Hidden maintenance user – The DR Agent supports the Replication Server hidden maintenance user password. However, Replication Server periodically changes the maintenance user password. After executing sap_teardown, you may need to reset the Adaptive Server maintenance user password in the primary and standby server before configuring a new replication environment. Log into the primary and standby server and issue sp_password. For example, this resets the password for the Sybase123 user: sp_password Sybase123, Sybase123, D01_maint See the Adaptive Server Reference Manual: Procedures • • Note: The hidden maintenance user configuration task was moved to the ERP materialization process. Executing commands simultaneously from local and remote DR Agents – Do not execute DR Agent administrative commands (for example, sap_setup_replication and sap_failover) simultaneously from local and remote DR agents. However, you can issue the sap_status command concurrently from local or remote DR Agents. Adaptive Server and Replication Server Domains – The DR Agent assumes that the Adaptive Server and the Replication Server on a logical host use the same network domain. If they do not use the same network domain, the HADR setup fails during ERP database materialization, and issues an error stating the local Adaptive Server could not communicate with the remote Adaptive Server. Issue sp_hadr_admin to manually change the network name. Use sap_set to verify the network name after you add the hosts with sap_set_host (see the DR Agent Users Guide). Examine the rs_hostname and ase_hostname values for each logical hostname to confirm the fully qualified domain names have the same suffixes. HADR Users Guide 31 CHAPTER 4: Configuring and Administering HADR with the DR Agent • • 32 Adaptive Server Dump Directory – The DR Agent cannot validate the location or permissions of the dump directory when Adaptive Server and the Replication Server are located on separate host computers. .sqlanywhere12 Directory – Replication Server uses a SQL Anywhere database to host the embedded RSSD. When you create the SQL Anywhere database, a directory named .sqlanywhere12 is created in the user’s home directory. Because the SQL Anywhere eRSSD functions correctly if this directory is deleted,. you need not backup this directory. SQL Anywhere write the directory in another location if the home directory does not exist. Adaptive Server Enterprise CHAPTER 5 Failover Failover is switching activity to a standby server from the primary server. Note: The steps below describe how to failover using isql. See "Performing Failover", above for information about performing failover with the DR Agent. Failovers are planned or unplanned. A planned failure is scheduled (for example, to perform maintenance on a primary host computer), and allows for an orderly sequence of steps to be performed to move processing to the standby site, which becomes the new primary server. Failover includes changes in direction for Replication Server and other applications to connect to the new primary server. Unprivileged users cannot connect to a standby server until it is promoted to a primary server and put into active mode. Unplanned failovers result in the same chain of events as for a planned failover. When it becomes available, the original primary server is updated with client and application activity performed on the standby server. To return processing to the original primary server, perform another failover. Draining the Transaction Log The HADR system automatically drains Adaptive Server transaction logs during the deactivation process once a server switches to an inactive state (unless you specify the nodrain parameter), or if you execute sp_hadr_admin drain from an inactive server. sp_hadr_admin drain initiates the log drain to the Replication Server. The log is subsequently applied to the standby server. Include the label parameter to mark the drain point in the transaction logs. After you issue sp_hadr_admin drain, 'label': • • Wait for the drain operation to complete on the HADR database on the primary server (issue sp_hadr_admin status on the primary server to monitor the progress of the drain operation). Wait for replication to complete on the standby server for the HADR database (issue sp_hadr_admin status, 'label' on the standby server to monitor replication status). HADR Users Guide 33 CHAPTER 5: Failover Performing a Planned Failover Perform planned failover after the current primary has been deactivated and the Replication Agent has drained the transaction logs of its HADR databases. Prerequisites Check the queue processing time (see admin sqm_process_time in the Replication Server Reference Manual). If can fail over within the allotted time constraints, change the state of the primary server to inactive. Task 1. Deactivate the primary server: sp_hadr_admin deactivate, 'timeout', 'label' 2. Check that all HADR databases have been drained by the Replication Agent on the primary server: sp_hadr_admin status 3. Verify that replication is complete on the standby server by issuing, on the standby server: sp_hadr_admin status, 'label' 4. Move the current primary server to the standby role. On the primary server, issue: sp_hadr_admin standby 5. Move the standby server to primary mode. The new primary server remains in an inactive state and does not allow any transaction activity by unprivileged connections. On the standby, issue: sp_hadr_admin primary 6. Reverse the direction for replication (see the Replication Server Reference Manual for information about commands below). a) Start replication from the new primary server (which was the old standby server) for each HADR database: 1. Enable the secondary truncation point in each HADR database in the primary server: dbcc settrunc(ltm, 'end') 2. Start the Replication Agent in each HADR database in the primary server: sp_start_rep_agent database_name b) Stop replication from the new standby server (which was the old primary server) for each HADR database: 1. Stop the Replication Agent in each HADR database in the standby server: sp_stop_rep_agent database_name 2. Disable the secondary truncation point for each HADR database in the standby server: 34 Adaptive Server Enterprise CHAPTER 5: Failover dbcc settrunc(ltm, ignore) 3. Increase the generation ID in each HADR database in the standby server by incrementing the current value by one: dbcc settrunc ('ltm', 'gen_id', <?>) 4. Clear Replication Server of the secondary truncation point from the standby server: rs_zeroltm server_name, database_name 7. Activate the new primary server: sp_hadr_admin activate See also • Chapter 6, Upgrading Adaptive Server in an HADR System on page 39 • Chapter 8, sp_hadr_admin Syntax on page 61 Performing an Unplanned Failover Perform an unplanned failover only after you determine that you cannot restart the primary server. Any items in the original primary server's transaction log that have not been replicated are lost. If the primary Replication Server is also down, any data in the Replication Server queue not yet applied to the standby server is also lost. You must manually rematerialize the old primary server and bring it back online as standby Drain the Replication Queues If the primary server is unavailable, finish replicating the data that is already in the Replication Server queues before switching to the standby server. If the primary Replication Server is available, use the Replication Server sysadmin command to insert a drain marker (which signals the end of replication for a database) into its inbound queue for each replicated user database: sysadmin issue_ticket, active_ase, active_db, 1, @h1=repdbsync, @h4="label" See the Replication Server Reference Manual. If the primary Replication Server is unavailable, insert a drain marker into the replicate Replication Server outbound queue for each replicated user database: sysadmin issue_ticket, standby_ase, standby_db, 0, @h1=repdbsync, @h4="label" Note: Labels longer than 10 characters are truncated. The drain marker is appended at the end of the queues and replicated to the standby server as a call to sp_repdbsync 'label'. The dataserver name (designated by active_ase and standby_ase, and configured with connect dataserver) must be known to Replication Server. HADR Users Guide 35 CHAPTER 5: Failover Verify Drain Progress The queue drain is complete when the drain marker is sent to the replicate database, sp_repdbsync is issued on that database, and the server status is updated. There are multiple databases and replication paths to drain. Drain markers are logged in the Replication Server log. Use the admin health and admin quiesce_check Replication Server commands to check the status of the drain marker. If all queues are drained, admin health shows: admin health Mode Quiesce Status ---------------- -------- -------NORMAL TRUE HEALTHY If some queues are not drained, admin health shows: admin health Mode Quiesce Status ---------------- -------- -------NORMAL FALSE HEALTHY admin health makes no report if Replication Agent has drained its log. However, admin quiesce_check returns a message similar to this if the log is not drained: admin quiesce_check Msg 4076, Level 12, State 0: Server 'ost_replnxb15_02': RSI is not quiesced. RSI for 16777318:0 sent 3067 unacknowledged bytes When the log is drained, move the standby server to primary mode. The new primary remains in the inactive state and does not allow any transaction activity from unprivileged connections. On the standby server, issue: sp_hadr_admin 'primary' Disable Replication from the Original Primary Server Data in the primary server and the Replication Server that is not replicated before the standby server is promoted to the primary mode is lost. To prevent replication of new data that users enter after you restart the original primary server, issue this command from the standby server: suspend connection to DS.db Set the Secondary Truncation Point and Start the Replication Agents 1. Enable the secondary truncation point in each HADR database in the new primary server: dbcc settrunc(ltm, 'end') 2. Start the Replication Agent in each HADR database in the new primary server: 36 Adaptive Server Enterprise CHAPTER 5: Failover sp_start_rep_agent database_name Activate the Newly Promoted Primary Server Activate the new primary server with sp_hadr_admin activate so it can accept user connections. Disable the Truncation Point Disable the secondary truncation point on the new standby server (the original primary server) at start-up. 1. Stop the Replication Agent in each HADR database in the standby server: sp_stop_rep_agent database_name 2. Disable the secondary truncation point in each HADR database in the standby server: dbcc settrunc(ltm, ignore) 3. Increase the generation ID in each HADR database in the standby server by incrementing the current value by one: dbcc settrunc ('ltm', 'gen_id', <?>) 4. Clear Replication Server of the secondary truncation point from the standby server: rs_zeroltm server_name, database_name Restart the Primary Replication Server Restart the primary Replication Server. See your Replication Server documentation. Drain Replication Server Queues During unplanned failovers, Adaptive Server may attempt to begin replication when it returns to service. After the Replication Agents start and the secondary truncation point is set, purge any data from the Replication Server queues that was replicated by Adaptive Server: 1. Place both Replication Servers in hibernation mode: sysadmin hibernate_on 2. Purge the inbound queues from the old primary Replication Server for each HADR database: sysadmin purge_queue 3. Purge the outbound queues from the old standby Replication Server for each HADR database: sysadmin purge_queue 4. Restore both Replication Servers from hibernation mode: sysadmin hibernate_off Prepare for Subsequent Failovers Reverse the steps you took in "Disable Replication from the Original Primary Server" now that Replication Agents have been stopped and the queues are purged. Issue this for each HADR database on the new primary server to support replication during subsequent failovers: HADR Users Guide 37 CHAPTER 5: Failover resume connection to DS.db 38 Adaptive Server Enterprise CHAPTER 6 Upgrading Adaptive Server in an HADR System Upgrading an Adaptive Server to a new version in an HADR system requires that you perform a number of prerequisites, including upgrading the standby Adaptive Server, and finally, upgrading the primary Adaptive Server. Before you begin the upgrade steps: • If you have disabled the sa login and use a different user to perform system administration, ensure that login has the allow hadr login privilege so it is not redirected to the primary server when connecting to the standby server. To verify that the user has the allow hadr login, issue: select has_privilege("allow hadr login") or: sp_helprotect To grant the allow hadr login privilege to the sapsso role, issue this command, as sa: grant allow hadr login to sso_role • • Verify that login redirection is disabled when you connect to Adaptive Server to perform administration or configuration activities so that, if allow hadr login is not granted, logins to the standby server fail instead of moving to the primary server. For example, use isql to set this property: [isql] CS_PROP_REDIRECT = CS_FALSE in $SYBASE/$SYBASE_OCS/config/ ocs.cfg Use select asehostname to verify the host to which you connect. To verify the host for the sapsso user, log in as sapsa and issue this command on both the primary and standby servers: grant select on asehostname to sso_role • For ERP configuration, the sapsso user does not have the allow hadr login privilege. This restriction prevents accidental changes on the standby server. To unlock the sa login that is used for the Adaptive Server upgrade (and lock it afterwards): 1. Log in to the standby server as sapsa and issue: grant allow hadr login to sso_role 2. 3. 4. 5. Log in to the standby server as sapsso and unlock the sa login, Log in to the standby server as the sa user and perform the upgrade steps. Log in to the standby server as sapsso and lock the sa login. Log in to the standby server as sapsa and issue: HADR Users Guide 39 CHAPTER 6: Upgrading Adaptive Server in an HADR System revoke allow hadr login from sso_role Upgrading the Primary Server or Performing a Rolling Upgrade Perform the steps below to upgrade the primary server. A rolling upgrade lets you update the Adaptive Server or operating system version without taking your HADR system offline. 1. Suspend replication to the standby server. From Replication Server, issue: suspend connection to server_name.database_name 2. Upgrade the standby server or the operating system on the standby machine using the documentation for your platform. 3. Restore replication to the standby server. From Replication Server, issue: resume connection to server_name.database_name 4. Perform a planned failover of Adaptive Server. 5. Suspend replication to the new standby server: suspend connection to server_name.database_name 6. Upgrade the new standby server or the operating system on the new standby machine according to the documentation for your platform. 7. Restore replication to the new standby server: resume connection to server_name.database_name See also • Chapter 8, sp_hadr_admin Syntax on page 61 • Initializing the HADR Nodes on page 19 • Performing a Planned Failover on page 34 Upgrading the Standby Adaptive Server Upgrading standby servers requires that you suspend replication. 1. Log in to the standby Replication Server and suspend replication to the databases that are participating in the HADR system: suspend connection to server_name.database_name For example, this suspends replication for the D01 and master databases on the D01_standby server: suspend connection to D01_standby.D01 suspend connection to D01_standby.master 2. Upgrade Adaptive Server following the instructions in the Adaptive Server Enterprise Installation Guide for your platform. 3. Log in to the standby Replication Server and resume replication to the participating databases: resume connection to server_name.database_name 40 Adaptive Server Enterprise CHAPTER 6: Upgrading Adaptive Server in an HADR System For example, this resumes replication for the master, SID, and saptools databases on the D01_standby server: resume connection to D01_standby.D01 resume connection to D01_standby.master resume connection to D01_standby.saptools HADR Users Guide 41 CHAPTER 6: Upgrading Adaptive Server in an HADR System 42 Adaptive Server Enterprise CHAPTER 7 SDK and Open Server Support for HADR SDK 15.7, Open Client 15.7, and Open Server 15.7 support Adaptive Server HADR. SDK Support for HADR Open Client 15.7, jConnect™ for JDBC™ 7.07, and Adaptive Server ODBC Driver 15.7 support Adaptive Server high availability disaster recovery (HADR). An HADR configuration consists of two or more servers, one of which is designated as the primary, on which all the transaction processing by user applications takes place, and the other as a standby, which acts as warm standby to the primary server. Any state changes of the primary server (for example, changing to deactivating or deactivated), activates the standby server (which becomes the new primary server), and cancels deactivation of the original primary server. Adaptive Server sends specific HADR state change messages. The client application must act on these messages to decide the course of action. To support this functionality, jConnect and Adaptive Server ODBC Driver have added new connection properties. Open Client Services There are two Open Client services that support disaster recovery (DR). • • The nokill capability allows applications to hold on to a connection to a primary server in a deactivating state until the primary server is activated. Use this capability for applications that create multiple connections to a server so that network bandwidth is not consumed by applications that are trying to reconnect while the primary server is deactivated. A callback receives notifications about DR map changes, which allows an application to provide connection information directly to the Client-Library. Note: DR Map is the client side view of the HADR membership list. It allows a client application to determine address information for each member of the HADR system. DR Map Callback When a DR map callback routine is installed on a server connection that supports DR map messages, the callback routine is called asynchronously when the DR map changes. The call signature: CS_RETCODE CS_PUBLIC (*CS_DR_MAP_CB)(CS_CONNECTION *conn, CS_INT info_count, CS_CTLINFO_T *info_ptr) HADR Users Guide 43 CHAPTER 7: SDK and Open Server Support for HADR This callback routine must be installed before connecting to the server. Installing this callback sets the CS_RESP_LIST_DR_MAP capability, which requests that the server send DR map information messages. If the server being connected to has existing map information, the routine is called during ct_connect(). At anytime during the connection, including during the ct_close() call, the callback routine may be invoked. Client-Library has removed the need for the application to look at the parameter name unless it is an unknown parameter that is added in the future release. Applications use CTL type information to determine the parameter significance. The message parameter contains the mapping information. To attach semantic meaning to the various values, Client-Library examines the parameter names and map them to CTL types. Then the application applies the defined meaning to the value. For example, Servername for CS_CTL_DSNAME. CS_CTLINFO_T Parameters The expected parameter names that Adaptive Server sends correspond to the CS_CTLINFO_T field. 44 Parameter Name Description CTL type Datatype Group Name The name of the group CS_CTL_GROUPfor the DR map. Adap- NAME tive Server can send only a single instance of Group Name. Character string Generation Number The generation numCS_CTL_GENERAber of the DR map. TION Adaptive Server can send only a single instance of Generation Number. The Generation Number monotonically increases for each Group Name and must be greater than or equal to 0. Integer Adaptive Server Enterprise CHAPTER 7: SDK and Open Server Support for HADR Parameter Name Description CTL type Datatype Data Source Name A map node name, CS_CTL_DSNAME which may correspond directly to a server name in the interfaces file. You can expect one or more instances of this parameter from Adaptive Server. Character string Data Source Address A valid address for the CS_CTL_ADDRSTR most recently viewed Data Source Name. Multiple Data Source Address parameters can follow a Data Source Name. The format of the address is “protocol address”, for example, tcp serverhost 3000. Character string HAFailover The HAFailover paCS_CTL_HANAME rameter contains a Data Source Name that can be used with an HA-aware client. This information is used by Client-Library fail over to an alternate data source. Character string Client libraries do not handle multiple HAFailover names. They need to be coalesced into a single name. DR Map Callback Example Example of the sequence of events within the DR map callback. for (i = 0; i < info_count; i++) { switch (info_ptr[i].ctl_type) { case CS_CTL_GROUPNAME: HADR Users Guide 45 CHAPTER 7: SDK and Open Server Support for HADR } … } validate values and perform post-processing return CS_SUCCEED } Note: The group name and generation number can be examined before the for loop, since they are expected to be the first two entries in the array. However, Open Client does not enforce this. The only time you expect to see a map sent from the server with a previously seen generation number is during login. Open Client Capabilities Open Client capabilities that support HADR. Capabilities Description CS_RES_DR_NOKILL This response capability requests the server to not terminate the connection when terminating new commands sent to the primary server, if the server is active or inactive. Simultaneously, this capability also allows logins to the primary server. The default connection of the server is inactive. The connection survives until the server becomes active or a new primary server is chosen. The server defines the usage of this capability. This capability modifies the messages that an application receives, and enables queue operation on the server. See the server documentation. CS_ RES_LIST_DR_MAP This response capability requests the server to send the DR map information at login or when the DR map is updated. This information may be sent asynchronously from the server. This bit is set only if there is a CS_DR_MAP_CALLBACK handler installed. Any direct attempt to set this bit through ct_capability(), is silently ignored. CS_REQ_READONLY This request capability sets CS_PROP_READONLY to CS_TRUE. See the property description. CS_PROP_READONLY Connection Property An application can set the CS_PROP_READONLY connection property to indicate that the requests sent on this connection conform to the server definition of a read-only client. Connection Properties for HADR in jConnect The HADR_MODE property enables jConnect to use the HADR functionality of Adaptive Server. The HADR_MODE property allows users to set values to enable or disable HADR. By default, HADR mode is disabled. Valid settings for this property include: 46 Adaptive Server Enterprise CHAPTER 7: SDK and Open Server Support for HADR • • • • • None (or false) – jConnect does not record any HADR state change messages. HADR_MAP – jConnect requests the Adaptive Server to send the HADR address list to the application upon login as well as asynchronously when the list is modified, informing the application of the available primary and standby servers, and whether any companion servers are available. HADR_NOKILL – by setting this property jConnect requests Adaptive Server, not to terminate an existing connection while current primary is deactivating or deactivated. The connection will not be reset by Adaptive Server, until a new server is made Primary. HADR_NOKILL_WITH_MAP – jConnect requests the Adaptive Server to not terminate the connection. When cancelling new commands is issued by the connection while the primary is in deactivated state or undergoing deactivation. Additionally, HADR_NOKILL_WITH_MAP requests Adaptive Server to send the HADR address list to the application upon login as well as asynchronously when the list is modified, informing the application of the available primary and standby servers, and whether any companion servers are available. HADR_NOKILL_WITH_MAP property is equivalent of setting HADR_MAP and HADR_NOKILL properties. HADR_RECONNECT – jConnect implicitly sets HADR_NOKILL_WITH_MAP. jConnect supports reconnection from the old primary to the newly activated primary if the HADR configuration has at least one standby Adaptive Server. After the reconnection is done, an exception JZ0F3 SQLState is thrown. To use this connection, the application must catch the JZ0F3 SQLException, and restore the context, for example, whether the database in use, set options if any, reprepare statements, cursors, and so on. If any transaction is cancelled due to the force option used in the sp_hadr_admin deactivate command, the application is informed, and must retry the transaction. See Adaptive Server Documentation for force option. See HADR reconnection in jConnect section. Retrieve HADR State Change Messages from jConnect jConnect receives messages about HADR state changes, which are sent by Adaptive Server. To retrieve these messages, jConnect application must implement the SybMessageHandler interface: import com.sybase.jdbcx.SybMessageHandler; public interface SybMessageHandler { public SQLException messageHandler(SQLException sqe); } The jConnect application must register the SybMessageHandler class using the SybConnection.setSybMessageHandler(SybMessageHandler hndlr) method. jConnect calls this handler whenever it receives any messages from Adaptive Server, or any other error messages, including state change messages. See Messages about HADR state change in jConnect and Adaptive Server ODBC Driver section for message list. The example shows how to code the message handler: HADR Users Guide 47 CHAPTER 7: SDK and Open Server Support for HADR class HADRMsgHandler implements SybMessageHandler { public SQLException messageHandler(SQLException sqe) { System.out.println("============="); System.out.println(sqe.getMessage()); int errorCode = sqe.getErrorCode(); System.out.println(errorCode); if(errorCode == 2378) { ResultSet rs = ((SybSQLException) sqe).getEedParams(); try { System.out.println("Deactivation time : " + rs.getTimestamp(1)); } catch (SQLException s) { } } return sqe; } } //register the SybMessageHandler implementation. _conn.setSybMessageHandler(new HADRMsgHandler()); Retrieve DR_LIST_MAP Connection Property from jConnect You must set HADR_MODE to HADR_MAP or HADR_NOKILL_WITH_MAP in jConnect to extract HADR_LIST_MAP out of these properties. When jConnect sets HADR_MODE to HADR_MAP or HADR_NOKILL_WITH_MAP, it receives the DR_LIST_MAP during login, and whenever there is a topology change. To retrieve DR_LIST_MAP, jConnect application must call SybConnection.getClientInfo() method. SybConnection.getClientInfo() returns the property object. jConnect application must extract HADR_LIST_MAP out of these properties, which returns: LinkedHashMap<string, object> To retrieve the HADR_LIST_MAP components from this LinkedHashMap, the keys are to be retrieved: GroupName GenerationNumber Primary Standby_1 Standby_2 Standby_3 . . . . Standby_n 48 Adaptive Server Enterprise CHAPTER 7: SDK and Open Server Support for HADR The GroupName, GenerationNumber, and Primary key names are fixed. To get the exact number standby servers in the list (Standby_n), jConnect must execute props.size()-3. This gives an exact number of standby servers in the list. public static void testHadrMap() throws Exception { Class.forName("com.sybase.jdbc3.jdbc.SybDriver"); SybConnection conn = (SybConnection) DriverManager.getConnection( "jdbc:sybase:Tds:SERVER:PORT? ENABLE_HADR_NOKILL=TRUE&ENABLE_HADR_LIST_MAP=TRUE", "USER", "PASSWORD"); Properties props =((SybConnection) conn).getClientInfo(); LinkedHashMap hadrMap = (LinkedHashMap)props.get("HADR_LISTMAP"); //Populate the ListMap System.out.println("Group Name : " + hadrMap.get("GroupName")); System.out.println("Generation Number: " + hadrMap.get("GenerationNumber")); printDSN(hadrMap, "Primary"); for(int i=1; i<=hadrMap.size()-3; i++) { printDSN(hadrMap, "Standby_" + i); } } public static void printDSN(LinkedHashMap hadrMap, String dsnType) { System.out.println("_________" + dsnType + "_________"); Properties dsnInfo = (Properties)hadrMap.get(dsnType); System.out.println(dsnInfo.get("DataSourceName")); LinkedList dataSourceAddressList = (LinkedList)dsnInfo.get("AddressList"); LinkedList haFailoverList = (LinkedList)dsnInfo.get("HAFailoverList"); int flags = ((Integer)dsnInfo.get("Flag")).intValue(); for(int i=0; i< dataSourceAddressList.size(); i++) { System.out.println(dataSourceAddressList.get(i)); } for(int i=0; i< haFailoverList.size(); i++) { System.out.println(haFailoverList.get(i)); } } System.out.println("Flag: " + flags); HADR Users Guide 49 CHAPTER 7: SDK and Open Server Support for HADR HADR Reconnection in jConnect jConnect supports reconnecting from the old primary to the newly activated primary, if the HADR configuration includes at least one standby Adaptive Server. To use the reconnection feature, application users must set HADR_MODE to HADR_RECONNECT. If HADR failover occurs and a new primary server is activated, jConnect internally connects to the newly activated primary server and throws a SQLException with SQLState, JZ0F3. The application must catch this SQLException, check the SQLState, and determine the appropriate course of action. The original connection object is available for use. If the force option was not used during deactivation, any transactions in progress must have been already completed. Any client that has had transactions cancelled due to the force option used in the deactivate command, receives a message, and must retry the transaction. Application must restore the context like database is in use, set options if any, reprepare statements, cursors and others. For example: public static void reconnect() throws Exception { SybConnection conn = null; try { Class.forName("com.sybase.jdbc4.jdbc.SybDriver"); conn = (SybConnection) DriverManager.getConnection( "jdbc:sybase:Tds:linuxjconnbld64:15780/jdbc_test_bacardi? HADR_MODE=HADR_NOKILL_WITH_MAP&CONNECT_READONLY=false&ENABLE_REDIRE CTION=true", "jladhe", "asehadr"); System.out.println("Connected to ASE."); ResultSet rs1 = conn.createStatement().executeQuery("select @@servername"); rs1.next(); System.out.println("ASE Name: " + rs1.getString(1)); InputStreamReader isReader = new InputStreamReader(System.in); BufferedReader reader = new BufferedReader(isReader); \n"); System.out.println("\nBefore entering the following input. 'myhadr'"); System.out.println("isql with sa to primary"); System.out.println("sp_hadr_admin deactivate, '2', System.out.println("sp_hadr_admin standby"); System.out.println("isql with sa to standby"); System.out.println("sp_hadr_admin primary"); System.out.println("sp_hadr_admin activate\n"); // Explicitly put an user input here to emulate long applications. 50 Adaptive Server Enterprise CHAPTER 7: SDK and Open Server Support for HADR and // This will give time slice to deactivate current primary separate // promote standby to primary and then activate it from // isql session. System.out.println("Please enter a number:"); int firstInt = Integer.parseInt(reader.readLine()); //This code will not be exercised conn.createStatement().executeUpdate("create table deleteMe(a int)"); ResultSet rs = conn.createStatement().executeQuery( "select @@version"); rs.next(); System.out.println("ASE Version: " + rs.getString(1)); } catch (SQLException sqe) { if(sqe.getSQLState().equalsIgnoreCase("JZ0F3")) { ResultSet rs = conn.createStatement().executeQuery("select @@servername"); rs.next(); System.out.println("ASE Name after new connection: " + rs.getString(1)); } } } Connection Properties for HADR in Adaptive Server ODBC Driver Several properties in Adaptive Server ODBC Driver support the HADR functionality of Adaptive Server. • • • DRNoKillDuringDeactivation – when this property is set to 1 (the default is 0), Adaptive Server ODBC Driver requests the Adaptive Server to not terminate the connection when cancelling new commands issued by the connection while the primary is in deactivated state or undergoing deactivation. HADRlist – when this property is set to 1 (the default is 0), Adaptive Server ODBC Driver requests the Adaptive Server to send the HADR address list to the application upon login as well as asynchronously when the list is modified, informing the application of the available primary and standby servers, and whether any companion servers are available. EnableRedirection – EnableRedirection connection property is 1 (the default). When enabled, Adaptive Server ODBC Driver allows the server to redirect the connection to an alternate server (cluster) or the primary server (HADR). Setting EnableRedirection to 0 disables redirection. HADR Users Guide 51 CHAPTER 7: SDK and Open Server Support for HADR HADR Information Messages from Adaptive Server ODBC Driver Adaptive Server ODBC Driver is informed of the start of a deactivation process, cancellation of an ongoing deactivation process, successful transition to a deactivated state, or the transition from deactivated state to active state by the Adaptive Server. Connections that enable the HADRList or DRNoKillDuringDeactivation connection property receive these messages when a state change occurs. See Messages about HADR state change in jConnect and Adaptive Server ODBC Driver section. Using the following connection attributes, the application can monitor the current state of the server: • SQL_ATTR_DR_INFO_MSG – retrieves the current state of the connection. An application can poll the connection by calling SQLGetConnectAttr function and passing in the SQL_ATTR_DR_INFO_MSG connection attribute. The value is set to a SQLINTEGER value of the most recent informational message received: SQL_DR_ACTIVATED, SQL_DR_DEACTIVATION_STARTED, SQL_DR_DEACTIVATED, SQL_DR_DEACTIVATION_CANCELED, or SQL_DR_NO_MESSAGE_RECEIVED. • SQL_ATTR_DR_DEACTIVATION_TIMEOUT – retrieves the time at which the deactivation will occur. An application can call SQLGetConnectAttr function and pass in the SQL_ATTR_DR_DEACTIVATION_TIMEOUT connection attribute. The value is set to a SQL_TIMESTAMP_STRUCT containing the exact time (in the universal time zone) when the deactivation timeout will occur. SQL_ATTR_DR_INFO_CALLBACK – applications that link directly to the Adaptive Server ODBC Driver can avoid polling by registering a callback function using SQLSetConnectAttr and setting SQL_ATTR_DR_INFO_CALLBACK to the address of the callback function. This function is called when an HADR informational message is received. The callback function will not be called on inactive connections because the connection is not proactively monitored. When the application executes a statement or fetches rows from the resultset, messages will be received. The syntax for the state events callback function is: • void HADRInfoCallback(HDBC conn, SQLINTEGER new_state); Applications that do not monitor the information messages will still be notified of some of the events as errors when executing statements. The connection property DRNoKillDuringDeactivation controls the nature of the messages. • • 52 DRNoKillDuringDeactivation =0 – primary server starts deactivation, it kills the connection to the server once it has completed its current transaction. Before killing the connection, the server sends information to the client that is about to be killed. Adaptive Server ODBC Driver sends an error that is added to the error list reported by ODBC. No client error is reported. DRNoKillDuringDeactivation =1 – primary server starts deactivation, it does not kill the connection until a new primary server is active. Instead of killing the connection, the server rejects any new request (once all active transactions have completed). The error: Adaptive Server Enterprise CHAPTER 7: SDK and Open Server Support for HADR Primary server is being deactivated When a server becomes the new primary server, the server informs the client, then kills the connection. It is the responsibility of the application to establish a new connection with the new primary server. When the server completes deactivation, it sends the client a message and rolls back any open transactions. There may be open transactions if the deactivation was done with the force option, and if the timeout is triggered. If this server later becomes the active server, the connection starts executing the commands in a new transaction. HADRList Connection Property from Adaptive Server ODBC Driver Connections that enable the HADRList property receive a message from the server upon login and any time the HADR data source list changes. The data source list contains the current primary server (listed first), followed by all available standby servers. Each data source enumerates a list of addresses (which refer to the same server) and a list of high-availability companion data sources available for that data source. To retrieve these messages, Adaptive Server ODBC Driver application can poll the connection by calling the SQLGetConnectAttr function: • • SQL_ATTR_DR_LIST_GENERATION – this connection property returns a SQLINTEGER identifying the generation of the data source list. The application uses this property to decide if it already has the current list before getting the SQL_ATTR_HADR_LIST attribute. SQL_ATTR_DR_LIST – this connection property returns a structure containing the details of the data source list. SQLGetConnectAttr function copies the data into the memory provided through the ValuePtr parameter. If the BufferLength is not large enough, SQLGetConnectAttr sets StringLengthPtr to the size needed and returns SQLERROR with a data overflow error: SQLState=HY000, MessageText=Data overflow. Increase specified column size or buffer size, NativeError=30128 • If the data is copied, StringLengthPtr is set to the number of bytes used. ValuePtr must be aligned on a 16-byte boundary. SQL_ATTR_HADR_LIST_CALLBACK – applications that link directly to the Adaptive Server ODBC Driver can avoid polling by registering a callback function using SQLSetConnectAttr and setting SQL_ATTR_HADR_LIST_CALLBACK to the address of the callback function. This function is called when an updated list is received from the server. The syntax of the callback function is: void HADRListAllocCallback(HDBC conn, SQLINTEGER generation_number, SQLLEN size_needed); where: • conn – is the connection handle on which the message was received. HADR Users Guide 53 CHAPTER 7: SDK and Open Server Support for HADR generation_number – is the generation number of the new list which determines whether the application must retrieve the new list or if it already has it from a different connection. • size_needed – is the amount of memory needed to hold the new list. When the callback function is called, the application, if it decides to update its list, may call SQLGetConnectAttr function and retrieve the SQL_ATTR_HADR_LIST attribute to get the new list. • DataSourceList structure struct SQLHADRDataSourceList { // The generation number of this list SQLINTEGER generation; // The group name of this list. Regardless of the setting of // SQL_OUTPUT_NTS, this name is null terminated. SQLWCHAR* group_name; // The length of the group_name in bytes. SQLLEN group_name_length; // The number of data sources in the data source list. SQLLEN number_of_data_sources; }; // An array of size number_of_data_sources containing pointers to // each SQLHADRDataSource in the list. SQLPOINTER* data_source_list; Struct SQLHADRDataSource { // The name of the data source. Regardless of the setting of // SQL_OUTPUT_NTS, this name is null terminated. SQLWCHAR* data_source_name; // The length of data_source_name in bytes SQLLEN data_source_name_length; // The number of address for this data source. Each address refers // to the same data source (server) SQLLEN number_of_addresses; // An array of size number_of_addresses containing pointers to each // address in the array. The addresses are in the same format as // addresses in the interfaces file. Regardless of the setting of // SQL_OUTPUT_NTS, the addresses are null terminated. SQLWCHAR** address_list; // An array of size number_of_addresses containing the byte length of // each element in the address_list array. SQLLEN* address_list_lengths; // The number of HA companions available for this data source. 54 Adaptive Server Enterprise CHAPTER 7: SDK and Open Server Support for HADR SQLLEN number_of_ha_companions; to // An array of size number_of_ha_companions containing pointers // the SQLHADRDataSource for each of the HA companion servers. SQLHADRDataSource** ha_companion_list; // This SQLINTEGER is treated as a set of flags for the data source. // Currently, the only flag defined is SQL_DR_READONLY. SQLINTEGER flags; }; Messages About HADR State Change in jConnect and Adaptive Server ODBC Driver jConnect and Adaptive Server ODBC Driver receive messages about HADR state changes, which are sent by Adaptive Server. Error Code Message Description 2376 New primary has been ac- • tivated. Available through ODBC SQL_ATTR_DR_INFO_MSG or SQL_ATTR_DR_INFO_CALLBACK • Available through jConnect Syb- Action Client application must establish a new connection to the newly activated primary. Connection.getClientInfo() 2377 Primary server is being deactivated. • Available through ODBC SQL_ATTR_DR_INFO_MSG or SQL_ATTR_DR_INFO_CALLBACK • Application should complete the current transaction and avoid any new transactions. Available through jConnect SybConnection.getClientInfo() 2378 Deactivation did not com- • plete successfully. Primary is once again active. Available through ODBC SQL_ATTR_DR_INFO_MSG or SQL_ATTR_DR_INFO_CALL- Client application may resume normal processing. BACK • Available through jConnect SybConnection.getClientInfo() HADR Users Guide 55 CHAPTER 7: SDK and Open Server Support for HADR Error Code Message Description 2379 Primary has been success- • fully deactivated. Action Available through ODBC SQL_ATTR_DR_INFO_MSG or SQL_ATTR_DR_INFO_CALLBACK. • Available through jConnect Syb- Client application should roll back any open transactions and avoid any new transactions. Connection.getClientInfo() 2380 Primary has been reactiva- • ted. Available through ODBC SQL_ATTR_DR_INFO_MSG or SQL_ATTR_DR_INFO_CALL- Client application may resume normal processing. BACK • Available through jConnect SybConnection.getClientInfo() 9662 Available through jConnect SybCon- See Adaptive Server Enterprise System Administration Guide. nection.getClientInfo() 9663 The RepAgent Thread for Available through jConnect SybCon- See Adaptive Server database '%.*s' is not run- nection.getClientInfo() Enterprise System ning. Administration Guide. 9664 At least one RepAgent Available through jConnect SybConThread did not complete nection.getClientInfo() its processing of the transaction log. See Adaptive Server Adaptive Server is runAvailable through jConnect SybConning in 'Standby' mode. nection.getClientInfo() The user does not have 'allow hadr login' privilege. See Adaptive Server Adaptive Server is runAvailable through jConnect SybConning in 'Primary Inactive' nection.getClientInfo() mode. The user does not have 'allow hadr login' privilege nor the 'hadr_gentle_cap' capability is set on the connection. See Adaptive Server 9665 9666 56 Unable to issue a checkpoint on database '%.*s'. Enterprise System Administration Guide. Enterprise Security Administration Guide. Enterprise Security Administration Guide. Adaptive Server Enterprise CHAPTER 7: SDK and Open Server Support for HADR Error Code Message 9667 Adaptive Server is runAvailable through jConnect SybConning in 'Primary Deactinection.getClientInfo() vating' mode. The user does not have 'allow hadr login' privilege nor the 'hadr_gentle_cap' capability is set on the connection. 9668 9670 9672 9673 9674 9671 Description Action See Adaptive Server Enterprise Security Administration Guide. Login redirection to Primary server failed. Available through jConnect SybCon- See Adaptive Server The truncation point of database '%.*s' has not been established with DBCC SETTRUNC. Available through jConnect SybCon- See Adaptive Server Server reached INACTIVE state. Available through jConnect SybCon- See Adaptive Server Initiating log drain mechanism. Available through jConnect SybCon- See Adaptive Server User connections statistics:: %d in xact, %d in chained mode, %d in unchained mode, %d holding server side cursors. Available through jConnect SybCon- See Adaptive Server Deactivation failed due to %s. Resetting state to %s. Available through jConnect SybCon- See Adaptive Server HADR Users Guide nection.getClientInfo() nection.getClientInfo() nection.getClientInfo() nection.getClientInfo() nection.getClientInfo() nection.getClientInfo() Enterprise System Administration Guide. Enterprise System Administration Guide. Enterprise System Administration Guide. Enterprise System Administration Guide. Enterprise System Administration Guide. Enterprise System Administration Guide. 57 CHAPTER 7: SDK and Open Server Support for HADR Error Code Message Description Action 3957 New transaction cannot be Available through jConnect SybConstarted due to an ongoing nection.getClientInfo() HADR deactivate operation. The command could not be completed. See Adaptive Server Enterprise System Administration Guide. 3958 Ongoing transactions Available through jConnect SybCon- See Adaptive Server have been rolled back due nection.getClientInfo() Enterprise System to HADR deactivate. Administration Guide. 15968 %s: parameter %d is not specified. Available through jConnect SybCon- See Adaptive Server Enterprise System Administration Guide nection.getClientInfo() Open Server Support for HADR Open Server provides new features to support DR functionality. New Capabilities Three new capabilities such as, CS_RES_DR_NOKILL, CS_RES_LIST_DR_MAP, and CS_REQ_READONLY are added for Open Server. These are the same capabilities that Client-Library has. When these capabilities are set, Open Server reports the setting of the capabilities to the application. Note: The externally visible values for these capabilities are the same for both Open Server and Open Client. DR message Use the control information message named SRV_CTL_DRMAP with srv_send_ctlinfo(). Existing control parameters continue to provide address-related parameters while additional control types are added which map to the above described parameters: 58 Control Parameter Type DR Map Message Parameter SRV_CT_SERVERNAME Data Source Address SRV_CT_TRANADDR Data Source Address SRV_CT_ADDRSTR Data Source Address SRV_CT_OPTION Data Source Flags Adaptive Server Enterprise CHAPTER 7: SDK and Open Server Support for HADR Control Parameter Type DR Map Message Parameter SRV_CT_HANAME HAFailover SRV_CT_GROUPNAME Group Name SRV_CT_GENERATION Generation Number SRV_CT_DSNAME Data Source Name HADR Users Guide 59 CHAPTER 7: SDK and Open Server Support for HADR 60 Adaptive Server Enterprise CHAPTER 8 sp_hadr_admin Syntax Configures and maintains the HADR system. Syntax Specifies the login name to use for HADR system communication. sp_hadr_admin setlogin, 'login_name' Creates the HADR group, setting the group name to group_name and the HADR server name to local_HADR_server_name. sp_hadr_admin init, 'group_name' [, 'local_HADR_server_name'] Adds a server to the HADR member list: sp_hadr_admin addserver, 'HADR_server_name' [,[pname] [,'nopropagate' ]] Removes the server name from the member list: sp_hadr_admin dropserver, 'HADR_server_name' [, 'nopropagate'] Displays the local server's member list: sp_hadr_admin listserver Distributes member list changes to all standby servers and hadr_list_cap enabled user connections: sp_hadr_admin propagate Promotes the standby server to a primary server: sp_hadr_admin primary, ['force'] Resumes user application transaction activity on the new primary member after fail over completes: sp_hadr_admin activate Moves the primary server to an inactive state: sp_hadr_admin deactivate, 'timeout_period', 'label' [,{'force' | NULL} [,'nodrain']] Stops the deactivation process and moves the server back to an active state: sp_hadr_admin cancel Changes the mode of an inactive HADR member from primary to standby mode. If the state of the primary server is not already inactive, use the deactivate parameter to change the state to inactive before issuing standby: sp_hadr_admin standby HADR Users Guide 61 CHAPTER 8: sp_hadr_admin Syntax Initiates a log drain to Replication Server: sp_hadr_admin drain, 'label' Checks the progress of the deactivation process from primary and standby servers: sp_hadr_admin status [, 'label'] Retrieves member information: sp_hadr_admin map Removes the specified HADR group and stops the local node from participating in an HADR system: sp_hadr_admin dropgroup, 'group_name' Estimates of the amount of time it takes to roll back transactions: sp_hadr_admin fail over_timeestimate [,standby_server_name] Parameters • • • • • • • • • • • • • • setlogin – configures the login name to use for HADR system communication. login_name – specifies the name of the login account you are configuring for HADR communication. init – creates an HADR group on the local machine. group_name – specifies the HADR group you are adding or dropping. local_HADR_server_name – specifies the HADR server name. The default is @@servername. addserver – adds a server to the HADR system and member list. HADR_server_name – server you are adding to–or dropping from–the HADR system or member list. pname – name specified in the interfaces file for the server named HADR_server_name. nopropagate – disables automatic propagation of the server name to the member list for standby servers and to hadr_list_cap enabled user connections (user connections with HADR capabilities). listserver – displays the local server's member list. propagate – distrubutes member list changes to standby servers and to hadr_list_cap enabled user connections (user connections with HADR capabilities). dropserver – drops a server from the member list. primary – promotes the standby server to primary mode. force – • • 62 Forces a standby server to the role of a primary server if some members of the HADR system are unreachable, or, Forces a transition to an inactive state if there are ongoing (unprivileged or otherwise) transactions when the timeout expires. Adaptive Server Enterprise CHAPTER 8: sp_hadr_admin Syntax • • • • activate – activates the inactive primary server, and allows it to resume transaction activity by user applications. deactivate – moves the primary server to an inactive state. All transactional activity is gracefully terminated on HADR and non-HADR databases by all connections (including privileged connections), and the transaction logs for all HADR databases are drained. timeout_period – specifies the duration, in seconds, during which the primary server is allowed to remain in the deactivating state. If timeout_period expires before the server moves to the inactive state, it is moved back to the active mode. You must use quotes for the numerical value. label – • • • • • • • • • • Specifies the string used by the log drain mechanism to mark the transaction logs of those HADR databases that have been successfully drained by Replication Agent during the deactivation or drain process, or, • Specifies the string used to retrieve the status of transaction replication on the standby server. label is ignored if you issue sp_hadr_admin status [,'label'] on the primary server. nodrain – allows deactivation to occur without initiating a log drain. By default, if you do not include the nodrain parameter, the server initiates a log drain. cancel – aborts the deactivation process. standby – moves the primary server to standby mode. drain – drains the transaction log. status – checks the progress of the deactivation process. The information status reports depends on the mode of the server (primary or standby). map – retrieves information about client connections for which you have set hadr_list_cap capabilities. dropgroup – drops the HADR group. You must drop all the HADR members before issuing dropgroup. fail over_timeestimate – estimates the amount of time it takes to roll back open transactions. standby_server_name – unused in this version. Examples • Example 1 – configures hadruser as the login for HADR communication: create login hadruser with password hadruser123 go grant role ‘sa_role’ to ‘hadruser’ go sp_hadr_admin setlogin, ‘hadruser’ go • Example 2 – initializes the HADR system and sets the HADR group name to IT_GROUP: sp_hadr_admin init, 'IT_GROUP' HADR Users Guide 63 CHAPTER 8: sp_hadr_admin Syntax • Example 3 – initializes the HADR system, sets the HADR group name to IT_GROUP, and sets the HADR server name to PARIS: sp_hadr_admin init, 'IT_GROUP', 'PARIS' • Example 4 – adds a server named PARIS to the member list: sp_hadr_admin addserver, PARIS (return status = 0) Adding server 'PARISDR', physical name 'PARIS' Server added. Command 'addserver' successful. (1 row affected • Example 5 – adds the PARISDR server to the member list using the pname format hostname:port: sp_hadr_admin addserver, PARIS, 'daisy:4901' (return status = 0) Adding server 'PARISDR', physical name 'daisy:4901' Server added. Command 'addserver' successful. (1 row affected) • Example 6 – drops the server PARIS from the HADR group without propagating the change to the other members in the group: sp_hadr_admin dropserver, PARIS, 'nopropagate' • Example 7 – displays the current member list: sp_hadr_admin listserver name network_name ------------ -------------------------------LONDON lily:4901 PARIS daisy:4901 • Example 8 – propagates membership list changes to the standby server: sp_hadr_admin propagate (return status = 0) (1 row affected) (return status = 0) (1 row affected) (return status = 0) Adding server 'LONDONDR', physical name 'lily:4901' Server added. Adding server 'PARISDR', physical name 'daisy:4901' Server added. (1 row affected) (1 row affected) (return status = 0) (1 row affected) (0 rows affected) (return status = 0) Command 'propagate' successful. (1 row affected) sp_hadr_admin listserver now displays the updated member list when issued on the standby server: 64 Adaptive Server Enterprise CHAPTER 8: sp_hadr_admin Syntax sp_hadr_admin listserver name network_name ------------ -------------------------------LONDON lily:4901 PARIS daisy:4901 • Example 9 – attempts to promote the server LONDON to primary mode, but the HADR system cannot connect to the PARISDR server: sp_hadr_admin primary Msg 11206, Level 16, State 1: Server 'JAIDB', Line 1: Unable to connect to server 'PARISDR'. Cannot promote the server to PRIMARY mode due to split brain check error. Use the 'force' option to force the promotion. Msg 19842, Level 16, State 1: Server 'MYSERVER', Procedure 'sp_hadr_admin', Line 531: 'primary' encountered an error and could not succeed. • Example 10 – forcefully promotes the server LONDON to primary: sp_hadr_admin primary, 'force' Command 'primary' successful. • Example 11 – activates the LONDON primary server: sp_hadr_admin activate (return status = 0) Command 'event' successful. (1 row affected) (0 rows affected) (return status = 0) Command 'activate' successful. (1 row affected) • Example 12 – deactivates the current server LONDON with a timeout period of 30 seconds: sp_hadr_admin 'deactivate','30','scheduled_offline' User connections statistics:: 0 in xact, 0 in chained mode, 9 in unchained mode, 0 holding server side cursors. Server reached INACTIVE state. Initiating log drain mechanism. Command 'deactivate' successful. (return status = 0) • Example 13 – cancels an ongoing deactivation process: sp_hadr_admin cancel Command ‘cancel’ successful. (return status = 0) • Example 14 – changes the mode of the inactive primary server LONDON to standby: HADR Users Guide 65 CHAPTER 8: sp_hadr_admin Syntax sp_hadr_admin standby Command 'standby' successful. (return status = 0) • Example 15 – initiates the log drain using the string scheduled_offline_07092013 as the label: sp_hadr_admin drain, 'scheduled_offline_07092013' Initiating log drain mechanism. Command 'drain' successful. (return status = 0) • Example 16 – displays the HADR map for the current system: sp_hadr_admin map Group Name Generation Number ---------- ----------------IT_GROUP 0 1 row affected) Data Source Name ---------------PARISDR Data Source Address ------------------tcp daisy 4901 Data Source Name ---------------LONDONDR Data Source Address ------------------tcp lily 4901 Command 'map' successful. (return status = 0) • Example 17 – drops the IT_GROUP group: sp_hadr_admin dropgroup, ‘IT_GROUP’ Command ‘dropgroup’' successful. (return status = 0) • Example 18 – estimates the amount of time required to roll back transactions: sp_hadr_admin fail over_timeestimate total potential rollback time (mins) -----------------------------------0 (1 row affected) dbid rep_drain_time ------ -------------(0 rows affected) total potential rep drain time 66 Adaptive Server Enterprise CHAPTER 8: sp_hadr_admin Syntax -----------------------------NULL (1 row affected) Command 'fail over_timeestimate' successful. (return status = 0) Usage • login_name must exist on all HADR servers and be granted the sa_role. Use sp_addexternlogin to add login_name as an external login for each remote HADR server. • • • • • • • • • • • • If you do specify the pname, sp_hadr_admin uses the HADR_server_name. Use this format to specify the host name or IP address and port for the HADR_server_name server: • hostname:port • ipaddress:port The server on which you issue the propagate command must be in primary mode (that is, @@hadr_mode must return a value of 1). force does not promote the standby server to a primary server if the HADR system detects an existing primary server. The administrator must first demote the existing primary server before reissuing the force parameter. You can run sp_hadr_admin primary only on the standby server. You can issue sp_hadr_admin active only on the inactive primary server The deactivate parameter triggers a transition from the active to the deactivating state, and then to the inactive state. sp_hadr_admin deactivate ignores the label parameter when you include the nodrain parameter. If the deactivation cannot complete in the period of time indicated by timeout_period, the server returns to active mode. Monitor the progress of replication by searching for the label 'scheduled offline' label in the Replication Server log. You can execute sp_hadr_admin cancel only on primary servers in a deactivating state (that is, while sp_hadr_admin deactivate is executing). To initiate a log drain, the server must be in an inactive state. You must issue sp_hadr_admin drain after issuing sp_hadr_admin deactivate nodrain. To check the label on the standby server and monitor the replication status, use the sp_hadr_admin status parameter. To avoid confusion, use a different label for each deactivation process. . sp_hadr_admin map retrieves HADR member information about client connections with hadr_list_cap capabilities: • Group name – group name that is associated with the HADR system. • Generation number – the number for the most recent version of the member address list. • Data source name – name of the HADR node. • Data source address – address for data source name. HADR Users Guide 67 CHAPTER 8: sp_hadr_admin Syntax • • • Run sp_hadr_admin map from the primary server. Run sp_hadr_admin failover_timeestimate from the primary server. Issuing sp_hadr_admin status [, 'label'] on the primary displays the database name and the log drain status for each HADR database. The label parameter is ignored on the primary server. The log drain status is one of: • completed – Replication Agent has processed the deactivation checkpoint marker. • in progress – a drain operation has started and Replication Agent has not yet processed the checkpoint marker. • inactive – there is no drain operation in progress. For example: sp_hadr_admin status Database Name Log Drain Status -------------- ---------------db1 completed db2 completed master completed Log Pages Left ---------------------------0 0 0 Issuing sp_hadr_admin status [, 'label'] on the standby server shows the: • Status of the replication process corresponding to the specified label, allowing a privileged user to determine if replication is complete. • Database name and status of replication for each HADR database. Status is listed as completed or pending. completed means the replicated data is applied to the standby database up to the deactivation checkpoint marker and its associated label. pending means this process is in progress. This example displays the status of a standby server using the label 'deactivate 2012/10/10': sp_hadr_admin status, 'deactivate 2012/10/10' Database Name Replication Status --------------------------------------------------------------------db1 pending db2 pending master completed See also • Chapter 6, Upgrading Adaptive Server in an HADR System on page 39 • Initializing the HADR Nodes on page 19 • Performing a Planned Failover on page 34 68 Adaptive Server Enterprise Index Index unplanned 33 @@hadr_mode global variable 7 @@servername for HADR server names 3 A active mode 5 Adaptive Server adding users 18 validating 17 allow hadr login privilege 10 applications during failover 33 in an HADR system 1 C capabilities hadr_list_cap 3 checkpoint marker, transaction log 33 clients during failover 33 Component Integration Services 16 configuration parameters HADR mode 7, 8 connections privileged 10 D databases and Replication Agent 15 participating in the HADR system 15 dbcc settrunc 35 deactivated mode 5, 13 deactivating mode 5 drain markers, recovering from unplanned failover 35 drain progress, verifying 35 E external mode 5 F failover planned 33 HADR Users Guide G generation number 3 global variables @@hadr_mode 7, 8 H HADR login account 16 HADR mode configuration parameter 7, 8 HADR server names recommendations 3 HADR system adding members 22 adding the HADR user login 19 adding users 18 applications 1 configuring nodes 20 deactivating 13 description 1 dropping members 22 failover 33 identifying members 3 initializing nodes 19 installing Replication Server 21 member address list 3 member modes 5 member modes, determining 7 naming nodes 19 participating databases 15 planned failover 34 privileges and roles 10 quiescing 13 recovering from unplanned failover 35 rolling upgrade 39 server name recommendations 3 split brain, preventing 4 upgrading 39 upgrading the standby server 40 validating Adaptive Server 17 HADR user login adding 19 hadr_list_cap capability 3 69 Index hadr_mode global variable 7, 8 HADR_server_name, recommendations 3 configuring as HADR member 20 initializing 19 naming 19 I internal mode 5 M maintenance user and privileges 10 password encryption 18 password expiration 18 privileges 11 manage hadr privilege 10 master database materializing for replication 21 materializing the master database 21 member address list adding members 22 determining 3 dropping members 22 during failover 34 generation number 3 propagating 3 member modes active 5 changing 8 changing with sp_hadr_admin 5 deactivated 5 deactivating 5 determining 7 external 5 forcing change 8 internal 5 primary 5 standby 5 members identifying 3 modes changing 8 changing with sp_hadr_admin 5 defined 5 multiple primaries, preventing 4 N naming servers 3 nodes adding the HADR user login 19 70 P password encryption maintenance user 18 password expiration maintenance user 18 planned failover 33 performing 34 queue process time 34 primary mode 5 primary server during failover 34 preventing multiple 4 privileged connections 10 privileges allow hadr login 10 granting 10 maintainance user 10 maintenance user 11 manage hadr 10 propagating the member list 3 Q queue process time 34 R remote connections, configuring 16 replication disabling 35 queues, draining 35 reversing direction 34 Replication Agent and transaction logs 33 as part of HADR system 1 databases participating in the HADR system 15 draining logs 13 during failover 33, 34 Replication Server 33 as part of HADR system 1 during failover 33 installing 21 materializing the master database 21 Adaptive Server Enterprise Index monitoring 4 restarting after unplanned failover 35 reversing replication direction 34 role during standby upgrade 40 role during unplanned failover 35 starting replication 21 rolling upgrade for HADR system 39 S server_net_name, recommendations 3 servers naming 3 soft quiesce 13 sp_hadr_admin 35 changing mode 5 changing server modes 8 determining modes 7 drain parameter 33 draining the transaction log 33 inactive state 13 member address, determining 3 nodrain parameter 33 performing failover 34 sp_repdbsync, unplanned failover 35 split brain, preventing 4 standby mode 5 standby server during failover 34 HADR Users Guide promoting to primary mode 35 upgrading 40 states inactive 13 sysattributes system table 3 sysservers system table 8 system tables sysattributes 3 sysservers 3, 8 T transaction log checkpoint marker 33 draining 33 truncation point, disabling 35 U unplanned failover 33 recovering from 35 transaction log 35 upgrading HADR system 39 rolling 39 standby server 40 users, adding to Adaptive Server 18 71 Index 72 Adaptive Server Enterprise