Download Oracle provided Application HA features

Application High Availability
with Oracle
Aychin Gasimov
02/2014
Application High Availability
• Application must be able to provide
uninterrupted service to its end users.
• Application must be able to handle below
listed cases:
– Member instance of the service failure
– All instances of the service failure
– Node/Site failure
– Planned downtimes
Required components
• Oracle Clusterware, Oracle Restart, Oracle Data
Guard
• FAN
• ONS
• Services
• UCP
• LBA and different types of load balancing
• FCF
• TAF
FAN Fast Application Notification
• FAN is a notification mechanism that Oracle
Clusterware uses to notify other processes
• FAN publishes service/instance/node state
change events, like UP and DOWN
• FAN also publishes load balancing advisory events
• FAN events are published using Oracle
Notification Service and Oracle Streams
Advanced Queuing.
• Oracle Net Services listeners are integrated with
FAN events
FAN Fast Application Notification
FAN publishes service/instance/node state change
events, like UP and DOWN
FAN notifies about configuration and service level information that
includes service status changes, such as UP or DOWN events. Applications can
respond to FAN events and take immediate action. FAN UP and DOWN events can
apply to instances, services, and nodes.
For cluster configuration changes, the Oracle RAC high availability
framework publishes a FAN event immediately when a state change occurs in the
cluster. Instead of waiting for the application to poll the database and detect a
problem, applications can receive FAN events and react immediately. With FAN,
in-flight transactions can be immediately terminated and the client notified when
the instance fails.
FAN Fast Application Notification
FAN publishes load balancing advisory events
FAN also publishes load balancing advisory events. Applications can take
advantage of the load balancing advisory FAN events to direct work requests to
the instance in the cluster that is currently providing the best service quality.
Listeners are integrated with FAN events
Oracle Net Services listeners are integrated with FAN events,
enabling the listener and CMAN to immediately de-register services provided
by the failed instance and to avoid erroneously sending connection requests
to failed instances.
SUBSCRIBE_FOR_NODE_DOWN_EVENT_listener_name=ON (default)
ONS Oracle Notification Service
• A publish and subscribe service for
communicating information about all FAN events.
• Oracle Notification Service is included as part of
the Oracle Clusterware and Client software
(ons.jar).
• Maintained as the Clusterware resource
• One ONS process per node
• Can communicate ONS processes on other nodes
and on client side
Services
• A named representation of one or more database instances. The service
name for an Oracle database is normally its global database name. Clients
use the service name to connect to one or more database instances.
• Logical abstractions for managing workloads in Oracle Database
• The services are tightly integrated with Oracle Database and are
maintained in the data dictionary.
• Connection requests can include a database service name.
• Services enable you to configure a workload, administer it, enable and
disable it, and measure the workload as a single entity.
• AWR records service performance. Each service has quality-of-service
thresholds for response time and CPU consumption.
• Database Resource Manager can map services to consumer groups.
Therefore, you can automatically manage the priority of one service
relative to others.
• Services can be created by DBMS_SERVICE package or srvctl utility
Services
Oracle Cluster CLS1
Applications
Node 1
Instance1
30% RTPC 0.5s CPUPC 0.3s
70% RTPC 0.7s CPUPC 0.5s
Srv1_db
Srv2_db
Node 2
Instance2
100% RTPC 0.3s CPUPC 0.2s
Srv3_db
Node 3
Instance3
60% RTPC 0.8s CPUPC 0.6s
40% RTPC 0.5s CPUPC 0.3s
• Using Resource Manager to distribute resources between services
• Setting thresholds on Response Time per sec and CPU per sec for the services
UCP Universal Connection Pool
• UCP for JDBC provides a connection pool implementation
for caching JDBC connections. Java applications that are
database-intensive use the connection pool to improve
performance and better utilize system resources.
• A UCP JDBC connection pool can use any JDBC driver to
create physical connections that are then maintained by
the pool.
• The pool also leverages many high availability and
performance features available through an Oracle Real
Application Clusters (RAC) database. These features include
Fast Connection Failover (FCF), run-time connection load
balancing, and connection affinity.
• Documented in Oracle® Universal Connection Pool for
JDBC Developer's Guide
Requirements for UCP
• JRE 1.5 or higher
• A JDBC diver or a connection factory class
capable of returning a java.sql.Connection and
javax.sql.XAConnection object
– Oracle drivers from releases 10.1 or higher are
supported. Advanced Oracle Database features, such
as Oracle RAC and Fast Connection Failover, require
the Oracle Notification Service library (ons.jar) that is
included with the Oracle Client software.
• The ucp.jar library must be included in the
CLASSPATH of an application.
LBA Load Balancing Advisory
• The Load Balancing Advisory provides information to applications or
clients about the current service levels that the Oracle RAC database
instances are providing. (v$servicemetric.goodness)
• Load balancing advisory is integrated with the AWR. AWR measures
response time and CPU consumption for each service
• The advice given by the LBA takes into account the power of the server
and the current workload of the service
• Integrated with Oracle 11g JDBC, ODP.NET and OCI
• Applications can take advantage of the load balancing FAN events to direct
work requests to the instance in the cluster that provides the best
performance based on the workload management directives defined for
that service.
• Configured by defining service-level goals for the Service. It enables the
LBA for that service and enables the publication of FAN load balancing
events.
• Listener also can use the load balancing advisory when it balances the
connection loads if LBA enabled and clb_goal is set to SHORT for the
Service.
RLB Run-time Load Balancing
• RLB is a feature of Oracle connection pools that can distribute client work
requests across the instances in an Oracle RAC, based on the LBA
information. It allocates connections, based on the current performance
levels. This provides load balancing at the transaction level.
• There are two types of service-level goals for Run-time Connection Load
Balancing
– Service Time (SERVICE_TIME)—Attempts to direct work requests to instances
according to response time. Load balancing advisory data is based on elapsed
time for work done in the service plus available bandwidth to the service. An
example for the use of SERVICE_TIME is for workloads such as internet
shopping where the rate of demand changes. (v$servicemetric.dbtimepercall)
srvctl modify service -d DB -s app_srvc -B SERVICE_TIME -j SHORT
– Throughput (THROUGHPUT)—Attempts to direct work requests according to
throughput. The load balancing advisory is based on the rate that work is
completed in the service plus available bandwidth to the service. An example
for the use of THROUGHPUT is for workloads such as batch processes, where
the next job starts when the last job completes. (v$servicemetric.callspersec)
srvctl modify service -d DB -s batch_srvc -B THROUGHPUT -j LONG
CLB Connection Load Balancing
• Provides load balancing at the time of the initial database
connection
• Listener directs a connection request to the best instance currently
providing the service
• For each service, you can define the method the listener uses for
load balancing by setting the connection load balancing goal.
– SHORT --Connection load balancing uses Load Balancing Advisory,
when Load Balancing Advisory is enabled (either goal_service_time or
goal_throughput). When GOAL=NONE (LBA disabled), connection load
balancing uses an abridged advice based on CPU utilization.
– LONG --Balances the number of connections per instance using
session count per service. This setting is recommended for
applications with long connections such as forms.
• Controlled by clb_goal property of the Service
Client-Side Load Balancing
• Client-side load balancing balances the connection requests
across the listeners.
• Client-side load balancing is defined in client connection
definition by setting the parameter LOAD_BALANCE=ON
• Oracle client randomly selects an address from the address
list, and connects to that node's listener
• Client-side load balancing includes connection failover.
• LOAD_BALANCE is ON by default for DESCRIPTION_LIST
only. This parameter by default is OFF for an address list
within a DESCRIPTION. Setting this ON for a SCAN-based
address implies that new connections will be randomly
assigned to one of the 3 SCAN-based IP addresses resolved
by DNS.
Client-Side Failover and Load Balancing
100.125.200.21
3
•
•
•
•
•
•
•
•
3
100.125.200.22
100.125.200.23
DB =
(DESCRIPTION =
(FAILOVER = on)
(LOAD_BALANCE = off)
(CONNECT_TIMEOUT = 5)
(TRANSPORT_CONNECT_TIMEOUT = 2)
(RETRY_COUNT = 2)
(ADDRESS = (PROTOCOL = TCP)(HOST = scan1)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = scan2)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = scan3)(PORT = 1521))
(CONNECT_DATA =
(SERVICE_NAME = myservice)
)
)
CLB occurs on client side
FAILOVER option is set to ON, it is default value.
Connection will be tried to first SCAN address
Then within this 3 SCAN IPs connection will be tried to first IP, if it will fail then second IP will be tried, each try
will have 2 sec TCP timeout and 5 sec overall timeout to connect and this 3 IPs will be traversed 3 times, 1 time
+ 2 RETRY_COUNT. It means that if all 3 SCAN IPs will fail it will take up to 2 * 3 * 3 = 18 sec to try next SCAN
address. If TCP connection will success in 2 sec then we will have additional time (CONNECT_TIMEOUT TRANSPORT_CONNECT_TIMOUT) to establish connection to the instance.
If next SCAN address will success then connection will be established
If all subsequent address will fail then all addresses will be tries 2 more times. Overall number of tries will be 3.
Addresses will be tried one by one in sequential order (LOAD_BALANCE=off), in this particular case the load
balancing between 3 SCAN IPs also will not be performed, it will try to connect to the first IP returned from
DNS
To enable CLB set LOAD_BALANCE=ON, then address will be randomly chosen from 3 addresses and also it will
randomly choose between 3 SCAN IPs.
How it works together
pds.setURL(„jdbc:oracle:thin:@(DESCRIPTION=
(LOAD_BALANCE=ON)
(ADDRESS = …(host=db-scan)…)
...
(CONNECT_DATA=(SERVICE_NAME=service1))“);
Application
SCAN Listener
service1
UCP
service2
ONS
Instance 1
ONS
Instance 2
SCAN Listener
service1
service2
ONS
Instance 3
FAN LBA event
•
•
•
UCP will create physical connections to the instances using provided connection description. Client side load balancing will distribute new
connection requests between different SCAN listeners (3 IPs) because LOAD_BALANCE=ON
Connection request arrives to the Listener, now according to the Services clb_goal value it will redirect it to the appropriate instance, it is serverside connection load balancing. If clb_goal is SHORT and LBA is enabled for the Service then listener will use the services GOODNESS information
which it receives from serving instances to decide to which instance to redirect the connection. If clb_goal is LONG then Listener will balance
connections by number of sessions per service. If connection pools physical connections count is constant then we can use clb_goal=LONG with
UCP, if this number is dynamic then clb_goal=SHORT must be used, because each new connection request from UCP must be accurately redirected
according to the LBA advice and goal (goal can be SERVICE_TIME or THROUGHPUT)
ONS from each node periodically sends LBA FAN events to UCP. This way UCP is aware about current service levels on each instance, like Listener.
According to this information Run-time load balancing mechanism distributes workload between different instances during application life.
How it works together
• Run-Time Load Balancing and Connection load balancing are related if clb_goal
of the Service is set to SHORT in:
 They both use Load Balancing Advisor.
 They both use same balancing goal defined in the Service definition by –B
key, i.e. SERVICE_TIME or THROUGHPUT.
• Database using AWR data will calculate the GOODNESS for each service based on
the runtime load balancing goal or clb_goal for that service. Current GOODNESS
number can be found in the V$SERVICEMETRIC.GOODNESS field.
• If clb_goal is:
 LONG, LBA will not be used for server-side load balancing, GOODNESS field
will contain just the number of current sessions for this service in current
instance.
 SHORT, LBA will be used for server-side load balancing, GOODNESS will be
calculated based on the load balancing goal, SERVICE_TIME or
THROUGHPUT.
• If clb_goal is SHORT and LBA is not enabled –B NONE then listener will consider
the node load to equalize CPU usage when distributing connections.
FCF Fast Connection Failover
• FCF designed for fast instance and database failover and switchover with
Oracle RAC and Oracle Data Guard.
• FCF receives FAN availability events and immediately clears affected
connections from the pool.
• Requires the use of an Oracle JDBC driver for JAVA applications and an
Oracle RAC database or an Oracle Restart.
• Can be used with Session and Connection pools of OCI applications
• It was introduced as part of pooling feature “Implicit Connection Cache”
that available from JDBC 10g
• Starting from 11gR2 Implicit Connection Caching is deprecated in favor of
UCP
• Now UCP must be used to benefit from FCF and RLB.
• FCF supports planned (instance relocation or shutdown in RAC database)
and unplanned outages
• Application logic must be used to make outages transparent for the end
users.
FCF
• Planned outage
– Stale borrowed connections are marked and removed after they
are returned to the pool
– On-going transactions proceed to complete
• Unplanned outage
– Detect and remove stale connections from pool
– Borrowed connections are immediately aborted and closed
– On-going transactions immediately receive an exception
• FCF supports RAC database, Data Guard and Single Instance
with Oracle Restart, they all can publish FAN messages
• Set oracle.net.ns.SQLnetDef.TCP_CONNTIMEOUT_STR
property in milliseconds.
FCF planned outage
Node 1
ONS
Application
UCP
Interconnect
Borrowed
connections
Instance 1
Service
Node 2
ONS
Service
Instance 2
• Application uses UCP, there is 9 physical connections in the pool
• Connections are distributed between 2 RAC instances
Now execute:
srvctl stop service –d DB –s Service –I Instance1
FCF planned outage
Node 1
ONS
Application
UCP
Interconnect
Borrowed
connections
Instance 1
Service
ONS
Service
Node 2
Instance 2
• Service on Instance 1 went down, evmd publishes service DOWN event
FCF planned outage
FAN servc DOWN
Application
Instance 1
Service
UCP
Interconnect
Borrowed
connections
Node 1
ONS
ONS
Service
Node 2
Instance 2
• ONS publishes FAN availability event about service DOWN on Instance 1
FCF planned outage
Node 1
ONS
Application
UCP
Interconnect
Borrowed
connections
ONS
Service
•
•
Instance 1
Service
Node 2
Instance 2
UCP received FAN event and immediately marks borrowed connections to the Instance 1 as to be
cleared, not borrowed connects are cleared and if needed reestablished to the available instance
Physical connections is still there, because there is borrowed connections in use. It is possible
because when we do normal service shutdown already active connections are not disconnected
and it is up to client (UCP) when to disconnect.
FCF planned outage
Node 1
ONS
Application
UCP
Interconnect
Borrowed
connections
Instance 1
Service
Node 2
ONS
Service
Instance 2
• As soon as application closes borrowed connection UCP will clear it
FCF planned outage
Node 1
ONS
Application
UCP
Interconnect
Borrowed
connections
Instance 1
Service
ONS
Service
Node 2
Instance 2
• If the pool min size will be reached new connection will be reestablished immediately
to the available Instance
• After Service on Node 1 will be started new connections will be placed to it by SLB
FCF unplanned outage
Node 1
ONS
Application
UCP
Interconnect
Borrowed
connections
Instance 1
Service
ONS
Service
• Application uses UCP, there is 9 physical connections in the pool
• Connections are distributed between 2 RAC instances
Node 2
Instance 2
FCF unplanned outage
Node 1
ONS
Application
UCP
Interconnect
Borrowed
connections
ONS
Service
• Node 1 fails, evmd publishes DOWN event
Instance 1
Service
Node 2
Instance 2
FCF unplanned outage
Node 1
ONS
Application
UCP
Interconnect
Borrowed
connections
Instance 1
Service
ONS
Service
Node 2
Instance 2
• Connections to the Instance 1 will fall into TCP retransmission cycle and will be in this
state until TCP timeout will expire which can take several minutes, but …
FCF unplanned outage
Node 1
ONS
Application
UCP
Interconnect
Borrowed
connections
Instance 1
Service
ONS
Service
FAN DOWN event
• ONS will distribute DOWN event immediately
Node 2
Instance 2
FCF unplanned outage
Node 1
ONS
Application
UCP
Interconnect
Borrowed
connections
ONS
Service
•
•
•
Instance 1
Service
Node 2
Instance 2
UCP will receive DOWN event and will immediately break affected connections out of TCP timeouts by
disconnecting physical connections
Application will immediately receive error, all not committed work is already rolled back by Instance 2.
Application do not need to execute rollback.
Application must:
 Retry the connection request, because the old one is no longer open
 Replay the transaction
Usage model of UCP/FCF
1. Get connection from the pool
2. Perform activity on it
3. Get exception from failure of some
component
4. Check with isValid() function if connection
still valid
5. If not, reconnect and recover lost actions
For information about how to configure UCP in your java app refer to:
Oracle® Universal Connection Pool for JDBC Developer's Guide 11g Release 2 (11.2)
FCF with Data Guard failover
1. Primary site lost! Connections fall into hang-state.
2. After failover complete. Respective database services
will start and DG Broker publish FAN availability
event
3. FCF will break connections out from TCP time out,
clear stale connections and throw error to the
application
4. Application will retry connection and replay lost
transactions if any
FCF not needed for DG switchover
•
•
For DG switchover FCF is not needed because its
primary role is to break connections from TCP
timeouts. Which is not a case when planned
switchover occurs.
Switchover steps:
1.
2.
3.
4.
Primary converts to physical standby and disconnects all
sessions
Client sessions receive ORA-3113 and begin going through
their retry logic (TAF for OCI and Application logic for JDBC)
Standby converted to primary database
As new primary opened the respective services are started
and clients now see the services as available and connect.
Replay lost actions if any.
TAF Transparent Application Failover
•
•
•
•
•
•
Client side feature of the OCI driver
Transparently fails over read-only sessions
Can use FAN events distributed by Streams AQ
Do not restore sessions state (ALTER SESSION)
Do not support DML
Provides callback functions to manage failover
steps
• Can be configured on client as well as on the
server side using database Services
TAF Transparent Application Failover
• To use FAN with OCI next conditions must met:
 Initialize the OCI Environment in OCI_EVENTS mode
 Connect to the Service that have AQ HA notifications
 Link with a thread library
• TAF have 2 failover types
 SESSION, when new sessions will be reestablished by
TAF but no select operation recovery
 SELECT, new sessions will be reestablished and
enables users with open cursors to continue fetching
after failover. Involves overhead on the client side in
normal select operations
TAF Transparent Application Failover
•
Sessions with active update transactions (UPDATE,
INSERT, DELETE) at the time of the failure:
 Will be reconnected to a new session
 Uncommitted transactions will be rolled back
 Error message will be returned to the application, stating
that a rollback must be issued
 Application must rollback and reissue the transaction
•
•
TAF also provides the ability, with the RETRIES and
DELAY parameters, to automatically retry reconnecting on failover
Example of TAF configured service creation:
srvctl add service -d DB -s taf_service -q TRUE -e SESSION -m BASIC -w 10 -z 50
•
•
•
•
•
“-q TRUE” enables AQ HA notifications
“-e SESSION” sets failover type to SESSION
“-m” set failover method to BASIC
“-w” set failover delay to 10 sec
“-z” set failover retries to 50
Oracle 12c Application Continuity
•
•
•
•
•
•
•
Restores full session including all states, cursors,
variables and last transaction if there was any.
Supports planned and unplanned outages
Performed automatically, minimal application change
Supported for Oracle RAC, Data Guard, Active Data
Guard and WebLogic Server in conjunction with the
JDBC Thin Driver or the UCP.
It applies only to JDBC Thin connections (JDBC OCI is
not supported).
Requires JDBC Replay driver
Service properties: FAILOVER_TYPE=TRANSACTION,
COMMIT_OUTCOME=TRUE,NOTIFICATION=TRUE
Oracle 12c Application Continuity
Node 1
ONS
Application
Continuity
Directory
Interconnect
LTXID
Replay
Context
UCP
JDBC Replay Driver
Borrowed
connections
Instance 1
Service
ONS
Service
LTXID
Node 2
Instance 2
Continuity
Directory
LTXID