Download 2006-08-15_Cecchet_ApacheConAsia2006

Building Highly Available
Web Applications
Emmanuel Cecchet
Chief architect - Continuent
http://www.continuent.org
High availability
• The magic nines
Percent uptime
1
Downtime/month
Downtime/year
99.0%
7.2 hours
3.65 days
99.9%
43.2 minutes
8.76 hours
99.99%
4.32 minutes
52.56 minutes
99.999%
0.43 minutes
5.26 minutes
99.9999%
2.6 seconds
31 seconds
Few definitions
• MTBF
• Mean Time Between Failure
• Total MTBF of a cluster must combine MTBF of its
individual components
• Consider mean-time-between-system-abort (MTBSA)
or mean-time-between-critical-failure (MTBCF)
• MTTR
•
•
•
•
•
2
Mean Time To Repair
How is the failure detected?
How is it notified?
Where are the spare parts for hardware?
What does your support contract say?
Problem: Database is the weakest link
• Clients connect to the application server
• Application server builds web pages with data coming
from the database
• Application server clustering solves application server
failure
• Database outage causes overall system outage
Application
servers
Internet
3
Database
Database
Disk
Disk replication/clustering
• Eliminates the single point of failure (SPOF)
on the disk
• Disk failure does not cause database outage
• Database outage problem still not solved
Application
servers
Database
Internet
Database disks
4
Database clustering with shared disk
•
•
•
•
Multiple database instances share the same disk
Disk can be replicated to prevent SPOF on disk
No dynamic load balancing
Database failure not transparent to users (partial
outage)
• Manual failover + manual cleanup needed
Application
servers
Internet
5
Databases
Database
Disks
Master/slave replication
•
•
•
•
•
Lazy replication at the disk or database level
No scalability
Data lost at failure time
System outage during failover to slave
Failover can take several hours
Application
servers
Internet
Master
Database
Database
Disks
hot standby
log shipping
Slave Database
6
Scaling the database tier
Master-slave replication
• Cons
• failover time/data loss on master failure
• read inconsistencies App.
Master
server
• scalability
Web
frontend
Internet
7
• Cons
Scaling the database tier
Atomic broadcast
• atomic broadcast scalability
• no client side load balancing
• heavy modifications of the database engine
Internet
Atomic
broadcast
8
Scaling the database tier – SMP
• Cons
• Scalability limit
• Cost
Web
frontend
App.
server
Database
Internet
Well-known
database
vendor here
Well-known hardware +
database vendors here
9
Sequoia motivations
• Database tier should be
•
•
•
•
•
scalable
highly available
without modifying the client application
database vendor independent
on commodity hardware
Internet
10
Transparent failover
• Failures can happen
• in any component
• at any time of a request execution
• in any context (transactional, autocommit)
• Transparent failover
• masks all failures at any time to the client
• perform automatic retry and preserves consistency
Internet
Sequoia
11
Sequoia competitive advantages
•
•
•
•
•
•
•
•
12
High availability with fully transparent failover
Online upgrade and maintenance operations
Scalability with dynamic load balancing
Commodity hardware
No modification to existing client applications
Database vendor independent
Heterogeneity support
Platform independent
Open source database clustering requirements
Feature
Commodity
hardware
MySQL
replication
Yes
MySQL
cluster
Yes
Application
Yes if reading Yes (NDB)
modifications from slaves
13
Slony-I
Yes
Sequoia
Yes
Yes if
No but driver
reading
update
from slaves needed
Database
No (built-in)
modifications
No (built-in) Recompile
No
DB support
MySQL 3.23,
4.x or 5.x
MySQL 5.x
Derby, MySQL,
PostgreSQL
any version
License
GPL/Comm.
GPL/Comm. BSD
PostgreSQL
7.x, 8.x
APL
Open source database clustering features
Feature
14
MySQL
replication
MySQL
cluster
Slony-I
Sequoia
Data loss on
failure
Yes
No
Yes
No
Transparent
failover
No
No
No
Yes
Disaster
recovery
Yes
Yes
Yes
Yes
Queries load
balancing
No
Yes, static
No
Yes, dynamic
Add node on
the fly
Yes (slave)
No
Yes (slave)
Yes
Online
upgrades
No
No
No
Yes
Outline
• RAIDb
• Sequoia
• Building an HA solution
• Scalability
• High availability
15
RAIDb concept
• Redundant Array of Inexpensive Databases
• RAIDb controller
• gives the view of a single database to the
client
• balance the load on the database backends
• RAIDb levels offers various tradeoff of
performance and fault tolerance
16
RAIDb levels
• RAIDb-0
• partitioning
• no duplication and no fault tolerance
• at least 2 nodes
SQL requests
RAIDb controller
table 1
17
table 2 & 3
table ...
table n-1
table n
RAIDb levels
• RAIDb-1
• mirroring
• performance bounded by write broadcast
• at least 2 nodes
SQL requests
RAIDb controller
Full DB
18
Full DB
Full DB
Full DB
Full DB
RAIDb levels
• RAIDb-2
• partial replication
• at least 2 copies of each table for fault tolerance
• at least 3 nodes
SQL requests
RAIDb controller
Full DB
19
table x
table y
table x & y
table z
Outline
• RAIDb
• Sequoia
• Building an HA solution
• Scalability
• High availability
21
Sequoia overview
• Middleware implementing RAIDb
• 100% Java implementation
• open source (Apache v2 License)
• Two components
• Sequoia driver (JDBC, ODBC, native lib)
• Sequoia Controller
• HA and load balancing features
• Database neutral (generic ANSI SQL)
22
Sequoia architectural overview
Sequoia
controller
Derby
Sequoia
JDBC
JDBCDriver
driver
JVM
23
Derby
JVM
Sequoia drivers (1/2)
• Visit http://carob.continuent.org
Java client
PHP
client
.Net
client
ODBC
client
ODBSequoia
Sequoia JDBC
driver
libmysql
libmy
sequoia
Custom
C/C++
client
Carob C++ API
Sequoia protocol
Sequoia controller
Derby
JDBC driver
Derby JDBC
driver
Derby protocol
24
Sequoia controller
MySQL
JDBC driver
MySQL
JDBC driver
MySQL protocol
Sequoia drivers (2/2)
• Provide load balancing between controllers
for connection establishment
• Controller failure detection
• Transparent failover for any request in any
context (including metadata) on controller
failure
• Transparent SQL proxying
• Optional SSL if needed
25
Inside the Sequoia Controller
Client application
(Servlet, EJB, ...)
Client application
(Servlet, EJB, ...)
Sequoia driver
Sequoia driver
Client application
(Servlet, EJB, ...)
Sequoia driver
Sequoia Controller
HTTP
RMI
...
JMX administration console
Virtual database 1
Virtual database 2
Authentication Manager
Authentication Manager
Request Manager
Request Manager
Configuration
Monitoring
JMX Server
Scheduler
Scheduler
Recovery
Query result cache
Log
Load balancer
Database dumps
management
Checkpointing
service
26
Recovery
Query result cache
Log
Load balancer
Database
Backend
Database
Backend
Database
Backend
Database
Backend
Database
Backend
Connection
Manager
Connection
Manager
Connection
Manager
Connection
Manager
Connection
Manager
DB2 JDBC
driver
DB2 JDBC
driver
DB2 JDBC
driver
MS SQL
JDBC driver
Oracle
JDBC driver
DB2
DB2
DB2
MS SQL
Oracle
Virtual Database
Java client
program
(Servlet, EJB, ...)
Sequoia
driver
Virtual database
Authentication Manager
Request Manager
Scheduler
Recovery
Log
Request Cache
Load balancer
27
Database
Backend
Database
Backend
Database
Backend
Connection
Manager
Connection
Manager
Connection
Manager
DB2 JDBC
driver
DB2 JDBC
driver
DB2 JDBC
driver
DB2
DB2
DB2
• Gives the view of a single
database
• Establishes the mapping between
the database name used by the
application and the backend
specific settings
• Backends can be added and
removed dynamically
• Backends can be transferred
between controllers
• configured using an XML
configuration file
Authentication Manager
Java client
program
(Servlet, EJB, ...)
Sequoia
driver
Virtual database
Authentication Manager
Request Manager
Scheduler
Recovery
Log
Request Cache
Load balancer
28
Database
Backend
Database
Backend
Database
Backend
Connection
Manager
Connection
Manager
Connection
Manager
DB2 JDBC
driver
DB2 JDBC
driver
DB2 JDBC
driver
DB2
DB2
DB2
• Matches real login/password
used by the application with
backend specific login/
password
• Administrator login to manage
the virtual database
• Transparent login proxying
available
Scheduler
Java client
program
(Servlet, EJB, ...)
Sequoia
driver
Virtual database
Authentication Manager
Request Manager
Scheduler
Recovery
Log
Request Cache
Load balancer
29
Database
Backend
Database
Backend
Database
Backend
Connection
Manager
Connection
Manager
Connection
Manager
DB2 JDBC
driver
DB2 JDBC
driver
DB2 JDBC
driver
DB2
DB2
DB2
• Manages concurrency control
• Provides support for clusterwide checkpoints
• Assigns unique ids to requests
and transactions
• Ensure serializable
transactional order
• Relies on database (row-level)
locking
Request cache
Java client
program
(Servlet, EJB, ...)
• 3 optional caches
Sequoia
driver
• tunable sizes
• parsing cache
Virtual database
Authentication Manager
Request Manager
Scheduler
Recovery
Log
Request Cache
Load balancer
30
Database
Backend
Database
Backend
Database
Backend
Connection
Manager
Connection
Manager
Connection
Manager
DB2 JDBC
driver
DB2 JDBC
driver
DB2 JDBC
driver
DB2
DB2
DB2
• parse request skeleton only once
• INSERT INTO t VALUES (?,?,?)
• metadata cache
• column metadata
• fields of a request
• result cache
• caches results from SQL requests
• tunable consistency
• optimizations for findByPk requests
Load balancer 1/2
• RAIDb-0
Java client
program
(Servlet, EJB, ...)
• query directed to the backend having the
needed tables
Sequoia
driver
• RAIDb-1
Virtual database
Authentication Manager
Request Manager
Scheduler
Recovery
Log
Request Cache
Load balancer
Database
Backend
Database
Backend
Database
Backend
Connection
Manager
Connection
Manager
Connection
Manager
DB2 JDBC
driver
DB2 JDBC
driver
DB2 JDBC
driver
• read executed by current thread
• write multicast through group
communication and executed in parallel
• asynchronous execution possible
• if one node fails but others succeed, failing
node is disabled
• RAIDb-2
• same as RAIDb-1 except that writes are sent
only to nodes owning the updated table
DB2
32
DB2
DB2
Load balancer 2/2
Java client
program
(Servlet, EJB, ...)
• Static load balancing policies
Sequoia
driver
• Round-Robin (RR)
• Weighted Round-Robin (WRR)
Virtual database
Authentication Manager
Request Manager
Scheduler
Recovery
Log
Request Cache
Load balancer
33
Database
Backend
Database
Backend
Database
Backend
Connection
Manager
Connection
Manager
Connection
Manager
DB2 JDBC
driver
DB2 JDBC
driver
DB2 JDBC
driver
DB2
DB2
DB2
• Least Pending Requests First
(LPRF)
• request sent to the node that has
the shortest pending request queue
• efficient even if backends do not
have homogeneous performance
Connection Manager
Java client
program
(Servlet, EJB, ...)
• Sequoia JDBC driver provides
transparent connection pooling
• Connection pooling for a backend
Sequoia
driver
Virtual database
Authentication Manager
Request Manager
Scheduler
Recovery
Log
Request Cache
Load balancer
Database
Backend
Database
Backend
Database
Backend
Connection
Manager
Connection
Manager
Connection
Manager
DB2 JDBC
driver
DB2 JDBC
driver
DB2 JDBC
driver
•
•
•
•
no pooling
blocking pool
non-blocking pool
dynamic pool
• Connection pools defined on a
per login basis
• resource management per login
• dedicated connections for admin
DB2
34
DB2
DB2
Recovery Log
Java client
program
(Servlet, EJB, ...)
Sequoia
driver
Virtual database
Authentication Manager
Request Manager
Scheduler
Recovery
Log
Request Cache
Load balancer
35
Database
Backend
Database
Backend
Database
Backend
Connection
Manager
Connection
Manager
Connection
Manager
DB2 JDBC
driver
DB2 JDBC
driver
DB2 JDBC
driver
DB2
DB2
DB2
• Transactional log for
recovery/resync
• Checkpoints are associated
with database dumps
• Record all updates and
transaction markers since a
checkpoint
• Store log information in a
database
Functional overview
•
•
•
•
•
•
36
Connection establishment
Read request
Write request
Failure handling
Recovery
Summary
Connection establishment
• Sequoia JDBC URL
• jdbc:sequoia://controller1[:port1],controller2[:port
2]/vdbname
• Driver connection to a controller according to a
policy defined in the URL (random, roundrobin, ordered, …)
• Controller connections to a physical database
use a connection pool defined per backend
• All queries on a driver connection go to the
same controller but controller load balance
queries on the underlying backends
37
Sequoia read request
Client application
(Servlet, EJB, ...)
Client application
(Servlet, EJB, ...)
Sequoia driver
Sequoia driver
connect
connect
login,
password
execute SELECT
*myDB
FROM t
Sequoia Controller
Virtual database 1
Authentication Manager
Request Manager
ordering
Scheduler
Recovery
Query result cache
Log
RR, WRR, LPRF, …
get connection
Database
Database
Database
Backend cache
Backend
Backend
update
from pool
Connection
Connection
Connection
Manager
Manager
Manager
(if available)
exec
Load balancer
38
Derby JDBC
driver
Derby JDBC
driver
Derby JDBC
driver
Derby
Derby
Derby
Write request
• Query broadcast using total order between controllers
• Backends execute query in parallel (possibly
asynchronously)
• Automatically disable backends that fail if others
succeed
• Update recovery log (transactional log) asynchronously
• Non-conflicting transactions execute in parallel
• table-based locking used as a hint for detecting conflicts
• conflicting queries execute in a single-threaded queue
• stored procedures execute one at a time
39
Controller replication
Client application
(Servlet, EJB, ...)
Client application
(Servlet, EJB, ...)
Client application
(Servlet, EJB, ...)
Sequoia driver
Sequoia driver
Sequoia driver
jdbc:sequoia://node1,node2/myDB
Sequoia Controller
Sequoia Controller
Total order reliable
Distributed Request Manager
multicast Virtual database 1
Distributed Request Manager
Virtual database 2
Authentication Manager
Authentication Manager
Request Manager
Request Manager
Scheduler
Recovery
Database
Embedded
Derby
40
Recovery
Query result cache
Log
Load balancer
Scheduler
Recovery
Database
Embedded
Derby
Recovery
Query result cache
Log
Load balancer
Database
Backend
Database
Backend
Database
Backend
Database
Backend
Database
Backend
Database
Backend
Database
Backend
Database
Backend
Connection
Manager
Connection
Manager
Connection
Manager
Connection
Manager
Connection
Manager
Connection
Manager
Connection
Manager
Connection
Manager
Derby JDBC
driver
Derby JDBC
driver
Derby JDBC
driver
Derby JDBC
driver
Derby JDBC
driver
Derby JDBC
driver
Derby JDBC
driver
Derby JDBC
driver
Derby
Derby
Derby
Derby
Derby
Derby
Derby
Derby
Sequoia failure handling (1/2)
• Failure of a connection to a controller
• transparently reconnected to another controller
(according to user defined policy)
• failure inside a transaction restores the transactional
context upon reconnection
• backend failure
• backend removed from load balancer (no noticeable
failover time)
• failure during read: retry on another backend
• failure during write: ignored if successful on other
backends
41
Sequoia failure handling (2/2)
• controller failure
• database backends attached to dead
controller are lost
• clients are transparently reconnected to
another controller (according to user defined
policy)
• failure during read: transparently retried
(even within a transaction)
• failure during write: retrieve from remaining
controller(s) failover cache
42
Node failure with Sequoia
execute INSERT INTO t …
• No 2 phasecommit
• parallel
transactions
• failed nodes are
automatically
disabled
43
Client application
(Servlet, EJB, ...)
Client application
(Servlet, EJB, ...)
Sequoia driver
Sequoia driver
Sequoia Controller
Virtual database 1
Authentication Manager
Request Manager
Scheduler
Recovery
Query result cache
Log
Load balancer
Database
Backend
Database
Backend
Database
Backend
Connection
Manager
Connection
Manager
Connection
Manager
Derby JDBC
driver
Derby JDBC
driver
Derby JDBC
driver
Derby
Derby
Derby
Outline
• RAIDb
• Sequoia
• Building an HA solution
• Scalability
• High availability
44
Highly available web site
• Apache clustering
• L4 switch, RR-DNS, One-IP techniques, LVS, Linux-HA, …
• Web tier clustering
• mod_jk (T4), mod_proxy/mod_rewrite (T5), session replication
• Database clustering
• Sequoia + any database
RR-DNS
mod-jk
Parsing cache
Result cache
Metadata cache
Internet
embedded
45
Highly available web applications
• Consider MTBF (Mean time between failure) of
every hardware and software component
• Take MTTR (Mean Time To Repair) into account to
prevent long outages
• Tune accordingly to prevent trashing
Internet
Sequoia
46
Group communication
• Group communication is the base for
replication
• Different tiers might have different needs
• Ordering (FIFO, Total, causal, …)
• Delivery (Best effort, reliable, …)
• Group Membership (View synchrony, …)
• Network protocols (UDP, TCP, …)
• LAN vs WAN
47
Appia
• Flexible group communication library available
under APL v2 (http://appia.continuent.org)
• Composed by
• A core used to compose protocols
• A set of protocols
• group communication, ordering guarantees, atomic
broadcast, among other properties
• Created and supported by the Appia team
• University of Lisbon, Portugal
• DIALNP research group
48
Appia Protocol Composition
(example)
Application
Total Order
Group Communication
TCP
Atomic Broadcast
Channel
49
Group Communication
Channel
Appia features (1/2)
• Possible transport protocols
• UDP / IP Multicast
• TCP
• TCP+SSL
• Existing protocols
• Causal order
• Total order
•
•
•
•
•
50
Sequencer-based;
Token-based;
Total-Causal;
Hybrid (Adaptive Sequencer/Causal);
Optimistic (sequencer-based) Uniform Total Order
Appia features (2/2)
• Event-based delivery model
• Support for asynchronous events
• Optional view synchrony with failure
detector
• Support for QoS constraints
51
Sequoia/Java configuration
• Keep all original database specific
settings as is (SQL dialect, connection
test statements, …)
• Copy sequoia-driver.jar in classpath
• Set driver class name to
org.continuent.sequoia.driver.Driver
• Set JDBC URL to
jdbc:sequoia://host1,host2/mydb
52
Sequoia/PHP configuration
• ODBC
• Copy libodbsequoia.so in /usr/lib
• Configure odbc.ini (look at the examples)
• MySQL native library
• Install libMySequoia rpm or dbm package
• LD_PRELOAD=/usr/lib/libmysequoia.so
your_program
• $link = mysqli_connect("node1 node2", "user",
"password", "db1");
• Perl
• Use DBD:JDBC module
53
Configuring Sequoia with Derby
Network server
• Virtual database configuration file
<VirtualDatabase name="myDB">
<Distribution>
<MessageTimeouts/>
</Distribution>
…
<DatabaseBackend name=“derby1”
driver=“org.apache.derby.jdbc.ClientDriver”
url=
“jdbc:derby:net://localhost:1527/xpetstore;create=tru
e;retrieveMessagesFromServerOnGetMessage=true;”
connectionTestStatement="values 1">
<ConnectionManager …/>
</DatabaseBackend>
54
Configuring Sequoia Clustering
with Derby (1/2)
<RequestManager>
<RequestScheduler>
<RAIDb-1Scheduler level=“passThrough"/>
</RequestScheduler>
<RequestCache>
<MetadataCache/>
<ParsingCache/>
<ResultCache granularity="table" />
</RequestCache>
<LoadBalancer>
<RAIDb-1>
<RAIDb-1-LeastPendingRequestFirst/>
</RAIDb-1>
</LoadBalancer>
55
Configuring Sequoia Clustering
with Derby (2/2)
<RecoveryLog>
<JDBCRecoveryLog
driver="org.apache.derby.jdbc.ClientDriver “
url="jdbc:derby:net://localhost:1529/xpetstore;
create=true;retrieveMessagesFromServerOnGetMessage=true;"
login="APP" password="APP">
<RecoveryLogTable tableName="RECOVERY"
idColumnType="BIGINT NOT NULL" sqlColumnName="sqlStmt“
sqlColumnType="VARCHAR(8192) NOT NULL“
extraStatementDefinition=",PRIMARY KEY (id)"/>
<CheckpointTable tableName="CHECKPOINT"/>
<BackendTable tableName="BACKENDTABLE"/>
<DumpTable tableName=“DUMPTABLE”/>
</JDBCRecoveryLog>
</RecoveryLog>
</RequestManager>
</VirtualDatabase>
56
Sequoia to access Derby embedded
Java client
Sequoia JDBC
driver
PHP
client
.Net
client
ODBC
client
ODBSequoia
DB X
native client
DB X Native
API.
Custom
C/C++
client
Carob C++ API
JVM
Sequoia protocol
Sequoia controller
Embedded Derby
57
jdbc:derby:path;create=true
Outline
• RAIDb
• Sequoia
• Building an HA solution
• Scalability
• High availability
58
Sequoia vertical scalability
• allows nested RAIDb levels
• allows tree architecture for scalable write
broadcast
• Sequoia driver re-injected in Sequoia
controller
59
Advanced Configurations
Vertical Scalability
Client
program
Client
program
Sequoia
driver
Sequoia
driver
JVM
JVM
Mixed Horizontal and Vertical Scalability
Client
program
Sequoia
driver
Client
program
Client
program
Client
program
Sequoia
driver
JVM
Sequoia
driver
Sequoia
driver
JVM
JVM
Sequoia controller
Full replication
Sequoia controller
Full replication
JVM
DB native
JDBC driver
Sequoia
driver
Sequoia controller
Partial replication
Sequoia driver
Sequoia controller
Full replication
Sequoia driver
DB 1
DB 2
Sequoia controller
Full replication
Sequoia controller
Full replication
Sequoia controller
Full replication
Sequoia controller
Full replication
DB native JDBC driver
DB native JDBC driver
DB native JDBC driver
Sequoia
driver
DB native
JDBC driver
DB native JDBC driver
DB 8
DB 9
DB 10
Sequoia controller
Full replication
DB native JDBC driver
DB 1
DB 2
DB 3
DB 4
DB 5
DB 6
DB7
DB 3
DB 4
Sequoia controller
Full replication
DB native JDBC driver
DB 5
60
DB 6
DB 7
DB 11
DB 12
DB 13
TPC-W benchmark (Amazon.com)
Throughput in requests per minute
• Nearly linear speedups with the shopping mix
1600
1400
1200
1000
Single Database
800
Full replication
600
Partial replication
400
200
0
0
2
4
Num ber of nodes
61
6
Performance vs Scalability
• Sequoia is not a parallel database
• No parallelization of query execution plan
• Query do not go faster when database is not
loaded
• Sequoia distributes the load evenly
• Constant response time when load increases
• Better throughput when load surpasses
capacity of a single database
62
Understanding scalability (1/2)
Performance vs. Time
500
450
400
Response time
350
300
1 Database - Load in users
Single DB
1 Database - Response time
Sequoia 2 DBs - Load in users
Sequoia 2 DBs - Response time
250
Sequoia
200
150
20 users
100
50
0
00:00:00
01:12:00
02:24:00
03:36:00
04:48:00
06:00:00
Time (sec.)
63
07:12:00
08:24:00
09:36:00
10:48:00
Understanding scalability (2/2)
Performance vs. Time
2500
1 DB - Load in users
Single DB
1 DB - Response time
2000
Sequoia 2DB - Load in users
Response time
Sequoia 2 DB - Response time
1500
Sequoia
1000
500
90 users
0
00:00:00
00:28:48
00:57:36
01:26:24
Time (sec.)
64
01:55:12
02:24:00
02:52:48
Outline
• RAIDb
• Sequoia
• Building an HA solution
• Scalability
• High availability
65
Initializing the cluster
EJB Container
• Dump initial Derby
database using backuper
• RecoveryLog tables are
initialized
dump
for initial
checkpoint
JOnAS, WebLogic,
JBoss, WebSphere, ...
C-JDBC driver
JVM
C-JDBC Controller
Recovery
Log
Derby JDBC driver
Derby
disabled
66
disabled
Logging
• Backend is enabled
• All database updates
are logged (SQL
JDBC
statement, user,
Recovery Log
transaction, …)
dump
for initial
checkpoint
EJB Container
JOnAS, WebLogic,
JBoss, WebSphere, ...
C-JDBC driver
JVM
C-JDBC Controller
Recovery
Log
Derby JDBC driver
Derby
enabled
67
enabled
Adding new backends 1/3
• Add new
backends while
system online
• Restore dump
EJB Container
JOnAS, WebLogic,
JBoss, WebSphere, ...
C-JDBC driver
JVM
JDBC
Recovery Log
enabled
C-JDBC Controller
Recovery
Log
dump
for initial
checkpoint
Derby JDBC driver
Derby
disabled
68
Derby
enabled
Derby
disabled
Adding new backends 2/3
EJB Container
• Replay
updates from
the log
JOnAS, WebLogic,
JBoss, WebSphere, ...
C-JDBC driver
JVM
JDBC
Recovery Log
enabled
C-JDBC Controller
Recovery
Log
Derby JDBC driver
dump
for initial
checkpoint
Derby
disabled
69
Derby
enabled
Derby
disabled
Adding new backends 3/3
EJB Container
• Auto-enable
backends when
done
JOnAS, WebLogic,
JBoss, WebSphere, ...
C-JDBC driver
JVM
JDBC
Recovery Log
enabled
C-JDBC Controller
Recovery
Log
Derby JDBC driver
dump
for initial
checkpoint
Derby
enabled
70
Derby
enabled
Derby
enabled
Taking new dumps (1/3)
• Calling backup command
on a backend will
automatically
disable/backup/reJDBC
enable
Recovery Log
dump
for initial
checkpoint
...
EJB Container
JOnAS, WebLogic,
JBoss, WebSphere, ...
C-JDBC driver
JVM
C-JDBC Controller
Recovery
Log
Derby JDBC driver
dump
for dump
last
for
last
checkpoint
checkpoint
Derby
disabled
71
enabled
Derby
enabled
Derby
enabled
Taking new checkpoints (2/3)
• Replay missing updates
from log
EJB Container
JOnAS, WebLogic,
JBoss, WebSphere, ...
C-JDBC driver
JVM
JDBC
Recovery Log
dump
for initial
checkpoint
...
enabled
C-JDBC Controller
Recovery
Log
Derby JDBC driver
dump
for dump
last
forfor
last
checkpoint
checkpoint
Derby
disabled
72
Derby
enabled
Derby
enabled
Making new checkpoints (3/3)
EJB Container
• Auto-enable
backend when
done
dump
for initial
checkpoint
JOnAS, WebLogic,
JBoss, WebSphere, ...
C-JDBC driver
JVM
JDBC
Recovery Log
...
enabled
C-JDBC Controller
Recovery
Log
Derby JDBC driver
dump
for dump
last
for
for
last
checkpoint
checkpoint
Derby
enabled
73
Derby
enabled
Derby
enabled
Handling failures
• A node fails!
• Automatically
disabled but
administrator fix
needed
dump
for initial
checkpoint
EJB Container
JOnAS, WebLogic,
JBoss, WebSphere, ...
C-JDBC driver
JVM
JDBC
Recovery Log
...
C-JDBC Controller
Recovery
Log
Derby JDBC driver
dump
for dump
last
for
for
last
checkpoint
checkpoint
Derby
disabled
74
enabled
Derby
enabled
Derby
enabled
Recovery 1/3
EJB Container
• Restore latest
dump
JOnAS, WebLogic,
JBoss, WebSphere, ...
C-JDBC driver
JVM
JDBC
Recovery Log
dump
for initial
checkpoint
...
C-JDBC Controller
Recovery
Log
Derby JDBC driver
dump
for dump
last
for
last
checkpoint
checkpoint
Derby
disabled
75
enabled
Derby
enabled
Derby
enabled
Recovery 2/3
EJB Container
• Replay missing
updates from log
JOnAS, WebLogic,
JBoss, WebSphere, ...
C-JDBC driver
JVM
JDBC
Recovery Log
dump
for initial
checkpoint
...
C-JDBC Controller
Recovery
Log
Derby JDBC driver
dump
for dump
last
for
for
last
checkpoint
checkpoint
76
enabled
Derby
disabled
Derby
enabled
Derby
enabled
Recovery 3/3
• Re-enable backend
when done
EJB Container
JOnAS, WebLogic,
JBoss, WebSphere, ...
C-JDBC driver
JVM
JDBC
Recovery Log
dump
for initial
checkpoint
...
C-JDBC Controller
Recovery
Log
Derby JDBC driver
dump
for dump
last
for
for
last
checkpoint
checkpoint
Derby
enabled
77
enabled
Derby
enabled
Derby
enabled
Current limitations
•
•
•
•
Distributed joins
Some JDBC 3.0 extensions
XA support through XAPool only
Triggers and updateable views supported
with limitations (fully available in
Sequoia 3.0)
• WAN network partition and reconciliation
not supported
78
Summary
• Multi-tier HA
• replication needed at each tier
• Appia group communication
• Sequoia
• high availability with fully transparent
failover
• performance scalability
• Visit http://www.continuent.org
79
Q&A
_________
Thanks to all users and
contributors ...
http://www.continuent.org
80