Download ppt - C-JDBC

status
Emmanuel Cecchet
[email protected]
JOnAS developer workshop – http://www.objectweb.org - [email protected]
1 - 23/02/2004
Outline
Overview
Advanced concepts
Query caching
Horizontal scalability
Monitoring
Perspectives
JOnAS developer workshop – http://www.objectweb.org - [email protected]
2 - 23/02/2004
Motivations
J2EE performance scalability bounded by
database performance
Database tier must be
–scalable
–fault tolerant (high availability + failover)
–without modifying the client application
–using open source databases
–on commodity hardware
JOnAS developer workshop – http://www.objectweb.org - [email protected]
3 - 23/02/2004
How do we want to use
?
• end-to-end open source solution for large scale J2EE
clustered application servers
• cost-effective solution for both small and large
configurations
Apache
Internet
MySQL
JOnAS developer workshop – http://www.objectweb.org - [email protected]
4 - 23/02/2004
Redundant Array of
Inexepensive Databases
RAIDb controller
– gives the view of a single database to the client
– balance the load on the database backends
RAIDb levels
–
–
–
–
RAIDb-0: full partitioning
RAIDb-1: full mirroring
RAIDb-2: partial replication
composition possible
JOnAS developer workshop – http://www.objectweb.org - [email protected]
5 - 23/02/2004
C-JDBC
Middleware implementing RAIDb
Two components
– generic JDBC 2.0 driver (C-JDBC driver)
– C-JDBC Controller
C-JDBC Controller provides
–
–
–
–
performance scalability
high availability
failover
caching, logging, monitoring, …
Supports heterogeneous databases
JOnAS developer workshop – http://www.objectweb.org - [email protected]
6 - 23/02/2004
Using JOnAS with C-JDBC
• Without C-JDBC
Database
JDBC
Driver
JOnAS developer workshop – http://www.objectweb.org - [email protected]
7 - 23/02/2004
Using JOnAS with C-JDBC
Update the datasource to use the C-JDBC
driver
JOnAS
Virtual
Database Controller
JDBC
Driver
Driver
Cache
Pooling
JOnAS developer workshop – http://www.objectweb.org - [email protected]
Backend
8 - 23/02/2004
What C-JDBC offers
No application modification
No JOnAS modification
No database modification
Load balancing
Virtual
VirtualDatabase
Database Controller
Driver
JDBC
Driver
Cache
Pooling
JOnAS developer workshop – http://www.objectweb.org - [email protected]
Backend
Backend recovery
9 - 23/02/2004
Outline
Overview
Advanced concepts
Query caching
Horizontal scalability
Monitoring
Perspectives
JOnAS developer workshop – http://www.objectweb.org - [email protected]
10 - 23/02/2004
Inside the Controller
Java client
program
(Servlet, EJB, ...)
C-JDBC
driver
RMI
Sockets
Sockets
RMI
C-JDBC Controller
XML configuration file
Administration
console
JMX
Configuration
&
administration
Virtual database
Virtual database
Authentication Manager
Authentication Manager
Request Manager
Request Manager
XML engine
Scheduler
Recovery
Log
Request Cache
Scheduler
Recovery
Log
Request Cache
Load balancer
Load balancer
Database
Backend
Connection
Manager
Database
Backend
Connection
Manager
Database
Backend
Connection
Manager
Database
Backend
Database
Backend
Connection
Manager
Connection
Manager
MySQL
JDBC driver
MySQL
JDBC driver
MySQL
JDBC driver
Oracle
JDBC driver
Oracle
JDBC driver
MySQL
MySQL
MySQL
Oracle
Oracle
JOnAS developer workshop – http://www.objectweb.org - [email protected]
11 - 23/02/2004
Fault tolerant recovery log
Client application
(Servlet, EJB, ...)
C-JDBC driver
C-JDBC Controller
Checkpointing
service
Recovery virtual database
Authentication Manager
Request Manager
Scheduler
Load balancer
Database
Backend
MySQL
Database
Backend
Application virtual database
Authentication Manager
Request Manager
Recovery
Log
Scheduler
Request Cache
C-JDBC
driver
Database
Backend
MySQL
JOnAS developer workshop – http://www.objectweb.org - [email protected]
Load balancer
Database Database
Backend Backend
MySQL
MySQL
12 - 23/02/2004
Cascading controller
Client
program
Client
program
Client
program
C-JDBC
driver
JVM
C-JDBC
driver
C-JDBC
driver
JVM
JVM
C-JDBC controller
Full replication
C-JDBC controller
Full replication
DB native
JDBC driver
C-JDBC
driver
C-JDBC driver
C-JDBC controller
Full replication
DB 1
DB 2
DB native
C-JDBC
JDBC driver
driver
C-JDBC controller
Full replication
DB native JDBC driver
DB 3
DB 4
JOnAS developer workshop – http://www.objectweb.org - [email protected]
DB 5
DB 6
DB 7
13 - 23/02/2004
Heterogeneity support
Servlet container
 application already
written for a specific
[commercial] database
 user defined rules
for on-the-fly query
rewriting to execute
on heterogeneous
backends
Tomcat, Jetty, ...
C-JDBC driver
Java client
program
EJB Container
JOnAS, WebLogic,
JBoss, WebSphere, ...
JVM
C-JDBC driver
C-JDBC
driver
JVM
JVM
C-JDBC Controller
RAIDb-2
Oracle JDBC driver
Oracle
JOnAS developer workshop – http://www.objectweb.org - [email protected]
MySQL JDBC driver
MySQL
MySQL
MySQL
14 - 23/02/2004
Outline
Overview
Advanced concepts
Query caching
Horizontal scalability
Monitoring
Perspectives
JOnAS developer workshop – http://www.objectweb.org - [email protected]
15 - 23/02/2004
Query caching status
Cache contains a list of SQL->ResultSet
Policy defined by queryPattern->Policy
3 policies
– EagerCaching: variable granularities for invalidations
– RelaxedCaching: invalidations based on timeout
– NoCaching: never cached
RUBiS bidding mix
with 450 clients
No cache
Throughput (rq/min)
Coherent
cache
Relaxed
cache
3892
4184
4215
801 ms
284 ms
134 ms
Database CPU load
100%
85%
20%
C-JDBC CPU load
-
15%
7%
Avg response time
JOnAS developer workshop – http://www.objectweb.org - [email protected]
16 - 23/02/2004
RUBiS - Tomcat without
C-JDBC caching
Throughput: 3900 pages/min
1 Database
100% cpu
Tomcat
~50% cpu
JOnAS developer workshop – http://www.objectweb.org - [email protected]
17 - 23/02/2004
RUBiS - Tomcat with
C-JDBC relaxed caching
Throughput: 4200 pages/min
Tomcat
~55% cpu
1 Database
~20% cpu
C-JDBC
<10% cpu
JOnAS developer workshop – http://www.objectweb.org - [email protected]
18 - 23/02/2004
JOnAS and C-JDBC query
caching
RUBiS
Throughput
(req/min)
No cache
Response time
(ms)
No cache
Servlet
3890
4140
744
310
JOnAS SF
CMP 1.1
2947
3418
2929
2045
JOnAS
CMP 2.0
2178
2450
6372
4630
JOnAS developer workshop – http://www.objectweb.org - [email protected]
19 - 23/02/2004
Outline
Overview
Advanced concepts
Query caching
Horizontal scalability
Monitoring
Perspectives
JOnAS developer workshop – http://www.objectweb.org - [email protected]
20 - 23/02/2004
Horizontal scalability
JGroups for controller synchronization
Groups messages for writes only
Client 1
Client...
Client n
DB2
DB3
Client...
Controller1
Controller1
DB1
Client 1
DB4
JOnAS developer workshop – http://www.objectweb.org - [email protected]
DB1
DB2
Client n
Controller2
DB3
DB4
21 - 23/02/2004
Horizontal scalability
Centralized write approach issues
commit
Controller1
DB1
DB2
commit
Controller2
DB3
DB4
Controller1
DB1
DB2
Controller2
DB3
DB4
commit
commit
Controller2
DB1
DB2
DB3
DB4
?
?
?
?
Issues with transactions assigned to
T1(Write)
connections T1(Read)
Controller1
Controller2
Controller1
Controller2
Controller1
Controller2
DB1
DB2
DB1
DB2
DB1
DB2
JOnAS developer workshop – http://www.objectweb.org - [email protected]
T1(R)
T1(W)
T1(R) T2(W)
22 - 23/02/2004
Horizontal scalability
General case for a write query
– 3 multicast + 2n unicast
DistributedVirtualDatabase@controller2
DistributedVirtualDatabase@controller1
DistributedRequestManager
execWriteRequest(AbstractWriteRequest)
return result
Distributed
scheduling handler
DistributedRequestManager
Scheduler
Write query handler
Scheduler
Distributed
scheduling handler
Send request to all
controllers
Wait for schedulers replies
Load Balancer
Distributed load
balancer handler
Send execute request
order to all controllers
Wait for load balancer replies
Notify scheduler of completion
Notify controllers of
completion success or not
Return result
Distributed load
balancer handler
Load Balancer
Notify scheduler of completion
DistributedVirtualDatabase@controller3
DistributedRequestManager
Distributed
scheduling handler
Distributed load
balancer handler
Scheduler
Load Balancer
Notify scheduler of completion
JOnAS developer workshop – http://www.objectweb.org - [email protected]
23 - 23/02/2004
Horizontal scalability
Solution: No backend sharing
– 1 multicast + n unicast [+ 1 multicast]
Controller1
DB1
DB2
Controller2
DB3
DB4
Controller3
DB5
DB6
DistributedVirtualDatabase@controller2
DistributedVirtualDatabase@controller1
DistributedRequestManager
execWriteRequest(AbstractWriteRequest)
return result
DistributedRequestManager
Distributed
scheduling handler
Scheduler
Distributed load
balancer handler
Load Balancer
Write query handler
Scheduler
Distributed
scheduling handler
Send request to all
controllers
Wait if all backends failed
Load Balancer
DistributedVirtualDatabase@controller3
Distributed load
balancer handler
Wait for load balancer replies
Wait if all backends failed
Notify controller(s) that have
failed if other have succeeded
Return result
DistributedRequestManager
Distributed
scheduling handler
Distributed load
balancer handler
Scheduler
Load Balancer
Wait if all backends failed
JOnAS developer workshop – http://www.objectweb.org - [email protected]
24 - 23/02/2004
Horizontal scalability
Issues with JGroups
– resources needed by a channel
– instability of throughput with UDP
– performance scalability
TCP better than UDP but
– unable to disable reliability on top of TCP
– unable to disable garbage collection
– ordering implementation is sub-optimal
Need for a new group communication
layer optimized for cluster
JOnAS developer workshop – http://www.objectweb.org - [email protected]
25 - 23/02/2004
Horizontal scalability
 JGroups performance on UDP/FastEthernet
JOnAS developer workshop – http://www.objectweb.org - [email protected]
26 - 23/02/2004
Outline
Overview
Advanced concepts
Query caching
Horizontal scalability
Monitoring
Perspectives
JOnAS developer workshop – http://www.objectweb.org - [email protected]
27 - 23/02/2004
Design choices
JMX for both administration and
monitoring
– JMX server is MX4J
HTTP console for basic features but not
suited for monitoring
Swing console for monitoring
– uses JMX/RMI connector
Currently working on a scalable
monitoring infrastructure
JOnAS developer workshop – http://www.objectweb.org - [email protected]
28 - 23/02/2004
Online Monitoring
JOnAS developer workshop – http://www.objectweb.org - [email protected]
29 - 23/02/2004
Viewing logs remotely
JOnAS developer workshop – http://www.objectweb.org - [email protected]
30 - 23/02/2004
Outline
Overview
Advanced concepts
Query caching
Horizontal scalability
Monitoring
Perspectives
JOnAS developer workshop – http://www.objectweb.org - [email protected]
31 - 23/02/2004
Current status
C-JDBC 1.0 rc2 release
Web site: >100.000 hits per month
Downloads: >1200 per month
Mailing lists: 134 subscribers
Ready for production environments
JOnAS developer workshop – http://www.objectweb.org - [email protected]
32 - 23/02/2004
Ongoing work
 Scalable group communication middleware for clusters
– Needed for horizontal scalability
– Decouple application streams from network channels
 Monitoring
– Continuous monitoring stored in repositories
– Generic monitoring console for both online monitoring and monitoring history
browsing
 Administration
– Fully featured administration console
– Graphical configuration and deployment of centralized/distributed backeds
and controllers (offline/online)
– Dynamic reconfiguration
 Misc.
– Automated Load testing
– RPM packaging (JPackage)
JOnAS developer workshop – http://www.objectweb.org - [email protected]
33 - 23/02/2004
Questions & Answers
_________
Thanks to all users and
contributors ...
http://c-jdbc.objectweb.org
JOnAS developer workshop – http://www.objectweb.org - [email protected]
34 - 23/02/2004
Virtual Database
Java client
program
(Servlet, EJB, ...)
C-JDBC
driver
Virtual database
Authentication Manager
Request Manager
Scheduler
Recovery
Log
Request Cache
Load balancer
Database
Backend
Connection
Manager
Database
Backend
Connection
Manager
Database
Backend
Connection
Manager
MySQL
JDBC driver
MySQL
JDBC driver
MySQL
JDBC driver
MySQL
MySQL
MySQL
 gives the view of a single
database
 establishes the mapping
between the database name
used by the application and the
backend specific settings
 backends can be added and
removed dynamically
 configured using an XML
configuration file
JOnAS developer workshop – http://www.objectweb.org - [email protected]
35 - 23/02/2004
TPC-W Performance
(Amazon.com)
Throughput in requests per minute
1600
1400
1200
Single DB
1000
RAIDb-0
800
RAIDb-1
600
RAIDb-2
400
200
0
0
1
2
3
4
5
6
Num ber of nodes
JOnAS developer workshop – http://www.objectweb.org - [email protected]
36 - 23/02/2004
Outline
Overview
Fault tolerance
Performance
New features
Roadmap
JOnAS developer workshop – http://www.objectweb.org - [email protected]
37 - 23/02/2004
Building the initial state
 Octopus is an ETL tool
 Use Octopus to store a dump of
EJB Container
JOnAS, WebLogic,
JBoss, WebSphere, ...
C-JDBC driver
the initial database state
JVM
disabled
C-JDBC Controller
Octopus
dump
for initial
checkpoint
JOnAS developer workshop – http://www.objectweb.org - [email protected]
Recovery
Log
PostgreSQL JDBC driver
PostgreSQL
disabled
38 - 23/02/2004
Journaling
 Backend is enabled
 All database updates are logged
(SQL statement, user,
transaction, …)
EJB Container
JOnAS, WebLogic,
JBoss, WebSphere, ...
C-JDBC driver
JVM
JDBC
Recovery Log
Octopus
dump
for initial
checkpoint
enabled
C-JDBC Controller
Recovery
Log
PostgreSQL JDBC driver
PostgreSQL
enabled
JOnAS developer workshop – http://www.objectweb.org - [email protected]
39 - 23/02/2004
Adding backend on the fly
Add new backends
EJB Container
while system online
Restore dump
corresponding to
JDBC
initial checkpoint Recovery
Log
with Octopus
Octopus
JOnAS, WebLogic,
JBoss, WebSphere, ...
C-JDBC driver
JVM
enabled
C-JDBC Controller
Recovery
Log
PostgreSQL JDBC driver
dump
for initial
checkpoint
PostgreSQL PostgreSQL
disabled
enabled
JOnAS developer workshop – http://www.objectweb.org - [email protected]
PostgreSQL
disabled
40 - 23/02/2004
Synchronizing backends
EJB Container
Replay updates
JOnAS, WebLogic,
JBoss, WebSphere, ...
from the log
C-JDBC driver
JVM
JDBC
Recovery Log
enabled
C-JDBC Controller
Recovery
Log
Octopus
PostgreSQL JDBC driver
dump
for initial
checkpoint
PostgreSQL
disabled
JOnAS developer workshop – http://www.objectweb.org - [email protected]
PostgreSQL
enabled
PostgreSQL
disabled
41 - 23/02/2004
Expanded Cluster
Enable backends
EJB Container
JOnAS, WebLogic,
JBoss, WebSphere, ...
when done
C-JDBC driver
JVM
JDBC
Recovery Log
enabled
C-JDBC Controller
Recovery
Log
Octopus
PostgreSQL JDBC driver
dump
for initial
checkpoint
PostgreSQL
enabled
JOnAS developer workshop – http://www.objectweb.org - [email protected]
PostgreSQL
enabled
PostgreSQL
enabled
42 - 23/02/2004
Handling a backend failure
A node fails!
Automatically
EJB Container
JOnAS, WebLogic,
JBoss, WebSphere, ...
disabled but should
be fixed or changed
JDBC
by administrator
Recovery Log
C-JDBC driver
JVM
enabled
C-JDBC Controller
Recovery
Log
...
dump
for initial
checkpoint
Octopus
PostgreSQL JDBC driver
dump
for dump
last
for
for
last
checkpoint
checkpoint
PostgreSQL
disabled
JOnAS developer workshop – http://www.objectweb.org - [email protected]
PostgreSQL
enabled
PostgreSQL
enabled
43 - 23/02/2004
Restoring a backend
Restore latest dump
with Octopus
EJB Container
JOnAS, WebLogic,
JBoss, WebSphere, ...
C-JDBC driver
JVM
JDBC
Recovery Log
enabled
C-JDBC Controller
Recovery
Log
...
dump
for initial
checkpoint
Octopus
PostgreSQL JDBC driver
dump
for dump
last
for
last
checkpoint
checkpoint
PostgreSQL
disabled
JOnAS developer workshop – http://www.objectweb.org - [email protected]
PostgreSQL
enabled
PostgreSQL
enabled
44 - 23/02/2004
Re-synchronization
Replay missing updates
EJB Container
JOnAS, WebLogic,
JBoss, WebSphere, ...
from log
C-JDBC driver
JVM
JDBC
Recovery Log
enabled
C-JDBC Controller
Recovery
Log
...
dump
for initial
checkpoint
Octopus
PostgreSQL JDBC driver
dump
for dump
last
for
for
last
checkpoint
checkpoint
PostgreSQL
disabled
JOnAS developer workshop – http://www.objectweb.org - [email protected]
PostgreSQL
enabled
PostgreSQL
enabled
45 - 23/02/2004
Healed Cluster
Re-enable backend
EJB Container
JOnAS, WebLogic,
JBoss, WebSphere, ...
when done
C-JDBC driver
JVM
JDBC
Recovery Log
enabled
C-JDBC Controller
Recovery
Log
...
dump
for initial
checkpoint
Octopus
PostgreSQL JDBC driver
dump
for dump
last
for
for
last
checkpoint
checkpoint
PostgreSQL
enabled
JOnAS developer workshop – http://www.objectweb.org - [email protected]
PostgreSQL
enabled
PostgreSQL
enabled
46 - 23/02/2004
Prototype
 C-JDBC
Management
Framework
Shared design
JOnAS developer workshop – http://www.objectweb.org - [email protected]
47 - 23/02/2004
Making new checkpoints
Disable one backend to
have a coherent snapshot
Mark the new checkpoint
entry in the log
Use Octopus to store
the dump
JDBC
Recovery Log
EJB Container
JOnAS, WebLogic,
JBoss, WebSphere, ...
C-JDBC driver
JVM
enabled
C-JDBC Controller
Recovery
Log
...
dump
for initial
checkpoint
Octopus
PostgreSQL JDBC driver
dump
for dump
last
for
last
checkpoint
checkpoint
PostgreSQL
disabled
JOnAS developer workshop – http://www.objectweb.org - [email protected]
PostgreSQL
enabled
PostgreSQL
enabled
48 - 23/02/2004
Making new checkpoints
Replay missing updates
EJB Container
JOnAS, WebLogic,
JBoss, WebSphere, ...
from log
C-JDBC driver
JVM
JDBC
Recovery Log
enabled
C-JDBC Controller
Recovery
Log
...
dump
for initial
checkpoint
Octopus
PostgreSQL JDBC driver
dump
for dump
last
for
for
last
checkpoint
checkpoint
PostgreSQL
disabled
JOnAS developer workshop – http://www.objectweb.org - [email protected]
PostgreSQL
enabled
PostgreSQL
enabled
49 - 23/02/2004
Making new checkpoints
Re-enable backend
EJB Container
JOnAS, WebLogic,
JBoss, WebSphere, ...
when done
C-JDBC driver
JVM
JDBC
Recovery Log
enabled
C-JDBC Controller
Recovery
Log
...
dump
for initial
checkpoint
Octopus
PostgreSQL JDBC driver
dump
for dump
last
for
for
last
checkpoint
checkpoint
PostgreSQL
enabled
JOnAS developer workshop – http://www.objectweb.org - [email protected]
PostgreSQL
enabled
PostgreSQL
enabled
50 - 23/02/2004