Download Pooling your connections

Pooling your connections
Tuomas Pystynen
Deepbase Consulting Oy
Who am I ?
Independent consultant since 1997
–
–
Database and application server expertise
–
–
Oracle Server Technologies
Oracle7 testing, RMAN development
–
Oracle experience since version 4
–
–
Nokia, Oracle Finland, ...
Member of the Oaktable Network
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
2
Database application architectures
●
●
Stateful database applications
–
each user session needs a database connection
–
client/server, Forms, HTML DB, batch jobs
–
not suitable for connection pooling
Stateless database applications
–
user session needs a database connection only
when executing database request
–
3-tier, Web applications, J2EE, Tuxedo
–
ideal for connection pooling
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
3
Database connection
●
●
●
A database connection
–
opens a socket to database server
–
creates a process/thread on database server
–
allocates memory/resources on database server
Number of concurrent connections is limited by
–
number of processes/threads
–
memory and other resources
Opening a database connection is slow
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
4
Scalability
●
●
High-end database servers can support
–
thousands of database connections
–
using a 64-bit operating system helps
Large number of connections
–
increase contention on Oracle and OS resources
–
increase CPU scheduling overhead
–
increase paging and swapping
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
5
Client-Server applications
●
Each user session has its own database
connection
–
●
●
not suitable for connection pooling
Can write stateful applications
–
keep transactions and locks
–
keep result sets open
Limited scalability
–
can use Oracle MTS
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
6
Oracle Multi-Threaded Server
●
●
Multiple connections share one server process
–
reduces number server processes
–
reduces memory usage (process stack)
If a connection waits for a resource
–
●
other connections using same process are blocked
MTS advice
–
designed for client/server applications
●
–
suitable for long living mostly idle connections
do not use with connection pooling
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
7
Stateless database applications
●
Users do not own database connections
–
borrow a connection to serve a request
●
–
can not keep transactions open and hold locks
between requests
●
–
must use optimistic “locking”
can not keep result sets open between requests
●
●
from a connection pool
need special techniques for large result sets
Many web applications use stateless model
–
better scalability
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
8
Connection Pooling
●
Application server maintains a connection pool
●
When application processes a request
●
–
gets a connection from the pool
–
processes the request
–
releases the connection back to the pool
A connection can serve many sessions
–
●
one at a time
A session can be served by many connections
–
each request by a different connection
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
9
Connection pooling
connection 1
connection 2
© 2007 Deepbase Consulting
session 33
session 14
session 45
session 33
session 77
session 25
OUGF Kevätseminaari 24.4. 2007
10
Writing stateless applications
●
Close statements before closing connection
–
avoid statement leaks
●
●
●
ORA-01000: maximum open cursors exceeded
Commit or rollback before closing connection
–
explicitly or implicitly depending on API
–
make sure auto-commit mode is off
Close connection when done
–
avoid connection leaks
●
ORA-00020: maximum number of processes exceeded
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
11
Max connections
●
OC4J “max-connections” attribute
–
new connections are opened as needed
●
●
–
if no unused connections are in pool
until “max-connections” is reached
get connection will wait
●
●
●
if “max-connections” are open and all are in use
until a connection is released back to pool
or until it times out (used-connection-wait-timeout)
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
12
Sizing connection pool
●
Stateless application should process the
database request and release the connection
–
●
no wait or think time while the connection is in use
Cary Millsap article
–
Batch Queue Management and the Magic of '2'
●
●
Batch jobs by definition are jobs with no think time during
an observation interval
If more than 2 batch jobs per CPU improves system
throughput, CPU is not your bottleneck.
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
13
Connection pool sizing test
●
100 simulated test clients with 1 sec think time
●
Test servlet that executes
–
●
Application server
–
●
select table_name from all_tab_columns where
column_name = ?
OC4J 10.1.2.0.2 on Windows XP
Database server
–
Intel Core 2 Duo E6600 with 4GB memory
–
Oracle EE 10.2.0.3 on RHEL 4 U4
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
14
Connection Pooling test results
max-size
-------1
2
4
8
16
32
64
128
getConnection
------------1712 ms
581 ms
356 ms
288 ms
152 ms
61 ms
20 ms
15 ms
executeQuery responseTime
------------ -----------26 ms
1738 ms
31 ms
612 ms
55 ms
411 ms
111 ms
399 ms
216 ms
368 ms
326 ms
387 ms
369 ms
389 ms
396 ms
411 ms
* average elapsed times measured in servlet code
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
15
Sizing connection pool
●
Allow 10 - 20 connections per db server CPU
–
too few connections will increase response times
●
–
long running requests may reserve all connections
●
●
–
●
especially if short and long running requests are mixed
tune them
create a separate connection pool for them
too many connections will increase contention
Consider also limiting concurrent sessions or
requests at application server level
–
if the database server cannot handle the load
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
16
Inactivity timeout
●
●
Causes OC4J to close inactive connections
–
which are unused for “inactivity-timeout” seconds
–
let OC4J close connections before firewall does
Can also restart application server periodically
–
to release resources
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
17
Statement caching
●
●
Stateless applications can not keep references
to prepared statements
–
application may be served by another connection
next time
–
statement caching helps to reduce parsing
OC4J connection pool can keep statement
cache for each connection
–
when statement is closed it is put to cache
–
when statement is prepared the cache is checked
first
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
18
Statement caching
●
Use statement parameters!
–
●
Statement caching helps only if identical SQL
statement is prepared many times
–
●
aka bind variables
statements using different literals are not identical
Statement caching works more efficiently if a
connection is reused by the same application
–
may create application specific connection pools
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
19
OC4J Spy servlet
●
●
Monitor OC4J statistics
–
http://localhost:8888/dms0/Spy or
–
dmstool
Show information about
–
connections
–
statements
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
20
OC4J Connection pool tracing
●
Start OC4 with debug properties
java -Ddatasource.verbose=true -Djdbc.debug=true
-Djdbc.connection.debug=true -jar oc4j.jar
●
Debug output is written to console
–
●
may be redirected to a file
Works only with OC4J 10g R2 and older
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
21
OC4J Connection pool tracing
●
OC4J 10g Release 3 connection pool tracing
–
enable using EM AS Control or
–
editing j2ee-logging.xml
<logger name='oracle.j2ee.datasource' level='TRACE:32'
–
by default output is written to oc4j/log.xml file
–
can redirect also to the console
java.util.logging.ConsoleHandler.level = ALL
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
22
Monitoring database sessions
●
Identifying sessions in database is not easy
●
Create a database service for connection pool
–
●
●
helps monitoring connection pool at db level
Use DBMS_APPLICATION_INFO
–
set module, action and client_info
–
visible in v$session, v$sql, v$sqlarea
–
written to SQL_TRACE files
In single-user test environment
–
set max-connections to 1
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
23
SQL_TRACE
●
Each database connection has its own trace file
–
●
users SQL_TRACE output is scattered across
many trace files
Enable SQL_TRACE using DBMS_MONITOR
–
enable for given service, module, action
–
use trcsess utility to consolidate trace files
–
use tkprof or another profiler to analyze
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
24
Error handling
●
●
Handle all errors
–
rollback
–
close statements and the connection
Can mask transient errors from users
–
retry a few times on error
rollback
close the connection
get a new connection
re-execute the request
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
25
OC4J Fatal error codes
●
Database connection may encounter fatal error
–
which indicate database instance failure
–
for example
●
●
OC4J closes all unused connections
–
●
ORA-03113 "end-of-file on communication channel"
when fatal error is encountered
Transparent to the application
–
it simply gets a SQLException
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
26
Fatal error codes
●
Predefined fatal error codes
ORA-01033: Oracle initialization or shutdown in progress
ORA-01034: ORACLE not available
ORA-01089: immediate shutdown in progress no operations are permitted
ORA-01090: shutdown in progress - connection is not permitted
ORA-03113: end-of-file on communication channel
ORA-03114: not connected to ORACLE
●
May define additional fatal error codes
ORA-01012: not logged on to Oracle
ORA-12571: TNS: packet writer failure
ORA-04068: Existing stage of package discarded
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
27
Instance failures
●
●
Opening a connection fails
–
while the database instance is down
–
while listener or network is down
OC4J can try opening a connection many times
–
max-connection-attempts
–
connection-retry-interval
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
28
Fast Connection Failover
●
RAC aware connection pooling
–
when an instance fails
●
●
–
when an instance is started up
●
●
ONS sends a DOWN event
all free connections to the failed instance are cleaned up
ONS sends an UP event
Application may get an error
–
if the instance dies in the middle of a request
–
should retry when an error occurs
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
29
Transparent Application Failover
●
Not connection pool aware
–
●
Tries to hide the error from application
–
●
works at individual connection level
but transactions and PL/SQL state is lost
Requires OCI
–
does not work with JDBC Thin driver
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
30
JDBC Driver
●
●
●
JDBC Thin driver
–
easy to install
–
limited functionality
JDBC OCI driver
–
requires client libraries (Oracle 10g instant client)
–
provides full functionality (Net, TAF, lobs)
OC4J bundles often old JDBC drivers
–
can upgrade, but requires testing
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
31
OC4J 10g Release 3
●
New connection pooling implementation
●
Managed data sources
●
–
OC4J connection and statement caching
–
new syntax in data-sources.xml
Native data sources
–
connection caching provided by JDBC driver
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
32
Summary
●
Get a connection, make database calls and
close the connection
–
●
Close statements and connections
–
●
even in error cases
Use statement caching
–
●
keep the connection for minimum time
use statement parameters instead of literals
Handle errors
–
retry when it makes sense
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
33
Summary
●
Use DBMS_APPLICATION_INFO
–
make your application traceable
●
Measure and log response times
●
Create a database service for each pool
●
Set max-connections to reasonable value
●
Monitor connections and open cursors
●
Restart application servers regularly
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
34
Questions?
© 2007 Deepbase Consulting
OUGF Kevätseminaari 24.4. 2007
35