* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Efficient, hot & automatic oracle database cloning
Survey
Document related concepts
Microsoft Access wikipedia , lookup
Entity–attribute–value model wikipedia , lookup
Serializability wikipedia , lookup
Microsoft SQL Server wikipedia , lookup
Team Foundation Server wikipedia , lookup
Open Database Connectivity wikipedia , lookup
Extensible Storage Engine wikipedia , lookup
Ingres (database) wikipedia , lookup
Functional Database Model wikipedia , lookup
Relational model wikipedia , lookup
Microsoft Jet Database Engine wikipedia , lookup
Oracle Database wikipedia , lookup
Versant Object Database wikipedia , lookup
Concurrency control wikipedia , lookup
Database model wikipedia , lookup
Transcript
Efficient, hot & automatic oracle database cloning “Discover how to clone your production database without disruption, in a totally automated & efficient way” Josep Vidal Canet Universtitat de València Motivation • Useful for the DBA or system administrator who wants to give his developers a full-sized TEST and DEV instance by cloning the PROD instance into the development server system • To train new, inexpert DBAs • To test backup and recovery strategies, workloads, software upgrades, migrations, etc Universitat de Valencia • Spain’s third-largest university – 45.000 students – 3.500 teachers – 2.000 workers • Different RDBMS: – DB2 / ZOS mainly OLTP – The biggest amount of data is stored in Oracle / Unix / Linux • Both OLTP & OLAP – Postgres What’s a database clone? • A database clone is a complete and separate copy of a database system that includes the business data, applications and the DBMS software (wikipedia) • The cloned DB is both fully functional and separate in its own right DB cloning strategies • Physical copy -> same server architecture & oracle version • Export / import : different architectures or oracle versions The Big Picture I: GOAL Prod Server or LPAR (Source) Test Server or LPAR (Target) Prod Instance Cloned Instance SGA SGA PMON SMON DBWR LGWR CKPT Others Data Files Database Control Files Redo Log Files To produce a physical copy of prod DB Automatically Without disruption Efficient Reliable Secure PMON SMON DBWR LGWR CKPT Others Data Files Database Control Files Redo Log Files pclone.sh • A script that automates the whole process: – Without disrupting production systems • Remote Hotbackup • If the OS allows it (AIX, HP-UX, Solaris) WLM controls are imposed to guarantee QoS – Efficient – Easy to schedule as a crontab job – Only differences (modified blocks) from prod to test database are copied – Parallel programming techniques are applied to reduce the amount of time required • The whole process is triggered by target system, which queries source database catalog and launches a remote hotbackup • Once completed, it is recovered in target system The Big Picture II: Procedure Prod Server or LPAR Test Server or LPAR Cloned Instance Prod Instance SGA SGA PMON SMON DBWR LGWR CKPT Others Data Files Database Control Files Redo Log Files 1: Issue a remote Hotbackup 2: Automatically recover it on target system PMON SMON DBWR LGWR CKPT Others Data Files Database Control Files Redo Log Files pclone.sh • No special hardware or software required • Data is moved using a SSL encrypted connection • Used at UV to clone/update test instances from production systems in a wide range of platforms (Linux/x86, Solaris/SPARC, AIX/Power). • Easy to use: pclone.sh DB user/password • The tool can be downloaded from: – http://www.uv.es/vijo/pclone.sh • Paper: – http://www.uv.es/vijo/cloning_oracle_database.pdf Requirements • Same ORACLE & OS versions & system architecture • ssh & rsync tools, configured so you can login without password • Time synchronization in both servers – NTP (Network Time protocol). • Archive log mode • Prod instance reachable from test system with tnsping utility Main steps • • Automated using bash scripting Executed in the target system: 1. Queries source database catalog to obtain physical database objects 2. Copies each database object from source to target system ensuring database consistency 3. Starts the source database in the target system (DB recovery) Step 1: Database physical layout • Source DB discovery – Find server & $ORACLE_HOME with tnsping & /etc/oratab – Test source DB status • Query target DB to determine physical layout – tablespaces, datafiles, redo logs, archive logs, init.ora, etc … Step 1: Database physical layout • The underlying directory structure must be created on target system, before copying physical objects – mkdir -p `dirname $f` Oracle Database /OraHome1/dbs/initDB.ora Data Files Control Files Redo Log Files Parameter File Archived Log Files Password File /OraHome1/dbs/orapw /data01/oracle/oradata/DB/control01.dbf /data03/oracle/oradata/DB/tbs_03.dbf Step 2: Copying physical database objects between systems • Goal: To copy each type of database object from source to target ensuring database consistency • Data transfer utility: rsync – Capable of copying files between remote computers by transferring only file differences rsync • Utility: To copy files between two systems • Performs a block level comparison of two files • Transfers only modified blocks – huge benefit if you are transferring large files like dafafiles over a network link • Speeds up file transfers when the destination file already exists • rsync remote-update protocol – allows rsync to transfer just the differences between two sets of files across the network link, using an efficient checksum search Step 2: Copying physical database objects between systems • Idea: To launch a remote hotbackup from the target system – Physical database objects are copied from the source to target ensuring consistency – File paths are kept identical in both systems rsync -taupog source:/$dir/tbs01.dbf target:/$dir/tbs01.dbf Oracle database main files • Some files (dump, archives, init.ora) can be copied without taking care of database consistency • Others, like datafiles, must be in a consistent state before being copied Oracle Database Data Files Parameter File Password File Control Files Redo Log Files Archived Log Files Step 2: Moving physical database objects between systems remote_backup(){ STATUS=`target_db_status` if [ "$STATUS" == "OK" ]; then shutdown_db "IMMEDIATE" fi sync_dump_dirs sync_initora sync_temporary_datafiles sync_db_datafiles sync_db_ctrl_and_log_files } Moving datafiles Prod Server or LPAR (Source) Test Server or LPAR (Target) Prod Instance sqlplus client SGA PMON LGWR Data Files tbs01 tbs02 SMON CKPT alter TBS begin backup DBWR Others Database Control Files Redo Log Files rsync -taupog source:/dir/tbs01.dbf target:/dir/tbs01.dbf rsync -taupog source:/dir/tbs02.dbf target:/dir/tbs02.dbf rsync -taupog source:/dir/tbs03.dbf target:/dir/tbs03.dbf tbs03 Alter TBS end backup shell Data Files Database Moving datafiles • The algorithm can be summarized as: – for each Tablespace in the source database do: • Set tablespace in backup mode • Copy (using rsync) each datafile from source to target system using parallel techniques • End tablespace backup mode • A similar approach is used to move both control and redo log files Step 2: Copying datafiles sync_db_datafiles(){ TBS=`get_tablespaces_name ${SOURCE_DB_CONNECT_STRING}` switch_db_logfile _ini # Parallelism related variables & locks initialization for T in $TBS; do DATAFILES=`get_tbs_datafiles ${SOURCE_DB_CONNECT_STRING} ${T}` begin_tbs_backup ${SOURCE_DB_CONNECT_STRING} ${T} for d in $DATAFILES ; do _maximum_parallelism_barrier mkdir -p `dirname $d` (rsync -e 'ssh -c blowfish' -tapogL $HOST:$d $d; _sub;exit;)& done _wait_for_all_children_to_finish_barrier end_tbs_backup ${SOURCE_DB_CONNECT_STRING} ${T} done } Parallelism • Cloning large databases can take a long time • Why not just split the amount of objects needed to be copied among different tasks? – For each tablespace: • Create a process for each datafile to be copied until a maximum level of parallelism is reached (_maximum_parallelism_barrier) • Once the maximum parallelism level is reached, a new process is created when a running one ends • The backup state is released for a tablespace when every task is completed (_wait_for_all_children_to_finish_barrier ) • Do the same for the remaining database files (archived logs, dumps, etc …) Step 3: Recovering database • Goal: To restore database integrity in target system • How: by recovering the physical copy of the source database • Source database catalog is queried to determine which datafiles need to be recovered • For each datafile a ‘media recovery’ is issued against the physical copy • Main steps: – Database is mounted on target system – Media recovery is issued for each datafile – Database is opened Step 3: Recovering database recover_db(){ export ORACLE_SID=$SOURCE_DB startup_db "mount" TBS=`get_tablespaces_name ${SOURCE_DB_CONNECT_STRING}` for T in $TBS; do DATAFILES=`get_tbs_datafiles ${SOURCE_DB_CONNECT_STRING} ${T}` for d in $DATAFILES ; do recover_datafile $d "AUTOMATIC" done done open_db } Conclusions • Database cloning can be used for different purposes like testing, developing, database maintenance tests or DBA training • pclone.sh automates the whole process without disrupting production systems in an efficient manner • The whole process, major steps and the necessary code to implement it, has been discussed Conclusions II • Performance – The time needed to complete the whole process depends on: • database size & available computational resources (CPUs & disks) • database update rate as well as cloning interval – Example: • 150 GB accounting production database, running in a p570 server uncapped LPAR (6 power5 CPU’s) • After an initial cloning, no additional synchronization was performed for 6 months • Afterwards, the test database was synchronized from the prod one • The whole process took less than an hour