Download disaster recovery procedure.

MISU2 DISASTER RECOVERY PROCEDURE
PRELIMINARIES
This document contains disaster recovery procedures for all critical systems on MISU2, in the OB9
machine room. For a list of systems covered, see Appendix B.
This document assumes that you are familiar with standard unix and Oracle commands and utilities.
Oracle documentation is available online at technet.oracle.com. For unix, see the man pages.
A complete list of unix and database passwords is kept in the fire safe in the Laws Building machine
room. Contact Security if you need help with the door keys or alarm code.
The core admin systems are currently at Oracle version 7.3.4, and run on MISU2 in the OB9 machine
room. There is also a single 8.1.7 database (EST82) on the same box, running a critical third-party app
for Estates. Development and testing takes place on MISU1, in the Laws Building machine room.
There is a full list of live databases and file locations in Appendix B.
Tape backups are taken nightly, with tapes re-used on a five-week cycle. Backup tapes from misu2 are
kept in the Laws Building fire safe. Daily exports of the databases are also made to disk on misu2 – see
Appendix C for locations.
RECOVERING ON MISU2
In cases of data corruption, disk failure or other problems that leave misu2 still available you have two
options for restoring the affected databases:
Recovery with database export(s)
Full exports of all the live databases are written to disk every night – see Appendix C for export file
locations. These are your best bet if you’re comfortable with the import utility.
1.
2.
3.
4.
5.
Locate the relevant export file on disk, or restore it from tape if necessary.
Drop the schema owner in the relevant database, with the CASCADE option.
Perform a full database import as user SYSTEM with IGNORE=y, COMMIT=y and
INDEXES=n options set.
Re-run the import with INDEXFILE=<filename> to generate an index creation script.
Log on to the database as SYSTEM and run the index creation script.
Recovery from cold backup(s)
If you’ve completely lost one or more databases or you’re not sufficiently familiar with the import
utility, you can restore database files from tape. Note that you’ll have to restore all the files for a given
database, even if some are still present.
1.
2.
3.
4.
Make a note of the files to be restored, and check that the it’s possible to restore them to their
exact original locations. If not (for example, because of defective disks) then see the
instructions for restoring to MISU1 in the next section.
If they’re still up, shut down the affected databases. If in doubt, shut down all the live
databases with the script:
/opt/admin/scripts/dbshutdown.rc
Get the required tape from storage. Make sure the red tab is showing (which write protects the
backup tape). Pull it over if its not.
Load the tape into the tape changer.
Press the load button on the box. (2nd button on the RHS), or use the command:
robot load drive 0 slot 0
6. Each backup tape holds several backup sets, each of which corresponds to a filesystem. From
Appendix A, find the sets containing the files you want to restore.
7. To find a particular backup set on a tape in the drive, use:
mt fsf <set>
(where <set> is the set number)
8. Backups are made with vdump, so to restore you will need to use vrestore. Full details
can be found in the man page, but the following examples may be useful:
vrestore –l –f /dev/nrmt0h
- lists the contents of the current fileset.
vrestore –i –f /dev/nrmt0h
- choose files to restore in interactive mode. Type ‘help’ for a list of available
commands, or ‘q’ to quit.
vrestore –x –f /dev/nrmt0h <list of files>
- restore one or more named files, or the entire set if you don’t specify any.
9. If you need to restore from another fileset, you’ll need to rewind the tape before proceeding:
mt rewind
10. When you’re finished eject the tape:
mt offline
11. Finally, start the newly-restored databases, or, if you shut everything down, start all the live
databases with:
/opt/admin/scripts/dbshutdown.rc
5.
RECOVERING MISU2 TO MISU1
If MISU2 is unusable, you will need to restore the live databases to MISU1, the test machine in the
Laws Building. This is essentially the same procedure as ‘recovery from cold backups’ above.
Prepare MISU1
1.
2.
3.
Shutdown all development databases. Use the script:
/opt/admin/scripts/dbshutdown.rc
Shutdown the SQL*Net listener:
lsnrctl stop
From the information in Appendix B, create filesystems on MISU1 that reflect the original
filesystems on MISU2. This may involve the destruction of one or more development
databases to free up space.
Restore files from tape
1.
2.
3.
4.
5.
6.
Get the required tape from storage. Make sure the red tab is showing (which write protects the
backup tape). Pull it over if its not.
Load the tape into the tape changer.
Press the load button on the box. (2nd button on the RHS), or use the command:
robot load drive 0 slot 0
Each backup tape holds several backup sets, each of which corresponds to a filesystem. From
Appendix A, find the sets containing the files you want to restore. This should include
everything listed in Appendix B, including the files listed under ‘Oracle Networking’.
To find a particular backup set on a tape in the drive, use:
mt fsf <set>
(where <set> is the set number)
Backups are made with vdump, so to restore you will need to use vrestore. Full details
can be found in the man page, but the following examples may be useful:
vrestore –l –f /dev/nrmt0h
- lists the contents of the current fileset.
vrestore –i –f /dev/nrmt0h
- choose files to restore in interactive mode. Type ‘help’ for a list of available
commands, or ‘q’ to quit.
vrestore –x –f /dev/nrmt0h <list of files>
- restore one or more named files, or the entire set if you don’t specify any.
7.
8.
If you want to restore from another fileset, you’ll need to rewind the tape before proceeding:
mt rewind
When you’re finished eject the tape:
mt offline
Post-restore actions
1.
2.
3.
Start up the newly-restored databases using the pre-existing script on MISU1:
/opt/admin/scripts/dbstartup_MISU2.rc
Start the SQL*Net listener:
lsnrctl start
Contact the network support team and arrange for the DNS entry for MISU2 to be pointed at
MISU1, so that end-users can access the restored system.
APPENDIX A: MISU2 Tape Backup Filesets
(Last updated: 01/10/2003)
Set
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Filesystem
/
/usr
/users
/oracle
/d01/logs
/d01/exports
/d02
/d03
/d04
/d05
/d06
/d07
/d08
/d30
/oratmp
APPENDIX B: MISU Database File Locations
(Last updated: 01/10/2003)
ACCOM2
Datafile
/d04/oradata/ACCOM2/data01ACCOM2.dbf
/d02/oradata/ACCOM2/idx01ACCOM2.dbf
/d03/oradata/ACCOM2/rbs01ACCOM2.dbf
/d03/oradata/ACCOM2/rbs02ACCOM2.dbf
/d06/oradata/ACCOM2/sys01ACCOM2.dbf
/d05/oradata/ACCOM2/tmp01ACCOM2.dbf
/d04/oradata/ACCOM2/tools01ACCOM2.dbf
Control Files and Redo Logs
/d07/oradata/redo/ACCOM2/redo01ACCOM2.dbf
/d07/oradata/redo/ACCOM2/redo02ACCOM2.dbf
/d07/oradata/redo/ACCOM2/redo03ACCOM2.dbf
/d05/oradata/redo/ACCOM2/redo01ACCOM2b.dbf
/d05/oradata/redo/ACCOM2/redo02ACCOM2b.dbf
/d05/oradata/redo/ACCOM2/redo03ACCOM2b.dbf
/d02/oradata/control/ctrl1ACCOM2.ctl
/d03/oradata/control/ctrl2ACCOM2.ctl
/d04/oradata/control/ctrl3ACCOM2.ctl
Size (KB)
6656000
2560000
1024000
573440
51200
12672000
51200
Configuration Files
/oracle/app/oracle/product/7.3.4/dbs/initACCOM2.ora
/oracle/app/oracle/product/7.3.4/dbs/configACCOM2.ora
ALUM
Datafile
/d07/oradata/ALUM/big_rb_ALUM.dbf
/d02/oradata/ALUM/data01ALUM.dbf
/d02/oradata/ALUM/data02ALUM.dbf
/d03/oradata/ALUM/idxs01ALUM.dbf
/d07/oradata/ALUM/rbs01ALUM.dbf
/d05/oradata/ALUM/sys01ALUM.dbf
/d04/oradata/ALUM/tmp01ALUM.dbf
/d04/oradata/ALUM/tools01ALUM.dbf
Size (KB)
51200
4669440
128000
153600
40960
51200
21504000
163840
Control Files and Redo Logs
/d06/oradata/redo/ALUM/redo01ALUM.dbf
/d06/oradata/redo/ALUM/redo02ALUM.dbf
/d06/oradata/redo/ALUM/redo03ALUM.dbf
/d04/oradata/redo/ALUM/redo02ALUMb.dbf
/d04/oradata/redo/ALUM/redo01ALUMb.dbf
/d04/oradata/redo/ALUM/redo03ALUMb.dbf
/d04/oradata/control/ctrl1ALUM.ctl
/d02/oradata/control/ctrl2ALUM.ctl
/d03/oradata/control/ctrl3ALUM.ctl
Configuration Files
/oracle/app/oracle/product/7.3.4/dbs/initALUM.ora
/oracle/app/oracle/product/7.3.4/dbs/configALUM.ora
EST82
Datafile
/d03/oradata/EST82/gva_audit01.dbf
/d07/oradata/EST82/gva_dat01.dbf
/d08/oradata/EST82/gva_idx01.dbf
/d05/oradata/EST82/indx01.dbf
/d04/oradata/EST82/rbs01.dbf
/d02/oradata/EST82/system01.dbf
/d06/oradata/EST82/temp01.dbf
/d03/oradata/EST82/tools01.dbf
/d03/oradata/EST82/users01.dbf
Size (KB)
307200
41369600
1459200
55296
528384
266240
69632
8192
106496
Control Files and Redo Logs
/d30/oradata/EST82/redo01.log
/d30/oradata/EST82/redo02.log
/d30/oradata/EST82/redo03.log
/d02/oradata/EST82/control01.ctl
/d05/oradata/EST82/control02.ctl
/d30/oradata/EST82/control03.ctl
Configuration Files
/oracle/app/oracle/product/8.1.7/dbs/initEST82.ora
SPS2
Datafile
/d02/oradata/SPS2/bigrbs01SPS2.dbf
/d05/oradata/SPS2/data01SPS2.dbf
/d06/oradata/SPS2/discdata01SPS2.dbf
/d03/oradata/SPS2/idxs01SPS2.dbf
/d02/oradata/SPS2/rbs01SPS2.dbf
/d02/oradata/SPS2/rbs02SPS2.dbf
/d02/oradata/SPS2/rbs03SPS2.dbf
/d02/oradata/SPS2/rbs04SPS2.dbf
/d07/oradata/SPS2/sys01SPS2.dbf
Size (KB)
102400
81920
20480
81920
40960
40960
40960
40960
71680
SPS2 (cont.)
Datafile
/d06/oradata/SPS2/tmp01SPS2.dbf
/d07/oradata/SPS2/tools01SPS2.dbf
Size (KB)
6144000
204800
Control Files and Redo Logs
/d04/oradata/redo/SPS2/redo01SPS2.dbf
/d04/oradata/redo/SPS2/redo02SPS2.dbf
/d04/oradata/redo/SPS2/redo03SPS2.dbf
/d06/oradata/redo/SPS2/redo01SPS2b.dbf
/d06/oradata/redo/SPS2/redo02SPS2b.dbf
/d06/oradata/redo/SPS2/redo03SPS2b.dbf
Configuration Files
/oracle/app/oracle/product/7.3.4/dbs/initSPS2.ora
/oracle/app/oracle/product/7.3.4/dbs/configSPS2.ora
SRS2
Datafile
/d04/oradata/SRS2/bigrbs01SRS2.dbf
/d06/oradata/SRS2/data01SRS2.dbf
/d07/oradata/SRS2/data03SRS2.dbf
/d07/oradata/SRS2/data04SRS2.dbf
/d02/oradata/SRS2/discdata01SRS2.dbf
/d02/oradata/SRS2/discdata02SRS2.dbf
/d07/oradata/SRS2/idx01SRS2.dbf
/d04/oradata/SRS2/rbs01SRS2.dbf
/d04/oradata/SRS2/rbs02SRS2.dbf
/d04/oradata/SRS2/rbs03SRS2.dbf
/d04/oradata/SRS2/rbs04SRS2.dbf
/d03/oradata/SRS2/sys01SRS2.dbf
/d02/oradata/SRS2/tmp01SRS2.dbf
/d03/oradata/SRS2/tools01SRS2.dbf
Size (KB)
3932160
38912000
204800
2457600
307200
30720
29491200
860160
307200
860160
860160
122880
171008000
7741440
Control Files and Redo Logs
/d05/oradata/redo/SRS2/redo01SRS2.dbf
/d05/oradata/redo/SRS2/redo02SRS2.dbf
/d05/oradata/redo/SRS2/redo03SRS2.dbf
/d02/oradata/redo/SRS2/redo01SRS2b.dbf
/d02/oradata/redo/SRS2/redo02SRS2b.dbf
/d02/oradata/redo/SRS2/redo03SRS2b.dbf
/d02/oradata/control/ctrl1SRS2.ctl
/d03/oradata/control/ctrl2SRS2.ctl
/d04/oradata/control/ctrl3SRS2.ctl
Configuration Files
/oracle/app/oracle/product/7.3.4/dbs/initSRS2.ora
/oracle/app/oracle/product/7.3.4/dbs/configSRS2.ora
UACC
Datafile
/d03/oradata/UACC/DATA01.dbf
/d06/oradata/UACC/rbs01.dbf
/d07/oradata/UACC/system.dbf
/d07/oradata/UACC/temp01.dbf
/d02/oradata/UACC/tools01.dbf
Size (KB)
15360
5120
51200
102400
122880
Control Files and Redo Logs
/d02/oradata/redo/UACC/redo01a.dbf
/d03/oradata/redo/UACC/redo01b.dbf
/d02/oradata/redo/UACC/redo02a.dbf
UACC (cont.)
Control Files and Redo Logs
/d03/oradata/redo/UACC/redo02b.dbf
/d02/oradata/redo/UACC/redo03a.dbf
/d03/oradata/redo/UACC/redo03b.dbf
/d05/oradata/control/ctrl1UACC.ctl
/d06/oradata/control/ctrl2UACC.ctl
/d07/oradata/control/ctrl3UACC.ctl
Configuration Files
/oracle/app/oracle/product/7.3.4/dbs/initUACC.ora
/oracle/app/oracle/product/7.3.4/dbs/configUACC.ora
UCAS2
Datafile
/d06/oradata/UCAS2/bigrbs01UCAS2.dbf
/d07/oradata/UCAS2/PGASdata01.dbf
/d07/oradata/UCAS2/PGASdata02.dbf
/d04/oradata/UCAS2/UCAS94data01.dbf
/d05/oradata/UCAS2/discdata01.dbf
/d06/oradata/UCAS2/PGASidx01.dbf
/d06/oradata/UCAS2/PGASidx02.dbf
/d05/oradata/UCAS2/UCAS94idx01.dbf
/d05/oradata/UCAS2/UCAS94idx02.dbf
/d06/oradata/UCAS2/rbs01UCAS2.dbf
/d06/oradata/UCAS2/rbs02UCAS2.dbf
/d06/oradata/UCAS2/rbs03UCAS2.dbf
/d06/oradata/UCAS2/rbs04UCAS2.dbf
/d02/oradata/UCAS2/sys01UCAS2.dbf
/d07/oradata/UCAS2/tmp01UCAS2.dbf
/d07/oradata/UCAS2/tmp02UCAS2.dbf
/d02/oradata/UCAS2/tools01UCAS2.dbf
Size (KB)
163840
61440
51200
41932800
2426880
35840
30720
491520
491520
368640
614400
368640
614400
737280
10240000
10240000
327680
Control Files and Redo Logs
/d03/oradata/redo/UCAS2/redo01UCAS2.dbf
/d03/oradata/redo/UCAS2/redo02UCAS2.dbf
/d03/oradata/redo/UCAS2/redo03UCAS2.dbf
/d05/oradata/redo/UCAS2/redo01UCAS2b.dbf
/d05/oradata/redo/UCAS2/redo02UCAS2b.dbf
/d05/oradata/redo/UCAS2/redo03UCAS2b.dbf
/d05/oradata/control/ctrl1UCAS2.ctl
/d06/oradata/control/ctrl2UCAS2.ctl
/d07/oradata/control/ctrl3UCAS2.ctl
Configuration Files
/oracle/app/oracle/product/7.3.4/dbs/initUCAS2.ora
/oracle/app/oracle/product/7.3.4/dbs/configUCAS2.ora
APPENDIX C: EXPORT FILE LOCATIONS
ACCOM2
ALUM
EST82
SPS2
SRS2
UACC
UCAS2
/d01/exports/ACCOM2
/d01/exports/ALUM
/d01/exports/EST82
/d01/exports/SPS2
/d01/exports/SRS2
/d01/exports/UACC
/d01/exports/UCAS2