Download eHealth space related issue troubleshooting guide

Troubleshooting Guide: eHealth Disk Space Related Issues
This paper covers following topics:
What are the symptoms that usually indicate an eHealth server is possibly running out
of disk space?
OS commands that can be used to identify the disk usage on Linux and UNIX
Ehealth utilities for obtaining disk space used by each Oracle tablespace and datafile?
Tips and tricks you need to know when adding and moving Oracle data file
Guide on how to Troubleshoot common issue of redo log files filling up the
ArchiveLogs directory
Some useful SQL queries for obtaining space usage for oracle tablespace and datafiles
What are the symptoms that usually indicate an eHealth server is possibly running out of disk
space
The following is a list of symptoms or situations that usually indicate an eHealth application may
or will be run out disk space soon.
1. Disk usage is reaching 90% on any partitions, and free space in Oracle home and eHealth
home partitions each is less than 5G.
2. Error of unable to add elements in system.log or eHealth console usually indicates that
that oracle datafiles could not be extended due to either disk is full or a datafile is
reaching maximum size of 32G. This symptom mainly occurs for datafiles in NH_DATA01
and NH_INDEX tablespaces. Sometimes, this symptom also occurs when the tmp or
rollback datafiles are approaching 30G in size.
3. Stats rollup had been failing for several weeks or a few months
4. ArchiveLogs directory had been filled up due to deleteArchive log job failure or due to a
generation of many redo log files in a short time range. A typical situation we have seen
is that one to several redo log files are generated within one minute.
5. Other symptoms, in addition to those mentioned above, occurs less frequently include
generation of huge amount of oracle trace files and generation of some big core dump
files. Also availability backfill can usually use all space allocated for $NH_HOME.
OS commands that can be used to identify the disk usage on Linux and UNIX
1. You can use df –k, df –m and df –h to list space usage in kilobyte, megabyte and
gigabyte
2. You can use df -h $NH_HOME (the place where eHealth application is installed) or df –h
$NH_ORACLE_HOME (the place where Oracle 10g or 11g is installed) to list space usage
for eHealth home or Oracle home partition in gigabyte respectively.
Following example command shows Oracle home partition has 15G free with a usage of
74%
df -h $NH_ORACLE_HOME
Filesystem
/dev/mapper/VolGroup00-LogVol00
Size
57G
Used
40G
Avail
15G
Use%
74%
Mounted on
/
3. You can use du command to list space usage for a specific directory and its
subdirectories
Following example command shows directory $NH_HOME/web has used 382
megabytes
du -sm $NH_HOME/web
382 /eHealth/web
Following example command shows everything including files and subdirectories under
$NH_HOME/web has used 390648 kilobytes. Please note that this does not means
$NH_HOME has used 390648.
du -s $NH_HOME/web
390648 /eHealth/web
Following example command displays the file size in kilobyte under $NH_HOME/web.
For example, the file size for file of /eHealth/web/jars/web.jar is 9848 kilobytes. Note
that $NH_HOME is /eHealth
du -ak $NH_HOME/web
……………………..
………………………
24
/eHealth/web/tomcat/bin/commons-daemon.jar
4
/eHealth/web/tomcat/bin/digest.sh
8
/eHealth/web/tomcat/bin/daemon.sh
448
/eHealth/web/tomcat/bin
120528 /eHealth/web/tomcat
4
/eHealth/web/tmp
9848 /eHealth/web/jars/web.jar
9852 /eHealth/web/jars
390648 /eHealth/web
[eHealth@rontr01-U109701 eHealth]$ ls -l /eHealth/web/jars/web.jar
-r--r--r-- 1 eHealth eHealth 10065032 Jun 17 20:29 /eHealth/web/jars/web.jar
Ehealth utilities for obtaining disk space used by each Oracle tablespace and datafile
You can run nhDbStatus or nhiShowDbSpace from command line to obtain disk space used by
each tablespace and datafile. You can also retrieve the same or similar information from the
database status tab in OneClick. You can use this information to determine if you need to add a
new datafile to a certain tablespace or move a datafile from one partition to another. Below are
some examples showing how to use this information to manage datafiles. The important points
you need to know are that the maximum size for a single eHealth datafile is 32G. A datafile can
be auto extended to 32G as needed. And you must add a new datafile into a tablespace if each
one of the datafiles in the tablespace is 32G or close to 32G.
Below is an example output of nhDbStatus that shows free space in each datafile, but the output
may not be that useful because free space in zero byte for EHEALTH/NH_DATA02a.dbf does not
necessarily mean this datafile is reaching maximum size of 32G. Therefore, please don’t use the
free space in the output as the only measure to determine if a new datafile needs to be added.
[eHealth@rontr01-U109701 eHealth]$ nhDbStatus
Database Name: EHEALTH
Database Size: 8174829568.00 bytes
RDBMS Version: 11.2.0.3.0
+---------------+--------------------+--------------------+--------------------------------Tablespace
Free Space In Byte
Free Space In Byte On Device
+---------------+--------------------+--------------------+--------------------------------NH_INDEX
243023872.00
15275810816.00
/eh_database/oradata/EHEALTH/NH_INDEX01.dbf
NH_SAMPLE
3144704000.00
15275810816.00
/eh_database/oradata/EHEALTH/NH_SAMPLE01.dbf
NH_VC_DATA
2662400.00
15275810816.00
/eh_database/oradata/EHEALTH/NH_VC_DATA01.dbf
NH_ROLLBACK
392953856.00
15275810816.00
/eh_database/oradata/EHEALTH/NH_ROLLBACK01.dbf
NH_VC_PR8_INDEX01 97280000.00
15275810816.00
/eh_database/oradata/EHEALTH/NH_VC_PR8_INDEX01a.dbf
NH_USERS
43663360.00
15275810816.00
/eh_database/oradata/EHEALTH/NH_USERS01.dbf
NH_VC_INDEX
3940352.00
15275810816.00
/eh_database/oradata/EHEALTH/NH_VC_INDEX01.dbf
SYSTEM
163905536.00
15275810816.00
/eh_database/oradata/EHEALTH/SYSTEM01.dbf|
NH_REG_INDEX02
459317248.00
15275810816.00
/eh_database/oradata/EHEALTH/NH_REG_INDEX02a.dbf
NH_REG_INDEX01
434503680.00
15275810816.00
/eh_database/oradata/EHEALTH/NH_REG_INDEX01a.dbf
NH_REG01
385024000.00
15275810816.00
/eh_database/oradata/EHEALTH/NH_REG01a.dbf
Data file|
NH_REG02
409600000.00
15275810816.00
/eh_database/oradata/EHEALTH/NH_REG02a.dbf
NH_DATA01
403456000.00
15275810816.00
/eh_database/oradata/EHEALTH/NH_DATA01a.dbf
NH_DATA02
00.00
15275810816.00
/eh_database/oradata/EHEALTH/NH_DATA02a.dbf
NH_CRN
104448000.00
15275810816.00
/eh_database/oradata/EHEALTH/NH_CRN01.dbf
NH_VC_PR8_DATA01 60416000.00
15275810816.00
/eh_database/oradata/EHEALTH/NH_VC_PR8_DATA01a.dbf|
SYSAUX
67239936.00
15275810816.00
/eh_database/oradata/EHEALTH/SYSAUX01.dbf
+---------------+--------------------+--------------------+--------------------------------Below is an example output of nhiShowDbSpace for datafiles that shows total and free space in
each datafile. The output indicates that DATA01a.dbf and DATA01b.dbf in tablespace
NH_DATA01 are full and a new datafile needs to be added to NH_DATA01 tablespace.
nhiShowDbSpace datafiles
[eHealth@rontr01-U109701 sys]$ ./nhiShowDbSpace datafiles
Datafile Report
Datafile name
Total file size in Megabyte
/eh_database/oradata/EHEALTH/NH_DATA01a.dbf
/eh_database/oradata/EHEALTH/NH_DATA01b.dbf
/eh_database/oradata/EHEALTH/NH_DATA02a.dbf
/eh_database/oradata/EHEALTH/NH_INDEX01.dbf
/eh_database/oradata/EHEALTH/NH_ROLLBACK01.dbf
…………………………….
………………………….
32023M
32093M
28900M
22000M
4025M
Below is another example output of nhiShowDbSpace for datafiles that indicates that there are
three datafiles in NH_DATA01 tablespace, they are DATA01a.dbf DATA01b.dbf and
DATA01c.dbf. Even though the first two datafiles are reaching the maximum size of 32G, the
third one is only about 3G and it can still be auto-extended to 32G, thus there is no need to add
a new datafile into the NH_DATA01 tablespace.
nhiShowDbSpace datafiles
[eHealth@rontr01-U109701 sys]$ ./nhiShowDbSpace datafiles
Datafile Report
Datafile name
Total file size in Megabyte
/eh_database/oradata/EHEALTH/NH_DATA01a.dbf
/eh_database/oradata/EHEALTH/NH_DATA01b.dbf
32023M
32093M
/eh_database/oradata/EHEALTH/NH_DATA01c.dbf
/eh_database/oradata/EHEALTH/NH_DATA02a.dbf
/eh_database/oradata/EHEALTH/NH_INDEX01.dbf
/eh_database/oradata/EHEALTH/NH_ROLLBACK01.dbf
…………………………….
3009M
28900M
22000M
4025M
Methods of adding or moving an Oracle datafile
eHealth uses nhManageDbSpace utility to add a new datafile, please run nhManageDbSpace –h
to understand its usage. Below is an example command used to add a new datafile, please note
that you need to use gigabyte for the size option and only need to use H:/OracleDb as the
newPath in the example below as the command will append
/oradata/EHEALTH/NH_DATA01c.dbf. If you use H:/OracleDb/oradata/EHEALTH as the
newPath, then the new datafile of NH_DATA01c.dbf would be added into
H:/OracleDb/oradata/EHEALTH/oradata/EHEALTH.
nhManageDbSpace -add -newPath H:/OracleDb -tablespace NH_DATA01 -size 2G
Created: H:\OracleDb\oradata\EHEALTH\\NH_DATA01c.dbf
Created: E:\eHealth62\oracle\database\GEN0VMMSAPP08_EHEALTH.lcf.2010-07-26-1706
Updated: E:\eHealth62\oracle\database\GEN0VMMSAPP08_EHEALTH.lcf
eHealth also uses nhManageDbSpace utility to move a datafile. Below is an example command
used to move a datafile d:/ca/oradata/EHEALTH/NH_INDEX01.dbf from d drive to e drive. Please
note that you need to stop eHealth server, then run the command below, then start the server.
This command only moves one datafile, you need to run the command again if you want to
move another datafile. Please also note that the command will move the datafile to
e:/ca/oradata3/oradata/EHEALTH/NH_INDEX01.dbf, but you only need to specify
e:/ca/oradata3 as the newPath because the command will append the rest of the path.
nhManageDbSpace -move -datafile 'd:/ca/oradata/EHEALTH/NH_INDEX01.dbf'
-newPath e:/ca/oradata3
You are not allowed to use nhManageDbSpace utility to move system, temporary or rollback
datafiles because moving these datafiles require shutting down the database, and bring the
database back to the mount state to allow the control file to be updated while the database is
not in use. Below is the example procedure that can be used to move a system datafile
‘d:/ca/oradata/EHEALTH/SYSTEM01.dbf’ from d drive to e drive.
sqlplus $NH_USER/$NH_USER
Shutdown immediate;
Startup mount;
Exit the sql session
mv d:/ca/oradata/EHEALTH/SYSTEM01.dbf e:/ca/oradata/EHEALTH ( this is OS command)
sign into sql session
alter database rename file 'd:/ca/oradata/EHEALTH/SYSTEM01.dbf ' to
' e:/ca/oradata/EHEALTH/SYSTEM01.dbf ';
alter database open;
Now you can start the ehealth server that will use the system01.dbf file in the new location
Troubleshooting issue of redo log files that fills up the ArchiveLogs directory
Normally the REDO log switches take place every 10-15 minutes. That means about 4 to 6 redo
log files would be created in one hour. But sometimes it can create 1 to several redo log files
within one minute due to a heavy database activity or a big database transaction. This rapid
creation of redo log files can fill up the ArchiveLogs directory in less than one hour, and hence
cause the eHealth running out of disk space because most eHealth only allocates 10 to 50G for
ArchiveLogs directory. The immediate workaround is to stop eHealth server, then remove all of
the redo log files under ArchiveLogs directory, then start the server again. Then check the
followings to figure out why.
1. Check to see if the deleteArchivelog job has been successfully run, if not, fix it
2. Check to see if the weekly Db Maintenance job has been successfully run, if not, fix
it.
3. Check to see how old and how frequently the redo log files are generated and what
is size of the redo log file, and send the output of ls –l.
4. Is a binary incremental save is scheduled, if so, then the archive logs will not be
deleted until full save is run
5. Check error messages in the system log and oracle alert log file during the time
range when the redo log files accumulate. The error may give a clue regarding what
database transaction, or what sql may trigger the redo log file accumulation.
Below are the possible solutions depending on the specific situation
1. Remove the redo logs in ArchiveLogs directory, stop eHealth server, and
database, followed by restarting them
2. Fix deleteArchiveLog job
3. Found what heavy database transaction that may cause redo log file
accumulation, and take action
4. Resize the redo log file size to make it larger that would reduce the number of
log switches
5. Destroy, create and reload a recent database save that was saved prior to redo
log accumulation.
Some useful SQL queries for obtaining space info and for troubleshooting issues
1. A sql query to report space usage for each tablespace/datafile
Following sql query can be used to report eHealth database space usage information.
For each Oracle tablespace/datafile, the SQL query reports the total disk space
allocated, disk space used, free disk space and the percent free, number of free
fragments, size of largest free fragment and the status of auto extend. The largest
fragment represents the largest contiguous chunk of free space. When an object tries to
allocate the next extent, it first looks for the largest fragment, if the size of the largest
fragment is less than that of the next extent, the datafile get extended. If the auto
extend for the datafile is off or if no free space on device is available for the autoextension, Oracle reports error about unable to extend the object in the oracle alert log
file, as well as in ehealth system log file.
nhisql “select b.file_name "DataFile",
b.tablespace_name "TableSpace",
b.bytes/1048576 "TotalAlloc(M)",
round((b.bytes - sum(nvl(f.bytes,0)))/1048576,2) "Used(M)",
round(sum(nvl(f.bytes,0))/1048576,2) "Free(M)",
count(f.bytes) "NumFrag",
round(max(nvl(f.bytes,0))/1048576,2) "MaxFrag(M)",
round((sum(nvl(f.bytes,0))/(b.bytes))*100,2) "%Free",
b.autoextensible "AautoExt"
from dba_free_space f, dba_data_files b
where f.file_id(+) = b.file_id
group by b.tablespace_name, b.file_name, b.bytes,b.autoextensible
order by b.tablespace_name;”
Below is an example output from the sql above that indicates a new datafile needs
to be added to NH_INDEX tablespace.
Space usage report by Tablespace and Datafile
DataFile
----------------------------------------------------------------------------------------------------------------------------------------------------TableSpace TotalAlloc(M) Used(M) Free(M) NumFrag MaxFrag(M) %Free Aau
------------- ------------- --------- --------- --------- ---------- --------- --/oracle/oradata/EHEALTH/nh_data11.dbf
NH_DATA01
9720.00 4348.63 5371.37 226.00 496.00 55.26 YES
/oracle/oradata/EHEALTH/nh_data12.dbf
NH_DATA01
32000.00 4252.48 27747.52 258.00
496.00
86.71 YES
/oracle/oradata/EHEALTH/nh_data13.dbf
NH_DATA01
32000.00 5133.09 26866.91 264.00
496.00
83.96 YES
/oracle/oradata/EHEALTH/nh_index02.dbf
NH_INDEX
32765.64 4040.45 28725.20 677.00
496.00
87.67 YES
/oracle/oradata/EHEALTH/nh_indx01.dbf
NH_INDEX
32761.23 2607.45 30153.78 654.00
/oracle/oradata/EHEALTH/nh_rbs01.dbf
NH_ROLLBACK
1900.00 80.31 1819.69
496.00
92.04 YES
15.00 1769.94
95.77 YES
/oracle/oradata/EHEALTH/nh_users01.dbf
NH_USERS
1316.00 457.36 858.64 997.00
/oracle/oradata/EHEALTH/system01.dbf
SYSTEM
400.00 253.77 146.23
1.00
76.56
146.23
65.25 YES
36.56 YES
2. A sql query to obtain the configured statistic rollup schedule
The statistics rollup schedule is a very important parameter that determines how long
raw data, hourly data, and daily data is maintained in eHealth database. This
parameter, in turn, determines in great degree, the database size. This parameter can
be found in eHealth OneClick. But it can also be obtained from running a sql below.
nhisql “SELECT DECODE (RLP_STAGE_NMBR, 0, 'raw data', 1, 'hourly data', 2, 'daily
data') AS DATA_TYPE, DURATION_SIZE/86400 days
FROM nh_rlp_plan
WHERE RLP_TYPE = 'ST'”
The table below shows an example of the output the query might display.
DATA_TYPE
DAYS
----------- ----------------raw data
2
hourly data
42
daily data
490
The output shows that eHealth keeps 2 days raw data (also called as-polled data), 42
days hourly data, and 490 days daily data.
3. A sql query to verify autoextend Is enabled for a datafile
nhisql “SELECT file_name, autoextensible
FROM dba_data_files”
The table below shows an example of the output the query might display.
File_name
autoextensible
-------------------------------------------------------------------C:\...\NH_VC_PR8_DATA01A.DBF
YES
D:\...\NH_VC_PR8_DATA01B.DBF
NO
…………..
………….
The output shows that file ‘C:\...\NH_VC_PR8_DATA01A.DBF ‘ is autoextensible. But file
‘D:\...\NH_VC_PR8_DATA01B.DBF’ is not autoextensible.
4. A sql query to Enable autoextend on a data File and on a tablespace
Following sql can be used to enable autoextend capabilities of any of the
datafiles (as sysdba). So that more space get added automatically from device when the
datafile gets full.
nhisql “ALTER DATABASE DATAFILE '/u05/app/oracle/oradata/R816/rbs01.dbf '
AUTOEXTEND ON”
Enable autoextend on a Tablespace (as sysdba)
nhsql “alter TABLESPACE NH_DATA01 AUTOEXTEND ON”
5. A sql query to obtain space usage for a table within a tablespace
Following sql reports space usage by each table for a specified tablespace, please
replace the tablespace name to the one you want to query for. It’s often asked why
NH_DATA01 tablespace get filled quickly when the stats rollup had been failed for a
days, or weeks. The answer is that there are too many stats0 tables in the tablespace
that are beyond the rollup schedule supposed to keep. This sql output provides table
name and its size for each table in a specified tablespace.
nhisql “SELECT a.segment_name,b.max_extents, a.extents,
a.bytes/1024 tot_size,a.next_extent/1024 next, b.num_rows
FROM dba_segments a, dba_tables b
WHERE a.owner='EHEALTH'
AND a.tablespace_name = 'NH_DATA01'
AND a.segment_type='TABLE'
ORDER BY b.table_name”
6. A sql query to obtain space usage for each index within a tablespace
Following sql to report space usage for each index for a specified tablespace, please
replace the tablespace name to the one you want to query for. An index is sized in the
same way as a table except that oracle calculates the average row length only for the
column that are part of the index. This sql can be used to demonstrate that a NH_INDEX
datafile is getting big in size or the datafile can be filled up so quickly when the stats
rollup job or the stats index job is failed.
nhisql “SELECT a.segment_name,b.max_extents, a.extents, a.bytes/1024 tot_size,
a.next_extent/1024 next, b.num_rows
FROM dba_segments a, dba_indexes b
WHERE a.owner='EHEALTH'
AND a.segment_type='INDEX'
AND a.segment_name=b.index_name
AND a.tablespace_name = 'NH_INDEX'
ORDER BY b.index_name”