Download Oracle For Openview troubleshooting guide

Oracle For Openview troubleshooting guide
Possible trouble Areas
Installation
Configuration
Maintenance
Troubleshoot
Tools
Communication SQL*Net/Net8
Ovo Server
opcctlm
User
User
Oracle DataBase
System Global Area (SGA)
DB Buffer Redo
Cache
Shared
Log
Pool
Buffer
opcdistm
User
opcmsgm
Dedicated Server
Background Processes
User
Redo-Log Files
opcdispm
Data-Files
Control-Files
Parameter File
Version 2 updated on February 6, 2007
Table of Contents
1 About OVO/UNIX and ORACLE.............................................................................................................................. 3
1.1 Introduction..................................................................................................................................................... 3
1.2 Combinations of the database and the server compatibles..................................................................................... 3
1.3 Installation – OVOU 7.X with Oracle 8i/9i.......................................................................................................... 4
1.4 Installation – OVOU 8.X with Oracle 9i/10g....................................................................................................... 5
1.5 Configuration.................................................................................................................................................. 6
1.5.1 OVOU 8.X installtion Script overview:........................................................................................................... 6
1.5.2 How can I check the Oracle version?............................................................................................................ 7
1.5.3 How can you find the Openview DB files?..................................................................................................... 7
1.5.4 How can you check which files are expandable?........................................................................................... 7
1.5.5 What is the maximum size of the OpenView DB files?..................................................................................... 8
1.5.6 Where are the OVO tables described?......................................................................................................... 8
1.5.7 Which are the ORACLE Database Users?...................................................................................................... 9
1.6 Maintenance................................................................................................................................................. 10
1.6.1 How can I do a manual reorganization?..................................................................................................... 10
1.6.2 Moving a data file to a different location .................................................................................................... 11
1.6.3 Moving the OpenView database into an existing ORACLE database.............................................................. 12
Copyright 2007
Page 1 of 38
1.7 Troubleshooting............................................................................................................................................. 12
1.7.1 What can I check?................................................................................................................................... 12
1.7.2 What can I do when Oracle does not shutdown?........................................................................................ 17
1.8 Tuning the database....................................................................................................................................... 17
1.9 Tools............................................................................................................................................................ 18
1.9.1 /opt/OV/contrib/OpC/opc_odc.............................................................................................................. 18
1.9.2 /opt/OV/contrib/OpC/opcdbck.............................................................................................................. 18
1.9.3 /opt/OV/bin/OpC/utils/opcchgaddr........................................................................................................ 18
1.9.4 /opt/OV/bin/OpC/opcdbidx.................................................................................................................. 19
1.9.5 Tfsulock................................................................................................................................................... 19
1.9.6 /opt/OV/bin/OpC/opcdbpwd................................................................................................................. 19
1.9.7 OERR..................................................................................................................................................... 20
2 Oracle For Openview troubleshooting guide........................................................................................................... 21
2.1 Understanding the communication mechanism used by OVO when connecting to the DB........................................ 21
2.1.1 Introduction............................................................................................................................................. 21
2.1.2 IPC protocol adapter................................................................................................................................ 21
2.1.3 TPC protocol adapter................................................................................................................................ 22
2.1.4 SQL/NET connection................................................................................................................................ 23
2.1.5 SQL*Net/ Net8 Configuration................................................................................................................... 23
2.1.6 Installation and configuration of SQL*Net/Net8.......................................................................................... 23
2.2 Troubleshooting SQL*Net/Net8...................................................................................................................... 24
2.2.1 Starting the listener and testing a client connection....................................................................................... 24
2.2.2 What can I do if the installation fails during the SQL*Net/Net8 configuration?................................................ 25
2.2.3 Oracle Networking tracing........................................................................................................................ 26
2.3 Tools............................................................................................................................................................ 26
2.3.1 tnsping utility........................................................................................................................................... 26
2.3.2 TRCROUTE utility...................................................................................................................................... 28
3 OVO Server........................................................................................................................................................ 30
3.1 Maintenance................................................................................................................................................. 30
3.1.1 Backup of OVO Data............................................................................................................................... 30
3.1.2 How to check the time elapsed for a history download?................................................................................ 31
3.1.3 How to check how many messages were downloaded?................................................................................ 31
3.2 Troubleshooting............................................................................................................................................. 31
3.2.1 Creation and configuration of the database during the OVO installation......................................................... 31
3.2.2 OVO automated backup........................................................................................................................... 33
3.2.3 opccfgdwn/opccfgupld............................................................................................................................. 33
3.2.4 SQL tracing in OVO................................................................................................................................. 34
3.2.5 Oracle runs, but OVO doesn’t connect........................................................................................................ 36
3.3 Tools............................................................................................................................................................ 36
3.3.1 /opt/OV/contrib/OpC/itochecker............................................................................................................ 36
Copyright 2007
Page 2 of 38
1 About OVO/UNIX and ORACLE
1.1 Introduction
All OVO server configurations require access to an ORACLE database.
In most setups this database resides on the same system as the OVO management server software. This is the default setup as
supported by the OVO installation and configuration procedures.
On HP-UX an ORACLE session appears as 2 processes: the application (OVO processes, sqlplus, ...) and a corresponding
ORACLE shadow process. Due to this dual process implementation a mechanism has to exist for communication of data and
commands between them.
The purpose of this document is to understand and troubleshoot the Oracle for OVOUX product in the following parts:
- Oracle database
- Communication with SQL*Net (listener)
- Application with OVO server
1.2 Combinations of the database and the server compatibles
HP—UX 11.11 PA-RISC OVOU 8.10
OVOU 7.X Oracle 9.2.0.2
OVOU 8.X Oracle 9.2.0.2,
Oracle 10.1.0.x where x is = or > than 4,
Oracle 10.2.0.y where y is = or > than 2
HP—UX 11.23 PA-RISC OVOU 8.20
Oracle 10.1.0.x where x is = or > than 4,
Oracle 10.2.0.y where y is = or > than 2
HP—UX Itanium 8.23 OVOU 8.20
Oracle 10.1.0.x where x is = or > than 4,
Oracle 10.2.0.y where y is = or > than 2
Solaris 2.8, 2.9, OVOU 8.10
Oracle 9.2.0.x where x is = or > than 2,
Oracle 10.1.0.y where y is = or > than 4
Solaris 2.10, OVOU 8.10
Oracle 10.1.0.x where x is = or > than 4,
Oracle 10.2.0.y where y is = or > than 2
Note:
Always check the most current OVOU Release Notes to verify the current Oracle Version/Patch Level of Support!
With OVOU 8.X you will need to install the latest Management Server Consolidated Patch while running
the ovoinstall script. By doing so you will have the latest scripts that ovoinstall calls when configuring the
Copyright 2007
Page 3 of 38
the OVOU Database. Be aware that in the Release Notes it may provide you with further instructions to
Either modify the ovoinstall script, an ftpsite do download a new one, or other details that are required for
a successfully install.
Copyright 2007
Page 4 of 38
Oracle Database
1.3 Installation – OVOU 7.X with Oracle 8i/9i
The principal steps for installing the Oracle database are the following:
- First of all, checking the Kernel parameters and the required patches.
-
-
Preparing an oracle database for installation:
o Creating a oracle user with user id > 100
o Creating a UNIX group = dba with a group id > 100
o Make the user oracle a member of the group dba
o As the home directory of the oracle user = /home/oracle
o Set umask 022 to allow user to access the oracle binaries.
o Create the Optimal Flexible Architecture (OFA) directory structure recommended by Oracle
/opt/oracle/product/<version> (=8.1.7 or 9.0.2 for OVO 7)
o Set the oracle environment variables in the /home/oracle/.profile of user oracle:
 ORACLE_BASE=/opt/oracle
 ORACLE_HOME=$ORACLE_BASE/product/<version> (8i or 9i for OVO 7)
 ORACLE_SID=openview
 ORACLE_TERM=hp|ansi
(ansi if you user dtterm)
 NLS_LANG= america.america.WE8ISO8859P15
 SHLIB_PATH=$ORACLE_HOME/lib for oracle 8i
 SHLIB_PATH=$ORACLE_HOME/lib32 for oracle 9i
Installing an oracle database
o Open 2 terminal windows
o Log on user root for the first one
o Log on user oracle for the second one
o Mount the CDROM as user root
o Run /SD_CDROM/install/hpunix/runInstaller & as user oracle
o
o
o
o
o
o
o
o
Copyright 2007
In the Oracle Universal Installer window follow the steps :
 Verify the path in the Inventory location window (/opt/oracle/oraInventory)
 Leave the field empty in the Unix group name window
Run the orainst.sh script as user root
Return to the Oracle Universal Installer window and Select product component (see the documentation Oracle
8i or 9i installation guide)
Install the components
Umount the first CDROM
Mount the second CDROM
Run root.sh script
Umount the second
Page 5 of 38
1.4 Installation – OVOU 8.X with Oracle 9i/10g
Follow tine Oracle Database instructions in the current Openview for Operations
Installation Guide. As the intent of this guide is to help with Oracle has it is
used in the HP Openview Environment it is best to keep current with those
published in the HP Openview Operation Guide along with the most current
Release notes for the OS you are installing OVOU on. The release notes will
frequently identify issues that have come up since the last publication of the
Installation Guide.
Recommend you set at least the following in both root’s and oracle user’s profile:
ORACLE_HOME
ORACLE_BASE
LANG
NSL_LANG
ORACLE_SID
Others that will be useful to set during the installation of OVOU with Oracle 9i:
ORA_NLS33
SHLIB_PATH
HP-UX
SHLIB_PATH=$ORACLE_HOME/lib32:$ SHLIB_PATH
LD_LIBRARY_PATH=${ORACLE_HOME}/lib
Oracle 32 Bit ... Solaris 64 Bit OS
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
Oracle 64 Bit ... Solaris 64 Bit OS
export LD_LIBRARY_PATH=$ORACLE_HOME/lib32
export LD_LIBRARY_PATH_64=$ORACLE_HOME/lib
Ref:
Doc ID: Note:131321.1 The note says:
Oracle 9.X.X (64Bit) on Solaris (64Bit) OS –
Set LD_LIBRARY_PATH=$ORACLE_HOME/lib32
Set LD_LIBRARY_PATH_64=$ORACLE_HOME/lib
Copyright 2007
Page 6 of 38
1.5 Configuration
1.5.1 OVOU 8.X installtion Script overview:
Overall Flow of Scripts that the ovoinstall runs for the installation of OVOU 8 as
it relates to the Oracle Database Creation:
::::ovoinstall ==> opcconfig ==> opcdbsetup ==> ovdbsetupo1_opc.sh - crdbov1.sql
::::
::::
==> ovdbsetupo2_opc.sh - crdbov2.sql
::::
::::
==> opcdbinst
::::
==> opcdbinit
::::
==> opcsqlnetconf
ovoinstall - Master install script for the OVOU Management Server.
Per Manpage:
* ask numbers for users/GUIs/agents
* check memory, kernel parameters and required patches
* ask various questions about the installation for OVO and OSSPI
* wrap the installation of NNM
* install OVO and - if desired - the OSSPI
* set up OVO
* deploy the local agent and templates - if desired
opcconfig – Main OVOU DB creation script which calls all required scripts necessary to setup the DB.
opcdbsetup - This script creates the OVO tablespaces, users and tables and loads the initial configuration.
opcdbinst - Generates the HP OpenView Operations (OVO) database schema
opcdbinit – load initial OVO configuration, insert MS coreid into DB
opcsqlnetconf - Shell script to configure Oracle Net listener
Note, with the OVOU 7.X installation the script opcconfig is run, the ovoinstall
script for OVOU 8.X will call opcconfig and in turn opcconfig will call:
opcdbsetup. Of the above all are scripts with the exception of opcdbinst which
is a shared executable called by opcconfig to generate the OVO database
schema.
Copyright 2007
Page 7 of 38
1.5.2 How can I check the Oracle version?
# su - oracle
$ sqlplus opc_op/<passwd of opc_op>@ov_net
SQL*Plus: Release 9.2.0.2.0 - Production on Tue Jun 1 19:15:52 2004
Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.
Connected to:
Oracle9i Enterprise Edition Release 9.2.0.2.0 - 64bit Production
JServer Release 9.2.0.2.0 - Production
SQL> exit
This gives both, the SQL*Plus version (which is usually the same as the client version) and the server version.
Note, that the newest itochecker (version A.07.19 and up) will collect this kind of data.
Also, at this
-
point, checking if the recommended or required Oracle patch sets are installed:
For Oracle 8.1.7, Oracle 8.1.7.4 patch set or higher is recommended
For Oracle 9.0.1, Oracle 9.0.1.4 patch set or higher is recommended
For Oracle 9.2.0, Oracle 9.2.0.2 patch set or higher is required.
If a remote database installation is used, this patch set is required on the database server system and on the database client
system.
1.5.3 How can you find the Openview DB files?
There are 2 different ways to locate the files:
- Checking the configuration file /etc/opt/OV/share/conf/ovdbconf
- Looking for ITO_DATADIR and ITO_INDEX
As user oracle call:
$sqlplus /nolog
>connect / as sysdba;
>select name from v$datafile;
or
>select file_name from dba_data_files;
The tables are described in the following manual:
HP OpenView VantagePoint Operations for UNIX Reporting and Database Schema
1.5.4 How can you check which files are expandable?
Copyright 2007
Page 8 of 38
Oracle 7.2 introduces the concept of "growing" DB files ("AUTOEXTEND" option). You can now create datafiles or alter
existing datafiles, so that they automatically increase in size when more space is needed in the database. The files increase in
specified increments up to a specified maximum.
Setting the datafiles to extend automatically results in the following:
* reduces the need for immediate intervention when a tablespace runs out of space
* ensures applications will not halt because of failures to allocate extents
It can be checked by connecting in the Oracle database as:
$sqlplus /nolog
>connect / as sysdba ;
>select file_name, autoextensible from dba_data_files;
1.5.5 What is the maximum size of the OpenView DB files?
Currently this is 500 MB and is hard coded in the installation scripts.
For how to change this, please see the man page of opcdbreorg.
1.5.6 Where are the OVO tables described?
There are described in the file /opt/OV/bin/OpC/install/opctabledef.
Example:
more /opt/OV/bin/OpC/install/opctabledef
CREATE TABLE OPC_USER_DATA
(
USER_ID
VARCHAR2(36) NOT NULL PRIMARY KEY,
SYMBOL_TYPE_ID
VARCHAR2(36) NOT NULL,
NODEHIER_ID
VARCHAR2(36),
USER_ROLE
NUMBER(3)
NOT NULL,
NAME
VARCHAR2(20) UNIQUE NOT NULL,
PASSWORD
VARCHAR2(40),
REALNAME
VARCHAR2(60),
DESCRIPTION
VARCHAR2(254),
LABEL
VARCHAR2(32))
TABLESPACE OPC_4;
Copyright 2007
Page 9 of 38
1.5.7 Which are the ORACLE Database Users?
Each ORACLE database has a list of valid database users. To access a database, a user must run a database application and
connect to the database instance using a valid user name defined in the database.
A user connecting to an instance can be authenticated by either the operating system or by Oracle. The openview DB is
created by the UNIX user oracle who is the default DBA user.
A newly created database has 2 default users, SYS and SYSTEM (With Oracle9 there is an additional one: OUTLN). The
passwords for these two usernames should be changed soon after the database is created; Default user/passwords are:
sys/change_on_install and system/manager. DBSNMP is an Oracle default user used by the (optional) Oracle SNMP agent.
The ITO installation scripts create 2 database users (opc_op and opc_report) in the OpenView DB. The passwords for these
users can be chosen when configuring ITO with the opcconfig/opcdbsetup scripts.
The user opc_report is a read-only user currently used by OV Service Reporter. The password of this user can be changed in
Oracle as the password of any other Oracle user (ALTER USER command...).
The Oracle user opc_op is used by ITO to connect to the DB.
The password is encrypted and stored in a file. When the ITO processes access the database, they will use the password from
this file.
The ITO user is not required to specify the DB password. This password is specified by the installation user during the setup or
during the upgrade. Use opcdbpwd(1M) to change the password of the opc_op database. This changes the password in the
database and updates the encrypted password in the file. Do not change the password using an Oracle tool like sqlplus as
ITO is no longer able to connect in this case.
(see also the notes on opcdbpwd in the tools chapter)
All users allowed to connect to an Oracle DB can be listed by checking the dictionary view dba_users, as follows:
# su – oracle
$ sqlplus /nolog
> connect / as sysdba;
Connected.
> select username,password from dba_users;
USERNAME
PASSWORD
---------------------------------------------------SYS
D4C5016086B2DC6A
SYSTEM
D4DF7931AB130E37
DBSNMP
E066D214D5421CCC
OPC_OP
9CF3CA19975AFA5C
OPC_REPORT
C791A4C37B125C26
5 rows selected.
> exit;
Server Manager complete.
To check the DB connection for the ORACLE user opc_op (this can be done as whatever UNIX user):
$ sqlplus
>connect opc_op/<password>
Copyright 2007
Page 10 of 38
The password is the password set at installation time (opcconfig) or as modified by using opcdbpwd.
Starting with ITO 5.0 all ITO processes connect to the Oracle database via sql*net. To check this connection please connect
the following way:
Sqlplus
> connect opc_op/<password>@ov_net
1.6 Maintenance
1.6.1 How can I do a manual reorganization?
After working with an Oracle DB for some time (and updating data) the data in an Oracle table is spread over different
"extents" or in different places within an "extent" with a lot of free/unused space in between.
In order to remove this fragmentation (and there is a whole lot of different types of fragmentation with Oracle) the standard
way is to export the data to a file, recreate each table having one "extent" only with the required size and import the data
back. But this could be done also by using the opcdbinst command as follow:
1.6.1.1 opcdbinst command
opcdbinst -act
It clears the active message tables in the OVO database. Some tables are shared with the history messages. They are only
cleared if the active and history messages are cleared.
opcdbinst -hist
It clears the history message tables in the OVO database. Some tables are shared with the active messages. They are only
cleared if the active and history messages are cleared.
1.6.1.2 Export / Import (example with the opc_act_messages table)
1- Shutdown OVOUX and all GUIs
The database must not be shutdown. To check:
ps- ef] grep ora
2- Create a temporary parameter file
Parameter file name: /tmp/parfile
Contents of the file:
Compress=Y
Grants=Y
Contraints=Y
Files=«/tmp/expdat.dmp»
Tables=(opc_act_messages)
Indexes=Y
3- Run oracle export
Copyright 2007
Page 11 of 38
$ORACLE_HOME/bin/exp opc_op/OpC_op parfile=/tmp/parfile
4- Drop the tablespace to be reorganized
Switch to user oracle and start the sqlplus utility:
Su - oracle
Sqlplus /nolog
>Connect / as sysdba;
>Drop tablespace OPC_1 including contents;
>Exit;
5- Check the size of the data files per tablespace
Identify the location of the OVOUX data file by looking into the configuration file ovdbconf:
Cd /etc/opt/OV/share/conf
Ll ovdb*
-rw-r----- 1 root
sys
818 Oct 30 10:29 ovdbconf
fgrep DATA_DIR ovdbconf
DATA_DIR /disc1/oracle/oradata/openview
Ll /disc1/oracle/oradata/openview/OPC_1*.dbf
-rw-r----- 1 oracle
dba
16785408 Jan 12 16:14 /disc1/oracle/oradata/openview/OPC_1_1.dbf
-rw-r----- 1 oracle
dba
6299648 Jan 11 04:11 /disc1/oracle/oradata/openview/OPC_1_2.dbf
Delete these files
The total size of all files belonging to OPC_1 is 23 MB.
6- Recreate the Oracle data file
Su -oracle
Sqlplus /nolog
>connect / as sysdba;
create tablespace opc_1 datafile ‘/disc1/oracle/oradata/openview/OPC_1_1.dbf’ size 23M
default storage (
initial 1M
next 1M
pctincrease 0);
>exit;
exit
7- Run Oracle import
${ORACLE_HOME}/bin/imp opc_op/OpC_op full=Y file=/tmp/expdat.dmp
8- Restart OVOUX and all GUIs
9- Check whether import was successful then delete the expdat.dmp file.
1.6.2 Moving a data file to a different location
In this scenario the file OPC_1_1.dbf is moved to a different disk.
1- Take the tablespace OPC_1 offline:
Su - oracle
Sqlplus /nolog
>connect / as sysdba;
>alter tablespace opc_1 offline normal;
2- In a second window move the file to the new location:
Mv /disc1/oracle/oradata/openview/OPC_1_1.dbf
Copyright 2007
Page 12 of 38
/disc2/oracle/oradata/openview/OPC_1_1.dbf
3- Back in the first window in sqlplus session, rename the data file using the ALTER TABLESPACE command:
>alter tablespace opc_1 rename datafile
‘/disc1/oracle/oradata/openview/OPC_1_1.dbf’ to
‘/disc2/oracle/oradata/openview/OPC_1_1.dbf’;
4- Bring the tablespace OPC_1 back online:
>alter tablespace opc_1 online
>exit;
5- Make sure that DATA_DIR is correctly set in the ovdbconf file:
# more ovdbconf|grep DATA_DIR
DATA_DIR /disc2/oracle/oradata
1.6.3 Moving the OpenView database into an existing ORACLE database
1- Acknowledge and download all OVOUX messages
2- Create an Oracle export parameter file
Contents of the file:
Compress=Y
Grants=Y
Contraints=Y
Files= « /tmp/openview.dmp »
Log=openview.log
Indexes=Y
3- Unload the existing openview database to a file by using the oracle exp utility
$ORACLE_HOME/bin/exp opc_op/OpC_op parfile=/tmp/parfile
4- In the database, create all OVOUX tablespaces
/opt/OV/bin/OpC/opcdbsetup
5- Reload the OVOUX table contents from the file in step 3
${ORACLE_HOME}/bin/imp opc_op/OpC_op full=Y file=/tmp/openview.dmp
6- Enable access to the database via sql*net
/opt/OV/bin/OpC/opcsqlnetconf
7- Modify the opcsvinfo file with the sql*net info
1.7 Troubleshooting
1.7.1 What can I check?
1.7.1.1 Check if the environnement variables are set correctly.
Verify that the environment variables ORACLE_HOME and ORACLE_SID for Oracle are correctly set.
Also, if the following variables are set, check that they are correctly set:
NLS_LANG
(e.g. america.WE8ISO8859P15)
SHLIB_PATH
($ORACLE_HOME/lib on Oracle8, $ORACLE_HOME/lib32 on Oracle9)
Copyright 2007
Page 13 of 38
ORA_NLS33
($ORACLE_HOME/ocommon/nls/admin/data)
The environment variables are described in the file /etc/opt/OV/share/conf/ovdbconf. The variables are listed in ovdbconf
and read from this file by some OV processes, but users who want to connect to the DB still need to set the variables in their
environment manually (or through .profile)
Example of ovdbconf :
DB_VENDOR Oracle
DB_NAME openview
DB_RELEASE 8.1.7
DB_TIME_STAMP "Tue Nov 26 15:38:43 MET 2002"
DB_USER ovdb
ORACLE_SID openview
ORACLE_HOME /u01/oracle/product/8.1.7
ORACLE_BASE /u01/oracle
DBA_USER oracle
DATA_DIR /var/opt/oradata/nnm
CREATE_DIR /u01/oracle/admin/openview/create
INDEX_DIR /var/opt/oradata/openview
ADMIN_DIR /u01/oracle
OS_AUTHENT_PREFIX
CHARACTER_SET WE8ISO8859P1
BASE_DATA_TS_SIZE 25
BASE_INDEX_TS_SIZE 5
DATA_TS_SIZE 25
INDEX_TS_SIZE
TEMP_TS_SIZE 1
DATA_TS_EXTENT_SIZE 1
DATA_TS_MAX_SIZE 500
INDEX_TS_EXTENT_SIZE
ECHO_CMD /usr/5bin/echo
PROMPT TRUE
DBA_PROGRAM svrmgrl
OV_USER ovdb
DBA_LOGFILE /var/opt/OV/share/log/svrmgrl_log
ORACLE_BASE_REV 8
ORACLE_SECOND_REV 1
NLS_LANG american_america.WE8ISO8859P1
ITO_DATADIR /opt/oradata/openview
ITO_INDEXDIR /opt/oradata/openview
1.7.1.2 Check the error Log files
Check the Oracle alert files and the Oracle trace files.
Alert file :
Copyright 2007
Page 14 of 38
It is a log file that records information about internal errors and administrative activities, such as backup, archiving …In the
event of an internal error, Oracle writes some information on the error to the alert file and a pointer to the trace file generated
because of this error.
The location of the parameter file is specified by the BACKGROUND_DUMP_DEST initialization parameter in initopenview.ora
file (if the instance is openview).
The default location is ${ORACLE_HOME}/rdbms/log/alert_openview.log (if the instance is openview).
Trace file:
Each time an Oracle instance starts, or an unexpected event occurs in a user process or background process, a trace file is
created.These files are not trace files in the common sense but are error log files. The name of the trace file includes the
process name and the process ID. It looks like:
Dbwr_23476.trc -> db writer trace file
Lgwr_23566.trc -> log writer trace file
Two initialization parameters specify where the trace files are stored:
BACKGROUND_DUMP_DESTand USER_DUMP_DEST in the initopenview.ora file.
1.7.1.3 Check for details the ORACLE errors
Running on the OVO server the tool:
Oerr ora <error number>
(See an example in tools chapter)
1.7.1.4 Check permission and ownership of the files.
All common system files and installation files will be owned by user oracle and will have the privileged group dba. All files or
directories should be written by oracle. To maintain access to all data, database, redo log files and control files must be
readable by oracle owner and group only. They must have mode 660 maintained.
Example:
-rw-r----- 1 oracle
dba
6905856 Jun 14 17:00 control03.ctl
1.7.1.5 Check the start / stop procedure of Oracle
Check that all processes accessing the database are stopped: opcsv -stop
As user root: /sbin/init.d/ovoracle start
/sbin/init.d/ovoracle stop
1.7.1.6 Check whether you have access to the database
Ps -ef|grep ora
This the typical output of OVO/UX running with Oracle
Copyright 2007
Page 15 of 38
Example:
#ps -ef|grep ora
oracle 1696
1 0 Mar 26 ?
root 1710 1599 0 Mar 26 ?
/var/opt/OV/dbspi/dsi/oracle
root 1711 1599 0 Mar 26 ?
/var/opt/OV/dbspi/dsi/oracl
oracle 1744
1 0 Mar 26 ?
oracle 1747
1 0 Mar 26 ?
oracle 1749
1 0 Mar 26 ?
oracle 1751
1 0 Mar 26 ?
oracle 1753
1 0 Mar 26 ?
oracle 1755
1 0 Mar 26 ?
oracle 1875
1 0 Mar 26 ?
oracle 1882
1 0 Mar 26 ?
oracle 1887
1 0 Mar 26 ?
oracle 1890
1 0 Mar 26 ?
oracle 1893
1 0 Mar 26 ?
oracle 1895
1 0 Mar 26 ?
0:01 /u01/oracle/product/8.1.7/bin/tnslsnr LISTENER -inherit
0:00 /opt/perf/bin/rep_server -t ORADB_FROV128_OPENVIEW
0:00 /opt/perf/bin/rep_server -t ORAOSM_FROV128_OPENVIEW
0:00
0:00
0:00
0:47
0:01
0:00
0:00
0:12
0:00
0:01
0:00
0:00
ora_pmon_openview
ora_dbw0_openview
ora_lgwr_openview
ora_ckpt_openview
ora_smon_openview
ora_reco_openview
oracleopenview (LOCAL=NO)
oracleopenview (LOCAL=NO)
oracleopenview (LOCAL=NO)
oracleopenview (LOCAL=NO)
oracleopenview (LOCAL=NO)
oracleopenview (LOCAL=NO)
All the processes starting with ora_ are Oracle system processes and oracleopenview processes are the
shadow/backgroundprocesses for the
OVO processes connected to the database.
1.7.1.7 Check OVOUX processes connected to the Oracle database
Bellow is a list of processes, which are opening a session with the DB.
/opt/OV/bin/OpC/opcmsgm
/opt/OV/bin/OpC/opcctlm
/opt/OV/bin/OpC/opcttnsm
/opt/OV/bin/OpC/opcdistm
/opt/OV/bin/OpC/opcsvcm
/opt/OV/bin/OpC/opcactm
/opt/OV/bin/OpC/opcdispm
/opt/OV/bin/OpC/opcuiop
(one operator connection)
/opt/OV/bin/OpC/opcuiadm
(administrator connection)
/opt/OV/bin/OpC/opcuiopadm
(administrator connection)
/opt/OV/bin/OpC/opcuiwww
(one javagui connection)
/opt/OV/bin/OpC/opcforwm
(MOM configuration)
/opt/OV/bin/OpC/opccmm
((MOM configuration)
Besides the manager processes permanently running while OVO is up, there are other processes, which are connecting to the
database.
Examples: opcnode (this tool maintains node, node groups and template assignments), opcack (this tool acknowledges
externally active messages)
Copyright 2007
Page 16 of 38
1.7.1.8 Check login as oracle and as DBA
There are 2 login checks that should be done:
Check login as DBA as user oracle:
#Su - oracle
$sqlplus /nolog
SQL> connect / as sysdba
Check login as Oracle user opc_op, e.g. :
#su - opc_op
$export ORACLE_HOME=…
$$ORACLE_HOME/bin/sqlplus opc_op<passwd of opc_op>@ov_net
1.7.1.9 Trace one Oracle process
You need to find the PID of the process you want to trace (either background or user processes) .
You use SQL to find the SID and the serial# as follow :
Select sid, serial# from v$session where process = <pid>;
You turn on tracing by:
Execute dbms_system.set_sql_trace_in_session(<sid>,<serial#>,TRUE)
You go to the directory specified as the user_dump_dest in init<SID>.ora
(Typicaly $ORACLE_HOME/admin/$ORACLE_SID/udump)
You look for a file <SID>_ora_<PID>.trc
You turn off tracing by :
Execute dbms_system.set_sql_trace_in_session(<sid>,<serial#>,FALSE)
1.7.1.10Trace all Oracle processes
In the init<SID>.ora file, you add the line:
SQL_TRACE = TRUE
You shutdown and restart the database.
You go to the directory specified as the user_dump_dest in init<SID>.ora
(Typicaly $ORACLE_HOME/admin/$ORACLE_SID/udump)
You look for a file <SID>_ora_<PID>.trc
Copyright 2007
Page 17 of 38
1.7.2 What can I do when Oracle does not shutdown?
There are
shutdown
shutdown
shutdown
three ways to shutdown Oracle:
-- normal shutdown, will wait until all Oracle connections are closed
immediate -- will wait until all transactions are complete and then shutdown, even if connections are still open
abort -- will abort the Oracle instance (as if they crashed). This will require instance recovery when staring.
You should first try 'shutdown immediate' :
$sqlplus /nolog
>connect / as sysdba ;
>shutdown
Only if this is not working you might want to use:
>shutdown abort.
After that, please check the following:
- Any oracle processes(oracleopenview) still running (if so kill them)
- Any shared memory/semaphores left (ipcs and ipcrm to clean this)
- Does the file $ORACLE_HOME/dbs/sgadef<oracle_sid>.dbf still exist?
- Though the DB is down? (delete it if the instance is down !)
Note: this file is not used with Oracle 8.1.7 and above
1.8 Tuning the database
OVO ships a file with Oracle database tuning hints:
/opt/OV/ReleaseNotes/opc_db.tuning.
The purpose of this document is to help users improve the performance
of the RDBMS used by VPO. These are general suggestions and guidelines.
While they apply to most installations, there are certainly situations
in which they do not apply. Additional information on DB tuning can be
found in various documents. Some are listed at the end of the opc_db.tuning document.
The information in those documents should be reviewed to fully
understand the pros and cons of the following suggestions.
This document contains some general notes on performance and a RDBMS
specific section.
Copyright 2007
Page 18 of 38
1.9 Tools
1.9.1 /opt/OV/contrib/OpC/opc_odc
The tool usage:
Running /opt/OV/contrib/OpC/opc_odc
The result is detailed in /tmp/opc_odc.log file.
This tool collects information related to the configuration of an Oracle DBMS. Opc_odc collects various information and
performs some checks:
• Get generic system data (OS, memory, swap, disk space…) to see the reason for resource and performance problems
• Check environment variables
• Check configuration files and file permissions
• Check Oracle DBA and opc_op user
• Check connection to the database as connect internal and as opc_op
• Check ITO tables and get number of rows for each table (to see if the problem occurs with a large configuration)
• Print all indexes on the ITO tables (to see if some indexes are missing)
• Show the foreign keys on the ITO tables (to see if a foreign key is missing or creating and causes the problem)
• Show the used blocks for the ITO tables and indexes. This allows seeing if the tables are fragmented and an
opcdbreorg is needed.
• Check SQL*NET : if opc_odc detects problems, it writes an error or a warning to the logfile.
• Contain also the content of several configuration files, Oracle parameters, information about the tablespaces, tables…
The best way to analyze the logfile is to compare it with the output on a “clean” system.
1.9.2 /opt/OV/contrib/OpC/opcdbck
The tool usage:
Opcdbck <opc.checkfile > opcdbck.out
The result is detailed in opcdbck.out file.
The opcdbck tool checks the ITO database for referential integrity. It checks for missing entries in related tables, for no
allowed NULL values and for orphans.
Opcdbck reads its actions from the configuration file opc.checkfile and writes progress messages to stderr and detailed
information to stdout.
Basically, this tool is only useful if you know that you have an inconsistency to get the Ids. It can report problems that are not
really problems.
1.9.3 /opt/OV/bin/OpC/utils/opcchgaddr
The tool usage:
/opt/OV/bin/OpC/utils/opcchgaddr [-force] [-sync] [-label <label>] IP <addr> <name> IP/OTHER <new_addr>
<new_name>
Copyright 2007
Page 19 of 38
(See man page for more detailed)
The opcchgaddr tool is used to change the node address of a node in the database. It can be used if many node addresses
need to be changed or to enforce a certain node address although it is not resolvable.
By default, the opcchgaddr checks that the IP and hostname provided by the name service are consistent with the provided
values. If they are not consistent, it does not change the. If you want to change the node address to a value that is not known
by the name service, you may use the -force option.
1.9.4 /opt/OV/bin/OpC/opcdbidx
The tool usage:
/opt/OV/bin/OpC/opcdbidx -all
/opt/OV/bin/OpC/opcdbidx [-dupl_check|-dupl_fix][-group][-orphan][-flags][-hierarchy]
[-application|-appl_check][-node][-lower][-config]
(See man page for more detailed)
The opcdbidx tool checks and fixes various problems likes:
• Check for and fix duplicates in the opc_node_names
• Check and fix inconsistencies with node_groups
• Remove orphans in opc_node_names
• Set software and configuration flags in database
• Add missing nodes to node hierarchies
• Check for duplicate applications or replace them by links
• Check for node inconsistency
1.9.5 Tfsulock
This tool detects the Oracle locks in the ITO database.
See the SSO document OV-EN004565 for the usage and example.
1.9.6 /opt/OV/bin/OpC/opcdbpwd
This tool is used to set the password of the opc_op database user in the database and to execute database programs that
need the password of the opc_op database user.
The tool usage:
opcdbpwd [<option>] [<program>] [<parm1>] [<parm2>] ...]
where <option> ::= { -set | -var|-exe | -rep } (Abbreviations allowed)
(Default option: -set)
<program> ::= { sqlplus | exp | imp }
<parm1>] [<parm2>] ...] ::= "parameterlist passed to <program>"
PLEASE DO NOT USE ANY ORACLE TOOL TO CHANGE THE PASSWORD of the opc_op user in the Oracle DB.
Copyright 2007
Page 20 of 38
1.9.7 OERR
The tool usage:
Oerr ora <error numer>
This tool describes the Oracle error number , the cause and proposes some actions to solve the problem.
Example:
oerr ora 12154
12154, 00000, "TNS:could not resolve service name"
//*Cause: The service name specified is not defined correctly in the
//TNSNAMES.ORA file.
//*Action: Make the following checks and correct the error:
//Verify that a TNSNAMES.ORA file exists and is in the proper
//place and accessible. See the operating system specific manual
//for details on the required name and location.
//Check to see that the service name exists in one of the
//TNSNAMES.ORA files and add it if necessary.
//Make sure there are no syntax errors anywhere in the file.
//Particularly look for unmatched parentheses or stray characters.
//Any error in a TNSNAMES.ORA file makes it unusable. See
//Chapter 4 in the SQL*Net V2 Administrator's Guide. If
//possible, regenerate the configuration files using the Oracle
//Network Manager.
Copyright 2007
Page 21 of 38
2 Oracle For Openview troubleshooting guide
2.1 Understanding the communication mechanism used by OVO when connecting to the
DB
2.1.1 Introduction
The OVO 7.X installation configures SQL*Net to use IPC Protocol Adapter by default. If this connection method does not
work, a connection via the TCP/IP Protocol Adapter is attempted.
SQL*Net must be configured with any OVO 7.X installation. The default installation is automatically performing this extra step.
SQL*Net is the mechanism used for both inter-node communication as well as for intra-node communication. With ORACLE 8
the connectivity software was renamed to Net8.
ORACLE's connectivity software is based on a technology called Transparent Network Substrate (TNS). TNS provides a
common interface into which you can plug in adapters for different network protocols.
When the communication occurs over a network, a network protocol such as TCP/IP is utilized through an appropriate
Protocol Adapter.
Besides these networks Protocol Adapters ORACLE has created 2 'local' protocols: IPC and BEQ ('Bequeath').
2.1.2 IPC protocol adapter
The IPC adapter is the default local means of communicating over SQL*Net. What this means is that by default SQL*Net V2
will attempt to make connections over IPC first.
The IPC adapter is very similar to the 'Bequeath' protocol adapter in that it uses the same fast, platform specific communication
mechanism. The difference is that it does not require the process spawning mechanism of the 'Bequeath' adapter. It does,
however require that the ORACLE Network Listener process be already running.
The sqlplus example above will show the following in this case:
$ sqlplus opc_op/opc_op@ov_net
# ps -ef|grep ora |sort
oracle 6190 6164 0 17:32:12 pts/0
oracle 6192
1 0 17:32:12 ?
0:00
sqlplus opc_op/opc_op@ov_net
0:00 oracleopenview (LOCAL=NO)
3>
Copyright 2007
Page 22 of 38
2.1.3 TPC protocol adapter
In a multi-node configuration a network Protocol Adapter is needed. In ITO environments this is the TCP/IP Protocol Adapter.
Since TCP/IP allows 'loopbacks' to the same machine this Protocol Adapter can be used for local interprocess communication
as well. However a certain overhead is associated with the network protocol stack.
The sqlplus example from above shows the identical output as in the IPC case:
$ sqlplus opc_op/opc_op@ov_net
# ps -ef|grep ora |sort
oracle 6190 6164 0 17:32:12 pts/0
0:00
sqlplus opc_op/opc_op@ov_net
oracle 6192
1 0 17:32:12 ?
0:00 oracleopenview (LOCAL=NO)
To determine the real communication mechanism the output of lsnrctl status or the listener log file must be checked as follows:
$ORACLE_HOME/bin/lsnrctl status
LSNRCTL for HPUX: Version 10.1.0.4.0 - Production on 27-JAN-2007 10:30:39
Copyright (c) 1991, 2004, Oracle. All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=IPC)(KEY=openview))
STATUS of the LISTENER
-----------------------Alias
LISTENER
Version
TNSLSNR for HPUX: Version 10.1.0.4.0 - Production
Start Date
23-JAN-2007 14:33:26
Uptime
3 days 19 hr. 57 min. 13 sec
Trace Level
admin
Security
ON: Local OS Authentication
SNMP
OFF
Listener Parameter File /etc/listener.ora
Listener Log File
/oracle/product/10g/network/log/listener.log
Listener Trace File
/oracle/product/10g/network/trace/listener.trc
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=openview)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ovstar01.rose.hp.com)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ovstar01.rose.hp.com)(PORT=1526)))
Services Summary...
Service "openview" has 1 instance(s).
Instance "openview", status READY, has 1 handler(s) for this service...
Service "openview.ovstar01" has 1 instance(s).
Instance "openview", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
Copyright 2007
Page 23 of 38
2.1.4 SQL/NET connection
In order to establish a connection to a database via SQL*NET the application has to know a connect descriptor.
A connect descriptor is a specially formatted descriptor of the destination for a network connection. It consists of two parts:
The network route information pointing to the location of the listener through use of a network address,
e.g. address=(protocol=tcp)(host=db_server)(port=1521)
The Oracle SID or service name (Oracle 8i),
e.g. oracle_sid=openview
Connecting by a complete connect descriptor creates a very lengthy connection string, e.g.
ovma_admin/ovms@(descriptor=(address=(protocol=tcp)(host=db_server)(port=1521))(connect_data=(oracle_sid=openview)))
In order to ease handling the connect descriptor is mapped to a shorter net service name for example by the tnsnames.ora file.
2.1.5 SQL*Net/ Net8 Configuration
A set of configuration files like listener.ora, sqlnet.ora, tnsnames.ora are needed with SQL*Net/Net8. These files can be
edited manually or generated by the ORACLE Configuration Tools.
According to the ORACLE documentation SQL*Net will attempt to make an IPC connection first, but will subsequently attempt
the networked connection, resolving the alias in tnsnames.ora.
2.1.6 Installation and configuration of SQL*Net/Net8
The OVO 7.X installation includes an extra step which configures SQL*Net on the ITO management server and starts the
SQL*Net listener. In the /opt/OV/bin/OpC/install/opcsvinfo file the parameter DATABASE will always be set (default:
DATABASE ov_net).
A new configuration script called opcsqlnetconf is provided to create the following files:
/etc/listener.ora
/etc/sqlnet.ora
/etc/tnsnames.ora
/etc/tnsnav.ora.
The files can reside either in /etc, $ORACLE_HOME/network/admin, or $HOME or in a directory pointed to by the
$TNS_ADMIN shell variable.
If the files already exist, opcsqlnetconf asks whether they should be overwritten. The script can be called at any point in time
to reconfigure SQL*Net if this is needed. For details on opcsqnetconf please see the corresponding man page.
Care must be taken if SQL*Net was already configured on this system and the files in /etc should not be overwritten or if the
SQL*Net configuration files reside in a different directory (like $ORACLE_HOME/network/admin, $HOME, etc).
In these special cases both the existing files must be edited and the openview DB settings included into these files.
The configuration file used by a listener is displayed by the command:
$ lsnrctl status
Listener Parameter File /etc/listener.ora
<-------------
Check to see that the SQLNET_ALIAS is correct.
Copyright 2007
Page 24 of 38
Verify the Management Server configuration using the tool ovconfget:
OVOU 8.X:
ovconfget -ovrg server opc DATABASE
ov_net <=== returns SQLNET_ALIAS
OVOU 7.X:
grep DATABASE /opt/OV/bin/OpC/install/opcsvinfo
DATABASE ov_net
Verify the setting in the ovdbconf
grep SQLNET_ALIAS /etc/opt/OV/share/conf/ovdbconf
SQLNET_ALIAS ov_net
Verify in the tnsnames.ora
grep -v ^# $ORACLE_HOME/network/admin/tnsnames.ora | head -3
ov_net = <=====Note SQLNET_ALIAS
(DESCRIPTION=
(ADDRESS_LIST=
Note sometimes the Oracle Administrator will use the TNS_ADMIN variable, if so then that is where
you will need to look for the Oracle network related files in place of the default $ORACLE_HOME/network/admin.
2.2 Troubleshooting SQL*Net/Net8
2.2.1 Starting the listener and testing a client connection
After setting up the Oracle configuration files do the following:
1. Enter a port for the SQL*Net service in the /etc/services file:
listener 1523/tcp
# Oracle SQL*NET
2. Start/Stop/or check the status of the listener process:
switch
lsnrctl
lsnrctl
lsnrctl
to oracle user: su - oracle
start
stop
status
Or as root user issue:
su - oracle -c "$ORACLE_HOME/bin/lsnrctl status"
su - oracle -c "$ORACLE_HOME/bin/lsnrctl start"
su - oracle -c "$ORACLE_HOME/bin/lsnrctl stop"
Copyright 2007
Page 25 of 38
3. Verify the connection (test this from the server and the clients) as user oracle
$ $ORACLE_HOME/bin/sqlplus
>connect system/manager@ov_net
>exit
To manually check this connection, as whatever UNIX user: (call sqlplus and make sure ORACLE_HOME is set appropriately)
$sqlplus
>connect opc_op/<password>@ov_net
>select count(*) from user_tables;
>select count(*) from opc_node_names;
>exit;
Some quick queries to check on the database:
To verify messages counts from the root user prompt you can interrogate the Oracle Database using
the echo command along with the pipe and our opcdbpwd command to pass the select statement.
Example shown below getting the count of active and history messages:
echo "select count(*) from opc_op.opc_act_messages;" | /opt/OV/bin/OpC/opcdbpwd -e sqlplus -s
COUNT(*)
---------30
echo "select count(*) from opc_op.opc_hist_messages;" | /opt/OV/bin/OpC/opcdbpwd -e sqlplus -s
COUNT(*)
---------0
2.2.2 What can I do if the installation fails during the SQL*Net/Net8 configuration?
At the end of opcconfig script, the SQL*Net/Net8 is configured and tested. If this step fails, you can configure
SQL*Net/Net8 manually by editing the SQL*Net/Net8 files or you can call the opcsqlnetconf(1M) utility to configure
SQL*Net/Net8 manually. After this, run opcconfig again, but answer n (no) when asked if you want to configure the
database automatically (the DB configuration is at this point already done).
A problem that we’ve seen several times is that the SQL*Net/Net8 files already exist because the Oracle 8.1.7 installation
places default files in $ORACLE_HOME/network/admin. These default files (sqlnet.ora, listener.ora, tnsnames.ora and
tnsnav.ora) can be removed or renamed and then the SQL*Net/Net8 configuration will work fine.
Copyright 2007
Page 26 of 38
2.2.3 Oracle Networking tracing
Client/Oracle server process tracing and logging details are configured in sqlnet.ora :
LOG_DIRECTORY_CLIENT = <path to log file>
TRACE_DIRECTORY_CLIENT = <path to trace file>
TRACE_LEVEL_CLIENT = <what to trace>
TRACE_LEVEL_CLIENT is USER for tracing client process, ADMIN for tracing oracle server processes, or OFF for no tracing.
Example:
For the listener, configure listener.ora file as follow:
LOG_DIRECTORY_LISTENER = <path to log file>
TRACE_DIRECTORY_LISTENER = <path to trace file>
TRACE_LEVEL_LISTENER = <USER or OFF>
2.3 Tools
2.3.1 tnsping utility
The tmsping utility (usually named tnsping) can be used to determine whether or not a service on a TNS network can be
successfully reached. The service being contacted can be an Oracle database name, an Oracle Names server, or any other
TNS service. If tnsping successfully connects, it will display an estimate of the round trip time to reach the service. If it fails, it
will display a message describing the error that occurred. This allows you to see the network error that is occurring without
the overhead of a database connection. Check the tnsnames.ora for the service name identifier or alias for the server and
database you want to connect to.
tnsping is typically invoked on the command line as follows:
tnsping <address or service name> <count>
The <address> argument can either be a name, in which case it will be resolved in tnsnames.ora or the name service in use,
or an explicit NVstring address. Note that if an NVstring is given, it may need to be quoted in some environments (such as on
UNIX under csh.)
The <count> argument determines how many times the program will attempt to reach the server.
If the address given is a database name, tnsping attempts to contact the corresponding network listener. It does not actually
determine whether or not the database itself is running.
For our example on our Openview Management Server located in the ovdbconf file you can see the value setting for the
alias: SQLNET_ALIAS ov_net
In our tnsnames file we identify that also and thus make the association to the host the database is on, the ORACLE SID,
then PORT (1521 default) used along with the protocol:
ov_net = <============= SQLNET_ALIAS
(DESCRIPTION=
Copyright 2007
Page 27 of 38
)
(ADDRESS_LIST=
(ADDRESS = (PROTOCOL = IPC)(KEY =openview))
(ADDRESS =
(COMMUNITY=OPENVIEW_COMMUNITY)
(PROTOCOL=TCP)
(HOST=ovo3600)
(PORT=1521)
)
)
(CONNECT_DATA=
(SID=openview)
)
1. tnsping connecting to the "ov_net" service or the sqlnet_alias:
tnsping ov_net
TNS Ping Utility for HPUX: Version 10.2.0.2.0 - Production on 27-JAN-2007 16:06:35
Copyright (c) 1997, 2005, Oracle. All rights reserved.
Used parameter files:
/oracle/product/10.2.0/network/admin/sqlnet.ora
Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION= (ADDRESS_LIST= (ADDRESS = (PROTOCOL = IPC)(KEY =openview)) (ADDRESS =
(COMMUNITY=OPENVIEW_COMMUNITY) (PROTOCOL=TCP) (HOST=ovlead1) (PORT=1521))) (CONNECT_DATA=
(SID=openview)))
OK (0 msec)
2. tnsping connecting to the "ov_net" service with a count of 3.
tnsping ov_net 3
TNS Ping Utility for HPUX: Version 10.2.0.2.0 - Production on 27-JAN-2007 16:07:21
Copyright (c) 1997, 2005, Oracle. All rights reserved.
Used parameter files:
/oracle/product/10.2.0/network/admin/sqlnet.ora
Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION= (ADDRESS_LIST= (ADDRESS = (PROTOCOL = IPC)(KEY =openview)) (ADDRESS =
(COMMUNITY=OPENVIEW_COMMUNITY) (PROTOCOL=TCP) (HOST=ovlead1) (PORT=1521))) (CONNECT_DATA=
(SID=openview)))
OK (0 msec)
OK (0 msec)
OK (0 msec)
Copyright 2007
Page 28 of 38
3. tnsping attempting to connect to an invalid name.
tnsping invalid_address_or_alias
TNS Ping Utility for HPUX: Version 10.2.0.2.0 - Production on 27-JAN-2007 16:08:02
Copyright (c) 1997, 2005, Oracle. All rights reserved.
Used parameter files:
/oracle/product/10.2.0/network/admin/sqlnet.ora
TNS-03505: Failed to resolve name
4. tnsping attempting to connect to a name that's valid, but resolves to
an address where no server is located.
% tnsping testing
TNS Ping Utility for HPUX: Version 2.3.4.0.0 - Production on 27-OCT-98 09:33:44
Copyright (c) Oracle Corporation 1995. All rights reserved.
Attempting to contact (ADDRESS=(PROTOCOL=tcp)(HOST=spot)(PORT=1599))
TNS-12541: TNS:no listener
2.3.2 TRCROUTE utility
The Trace Route Utility (TRCROUTE) enables administrators to discover what path or route a connection is taking from a client
to a server.
If TRCROUTE encounters a problem, it returns an error stack to the client instead of a single error. These additional error
messages make troubleshooting easier.
TRCROUTE is different from TNSPING in that it travels as a special type of connect packet, and is routed as such. As it travels
toward its destination, the TRCROUTE connect packet collects the TNS addresses of every node it travels through. If an error
occurs, TRCROUTE collects error information that shows where the error occurred. The Trace Route Utility displays the
information collected on the client screen. You can redirect the TRCROUTE output to a file, and print it if you wish.
•
Requirements
Trace Route works only over Net8 and SQL*Net version 2.3(ORACLE 7.3) and later. Every node along the route from client to
server must use SQL*Net version 2.3 or later. If a pre-2.3 node is on the path, the following error is displayed:
TNS-03603: Encountered a node with pre-2.3 version of SQL*Net
TRCROUTE shows what node along the path is responsible for any errors.
•
Effect on Performance
The Trace Route Utility uses minimal resources. It gathers information in the connect data of a special connect packet; standard
connect packets are not affected.
The server is not affected by TRCROUTE. The listener receives a processes the TRCROUTE connect packet. It returns the
information to the client by putting it into a refuse packet. The server does not need to start up any new processes or deal with
dummy connections.
Starting the Trace Route Utility
To invoke TRCROUTE, type the following from the command line:
Copyright 2007
Page 29 of 38
trcroute service_name
Examples of Trace Route Output
* A successful Trace Route packet that traveled from a client to a listener
* An unsuccessful Trace Route packet that could not reach the listener
because the listener was not up. Successful Trace Route
$ trcroute ov_net
Trace Route Utility for HPUX: Version 2.3.4.0.0 - Production 26-OCT-98 16:58:59
Copyright (c) Oracle Corporation 1995. All rights reserved.
Route of TrcRoute:
-----------------Node: Client
Time and address of entry into node:
------------------------------------------------------------26-OCT-98 16:58:59 ADDRESS= COMMUNITY=OPENVIEW_COMMUNITY PROTOCOL=TCP
HOST=bigbaan.bbn.hp.com PORT=1523
Node: Server
Time and address of entry into node:
------------------------------------------------------------26-OCT-98 16:58:59
Trace Route with Error
~~~~~~~~~~~~~~~~~~~~~~
$ trcroute ov_net
Trace Route Utility for HPUX:
Version 2.3.4.0.0 - Production on 27-OCT-98 10:04:16
Copyright (c) Oracle Corporation 1995. All rights reserved.
Route of TrcRoute:
-----------------Node: Client
Time and address of entry into node:
------------------------------------------------------------27-OCT-98 10:04:16 ADDRESS= COMMUNITY=OPENVIEW_COMMUNITY PROTOCOL=TCP
HOST=bigbaan.bbn.hp.com PORT=1523
TNS-12203: TNS:unable to connect to destination
TNS-12541: TNS:no listener
TNS-12560: TNS:protocol adapter error
TNS-03601: Failed in route information collection
Copyright 2007
Page 30 of 38
3 OVO Server
3.1 Maintenance
3.1.1 Backup of OVO Data
OVO offers several ways to backup the configuration. All these commands are described in detail in the man pages.
3.1.1.1 Offline Backup - opc_backup/opc_recover
opc_backup shuts down all OVO and Oracle Processes and performs an offline backup. It is possible to backup only
configuration data or to backup the configuration data and the OpenView binaries. opc_backup is by default interactive.
Since VPO 6, it is possible to call opc_backup non-interactive and therefore have scheduled backups.
opc_recover allows to restore a backup that has been taken with opc_backup. It is always interactive.
3.1.1.2 Automated Backup - ovbackup.ovpl/ovrestore.ovpl
ITO integrates its own backup and restore scripts with those provided by Network Node Manager (NNM),
ovbackup.ovpl(1M), and ovrestore.ovpl(1M) in order to carry out a complete automated backup of the database while the ITO
GUI and server processes are running. Some activities are not possible during the backup because the NNM server processes
are paused. Therefore it is not possible to work on OVW submaps. But the OVO server processes remain fully operational.
Therefore it is still possible to receive messages and work on them. This automated backup is not interactive and may be run
with a cron job or an ITO scheduled action.
3.1.1.3 Configuration Download - opccfgdwn/opccfgupld
The OVO configuration download is not directly a backup method, but it can be used to backup most of the OVO
configuration data. This does not include NNM configuration or map customizations in the ovw maps. The configuration
download can be performed while the GUI and server processes are running. However, the administrator should not make
configuration changes while the data is downloaded. Through modifying the index file, it is possible to selectively restore only
parts of the data.
Copyright 2007
Page 31 of 38
3.1.2 How to check the time elapsed for a history download?
In the 'Database Maintenance' of the Admin GUI there is an option to send a notification for history downloads:
1- If this option is enabled, whenever the history download is started and completed it writes a message to the opcerror file on
the management server and to the ITO browser.
2- If this option is NOT enabled, the start time of a history download can be looked up from the 'Database Window' or from
the target download file name. Once the start time is known, the download time can be found by looking at the time the
history download file was accessed.
For example:
-rw-rw-rw- 1 root sys 3056010 Apr 26 08:56 h970326.03000
In this example the history download started at 3am on Apr 26 and completed at 8:56am.
3.1.3 How to check how many messages were downloaded?
Enter the following command as user root:
grep “===” h970326.030000 | wc -l
This command returns the number of messages downloaded.
3.2 Troubleshooting
3.2.1 Creation and configuration of the database during the OVO installation
After Oracle and the OVO binaries have been installed, then opcconfig needs to be executed. This will configure OVO, and
create the tables in the Oracle database.
The OVO processes are as follow:
Copyright 2007
Page 32 of 38
opcconfig
opcdbsetup
ovdbsetup1. sh
- createopenviewDB
- createovdbconffile
ovdbsetup2. sh
- create ITOtablespace
- create ITOsDB users
- create
cfg configfile
oracle_dbf.
- create systemtablespace
- configure DB s
opcdbinst
- create ITO
tables
opcdbinit
- load initial
configuration
opcsqlnetconf
- configuresql*net/net
8
-
If opcconfig is done, then install one or more nodes, and do the configuration within OVO. Once done it is advised to do an
opccfgdwn. If for whatever reason opcconfig is needed again, then some time can be saved by uploading the configuration
with opccdfupld.
If opcconfig fails during the phase of the database configuration, it is best to start from the beginning or set up at the last
failed step.
If a problem occurs during the creation of the core database (ovdbsetupo1.sh) or the configuration of the core database
(ovdbsetupo2.sh), it is usually the best, to completely drop the database using opcdbsetup –d, then fix the problem and start
again with the configuration.
We have seen the creation of the core database fail for several reasons:
 Wrong value of environment variables (NLS_LANG, ORACLE_HOME, etc.)
 No disk space
 Insufficient permissions
Once the core database is created and configured, opcconfig can be called again after a problem occurred and was fixed.
Opcconfig will continue where it stopped. You will get warnings that certain database objects are already created, but
opcconfig will continue. You can ignore these warnings. If some database tables have already initial configuration when
opcconfig wants to load the initial configuration, you can choose whether you want to clear the database and load the initial
configuration again or if you want to abort (for example if the database has already been successfully initialized before).
Very at the end of opcconfig, the SQL*Net/Net8 is configured and tested. If this step fails, you can configure SQL*Net/Net8
manually by editing the SQL*Net/Net8 files or you can call the opcsqlnetconf(1M) utility to configure SQL*Net/Net8
manually. After this, run opcconfig again, but answer n (no) when asked if you want to configure the database automatically
(the DB configuration is at this point already done).
Example:
Copyright 2007
Page 33 of 38
The installation fails and shows the following errors :
Database: ORA-12154: TNS:could not resolve service name
(OpC50-15)
Could not connect to database ov_net.
Please look if the database processes are running. (OpC50-2)
This error is described in the following SSO documents: OV-EN009932, OV-EN008235
3.2.2 OVO automated backup
The automated backup will write it’s output to the logfile /var/opt/OV/tmp/ovbackup.log.
To get a more verbose output of the OVO integration, you can export following environment variables before calling the
automated backup
export VERBOSE=1
export TRACE=1
VERBOSE will produce high level information, TRACE will produce very detailed information.
3.2.3 opccfgdwn/opccfgupld
Here are the more common problems:
 A problem that occurs sometimes when using opccfgupld is that the agent platform of an uploaded node or node
default is missing. In this case, the node or node default is skipped (which in turn may produce other problems).
Depending on the installed ITO bundle (ITOEngOraAll, ITOEngOraMin), you have different agent platforms on the
management server. It is also possible, that an agent platform that was supported by previous ITO releases is
obsolete. There are following ways to solve this problem:
o Add the missing AgentPlatform and load the agent platform information into the ITO database, then re-run
opccfgupld.
o Download the configuration again, but exclude the node with the missing platform and upload this new
configuration again.
o (Temporary) change the platform of the node in the ITO node bank on the system where you performed the
download, download the data again and upload it on the target management server again.

Similar to missing platforms, another missing object that is referenced by an uploaded element may produce an error
and the object is skipped. For example, an instruction interface cannot be uploaded if the node on which the
instruction interface shall run is not in the node bank or a template assignment is skipped because the template is not
in the database.This problem can be fixed by either adding the missing object manually or by downloading the data
again including the missing object.If you upload many operators with a big node hierarchy, opccfgupld may fail
because it runs out of memory. It then sometimes aborts with an internal error. This problem is mainly an ITO 4
problem because ITO 5 uses operator independent node hierarchies. There are two ways how to fix this problem:
o opccfgupld uploads the user bank in one transaction and cumulates all memory within the transaction. If
selected users are uploaded, each user is performed in a separate transaction and less memory is cumulated.
Copyright 2007
Page 34 of 38
o
Instead of downloading the complete user bank, select all users and download selected users and upload this
new configuration.
Increase the value of the maxdsize and maxssize kernel parameters and perform the upload again
3.2.4 SQL tracing in OVO
There are two SQL traces available:
- The ITO SQL trace
This trace uses that it is possible to get the current SQL statement. It is in the central ITO routine that checks the SQL return
code which is usually called after every SQL command. Unfortunately not all statements appear in the used Oracle structure.
This SQL trace can be used to see the order of statements performed and what return code they had.
- The Oracle SQL traces
Oracle offers a SQL trace that allows collecting timing information and execution plan information. This trace shows how often
a SQL statement was executed and the total used time. Therefore this trace can be used to find performance problems. It does
however not show the exact order of the statements.
3.2.4.1 ITO SQL Trace (VPO >= A.06.07)
The ITO SQL Trace dumps most of the SQL statements in the timely order of their occurrence including the result (Error, no
row). It is very useful to isolate the location of a problem (what was the last SQL statement before aborting).
Starting with VPO 6.07, the ITO SQL Trace is handled as any other ITO trace. It is enabled by using the DEBUG area SQL.
3.2.4.2 ITO SQL Trace (VPO < A.06.07)
The ITO SQL Trace dumps most of the SQL statements in the timely order of their occurrence including the result (Error, no
row). This trace has no timing information. It is very useful to isolate the location of a problem (what was the last SQL
statement before aborting).
When enabled, it creates for each ITO program that accesses the database a file named process_name.sqltrc.
If such a file already exists, a number is appended to make the name unique.
These are the steps to use this trace:
1. Add following parameter to the opcsvinfo file:
DB_TRACE_FLAG TRUE
2. The default directory is /var/opt/OV/share/tmp/OpC/mgmt_sv.
If you want to use a different directory, add a line like the following to opcsvinfo file:
DB_TRACE_DIR /tmp
Copyright 2007
Page 35 of 38
3. (Re-)start the process(es) that you are interested in.
This generates files like below:
-rw-rw-rw- 1 root
-rw-rw-rw- 1 root
-rw-rw-rw- 1 root
sys
sys
sys
10311 Oct 22 10:43 opcdispm.sqltrc
28650 Oct 22 10:43 opcdistm.sqltrc
10228 Oct 22 10:43 opcforwm.sqltrc
NOTE: By default, this tracing does not work with Oracle 8.0.5. The function sqlglst() that is used with the ITO SQL Trace
is missing in the shared library. You may perform following steps to re-create an Oracle shared library that includes this
function:
- Edit $ORACLE_HOME/precomp/admin/shrept.lst, add a line for sqlglst:
precomp : sqlglst
- Re-build the Oracle shared library as user oracle:
$ORACLE_HOME/bin/genclntsh
3.2.4.3 Oracle trace
Oracle offers a SQL trace that allows collecting timing information and execution plan information. This trace shows how often
a SQL statement was executed and the total used time. Therefore this trace can be used to find performance problems. It does
however not show the exact order of the statements.
To activate the Oracle trace, perform following steps:
- Stop all ITO processes: /opt/OV/bin/ovstop
- shutdown Oracle
- Add following lines to $ORACLE_HOME/dbs/init${ORACLE_SID}.ora:
sql_trace = true
timed_statistics = true
- restart Oracle
- Create explain plan table as DB user opc_op:
sqlplus /nolog
connect / as opc_op;
@?/rdbms/admin/utlxplan.sql
exit
- Each process that connects to Oracle creates a trace file in the Oracle user dump directory that is named like ora_15272.trc.
It is difficult to find out which file belongs to which process. So, if you have a tool that does not need the server processes
running, only start this tool. If you need to start the server and GUI processes, start everything, delete all trace files, start the
process that you want to trace and the created trace file is from this process.
- You need to process the generated trace file to get readable information, for example:
tkprof ora_15272.trc opc.tk sys=no explain=opc_op/opc_op
Copyright 2007
Page 36 of 38
3.2.5 Oracle runs, but OVO doesn’t connect
In the opcerror file on the server, you see errors like:
Database : (OpC50 -15) Could not connect to database ov_net .
Please look if the database processes are running. (OpC50 -2)
can not connect to database ( OpC40 -557)
1. Check if the Oracle processes are running:
# ps -ef |grep oracle
oracle 19606
1 0 May 28 ?
0:46 ora_ckpt_openview
oracle 19604
1 0 May 28 ?
0:13 ora_lgwr_openview
oracle 19608
1 0 May 28 ?
0:01 ora_smon_openview
oracle 19602
1 0 May 28 ?
0:14 ora_dbw0_openview
oracle 19600
1 0 May 28 ?
0:17 ora_pmon_openview
oracle 19558
1 0 May 28 ?
0:00 /opt/oracle/product/8.1.7/bin/tnslsnr LISTENER -inherit
oracle 19610
1 0 May 28 ?
0:00 ora_reco_openview
2. Check if Oracle environment variables are set
(if they are not set, that's OK - the values
from ovdbconf will be used).
3. Check the /etc/opt/OV/share/conf/ovdbconf file
4. Enable tracing:
OPC_TRACE TRUE
OPC_TRACE_TRUNC FALSE
OPC_TRACE_AREA ALL,DEBUG
OPC_TRC_PROCS opc_dflt_lang
OPC_DBG_PROCS opc_dflt_lang
4. Run opc_dflt_lang
# /opt/OV/bin/OpC/install/opc_dflt_lang 2>&1 | tee /tmp/opc_dflt_lang.log
5. Gather opcerror, trace, opcsvinfo, ovdbconf and opc_dflt_lang.log
3.3 Tools
3.3.1 /opt/OV/contrib/OpC/itochecker
The tool usage:
Running /opt/Ov/contrib/OpC/itochecker [ -1234567 [-l] [-t] [-a] [-m] [-s nodename] ]
Copyright 2007
Page 37 of 38
Itochecker is a tool, which is designed to describe some basic configuration data for an OpenView VPO management server.
Further this tool attempts to identify configuration items, which may cause problems in the use of ITO.
This script is only a report tool. It doesn't make any changes on the system it checks. Only report files will be created in the
output directory, which is set to /tmp/ito_rpt by default.
Options related to the Database are the following:
-1 System Environment Check:
Domain name, whether DNS is in use, number of lines in /etc/hosts, whether nslookup is working for user opc_op,
Server Memory, Swap space, and product versions, Number of processes, DCE status and patchlevel, if portmapper is
running, Installed OS patches and patchbundles, if required patches are installed.
-2 VPO environment check:
Installed OVO bundles, OVO and ECS designer versions, what processes are running on the server, Server kernel
parameters and whether the requirements are met, installed OVO and NNM patches, version and patchlevel of all OVO
binary and library files, Disk space in DB and OVO directories, Pending agent data in the distribution directory, Decoded
agent configuration files, cluster information, core file information, opcsvinfo/opcinfo and opcerror files, elements in server
and agent queues, file permissions of some critical files.
-3 General Database Check:
Database version, How many messages are in the active and history message tables, whether all node group /
message group pairs are assigned to some operator, test the number of files in the audit directory of Oracle (a large number
of files in this directory can cause the monitor mondbfile to fail).
-4 Database / Node Verification:
Compares agent entries in the database with the file system, gets the free disk space in the oracle directory.
Copyright 2007
Page 38 of 38