Download LBTO software startup/shutdown and troubleshooting

LBTO software
startup/shutdown
and troubleshooting
July 18, 2006
Chris Biddick
1 cjb
TCS servers







2 cjb
lbtmu01 (TO station)
lbtmu102
lbtmu103
lbtmu104
lbtmu105
lbtmu107
jet (special)
Start TCS networks


netconfig start (on all servers, but not jet)
•
•
•
networkserver
rpcserver
gshmserver
subsystem manager
RPC manager
reflective memory manager
netconfig start passive (TO station only)
prevents subsystems from running here
3 cjb
Start MCSPU on jet



User telescope
rpcconfig start 10.144.0.20 [passive]
./gotemp (used)
Will start mcstemp (most recent control program) and the
engineering interface program mcsDisplay.

./go
(not used)
Will start mcsworking (stable control program) and the
engineering interface program
4 cjb
Start subsystems

netconfig start [left | right] SUB
Starts subsystem SUB on lightest load server

netconfig start [left | right] SUB on address
Starts subsystem SUB on server ‘address’ – name or IP address

netconfig start [left | right] SUB on local
Starts subsystem SUB attached to local terminal

5 cjb
Alternate side notation:
• left SUB
SUBL
• right SUB
SUBR
Start subsystems

When subsystems are started normally

When started ‘on local’
• stdout is directed to /dev/null
• stderr is available (but not much use)
• both stdout and stderr are available and
attached to controlling terminal
6 cjb
Current subsystems








7 cjb
LSS
ECS
CSQ
PMC left
PSF left
MCS
PCS
SDB
Logging control
Enclosure control
IIF receiver
Primary mirror control
Active optics control
Mount control
Pointing control
Star database
Start GUIs




8 cjb
GUIs are normal executables installed in
a system path
ECSGUI [READ_WRITE]
ECSGUI [READ_WRITE] &
For sided subsystems add ‘left’ or ‘right’
as argument: PSFGUI left
Current GUIs









9 cjb
ECSGUI [READ_WRITE]
PMCGUI {left | right}
PSFGUI {left | right}
MCSGUI (old)
mcsgui (OpenGL)
PCSGUI (old)
pcsgui (OpenGL)
presetgui (OpenGL)
WXGUI
TO station start buttons









10 cjb
ECSGUI
MATGUI
MCSGUI
PCSGUI
PMCGUI
PRESETGUI
PSFGUI
TELGUI
WXGUI
TEL, ECS, ECSGUI READ_WRITE
CSQ, MATGUI
MCS, mcsgui (OpenGL)
PCS, pcsgui (OpenGL)
TEL, LSS, left PMC, PMCGUI left
PCS, MCS, SDB, presetgui
TEL, LSS, left PSF, PSFGUI left
TEL, TELGUI
ECS, WXGUI
Stop GUIs

11 cjb
Use ‘File exit’ option or title bar ‘X’
Stop subsystems

netconfig stop [left | right] SUB
Stops subsystem SUB – times out in seven seconds

netconfig stop all
Stops all subsystems

netconfig kill [left | right] SUB
Sends SIGKILL (9) instead of SIGINT (2)

12 cjb
If GUI is left running when SUB stopped
GUI should turn red
TO station stop buttons

13 cjb
TCSstop
all subsystems
Stop MCSPU on jet


14 cjb
Command ‘stop’ in engineering interface
will stop just the interface
Command ‘stopall’ will stop both the
engineering interface and mcstemp
Stop TCS networks

netconfig stop
• Stop all TCS network processes on local
•
15 cjb
(on each server)
server
You may stop TCS networks without stopping
subsystems – network state is kept in files on
disk
TCS state files

LBT.conf
Overall configuration parameters in /usr/commonsw/NetConfig

rpc_IPAddr.conf
RPC functions and aliases on local server in /var/state/LBT

nameprocs.conf
Subsystem running on local server in /var/state/LBT

LSSEventClientList.dat
List of event callbacks on local server in /var/state/LBT
16 cjb
Troubleshooting - netconfig

netconfig ps

netconfig –l
• Shows all TCS processes on local server
• Shows all known TCS subsystems on all
servers

netconfig –s
• Shows all servers with subsystem count and
load
17 cjb
Troubleshooting - rpcconfig





18 cjb
rpcconfig stop [address]
•
Stops rpcserver on ‘address’
rpcconfig start address [cold] [passive]
•
•
•
Starts rpcserver on ‘address’
‘cold’ skips reading state file
‘passive’ disables rpc health check
•
Shows number of registered functions and aliases on local server
•
Lists all functions on server address
•
Lists all aliases on server address
rpcconfig –l
rpcconfig –f [address]
rpcconfig –a [address]
Troubleshooting - gshmconfig




19 cjb
gshmconfig stop
•
Stop gshmserver on local server
gshmconfig start
•
Start gshmserver on local server
gshmconfig –b
•
Shows reflective memory IP broadcast address on
local server
gshmconfig –l
•
•
Lists segment key, size, and all servers currently on
the reflective memory network
Repeat and check that all servers stay at same ID
Troubleshooting - monitor

monitor –o
• Shows computers as they join or drop off the
•
20 cjb
shared memory network
Shows restart requests
Troubleshooting - syslog


21 cjb
/var/log/messages on local server
‘tcslog’ will show messages with LBT
designation as they occur
Troubleshooting - DDViewerGUI



22 cjb
DDViewerGUI shows hierarchical view of
reflective memory on local server
Available on every server and TO station
Updates about every two seconds
Troubleshooting - Events

No LSS - XML files on local server

LSS - MySql database
• /tmp/SUB.log
• mysql –u lss –p –h mysql LBT_log
• password lss
• select date,description from Event where
(date like ‘%Jul 18%’ and description like
‘%left PSF%’);
23 cjb
Mountain subsystem debugging






24 cjb
Use your account on a server
Get current CVS tree
Modify subsystem code
Build and install (may require removal of
existing package)
netconfig stop SUB
netconfig start SUB on server
Mountain GUI debugging





25 cjb
Use your account on a server or TO
station
Get current CVS tree
Modify GUI code
Build GUI (install not needed)
Run GUI directly