LBTO software startup/shutdown and troubleshooting

1. LBTO software startup/shutdownand troubleshooting July 18, 2006 Chris Biddick 1 cjb

2. TCS servers • lbtmu01 (TO station) • lbtmu102 • lbtmu103 • lbtmu104 • lbtmu105 • lbtmu107 • jet (special) 2 cjb

3. Start TCS networks • netconfig start (on all servers, but not jet) • networkserver subsystem manager • rpcserver RPC manager • gshmserver reflective memory manager • netconfig start passive (TO station only) prevents subsystems from running here 3 cjb

4. Start MCSPU on jet • User telescope • rpcconfig start 10.144.0.20 [passive] • ./gotemp (used) Will start mcstemp (most recent control program) and the engineering interface program mcsDisplay. • ./go (not used) Will start mcsworking (stable control program) and the engineering interface program 4 cjb

5. Start subsystems • netconfig start [left | right] SUB Starts subsystem SUB on lightest load server • netconfig start [left | right] SUB on address Starts subsystem SUB on server ‘address’ – name or IP address • netconfig start [left | right] SUB on local Starts subsystem SUB attached to local terminal • Alternate side notation: • left SUBSUBL • right SUBSUBR 5 cjb

6. Start subsystems • When subsystems are started normally • stdout is directed to /dev/null • stderr is available (but not much use) • When started ‘on local’ • both stdout and stderr are available and attached to controlling terminal 6 cjb

7. Current subsystems • LSS Logging control • ECS Enclosure control • CSQ IIF receiver • PMC left Primary mirror control • PSF left Active optics control • MCS Mount control • PCS Pointing control • SDB Star database 7 cjb

8. Start GUIs • GUIs are normal executables installed in a system path • ECSGUI [READ_WRITE] • ECSGUI [READ_WRITE] & • For sided subsystems add ‘left’ or ‘right’ as argument: PSFGUI left 8 cjb

9. Current GUIs • ECSGUI [READ_WRITE] • PMCGUI {left | right} • PSFGUI {left | right} • MCSGUI (old) • mcsgui (OpenGL) • PCSGUI (old) • pcsgui (OpenGL) • presetgui (OpenGL) • WXGUI 9 cjb

10. TO station start buttons • ECSGUI TEL, ECS, ECSGUI READ_WRITE • MATGUI CSQ, MATGUI • MCSGUI MCS, mcsgui (OpenGL) • PCSGUI PCS, pcsgui (OpenGL) • PMCGUI TEL, LSS, left PMC, PMCGUI left • PRESETGUI PCS, MCS, SDB, presetgui • PSFGUI TEL, LSS, left PSF, PSFGUI left • TELGUI TEL, TELGUI • WXGUI ECS, WXGUI 10 cjb

11. Stop GUIs • Use ‘File exit’ option or title bar ‘X’ 11 cjb

12. Stop subsystems • netconfig stop [left | right] SUB Stops subsystem SUB – times out in seven seconds • netconfig stop all Stops all subsystems • netconfig kill [left | right] SUB Sends SIGKILL (9) instead of SIGINT (2) • If GUI is left running when SUB stopped GUI should turn red 12 cjb

13. TO station stop buttons • TCSstop all subsystems 13 cjb

14. Stop MCSPU on jet • Command ‘stop’ in engineering interface will stop just the interface • Command ‘stopall’ will stop both the engineering interface and mcstemp 14 cjb

15. Stop TCS networks • netconfig stop (on each server) • Stop all TCS network processes on local server • You may stop TCS networks without stopping subsystems – network state is kept in files on disk 15 cjb

16. TCS state files • LBT.conf Overall configuration parameters in /usr/commonsw/NetConfig • rpc_IPAddr.conf RPC functions and aliases on local server in /var/state/LBT • nameprocs.conf Subsystem running on local server in /var/state/LBT • LSSEventClientList.dat List of event callbacks on local server in /var/state/LBT 16 cjb

17. Troubleshooting - netconfig • netconfig ps • Shows all TCS processes on local server • netconfig –l • Shows all known TCS subsystems on all servers • netconfig –s • Shows all servers with subsystem count and load 17 cjb

18. Troubleshooting - rpcconfig • rpcconfig stop [address] • Stops rpcserver on ‘address’ • rpcconfig start address [cold] [passive] • Starts rpcserver on ‘address’ • ‘cold’ skips reading state file • ‘passive’ disables rpc health check • rpcconfig –l • Shows number of registered functions and aliases on local server • rpcconfig –f [address] • Lists all functions on server address • rpcconfig –a [address] • Lists all aliases on server address 18 cjb

19. Troubleshooting - gshmconfig • gshmconfig stop • Stop gshmserver on local server • gshmconfig start • Start gshmserver on local server • gshmconfig –b • Shows reflective memory IP broadcast address on local server • gshmconfig –l • Lists segment key, size, and all servers currently on the reflective memory network • Repeat and check that all servers stay at same ID 19 cjb

20. Troubleshooting - monitor • monitor –o • Shows computers as they join or drop off the shared memory network • Shows restart requests 20 cjb

21. Troubleshooting - syslog • /var/log/messages on local server • ‘tcslog’ will show messages with LBT designation as they occur 21 cjb

22. Troubleshooting - DDViewerGUI • DDViewerGUI shows hierarchical view of reflective memory on local server • Available on every server and TO station • Updates about every two seconds 22 cjb

23. Troubleshooting - Events • No LSS - XML files on local server • /tmp/SUB.log • LSS - MySql database • mysql –u lss –p –h mysql LBT_log • password lss • select date,description from Event where (date like ‘%Jul 18%’ and description like ‘%left PSF%’); 23 cjb

24. Mountain subsystem debugging • Use your account on a server • Get current CVS tree • Modify subsystem code • Build and install (may require removal of existing package) • netconfig stop SUB • netconfig start SUB on server 24 cjb

25. Mountain GUI debugging • Use your account on a server or TO station • Get current CVS tree • Modify GUI code • Build GUI (install not needed) • Run GUI directly 25 cjb