TelePresence Microscopy at CSIRO Minerals

1. TelePresence Microscopy at CSIRO Minerals Colin MacRae CSIRO Minerals - Microbeam Laboratory www.minerals.csiro.au/microbeamlab

2. Talk outline • What and why? • History of TelePresence at CSIRO Minerals • 3rd generation TelePresence system • System architecture • Control interfaces • Video streaming technology • Security features • Administration • Modular installation of new functions • Conclusions, where to from here? Remote Access and automation Workshop 2005

3. What is TelePresence? • Tele-conferencing • Video + Audio • Tele-control • Microscope controls • On-line data collection + analysis • Tele-control is provided by some manufacturers • Ideally, TelePresence should be: • Interoperable with many platforms • ‘Shrink-wrapped’ software • Looks professional, with easy to use interfaces • Fast and responsive • Stable and secure Remote Access and automation Workshop 2005

4. Why do we want TelePresence? • Internal clients (Brisbane, Hobart, Perth) • External clients (Melbourne, Sydney, Brisbane) • For analyses, our clients must either: • Mail / courier samples, and wait for our report (slow) • Travel to our laboratory (expensive) • TelePresence can help • Provide results to client in real-time, and also allows instant feedback from client • Attract new clients CSIRO Minerals CSIRO Minerals, Exploration & Mining BHP-Billiton CSIRO Marine Research Comalco & RioTinto Remote Access and automation Workshop 2005

5. What do we want to connect? • Scanning Electron Microscopes • Electron Microprobes & Proton Microprobe • X-ray detectors –EDS & WDS • Cathodoluminescence systems • Optical microscopes Remote Access and automation Workshop 2005

6. What do we want to offer • Clear fast images • On-line microanalysis • On-line x-ray spectra • Large area optical views • Streaming maps • Security • Voice over IP • Data storage • Data analysis/presentation software Remote Access and automation Workshop 2005

7. What do we need from TelePresence? • Provision of: • Real-time video from multiple sources (SEM, room camera) • Streaming EDS spectra and other data ( + controls) • Control of microscope stage + column (lower priority) • Minimal hardware requirements • PC or laptop (no workstations) • ADSL or cable modem, ~1Mbit/s (not Internet2, GRID) • ‘Shrink-wrap’ software • Easy to set-up • Very easy to learn and use, with generic controls/interface • Very, very stable • Extraordinary security • Low maintenance • Must not affect normal operation of instruments! Remote Access and automation Workshop 2005

8. TelePresence history TelePresence Microscopy 1999 • Nestor J. Zaluzec, Argonne National Laboratory • Web-based, ‘pervasive’ • Web based notebook • Video with “GTS” streaming technology Remote Access and automation Workshop 2005

9. TelePresence history TelePresence Microscopy 1999 • Nestor J. Zaluzec, Argonne National Laboratory • Web-based, ‘pervasive’ Java system 2001 • Platform independent, ‘safe’ code • MJPEG video (TCP-IP) • Web based notebook with embedded Java applets for Images, Spectra and maps • Successful, but limited Remote Access and automation Workshop 2005

10. TelePresence history TelePresence Microscopy - 1999 • Nestor J. Zaluzec, Argonne National Laboratory • Web-based, ‘pervasive’ Java system - 2001 • Platform independent, ‘safe’ code • MJPEG video • Web based notebook with embedded Java applets for Images, Spectra and maps • Successful, but limited C++ system – 2004  • Widely used language, with broad library support • MPEG-4 video compression in software, changeable video size • New, secure architecture Remote Access and automation Workshop 2005

11. Internet protocols: UDP versus TCP Sender Router TCP packet acknowledgement TCP Lost packet? UDP packet UDP TCP = Transport Control Protocol UDP = User Datagram Protocol Remote Access and automation Workshop 2005

12. Video latency • Definition: • Latency = time displayed – time captured • What do we want? • <500ms for focussing (or better) • <200ms preferred • Causes of latency • Video compression / decompression • Hardware accelerated rendering (done) • Hardware compression (cheap MPEG-4 hardware compression coming soon) • Router delays (congestion) • Ultra-broadband internet (eg GRID) • Reduce stream size (better codec) • inter-router delays (speed / distance) • More bandwidth wont help • Use faster internet protocol (i.e. TCP  UDP) Remote Access and automation Workshop 2005

13. Another real life problem : The Firewall Firewall Firewall Internet Client network Our network port 8080 (UDP), from specified addresses Server Client Video port 8080 (TCP) Host ‘DMZ’ Remote Access and automation Workshop 2005

14. Functionality Low Moderate High Execution speed Low High High Development pain Low High Moderate The vibe Poor So so Yeah groovy Platform independent? Yes Yes No Trusted? Somewhat Yes Yes, with security Comparison of platforms Web browser Java C++ Remote Access and automation Workshop 2005

15. JEOL 8900R Electron Microprobe FEI Quanta FEG-Environmental SEM Hitachi S-5000 In-Lens Cold FEG SEM PIXE Proton Microprobe System architecture Password Client Host Server (TCP socket) commands & data Server Video (UDP stream) Server Generic interface Instrument specific Drivers Remote Access and automation Workshop 2005

16. Network transmission sockets Timing system (frame-rate) Video capture Compression codec DirectShow video streaming ‘GraphEdit’ component of Microsoft DirectX software development kit (SDK) Remote Access and automation Workshop 2005

17. DirectShow video streaming • Very easy to configure and re-configure ‘graphs’ • Other stream designs are possible, eg: • Saving stream to HDD • Motion detection triggering • Adaptable to as yet unforseen requirements • All components are upgradeable and replaceable • New codecs can be adopted as they become available • ‘Future proof’ • We didn’t have to write any of these components (only wire them together) • Some DirectShow components are hardware-accelerated • Fast video rendering Remote Access and automation Workshop 2005

18. TelePresence interfaces • Video • Camera control Tilt-pan-zoom controls Used in conjunction with video window • EDX spectrometers Generic, easy to use interface Fast (2-5 updates/s) • Text streams • Miscellaneous others • Stage and Electron optical System Remote Access and automation Workshop 2005

19. Themeda Australis Kangaroo grass Variations in humidity cause grass to spiral and “dig” into the soil Video streaming technology • Microsoft DirectShow • Widely supported standard with pre-compiled modules • Hardware-accelerated rendering • MPEG-4 encoding • 320  240 at 25 fps is ~500kbit/s (~ADSL) • (~12 Mbit/s by MJPEG) • (~59 Mbit/s for raw video) • UDP transmission • Lowest possible latency over IP (~200ms) Remote Access and automation Workshop 2005

20. Video cameras • Canon VC-C1 and VC-C3 cameras • 450 x 350 pixels at 30 frames/s. • ±90º pan, ±25º tilt, 10 zoom • Good low-light performance Remote Access and automation Workshop 2005

21. Bench-top camera • ‘ELMO’ camera • XGA video, 20 frames/s • www.elmousa.com/presentation Remote Access and automation Workshop 2005

22. Client feedback • Problem: Video stream is one-way. How do clients tell us what they want to see? • Verbal feedback (telephone) • “that thing, over to the left. Up, say, half an inch…” • Not very effective • Graphical feedback • Video overlay marker • Visible to all other clients and the operator • More tools to follow Remote Access and automation Workshop 2005

23. Security • Firewalls • Blocks all direct communication with servers and drivers • Host and servers are in separate ‘DMZ’ network, not mounted on CSIRO file system • Socket encryption: • Sockets are wrapped with ‘Rijndael’ cipher, NIST advanced encryption standard (AES 2001). • Certified for all levels of classification, up to and including ‘top-secret’. • Hackers must break AES to read data, send false data, or issue commands Remote Access and automation Workshop 2005

24. On-line help Remote Access and automation Workshop 2005

25. Administration Remote Access and automation Workshop 2005

26. Web based administration Remote Access and automation Workshop 2005

27. Software based Administration tools Remote Access and automation Workshop 2005

28. Administration tools – Modular New instruments can be added and configured Remote Access and automation Workshop 2005

29. Summary • TelePresence allows Microscopy facilities providing a alternative service to clients • High-quality real-time video is achievable over cable modem using MPEG-4 codec's and UDP transmission • Internet security is ensured using firewalls and encryption of communications • Architecture is extensible and applicable to any instrument for which DLLs or control code are available Remote Access and automation Workshop 2005

30. Acknowledgements • CSIRO Minerals Aaron Torpy Nick Wilson • CSIRO ITS Tony Sanderson Andrew Smith • Engineering Firm Cymontkowski Marc Cymontkowski • Further information TPM web reference http://minerals.tpm.csiro.au/software/telepresence/ Remote Access and automation Workshop 2005

31. Questions? Remote Access and automation Workshop 2005