Correlator experiences with Mark 5 in operation

1. Correlator experiences with Mark 5 in operation Arno Müskens Geodetic Institute – University of Bonn Walter Alef, Dave Graham Max-Planck-Institute for Radioastronomy Kerry Kingham USNO Washington D.C. Third IVS General Meeting 2004, Ottawa, Canada

2. Processor / SU / Mk5a units Mk5a units Panorama on the Correlator Bonn Correlator - snapshots Third IVS General Meeting 2004, Ottawa, Canada

3. MK5 8-pack (discs memory from 120 to 200 Gbyte and more) MK5 8-pack disc library Mark 5 - snapshots Third IVS General Meeting 2004, Ottawa, Canada

4. Mark 5 A history Sept. 1999 Haystack Observatory. White paper states the goal of the next generation 1 Gbps VLBI data system May 2000 Haystack Observatory proposed the development of a new high- data-rate VLBI system based on COTS technology Jan. 2001 COTS-VLBI interim report states disks should be future recording medium Mar. 2001 First Mark 5 Demonstration System at 512 Mbps, built in 2 month June 2001 Proposal for support of Mk5 development to BKG, MPIfR and other institutions July 2002 First Mark 5 P System arrived at the Bonn Correlator Oct. 2002 Intensives started using Mark 5P at USNO Dec. 2002 First two Mark 5 A systems arrived at the Bonn Correlator May 2003 First successful EVN observation at 1 Gbps with four telescopes Sept. 2003 Eight Mark 5 A systems are implemented at the Bonn Correlator Febr. 2004 Eight Mark 5 A systems are implemented at USNO Correlator Third IVS General Meeting 2004, Ottawa, Canada

5. Advantages of the Mark 5 system • Robust operations • Easy transportable • Conformance to the VLBI Standard Interface (VSI) specification (Mk 5B) • Design based primarily on unmodified off-the-shelf subsystems and components • Unattended operation limited by available disc size • Cost for recording media (ATA discs) drops continuously • Rapid random access to any data • Instant synchronization on playback into the correlators • Easily upgradeable host platform • The data is written to all disk simultaneously in a ´round robin´ fashion • Data of broken disk is replaced by a fill-pattern Third IVS General Meeting 2004, Ottawa, Canada

6. Disadvantages of the Mark 5 system • The disc modules have to be handled carefully • Some kinds of discs are more susceptible to damage in transport as the head is parked on the disk patters • Disc modules must be inserted into Mark 5 unit without brute force as the 200-pin connector could get damaged • StreamStor card can occasionally hang Third IVS General Meeting 2004, Ottawa, Canada

7. In the end of 2002 we saw one of these fatal problems caused by one or two screws which were screwed too tightly. This reaction started from backplanes that were not constructed correctly. A new revision of the backplanes solved this problem. This screw caused panic Third IVS General Meeting 2004, Ottawa, Canada

8. 200 pin position connector (AMP) The backplane underlies constant mechanical stress due to the insertions and removals of the modules from the Mark 5 units Occasionally those condenser inside the Mark 5 8-pack module bypassed sometimes, which could load the power supply in perpetuity Third IVS General Meeting 2004, Ottawa, Canada

9. Correlator status with Mark 5 A at Bonn and USNO • Eight Mark 5 A units were integrated into both correlators. • Connected to the 2nd head-input of the SU, all 32-track modes can be correlated with either tape or disc, changed using just software switch • With simple re-cabling, single pass correlation of modes up to 1 Gbps can be done • Mark 5 A correlation is becoming a standard and is already more robust than the correlation with tape • Various modes were correlated successfully with mixed discs and tapes • About 30 % to 40 % of the correlated data is now on Mark 5 A • From November 2002, Intensives were Mark 5 only at USNO • Mark 5 Intensive Processing Factor (PF) is 1.0 with tapes it was 1.5 • Very few failures in more than 200 Intensives occurred Both correlators experienced (here IVS-R and other 24 hours experiments) a reduction in the PF by 20-25 % and the reprocessing by almost ~80 %. We ascribe this to growing number of Mark 5 stations (2-4) station in our 6 to 9 station experiments and to the reduced need of intervention from the operator Third IVS General Meeting 2004, Ottawa, Canada

10. Mark 5 correlation problems and notes • At the beginning and still today Mark 5 8-packs often arrived without proper labels. • All disc modules are uniquely identified with a VSN on the disc module outside and electronically on disk. Occasional saw that those two VSN were different. As a result the modules cannot identified properly which confused operators and software. • All disc modules are identified with disc serial numbers. Sometimes the observing log files created at the stations during observation contain wrong entries. • StreamStor card occasionally hang (Mark 5). • During multi streaming processing occasionally SU crashed. Haystack worked on fixing that problem. Third IVS General Meeting 2004, Ottawa, Canada

11. 1 Gigabit/sec data handling Recording • Initial problems were encountered due to faulty or outdated firmware, faulty electronics and excessive cable length inside the Mark 5 units • Cable length of 0.5 m and greater than 2 m produced problems. A length of about 1.5 m seems working fine. This might be due to bad impedance matching at the formatter output. These cables could become a source of RFI • Old cables that have been kicking around for many years and have been bent out or creased do not perform well at these high data rates. Third IVS General Meeting 2004, Ottawa, Canada

12. 1 Gigabit/sec data handling Playback • Mark 5 units are connected to the upper input connectors of the SU, originally designed for the second tape drive head. The single head of a tape drives than connected to the lower input pair of the SU. This enables easy software switching between tape and disc operation for each SU • At 1 Gbps the connection between Mark 5 and SU must be changed to connect 64 tracks • Correlation runs smoothly, expect when the online system already reported problems during recording. • JIVE correlator reports problems in recording data using the LSB channels and which has been recorded without accompanying USB channels. It is not clear if this affects all mk4 correlators. Third IVS General Meeting 2004, Ottawa, Canada

13. Mark 5 B • Will have VSI-compliant interface and command set • Based on the same physical platform as Mark 5 A • Will bypass the formatter and connect directly to the output of the samplers • Will allow maximum data rate of 1024 Mbps also for VLBA • Will be compatible with the planned digital BBC, which have only VSI interface • Will include simple Mark 4 station-unit (SU) capability, which will allow us to replace the bug afflicted station units (SU) which are the weakest points of the correlator • Will allow a relative inexpensive expansion of the number of stations at the correlator side • Will not be able to play back Mark 5 A recordings, but compatibility in the other direction is foreseen Third IVS General Meeting 2004, Ottawa, Canada

14. To Do list for Mark 5 • Support for interchangeability between 8-pack modules containing parallel ATA disks and/or serial ATA disk drives. • Robustness in the handling for stations and correlators • Adding extra information to the scan directory on Mark 5 recorded data • Make upgrading Mark 5 easier and less prone to error • Upgrade for NRAO "Track" Software • Revise of the VSN label specification • Mark 5 A and Mark 5 B disc recordings are not interchangeable ! • All correlators will have to ensure correlation both with Mark 5 A and Mark 5 B at the beginning. Third IVS General Meeting 2004, Ottawa, Canada

15. Conclusions In spite of some Mark 5 childhood diseases it is clear that : • The Mark 5 system has been shown to make VLBI data recording and correlation significantly more robust and at the same time reduces the cost for maintenance and media drastically. • Both geodetic and astronomical VLBI observing can soon profit from the increased sensitivity provided by recording data rates up to 1024 Mbit / s. • The throughput can be higher than with tapes and should allow IVS to reach it ambitious plans for observing in the next few years. Third IVS General Meeting 2004, Ottawa, Canada

16. Some important decisions in your life you will regret ! Third IVS General Meeting 2004, Ottawa, Canada

17. but the step to Mark 5 … N E V E R !! Thank you for your attention ! Third IVS General Meeting 2004, Ottawa, Canada