Digital FX Correlator

1. Digital FX Correlator EOVSA Technical Design Meeting Nimish Sane Center for Solar-Terrestrial Research New Jersey Institute of Technology, Newark, NJ

2. P, P2 • Calculation Overview Nimish Sane, New Jersey Institute of Technology

3. Hardware • Roach-2 board [1] • Virtex-6 SX475T FPGA (XC6VSX475T-1FFG1759C) • PowerPC 440EPx stand-alone processor to provide control functions • 2 x Multi-gigabit transceiver break out card slots, supporting up to 8x10Ge links which may be CX4 or SFP+ • 4 x 36 * 2M QDR II+ SRAMs connected to the FPGA • A single 72-bit DDR3 RDIMM slot connected to the FPGA • 2 x ZDOKs • An FTDI FT4232H USB to JTAG, serial and IIC • 8 boards with 2 antennas (dual-polarization) per board Nimish Sane, New Jersey Institute of Technology

4. ADC Comments from discussion with Dan Werthimer and David MacMahon • KAT ADC • ASIAA ADC • 1 signal at 5 GSamples/sec (GS/s) • 2 signals at 2.5 GS/s • Used at CFA (Jonathan Weintroub, Rurik Primiani) • 2 ADCs on the same board may cause cross-coupling issues. Nimish Sane, New Jersey Institute of Technology

5. F-Engine Fix 8_7 Ufix 36 even odd F – engine : 4096 Channel No. of F-engines per Roach board = 4 (2 antennas dual polarization) Output of an F-engine: Complex (even and odd channels), each with 18-bit real and 18-bit imaginary. Output data of F-engines per Roach board per FPGA clock cyle = 72 x 4 bits Total data rate of F-engines per Roach board = 7.2 Gbps (assuming FPGA clock period of 4 ns) Nimish Sane, New Jersey Institute of Technology

6. X-Engine UFix 36 Fix 4_3 (?) (32) (960) X – Unit Each Roach board will have 1 X-unit Each X-unit will handle 4096/8 = 512 frequency channels (256 even and 256 odd channels) No. of complex multipliers in each X-unit correlator block = No. of visibilities x No. of polarizations x Simultaneous even and odd channels per F-unit = 120 x 2 x 2 = 480 X-unit output data per integration = 480 x 2 x 256 x 4 bits = 983040 bits Total data rate at the output of X-unit per Roach board = 983040 x 50 = 49.152 Mbps (assuming integration time of 20 ms) Nimish Sane, New Jersey Institute of Technology

7. F-engine Comments Coarse delay on FPGA Fine delay off-line Keep P, P2 calculations separate from X-engine Phase switching is difficult on FPGA. Dan suggests doing it before ADC, and then undoing it on FPGA ATA Memo on fringe stopping after FFT GMRT does fringe stopping + coarse delay + fine delay + (possibly) phase switching, but not at 600 MHz Nimish Sane, New Jersey Institute of Technology

8. F and X-engine Connections • Various architectures have been proposed in [2]. • Current state of art: • F and X engines on different boards • F + X on the same board: (1) Can we fit the design? (2) Can F + X work in tandem? • Use full-duplex bidirectional capacity of 10 GbE link: Send output of F – engine to a switch that will distribute it to X – engines (even if F and X are on the same board) Nimish Sane, New Jersey Institute of Technology

9. X-engine Comments If CPUs are going to be used anyways, GPUs may be targets for X-engine No GPU correlator has been deployed yet. (PAPER and LEDA in progress). X-engine on FPGA is more straight-forward. Nimish Sane, New Jersey Institute of Technology

10. Other Issues • How to perform 4-bit correlation? • Ensuring that data is well-distributed over 16 levels • How to use block RAM (BRAM) blocks for scaling each frequency channel? How to determine the gain values for each frequency channel? • Synchronization with external clock Nimish Sane, New Jersey Institute of Technology

11. References Nimish Sane, New Jersey Institute of Technology https://casper.berkeley.edu/wiki/ROACH2 P. McMahon, et al. “CASPER Memo 017: Packetized FX Correlator Architectures,” September 2007.