A Unified Approach to Fast Digital Processing for Beam Dampers, Instrumentation, & Controls

1. A Unified Approach toFast Digital Processing for Beam Dampers, Instrumentation, & Controls Bill Foster Beam Instrumentation Workshop May 6, 2004

2. A Digital Manifesto Example Application: 3-coordinate Bunch-by-Bunch Beam Damper for Fermilab Main Injector Implemented on a Single Altera Stratix FPGA Five other applications using this same hardware Or,

3. Once Upon a Time, there was a job called:“Audio Frequency Analog Engineer” Their products: • Mixers • Equalizers • Crossover networks • Reverbs • Fuzz Boxes…. Bob Widlar, “inventor of the IC Op-Amp” and other analog gems http://www.elecdesign.com/Globals/PlanetEE/Content/3080.html

4. Nowadays, an “Audio Frequency Analog Engineer” is any high-school kid with a PC and SoundBlaster Card Their products: • Mixers • Equalizers • Crossover networks • Reverbs • Fuzz Boxes Emulated on a PC PLUS: • Synthesizers that can fool my ears • Time compressors that squeeze 20% off of a song’s play time without altering the pitch. • Real-time tone substitution makes even Leonard Cohen sing on key …try doing that with an op-amp!

5. What Unemployed the Audio Analog Engineers? • ADC Sampling Rates and Accuracies exceeded requirements • Audio requirements set by human ear • Digital Processing capability exceeded requirements at reasonable cost • 2 GHz CPU executes 50,000 instructions per audio waveform sample

6. Once Upon a Time, there was a job called: “Low-Level Radio Frequency Analog Engineer” Their products: • Mixers • Equalizers • Phase Shifters • Down converters • Phase-locked Loops Fermilab’s Booster Low-Level RF system as it exists today!

7. Nowadays, a “LLRF Analog Engineer” is (or should be) any old programmerwith a fast digitizer and an FPGA Their products: • Mixers • Equalizers • Phase Shifters • Down converters • PLLs Implemented in FPGA’s PLUS: • Direct Digital Synthesis of complex RF waveforms • Built-in system diagnostics • Digital Reproducibility (&spares!) • High Speed Serial Links • Multi-user support

8. What Unemploys the Analog RF Engineers? • ADC Sampling Rates and Accuracies exceed requirements ~ 4 samples per RF clock gives bunch-by-bunch phase and amplitude • Digital Processing capability exceeded requirements at reasonable cost • FPGAs and DSP’s  For ~100 MHz or less, it is: GAME OVER

9. Generic Hardware Concept for Accelerator Instrumentation & Control Clock, control, ... Cables from Tunnel INPUTS: BPM Stripline Pickup Resistive Wall Flying Wire PMT RF Fanback Kicker Monitor …etc. FAST ADC Minimal Analog Filter Monster FPGA CPU Bus VME/ VXI/ PCI/ PMC etc. OR SERIAL LINK . . . . . . . . . FAST ADC Minimal Analog Filter OUTPUTS: Stripline Kicker RF Fanout Analog Monitor …etc. FAST DAC

10. All-Coordinate Digital Damper 53 MHz, TCLK, MDAT,... 212 MHz Stripline Pickup FAST ADC Minimal Analog Filter Monster FPGA 12 Transverse Dampers Identical X & Y CPU: VME/ VXI/ PCI/ PMC etc. OR Serial LINK FAST ADC Minimal Analog Filter Stripline Kicker Power Amp 424 MHz FAST DACs 12 > 27 MHz Resistive Wall Monitor FAST ADC Minimal Analog Filter Longi- tudinal (Z) Damper Broadband Cavity Power Amp FAST DACs 12

11. The Board Alexi Seminov, Sten Hansen, Bill Ashmanskas, Dennis Nicklaus, Hyejoo Kang…

12. Some Example Applications using this same basic hardware: 1) Universal Beam Position Monitors (BPMs) • Handles full variety of FNAL beam RF structure 2) Generic instrumentation readout “Scope” • ex: Flying Wire readout for arbitrary bunches 3) Beam Loading Compensation 4) Universal Beam Dampers / Beamline Tuner 5) Entire Low-Level RF system

13. Fast, High Precision Pipelined ADC’s • AD6645: 14 Bits, 105 MHz • AD9430: 12 Bits, 210 MHz • AD12500: 12 Bits, 500 MHz (hybrid) • Several : 8 bits, ~1-2 GHz (‘scopes) Private opinion: it appears that ADCs are about to fall off of Moore’s Law curve the same way that CPU’s have…

15. AD6645 Functional Block Diagram • Two-Stage Pipelined ADC • Internal Track & Hold • Differential Analog Inputs

16. This ADC can sample 53 MHz signals at 4 samples per cycle to measure both In-Phase and Quadrature on each cycle

18. Board Layout for High-Speed ADCsis a Lot Easier Than it Used to Be • LVDS signals eliminate digital noise • 0.25V differential swing far quieter than TTL • Direct “glueless” interface to FPGAs • Fast input op-amps and surface mount components with small parasitics • Front-end layout is not critical since it is physically small

19. Clock Distribution for ADCsis a Lot Easier Than it Used to Be • Clock and Signal timing can be fixed ex post facto via FPGA firmware timing adjustments • Some A/Ds & D/As have internal PLLs to reduce or eliminate effects of clock jitter • FPGAs have high-quality clock distribution which can be used to drive external A/D & D/As • FPGA clock distribution can challenge Dedicated Clock Distribution Chips (on not…L. Dolittle)

20. Q: What ADC Clock Speed is needed? A: 4x RF Bunch Frequency • Minimum needed for bunch-by-bunch Phase and Amplitude measurement • In frequency domain, 4x RF sampling measures bothin-phaseandquadrature components. • For Fermilab’s 53 MHz RF  212 MHz ADC’s

21. 212 MHz Sampling of RWM Pulse Low-pass Filter Spreads signal +/-5ns in time so it will not be missed by ADC Filter Reduces ADC Dynamic Range requirement, since spike does not have to be digitized

22. 212 MHz Sampling of Stripline Signal Roles of “Phase” and “Amplitude” signals are reversed from unipolar case.

23. Repetitive Waveform looks like simple sine wave, but contains bunch-by-bunch phase and amplitude “A - B” gives bunch-by-bunch “in-phase” signal “D - (C+E)/2” gives bunch-by-bunch “out-of-phase” or “quadrature” signal Vector Sum sqrt(I**2 +Q**2) is insensitive to clock jitter

24. Bunch-By-Bunch Phase vs. Turn NumberMeasured with MI Digital Damper • Damper Output comes from derivative of individual bunch phase errors

25. Bunch-By-Bunch Intensity

26. Synchronous vs. Asynchronous ADC Sampling • The choice is between • N*53 MHz beam phase locked sampling, or • Asynchronous sampling at a (possibly) lower rate • Asynchronous sampling of a waveform will allow you to recover all the information, IF: • you know that the input is a pure sine wave, or • you know the input is repetitive (stored beam), or • the sampling rate is much higher than fMAX My belief is, undersampling is just a bad idea…

27. The Perils of Undersamplinga Single-Pass Beam If a single-pass beam does not have uniform bunch populations, the ADC input is NOT a good sine wave and an undersampled waveform can give an erroneous picture of the beam signal. The signal CAN be reconstructed with many passes of stored beam.

28. … dealing with this variety of beams would be painful in Analog…

29. Digital Filter looking at many samples can still extract individual bunch transverse positions

30. Advantages of Digital Processing • Digital filters more reproducible (=>spares!) • Inputs and Outputs clearly defined (& stored!) • filters can be developed & debugged offline • Digital filter can also operate at multiple lower frequencies ...simultaneously if desired. • Re-use Standard hardware with new FPGA code • or same code with different filter coefficients

31. Conclusions on ADC Clock Rate • A Bunch-by-Bunch processing system must sample the raw waveform at a minimum of 4x the Bunch frequency • You can never be: • too rich • too thin • or have too many ADC samples

32. What is an FPGA? • Reconfigurable Logic Array ~106 logic gates • Pre-built logic subassemblies: “Megafunctions” • Multipliers/Accumulators • Multi-port RAMs • Gigabit serial links • Entire CPUs • Phase Locked Loops • Complex I/O pads • More transistors than a Pentium • Impressive Support software

33. XYLINX And ALTERA Are the Industry Leaders

35. What is an FPGA Good At? • Big Synchronous Arithmetic Pipelines • 400 MHz multiply/accumulators, filters.. • High-Speed Interface with Modern Parts • ADCs, DACs, Serial Links • Built-in system diagnostics • “Digital Scope on every signal” • Flexibility and Multiple Applications • Use one board design for many applications • Add features without hardware changes

36. SYNCHRONOUS PIPELINES • When people say “Analog is simple”, they are often referring to the deterministic execution time (“propagation delay”). • Analog circuits never fail to respond in time because they are off servicing an interrupt. • FPGA synchronous pipelines provide dedicated logic which responds at a deterministic time. • This captures a big advantage of Analog.

38. FPGA Programming Languages:Graphical vs. Text-mode • Graphical Schematic Entry is useful for: • diagramming data flow • giving talks • Text Mode features • Text is faster to enter and more concise • Can “diff” two files to see what’s changed • Code management systems can handle text well I PERSONALLY RECOMMEND TEXT MODE

39. FPGA Programming Languages:Proprietary vs. Industry Standard • Proprietary languages often lock you into a single vendor. • I use one (Altera AHDL) anyway. • The “industry standard” VHDL language • It is extremely verbose & repetitive • Translating AHDL into VHDL increases the text length by a factor of ~2. LIKE TRANSLATING A DOCUMENT TO FRENCH

40. Some Development Models How do you Download & Talk to this Board that you’ve just built? • FPGA Programming cable • Firmware Serial Port Model • Crate Backplane Bus Access • On-Board CPU w/Ethernet • Compiled-in On-chip CPU w/ Ethernet

41. CPU Access to FPGA Registers • Usually want ADDRESS/DATA R/W model for CPU access to Control Registers • These “address and data busses” are synthesized in firmware Example (AHDL) for 32-bit read/write register: (Bus[],Outputs[]) = BUS_REG( ) WITH( ADDRESS = H“08402020”, WIDTH = 32 );

42. Development Model #1:FPGA Programming Cable • The programming cable needed to program the FPGA (usually through the PC printer port) can also be used for limited communication • Not clear how useful this is for real-time response since it works through serial port driver • Altera provides “compiled-in logic analyzer” which provides output that can be compared with simulation.

43. Development Model #2:Crate Backplane Bus Access • Crate Backplane Bus connections to FPGA can provide CPU access to registers in internal address space of FPGA • Internal Address Space is defined in firmware • Requires many bits of bus buffers, etc. • Be Careful of Backplane Noise (TTL)

44. Development Model #3:Serial Port to PC • A firmware-defined Serial Port can be used for 2-wire communication with the .COM port of a PC • Terminal emulator can provide simple read/write access the internal address space of the FPGA • Can also connect to spread sheets, etc via Visual Basic access to .COM port

45. Development Model #4:On-Board CPU w/Ethernet • “Postage Stamp” Ethernet CPU or homebuilt DSP can provide Ethernet and Web access to FPGA registers • Firewire is an alternative • Remote update of firmware possible • NIM-like modules without need of crate backplane

46. Development Model #5:Compiled-in On-chip CPU with Firmware-defined Ethernet • High-end end FPGAs have built-in or firmware-defined CPUs fast enough to support IP stack, Web Servers, etc. • These are available on Demo Boards • C-language programming of these is integrated into FPGA development environment (no new software!).

47. Adding a new ACNET Device 1) Add register(s) to FPGA Firmware  Takes about 10 minutes from concept to Fast-Time Plot 2) Start Recompile (takes ~6 minutes) 3) Meanwhile, use DABBEL/D80 to define properties of new ACNET device 4) Download Firmware & Reboot Crate (~2 min.)

48. Application #1: Universal BPM(Beam Position Monitor) • Measures position of each bunch on each pass around the ring with full-bandwidth FIR filter • (R-L)/(R+L) for each bunch measurement. • Multi-bunch averages available for lower noise • per batch, per turn, many turns, different bandwidths • Multiple users can share hardware w/o conflicts • ADC is always active, FPGA stores data many ways Same Hardware OK for Booster, MI, RR, TeV, & beamlines.

49. FPGA Based “Universal BPM” 53 MHz, TCLK, MDAT,... 212 MHz Split Plate Pickup #1 Minimal Analog Filter Monster FPGA(s) ADC R 14 CPU VME/ VXI/ PCI/ PMC Ethernet etc. Minimal Analog Filter L ADC . . . . . . Pickup #4 Minimal Analog Filter ADC T Minimal Analog Filter B ADC Serial Link to Real Time Orbit Control System Analog Position Monitor Test Point (Optional) OR Modulation Output for Synchronous Lock-in Detection Technique FAST DAC

50. “Universal BPM” Application: Signal Processing Steps Front End 1) Bandwidth-Limit input signal to ~53 MHz 2) 12 Bit Digitization at 212 MHz 3) FIR filter(s) to get single-bunch signal(s) 4) Sum & Difference of plate signals 5) (Difference / Sum) gives position 6) Linearization lookup table or polynomials 7) Bunch, Batch, Multiturn Averaging 8) “Scope Trace Buffers” on every signal Multiple users can be acquiring and filtering data multiple ways without conflicts Inside FPGA