All about CSC Timing

1. All about CSC Timing An immodest title and a work in progress. Try to cover as much as possible: • CSC Clock distribution • Inter-board communication timing • Trigger timing • Online DAQ readout timing • BX numbers in online synchronization • Data records: bx numbers and offline timing CMS Meeting CSC Timing

2. Current  status (May 2009) •  Many timing parameters can be determined without beam, some even without HV or gas in chambers •  Cosmic rays, radioactivity, and pulsing can be used variously, timing is often very similar •  Timing parameters agree very well (but not perfectly) with predictions based on copper and optical cable lengths •  Trigger and readout are usually efficient •  This was all exercised late last year when CSCs provided one of the main triggers for single beam operation CMS Meeting CSC Timing

3. Current  status (May 2009) •  CSC synchronization seems overly complicated •  Documentation is somewhat scattered • Write a formal and definitive CMS note? •  Some discrepancies are found • E.g. ½ bx optical cable length correction at SP • Transition to colliding beams looks non-trivial • TOF corrections not completely under control •  Finer touches are lacking • We are not continuously checking synchronization • We don’t check all the ALCT-TMB trigger and data lines • We haven’t ironed out all ALCT and TMB emulator discrepancies • We haven’t ironed out all trigger bit comparisons between DMB and SP readout CMS Meeting CSC Timing

4. CSC Clock Distribution • TTC clock distribution. • CFEB comparator clocking. • ALCT clocking. CMS Meeting CSC Timing

5. TTC Clock Distribution • Unfortunately, TTC signals are distributed to the 60 peripheral crates through fibers of various lengths and propagation times • Compensate for fiber length variation via programmable delays in TTCrx chips in CCBs • Relies on good optical fiber length measurement • July 2008 Optical fiber lengths were re-measured to ~cm level (S. Casenove, CERN), previously were poorly measured • Oct. 6, 2008 adjusted clocking to ns level (config: TTCrxFineDelay) • (previously, Aug. 15, 2008 adjusted clocking to nearest BX level, TTCrxCoarseDelay) CMS Meeting CSC Timing

6. TTC Clock Distribution • Time delay versus optical fiber length for CSC peripheral crates covers a range of about 11 bx: *** NB remove black and red points CMS Meeting CSC Timing

7. SP optical fiber AFD delays MPC resync resync TTCrx delays CCB TTC optical fiber Check of peripheral crate clocking • See diagram at right: • TTCrx delays now set for isochronous Peripheral Crate clocking • Compare signals arriving from MPCs through various MPCSP optical fibers • AFD delays should exactly compensate for various MPCSP fiber lengths • Almost the same as the TTCCCB fiber lengths *** Greg has better way of Indicating delays in small circles CMS Meeting CSC Timing

8. SP optical fiber AFD delays MPC resync resync TTCrx delays CCB TTC optical fiber Check of peripheral crate clocking • Details (MPCSP resync procedure): • Resync goes to CCB in Track Finder crate • Sent to all SPs in parallel via isochronous backplane • Resync goes to 60 Peripheral Crate CCBs via various optical fibers of various lengths • Resync (formerly “L1_Reset”) sent from CCB to MPC • MPC initiates optical link synchronization procedure • In Sector Processors there is an 80 MHz Alignment FIFO Delay (AFD) to be set so that optical link data are lined up in time. CMS Meeting CSC Timing

9. Check of peripheral crate clocking • Result looks good (0.5 bx steps every 2.52m)… *** NB remove blue points CMS Meeting CSC Timing

10. But results not perfect… : • MPCSP fibers not really same length as TTC CCB fibers • MPC-SP difference (meters): • Nov. 2008 results from Dan including MPCSP fiber lengths not as nice as plot on previous page: CMS Meeting CSC Timing

11. A good test to do: • Bc0 should be aligned in time at the peripheral crates • Check by sending Bc0 from TMBs to SP: • Diagram below: must set all mpc_tx_delay values to be same (0?) • Must send Bc0 from TMBs • Check at MPC: are all Bc0’s from TMBs aligned? • Check at SP: are all Bc0’s from MPCs aligned? • I believe that most if not all of TTCrxDelays are set correctly to 1ns level. • If Bc0’s turn out not to be aligned, then need to adjust AFDs, especially for cable lengths near 80 MHz edges. (for discussion) mpc_tx delay AFD delays MPC SP TMB bc0 TTCrx delays CCB TTC CMS Meeting CSC Timing

12. Details of BC0 check (1/4) • TMB setup: (called BX0 in documentation) • By default, sends BC0 every orbit to MPC • Can choose from TMB’s own BC0 or relay ALCT BC0. • TMB’s own BC0 is preferred • VME address CA contains several relevant parameters • alct_bx0_enable “default 1=Enable using alct bx0, else copy clct bx0” probably want to set to 0. • Also contains adjustable delays for BC0 (0-15 bx) CMS Meeting CSC Timing

13. Details of BC0 check (2/4) • MPC check BC0 alignment: • MPC2004_100808.pdf: “the BC0 signal that is transmitted to the SP, is a logical OR of 9 BC0 signals from all the TMB’s” • Can SP input check that BC0 signal is 1 bx wide? (see next slide) • If not, it will be a surprise and/or hardware problem in the peripheral crate • Finding offending chambers: • TMB allows to mask off BC0 for individual TMBs to find the offender(s). • Or, perhaps only enable one TMB at a time to send BC0. • Also TMB BC0 can be delayed to find the timing of the offender(s). CMS Meeting CSC Timing

14. Details of BC0 check (3/4) • SP check of BC0 alignment • LU-SP2DDU_Event_Record_Structure_4d2.pdf: • “BXx – Bunch Crossing Counter status bits for 15 MEx links; BXx bit is set to 1, when link’s BC0 timing mark arrives early or late, and it is reset back to 0, if any next BC0 timing mark arrives in time. So, this bit monitors the ALCT/TMB/MPC timing and post-marks irregular orbits” CMS Meeting CSC Timing

15. Details of BC0 check (4/4) • “BXx – Bunch Crossing Counter status bits for 15 MEx links; BXx bit is set to 1, when link’s BC0 timing mark arrives early or late, and it is reset back to 0, if any next BC0 timing mark arrives in time. So, this bit monitors the ALCT/TMB/MPC timing and post-marks irregular orbits” • Is this all that we have and/or need to monitor? • If always present, can we access by simple VME read(s)? • What are “early or late” relative to – previous orbit on that link, or some internal common BC0? • If some TMBs send BC0 early but other TMBs are correct, does this status bit from MPC get set and then immediately cleared so as to make the out-of-timeness invisible? CMS Meeting CSC Timing

16. Crate Master clock TMB TMB Master clock Latch input ALCT data ALCT section • Adjust ALCTtx for optimal latching of ALCT output data at TMB 2ns/bin Latch output ALCT commands ALCT -RX clock CCB test pulse commands • Adjust ALCTrx for optimal latching TMB output data at ALCT ALCT -TX clock TMB pass- through 2ns/bin ALCT commands ALCT data ALCT ALCT Master clock ALCT Main FPGA • Adjust Delay ASICs for max. probability for ALCTs to come in one BX Internal test pulse via VME command to TMB Asynch. test pulse from VME write to CCB Delay ASICs ALCT latch raw data Synch. test pulse from TTC command or VME write to CCB Main FPGA OR ~2.2ns/bin AFEB CSC Test Pulse Strips AFEB data Test pulse to AFEB amplifier or test strips (select via VME write to TMB to ALCT Slow Control FPGA register) ALCT Clocking from TMB • TMB sends two 40 MHz clock to ALCT • Clocks and bidirectional data are sent on the same 6-15m Skewclear cable • ALCT_tx clock adjusted so trigger data etc. is latched properly at TMB (like CFEB clock) • ALCT_rx clock adjusted so that TMB “command” data is latched properly at the ALCT • Adjust both in 2ns steps to middle of <12.5ns good data window CMS Meeting CSC Timing

17. Active Pulsing for ALCT Clock Timing • *** This is probably out of date. How done in ME1/1 with no test strips? • How it’s done: • CCB provides 500ns gate to make the ALCT test strip pulses. These pulses go through the ALCT (lemo connector) to the test strips. • Capacitive coupling between test strips and anode wire groups gives pulses on leading and trailing edges of 500ns pulse • All test strips fire on leading edge • Hot wire mask on ALCT board  Select ALCT patterns with 6 hits. Create 2 muons mask. • Like CFEB, vary 40 MHz receive and send phases until optimum data transmission from TMB to and from ALCT • Find optimum in 2D matrix of receive/send clock phases • Scan patterns across chamber to find bad AFEB/ALCT channels • Advantages: • Set up phases of 80 MHz clock TMB to and from ALCT with high reliability without HV, gas, or cosmic ray data • Check all AFEB, ALCT, and Skewclear channels through the system • The HV should be off. CMS Meeting CSC Timing

18. ALCT tx and rx phases are correlated: Good settings ALCT Clock Phases Determination CMS Meeting CSC Timing

19. ALCT clock phase check • Correct phase should depend only on the Skewclear cable length. • On Oct 16, 2008, there was only one outliers out of 468 chambers (off by at least 3 settings = 6 ns): • Recall phases can wrap around so that setting 12 = 0 • Outliers could be a problem with a few cable lengths in database (?) • We always use the measured values, not predicted • (*** note for self: why more than one outlier below?? Check size of blue bar below) Blue line shows typical size of good data window CMS Meeting CSC Timing

20. Also • ALCT rx versus tx delay – totally correlated, as expected CMS Meeting CSC Timing

21. Truly isochronous ALCT clocking? • Set ALCT-tx clock delays to values pre-calculated from Skewclear lengths: • So that the ALCTs would be clocked isochronously • Then AFEB fine delays only need to accommodate TOF+electronics delays • Would greatly simplify our understanding of our AFEB fine timing constants • Would simplify understanding of other timing constants as well (such as ALCT/CLCT relative timing) CMS Meeting CSC Timing

22. Globally isochronous after TTC fine delay corr’s Isochronous muons within a crate Isochronous muons after SP input FIFO delay Lev idea for synchronous ALCTs via BC0 • Pass bc0 through the electronics chain and check alignment at each stage: Clocks running downwards TTC CCB1 CCB2 TMB TMB TMB TMB Data: Muon+TOF ALCT ALCT ALCT ALCT Data: Muon+TOF TMB TMB TMB TMB MPC1 MPC2 CCB SP CMS Meeting CSC Timing

23. BC0 alignment at ALCT • Not aligned now because BC0 goes to ALCT on Skewclear cables of various lengths • UCLA TMB firmware update of 12-Aug-2008 has programmable delay (0-15 bx) for bx0, L1A, other TTC signals sent from TMB to ALCT. • Allows ~isochronous ALCTs. • However, doesn’t quite work at the sub-bx level because of ALCT transmit and receive clock phase adjustments, which depend on ALCT-TMB cable lengths. • Full chain of anode delays: mpc_tx delay AFD delays fine delay TMB SP MPC ALCT AFEB resync resync match delay TTCrx delays CCB TTC CMS Meeting CSC Timing

24. Crate Master clock TMB TMB Master clock Latch input ALCT data ALCT section • Adjust ALCTtx for optimal latching of ALCT output data at TMB 2ns/bin Latch output ALCT commands ALCT -RX clock CCB test pulse commands • Adjust ALCTrx for optimal latching TMB output data at ALCT ALCT -TX clock TMB pass- through 2ns/bin ALCT commands ALCT data ALCT ALCT Master clock ALCT Main FPGA • Adjust Delay ASICs for max. probability for ALCTs to come in one BX Internal test pulse via VME command to TMB Asynch. test pulse from VME write to CCB Delay ASICs ALCT latch raw data Synch. test pulse from TTC command or VME write to CCB Main FPGA OR ~2.2ns/bin AFEB CSC Test Pulse Strips AFEB data Test pulse to AFEB amplifier or test strips (select via VME write to TMB to ALCT Slow Control FPGA register) Possible pre-calculated ALCT-tx phase for isochronous ALCTs Implications: • ALCT-rx phase needs adjustment for TMBALCT command transmission • Part of current timing-in procedure, so not a problem • TMB inputs from ALCT would no longer be sync’ed to TMB main clock – need an additional input data synchronization stage CMS Meeting CSC Timing

25. Implications of possible pre-calculated ALCT-tx phase • TMB inputs from ALCT would no longer be sync’ed to TMB main clock – need an additional input data synchronization stage • This raises several questions: • Additional clock delay adjustment in FPGA and firmware– do we even have that possibility? • Is there additional latency? (But ALCT is not the critical path) • Should we do the same for CFEB clocking?? Less critical for timing, of course. CMS Meeting CSC Timing

26. Additional implications… • ALCT fine delays has to be optimized to maximize ALCTs in single bx (with small tail preferred in the +t direction?) • Unlikely that we can optimize all chambers in the SAME bx: • AFEB fine delay setting range is 0-15 (0-33 ns) • c=3*108 m/s = 0.3 m/ns = 7.5 m/bx • Specific example: • center of nearest chamber ME+1/1/nn optimal delay could be 5 (11ns) and cannot be set to 16 (~36 ns) • Further out chambers have larger TOF hence smaller delay by ~25ns  indicated adjustment would be negative • Therefore, keep in mind that some chambers will necessarily be “off” by 1bx intentionally – can be adjusted for by mpc_tx_delay etc. CMS Meeting CSC Timing

27. For now, can skip to conclusions slide CMS Meeting CSC Timing

28. TMB Crate Master clock TMB Master clock Latch data in CLCT section Comp. delay Data Delay Devices 2ns/bin 40 MHz clock Comparator data CFEB (1 of 5) Comparators CFEB Comparator Clocking • TMB sends 40 MHz clock to CFEBs • CFEBs return comparator data • Clock and data on same 6-15m Skewclear cable • Adjust comparator clock phase so data is latched correctly by TMB internal clock • Adjust in 2ns steps to middle of ~12ns good data window CMS Meeting CSC Timing

29. CFEB Comparator Clocking – how measured? • *** Is this still up-to-date? • Generate comparator hit patterns for all layers: • Buckeye ASIC - all channels have capacitors with 4 charge levels that can be preset (0,1,2,3) • “left half-strip” puts strip charges at ….,0,0,2,3,1,0,0… • “right half-strip” puts strip charges at …,0,0,1,3,2,0,0… • Load pattern and pulse height into the ASICs. Take staggering into account. • Single VME command to DMB pulses loaded channels. • Loaded patterns give e.g. 6-layer CLCTs. • Compare TMB readout with loaded pattern. • Vary 40 MHz clock phase from TMB to comparators until patterns correctly found. • Also, patterns can be swept across entire chamber • Checks all Buckeye, comparator, and CFEB-TMB Skewclear cable channels • No gas, HV, etc. needed. HV should be off. Interference with cosmic muons. CMS Meeting CSC Timing

30. (*** would like more up-to-date plot, this is really old) CFEB Clock Phase Determination CMS Meeting CSC Timing

31. CFEB clock phase check • Correct phase should depend only on the Skewclear cable length. • How to line up data at same phase at TMB with a longer cable to CFEB? • Example: cable B is 1ns longer than cable A • Send the clock down 2ns earlier. • It arrives at CFEB 1ns earlier. • Data arrives back at CFEB at the same time as with cable A. • So with longer cable delay, should have 2x earlier clock • *** List the formula CMS Meeting CSC Timing

32. CFEB Comparator clock phases compared with model values • *** This is an old slide, we have much better statistics now • Correlates well with CFEB cable length • Expected behavior: with longer cable delay, should have 2x earlier clock Recall that for a continuous clock, phases wrap around so that no need for settings 13-16 and setting 12 ~ setting 0 CMS Meeting CSC Timing

33. CFEB clock phase check • *** Check size of blue bar below, check the number of outliers • On Oct 16, 2008, there were about 5 outliers out of 468 chambers (off by at least 3 settings = 6 ns): • Recall for continuous clock, phases wrap around so that no need for settings 13-16 and setting 12 ~ setting 0 • Outliers could be a problem with a few cable lengths in database (?) • We always use the measured values, not predicted Blue line shows typical size of good data window CMS Meeting CSC Timing

34. Next section: Trigger Timing • ALCT-CLCT time coincidence • LCT time alignment between chambers before sending to MPC, parameter mpc_tx_delay • AFEB adjustments to fine delays • Between adjacent chambers from data (Dayong) • Greg covered this well in 071019_rakness_sync_summary.ppt • But only 18 to 36 slice test chambers then, vs. 468 now CMS Meeting CSC Timing

35. Next section: Trigger Timing • Further adjustments to timing for TOF • Equalized already at bx level between endcaps from HE comparison  • Note to self: did we overlook a possible fine timing adjustment between endcaps?  • Cosmic rays: top trigger sectors delayed by hand relative to bottom to match barrel trigger better: • 1 bx adjustment at present • NB. endcaps are 1.86 bx tall  2 bx adjustment better?? • Experiment from top of one endcap to bottom of the other is ~3 bx… • By-hand adjustment then gives a discontinuity at 0, p (or so) in f with respect to Dayong optimal time coincidences  CMS Meeting CSC Timing

36. Also • ALCT-CLCT relative timing, based on DQM data (Barashko) • JH “phase theory” predictions not precise, not yet understood. CMS Meeting CSC Timing

37. Also • Various delays from TMB to MPC (mpc_tx_delay parameter) • Predictions are not precise (mostly 0+-1 bx), not understood. CMS Meeting CSC Timing

38. Next section: DAQ Timing • TMB • L1A coincidence window to initiate TMB readout to DMB, parameter tmb_l1a_delay (0-255 bx) • TMB-DAV delay to tell DMB that data is being sent to its input FIFO, parameter tmb_dav_delay (?) • *** There is a little “scope window”, does DAV need perfect timing? • ALCT: • L1A coincidence window, parameter alct_l1a_delay (0-255 bx) • ALCT-DAV delay to tell DMB that data is being sent to its input FIFO, parameter alct_dav_delay (?) • CFEB: • L1A coincidence window(?) • ??CFEB-DAV delay to tell DMB that data is being sent to its input FIFO CMS Meeting CSC Timing

39. TMB-DMB Block Diagram TMB DMB CFEB TTC/CCB Crate Master Clock, L1A • (external L1A = LHC & Test Beam operation modes) TMB Master Clock, L1A DMB Master Clock, L1A CLCT DAV delay L1A*CLCT DAV Coinc. CLCT-DAV LCT-L1A Coinc. starts TMB readout CLCT Readout queue CLCT FIFO DMB-DDU readout Controller logic ALCT DAV delay L1A*ALCT DAV Coinc. ALCT-DAV LCT -read delay LCTs to MPC From ALCT readout queue ALCT FIFO CFEB DAV Coinc. CFEB Clock phase L1A delay L1A ALCT/ CLCT/ RPC Coincidence RPC delay 1 ns/bin CFEB DAV delay CFEBs “hit” Auto set Cable Equal. delay AFF (Active FEB Flags) Cable Equal. delay CFEB FIFOs (5) CFEB -DAV CLCT Final logic AFF-L1A Coinc. Starts CFEB digi. & readout RPC logic AFF delay ALCT delay Output FPGA fixed CFEB Clock SCAs, ADCs, Memories From RPC/ RAT Store SCA data command From ALCT CLCT pre-trigger logic Comparators CMS Meeting CSC Timing

40. Also • TMB DAV delay is trivial (TMB/DMB are a board pair) • TMB L1A delay - this parameter is predicted well: CMS Meeting CSC Timing

41. Also • ALCT L1A delay is predicted well (a few outliers) CMS Meeting CSC Timing

42. Next section: BX numbers in online synchronization • Nothing here yet. • Besides BC0 check in SP data record, which electronics can check bx alignment? • TMB has a “sync error” to compare ALCT vs. TMB bunch crossing or BC0… CMS Meeting CSC Timing

43. Data records: bx numbers and offline timing • BX numbers should align for data • Offline timing: what defines the position of data for: • CFEB comparator hits • CLCT • AFEB hits • ALCT • CFEB ADC data CMS Meeting CSC Timing

44. Conclusions • Would be great to do BC0 test at peripheral crate • More thinking required about possibility of BC0 alignment at ALCT • The rest of the talk is a first stab at documentation and needs much updating and refinement •  More people are starting to look at CSC timing • Documentation is therefore a high priority • More effort is needed to address the set of  items listed earlier CMS Meeting CSC Timing

45. That’s it for now • (lots of) backup slides follow CMS Meeting CSC Timing

46. TMB-ALCT block diagram Crate Master clock TMB TMB Master clock ALCT section Latch input ALCT data • Adjust ALCTtx for optimal latching of ALCT output data at TMB 2ns/bin Latch output ALCT commands ALCT -RX clock CCB test pulse commands • Adjust ALCTrx for optimal latching TMB output data at ALCT ALCT -TX clock TMB pass- through 2ns/bin ALCT commands ALCT data ALCT ALCT Master clock ALCT Main FPGA • Adjust Delay ASICs for max. probability for ALCTs to come in one BX Internal test pulse via VME command to TMB Asynch. test pulse from VME write to CCB Delay ASICs ALCT latch raw data Synch. test pulse from TTC command or VME write to CCB Main FPGA OR ~2.2ns/bin AFEB CSC Test Pulse Strips AFEB data Test pulse to AFEB amplifier or test strips (select via VME write to TMB to ALCT Slow Control FPGA register)

47. TMB-CFEB block diagram TMB Crate Master clock TMB Master clock Latch data in CLCT section Comp. delay Data Delay Devices 2ns/bin 40 MHz clock Comparator data CFEB (1 of 5) • Clock and data on same 6-15m Skewclear cable • Adjust comparator clock phase to middle of ~12ns window where data is latched correctly by TMB Comparators CMS Meeting CSC Timing

48. TMB-MPC Block Diagram TMB MPC … Crate Master 40 & 80 MHz clocks TMB Master clock MPC Master 40 &80 MHz clocks DLL makes 80 MHz, phase=0 80 MHz VCX0 Winner Bits 40 MHz Optical to Track Finder at 80 MHz Select Winner bits pointer MPC Master phase ALCT-CLCT-RPC coincidence logic LCT Readout Controller 0.25 ns/bin Sort best 3/18 logic LCTs at 80 MHz on backplane Clk. Mult. to 80 MHz 40-to-80 MHz MUX Latch and de-mux TMB LCT data • MPC Latch Delay is adjusted to middle of latch window for data from all 9 TMBs. • Winner bits come back to TMB about 8 clock cycles after LCTs sent to MPC (Pointer to data in pipeline should be fixed for all time) • Phase of Winner bits to TMB may need adjustment on TMB end. CMS Meeting CSC Timing

49. TMB-DMB Block Diagram TMB DMB CFEB TTC/CCB Crate Master Clock, L1A • (external L1A = LHC & Test Beam operation modes) TMB Master Clock, L1A DMB Master Clock, L1A CLCT DAV delay L1A*CLCT DAV Coinc. CLCT-DAV LCT-L1A Coinc. starts TMB readout CLCT Readout queue CLCT FIFO DMB-DDU readout Controller logic ALCT DAV delay L1A*ALCT DAV Coinc. ALCT-DAV LCT -read delay LCTs to MPC From ALCT readout queue ALCT FIFO CFEB DAV Coinc. CFEB Clock phase L1A delay L1A ALCT/ CLCT/ RPC Coincidence RPC delay 1 ns/bin CFEB DAV delay CFEBs “hit” Auto set Cable Equal. delay AFF (Active FEB Flags) Cable Equal. delay CFEB FIFOs (5) CFEB -DAV CLCT Final logic AFF-L1A Coinc. Starts CFEB digi. & readout RPC logic AFF delay ALCT delay Output FPGA fixed CFEB Clock SCAs, ADCs, Memories From RPC/ RAT Store SCA data command From ALCT CLCT pre-trigger logic Comparators CMS Meeting CSC Timing

50. Master Clock Distribution inCSC Peripheral Crates CCB TTC Phase adjustment 0.1 ns/bin (unused so far) TTCrx Crate Master clock: Isochronous backplane distribution … TMB 1 DMB 1 TMB 9 DMB 9 MPC • TTC command and data strobes are delayed along with the phase adjustment so as to remain within 25ns latch window CMS Meeting CSC Timing