Accelerator Operations and Efficiency

1. Accelerator Operations and Efficiency Presentation by Roger Erickson SLAC Operations Review SLAC, June 15-16, 2004

2. Outline • Accelerator Operations Section and Matrix Organization • Communications with Stakeholders • MCC Control Room Staffing • Operator Profile and Training • Conduct of Operations: Formality of Safety Procedures • Maintenance Management • ROD and Down Time Planning • Hardware Availability

3. Accelerator Operations Section

4. Matrix Management Accelerator programs are managed through a matrix organization structure with links to: • Program Coordinators for PEP-II and Linac Pulsed Programs. • Liaison persons representing each experiment. • System Physicists for each major section of the accelerator facility. • Area Managers – one for each major section, who report to the System Physicists. • Accelerator physicists and engineers responsible for subsystems or commissioning tasks. • Technical support departments with engineering and technician support for on-call maintenance. • Accelerator Department Safety Office. • ES&H Departments, including Radiation Physics and OHP.

5. Communications with Stakeholders • 8:00 meeting every day, seven days a week, in MCC conference room. • Operations report by EOIC, including run time statistics, progress in carrying out scheduled program, descriptions of problems, etc. • Reports by representatives of major experimental user groups. • Attended by Tech Div, Accel Dept, and Operations management reps, Program Deputy, System Physicists, Area Managers, representatives of major technical support departments, and interested users; typical attendance: 45 persons. • 8:15 Maintenance meeting, five days each week. • Run by Operations Section maintenance managers. • Attended by supervisors of all maintenance groups, Area Managers, and interested physicists and engineers. • Attendance: typically 45 persons. • Accelerator Plans Meeting, weekly. • Review of operational progress, problems, and plans. • Attended by Tech Div AD and Dept Heads, Program Coordinators, Accelerator Section Heads and ADSO.

6. Control Room Staffing • Program Deputy • Duty rotates among Accelerator Department physicists. • Oversees program for 3 or 4 day period. • Not always present in control room. • Operations Engineer • 1 EOIC per shift (from group of 7), on rotating 7-week cycle. • Responsible for safely carrying out the scheduled program. • Exempt position; directs the Accelerator Systems Operators. • Must be present if accelerator is capable of producing beam. • All have experience as accelerator systems operators. • Accelerator Systems Operators Typically 3 operators per shift on rotating 5-week cycle: • Linac and sources • PEP-II rings • Secondary program, PPS control, and backup.

7. Typical Operator Profile Typical applicant is a recent graduate with a bachelor’s degree in physics . • 15 have bachelor’s degrees • 4 have master’s degrees • 4 have a technical background and extensive SLAC experience. • Accelerator Systems Operator 1 • 1-2 years of experience • Accelerator Systems Operator 2 • 2-5 years of experience • Accelerator Systems Operator 3 • 5+ years of experience • Operations Engineer • 2+ years of experience

8. Operator Training Program • Workbook and evaluation for each classification: • Accelerator Systems Operator 1 • Accelerator Systems Operator 2 • Accelerator Systems Operator 3 • Operations Engineer • ES&H Training (including RWT-II and LRCA) • Accelerator Operations Safety Procedures Training (including PPS operation, searching, interlock testing for each zone.) • US Particle Accelerator School • Downtime seminars and special projects

9. Main Control Center

10. Documentation Hierarchy

11. Safety Procedures Operators are trained to carry out certain well-defined tasks, following formally controlled procedures. Examples: • Interlock Checklists • Search Procedures • Entry and Exit Procedures • Safety Inspection Checklists Safety procedures: • undergo a formal review and approval process. • are updated as conditions change. • are available on the AD Documentation web site, although the signed originals are preserved in paper form. Completed checklists and log books are archived.

12. AcceleratorDepartment Safety Documents Accelerator Operations BAS Key and Lock Procedures Hot Sheet Accelerator Operations BAS Instructions Incident Response Procedures Alarm Response Procedures Laser Safety Procedures Arcs PPS Log Linac etc Building Emergency Plan Beam Containment System Procedures Linac PPS Log BSOIC Certification Checklists Logging Requirement for Work… BSY PPS Log MCC Building Manager Checklists CID PPS Log MCC Emergency Plan DRIP/DR PPS Log MCC Key Safe Checkout Electrical Hazard Test Procedures PEP PPS Log Emergency Notification Form PEP PPS Overview Entry and Exit Procedures PLS Keys Procedures and Log Equipment Checkout Log Positron PPS Log ESA PPS Log PPS Interlock Checklists FFTB PPS Log PPS Zone Maps Final Focus PPS Log Safety Configuration Control Fire Alarm System Reference Manual Safety Inspection Checklists Guidelines for Work in SLAC Accelerator Housings Search Procedures Gun Test Lab Safety Procedure VVS Remote Turn-on Procedure High Voltage Test Facility Safety Procedures Warning Response Procedures

13. Maintenance Management • Problems that interrupt the scheduled program trigger immediate repairs to restore operations. • 24-hour on-site staff to repair: • Mechanical utilities • Power supplies • Modulators • Cryogenics • On-call staff for: • Electrical power • Magnets and other mechanical systems • RF systems • Vacuum systems • OHP • Extensive call-in lists identify first responders, alternates, and experts for all systems.

14. Opportunistic Maintenance • Opportunistic Approach • Whenever possible, identified maintenance, repair, and installation jobs are pre-planned, but deferred until an opportunity arises. • Coordination • Daily 8:15am meeting. • Coordinated by Operations Maintenance Office. • Area Managers are advocates for their areas. • Standby Maintenance List (SML) • Jobs that are planned and can be launched on short notice. • Provides options for EOIC to take advantage of unscheduled interruptions. • Repair Opportunity Day (ROD) • Launched only when required by a driving event. • Carefully planned, and typically involve 100 or more jobs. • ARTEMIS Electronic Database

15. ARTEMIS • ARTEMIS (Accelerator Remedy Trouble Entry and Maintenance Information System) is an Oracle database with a Remedy interface, accessible over the internet by everyone involved with SLAC accelerator maintenance. • Operators report problems as they occur by making entries in ARTEMIS. • Repair work is launched immediately if the problem is inhibiting the scheduled accelerator program. • New ARTEMIS entries are reviewed at the daily maintenance meeting. Open questions are addressed and jobs are assigned. • Jobs may be assigned to the Standby Maintenance List (SML), or to the Repair Opportunity Day (ROD) list, or marked for the next long down time. • Jobs are tracked using ARTEMIS until they are finally closed.

16. Daily Maintenance Meeting

17. Review of New Problems Sample from the June 10 maintenance meeting.

18. Standby Maintenance List

19. Repair Opportunity Days • The accelerator systems are not turned off for maintenance on a routine, scheduled basis. • A ROD list of pending repair jobs is derived from the ARTEMIS database and maintained by the Operations Section. Installation and PM jobs are added as needed. • All ROD jobs are reviewed in the daily maintenance meetings. • The ROD list can be viewed over the internet. • When a “driving event” occurs, a detailed plan can be assembled and launched quickly, using the ROD list. • A typical ROD day includes about 100 distinct jobs. • Three ROD days so far in 2004.

20. Long Down Time Planning • Long Down Times usually occur once each year (although the linac hasn’t seen a long down time for two years) and typically last two to three months. • Area Managers collect job requests from system physicists, project engineers, maintenance groups, and the ARTEMIS database. • A comprehensive schedule is prepared, updated, and reviewed at weekly meetings, starting about three months before the down time. • This schedule lists each job with an assigned shop, and lists the Project or Group Manager, Responsible Task Person, and Area Manager. • Schedule specifies time and duration of each job (with 1 day time resolution), and number of persons required. • Detailed (shift by shift) schedule of pre-run testing, safety certification, and turn-on activities is prepared by the Operations Section and revised as needed on a real-time basis.

21. Maintenance Statistics • ARTEMIS entries since Jan 1989: 66,883 (as of June 14, 2004) • Total identified problems not yet resolved: 620 Many of these involve non-critical problems that were observed, but went away without clear explanation; e.g., a flow switch trip or a power supply found to be out of tolerance. • ARTEMIS entries since September 1, 2003: • 2306 new problems (8.0 per day). • 339 problems resulted in lost PEP-II run time. • 2222 problems closed (7.7 per day). • Active jobs: • SML: 83 • Next ROD: 89 • Aug-Sept down time: 109

22. Accelerator Performance Statistics Revealed failures: malfunctions resulting in lost beam time. Also called “events”. Unscheduled down time:hours lost from scheduled program due to malfunctions. Mean Time to Fail: MTTF = Scheduled beam time Events Mean Time to Repair: MTTR = Unscheduled down time Events Availability = 1 - Unscheduled down time Scheduled beam time NOTE:PEP-II aborts are not counted as downtime, unless the event is reported; i.e., unless we stop to fix something and make a database entry.

23. Unscheduled Down StatisticsJan 10, 2000 – June 29, 2003

24. Down Time Analysis Details Example: PEP-II systems, Run 3

25. Availability, MTTF, and MTTR

26. PEP-II Time Accounting by Quarter Start of Trickle Charging