1 / 30

Team members Description of the TE-MPE Stand-By Service Statistics 2010 + 2011 SEU

TE-MPE Stand-By Service 2010 – 2011 lessons learnt Gert Jan Coelingh on behalf of TE-MPE Stand-By team,. Team members Description of the TE-MPE Stand-By Service Statistics 2010 + 2011 SEU DQHDS main switch “IN THE SHADOW” Personal touch Extrapolate 2011 – Improvements?

kieve
Download Presentation

Team members Description of the TE-MPE Stand-By Service Statistics 2010 + 2011 SEU

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. TE-MPE Stand-By Service 2010 – 2011 lessons learntGert Jan Coelingh on behalf of TE-MPE Stand-By team, Team members Description of the TE-MPE Stand-By Service Statistics 2010 + 2011 SEU DQHDS main switch “IN THE SHADOW” Personal touch Extrapolate 2011 – Improvements? Conclusion & Outlook

  2. Team Members • Reiner is responsible of the service, and Gert-Jan is the linkman • Stand-By members: In 2011, 8 members, with specific knowledge: • Giorgio D'Angelo  EI  ELQA & Heating Syst. C.L • Richard Mompo  EI  ELQA & Heating Syst. C.L • Grzegorz Seweryn  EI  ELQA & QPS Heating Syst. C.L • Steven Pemberton  EI  ELQA & Heating Syst. C.L • Kevin Priestnall  CP  QPS – Retirement 2011 • Vincent Froidbise  CP  QPS • Joaquim Mourao  CP  QPS • Gert-Jan Coelingh  CP  600A EE, 13 kA & QPS • Help of Mathieu Favre, Noel Fournier and Bozhidar Panev (CP)

  3. Team Members: Experts • Specialists: support for Stand-By members Very limited specialists (in number  ): • Knud Dahlerup-Petersen (EE & QPS) • Reiner Denz (QPS) • Zinur Charifoulline (nQPS informatics). • Jens Steckert (nQPS). • CP section: Other specialists on Stand-By • Current leads specialist all specialists on Stand-By  Now under EI responsibility;. • DFB specialists: are part of Stand-By members, Mateusz Bednarek since November 2010!! • MP3 members Stand-By • All Experts are on BEST-EFFORT

  4. MPE Stand-By mandate: • 1. Mission: • To put back in service, as rapid as possible, a faulty installation within the framework (QPS, Energy Extraction Systems and electrical components of the DFBs) during LHC run. • 2. Actions: • Intervene after a call from the Control Centre CERN (CCC), • Diagnostics and if possible solve the problem remotely • Organize the intervention with the operator/EIC CCC (RP) • If needed ask the “Expert” (if reachable) for support • Archive the intervention in the Logbook (QPS Logbook). • 3. Responsibility: • He must assure his own security as well as those he asked helping him and assure the correct way of intervening in the tunnel • The technician is responsible for the intervention. If he cannot find a solution he must inform the group hierarchy.

  5. MPE Expert mandate (1/2) The Expert (based on best effort) : • 1. Mission: • Support the first line intervention in case of complex problems • 2. Actions: • If judged intervene to help the Stand-By technician • Ask another Stand-By team member to join for intervention • 3. Indirectly related actions: • Decrease LHC machine down-time where possible • Implement solutions for problems • Firmware upgrades • Preventive maintenance • DQHDS main switch exchange campaign

  6. MPE Expert mandate (2/2) • 3. Indirectly related actions (cont.): • Not to be underestimated in hours and workload • Example: Arkadiusz Gorzawski developed an interface logging database – 600A EE signals - permits preventive maintenance on breakers with too many closing failures - Currently extended with intervention history per system Direct result = no breaker exchange so far this year HUGE EFFORT not visible in Stand-By statistics

  7. MPE Stand-By organization: • Documentation: • Each equipment specialist prepared a documentation on how to solve a typical problem. • An updated copy is always available in two Stand-By cars • Regular meeting and training: • Weekly meeting on Monday morning to transmit the “Piquet role” and share experience over the past week. • Training are organized in order to be as multidisciplinary as possible. • Two Stand-By cars available, equipped with tooling, docs and spare parts. • Storage place in Build.30, 281 and 936 (in front of CCC) • Yearly planning: On our repository, we have a yearly planning with foreseen vacations, official holidays, night and day “Stand-By” turn-over. • Day calls: transparent to CCC (same number 16.8801), assures intervention during normal hours. This allow night piquet to recover. • Night calls: outside normal hours and w-e, official holidays included.

  8. Intervention Flowchart Emergency call from CCC to Stand-By duty: Stand-By: 16.8801 Information exchange via telephone Stand-By / CCC operator Remote Analysis Difficult problem or Time needed > 3 hours ? Y Problem Solved remotely? Stand-By / CCC Y Call equipment Specialist or Responsible TE/MPE N Fill in QPS Logbook N Stand-By person to analyze max. delay (1hour) End of intervention Repair and restart the equipment Repair and restart the equipment or, feedback situation with CCC. Need more than one Person? Y Assistance via: Stand-By support Operator CCC* N Fill in QPS Logbook End of intervention *(en dehors des heuresouvrablesuniquement )

  9. TE-MPE Stand-By service for the LHCGert Jan Coelingh on behalf of TE-MPE Stand-By team,with inputs from R. Denz Team members Description of the TE-MPE Stand-By Service Statistics 2010 + 2011 SEU DQHDS “IN THE SHADOW” Personal touch Improvements Conclusion & Outlook

  10. Some statistics from 2010/2011: (1/..) • Total number of interventions (Remote / Local) and Total duration 219 hours 140 Interventions 87 Interventions 87 hours

  11. Some statistics from 2010/2011: (2/..)

  12. Some statistics from 2010/2011: (3/..)

  13. Some statistics from 2010/2011: (4/..)

  14. Some statistics from 2010: (5/..)

  15. TE-MPE Stand-By Service 2010 – 2011 lessons learntGert Jan Coelingh on behalf of TE-MPE Stand-By team, Team members Description of the TE-MPE Stand-By Service Statistics 2010 + 2011 SEU DQHDS “IN THE SHADOW” Personal touch Improvements Conclusion & Outlook

  16. SEU – increasing number • 17 treated by Stand-by team so far this year • Symptoms: QPS_OK lost (fill can be continued until beam dump) • continuous sending PM files or loss of communication • Action: • locally power cycle the crate • (and depending on time of intervention) • Update locally the firmware or • change board for one with new firmware) • Specialists actions: upgrade of firmware

  17. TE-MPE Stand-By Service 2010 – 2011 lessons learntGert Jan Coelingh on behalf of TE-MPE Stand-By team, Team members Description of the TE-MPE Stand-By Service Statistics 2010 + 2011 SEU DQHDSmain switch “IN THE SHADOW” Personal touch Improvements Conclusion & Outlook

  18. DQHDS main switch • 18 cases treated by Stand-by team so far • 26 treated in 2010 – we are NOT better in number but much better in time of intervention; almost always IN THE SHADOW causing no downtime to LHC operation • 1650 (26%) out of 6200 have been replaced • All DQHDS of all MQs done and regular campaigns on-going • Software interlock adapted to MB. One out of four DQHDS of a MB can be down. No immediate intervention needed. • Why not better in numbers? … • plastic is getting too hard?? We should hurry up • No faults in the new switches

  19. TE-MPE Stand-By Service 2010 – 2011 lessons learntGert Jan Coelingh on behalf of TE-MPE Stand-By team, Team members Description of the TE-MPE Stand-By Service Statistics 2010 + 2011 SEU DQHDS “IN THE SHADOW” Personal touch Improvements Conclusion & Outlook

  20. « IN THE SHADOW » • 30 out of 87 interventions “IN THE SHADOW” • less downtime for LHC operations • Mainly because • of upgraded Interlocking software in case of a MB DQHDS failure. • SEU can finish the fill before power cycling the crate • Preventive maintenance • Coordination between Equipment Responsible and Machine Coordination in case of longer downtime

  21. TE-MPE Stand-By Service 2010 – 2011 lessons learntGert Jan Coelingh on behalf of TE-MPE Stand-By team, Team members Description of the TE-MPE Stand-By Service Statistics 2010 + 2011 SEU DQHDS “IN THE SHADOW” Personal touch Improvements Conclusion & Outlook

  22. Personal Touch (updated until June 20) 3 x

  23. TE-MPE Stand-By Service 2010 – 2011 lessons learntGert Jan Coelingh on behalf of TE-MPE Stand-By team, Team members Description of the TE-MPE Stand-By Service Statistics 2010 + 2011 SEU DQHDS “IN THE SHADOW” Personal touch Extrapolate 2011 – Improvements? Conclusion & Outlook

  24. Extrapolate 2011 – improvement? 189 interventions = +35% 189 hours = -14% +178% +170% +17% 0% - 41% - 21%

  25. TE-MPE Stand-By Service 2010 – 2011 lessons learntGert Jan Coelingh on behalf of TE-MPE Stand-By team, Team members Description of the TE-MPE Stand-By Service Statistics 2010 + 2011 SEU DQHDS main switch “IN THE SHADOW” Personal touch Extrapolate 2011 – Improvements? Outlook Conclusion

  26. Outlook: • Improvement ongoing: • TUNNEL INTERVENTIONS • Upgrades on iQPSfirmware on SEU • Switches DQHDS replacement – 1600 done + 500 4-9 July • Software tools on QPS • Remote restart – done • Upgrade Firmware of new Busbar detector 4-9 July • Interlocks regarding DQHDS – interventions “in-the-shadow” • Potential bottlenecks: • More radiation to equipment SUE • Resources are limited, both experts and technicians • MP3 Stand-By canceled in 201x? Who takes over? • Back-up for Stand-By team based on best effort

  27. Conclusion : • Conclusion so far as in 2010 • Very efficient: < 2 hours/intervention on site, including access time !! • High reliability of the systems • No accident ! • Stand-By service interventions seems to be decrease in numbers of hours • BUT.. • A lot of work behind the scenes to get there

  28. MPE STAND-BY on DFS • All information about the Stand-By service can be found on the DFS folder: • Dfs:\\cern.ch\Departments\TE\Services\MPE_on-call-service

  29. QUESTIONS ? THANK YOU !!! LPCT Local Power Cycle Tool The absence of safety shoes and helmet is not representative for the MPE Stand-By team

  30. 1) Routing of faulty pieces from a TE/MPE installation towards a repair workshop - - Who?: Equipment responsible When: Weekly, when lower limit is reached UA (Zone Machine Supervisée) Repair work shop TE/MPE Bat. 30 Bat. 281 Bat. 287 External Who?: MPE on call Technician Radioactif ? Quand?: Uniquement lors d’une intervention When?: Only in cases of an intervention Quand?: Uniquement lors d’une intervention ‘Picomur’ Zonede Transit TE/MPE for non radio active pieces Bat. xx Yes Broken Modules Rack Les piècessontcontrôlées par un technicien DGS/RP RR/UJ (Zone Opérationnellecontrôlée) Who?: MPE on call Technician ‘Picomur’ When?: Only in cases of an intervention Buffer zone for pieces from LHC machine Labo de Réparationdédié aux équipementsfaiblementradioactifs TE-EPC Who?: MPE on call Technician LHC Machine (Zone Opérationnelle contrôlée) When?: Only in cases of an intervention ‘Picomur’ • SécuritéRadiologiqueOpérationnelle: Lignesdirectricesgénérales • Tout matérielquittant les zones de rayonnement CERN doitêtreconsidérécommepotentiellementradioactif, • Tout matérieldoitêtrecontrôlé par DGS/RP, • SeulDGS/RP, estautorisé à déclarer du matérielcomme non radioactif, • En cas de doute, contactez un technicienDGS/RP

More Related