1 / 13

FAX Deployment, Service and Storage Integration

FAX Deployment, Service and Storage Integration . Wei Yang. Overview. FAX Components and Services Redirector, LFC and monitoring Infrastructure Sites deployment Status Use cases Panda Cloud On-going Integration with Storage systems R&D Activities. Infrastructure and Services.

tana
Download Presentation

FAX Deployment, Service and Storage Integration

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. FAX Deployment, Service and Storage Integration Wei Yang US ATLAS Distributed Facility Meeting University of California Santa Cruz

  2. Overview FAX Components and Services • Redirector, LFC and monitoring Infrastructure • Sites deployment Status Use cases • Panda • Cloud On-going Integration with Storage systems R&D Activities US ATLAS Distributed Facility Meeting University of California Santa Cruz

  3. Infrastructure and Services • A Network/Tree of Redirectors • Allow a user to start from anyway and reach everywhere • Multiple levels of redirectors • Top level: EU & BNL • Country level: DE, FR, RU, UK • Regional level: US central (hosted by UC) • Site level: UC, SLAC • Read-only LFC services • Hosted by BNL (for US sites) and CERN (for EU sites) • Monitoring Data Collectors • Collect and send monitoring data to ATLAS dash board • Site specific/unique file for validation US ATLAS Distributed Facility Meeting University of California Santa Cruz

  4. http://ivukotic.web.cern.ch/ivukotic/FAX/index.asp BNL and EU redirectors are peers at top level due to network latency US ATLAS Distributed Facility Meeting University of California Santa Cruz

  5. The Monitoring Services • Availability Dashboard • Current running at UC, will be migrated to ATLAS SSB • Detail Monitoring Collector • A.K.A UCSD collector, collect info on every read • Aggregated info file level access info • Send to ATLAS monitoring dashboard via ActiveMQ • Summary Monitoring Collector • Based on MonaLisa, aggregated at data server level • Info used to compare with detail info and debugging • ATLAS Monitoring Dashboard for FAX • Integrate with AGIS US ATLAS Distributed Facility Meeting University of California Santa Cruz

  6. https://uct3-xrdp.uchicago.edu:8443/rsv/ US ATLAS Distributed Facility Meeting University of California Santa Cruz

  7. From Julia Andreeva FAX Dashboard and ML FAX repository comparison FAX Dashboard now includes EOS, which dominates over all other transfer/access This plot is showing overall traffic rate over last 12 hours group by source , excluding CERN (EOS) Aggregated xrootd traffic rate over last 12 hours according to FAX ML repository, excluding MWT2_UC and SLAC which are missing in Dashboard In general is a good agreement, as well as going site by site. Big progress over last couple of weeks

  8. http://dashb-atlas-xrootd-transfers.cern.ch/ui US ATLAS Distributed Facility Meeting University of California Santa Cruz

  9. Site Deployment • 8 sites in the US (all sites) • 4 sites in the UK • 3 sites in DE • 2 sites in RU • 1 site in Prague, CZ • working with IT cloud https://twiki.cern.ch/twiki/bin/view/Atlas/FaxSiteCertification US ATLAS Distributed Facility Meeting University of California Santa Cruz

  10. Use Cases • Interactive Access from Desktop/Laptop • Xrdcp or ROOT/ProofLite • From Panda Jobs • Prun: supply a list of files in global name • Panda pilot support • Phase I: replace missing files using FAX • See Paul’s talk. Expanding testtomore Tier 2 sites • Phase 2: use site cost matrix for job scheduling • Phase 3: beyond, a lot more opportunities … See Torre’s talk • By the Cloud • FAX is a nature choice for jobs in the Cloud to consume data • Inbound data traffic is free/low cost (outbound is expensive) • No need for long term storage in the Cloud US ATLAS Distributed Facility Meeting University of California Santa Cruz

  11. Storage System Integration • Have solutions for almost all ATLAS systems • Basic idea: • A dedicated xrootd machine to help the site joining FAX either as a helper, refer client to the site storage or a proxy, fetch data from site storage on client’s behave • Translate global file name to site storage file name • Support POSIX (NFS, Lustre, GPFS, etc.), Xrootd (including EOS), dCache, DPM • Working on Castor (RAL) • Support • tWiki and mailing list • https://twiki.cern.ch/twiki/bin/viewauth/Atlas/AtlasXrootdSystems • atlas-adc-federated-xrootd@cern.ch • Bi-weekly Vidyo meeting on deployment issues • Experts in the US for general Xrootd and dCache support • UK/DPM team support DPM integration to FAX • Some sites are creative and self support (EOS) • Cloud level support: e.g. DE and UK clouds US ATLAS Distributed Facility Meeting University of California Santa Cruz

  12. R&D • Driven by feature request/Operation feedback • Deployment and Operation are the focus • But some level of R&D is still needed for a while • Have experts in many R&D area in US and EU • R&D provides • New functions/features, e.g. f-stream • bug fixes • New models for site and ADC specific needs US ATLAS Distributed Facility Meeting University of California Santa Cruz

  13. Federated Xrootd deployment timeline …more dCachedev …new monitoring stream & integration issues As always, the docs could Be better From Rob Gardner

More Related