1 / 35

The National Virtual Observatory & Big Computing in Astronomy Roy Williams Caltech

The National Virtual Observatory & Big Computing in Astronomy Roy Williams Caltech NVO Co-Director. Outline. What is the VO Standard services Registry of services Palomar-Quest survey Griffith observatory “big picture” Big data Services Transforming Real-time astronomy

elmo
Download Presentation

The National Virtual Observatory & Big Computing in Astronomy Roy Williams Caltech

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The National Virtual Observatory & Big Computing in Astronomy Roy WilliamsCaltech NVO Co-Director

  2. Outline • What is the VO • Standard services • Registry of services • Palomar-Quest survey • Griffith observatory “big picture” • Big data • Services • Transforming • Real-time astronomy • VOEvent and VOEventNet

  3. What is the Virtual Observatory… • A set of international standards to share complex data • A modular set of tools to work with distributed data • An essential part of the research astronomer’s toolkit • A catalyst for world-wide access to astronomical archives • A vehicle for education and public outreach • NOT a replacement for building new telescopes and instruments • NOT a centralized repository for data • NOT a data quality enforcement organization

  4. Discover Compute Publish Collaborate Portals, User Interfaces, Tools Topcat VOPlot SkyQuery OASIS Mirage DIS Aladin conVOT interfaces to data Registry Layer Data Services Compute Services HTTP Services SOAP Services Grid Servicesstateless, registered & self-describing & persistent, authenticated Semantics (UCD) crossmatch visualization Bulk Access OAI ADS image data mining source detection OpenSkyQuery SIAP, SSAP FITS, GIF,… VOTable Virtual Data Digital LibraryOther registriesXML, DC, METS Workflow (pipelines) Authentication & Authorization Existing Data Centers VOSpace storage services Grid MiddlewareSRB, Globus, OGSASOAP, GridFTP Databases, Persistency, Replication Disks, Tapes, CPUs, Fiber

  5. Compute Services Registry Data Services

  6. VOTable • Full metadata representation • Hierarchy of RESOURCEs • containing PARAMs and TABLEs • UCD (unified content descriptor) • a has unit meter • a has UCDORBIT_SIZE_SMAJ (Semi-major axis of the orbit ) • Can reference remote and/or binary streams • Table can be • Pure XML • "Simple Binary" • FITS Binary Table

  7. Cone/SIAP/SSAP • Simple, pragmatic solutions • quickly Specified, Created, Registered, Utilized! • Cone • request is cone, response is VOTable with RA, Dec • many of these since 2/02 • SIAP • request is cone, response is VOTable of image links • SSAP • under development

  8. Simple Image Protocol • Specify box by position and size • SIAP server returns relevant images • Footprint • Logical Name • URL Can choose: standard URL: http://....... SRB URL srb://nvo.npaci.edu/…..

  9. Quasar ScienceAn NVO-Teragrid projectPennState, CMU, Caltech • 60,000 quasar spectra from Sloan Sky Survey • Each is 1 cpu-hour: submit to grid queue • Fits complex model (173 parameter) • derive black hole mass from line widths clusters NVO data services globusrun manager

  10. Source Extraction and Crossmatch Krughoff, Connolly http://us-vo.org/apps

  11. David Hockney Pearblossom Highway 1986 Image Mosaicking • Delivers custom, science grade image mosaics • An image mosaic is a combination of many images containing individual pixel data so that they appear to be a single image from a single telescope or spacecraft • User specifies projection, coordinates, spatial sampling, mosaic size, image rotation • Preserve astrometry (to 0.1 pixels) & flux (to 0.1%) • Public service will be deployed at Caltech • Order mosaics through web portal • Larger mosaics for more authentication

  12. Example Portal:Astronomical Image Mosaicking

  13. Service Lifecycle VOResource + VOStdService + WSDL Registry 1. publish fill in forms standard service types 2. find by white, yellow, green pages Provider Client 3. bind: request & response

  14. Registry Services • Publish • Caltech, NCSA, STScI registries, many international • Query • ADQL • XQuery/XPath • Harvest • OAI from NCSA, Caltech, JHU, Vizier (France) • What entities are described by registry? • Service • VO standard or arbitrary • Extension schema: eg VOEvent server • Project, Data Collection • (person, community, VD object, etc etc ...?)

  15. Authorization WSDL Description Service Semantics CURATION VOResource A mandatory form plus other supporting forms

  16. Distributed IVOA Registry Astrogrid CfA NCSA CDS ESO STScI/JHU NOAO ESAC Caltech HEASARC JapanVO NRAO IVOA registries as of Apr 05 Can resolve VOEvent identifier Redirect to specific VOEvent Database that has it

  17. Registry OAI Query Registry Registry OAI OAI Publish Publish Data Inventory Service • Federates multiple cone, SIAP services JHU/StSci NCSA 4 Caltech Goddard DIS 2 1 3

  18. Data Inventory Service Relevant Images and Catalogs NVSS Image ROSAT catalog

  19. Palomar-Quest

  20. Palomar-Quest SurveyDjorgovski, Baltay, Brunner, Mahabal, Graham, Williams Transient pipeline computing reservation at sunrise for immediate followup of transients Synoptic survey massive resampling for ultrafaint detection of hi-Z quasars P48 Telescope 50 Gbyte/night ALERT Caltech Yale TG  NCSA NCSA and Caltech and Yale run different pipelines on the same data 5 Tbyte

  21. Griffith Observatory“big picture” 158 feet 158 feet 158 feet Show tile b14 Show stacking and cleaning

  22. Big Data Services

  23. Big Data Services Big sky surveys online at Caltech with NVO protocol DPOSS 3TB 2MASS 9 TB PQ 13 TB Request from client http://nvo.caltech.edu/cgi-bin/dposs.cgi? POS=185,12&SIZE=1 Manager service nvo.caltech.edu Metadata service (ipac.caltech.edu) Data service tg-www.cacr.caltech.edu 80 TB Caltech datawulf

  24. Big Data Services • Multi-layer PQ data in seconds • Data stored in containers • All disk, no tape! • Zip containers (not tar) • By observation date • Image files built dynamically • Astrometric information • Cleaning Query: data from sky region 2004 09 19 Data: by observation date 2004 01 16 2005 03 26 2004 06 12

  25. Transforming Big Data • Services should be idempotent (Acting as if used once, even if used multiple times) • Source Target model • For all sources, check target for existence, age • Data transformation is “like a makefile” • Unpack containers on scratch disk tiny metadata server • Copy container • Unpack and transform • Pack up into container • copy back to mass store Mass store:PVFS, GPFS, SRB etc local scratch

  26. DPOSS flattening Source Target 2650 x 1.1 Gbyte files Cropping borders Quadratic fit and subtract ~20 minutes on 1 node

  27. VOEvent WorkshopAn Information Infrastructurefor Immediate Astronomical Eventsa Working Group of theInternational Virtual Observatory AllianceApril 13-14, 2005 Roy WilliamsCaltech andUS National Virtual Observatory 1604 image of Cas A

  28. Transient from PQ

  29. Exploration of the Time Domain … … and the advent of Synoptic Sky Surveys An example (from DPOSS) of a new type of a phenomenon which may be discovered in a systematic exploration of the Time Domain: A normal, main-sequence star which underwent an outburst by a factor of > 300. There is some anecdotal evidence for such megaflares in normal stars. The cause, duration, and frequency of these outbursts is currently unknown.

  30. Faint, Fast Transients (Tyson et al.) Some flaring M-stars, some extragalactic, …  A heterogeneous population!

  31. A Key Contaminant: Slow-Moving Asteroids (less of a problem if observing near the meridian/oposition) Asteroid Normal PSF  Coordination with the NEAT survey

  32. <VOEvent type="update" id="ivo://raptor.lanl/235649409/sn2005k" xmlns:stc="http://www.ivoa.net/xml/STC/stc-v1.20.xsd" xmlns:crd="http://www.ivoa.net/xml/STC/STCcoords/v1.20"> <Who> <Publisher> ivo://raptor.lanl/ </Publisher> <Date>2005-04-15T14:34:16</Date> </Who> <WhereWhen> <stc:ObservationLocation> <crd:AstroCoords coord_system_id="FK5-UTC"> <crd:Time unit="s"> <crd:TimeInstant> <crd:Timescale>UTC</crd:Timescale> <crd:ISOTime>2005-04-15T23:59:59</crd:ISOTime> </crd:TimeInstant> </crd:Time> <crd:Position2D unit="deg"> <crd:Value2>148.888 69.065</crd:Value2> <crd:Error1Circle><crd:Size>0.1</crd:Size></crd:Error1Circle> </crd:Position2D> </crd:AstroCoords> </stc:ObservationLocation> </WhereWhen> <What> <Param name="magnitude" ucd="phot.mag:em.opt.R" value="13.2" /> </What> <Why> <Classification probability="30" units="percent" type="ot"> Fast Orphan Optical Transient </Classification> </Why> </VOEvent> Simplest VOEvent

  33. <VOEvent type="update" id="ivo://raptor.lanl/235649409/sn2005k" xmlns:stc="http://www.ivoa.net/xml/STC/stc-v1.20.xsd" xmlns:crd="http://www.ivoa.net/xml/STC/STCcoords/v1.20"> <Who> <Publisher> ivo://raptor.lanl/ </Publisher> <Date>2005-04-15T14:34:16</Date> </Who> <WhereWhen> <stc:ObservationLocation> <crd:AstroCoords coord_system_id="FK5-UTC"> <crd:Time unit="s"> <crd:TimeInstant> <crd:Timescale>UTC</crd:Timescale> <crd:ISOTime>2005-04-15T23:59:59</crd:ISOTime> </crd:TimeInstant> </crd:Time> <crd:Position2D unit="deg"> <crd:Value2>148.888 69.065</crd:Value2> <crd:Error1Circle><crd:Size>0.1</crd:Size></crd:Error1Circle> </crd:Position2D> </crd:AstroCoords> </stc:ObservationLocation> </WhereWhen> <What> <Param name="magnitude" ucd="phot.mag:em.opt.R" value="13.2" /> </What> <Why> <Classification probability="30" units="percent" type="ot"> Fast Orphan Optical Transient </Classification> </Why> </VOEvent> Simplest VOEvent

  34. VOEventNet RAPTOR LANL (binocular) Palomar 48” (survey) Palomar 60” (follow-up) PAIRITEL Berkeley (infrared)

  35. VOEventNet: a Rapid-Response Telescope Grid VOEvent database eStar GRB satellites Palomar-Quest PQ next-daypipelines baselinesky Raptor catalog Palomar 60” PQ Event Factory Event Synthesis Engine VOEventNet Pairitel SDSS 2MASS known Variables known asteroids remote archives

More Related