Ay 119: Methods of Computational Science

1. Ay 119:Methods of Computational Science S. G. Djorgovski et al. Spring 2012

2. The Evolving Paths to Knowledge • The First Paradigm: Experiment/Measurement • The Second Paradigm: Analytical Theory • The Third Paradigm: Numerical Simulations • The Fourth Paradigm: Data-Driven Science

3. Exponential Growth of Data Volumes … and Complexity on Moore’s law time scales Understanding of complex phenomena requires complex data! From data poverty to data glut From data sets to data streams From static to dynamic, evolving data From anytime to real-time analysis and discovery From centralized to distributed resources From ownership of data to ownership of expertise

4. Transformation and Synergy • We are now in the second phase of the IT revolution: the rise of the information/data driven computing • In addition to the traditional numerically-intensive science • IT as a primary publishing and communication technology • All science in the 21st century is becoming cyber-science (aka e-Science) - and with this change comes the need for a new scientific methodology • The challenges we are tackling: • Management of large, complex, distributed data sets • Effective exploration of such data  new knowledge • These challenges are universal • A great synergy of the computationally . enabled science, and the science-driven IT

5. A Modern Scientific Discovery Process • Data Gathering (e.g., from sensor networks, telescopes…) • Data Farming: • Storage/Archiving • Indexing, Searchability • Data Fusion, Interoperability • Data Mining (or Knowledge Discovery in Databases): • Pattern or correlation search • Clustering analysis, automated classification • Outlier / anomaly searches • Hyperdimensional visualization } Database Technologies Key Technical Challenges Key Methodological Challenges • Data Understanding • New Knowledge +feedback

6. Information Technology  New Science • The information volume grows exponentially Most data will never be seen by humans! The need for data storage, network, database-related technologies, standards, etc. • Information complexity is also increasing greatly Most data (and data constructs) cannot be comprehended by humans directly! The need for data mining, KDD, data understanding technologies, hyperdimensional visualization, AI/Machine-assisted discovery … • We need to create a new scientific methodologyon the basis of applied CS and IT • Important for practical applications beyond science

7. Astronomy Has Become Very Data-Rich • Typical digital sky surveys generate ~ 10 - 100 TB each, plus a comparable amount of derived data products • PB-scale data sets are imminent • Astronomy today has ~ a few PB of archived data, and generates ~ 10 TB/day • Both data volumes and data rates grow exponentially, with a doubling time ~ 1.5 years • Even more important is the growth of data complexity • For comparison: Human Genome < 1 GB Human Memory < 1 GB (?) 1 TB ~ 2 million books Human Bandwidth ~ 1 TB / year (±)

8. … And It Will Get Much More So Large Synoptic Survey Telescope (LSST) ~ 30 TB / night Square Kilometer Array (SKA) ~ 1 EB / second (raw data) (EB = 1,000,000 TB)

9. There Are Lots Of Stars In The Sky… Modern sky surveys obtain ~ 1012 – 1015 bytes of images, catalog ~ 108 – 109 objects (stars, galaxies, etc.), and measure ~ 102 – 103 numbers for each

11. Numerical Simulations: A qualitatively different and necessary way of doing theory, beyond the analytical approach Theory is expressed as data, an output of a numerical simulation, not as a set of equations … and then must be matched against complex measurements

12. The Evolving Data-Rich Astronomy From “arts & crafts” to industry From data subsistence to an exponential overabundance Astronomy is now driven by the progress in information technology t2 ~ 1.5 yrs Synoptic sky surveys: from Terascale to Petascale data streams Telescope+instrument are “just” a front end to data systems, where the real action is

13. The Cyber-Infrastructure Movement (or e-Science) (aka “The Atkins Report”)

14. The Rise of Virtual Scientific Organizations Data Archives Compute Resources Analysis Tools • A grassroots response of scientific communities to the challenges and opportunities brought by the data glut • Domain-specific, not institution-based; inherently distributed • The human, data, and compute resources are distributed • A new type of a scientific organization • Virtual Observatory: a complete research environment for astronomy with massive and complex data sets

15. The Virtual Observatory Concept • A complete, dynamical, distributed, open research environment for the new astronomy with massive and complex data sets • Provide and federate content (data, metadata) services, standards, and analysis/compute services • Develop and provide data exploration and discovery tools • Harness the IT revolution in the service of astronomy • A part of the broader e-Science /Cyber-Infrastructure

16. Virtual Observatory Is Real! http://us-vo.org http:// ivoa.net http://www.euro-vo.org

17. Virtual Observatory Science Examples Combine the data from multi-TB, billion-object surveys in the optical, IR, radio, X-ray, etc. • Precision large scale structure in the universe • Precision structure of our Galaxy • Discover rare and unusual (one-in-a-million or one-in-a-billion) types of sources • E.g., extremely distant or unusual quasars, new types, etc. Match Peta-scale numerical simulations of star or galaxy formation with equally large and complex observations … etc., etc.

18. Understanding the Cosmic Microwave Bgd. and its Foregrounds Integrated SZ Grav. Lensing Sachs-Wolfe CMB Signal Galactic Thermal Gal. Nonthermal Galaxies (SF) Radio Sources

19. Parameter Space Exploration in Catalog Domain • In most surveys, image/pixel data are reduced to catalogs, using some kind of a processing pipeline, which detects sources and measures their attributes/parameters on the basis of position, flux, and light distribution • Nowadays typically we measure up to a few hundred parameters per source per survey • Typically we detect ~ 108 - 109 sources per survey • Data federation from different surveys increases these numbers • Data are then vectors in parameter spaces of hundreds of dimensions • They generally do not populate this space uniformly, but define clusters and correlations • Their description and analysis leads to scientific discoveries, and is also useful for quality control purposes

20. Parameter Spaces: Clustering Analysis • How many different types of objects are there? • Which ones are identifiable with known, physically distinct types (e.g., stars, galaxies, quasars at different redshifts, etc.)? • Are there rare and/or previously unknown classes, seen as outliers or distinct classes? • Are there intermediate or transition types? • Are there gaps (negative clusters)? • Anomalies possibly indicative of problems with the data?

21. A powerful, expandable, web-based user-friendly data mining service for astronomy Brought to you by: Currently at: http://dame.caltech.edu http://dame.dsf.unina.it

22. The Sky Is Also Flat • Professional Empowerment: Scientists and students anywhere with an internet connection would be able to do a first-rate science (access to data and tools) • A broadening of the talent pool in astronomy, leading to a substantial democratization of the field • They can also be substantial contributors of data and ideas, not only consumers • Riding the exponential growth of the IT is far more cost effective than building expensive big telescopes Probably the most important aspect of the IT revolution in science

23. Beyond Virtual Scientific Organizations: • The Rise of X-Informatics (X = Astro, Bio, Geo, ..) • Domain-specific amalgam fields (science + CS + ICT) • A mechanism for a broader community inclusion (both as contributors and as consumers) • A mechanism for interdisciplinary e-Science methodological sharing

24. The key role of data analysis is to replace the raw complexity seen in the data with a reduced set of patterns, regularities, and correlations, leading to their theoretical understanding However, the complexity of data sets and interesting, meaningful constructs in them is starting to exceed the cognitive capacity of the human brain

25. The Uses of Machine Intelligence:Science on the Carbon-Silicon Interface • Data mining, analysis, and understanding: • Clustering, classification, outlier / anomaly detection • Pattern recognition, hidden correlation search • Assisted dimensionality reduction for visualization • Workflow control in Grid- or Cloud-based apps • Data farming and data discovery:semantic web, etc. • Code design and implementation:from art to science? • Data processing: • Automated object / event classification, pattern recognition • Automated data quality control (anomaly/fault detection and repair)

26. The Fourth Paradigm Redux Is this really something qualitatively new, rather than the same old data analysis, but with more data? • The information content of modern data sets is so high as to enable discoveries which were not envisioned by the data originators • Data fusion reveals new knowledge which was implicitly present, but not recognizable in the individual data sets • Complexity threshold for a human comprehension of complex data constructs? Need new methods to make the data understanding possible Data Fusion + Data Mining + Machine Learning = The Fourth Paradigm

27. Some Thoughts About e-Science • Computational science ≠ Computer science • Data-driven science is not about data, it is about knowledge extraction (the data are incidental to our real mission) • Information and data are (relatively) cheap, but the expertise is expensive • Just like the hardware/software situation • Computer science as the “new mathematics” • It plays the role in relation to other sciences which mathematics did in ~ 17th - 20thcentury • Computation: an interdisciplinary glue/lubricant • Many important problems (e.g., climate change) are inherently inter/multi-disciplinary

28. The Revolution in Scholarly Publishing • Increasing complexity and diversity of scientific data and results • Data, archives, metadata, virtual data, simulations, algorithms, blogs, wikis, multimedia… • From static to dynamic: evolving and growing data sets • From print-oriented to web-oriented • Institutional, cultural, and technical challenges: • Curationby domain experts • Effective peer review and quality control • Persistency and integrity of data and pointers • Interoperability and metadata standards • From the ownership of storage media to the ownership of access to the bits As the science evolves, so does its publishing

29. Science in Cyberspace Theory and Simulations