Collaborative Online Embryology: The Visible Embryo Project

1. Collaborative Online Embryology:The Visible Embryo Project We are building a 10 PetaByte digital library to support education, research and clinical planning In embryology Presenters* andAuthors: Dr. Bill Lennon* Lawrence Livermore National Laboratory Dr. Adrianne Noe,* Elisabeth Lockett Armed Forced Institute of Pathology Dr. Mark Pullen* George Mason University Dr. Jeff Pentecost,* Dr. Kent Thornburg Oregon Health and Sciences University Dr. Mike Doyle,* Dr. Morey Piscatelli Eolas Inc Dr. Mark Holterman* University of Illinois Chicago Medical Center Dr. Reagan Moore San Diego Supercomputing Center Dr. Chuck Paidas, Robert Morreale Johns Hopkins University Medical Center Crew: George Pavel, Jeff Olsen, Brian Bodtker, and Paul Atwal Lawrence Livermore National Laboratory

2. OHSU Eolas UIC LLNL Johns Hopkins NIH NGI Supernet & Internet II NMHM /AFIP GMU NTON SDSC LSU This project has been made possible by high performance networks Source materials, archives, computing resources, and team members are located throughout the country.

3. .ppt TV TV TV TV TV TV TV TV TV TV TV TV “Global Grid” .ppt .ppt .ppt “Internet 2,” ESNET et. al. Audio conference Our team is using the global grid to present and demonstrate our collaboration NSF Noe* Lockett Paidas Argonne Doyle* Holterman* SC 2001 Lennon* Pullen* Moore LLNL Pentecost* Macaw Macaw Macaw • Application Window multicast on feed: • LLNL ACN Screen N.B. a “Macaw” is an NT 2000 workstation 11/01

4. Outline Application Demonstration Window multicast on feed: “LLNL ACN Screen” “Presentation feeds” will be “queued” over the “MUD” communication channel Presenters principal Site Topic Dr. Bill Lennon Denver, CO Structure of Presentation 5 Lawrence Livermore National Laboratory Dr. Adrianne Noe Arlington, VA Program Motivation 5 Armed Forced Institute of Pathology Dr. Mark Pullen Denver, CO Project Overview & Data Flow 7 George Mason University Dr. Jeff Pentecost Livermore, CA Research & Image Annotation 5 Oregon Health and Sciences University (& Collaboration Demo) Dr. Mike Doyle Argonne, IL Visualization 5 Eolas Inc. (& Collaboration Demo) Dr. Mark Holterman Argonne, IL Medical School Education 5 University of Illinois Chicago Medical Center & Clinical Planning Dr. Bill Lennon and Denver, CO Questions and Answers. 5 Dr. Mark Pullen With presenters, authors and team members in Arlington, Argonne, Denver, Livermore, and San Diego.

5. Project motivation Dr. Adrianne Noe Director, National Museum of Health and Medicine Deputy Director, Armed Forced Institute of Pathology Site: NCSA Access site (across the street from NSF) Presenter feed: ACCESS

6. National Library of Medicine NGI FundingApplications-DrivenEducationClinicalResearchCollaboratory model

7. Human Developmental Anatomy CenterNational Museum of Health and MedicineArmed Forces Institute of PathologyWashington, DC

8. Major General Daniel E. Sickles (and his limb)

9. Serial Sections

10. 2X Slide View Regions Of Interest

11. Registration • Fiducial markers on bromide images made at time of specimen preparation

12. Project Overview & Data Flow Dr. Mark Pullen George Mason University Professor of Computer Science Principal Investigator for Visible Embryo Project Site: SC2001 Presenter feed: SC Global Showcase

13. Visible Embryo Project Overview • Deploy a system of visualization/ collaboration workstations using high performance networking • Digitize embryo data from the Carnegie Collection • the basis for collaboration experiments. • Demonstrate the system in three medical collaborations: • annotation and modeling • embryology education • clinical management planning • The NGI digital library project is supported by NIH/NLM. • We are in our third year, emphasizing the medical applications -- Objective: 3 or more sites collaborating

14. Visible Embryo Project Participants • GMU - overall direction, collaboration technology • Eolas - technical coordination, software • AFIP/NMHM - data acquisition • SDSC - data repository and rendering • LLNL - network facilitation • OHSU, UIC, JHU - medical demonstrations

15. LSUMC OHSU AFIP Eolas UIC NLM LLNL GMU JHU & 11/01 SDSC Visible Embryo Project -- Current Connections Internet (Genuity+@home+BellSouth+ AlterNet+XONet) LANet Pacific Northwest GigaPOP MAX MREN CalREN-2 ESnet Abilene Net.Work. Virginia vBNS

16. Top level Data Flow to build digital library • Scan tiles @ 20X. • Combine into sections. • Annotate structures within sections. • Align sections. • Collaborate using visualization software. • e.g. Build 3D and “4D” models. • Capture all notes, etc. into archive during library use. Data capture station

17. Macaw flatbed scanners low power 20X 20X Origin 200 and RAID array local archiving NLM SDSC Data Capture and Storage Resource Broker (SRB) archiving • Imagewhere?TM software runs as a client on individual 20x capture stations. • At completion watchdogs: • detect and archive (index) all images. • move from local drive to RAID array. • use “marked up” 2X section locator images on slides and XML document to verify the files. • SRB Ingestion • SRB Ingestion scripts direct files to SDSC creating all appropriate directories. • After verification, files on capture RAID are erased. N.B. ingestion rates to SDSC approx. 4.8 MB/s

18. Team users and data base experts collaborated on the definition of the Metadata Catalog for the Archive at SDSC Dimen-unit Embryo related objects Embryo morphology Dimen-type Many-to-one Obj-type Used everywhere Used everywhere One-to-many Embryo physical Cr_activity Embryo_core Token Keys: Reserved words Material Slide physical Supports Person-Info Slide stain Stain Cr_activity Slide_core Slide desc Technique Condenser Microscope Anno structure Section_core Section image-core Annotation Lens Camera Image-filter Image-info physical zmap Resolution Anno Field Section Subjects Tile_core Tile image-core image Classification Labels Subject RM 3/01 Filter Image-type

19. SRB - Web Interface WAN API & CGI GMU/ AFIP EOLAS Windows NT Unix SRB - Windows Client Client Browser SRB - Windows Interface Scommands - Loading NTBrowser - Viewing Network Web Server SRB - Java SRB - CGI Sun - Ultra Sun - Ultra Browser Scripts LAN Images SRB SRB/MCAT Archived HPSS Server Server Here (MDA - 18) (TORAH) IBM - SP2 IBM - SP2 Sun - E10K Images Cached Here ARCHIVAL Metadata File System DEMBRYO EMCAT CACHE Metadata ) DATABASE FILE SYSTEM for (Oracle for & Integration MCAT Visual (Oracle) Embryo Data archive configuration at SDSC

20. AFIP web site serves as distribution point for external access to the Digital Embryo collection. • Data can be retrieved one at a time or in batch mode through scripts • AFIP web site serves as distribution point for external access to the Digital Embryo collection. • Data can be retrieved one at a time or in batch mode through scripts To Access Image Data (9/01 Approx. 155 GB stored – 26,500 images – both uncompressed and JPEG)

21. Collaboration Workstation (MACAW) Functions • Image annotation • 3D reconstruction indexed by the annotations • Collaborative annotation and data exploration • Facilitate educational applications • Scriptable interfaces to tools and data • Environment for clinical consultations • Be able to take a holistic view in context of development, at every stage of development

22. MACAW implementation • Equipment: • High-end Windows Pentium III workstation with • 1 GB RAM, 2x18 GB disk, backup tape, • video camera and mike, • scanner, graphic entry tablet • Appropriate network interface • (OC3 and up – IP/Sonet, ATM or Enet) • Software: • Operating system (Windows 2000)/utility(GroupKit) • Data access (streaming) • Existing collaboration (MBone tools) • Visualization/rendering (runs at SDSC) • Project developed software Today: Online workstation at LLNL. Laptops at ANL and SC2001.

23. Collaborative Image annotation (& Demonstration) Dr. Jeff Pentecost Oregon Health and Sciences University Heart Research Center Site: LLNL Presenter feed: LLNL ACN Presenter Other feeds: LLNL ACN Screen (application window multicast stream) SC Global Showcase ( Mark Pullen) ANL (Mike Doyle)

24. Create archives of tagged image data for visualization: • every voxel can be associated with many labels. • e.g. the organ system to which it belongs, the structure and tissue types represented, etc. • Applications: • Anatomic site mapping of gene expression • Analysis of embryonic cardiac hemodynamics • Integration of 3D graphics with symbolic knowledge for education, diagnosis, and treatment. Project Medical Applications

25. Initial review of serial images • Initial review of serial images • Variation of magnification • Registration • Segmentation • Symbolic data: nomenclature scheme Overview of Annotating Cardiopulmonary Anatomy Fiducial markers on bromide images made at time of specimen preparation

26. Allows two or more users to cooperatively annotate image(s) over the network • Full event synchronization among users • Users can collaboratively annotate structures within the image • Synchronization of file loading and metadata state • Define image areas using irregular polygons, circles and rectangles, to associate with metadata & unique object IDs • Session “catch up” for new users joining existing sessions Collaborative Image Browser/Annotator

28. Nomenclature Scheme • One goal of collaborative annotation is a logically organized database of embryologic terms • We adhere to such a scheme, although this is in itself a research area of which we are a part. • Collaborating with Cornelius Rosse and Jim Brinkley at the University of Washington, we are establishing a model semantic framework for naming embryologic structures For example: • embryo => organ system => cardiovascular system => heart =>primitive ventricle => future left ventricle • embryo => organ system => cardiovascular system => heart =>atrio-ventricular cushion tissue => ependymal cushion tissue =>bulbar cushion • embryo => organ system => cardiovascular system => heart =>bulbus cordis => rostral half =>conotruncus

29. Collaborative Visualization (& Demo) Dr. Mike Doyle Eolas Site: Argonne National Laboratory Presenter feed: LLNL ACN Screen (application window multicast) LLNL ACN Presenter (collaborator Jeff Pentecost) SC Global Showcase (collaborator Mark Pullen)

30. GroupVisualization Allows multiple users over the net to explore 3D annotated datasets • Visualization and knowledgebase query sessions are synchronized among users • Will be evolved into a platform for group consultations • Will be integrated with the annotation tools to facilitate gene expression mapping on canonical datasets

31. 3D Group Visualization • Synchronized control of rotation, slicing, and object-voxel queries

32. 3D Group Visualization • Forms a morphological matrix for spatial mapping of gene expression activity

33. Medical School Education & Clinical planning Dr. Mark Holterman University of Illinois at Chicago Medical Center Site: Argonne National Laboratory Presenter feed: ANL

34. Embryology Education Goals: • Improve the overall comprehension of human development • Provide the framework for the generation of new insights into human development • Recruit talented people into the field of neonatal health and development

35. Target Audience: • Embryology Educators and Researchers • AACA/BACA/AAA • Physicians in training- • UIC, Rush, MCW, Albert Einstein • Medical specialists • Perinatology/Pediatrics/Surgery • AAP/APSA Assembled team: • embryology educators: UIC, Rush, MCW, Albert Einstein • animators at UIC and JHU and MCW • clinical team: cardiology, surgery, pathology • medical students Education thrust:

36. Progress on Education Application Existing AFIP data sets at 2X • models • software techniques • animation • first three weeks Educational modules • interactive web based • feedback • testing and evaluations

37. Progress on Education Application Existing AFIP data sets at 2X • models • software techniques • animation • first three weeks Educational modules • interactive web based • feedback • testing and evaluations

38. Progress on Education Application Existing AFIP data sets at 2X • models • software techniques • animation • first three weeks Educational modules • interactive web based • feedback • testing and evaluations

39. Progress on Education Application Existing AFIP data sets at 2X • models • software techniques • animation • first three weeks Educational modules • interactive web based • feedback • testing and evaluations

42. Future Plans • Develop complete educational package for foregut, urogenital and cardiac systems • Framework of web-based educational tool with textual and clinical information enriched with animations, models and clinical images • Explore virtual reality modeling of embryos

43. Comparing ultrasound images with library will support clinical planning Carnegie Stage 18/7 wks

44. Questions Dr. Bill Lennon LLNL Dr. Mark Pullen GMU Site: SC2001 Presenter feed: SC Global Showcase Other feeds: LLNL ACN Presenter ANL ACCESS