1 / 45

Engineering Reports Day 1

Engineering Reports Day 1. November 2017. EN Reports. Day 1 Build 8a Release Tool Working Group Day 2 Digital Object Identifiers PDS Home Page Discussion Search Discussion 2018 Tech Session. Build 8a Release. November 2017. Executive Summary.

floresw
Download Presentation

Engineering Reports Day 1

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Engineering Reports Day 1 November 2017

  2. EN Reports • Day 1 • Build 8a Release • Tool Working Group • Day 2 • Digital Object Identifiers • PDS Home Page Discussion • Search Discussion • 2018 Tech Session

  3. Build 8a Release November 2017

  4. Executive Summary • Information Model V1.9 and core software services and tools were delivered to I&T on September 29; SR & DPH documents posted for V1.9 • I&T Completed on 10/23/17 • Deployment to EN Completed on 11/10/17 • V1.9 and software available to nodes, IPDA, and public

  5. Build 8a I&T Process SCRs (approved by CCB) IM V1.9.0.0 candidate Issues (in JIRA) Test Data Build 8a Software Build 8a Test Plan (test cases) IM I&T IM V1.9.0.0 Build 8a System I&T Issues (in JIRA) Build 8a Test Report

  6. Build 8a Artifacts

  7. I&T Input/Output • Input • IM V1.9.0.0 as a result of Node testing of candidate IM based on approved SCRs by CCB • Build 8a system software including upgrades and fixes of issues captured in JIRA • Build 8a test plan with modified test cases for new features and regression test • Build 8a test data • Output • New issues found have been added to JIRA • Fixed issues were closed in JIRA • Test result has been documented in the test report here (see next page)

  8. Updated Test Cases • PRP.1 – Added transformations to csv and PDS3 files • PRV.1 - Replaced test cases that used obsoleted schemas  • PRV.7 - Added steps to test more of validate -R's functionality • SRCH.3 – Added steps to test search-analytics • HVT.1 – Added steps to test harvest-search

  9. I&T Results • Total of 51 test cases exercised and passed, no failure. • 97 requirements met. • 0 major issues found, no show stopper. • 3 minor issues and 3 suggested improvements created in JIRA. All 3 minor issues were fixed and regression tested during I&T. • Test report and data posted at: https://pds-engineering.jpl.nasa.gov/content/build_8a_deliverables

  10. Details of Information Model Changes • CCB-162 - Move <md5_checksum> from Object_Statistics to Byte_Stream • CCB-165 - Ambiguity of ASCII_Numeric_Base • CCB-166 - Deprecate bit_mask from IM • CCB-171 - Split hardware-compatible ASCII numeric types from the unbounded • CCB-172 - Add movie format(s) to list of acceptable documents. • CCB-176 - Update Node names in the IM • CCB-178 - Internal Reference Implementation • CCB-179 - In PDS4 'pds' namespace, use ASCII_NonNegative_Integer • CCB-184 - The xs:choice element does not validate as expected • CCB-185 - Allow multiple instances of purpose and processing_level • CCB-191 - Add Indian Space Research Organisation (ISRO) to the Namespace Registry • CCB-192 - Add Product_Metadata_Supplemental (M.Gordon) • CCB-193 - Add Geosciences and Radio Science as permissible values SCRs that did not impact the IM • CCB-180 - Clarify whether namespace abbreviation in labels must correspond to namespace_id • CCB-181 - Clarify whether newlines are allowed in PDS DSV 1 • CCB-182 - Clarify whether CDATA is allowed • CCB-183 - Rejected

  11. PDS4 Information ModelDiscipline and Mission Dictionaries

  12. Build 8a Software Components Tools Services Registry and Search Includes support for Tool and Data Dictionary content Transport Backward compatible with the PDS-D Product Server Report Generating monthly HQ reports • Ingest • Harvest Tool • Catalog Tool • Preparation • Generate Tool • Transform Tool • Validate Tool • Core and PDS4 Tools Libraries • Inspect Tool (Prototype)

  13. Major Software Changes • Validate Tool • Added support for data content validation of tables against the label description • Transform Tool • Added support for transforming PDS3 labeled tables to CSV • Added support for transforming PDS4 label to PDS3 label (ODL) • Inspect Tool • Initial release supporting visualization of PDS4 products • Tracking Service • Initial release supporting API access to tracking services. • Search Service • Added performance improvement for index generation.

  14. Deployment • Build 8a has been successfully deployed at the EN early November. • V1.9 and software available to nodes, IPDA, and public

  15. Backup

  16. Summary of Information Model Changes • Common dictionary – Implemented 13 approved SCRs • Detailed list in backup. • CCB-184 requires the regeneration of LDDs (LDDTool) • CCB-178 further restricts the values for <local_identifiers> • Cartography LDD ingested • Ingested Rings LDD updated • Files exported from IM for tools and services. • Added “abstract” and “deprecated” flags to the JSON file • Local Data Dictionary (LDD) changes. • Ingested new version of Rings LDD - V1501_1800 • Spectral LDD released – V1800 • Spectral Library LDD released – V1000

  17. Software Change Detail • New Capabilities • [PDS-243] - Generate: Add support to use Product Tools library • [PDS-495] - PDS4-Tools: Add URL support[PDS-506] - Transform: Add support for transforming PDS3 tables to CSV • [PDS-507] - Transform: Add support for transforming a PDS4 label to a PDS3 label • [PDS-508] - Validate: Add support for validating data content of table products • [PDS-517] - Tracking: Develop a service-based interface for tracking product status • Modified Capabilities • [PDS-504] - Validate: Improve Operation guide examples[PDS-505] - Generate/Transform: Update PDS4 label transformation for the latest model • [PDS-509] - Search: Separate the protocol implementation into a separate component • [PDS-511] - Transform: Add support for reading targets from URLs • [PDS-512] - Transform: Integrate the tool with the Tools Service • [PDS-519] - Generate: The tool outputs an unexpected warning • [PDS-521] - Validate: The -R pds4.bundle option cancels -e "*.xml" • Corrected Capabilities • [PDS-467] - Core: Support passing in Schema jar URLs • [PDS-478] - Search: XSLT variable resolution is not working on the Windows platform • [PDS-501] - Core: Support bundle.xml and collection.xml file names to trigger their respective rule types • [PDS-515] - Validate: Handshake failure occurs when trying to validate against schemas located on the PDS machines • [PDS-523] - Transform: transformation of character table product to PDS3 label fails

  18. I&T Improvements Found

  19. PDS Tool Working Group(PDS-TWG) November 2017

  20. Agenda • Status Overview • Requirements • Tool Activities • Tool requirements and release plans • Inspection • Transformation • Validation • Open Source • Container Deployments • Next Steps Tool Working Group

  21. Status Overview • Since last MC F2F, the working group has focused on: • Final requirements and priorities for inspection • Discussion of Node Tool • Exploration of deployment options • Discussion of open source approach • Continuing to populate the Tool Registry Tool Working Group

  22. Requirements • Updated Level 1/2/3 requirements were approved April 2017 by MC and are posted • Individual requirements for validation, transformation, and inspection have been prioritized by the TWG • Mapped to builds Tool Working Group

  23. Tool Activities • The team is tracking status for actively developed tools at the EN and the DNs • A traceability matrix is maintained to map Level 3 requirements to tools • EN has been discussing core tools • Nodes were asked to begin presenting DN tools • Nodes were asked to identify tools beyond the list (on-going request) • This is facilitated with the following Google doc: • https://docs.google.com/spreadsheets/d/18oqtg3DEo2KrgvBOWLSOuqF2uZtq2XmByJwUknYSZUQ/edit#gid=1340660799 Tool Working Group

  24. PDS Maintained Tool Status Tool Working Group

  25. Inspection • The TWG has been discussing the inspection tool • The Inspect Tool will provides support for visualizing PDS4 products • Eventually planned to replace NASAView • Requirements have been reviewed by the TWG • Comments are being incorporated into an updated version of the document • This was facilitated with the following Google doc: • https://docs.google.com/spreadsheets/d/18oqtg3DEo2KrgvBOWLSOuqF2uZtq2XmByJwUknYSZUQ/edit#gid=807742626 • Summary shown on backup slide • An initial version has been included in Build 8a and will be tested by the TWG Tool Working Group

  26. Transformation • Today, the Transform Tools supports approximately 29 transformations as of the last Build release (8a) • The team was tasked with adding to and prioritizing the list of supported transformations for the Transform Tool • This was facilitated with the following Google doc: • https://docs.google.com/spreadsheets/d/18oqtg3DEo2KrgvBOWLSOuqF2uZtq2XmByJwUknYSZUQ/edit#gid=635102082 • Summary shown on backup slide 20 Tool Working Group

  27. Transformation Releases Plan • The prioritization exercise identified the following as higher priority (assigned to builds): • Build 8a (just released) • PDS3 labeled tables to CSV • PDS4 label to PDS3 label (ODL) • Build 8b • PDS3 labeled tables to PDS4 labeled tables • PDS4 labeled tables to PDS4 labeled tables (e.g., binary to character) • Build 9a • Array 2D Map to GeoTIFF • Array 3D Spectrum to ENVI Cube • Array 2D Image to FITS Tool Working Group

  28. Validation • The Validate Tool supports the following as of the Build 8a release: • Syntactic and semantic validation via the XML Schema and 350+ Schematron rules pertaining to PDS4 label structure and content • Bundle and Collection referential integrity checking • Data content validation of tables against the label description • The team was tasked with prioritizing the validation requirements • These focus primarily on specific content validation rules, many of which need to be individually implemented and scheduled for release • This was facilitated with the following Google doc: • https://docs.google.com/spreadsheets/d/18oqtg3DEo2KrgvBOWLSOuqF2uZtq2XmByJwUknYSZUQ/edit#gid=1268101288 • Summary shown on backup slide 21-23 Tool Working Group

  29. Validation Release Plan • Upcoming priorities (from TWG discussions): • Build 8b • Extended support for XML Catalog • Data content validation of arrays against the label description • Build 9a • Verify complex elements confrom to naming rules • Verify simple elements confrom to naming rules • Verify file names conform to naming rules • Verify schema location accessibility Tool Working Group

  30. PDS Validate Tool Support Tool Working Group

  31. Upcoming Tool Topics • Search Tools – Tracking and integration of search tools across PDS. • Migration Tools and Support – Tracking and planning of PDS-wide tools and approaches to support PDS3 to PDS4 migration needs.

  32. Open Source • All software related to validation has been approved for release to open source. • This includes the PDS3 and PDS4 validation libraries as well as the PDS4 Tools library. • These packages will be migrated to the nasa-pds project [1] on GitHub. • Plan to have source code migrated in February 2018 and discuss at the Tech Session [1] https://github.com/orgs/nasa-pds/ Tool Working Group

  33. Container Deployments • EN is investigating the use of container technology for deploying PDS4 software • Mainly focused on PDS4 services and not necessarily tools • Containers are different from virtual machines (see next slide) • Goal is to simplify deployments at the EN as well as the Nodes • Docker is a technology commonly in use at JPL and will be used in the first prototype • Will be included in tech session Tool Working Group

  34. VMs vs Containers Tool Working Group

  35. Next Steps • Coordinate use and governance rules around open source • Schedule search tools discussions • Schedule migration tools discussions • Continue scheduling DN tool discussions • Evaluate container technologies to improve software installations • Develop PDS3 tool maintenance plans Tool Working Group

  36. Questions/Comments Tool Working Group

  37. Inspection Requirements Tool Working Group

  38. Transformation Prioritization Tool Working Group

  39. Validation Prioritization Tool Working Group

  40. Validation Prioritization cont. Tool Working Group

  41. Validation Prioritization cont. Tool Working Group

  42. Charter • Charter • Maintaining the PDS Level 3 requirements for tools; • Reviewing and overseeing the PDS-wide tool inventory; • Identifying gaps and overlaps in tools; • Recommending tool priority and phasing for development; • Reviewing and commenting on specific tool requirements; • Maintaining PDS-wide Tool Schedule and Plan; • Supporting beta testing of tools as part of a release cycle; • Hosting a tool summit with the PDS technical group; and • Provide regular reports at the Management Council Face-to-Face meetings. Tool Working Group

  43. Members • D. Crichton, Chair (E. Law as alternate) • E. Guinness, Geosciences • Joni Johnson, Atmospheres • Todd King, PPI • Jordan Padams, IMG (Moses Milazzo as alternate) • Eric Palmer, PSI • Tanya Lim, PSA * Sean Hardman, EN as ex-officio advisor Tool Working Group

  44. Tool Registry • The application is available at: • http://pds.nasa.gov/tools/tool-registry • Migrations of the PDS Search Tools and IPDA Tools have been completed. • Beyond migrated entries, approximately 28+ new entries have been captured (current total of 75). • A couple new entries in the last couple of months. • Submissions have been reviewed by the TWG. Tool Working Group

  45. Types of PDS4 Validation Tool Working Group

More Related