Project Omniglean

1. Project Omniglean Team 09 CPR E CPR E CPR E SE Joe Briggie Kenny Trytek Abby Birkett Derek Woods Advisor: SimantaMitra Client: Kingland Systems

2. Project Description What Kingland Does: Master Copy • Hierarchy Master Data Management – Use many sources of information to determine which company owns whom to resolve stock ownership conflicts. Problem: • FFIEC and FDIC websites take a long time to traverse and update information manually. • Create a way of automatically “crawling” the sites and collecting the information contained therein.

3. Requirements and Specifications • Automatically traverse the FDIC and FFIEC websites • Store the results in a database • Expose the database to clients through web services • Provide “infinitely searchable” data in a user interface for client • Utilize a DAL to maintain a normalized database

4. Project Plan • Plan out basic features of the project • Create prototypes that implement the basic features • Revise requirements, continue with second iteration providing additional features • Revise requirements, final iteration • Verify final product’s functionality with client • Request client signoff

5. Analyst User Omniglean Internet Harvest & Store Client User Project Design • Omniglean mediates between the information and the client. • It harvests thedata and stores it in a database accessible through web services. • The project was broken into interchangeable modules for rapid development. • The modules are: harvester, Export, Transfer, and Load tool, Database Access Layer, database, web services, and Graphical User Interface. WWW

6. DAL Database ETL Tool Normalized External Client UI Kingland Data Analyst UI Web Svcs. No Conflicts? Project Design Scraper Tool WWW Data HTML Parser PDF Parser • The harvester module collects the data and passes it to the ETL. • The ETL transforms the data for the DAL. • The DAL does the work of storing the data in the database. • Web services expose the data to a client-facing UI. • The interface allows searching, organizing, and maintaining the data. Flat File Create Read Update Delete

7. Implementation - Harvester Parse HTML & container files to produce flat file. • FFIEC Website • Harvester runs on a daily schedule to get updates. • Harvester traverses FDIC and FFIEC websites, as well as PDF files contained therein. • Harvester outputs XML file readable by ETL tool. • Throws exception if the internet connection is unavailable to retrieve the updates, triggering alert for the system administrator to fix the connection. • PDF Files Flat File (XML) • FDIC Website • Institutions2.csv

8. Flat File (XML) ETL Tool DAL Implementation – ETL Tool • Extract data from XML files (created by Harvesters) • Transform data and load into database (via DAL) • Communicates w/ DAL via business logic objects Implementation: Testing Results: • Able to read data from XML documents • Able to load data into database • Need to check for malformed XML

9. Implementation – User Interface Implementation: • Allow “infinite searchability” on data • Currently prototyped

10. Database DAL Add() Find() Update() Delete() ETL Tool User Interface Implementation – DAL & Database • DAL is nearly complete • Need to add error checking • Database is created and normalized Implementation: Testing Results: • Successfully able to create, read,update, and delete records fromthe database

11. Semester Plan • Complete prototypes for each module • Integrate modules into a single system • Review/revise requirements with the client • Enhance prototypes with additional functionality • Confirm features are complete with the client • Request client signoff