1 / 18

Data Explorer

Data Explorer. - A data profiling tool. Agenda. Introduction Existing System Limitations of Existing System Proposed Solution Project Scope Block Diagram Implementation Technology Hardware and Software Requirements Features and Benefits Future Enhancement.

hamish
Download Presentation

Data Explorer

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data Explorer - A data profiling tool

  2. Agenda • Introduction • Existing System • Limitations of Existing System • Proposed Solution • Project Scope • Block Diagram • Implementation • Technology • Hardware and Software Requirements • Features and Benefits • Future Enhancement Data Explorer – A Data Profiling Tool

  3. Introduction (1/2) Data Profiling • Data profiling is the process of examining the data available in an existing data source (e.g. a database or a file) and collecting statistics and information about that data. • Data profiling is an analysis of the candidate data sources for a data warehouse to clarify the structure, content, relationships and derivation rules of the data.  Profiling helps to understand anomalies and to assess data quality, but also to discover, register, and assess enterprise metadata.  • The purpose of data profiling is both to validate metadata when it is available and to discover metadata when it is not.  • The result of the analysis is used both strategically, to determine suitability of the candidate source systems and give the basis for an early go/no-go decision, and tactically, to identify problems for later solution design, and to level sponsors’ expectations. Data Explorer – A Data Profiling Tool

  4. Introduction (2/2) Pourpose of Data Profiling • Find out whether existing data can easily be used for other purposes • Improve the ability to search the data by tagging it with keywords, descriptions, or assigning it to a category • Give metrics on data quality, including whether the data conforms to particular standards or patterns • Assess the risk involved in integrating data for new applications, including the challenges of joins • Assess whether metadata accurately describes the actual values in the source database • Understanding data challenges early in any data intensive project, so that late project surprises are avoided. Finding data problems late in the project can lead to delays and cost overruns. Data Explorer – A Data Profiling Tool

  5. Existing System • Initially the data Profiling activities used to be done by writing complicated SQL queries • This would be comfortable for analyst or user who knows to write SQL queries • Many of us do not know the proper syntax and format for writing SQL queries • To overcome this, Data Profiling tools were introduced • Data Profiling Tools, to a some extent overcome the limitations for writing complex queries • All types of profiling activities were not supported by the tools • User has to understand and learn how to use the tool Data Explorer – A Data Profiling Tool

  6. Development time is more. Need to understand the functionality for developing the queries. Results needs to be exported to excel or notepad for anlysis Traditional Approach Complex User Interface Limited Functionality. Setup and Installation. License Cost. Minimum Server Requirements Limitations of Existing System SQL Queries Existing Tools Data Explorer – A Data Profiling Tool

  7. Proposed Solution • Developing an Application performing all the types of profiling • Easy to use interface • Minimum system requirements • Feature to export the profiling results data to excel • Additional feature to indicate the Data Quality i.e. Data Quality Indicator • Supporting multiple Databases like Oracle 10g, Oracle 11g, MS SQL Server 2005, MS SQL Server 2008, My SQL etc Data Explorer – A Data Profiling Tool

  8. Project Scope • Keeping the Time Line and other factors in mind, the project will currently support only MS SQL Server • Also the project will have following types of Profiling: • Column Profiling • Empty Column Analysis • Null Rule Analysis • Constant Analysis • Frequency Analysis • Uniqueness Analysis • Primary/Composite Key Analysis Data Explorer – A Data Profiling Tool

  9. Business Users Management Analysis Team Data Explorer Reporting Data Profiling Central Metadata Repository Capture Issues and Notes Other Databases MS SQL Server Architecture Diagram Data Explorer – A Data Profiling Tool

  10. Implementation • The project will be implemented module wise. • Project will be having different modules. Each module will be developed individually and Unit Tested • After completion of all the modules and unit testing, all the modules will be integrated and System Integration Testing will be performed • There will be separate modules for Databases retrieval from server, Tables retrieval after selecting a database, Columns retrieval after selecting a Table • There will be separate module for each type of profiling discussed. Data Explorer – A Data Profiling Tool

  11. Implementation - Profiling Details • Column Profiling • This will help in discovering total no of records, null percentage, unique percentage, minimum and maximum value in the column, documented data type etc. • Constant Analysis • This will help in discovering those columns which has less than 4 and greater than 0 distinct values. • Null Rule Analysis • This will help in finding all the columns in a table which has 100% NULL values Data Explorer – A Data Profiling Tool

  12. Implementation - Profiling Details • Unique Analysis • This will help in finding all the columns in table which has 100% uniqueness. • Primary Key / Composite Key Analysis • It will help us to find out the possible primary or composite key columns which can be have unique combination. • Frequency Analysis • This will help in finding the no. of distinct values in the columns and the no. of time the value is repeated in a column. Data Explorer – A Data Profiling Tool

  13. Technology • Data Explorer will be developed on .NET platform using C# as a coding language. • .NET is Microsoft platform for developing advanced and Robust applications • .NET supports a wide range of library classes which eases the development efforts and hence more time can be utilized in other activities • .NET is called Language Independent Platform as it support 4 native languages and 21 non-native languages. • Native Languages are a Microsoft created languages i.e. C#. VB.Net. J#, VC++ • Non-Native or Non Microsoft Languages supported are Pearl, Ruby on Rails etc Data Explorer – A Data Profiling Tool

  14. Hardware and Software Requirements Data Explorer Software Hardware • Windows 2000/ Windows XP/ Windows Vista/ Windows 7 • Microsoft .NET Framework 3.5 • Microsoft Visual Studio 2008 • Pentium Core 2 Duo processor or above • 2 GB RAM • 20 GB HDD • Printer • Router for Internet Connection Data Explorer – A Data Profiling Tool

  15. Features • Supports multiple databases like MS SQL Server, Oracle • Different types of profiling like • Column Profiling • Constant Analysis • Unique Analysis • Null Rule Analysis • Frequency Analysis • Empty Column Analysis • Primary / Composite Key Analysis • Quickly Analyze and validate data issues Data Explorer – A Data Profiling Tool

  16. Benefits • Quick discovery of data issues • No more writing of queries to profile data • Time efficient • Shorten the implementation cycle of major projects • Improve understanding of data for the users • Discovering business knowledge • Improves data accuracy in corporate databases Data Explorer – A Data Profiling Tool

  17. Future Enhancement • Data Explorer can be further extended to support unstructured or semi-structured data like flat files, .csvfiles • It can also be extended to support other relation data bases like MS Access, MySQL, Sybase etc Time efficient • It can also be enhanced by including Data Quality reports on top of Data Quality Results • There can be mechanism to store the profiling results so that it can be used or referred later at any point of time Data Explorer – A Data Profiling Tool

  18. Thank You Data Explorer – A Data Profiling Tool

More Related