1 / 22

Computer Systems and Big Data Analysis

Computer Systems and Big Data Analysis. Motivating Examples. “Data is very important. The world in the future will be dominated by data”. Ma Yun “数据非常重要 , 未来的世界是数据的世界”。 马云 Guess which provinces are bikini best sold in China. Guangdong, Hainan? No….

fayre
Download Presentation

Computer Systems and Big Data Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Computer Systems and Big Data Analysis

  2. Motivating Examples • “Data is very important. The world in the future will be dominated by data”. Ma Yun • “数据非常重要,未来的世界是数据的世界”。 马云 • Guess which provinces are bikini best sold in China. • Guangdong, Hainan? • No…. • According to Taobao, there are Xinjiang and Inner Mongolia. • Explanation: Each man have told his wife/lover/girl friend that he would take her swimming in the sea. • Orbitz is a ticket-booking website. After data analysis, they found that customers’ ticket prices are related to their web browser: Safari highest, Chrome and Firefox similar. • They adjust the strategy accordingly. The Safari user will be given expensive tickets first.

  3. What Is Big Data? • There is not a consensus as to how to define big data “Big data exceeds the reach of commonly used hardware environments and software tools to capture, manage, and process it with in a tolerable elapsed time for its user population.” - Teradata Magazine article, 2011 “Big data refers to data sets whose size is beyond the ability of typical database software tools to capture, store, manage and analyze.” - The McKinsey Global Institute, 2011

  4. Where Is This “Big Data” Coming From ? 4.6 billioncamera phones world wide 30 billionRFID tags today (1.3B in 2005) 12+ TBsof tweet data every day 100s of millions of GPS enableddevices sold annually ? TBs ofdata every day 2+ billionpeople on the Web by end 2011 25+ TBs oflog data every day 76 millionsmart meters in 2009… 200M by 2014

  5. With Big Data, We’ve Moved into a New Era of Analytics 12+ 5+ million terabytes of Tweets create daily. trade eventsper second. Volume Velocity Variety Veracity 100’s Only1 in 3 of different types of data. decision makers trust their information.

  6. 3 Vs of Big Data • The “BIG” in big data isn’t just about volume

  7. Four Characteristics of Big Data Cost efficiently processing the growing Volume Responding to the increasing Velocity Collectively Analyzing the broadening Variety 30 Billion RFID sensors and counting 50x 35 ZB 80% of the worlds data is unstructured 2010 2020 Establishing the Veracityof big data sources 1 in 3 business leaders don’t trust the information they use to make decisions

  8. Big Data Analysis Example: Product arrangement • How does location tracking work? • Recognize the dead zone

  9. Usage Example in Big Data • In March 2012, The White House announced a national "Big Data Initiative" that consisted of six Federal departments and agencies committing more than $200 million to big data research projects. • PRISM is a clandestinemasselectronic surveillancedata mining program operated by the United States National Security Agency (NSA) since 2007. • It is reported that China is going to create a national policy about big data management.

  10. Why You Need to Tame Big Data • Analyzing big data is already standard (e.g. ecommerce) • Be left behind in a few years • So far, only missed the chance on the bleeding edge • Capturing data, using analysis to make decisions • Just an extension of what you are already doing today

  11. Filtering Big Data Effectively • Sipping from the hose Focus on the important pieces of the data It makes big data easier to handle

  12. Analytic With Data-In-Motion & Data At Rest Data Ingest 01011001100011101001001001001 0011010100100100100110100101010011100101001111001000100100010010001000100101 11000100101001001011001001010 01100100101001001010100010010 01100100101001001010100010010 11000100101001001011001001010 Opportunity Cost Starts Here 01100100101001001010100010010 BootstrapEnrich 01100100101001001010100010010 01100100101001001010100010010 01100100101001001010100010010 11000100101001001011001001010 01100100101001001010100010010 Forecast Forecast Nowcast 01100100101001001010100010010 01100100101001001010100010010 AdaptiveAnalyticsModel 01100100101001001010100010010 01100100101001001010100010010 11000100101001001011001001010 01100100101001001010100010010 01100100101001001010100010010 01100100101001001010100010010 11000100101001001011001001010

  13. Relational Data File Systems Content Management Email CRM Supply Chain ERP RSS Feeds Cloud Custom Sources Big Data Exploration: Value & Diagram • Find, Visualize & Understand all big data to improve business knowledge • Greater efficiencies in business processes • New insights from combining and analyzing data types in new ways • Develop new business models with resulting increased market presence and revenue Application/ Users DataExplorer

  14. Operations Analysis: Value & Diagram Indexing, Search Only store what is needed Statistical Modeling Raw Logs and Machine Data Root Cause Analysis Real-time Analysis Machine Data Accelerator Federated Navigation & Discovery

  15. The Need for Standards • Become more structured over time • Fine-tune to be friendlier for analysis • Standardize enough to make life much easier

  16. Today’s Big Data Is Not Tomorrow’s Big Data • Banking industries were very hard to handle even a decade ago • “BIG” will change: • Big data will continue to evolve

  17. Go to 'View > Header and Footer' to change this footer text to the event title IBMCase : How Computers Make Big Data dream to come true Built-In Expertise systems for Big Data analysis • Dedicated device • Optimized for purpose • Complete solution • Fast installation • Very easy operation • Standard interfaces • Low cost

  18. BigInsights and the data warehouse Traditional analytictools Big Data analytic applications From Cognos BI via Hive JDBC BigInsights • Query-ready archive for “cold” warehouse data Data Warehouse

  19. Analyze Streaming Data

  20. Analytic Applications BI / Reporting Exploration / Visualization FunctionalApp IndustryApp Predictive Analytics Content Analytics Visualization & Discovery Application Development Systems Management Accelerators HadoopSystem Data Warehouse Stream Computing Information Integration & Governance The Platform Advantage BI / Reporting IBM Big Data Platform

  21. How Much the Big Data Analysis Enhanced by IBM Project of T-Mobile Czech Rep. RESPONSE TIME MASSIVELY IMPROVED

  22. Resource • Ömer Sever (omers@tr.ibm.com) IBM SWG TR • Martin Pavlík (Martin_pavlik@cz.ibm.com) cz.ibm.com • iDB: Internet Database Lab

More Related