1 / 5

Big Data, Big Knowledge, and Big Crowd

Big Data, Big Knowledge, and Big Crowd. AnHai Doan University of Wisconsin . The world has changed; now everything is data centric everyone collects, stores, analyzes TBs and PBs of data To manage data in this new world, need 3B technologies:

miya
Download Presentation

Big Data, Big Knowledge, and Big Crowd

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Big Data, Big Knowledge, and Big Crowd AnHai Doan University of Wisconsin

  2. The world has changed; now everything is data centric • everyone collects, stores, analyzes TBs and PBs of data • To manage data in this new world, need 3B technologies: • lot of data  need big data technologies to scale up algorithms • data is noisy, unstructured, heterogeneous  need a lot of domain knowledge to understand such knowledge is often captured in big knowledge bases • algorithms are imperfect, certain things humans do better, need humans in the loop, scale is such that there is not enough human developers  need crowdsourcing with big crowd

  3. Examples • Semantic analysis of the Twitter stream • process 3000-6000 tweets per sec, need fast data infrastructure • to recognize entities, e.g., “go giant!”, need a big KB • KB being built in real time using crowdsourcing • Product matching for e-commerce • build 500+ matchers to match products one matcher per category: toy, electronics, clothes, etc. • match 500K electronics products with 500K  need Hadoop • use a KB to match numerous synonyms: soft cover = paperback, etc. • use crowdsourcing to generate training and testing data

  4. Big Knowledge Technologies • Everyone is now building KBs • IT companies: Google, Microsoft, … • e-retailers: Amazon, Walmart, … • stodgy behemoths: Johnson Control, GE, … • tiny startups, academia, … • User communities are building KBs (e.g., biomedical) • There will be not just data centers, but also knowledge centers • KBs and tools that use such KBs • critical for understanding data (e.g., tweets) • How do we help people build KBs? Knowledge centers? • a next important direction for data integration research

  5. Big Crowd Technologies • Industry has been doing these for years • For us it’s not a fad, it’s fundamental • as data management increasingly involves semantic problems • Have gotten off to a good start (platforms / problems) • Need hands-off crowdsourcing • no developer in the loop, otherwise will not scale • e.g., crowdsourcing 500 product matching problems, one per category • Need crowdsourcing for the masses • e.g., journalist wants to match two political lists of donors • Need “grand challenges” for crowdsourcing? • e.g., something like Wikipedia?

More Related