1 / 8

Crowdsourcing in Life Science Research

Crowdsourcing in Life Science Research. Divya Mistry 4/22/2011. Crowdsourcing. Crowd + Outsourcing Outsourcing a task or its subparts to individuals in a crowd who aren’t part of research group. Crowd consists of volunteers replying to “open call”

tiana
Download Presentation

Crowdsourcing in Life Science Research

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Crowdsourcing in Life Science Research DivyaMistry 4/22/2011

  2. Crowdsourcing • Crowd + Outsourcing • Outsourcing a task or its subparts to individuals in a crowdwho aren’t part of research group. • Crowd consists of • volunteers replying to “open call” • individuals, who are paid a small amount • Indirectly gathering live data from humans (e.g. twitter) • Wisdom of the crowd • Answering question through collective opinion instead of just an expert.

  3. Crowdsourcing in Science • 2 popular examples • Passive contribution • Active contribution • What they have in common • Cost of entry is practically zero. • Purpose of the project is understandable to common person • Burden of analysis/verification is not on participant

  4. Tools for Crowdsourcing • Paid Participation • Amazon Mechanical Turk • Define a goal with granular tasks • Recruit the crowd • Pay per response/task • Kaggle, InnoCentive, Netflix Prize • Competition based • Only winner gets paid • Unpaid Participation • Social networks • Web-based game • CMS’ and Wiki to track offline tasks • Utility tracking user behavior

  5. Examples • SCORE++. (Paid) • Finding r.o.i. in microscopy images • Recruit crowd from AMT at a low cost • Individuals manually define r.o.i. Consensus truth regions are found by comparing all responses.

  6. Examples • Phylo. (Unpaid) • Tetris/Bejeweled-like game format • Players match and sort blocks of similar color in short time. • Goal is to get better multiple sequence alignment

  7. The “Zero-Curation” Model • Technical Difficulties / Lessons Learned • iCAPTURer, iCAPTURer 2 • Skewed participation curve, i.e. few individuals do most of the work • Gaps in expertise. ~66% correct responses on given task • Highly biased towards one of the answers, e.g. “yes”. • Benefits • Cost advantage • Humans are amazing gamers • Engaging population in science

  8. References • Good et al. 2006. Fast, Cheap, and Out of Control: A Zero-Curation Model for Ontology Development. Pac SympBiocomput. 2006:128-39. • Good and Wilkinson. 2007. Ontology engineering using volunteer labor. Proceedings of the 16th Intl Conf on World Wide Web. • Snow et al. 2008. Cheap and Fast - But is it Good?: Evaluating Non-Expert Annotations for Natural Language Tasks. Proceedings of the Conference on Empirical Methods in Natural Language Processing. • SETI@Homehttp://setiathome.berkeley.edu/ • Folding@Homehttp://folding.stanford.edu/ • Galaxy Zoo http://www.galaxyzoo.org/ • AMT https://www.mturk.com/ • Kagglehttp://www.kaggle.com/ • InnoCentivehttps://www.innocentive.com/ • Netflix Prize http://www.netflixprize.com/ • ITK http://www.itk.org/ • SCORE++ proposal http://www.itk.org/Wiki/images/a/a4/Megason-A2D2proposal-SCORExx.pdf • Phylohttp://phylo.cs.mcgill.ca • iCAPTURer presentation http://www.slideshare.net/goodb/bio-logical-mass-collaboration3

More Related