Playing GWAP with strategies - using ESP as an example

1. Playing GWAP with strategies- using ESP as an example Wen-Yuan Zhu CSIE, NTNU

2. Motivation • “Games with a purpose” (GWAP) is an innovative concept in computer science • There are a lot of GWAP systems has been created • Research on enhancing GWAP systems is scarce

3. State of the art • “Human Computation” represents a new paradigm of applications • Some problems solved by human, not computer • “Games with a purpose” (GWAP) • created by Dr. Luis von Ahn (CMU) • the most popular application

4. Our Contribution • We propose an evaluation metric for GWAP systems • We study the inner properties of the ESP game using analysis • We propose an “Optimal Puzzle Selection Algorithm” (OPSA)

5. Our Contribution (2) • We implement a quasi ESP game, ESP Lite, to demonstrate our proposed algorithms • We confirm GWAP systems are more efficient if they are designed and played with strategies

6. Outline • Introduction • Analysis • Propose our algorithm • Evaluation & Simulation • Implementation • Result • Conclusion & Future work

7. Human Computation • There is a lot of things that human can easy do that computers can not yet do • Speech recognition • Natural language understanding • Computer graphics

8. Games with a Purpose • It combine computation with game • People spend a lot of time playing games • It makes Human Computation more efficient • There are a lot of GWAP systems has been created(e.g. ESP game and Google Image Labeler)

9. What is the ESP game? Alice Bob agreement on “flower” shoe flower rocks flower

10. What is the ESP game? (2) • it is efficient • 200,000+ players have contributed 50+ million labels • each player plays for a total of 91 minutes • 233 labels/human/hour (i.e. one label every 15 seconds) • Google bought a license to create its own version of the game in 2006

11. Outline • Introduction • Analysis • Propose our algorithm • Evaluation & Simulation • Implementation • Result • Conclusion & Future work

12. Objectives • Design a metric to measure the performing of the ESP game • Design proper strategies to improve the ESP game • Validate the proposed strategies in real world systems

13. Observations • The ESP game has two goals • Quantity : the system prefers to maximize the number of images which have been played • Quality : the system prefers to take as many labels as possible for each image • There is a trade-off between the two goals

14. System metric # of labeled images average scores per labeled image # of labels # of labels per labeled image

15. Modeling(2)

16. Outline • Introduction • Analysis • Propose our algorithm • Evaluation & Simulation • Implementation • Result • Conclusion & Future work

17. Puzzle Selection Algorithms • Optimal Puzzle Selection Algorithm (OPSA) • select an image based on our analysis • Random Puzzle Selection Algorithm (RPSA) • select a image by random • Fresh-first Puzzle Selection Algorithm (FPSA) • select a image that has been played least frequently

18. OPSA • The idea is there are optimal r labels per labeled image in system • We group images into 3 sets • contains all the images that have not been played • contains all the images that have been played at least once, but less than r rounds

19. OPSA (2)

20. Outline • Introduction • Analysis • Propose our algorithm • Evaluation & Simulation • Implementation • Result • Conclusion & Future work

21. Simulation • Setup • There are 100,000 images • Running 20 times simulation • Observation • T vs. r → OPSA • T vs. System gain → 3 strategies • Discussion

22. T vs. r

23. T vs. System Gain

24. Discussion • OPSA is superior to RPSA & FPSA in the simulation • A systematically & thorough study to verify the purposed strategies in real systems is highly desirable • To this end, we decide to implement the ESP Lite system

25. Outline • Introduction • Analysis • Propose our algorithm • Evaluation & Simulation • Implementation • Result • Conclusion & Future work

26. ESP Lite • We implement ESP using Flash/Java • It is a mimic ESP game • It implemented three playing strategies • RPSA, FPSA, OPSA

27. Flow chart the strategy selection process gives priority to the strategy that has been used least in terms of the # of rounds played previously

28. Client interface

29. Client interface (2)

30. Score System • The score system is used to measure the quality of the agreed words • The quality of each the agreed word should depends on its popularity • high frequency → low quality → low score • low frequency → high quality → high score

31. Score System (2) n words in the score table frequency of the word score of w_i i-th level of the w_i reserved for agreed words which are not in score table

32. Score System (3) • Apply the Porter Stemming Algorithm to remove common morphological and inflectional endings of English words • Prevent words with the same root, but receiving different scores (e.g. determinant and determine) • Reduce the plural form to the singular form (e.g. experiments and experiment)

33. Outline • Introduction • Analysis • Propose our algorithm • Evaluation & Simulation • Implementation • Result • Conclusion & Future work

34. Experiment Setting • 100,000 images from ESP dataset • ESP dataset • 100,000 images • Average 15 labels per image collected from the ESP game • Score system • Range of score (10 levels)

35. Experiment Setting (2) • Score system (cont.) • The 5,000 most frequency words from Brown Corpus (a general corpus in the field of corpus linguistics) • Processed by Porter Stemming Algorithm • 3,476 words in score table

36. Experiment • http://nrl.iis.sinica.edu.tw/GWAP/ESPLite/ • From 2009/3/9 to 2009/4/9 • 3,103 games • 9,376 labeled pictures • 12,312 agreements

37. Score Statistics

38. Behavior of OPSA

39. Behavior of OPSA (2) • What happened in each round?

40. Behavior of OPSA (2) • The operation of OPSA depend on r

41. Behavior of OPSA (3) • Observe players’ behavior

42. Performance

43. Outline • Introduction • Analysis • Propose our algorithm • Evaluation & Simulation • Implementation • Result • Conclusion & Future work

44. Conclusion • In this thesis • We propose a metric to evaluate the performance of GWAP • We play GWAP systems with strategies and make GWAP systems more efficient

45. Conclusion (2) • This is the first GWAP study that implements and evaluates an analytical model on real-world GWAP systems • our experiment results confirm that GWAP systems are more efficient if they are designed and played with strategies

46. Future work • Thinking about the time factor in analysis • Thinking about adaptive r for each image • Constructing the framework of GWAP systems • Developing the GWAP portal for fast accessing more GWAP systems • Developing the GWAP systems toward mobile environment

47. Thank You!