ART Testbed

1. ART Testbed • Join the discussion group at: http://www.art-testbed.net

2. ART Testbed Questions • Can agents request reputations about themselves? • Can an agent produce an appraisal without purchasing opinions? • Does the Testbed assume a common representation for reputations? • Does the Testbed prevent agents from winning via action-planning skills, as opposed to trust-modeling skills? • What if an agent can’t or won’t give a reputation value? • Why does it cost more to generate an accurate opinion than an inaccurate one? • Why not have a centralized reputation broker? • Isn’t it unrealistic to assume a true value of a painting can be known? Is art appraisal a realistic domain? • Why not design an incentive-compatible mechanism to enforce truth-telling?

3. “Really Good” ART Testbed Questions • Is there a consensus on the definitions of “trustworthiness” and “reputation”? • How can collusion be avoided? • Is truth-telling a dominant strategy? • Will the system reach equilibrium, at which point reputations are no longer useful? • What happens if client fee (100), opinion cost (10), and reputation cost (1) are changed? • Do any equilibria exist? • What happens when agents enter or leave the system? • When will agents seek out reputations? • Space of experiments is underexplored—that’s a good thing!

4. Questions about the Paper • What is a “trust model”? • How does q-learning work? How related to reinforcement learning? How do rewards tie in? • What is lambda? • How can experience- and reputation-based learning be combined to overcome the weaknesses of each (intermediate lambda values)? • What about different combinations of (more sophisticated) agents in a game? • Why the assumptions chosen? They seem too extreme. • Reputation decisions weren’t examined very well.

5. The Agent Reputation and Trust Testbed: Experimentation and Competition for Trust in Agent Societies Karen K. Fullam1, Tomas B. Klos2, Guillaume Muller3, Jordi Sabater4, Andreas Schlosser5, Zvi Topol6, K. Suzanne Barber1, Jeffrey S. Rosenschein6, Laurent Vercouter3, and Marco Voss5 1Laboratory for Intelligent Processes and Systems, University of Texas at Austin, USA 2Center for Mathematics and Computer Science (CWI), Amsterdam, The Netherlands 3Ecole Nationale Superieure des Mines, Saint-Etienne, France 4Institute of Cognitive Science and Technology (ISTC), National Research Council (CNR), Rome, Italy 5IT Transfer Office, Darmstadt University of Technology, Darmstadt, Germany 6Multiagent Systems Research Group—Critical MAS, Hebrew University, Jerusalem, Israel

6. Appraiser Agent Appraiser Agent Appraiser Agent Appraiser Agent Appraiser Agent Client Client Client Testbed Game Rules If an appraiser is not very knowledgeable about a painting, it can purchase "opinions" from other appraisers. For a fixed price, clients ask appraisers to provide appraisals of paintings from various eras. Agents function as art appraisers with varying expertise in different artistic eras. Opinions and Reputations Client Share Appraisers whose appraisals are more accurate receive larger shares of the client base in the future. Appraisers can also buy and sell reputation information about other appraisers. Appraisers compete to achieve the highest earnings by the end of the game.

7. Step 1: Client and Expertise Assignments • Appraisers receive clients who pay a fixed price to request appraisals • Client paintings are randomly distributed across eras • As game progresses, more accurate appraisers receive more clients (thus more profit)

8. Step 2: Reputation Transactions • Appraisers know their own level of expertise for each era • Appraisers are not informed (by the simulation) of the expertise levels of other appraisers • Appraisers may purchase reputations, for a fixed fee, from other appraisers • Reputations are values between zero and one • Might not correspond to appraiser’s internal trust model • Serves as standardized format for inter-agent communication

9. Provider Requester Request Accept Payment Reputation Step 2: Reputation Transactions Requester sends request message to a potential reputation provider, identifying appraiser whose reputation is requested Potential reputation provider sends “accept” message Requester sends fixed payment to the provider Provider sends reputation information, which may not be truthful

10. Step 3: Opinion Transactions • For a single painting, an appraiser may request opinions (each at a fixed price) from as many other appraisers as desired • The simulation “generates” opinions about paintings for opinion-providing appraisers • Accuracy of opinion is proportional to opinion provider’s expertise for the era and cost it is willing to pay to generate opinion • Appraisers are not required to truthfully reveal opinions to requesting appraisers

11. Provider Requester Request Certainty Payment Opinion Step 3: Opinion Transactions Potential provider sends a certainty assessment about the opinion it can provide - Real number (0 – 1) - Not required to truthfully report certainty assessment Requester sends request message to a potential opinion provider, identifying painting Requester sends fixed payment to the provider Provider sends opinion, which may not be truthful

12. Step 4: Appraisal Calculation • Upon paying providers and before receiving opinions, requesting appraiser submits to simulation a weight (self-assessed reputation) for each other appraiser • Simulation collects opinions sent to appraiser (appraisers may not alter weights or received opinions) • Simulation calculates “final appraisal” as weighted average of received opinions • True value of painting and calculated final appraisal are revealed to appraiser • Appraiser may use revealed information to revise trust models of other appraisers

13. 2006 ART TestbedCompetition Results Karen K. Fullam

14. Competition Organization • “Practice” Competition • Spanish Agent School, Madrid, April 2006 • 12 participants • International Competition • AAMAS, Hakodate, May 2006 • Preliminary Round • 13 Participants • 5 games each • Final Round • 5 Finalists • 10 games with all finalists participating

15. Bank Balances Iam achieves highest bank balances

16. Opinion Purchases Joey and Neil do not purchases opinions Sabatini purchases the most opinions

17. Opinion Earnings Sabatini and Iam provide the most opinions Neil and Frost do not provide many opinions

18. Opinion Sensing Costs Iam invests the most in opinions it generates

19. Expertise vs. Bank Balance Iam’s average expertise was not significantly higher than others’ Greater Expertise

20. Learning Trust Strategies in Reputation Exchange Networks Karen K. Fullam K. Suzanne Barber

21. Trust Decisions in Reputation Exchange Networks • Agents perform transactions to obtain needed resources • Transactions have risk because partners may be untrustworthy • Agents must learn whom to trust and how trustworthy to be • When agents can exchange reputations • Agents must also learn when to request reputations and what reputations to tell • Agents’ trust decisions affect each other • Difficult to learn each decision independently If I cheat A, and A tells B, will it hurt my interactions with B? How trustworthy should I be? Should I trust? Resources (goods, services, information) If I lie to others that C is bad, can I monopolize C’s interactions? What reputations should I tell? Reputations Which reputations should I listen to?

22. Enumerating Decisions in a Trust Strategy • Num agents = a • Num transaction types = e • Num choices/decision = n Trustee If these decisions affect each other, there are possible strategies! Truster Transaction Reputation Fundamental Should I tell an accurate reputation? How trustworthy should I be? Trustee Agent Role How to learn the best strategy with so many choices? Should I believe this reputation? Should I trust? Truster

23. Reinforcement Learning Select a strategy Expected Reward Strategy A B C D Strategy feedback influences expected reward Strategies with higher expected rewards are more likely to be selected

24. Learning In Reputation Exchange Networks Decision Expected Reward Tr(A) ⌐Tr(A) Decision Expected Reward Tr(B) ⌐Tr(B) Decision Expected Reward Tr(C) ⌐Tr(C) Strategy Expected Reward Tr(A),Tr(B),Tr(C)… ⌐Tr(A),Tr(B),Tr(C)… Tr(A),⌐Tr(B),Tr(C)… ⌐Tr(A),⌐Tr(B),Tr(C)… Tr(A),Tr(B),⌐Tr(C)… ⌐Tr(A),Tr(B),⌐Tr(C)… Tr(A),⌐Tr(B),⌐Tr(C)… ⌐Tr(A),⌐Tr(B),⌐Tr(C)… Removing interdepend-encies makes each decision in the strategy learnable . . . Because decisions are interdependent, there are . possible strategies! Use the ART Testbed as a case study

25. Many Interdependent Decisions Reputation Requester’s reputation costs Opinion Requester’s opinion costs Opinion Provider’s opinion order costs Opinion Requester Opinion Provider Reputation Provider Reputation Requester Number of requests received by Reputation Provider Accuracy of Reputation Requester’s trust models Accuracy of Opinion Requester’s appraisals Number of requests received by Opinion Provider Reputation Provider’s reputation revenue Opinion Requester’s client revenue Opinion Provider’s opinion revenue Other Appraisers’ client revenue When Reputation Requester is Opinion Requester

26. Opinion Requester Feedback Opinion Requester’s opinion costs Opinion Requester’s opinion costs Opinion Requester Assume: Client revenue feedback is wholly attributed to Opinion Requester decision Opinion Requester’s client revenue Opinion Requester’s client revenue Divide revenue (client revenue) among opinions based on opinion accuracy Opinion Purchase Costs Client Revenue Reward = – Make sure to bundle: average all decisions for same opinion provider and era

27. Opinion Provider Feedback Opinion Provider’s opinion order costs Opinion Provider’s opinion order costs Assume: Client revenue is not related to Opinion Provider decision Opinion Provider Opinion Provider’s opinion revenue Opinion Provider’s opinion revenue Other Appraisers’ client revenue • Bundle all n opinions by requesting agent and era (same cg decision) • Reward for this cg decision: Opinion Selling Revenue Opinion Generating Costs Reward = – Bundle all decisions for same requesting agent and era

28. Reputation Provider Feedback Assume: Client revenue is not related to Reputation Provider decision Reputation Provider Reputation Provider’s reputation revenue Reputation Provider’s reputation revenue Other Appraisers’ client revenue • Parameterize reputation accuracy: • Reward for this d value: (n = 1 if requested; n = 0 for each timestep until reputation is requested again) Reputation Selling Revenue Reward = • Identify reputations according to requesting agent, subject agent and era (there will be only 1 in each “bundle”) Parameterize reputation accuracy

29. ReputationRequesterFeedback Reputation Requester’s reputation costs Opinion Requester’s opinion costs Opinion Requester’s opinion costs Opinion Requester: which opinions to purchase Opinion Requester Reputation Requester Opinion Requester’s client revenue Opinion Requester’s client revenue l( ) Opinion Requester Reward Reputation Purchase Costs Reward = – l determines influence of: past experience vs. reputations in deciding to purchase opinions l = 0: Past experience only  Opinion-requesting decision  No reward for requesting reputations l = 1: Reputations only  Reputation-requesting decision  Full reward for requesting reputations