Measuring ROI on ACSI Surveys

1. Measuring ROI on ACSI Surveys Tim Evans Social Security Administration Office of Electronic Services tim.evans@ssa.gov

2. Introduction ROI on your surveys not directly attributable to your survey expenditures�you may be validating prior work Ways of using ACSI data for ROI purposes Identify user behavior that explicitly shows your site�s value Identify data that shows how you can improve your site to increase its value Identify and focus your (limited) resource priorities

3. SSA and ACSI Began Q1/04 Four current surveys Main site FAQ�s site Retirement Planner site Internet Social Security Benefit Application--ISBA (not published) Two more surveys planned Business Services On-line (employer wage reporting) Disability (largest single group of surveyed visitors say they came for disability info)

4. SSA ROI for ACSI Surveys--Summary Main survey shows high value + return on 2003 site redesign FAQ survey shows clear reduction in use of high-cost support channels as direct result of FAQ�s availability Retirement Planner survey shows high value of Planner + the opportunity to focus our resources ISBA Survey helps clarify resource priorities in limited-budget environment

5. Main Site Survey, Q3/04http://www.socialsecurity.gov Overall 75 Content 80 Functionality 76 Look & Feel 79 Navigation 73 Search 74 Site Performance 83

6. Compare to ForeSee Results Benchmarks Overall Satisfaction: +5 vis a vis Q2/04 Gov�t benchmark; +4 vis a vis overall Look & Feel: +4/+3 Content: +2/+2 Navigation: +5/+3

7. 2003 Site Redesign Site-wide adoption of CSS Style Sheets for common look & feel (colors, fonts, etc.) Added common header with top nav-links Primarily implemented via common DreamWeaver template for easy propagation site wide Context: ~80 SSA webmasters with central, but informal governance

8. SSA FAQ�s Survey, Q3/04http://ssa-custhelp.ssa.gov/cgi-bin/ssa.cfg/php/enduser/entry.php Overall Score 74 Content 81 Functionality 77 Look & Feel 79 Navigation 75 Search 75 Site Performance 84

9. Custom Questions Results Reveal ROI--#1 If there were no FAQ�s, how would you have contacted Social Security to ask your question? (N=1273) Send e-mail 16% Browse web site 8 Call 800 number 39 Call local SSA office 22 Visit local SSA office 10 Write SSA 1 Would not have contacted SSA 4

10. Custom Questions Results Reveal ROI--#2 After reading our answer(s), what do you plan to do next? (N=487) Nothing; question answered 36% Nothing; question not answered 4 Send SSA e-mail 6 Call SSA 800 number 14 Call SSA local office 9 Visit local SSA office 22 Try different question 4 Look elsewhere on site 3 Question re-worded, 8/25/04, to better focus it; this explains lower number of respondents. Prior wording had less precise answer choices, but trend was the generally clearQuestion re-worded, 8/25/04, to better focus it; this explains lower number of respondents. Prior wording had less precise answer choices, but trend was the generally clear

11. Key Comparisons for 2 Questions Send SSA e-mail 16% 6% -62% Call SSA 800 number 39 14 -65 Call SSA local office 22 9 -59 Visit local SSA office 10 22 +120 Refer to Pew Study to be discussed laterRefer to Pew Study to be discussed later

12. ROI�Reduction Use of High-Cost Support Channels Forester Research Estimates Cost of Services Knowledge-Based Self Service: $1.17/incident Telephone support: $33/incident E-mail support: $10/incident SSA Results, shown by ACSI data 65% reduction in calls to 800 number 59% reduction in calls to local office 62% reduction in e-mails 120% increase in local office visits 54 million 800 calls in FY 200354 million 800 calls in FY 2003

13. SSA Retirement Planner Survey, Q3/04http://www.socialsecurity.gov/r&m1.htm Overall Score 77 Content 82 Functionality 80 Look & Feel 81 Navigation 77 Site Performance 85 Tasks/Transactions 78

14. SSA Retirement Planner Survey, Q3/04--Futures Likely to Return 85 Recommend 83 File on-line 61 What�s wrong with this picture? Is it good news or bad?

15. Retirement Planner�Custom Question How do you plan to file for retirement benefits? (7/12-8/9 data; N=1127) On line 39% Face to face 37 Telephone 4 On line w/SSA help 2 More recent data shows �on line w/help� running ~13%, �on line� ~30%--totaling more than 40% (not enough data for conclusions) On line w/help option added 8/4On line w/help option added 8/4

16. Pew Internet Studyhttp://www.pewinternet.org/PPF/r/128/report_display.asp How Americans Get in Touch with Government (May, 2004) Telephone 40% Web 24 In person 13 E-mail 11 Write a letter 10

17. Pew Internet Study�Cont�d Why Americans contact Government Conduct Transactions 30% Get specific information 25 Register opinions 19 Help w/specific problems 11 For complex, problem-solving contacts, 62% prefer phone/in person Refer back to FAQ increase in local office visits; complex problem-solving contactsRefer back to FAQ increase in local office visits; complex problem-solving contacts

18. What Does This Tell Us About ROI for Retirement Planner? With ~40% interested in filing on line, SSA is doing quite well (compare Pew 30% transactions figure) Futures Score shows opportunity to improve on-line application, as Boomers age and interest in on-line filing rises New �On-line w/Help� Numbers strong indicator for future, bearing out Pew conclusions on complex contacts

19. Internet Social Security Benefit Application (ISBA) Retirement/spouse applicants vastly more satisfied (>2:1) than disability applicants Far more (5:1) retirement/spouse applicants successfully complete on-line application Lowest scores in Functionality, Tasks/Transactions; cut across board, especially among disability applicants Disability applicants have far lower level of Internet experience (5:1 �beginner�), and may need additional help Expectations count: level of Internet experience inversely proportional to satisfaction Pew study: only 40% of disabled use the Internet; 38% say their disability makes using the Internet harder or impossible; only 9% see Internet as preferred channelPew study: only 40% of disabled use the Internet; 38% say their disability makes using the Internet harder or impossible; only 9% see Internet as preferred channel

20. ISBA and ROI Lessons Retirement/spouse applicants >80% of ISBA customers; universe will grow as Boomers age and demand service (Pew data cited earlier) Disability applicants have lower Internet experience, need more support via high-cost channels The whole disability application process differs substantially from retirement application process With resources limited, best use lies in improving overall Functionality, Tasks/Transactions All applicants will benefit from general improvements in the on-line application Again, Pew survey on disabled Americans� reluctance to use the InternetAgain, Pew survey on disabled Americans� reluctance to use the Internet

21. Wrap ROI on your surveys not directly attributable to your survey expenditures Ways of using ACSI data for ROI purposes Identify user behavior that explicitly shows value of your site Identify data that shows how you can improve your site to increase its value Identify and focus your resource priorities

22. SSA Conclusions Main survey shows high value + return on 2003 site redesign FAQ survey shows clear reduction in use of high-cost support channels as direct result of FAQ�s availability Retirement Planner survey shows high value of Planner + the opportunity to focus our resources ISBA Survey helps clarify resource priorities in limited-budget environment