1 / 28

Strategic Approach to Web Evaluation

Strategic Approach to Web Evaluation. by Fred B. Wood Office of Health Information Programs Development National Library of Medicine NIH MEDLINE plus Advisory Group Meeting June 22, 2001. Overview--Web Evaluation Methods Tested.

martena-burnett
Download Presentation

Strategic Approach to Web Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Strategic Approach to Web Evaluation by Fred B. Wood Office of Health Information Programs Development National Library of Medicine NIH MEDLINEplus Advisory Group Meeting June 22, 2001 M+AdvGrp1.ppt

  2. Overview--Web Evaluation Methods Tested • Online random survey of MEDLINEplus users (in collaboration with CyberDialogue) • Nationwide telephone survey of online health information users (CyberDialogue, syndicated) • Online random survey of non-NLM health information users (PCData online panel) • Comparisons with other surveys and NLM web log data • Competitive analysis of MEDLINEplus (CyberDialogue online panel) • Internet audience measurement (PCData, netScore panels) • Internet connectivity performance tests

  3. MEDLINEplus Online Survey • NLM developed 22-question survey in collaboration with Cyber Dialogue, Inc. • First time visitors were asked only background and demographic questions, repeat visitors also were asked evaluative and impact questions • Survey instrument approved under OMB blanket clearance procedure • Site visitors were randomly intercepted and given opportunity to respond

  4. MEDLINEplus Online Survey (cont’d) • 93,852 visitors intercepted between February 14-27, 2001 • 2,969 completed the survey (3% overall response rate--typical of online web surveys) • Transient cookies used to preclude repeat intercepts, expired at end of survey, and no behavioral tracking (consistent with OMB cookie policy) • 1,307 (44% of total respondents) expressed interest in participating in future MEDLINEplus surveys and provided an e-mail address

  5. Cross-Comparison with Web Log Data--MEDLINEplus • Time-of-day/day-of-week variability--97% correlation between survey data and web log data. • US/non-US user split--73% US/27% non-US per survey, 65% US/35% non-US per web log data (but majority of log data cannot be resolved). • Repeat visitors (2+ times per month)--36% per survey, 21% per web log data (but log data subject to error factors both ways) • Survey non-response bias may be relatively minimal.

  6. MEDLINEplus Competitive Analysis • Compared MEDLINEplus, WebMD, Mayo Health, InteliHealth, and Medscape • Used CyberDialogue’s virtual online panel (120K) • Panelists randomly sampled to identify 950 online health site users • Each panelist asked to compare two of the five sites and complete an online survey • 520 completed the survey with 482 valid responses (51% net response rate)

  7. Internet Audience Measurement Online Panels • NLM contracted with PCDataOnline, Inc. for external usage data based on PCData’s online panel of 120K “home” users who agreed to monitoring of web usage • Results extrapolated to US home Internet market • PCData ceased operations in March 2001, client base bought by comScore • comScore offers netScore with a panel of 1.5M users covering home, work, and school • Can cross-compare PCData, netScore, & log data

  8. Illustrative Audience Measurement DataUS Home Internet Market--NIH Drill Down US Unique Home Users (% of NIH Total) [source: PCData] Dec 2000 Jan 2001 Feb 2001 Domains nlm.nih.gov 42.6% 45.3% 50.4% niddk.nih.gov 11.2 8.7 7.9 nci.nih.gov 7.1 7.4 8.7 nhlbi.nih.gov 4.4 3.8 6.0 Web sites MedlinePlus 14.2% 18.0% 17.5% PubMed 13.4 18.3 20.1 www.nih.gov 13.6 11.1 17.0 www.niddk.gov 11.2 8.7 7.9

  9. How Accurate Are These Data? • Different companies use different methodologies • Panel composition differs--home vs home/work/school • Raw data are extrapolated based on demographics • Of US Government web domains, NIH.gov ranked between #5 and #7 by PCData and netScore in Feb 2001. • NIH.gov ranked #3 (after USPS & NASA) by both companies, before tax and student aid seasons began, in Dec 2000 • Of health info domains, NIH.gov ranked #3 or #4 (after webMD, allhealth [iVillage], and ediets.com) by both companies in Feb 2001

  10. How Accurate Are These Data (cont’d)? • Relative rankings may be accurate, and absolute numbers may be in the ball park • Based on Feb 2001 NLM web log data: PubMed (~2M) + MEDLINEplus (~0.45M) + www.nih.gov (~0.45M) = 2.9M unique visitors • Compares with 2.5M for PCData and 2.6M for netScore, for the NIH.gov domain • netScore also measures international traffic, 2M unique visitors to NIH.gov in Feb 2001 or ~ 40% of total 4.6M • Assuming these three web sites ~ 55% of total NIH.gov visitors, extrapolated total from web log data = 5.3M • 4.6M vs 5.3M is within 20%--NOT BAD!

  11. CENDI Symposium--Co-Sponsored by NLM • Presentations by government, private sector, and non-profit organizations • Presentations available on the CENDI web site at http://www.dtic.mil/cendi/ click on “what’s new” • Or go to www.dtic.mil/cendi/activities/ 04_17_01_eval_pres_agenda.html Tuesday, April 17, 2001

More Related