U.S. National Computer Security Survey (NCSS)

1. Using the Dun & Bradstreet (D&B) Database as a Sampling Frame for Company SurveysSarah Cotton, Anil Bamezai

2. U.S. National Computer Security Survey (NCSS) • Sponsored by: • U.S. Department of Justice • U.S. Department of Homeland Security • Focused on collecting data on the nature, extent, and consequences of computer security incidents • Goal of producing reliable national estimates of the incidence and prevalence of cybercrime against businesses,and their resulting losses • Fielded by the RAND Corporation in 2006

3. Study Goals • Provide information about cybersecurity practices and cost of cybersecurity breaches to the US economy • Provide statistically meaningful contrasts of the above by broad industrial sectors • Representative sample of 36,000 companies across 36 industry sectors.

4. Previous Effort in 2001 • In 2001, the pilot Computer Security Survey was administered to 500 companies as a pilot test by BJS and Census, whose “findings…illustrate[d] the feasibility and utility of a data collection program to be initiated in 2004 among some 36,000 businesses…Results of this test demonstrated a need for an increased response rate to produce valid national estimates and a need to refine survey questions.”

5. Study Goals • We had to build on the resources available from the 2001 Pilot Survey • We had limited ability to tweak the existing survey instruments • We did, however, review the Pilot’s methods and assumptions, and also conducted cognitive interviews to assess how best to implement this survey

6. Establishment vs. Company • Cybersecurity practices generally set at corporate level, so surveying at establishment level was considered wasteful • Census also conducted the 2001 Pilot Survey at the company level • Census’s Business Register (BR) served as the frame, but this obviously was unavailable to us

7. Types of Companies • Three types of companies • Single-location companies (relatively straightforward) • Many establishments, with one company HQ, not in turn owned by another company (also relatively straightforward) • Companies with one or more establishments, in turn owned by another company (very complicated)

8. Why Complicated? • Company ownership patterns in flux due to M&A activity • Information security practices may vary across companies even though they may be part of the same corporate tree • Companies that are part of a complex corporate tree may belong to different industrial sectors • Our data suggests that approximately 20% of corporate trees span 2 or more industries

9. We Used D&B as Company Frame • D&B provides one of the most comprehensive list of companies operating in the US • Provides a detailed description of company ownership patterns to permit construction of complex corporate trees • Includes SIC and NAICS codes to indicate each company’s primary activity • Includes a variety of contact information

10. Surveying Corporate Trees • Research during the cognitive testing stage suggested IT practices variable across companies within the same tree • Not clear if the “ultimate” entity at the apex of a corporate tree could even report cybersecurity-related information for all subsidiaries below • Even if apex entity could, reporting burden in getting them to break out information by subsidiary likely to be excessive, thus low response • Breakdown by subsidiary was required, however, to permit industry-level estimates

11. Options Considered • Send survey first only to apex entity in a corporate tree, then to each sampled subsidiary for which apex was unable to report • Two-stage fielding strategy would have required more time and money • Treat every company within a corporate tree as an independent sampling and reporting unit

12. Key Survey Design Decisions • For complex corporate trees, a compromise was struck • We decided to treat each company within a tree as an independent sampling and reporting unit • However, while respondent was instructed to report only for his company, the instrument was modified to include space where respondent could indicate whether this was the case or subsidiary data had also been included, and if so, for which ones • Each company was assigned to one of 36 broad industrial sectors • Several types of companies were sampled with certainty

13. Scale of the Complex Company Challenge • Dealing with complex companies is a core issue in surveys of this type • Companies from complex corporate trees comprise roughly 25% of our sample • High overlap between complex and certainty companies—85% of complex companies sampled with certainty • Certainty companies comprise 32% of total frame employment, but 85% of total sample employment

14. Challenges Faced in Using D&B File (1) • Contact information for top leadership often repeated among companies within a corporate tree • 12% had the same address • 10% had the same telephone number • 8% had the same contact name • High CSO/CTO turnover • Data for the smallest companies also problematic because of known high business turnover

15. Challenges Faced in Using D&B File (2) • Required greater than anticipated effort to obtain clean contact information (pre-calling, web lookup) • Wave I Pre-Calling: • 2% confirmed company name and address • 29% updated their name and/or address • 2.5% were flagged for supervisor review • relatively inefficient - ~12 min per case • Web II Web Lookup: • 67% of company names and addresses confirmed • 29 % of name and/or address updated • % flagged for supervisor review   • Very efficient – less than 1 min per case

16. Lessons for Future Studies • Two stage fielding may generate a higher response rate and cleaner data, but will also cost more and increase fielding period • Aiming for a smaller sample but with more intensive pre-screening, follow-up, spread over a longer surveying window may represent a better allocation of resources • Considerable budget needs to be allocated for updating contact information in the D&B file, and also to identify companies that have closed • Smallest firms (less than 10 employees) may best be omitted from the frame