1 / 16

Cyber Metrics in the DoD or How Do We Know What We Don ’ t Know?

Cyber Metrics in the DoD or How Do We Know What We Don ’ t Know?. John S. Bay, Ph.D. Executive Director. Things People Have Asked Me. How much money should I spend this year on cyber defense technologies? How many attacks has your firewall repelled this month?

hmalcolm
Download Presentation

Cyber Metrics in the DoD or How Do We Know What We Don ’ t Know?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Cyber Metrics in the DoDorHow Do We Know What We Don’t Know? John S. Bay, Ph.D.Executive Director

  2. Things People Have Asked Me • How much money should I spend this year on cyber defense technologies? • How many attacks has your firewall repelled this month? • If I only had a dollar to spend on cyber, where should I spend it? • Why is cyber research such a slog?

  3. Answers(which did not go over well) • How much money have you got? • We repelled all of them … except that one you read about in the paper • Spend your dollar on upgrades • Cyber research is a slog because there is no physics theory underlying it all, liker Maxwells’ Equations or Newton’s Laws

  4. But really … it DEPENDS • The “threat” factor is common in cybersecurity, but mostly not elsewhere • … and it IS true that there is no useful PHYSICS for the problem

  5. DoD Taxonomy of Threats From: Defense Science Board, Resilient Military Systems and the Advanced Cyber Threat, January 2013

  6. And The Corresponding Criticality

  7. What Might the COSTS Be?

  8. So Then, What to Measure? • Qualitative • Capabilities • Missions lost • Quantitative • Performance • Cost • To achieve • Not achieving

  9. Capabilities and Maturity

  10. Dashboard Approach

  11. “Stoplight Chart” Assessments See: SPIDERS JCTD

  12. Costs to Us • All vulnerabilities are bugs • All code has bugs • Bugs are expensive • Exploits are cheap  the “asymmetry” problem

  13. Mission-Assurance Approach • Helps focus attention • Requires a “map” o the mission • Implies a prioritization on missions (something loses) • Requires reconfigurable systems and networks • Is not cheap From: DUSD(I&E) Office, HANDBOOK For SELF-ASSESSING SECURITY VULNERABILITIES & RISKS of INDUSTRIAL CONTROL SYSTEMS On DOD INSTALLATIONS, December 2012

  14. Just Good Enough (Incremental)Approach • How long would our red team take to penetrate the system? • An empirical measure, at best. • Implies a canonical red team Bad code prob(first vulnerability is discovered) Gamma distribution? Better code time

  15. The Accountability Approach • NIST 800-53 guidelines • The “did we do everything we know how to do” approach From: NIST Special Publication 800-53, rev. 4, Security and Privacy Controls for Federal Information Systems and Organizations, April 2013

  16. Conclusions: Which is Best? • None of them. They service somewhat orthogonal purposes. • But they can provide apples-to-apples comparisons • Can they answer the Generals’ questions? • No • … except maybe the one about the firewall • There is CERTAINLY no satisfactory “physics” to guide anybody • Cyber Metrics is still an extremely important and high-priority problem for OSD!

More Related