Countering Malicious Code: Building an Ideal Reference Monitor

1. Lecture 21: Countering Malicious Code David Evans http://www.cs.virginia.edu/evans CS588: Cryptography University of Virginia Computer Science

2. Menu • Reference Monitors • Java/.NET Security University of Virginia CS 588

3. Malcode Defenses • Constrain program behavior • Reference Monitors • In-line Reference Monitors • Prevent possibly harmful code from running • Safe Languages • Proof-Carrying Code University of Virginia CS 588

4. Monitor Speakers Network Disk Memory SuperSoaker 2000 Program Execution Program University of Virginia CS 588

5. Monitor Speakers Network Disk Memory SuperSoaker 2000 Program Execution Reference Monitor Program University of Virginia CS 588

6. Ideal Reference Monitor • Sees everything a program is about to do before it does it • Can instantly and completely stop program execution (or prevent action) • Has no other effect on the program or system Can we build this? Probably not unless we can build a time machine... University of Virginia CS 588

7. most things limited Real Ideal Reference Monitor • Sees everything a program is about to do before it does it • Can instantly and completely stop program execution (or prevent action) • Has no other effect on the program or system University of Virginia CS 588

8. Operating Systems • Provide reference monitors for most security-critical resources • When a program opens a file in Unix or Windows, the OS checks that the principal running the program can open that file • Doesn’t allow different policies for different programs • No flexibility over what is monitored • OS decides for everyone • Hence, can’t monitor inexpensive operations University of Virginia CS 588

9. Reference Monitor as Finite State Automaton [Schneider99] All other instructions All other instructions Aim Aim 2 0 1 Fire Aim All other instructions Fire STOP Policy Violation University of Virginia CS 588

10. What policies can be enforced? • Assume: • Security Automaton can see entire state of world, everything about instruction about to execute • Security Automaton has unlimited memory, can do unlimited computation • Are there interesting policies that still can’t be enforced? University of Virginia CS 588

11. What’s a Security Policy? • What’s a program? • A set of possible executions • What’s an execution? • A sequence of states • What’s a security policy? • A predicate on a set of executions University of Virginia CS 588

12. More Formally... •  : set of all possible executions (can be infinite) • S: set of executions possible by target program S • P: security policy set of executions  Boolean S is safe iff P (S ) is true. University of Virginia CS 588

13. Reference Monitors cannot enforce all Security Policies • Some policies depend on: • Knowing about the future • If the program charges the credit card, it must eventually ship the goods • Knowing about all possible executions • Information flow – can’t tell if a program reveals secret information without knowing about other possible executions • Reference Monitors can only know about past of this particular execution University of Virginia CS 588

14. Safety Policies • Reference monitors can only enforce safety policies • Safety policy is a predicate on a prefix of states (see Schneider98 for more formal definition) • Cannot depend on future: prefix means once it is false, it is always false • Cannot depend on other possible executions University of Virginia CS 588

15. Java SecurityReal or Decaf? University of Virginia CS 588

16. What is Java? • Island in Indonesia • A Programming Language (Java) • A Portable Low-Level Language (JVML) • A Platform (JavaVM) • A (semi-)successful marketing strategy • JavaScript is not related to Java or Java • Work on your projects • All of the above University of Virginia CS 588

17. Java : Programming Language “A simple, object-oriented, distributed, interpreted, robust, secure, architecture neutral, portable, high-performance, multithreaded, and dynamic language.” [Sun95] University of Virginia CS 588

18. What is a secure language? • Language is designed so it cannot express certain computations considered insecure. • Language is designed so that (accidental) program bugs are likely to be caught by the compiler or run-time environment instead of leading to security vulnerabilities. A few attempt to do this: PLAN, packet filters University of Virginia CS 588

19. Safe Programming Languages • Type Safety • Compiler and run-time environment ensure that bits are treated as the type they represent • Memory Safety • Compiler and run-time environment ensure that program cannot access memory outside defined storage • Control Flow Safety • Can’t jump to arbitrary addresses Which of these does C++ have? Not a new idea: LISP had these in 1960! University of Virginia CS 588

20. Java Safety • Type Safety • Most types checked statically • Coercions, array assignments type checked at run time • Memory Safety • No direct memory access (e.g., pointers) • Primitive array type with mandatory run-time bounds checking • Control Flow Safety • Structured control flow, no arbitrary jumps University of Virginia CS 588

21. Malicious Code Can a safe programming language protect you from malcode? • Code your servers in it to protect from buffer overflow bugs • Only allow programs from untrustworthy origins to run if the are programmed in the safe language University of Virginia CS 588

22. Safe Languages? • But how can you tell program was written in the safe language? • Get the source code and compile it (most vendors, and all malicious attackers refuse to provide source code) • Special compilation service signs object files generated from the safe language (SPIN, [Bershad96]) • Verify object files preserve safety properties of source language (Java) University of Virginia CS 588

23. Joe User JVML malcode.java Java Source Code malcode.class JVML Object Code javac Compiler JavaVM Joe wants to know JVML code satisfies Java’s safety properties. University of Virginia CS 588

24. No! This code violates Java’s type rules. Does JVML satisfy Java’s safety properties? iconst_2 push integer constant 2 on stack istore_0 store top of stack in variable 0 as int aload_0 load object reference from variable 0 arraylength replace array on top of stack with its length University of Virginia CS 588

25. Trusted Computing Base Joe User Bytecode Verifier malcode.class JVML Object Code Java Bytecode Verifier Invalid “Okay” STOP JavaVM University of Virginia CS 588

26. Bytecode Verifier • Checks JVML code satisfies Java’s safety properties • Type safe – stack and variable slots must store and load as same type • Memory safe (guaranteed by instruction set) • Control flow safe: jumps must be within function, or call/return University of Virginia CS 588

27. Are Java Bytecode Verifiers Complicated? • ~700 rules to enforce, JVML specification is (not all clearly specified) • Emin Gün Sirer found > 100 bugs in commercial bytecode verifiers (using automatic test generation) • At least 15 of them were security vulnerabilities • JVML includes jsr instruction (jump to subroutine), can be called with different types in variables and on stack University of Virginia CS 588

28. Joe User Java javac Compiler malcode.java malcode.class JVML Trusted Computing Base Java Bytecode Verifier Invalid “Okay” STOP JavaVM University of Virginia CS 588

29. JavaVM • Virtual machine – interpreter for JVML programs • Has complete access to host machine • Bytecode verifier ensures some safety properties, JavaVM must ensure rest: • Type safety of run-time casts, array assignments • Memory safety: array bounds checking • Resource use policy University of Virginia CS 588

30. JavaVM Policy Enforcment [JDK 1.0 – JDK 1.1] From java.io.File: public boolean delete() { SecurityManager security = System.getSecurityManager(); if (security != null) { security.checkDelete(path); } if (isDirectory()) return rmdir0(); else return delete0(); } University of Virginia CS 588

31. java.lang.SecurityManager /** Throws a SecurityException if the calling thread is not allowed to delete the specified file. This method is invoked for the current security manager by the delete method of class File. */ (Some other comments deleted.) public void checkDelete(String file) { throw new SecurityException(); } University of Virginia CS 588

32. Security Manager • Reference monitor • How well does it satisfy the requirements? • Complete mediation • Can stop execution/prevent action • Limited effect on execution until policy violation • User/host application creates a subclass of SecurityManager to define a policy University of Virginia CS 588

33. HotJava’s Policy (JDK 1.1.7) public class AppletSecurity extends SecurityManager { ... public synchronized void checkDelete(String file) { checkWrite(file); } ... } University of Virginia CS 588

34. Note: no checking if not inApplet! Very important this does the right thing. AppletSecurity.checkWrite(some exception handling code removed) public synchronized void checkWrite(String file) { if (inApplet()) { if (!initACL) initializeACLs(); String realPath = (new File(file)).getCanonicalPath(); for (int i = writeACL.length ; i-- > 0 ;) { if (realPath.startsWith(writeACL[i])) return; } throw new AppletSecurityException ("checkwrite", file, realPath); } } University of Virginia CS 588

35. JDK 1.0 Trust Model • When JavaVM loads a class from the CLASSPATH, it has no associated ClassLoader (can do anything) • When JavaVM loads a class from elsewhere (e.g., the web), it has an associated ClassLoader University of Virginia CS 588

36. JDK Evolution • JDK 1.1: Signed classes from elsewhere and have no associated ClassLoader • JDK 1.2: • Different classes can have different policies based on ClassLoader • Explict enable/disable/check privileges • SecurityManager is now AccessController University of Virginia CS 588

37. What can go wrong? • Java API doesn’t call right SecurityManager checks (63 calls in java.*) • Font loading bug, synchronization • ClassLoader is tricked into loading external class as internal • Bug in Bytecode Verifier can be exploited to circumvent SecurityManager • Policy is too weak and allows damaging behavior University of Virginia CS 588

38. Joe User Java javac Compiler malcode.java malcode.class JVML Trusted Computing Base Java Bytecode Verifier Invalid “Okay” STOP JavaVM University of Virginia CS 588

39. Bytecode Verifier • Checks JVML code satisfies safety properties • Simulates program execution to know types are correct, but doesn’t need to examine any instruction more than once • After code is verified, it is trusted: is not checked for type safety at run time (except for casts, array stores) Key assumption: when a value is written to a memory location, the value in that memory location is the same value when it is read. University of Virginia CS 588

40. Violating the Assumption … // The object on top of the stack is a SimObject astore_0 // There is a SimObject in location 0 aload_0 // The value on top of the stack is a SimObject If a cosmic ray hits the right bit of memory, between the store and load, the assumption might be wrong. University of Virginia CS 588

41. Improving the Odds • Set up memory so that a single bit error is likely to be exploitable • Mistreat the hardware memory to increase the odds that bits will flip • One bit flip allows you to violate type safety, compromise all security Sudhakar Govindavajhala and Andrew W. Appel, Using Memory Errors to Attack a Virtual Machine, July 2003. University of Virginia CS 588

42. Getting a Bit Flip • Wait for a Cosmic Ray • You have to be really, really patient… (or move machine out of Earth’s atmosphere) • X-Rays • Expensive, not enough power to generate bit-flip • High energy protons and neutrons • Work great - but, you need a particle accelerator • Hmm…. University of Virginia CS 588

43. Using Heat • 50-watt spotlight bulb • Between 80°-100°C, memory starts to have a few failures • Attack applet is successful (at least half the time)! • Hairdryer works too, but it frys too many bits at once Picture from Sudhakar Govindavajhala University of Virginia CS 588

44. Should Anyone be Worried? Java virtual machine University of Virginia CS 588

45. Recap • Verifier assumes the value you write is the same value when you read it • By flipping bits, we can violate this assumption • By violating this assumption, we can violate type safety: get two references to the same storage that have inconsistent types • By violating type safety, we can get around all other security measures University of Virginia CS 588

46. .NET University of Virginia CS 588

47. Major Security Vulnerabilities (Cumulative) 50 http://www.cs.virginia.edu/evans/pubs/acsac2004.html 45 40 35 30 25 Java VM 20 15 10 .NET VM 5 0 1996 1997 1998 1999 2000 2001 2002 2003 2004

48. Course Evaluations University of Virginia CS 588

49. CS588 Pledge I will provide useful feedback. I realize this is an evolving course and it is important that I let the course staff know what they need to improve the course. I will not wait until the end of the course to make the course staff aware of any problems. I will provide feedback either anonymously … or by contacting the course staff directly. I will fill out all course evaluation surveys honestly and thoroughly. University of Virginia CS 588

50. Course Evaluations • Fill out the SEAS Evaluation on-line • It may not be anonymous, but I promise not to peek • Respond based on whether you want me to get fired or promoted • (optional) Fill out a RateMyProfessors.com survey • Provide useful information to future students • Fill out my course-specific survey (handed out last day) • Help improve future versions of the course for later students University of Virginia CS 588