1 / 103

Software Testing

Dynamic Symbolic Execution (aka, directed automated random testing, aka concolic execution) Slides by Koushik Sen. Software Testing. Testing accounts for 50% of software development cost Software failure costs USA $60 billion annually

blakee
Download Presentation

Software Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Dynamic Symbolic Execution (aka, directed automated random testing, aka concolic execution)Slides by Koushik Sen

  2. Software Testing • Testing accounts for 50% of software development cost • Software failure costs USA $60 billion annually • Improvement in software testing infrastructure can save one-third of this cost “The economic impacts of inadequate infrastructure for software testing”, NIST, May, 2002 • Currently, software testing is mostly done manually

  3. Simple C code int double(int x) { return 2 * x; } void test_me(int x, int y){ int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  4. Automated testing • What do we need to do if we want to automatically test a piece of code (i.e., automatic unit testing)? • Figure out the environment of the application • What are the inputs? • What are the interfaces for interacting with other components • Automatically generate an environment for the application • Automatically generate the values that come from the environment

  5. Automatic Extraction of Interface • Automatically determine (code parsing) • inputs to the program • arguments to the entry function • variables: whose value depends on environment • external objects • function calls: return value depends on the environment • external function calls • For simple C code • want to unit test the function test_me • intx and int y : passed as an argument to test_me forms the external environment

  6. Automated Random Testing • Generate a test driver automatically to simulate random environment of the extracted interface • most general environment • C – code • Compile the program along with the test driver to create a closed executable. • Run the executable several times to see if assertion is violated

  7. main(){ int tmp1 = randomInt(); int tmp2 = randomInt(); test_me(tmp1,tmp2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } Random test-driver Random Test Driver

  8. main(){ int tmp1 = randomInt(); int tmp2 = randomInt(); test_me(tmp1,tmp2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } Random test-driver Random Test Driver Probability of reaching abort() is extremely low

  9. Limitations • Hard to hit the assertion violated with random values of x and y • there is an extremely low probability of hitting assertion violation • Can we do better? • Directed Automated Random Testing • White box testing

  10. DART (Directed Automated Random Testing) Approach main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  11. DART Approach concrete state symbolic state constraints t1=36 t1=m Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  12. DART Approach concretestate symbolicstate constraints t1=36, t2=-7 t1=m, t2=n Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  13. DART Approach concretestate symbolicstate constraints t1=36, t2=-7 t1=m, t2=n Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  14. DART Approach concretestate symbolicstate constraints x=36, y=-7 x=m, y=n Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  15. DART Approach concretestate symbolicstate constraints x=36, y=-7, z=72 x=m, y=n, z=2m Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  16. DART Approach concretestate symbolicstate constraints x=36, y=-7, z=72 x=m, y=n, z=2m Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } 2m != n

  17. DART Approach concretestate symbolicstate constraints x=36, y=-7, z=72 x=m, y=n, z=2m Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } 2m != n

  18. DART Approach concretestate symbolicstate constraints x=36, y=-7, z=72 x=m, y=n, z=2m Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } solve: 2m = n m=1, n=2 2m != n

  19. DART Approach concretestate symbolicstate constraints t1=1 t1=m Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  20. DART Approach concretestate symbolicstate constraints t1=1, t2=2 t1=m, t2=n Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  21. DART Approach concretestate symbolicstate constraints t1=1, t2=2 t1=m, t2=n Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  22. DART Approach concretestate symbolicstate constraints x=1, y=2 x=m, y=n Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  23. DART Approach concretestate symbolicstate constraints x=1, y=2, z=2 x=m, y=n, z=2m Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  24. DART Approach concretestate symbolicstate constraints x=1, y=2, z=2 x=m, y=n, z=2m Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } 2m = n

  25. DART Approach concretestate symbolicstate constraints x=1, y=2, z=2 x=m, y=n, z=2m Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } 2m = n m != n+10

  26. DART Approach concretestate symbolicstate constraints x=1, y=2, z=2 x=m, y=n, z=2m Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } 2m = n m != n+10

  27. DART Approach concretestate symbolicstate constraints x=1, y=2, z=2 x=m, y=n, z=2m Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } 2m = n m != n+10

  28. DART Approach concretestate symbolicstate constraints x=1, y=2, z=2 x=m, y=n, z=2m Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } solve: 2m = n and m=n+10 m= -10, n= -20 2m = n m != n+10

  29. DART Approach concretestate symbolicstate constraints t1=-10 t1=m Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  30. DART Approach concretestate symbolicstate constraints t1=-10, t2=-20 t1=m, t2=n Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  31. DART Approach concretestate symbolicstate constraints t1=-10, t2=-20 t1=m, t2=n Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  32. DART Approach concretestate symbolicstate constraints x=-10, y=-20 x=m, y=n Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  33. DART Approach concretestate symbolicstate constraints x=-10, y=-20, z=-20 x=m, y=n, z=2m Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  34. DART Approach concretestate symbolicstate constraints x=m, y=n, z=2m x=-10, y=-20, z=-20 Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } 2m = n

  35. DART Approach concretestate symbolicstate constraints x=m, y=n, z=2m x=-10, y=-20, z=-20 Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } 2m = n m = n+10

  36. DART Approach concretestate symbolicstate constraints x=m, y=n, z=2m x=-10, y=-20, z=-20 Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } Program Error 2m = n m = n+10

  37. DART Approach z==y x!=y+10 main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } N Y N Y Error

  38. DART in a Nutshell • Dynamically observe random execution and generate new test inputs to drive the next execution along an alternative path • do dynamic analysis on a random execution • collect symbolic constraints at branch points • negate one constraint at a branch point (say b) • call constraint solver to generate new test inputs • use the new test inputs for next execution to take alternative path at branch b • (Check that branch b is indeed taken next)

  39. More details • Instrument the C program to do both • Concrete Execution • Actual Execution • Symbolic Execution and Lightweight theorem proving (path constraint solving) • Dynamic symbolic analysis • Interacts with concrete execution • Instrumentation also checks whether the next execution matches the last prediction.

  40. Experiments • Tested a C implementation of a security protocol (Needham-Schroeder) with a known attack • 406 lines of code • Took less than 26 minutes on a 2GHz machine to discover middle-man attack • In contrast, a software model-checker (VeriSoft) and a hand-written nondeterministic model of the attacker took hours to discover the attack

  41. Larger Experiment • oSIP (open-source session initiation protocol) • http://www.gnu.org/software/osip/osip.html • 30,000 lines of C code (version 2.0.9) • 600 externally visible functions • Results • crashed 65% of the externally visible functions within 1000 iterations • no nullity check for pointers • Focused on oSIP parser • can externally crash oSIP server • osip_message_parse() : pass a buffer of size 2.5 MB with no 0 or “|” character • tries to copy the packet to stack using alloca(size) • this fails: returns NULL pointer • this NULL pointer passed to another function • does not check for nullity and crashes

  42. struct foo { int i; char c; } bar (struct foo *a) { if (a->c == 0) { *((char *)a + sizeof(int)) = 1; if (a->c != 0) { abort(); } } } Reasoning about dynamic data is easy Due to limitation of alias analysis “static analyzers” cannot determine that “a->c” has been rewritten Software model checker BLAST would infer that the program is safe DART finds the error Advantage of Dynamic Analysis over Static Analysis

  43. 1 foobar(int x, int y){ 2 if (x*x*x > 0){ 3 if (x>0 && y==10){ 4 abort(); 5 } 6 } else { 7 if (x>0 && y==20){ 8 abort(); 9 } 10 } 11 } static analysis based model-checkers would consider both branches both abort() statements are reachable false alarm Symbolic execution gets stuck at line number 2 DART finds the only error Further advantages

  44. void again_test_me(int x,int y){ z = x*x*x + 3*x*x + 9; if(z != y){ printf(“Good branch”); } else { printf(“Bad branch”); abort(); } } Let initially x = -3 and y = 7 generated by random test-driver Simultaneous Symbolic & Concrete Execution

  45. void again_test_me(int x,int y){ z = x*x*x + 3*x*x + 9; if(z != y){ printf(“Good branch”); } else { printf(“Bad branch”); abort(); } } Let initially x = -3 and y = 7 generated by random test-driver concrete z = 9 symbolic z = x*x*x + 3*x*x+9 take then branch with constraint x*x*x+ 3*x*x+9 != y Simultaneous Symbolic & Concrete Execution

  46. void again_test_me(int x,int y){ z = x*x*x + 3*x*x + 9; if(z != y){ printf(“Good branch”); } else { printf(“Bad branch”); abort(); } } Let initially x = -3 and y = 7 generated by random test-driver concrete z = 9 symbolic z = x*x*x + 3*x*x+9 take then branch with constraint x*x*x+ 3*x*x+9 != y solve x*x*x+ 3*x*x+9 = y to take else branch Don’t know how to solve !! Stuck ? Simultaneous Symbolic & Concrete Execution

  47. void again_test_me(int x,int y){ z = x*x*x + 3*x*x + 9; if(z != y){ printf(“Good branch”); } else { printf(“Bad branch”); abort(); } } Let initially x = -3 and y = 7 generated by random test-driver concrete z = 9 symbolic z = x*x*x + 3*x*x+9 take then branch with constraint x*x*x+ 3*x*x+9 != y solve x*x*x+ 3*x*x+9 = y to take else branch Don’t know how to solve !! Stuck ? NO : DART handles this smartly Simultaneous Symbolic & Concrete Execution

  48. void again_test_me(int x,int y){ z = x*x*x + 3*x*x + 9; if(z != y){ printf(“Good branch”); } else { printf(“Bad branch”); abort(); } } Let initially x = -3 and y = 7 generated by random test-driver Simultaneous Symbolic & Concrete Execution

  49. void again_test_me(int x,int y){ z = x*x*x + 3*x*x + 9; if(z != y){ printf(“Good branch”); } else { printf(“Bad branch”); abort(); } } Let initially x = -3 and y = 7 generated by random test-driver concrete z = 9 symbolic z = x*x*x + 3*x*x+9 cannot handle symbolic value of z Simultaneous Symbolic & Concrete Execution

  50. void again_test_me(int x,int y){ z = x*x*x + 3*x*x + 9; if(z != y){ printf(“Good branch”); } else { printf(“Bad branch”); abort(); } } Let initially x = -3 and y = 7 generated by random test-driver concrete z = 9 symbolic z = x*x*x + 3*x*x+9 cannot handle symbolic value of z make symbolic z = 9 (randomly chose a value) and proceed Simultaneous Symbolic & Concrete Execution

More Related