1 / 31

For Developer Testing

Separation of Concerns. For Developer Testing. Tao Xie Peking University, China North Carolina State University, USA. In collaboration with Nikolai Tillmann , Peli de Halleux , Wolfram Schulte @Microsoft Research and students @NCSU ASE. Background – Separation of Concerns.

nibaw
Download Presentation

For Developer Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Separation of Concerns For Developer Testing Tao Xie Peking University, China North Carolina State University, USA In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU ASE

  2. Background – Separation of Concerns • Separation of Concerns (SoC) • Aspect-Oriented Software Development (AOSD) • “Killer Apps” for SoC/AOSD? • Logging, Security Checking, … • Any Other Application Scenarios in Practice? • Tool Support for Software Testing?

  3. ? = Software Testing Setup + Expected Outputs Test inputs Program Outputs Test Oracles

  4. ? = Software Testing Problems + Expected Outputs Test inputs Program Outputs Test Oracles • Test Generation • Generating high-quality test inputs (e.g., achieving high code coverage)

  5. ? = Software Testing Problems + Expected Outputs Test inputs Program Outputs Test Oracles • Test Generation • Generating high-quality test inputs (e.g., achieving high code coverage) • Test Oracles • Specifying high-quality test oracles (e.g., guarding against various faults)

  6. The Recipe of Unit/Dev Testing • Three essential ingredients: • Data • Method Sequence • Assertions void TestAdd() { int item = 3; var list = new ArrayList(); list.Add(item); Assert.AreEqual(1, list.Count); }

  7. Outline • Parameterized Unit Testing • Separate Test Generation and Behavior Specification • Test Generalization • Localize Crosscutting Concerns of Behavior Under Test • Object Creation with Factory Method • Localize Crosscutting Concerns of Object Creation • Environment Isolation with Moles • Localize Crosscutting Concerns of Environment Dependency

  8. Parameterized Unit Testing [Tillmann&Schulte ESEC/FSE 05] • Parameterized Unit Test = Unit Test with Parameters • Separation of concerns • Data is generated by a tool • Developer can focus on functional specification void TestAdd(ArrayListlist, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); }

  9. Dynamic Symbolic Execution Choose next path • Code to generate inputs for: Solve Execute&Monitor void TestMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } Negated condition a==null F T a.Length>0 T F Done: There is no path left. a[0]==123… F T Data null {} {0} {123…} Observed constraints a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!=1234567890 a!=null && a.Length>0 && a[0]==1234567890 Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==1234567890

  10. Pex:Visual Studio Power Tool http://research.microsoft.com/projects/pex/ • Download counts (20 months)(Feb. 2008 - Oct. 2009 ) • Academic: 17,366 • Devlabs: 13,022 • Total: 30,388 Dynamic Symbolic Execution [Tillmann&deHalleux TAP 08]

  11. Pex for FunWeb-based Learning Tool Try it at http://www.pexforfun.com/ 669,584 clicked 'Ask Pex!‘ since 2010 summer

  12. Parameterized Unit TestingGetting Popular Parameterized Unit Tests (PUTs) commonly supported by various test frameworks • .NET: Supported by .NET test frameworks • http://www.mbunit.com/ • http://www.nunit.org/ • … • Java: Supported by JUnit 4.X • http://www.junit.org/ Generating test inputs for PUTs supported by tools • .NET: Supported by Microsoft Research Pex • http://research.microsoft.com/Pex/ • Java: Supported by Agitar AgitarOne • http://www.agitar.com/

  13. Parameterized vs. Conventional Unit Tests PUTs vs. CUTs • Benefits over CUTs • Help describe behaviours for all test arguments • Address two main issues with CUTs • Missing test data (that would exercise important behavior) • Low fault-detection capability • Redundant test data and scenarios(that exercises the same behaviour) • Redundant unit tests

  14. Outline • Parameterized Unit Testing • Separate Test Generation and Behavior Specification • Test Generalization • Localize Crosscutting Concerns of Behavior Under Test • Object Creation with Factory Method • Localize Crosscutting Concerns of Object Creation • Environment Isolation with Moles • Localize Crosscutting Concerns of Environment Dependency

  15. An Example using IntStack Three CUTs • CUT1 and CUT2 exercise push with different test data • CUT3 exercises push when stack is not empty • Two main issues with CUTs: • Fault-detection capability: undetected defect where things go wrong when passing a negative value to push • Redundant tests: CUT2 is redundant with respect to CUT1 public void CUT1() { intelem = 1; IntStackstk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count()); } public void CUT2() { intelem = 30; IntStackstk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count()); } public void CUT3() { int elem1 = 1, elem2 = 30; IntStackstk = new IntStack(); stk.Push(elem1); stk.Push(elem2); Assert.AreEqual(2, stk.Count()); }

  16. Test Generalization: IntStack An equivalent PUT Three CUTs public void CUT1() { intelem = 1; IntStackstk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count()); } public void PUT(int[] elem) { Assume.IsTrue(elem != null); IntStackstk = newIntStack(); for(inti in elem) stk.push(elem); Assert.AreEqual(elem.Length, stk.count()); } public void CUT2() { intelem = 30; IntStackstk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count()); } • No need to describe test data • Generated automatically • Single PUT replaces multiple CUTs • With reduced size of test code public void CUT3() { int elem1 = 1, elem2 = 30; IntStackstk = new IntStack(); stk.Push(elem1); stk.Push(elem2); Assert.AreEqual(2, stk.Count()); } [Thummalapenta et al. FASE 11]

  17. Ring a Bell? Refactoring Clones? • Refactoring clone instances (crosscutting concerns) to a single copy (aspects) • Need pointcuts to specify join points, i.e., which original code locations to weave the aspects • Test generalization is different • Original code locations (old CUTs) are thrown away • PUTs + Pex generate new CUTs  No need of join points or pointcuts

  18. Outline • Parameterized Unit Testing • Separate Test Generation and Behavior Specification • Test Generalization • Localize Crosscutting Concerns of Behavior Under Test • Object Creation with Factory Method • Localize Crosscutting Concerns of Object Creation • Environment Isolation with Moles • Localize Crosscutting Concerns of Environment Dependency

  19. Object Creation void TestAdd(ArrayListlist, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); }

  20. Object Creation • Pex uses public methods to configure non-public object fields • Heuristics built-in to deal with common types • User can help if needed [PexFactoryMethod(typeof(ArrayList))] public static ArrayList Create(intcapacity, object[] items) { varlist = new ArrayList(capacity); foreach (varitem in items) list.Add(item); return list; }

  21. “Join Point” for Factory Method void TestAdd(ArrayListlist, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); } [PexFactoryMethod(typeof(ArrayList))] public static ArrayList Create(intcapacity, object[] items) { varlist = new ArrayList(capacity); foreach (varitem in items) list.Add(item); return list; }

  22. Outline • Parameterized Unit Testing • Separate Test Generation and Behavior Specification • Test Generalization • Localize Crosscutting Concerns of Behavior Under Test • Object Creation with Factory Method • Localize Crosscutting Concerns of Object Creation • Environment Isolation with Moles • Localize Crosscutting Concerns of Environment Dependency

  23. Unit Testing is Not That Easy • Components depend on other components • Hidden Integration Tests boolIsFileEmpty(string file) { var content = File.ReadAllText(file); return content.Length == 0; } File.ReadAllText(file); void FileExistsTest() { File.Write(“foo.txt”, “”); var result = IsFileEmpty(“foo.txt”) Assert.IsTrue(result); } File.Write(“foo.txt”, “”);

  24. Isolation is Critical • Slow, complicated setup, non-deterministic tests • Solution: Replace by Simpler Environment (“mocking”) • Testable Design: Abstraction layer + Dependency Injection + Mocks for testing • Simply use virtual methods (i.e., overridenable) • Hard-coded Design: No abstraction layer, static methods, sealed types (i.e., not inheritable).

  25. Stubs and Moles Framework • Replace any .NET method with a Delegate • similar to a C++ function pointer • Method can be overridden? Use Stubs • Interfaces • Abstract classes • Virtual methods in non-sealed types • Method cannot be overridden? Use Moles

  26. Hard-coded Design • Existing external components cannot be re-factored • SharePoint, ASP .NET, … • Need mechanism to stub non-virtual methods • Static methods, methods in sealed types, constructors

  27. Moles– Delegate Based Detours • Method redirected to user delegate, i.e. moled • Requires code instrumentation MFile.ReadAllTextString = file => “”; expression lambda : (input parameters) => expression bool result = IsFileEmpty(“foo.txt”); Assert.IsTrue(result); [de Halleux&Tillmann TOOLS 10]

  28. Moles under the Hood mscorlib.dll File.ReadAllText(string name) { } .NET Runtime Just In Time Compiler File.ReadAllText(string name) { var d = GetDetour(); if (d != null) return d(); } push ecx push edx push eax ExtendedReflection

  29. Isolated Parameterized Unit Testing • Automated White box Analysis does not ‘understand’ the environment • Isolate Code using Stubs and Moles ??? if (DateTime.Now == new DateTime(2000,1,1)) throw new Y2KException(); DateTime.Now public void TestY2k(DateTimedt) {MDateTime.NowGet = () => dt ... } MDateTime.NowGet = () => dt DateTime.Now == dt

  30. Separation of Concerns for Developer Testing • Parameterized Unit Testing • Separate Test Generation and Behavior Specification • Test Generalization • Localize Crosscutting Concerns of Behavior Under Test • Object Creation with Factory Method • Localize Crosscutting Concerns of Object Creation • Environment Isolation with Moles • Localize Crosscutting Concerns of Environment Dependency Application Scenarios of Separation of Concerns in Practice

  31. Thank you! http://research.microsoft.com/projects/pex/ Questions ? http://www.pexforfun.com/ http://research.microsoft.com/en-us/projects/pex/community.aspx#publications http://pexase.codeplex.com/ https://sites.google.com/site/asergrp/

More Related