1 / 61

Jonathan W. Lartigue

Jonathan W. Lartigue. Course Objectives (Goals). Upon completion of this course, the student will be able to: Describe how software testing fits into the MIL-STD-498 process. Distinguish between Software Unit Testing and Software Integration Testing techniques.

reneethompson
Download Presentation

Jonathan W. Lartigue

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Jonathan W. Lartigue

  2. CourseObjectives(Goals) • Uponcompletionofthiscourse,thestudentwillbeableto: • DescribehowsoftwaretestingfitsintotheMIL-STD-498process. • DistinguishbetweenSoftwareUnitTestingandSoftware IntegrationTestingtechniques. • DescribethemanyapproachestoSoftwareUnitTesting, theirstrengthsandtheirlimitations. • Describethe several approachestoSoftwareIntegrationTestingandtheirstrengthsandlimitations.

  3. CourseOutline(Topics) • Topicswewillcoverinclude: • SoftwareTesting • StagesofaSoftware Project • TestPlanning • SoftwareUnitTesting • BlackBoxTestingMethods • WhiteBoxTestingMethods • SoftwareIntegrationTesting • IntegrationTestMethods • CSCIQualificationTesting

  4. SoftwareTesting(Introduction) • Howcanweestablishsomedegreeofconfidencethataprogramdoeswhatitissupposedtodo? • Withthorough,exhaustiveunitandfunctionaltesting. ‘Thereisnosingledevelopment,ineithertechnologyorinmanagementtechnique,thatbyitselfpromisesevenoneorder-of-magnitudeimprovementin productivity,inreliability,insimplicity.’ —FredBrooks

  5. SoftwareTestPlanning • Whenshouldwestartthinkingaboutsoftware testing? • Whenthesoftwareisalreadywrittenandduetobeshipped? • Whenwe’rewritingthesoftware? • Whenwe’redesigningthesoftware? • Whenwe’redesigningthesystemthesoftwarerunson?

  6. Needfortesting(Whybother?) • Startingattheverybeginningofaproject,theearlieranerrorordefectisdiscovered,themorecost-efficientitsremovalbecomes. Cost of Correcting Errors by Project Stage 250 200 150 100 50 0 Analysis Design Code Test Maintain

  7. Needfortesting(Whybother?) • ‘Theprobabilityoftheexistenceofmoreerrorsina sectionofaprogramisproportionaltothenumberoferrorsalreadyfoundinthatsection.’ • —GlenfordMyers • Probability ofundiscoverederrors versuserrorsfound 50 40 30 20 10 0 UndiscoveredErrors 10 20 30 ErrorsFound 40 Source:TheArt of SoftwareTesting, GlenfordMyers, 1979

  8. Needfortesting(Whybother?) • We’reonlyhuman… • Example: • Oct.1999lossofNASAMarsClimate Orbiter hasbeenattributedtoametricsystemtoEnglishsystemofmeasurementerror. • Engineersmistakenlyassumedmeasurementswereinadifferentsystem. • CosttoNASA:$125,000,000+ • Couldhavebeen(shouldhavebeen?)caughtduringtesting! Source: SoftwareQAandTestingResourceCenter

  9. Definitions(SoftwareTesting) • SoftwareTesting—theprocessofexecutingaprogramwiththeintentoffindingerrors. • Forevenwell-writtensoftware,testingistypicallythelongestphaseofadevelopmentcycle. • TestPlan—anartifactthatencapsulatesassociatedtestcasesandprovidesanoverviewoftherelationshipsofthetestset. • TestCase—anartifactthatdescribesaninput,action,oreventinthesystemsandtheexpectedresult(s). • Source: SoftwareQAandTestingResourceCenter

  10. Definitions(SoftwareTestingPhases) • Softwaretestingcanoftenbedividedintotwoseparatephases: • SoftwareUnitTestingistheprocessbywhichanindividualsoftwarecomponentistested,isolatedfromallothersoftwareinthesystem. • SoftwareIntegrationTestingistheprocessbywhich separatesoftwaremodulesaremergedandtheir interfacesaretestedagainstrequirements. • SoftwareUnitTestingandSoftwareIntegrationTestingaredistinctandseparate activities.

  11. Definitions(TestPlan) • Toperformsoftwaretesting,wefirstneedasoftwaretestplan. • Atestplanwilldescribeindetail: • Acceptancecriteria. • Anapproachtotestingandintegrating. • Taskstobeperformed. • Aschedule/timeline. • Individualresponsibilitiesofteammembers.

  12. Definitions(TestCase) • Testcases: • Asuccessfultestcaseisonethatdiscoversan as-yetundetectederror. • Itisimpossibletohavetestcasesforeverystateinasystem. • Thenumberofpossibleinputsintoasystemincreasesexponentially with thecomplexityofthatsystem.

  13. Definitions(TestCase) • Testcases: • Itoftenrequiresonelineoftestcodetovalidateeachlineofdeliverablecodeinasystem • Testcodeis sometimes called‘throwaway’code–itdoesn’tgetdeliveredtothecustomer. • Therefore,itisnotunusualforsoftwareengineerstodiscard50percentormorethecodetheywrite. • Manyengineersrealizethisandthereforeminimizetestingeffortsbecausetheydon’tproduce‘deliverablecode.’ • Thisisapoorphilosophytoadopt.

  14. UnitTestingvs.IntegrationTesting • UnitandIntegrationTesting: • S/WUnitTestingallowsforeasierdetectionandlocationoferrorsinthesystem. • Isiteasiertofindanerrorconfinedtooneprocedureorspanningseveral files? • S/WIntegrationallowstestingof interactions betweensoftwaremodules. • Candetectinconsistenciesinparalleldesign(assumptionsthatdifferacrossmultiplemodulesbecauseeachwaswrittenbyadifferentengineer). • UnitandIntegrationtestingareNOTthesame!

  15. Definitions(HigherOrderTesting) • CSCIQualificationTestingwillusuallypitthe integratedsystemasawholeagainstitsownsoftwaresystemrequirements. • “Ourcustomerstatedthatweneedtohavethiseventtakeplaceifthisspecifiedconditionoccurs.” • Doesthis happen, andhowdoweverifyit? • Example: • EithertheRedSwitchsoftwaremoduleXproperlycommunicates with moduleYoritdoesn’t.

  16. Definitions(HigherOrderTesting) • SystemTestingwilltestthesoftwarewiththerestofthesystemcomponentsandverifiesthesystemoperationmeetsitsspecifications. • “Ourcustomerstatedthatweneedtohavethiseventtakeplaceifthisspecifiedconditionoccurs.” • Doesthis happen, andhowdoweverifyit? • Example: • EithertheRedSwitchsoftwarestillfunctionswhencircuitboard • ‘X’fails,oritdoesn’t.

  17. Comparison(Conclusion) • Conclusion • SoftwareUnitTesting,SoftwareIntegration,andSoftwareSystemIntegrationeachserveadistinctandvaluablepurpose. • Theeffectivenessandefficiencyofanyoneofthethreebecomeslessenedifanotherisremovedfromtheoverallprocess. • Totestasoftwaresystemsuccessfully,we needtoimplementallthreeapproaches.

  18. SoftwareTesting

  19. S/WUnitTestMethods • Breakdownofunittestingapproaches: • UnitTesting StaticTechniques DynamicTechniques BlackBox Testingw/oexecutingcode •EquivalencePartitioning •BoundaryValueAnalysis •RandomTesting •Cause-EffectGraphing Reviews,Inspections, etc. CleanroomTesting White Box •Basis-PathTesting SymbolicExecution •Statement-level coverage •Branch-levelcoverage •Condition-level coverage Source:T. Hilburn, Embry-RiddleAero. Univ.

  20. Black-BoxTesting • Black-BoxTesting: • Alsoknownas‘data-driventesting’,itreliesonamoduledescriptiontodevisetestcases. • Usesthearchitecturaldesign,includinginputs,functionality,andoutputs. input output

  21. Black-BoxTesting • Black-BoxTesting: • Treatsmodulelikea • “blackbox” • i.e.,youcan’tseetheinnerworkingsofthebox,justwhatgoesin andcomesout. • Blackboxtestcasesshouldbedesigned beforethecoding/implementationphaseends. input output

  22. White-Box/Clear-BoxTesting • White-BoxTesting: • Oftencalled‘structuraltesting’,it reliesonanalysisofamodule’ssourcecodetodesignatestcase. • Morecloseanalysisofthedetaileddesignofamodule. input output

  23. White-Box/Clear-BoxTesting • White-BoxTesting: • Treatsmodulelikea • “white-box” • Wecanseeandtesttheinnerworkingsofthebox. • Testcasesdesignedasneeded,onacase-by-casebasis. • Whiteboxtesting shouldsupplementblackboxtesting. input output

  24. SoftwareTestingDiscussion

  25. Black-BoxTestingApproaches • EquivalencePartitioning: • Divideinputdomainofaprogramintoclassesfromwhichtestcasesarederived: • Theunit’sdesignisanalyzedtoidentifyvalidinputvalues. • Whencombined with thespectrumofpossiblevalues,oneormorediscrete ‘partitions’ canbeidentified. • Thesepartitionsrepresentrangesfromwhichtestcasevaluescanbeselected.

  26. EquivalencePartitioning,example • Equivalence1: • ◆x<65 • Equivalence2: • ◆65<=x<=90 • Equivalence3: • ◆91<=x<=96 • Equivalence4: • ◆97<=x<=122 • Equivalence5: • ◆122<x • Possible test case inputs: 23, 71, 94, 108,147

  27. Black-BoxTestingApproaches • Boundary-ValueAnalysis: • Testcasesaredesignedontheboundariesofacceptableinputdomain. • Studieshaveshownthatmoreerrorsoccurontheboundaryofaninputdomainthaninthecenter. • e.g.The‘infamous’off-by-oneerror. • Complementsequivalencepartitioningandisoftenusedin conjunctionwithit.

  28. Boundary-ValueAnalysis,example • Boundary1: • ◆65 • Boundary2: • ◆90 • Boundary3: • ◆97 • Boundary4: • ◆122 • Possible test case inputs: 64, 65, 66 • 89, 90, 91 • 96, 97, 98 • 121,122,123

  29. Black-BoxTestingApproaches • RandomTesting: • Sometimes, somepeoplearejust“good”attesting(culminationofintuition,experience,detailedknowledgeofsystem). • Norealprocedurefordesigningtestcases,couldbebasedonguessingwhereerrorsmightoccur. • Alsoknownas“errorguessing”. • Example:Sortingalgorithm • Somerandom,but‘good’testsmightbe: • Emptyinputlist,single entry inlist,allentrieshavesamevalue,entry listisalready sorted, etc.

  30. SoftwareTestingDiscussion

  31. White-BoxTestingApproaches • White-boxtesting: • isverytimeintensive,asit requiresdetailedanalysisofinternalsourcecode. • isdesignedtobeextremelythoroughandoftentoucheseverysinglelineofcode. input output • Whiteboxtestingisconsideredessentialformission-criticalsoftware(i.e.nosurprises)andistheonlyacceptedwaytocompletelyvalidatesoftware. • Remember,evenwhite-boxtestingcannottesteverypossible input combination.

  32. White-BoxTestingApproaches • Basis-PathTesting: • Consistsofsteps: • Convertunitundertestintoa‘flowgraph’ • Computeameasureofthemodule’slogicalcomplexity • » McCabeMetrics‘cyclomaticcomplexity’ • Usethemeasuretoderivea‘basis’setofexecutionpaths,whichareusedtogeneratetestcases.

  33. Basis-PathTesting,example start bool isLetter(char value) { if (value >= 65 return true; if (value >= 97 && value <= 90) // A-Z && FALSE 65>= value <= 122) TRUE return true; return false; // a-z FALSE <=90 } TRUE • Wecancreateaflowgraphtodetailthedecisionpathsthroughthecode. • Byidentifyingallpossiblepaths,wecanidentifythemaximumnumberoftestcasesneeded. FALSE 97>= returntrue TRUE FALSE <=122 TRUE returntrue returnfalse end

  34. Basis-PathTesting,example • Ifthefirstconditionofthefirstifstatementisfalse,therearethreepossiblepaths. • Remember,C++Booleanconditionalsare‘shortcircuited.’ start start start FALSE FALSE FALSE 65>= 65<>= 65>= TRUE TRUE TRUE FALSE FALSE <=90 FALSE <=90 <=90 TRUE FALSE TRUE TRUE FALSE 97>= FALSE 97>= 97>= return true TRUE return true return true TRUE TRUE FALSE FALSE <=122 FALSE <=122 <=122 TRUE TRUE TRUE return true return false return true return false return true return false end end end

  35. Basis-PathTesting,example • Whenthefirstconditionofthefirstifstatementistrue,butthesecondfalse,therearethreemorepathsavailable. start start start FALSE FALSE FALSE 65>= 65<>= 65>= TRUE TRUE TRUE FALSE FALSE <=90 FALSE <=90 <=90 TRUE FALSE TRUE TRUE FALSE 97>= FALSE 97>= 97>= return true TRUE return true return true TRUE TRUE FALSE FALSE <=122 FALSE <=122 <=122 TRUE TRUE TRUE return true return false return true return false return true return false end end end

  36. Basis-PathTesting,example • Finally,thereisonlyone pathifbothconditionsonthefirstifstatementaretrue. • Eventhoughwetestedonlythefirst‘if’statement,we produced7uniquepathsthroughthecode. • Luckily,wealsotestedthepossiblepathsthroughtheotherif statement. • Minimum‘basisset’ofpaths: • Withsomerefinement,wecould reducethenumberofpathsneededtotouchevery‘leg’ofthecodegraphtofour. • Thecode’s‘cyclomaticcomplexity,’ thenumberof‘basis set’ paths,is4. • Conclusion: • Basis-pathtestingiscomplex: • Thisresultsinanexponentialnumberofpossibletestcases. • Automatedtoolsareamust. start FALSE 65>= TRUE FALSE <=90 TRUE FALSE 97>= return true TRUE FALSE <=122 TRUE return true return false end

  37. White-BoxTestingApproaches • Statement-levelcoverage: • Testcasesinthiscategory,asthenameimplies,havethegoalofreachingeveryexecutablestatementinthecodeatleastonce. • Fewer testcasesarerequiredwiththisapproachthanareneededinbasis-pathtesting. bool isLetter(char value) { if (value >= 65 && value <= 90) { return true; } // A-Z else if (value >= 97 && value <= 122) { return true; } // a-z return false; } RECALL: C++ isshort circuited, so statement-level coverage may not fully test ifstatements.

  38. Statement-levelCoverage,example • Thefirstcasetestsapathtoreachthefirst‘true’return condition. • Possibleinputvalue:70 • Thesecondtestcasereachestheother‘true’return condition. • Possibleinputvalue:105

  39. Statement-levelCoverage,example • Finally,wemustmakesurethatourtestreachesthe‘false’ returncondition,statement3. • Possibleinputvalue: 40. • Conclusions: • Wemanagedtocoverallstatementsinourcodewith onlythreetestcases. • However,ourtestingisnotas‘bulletproof’asthebasis-path testcases.

  40. White-BoxTestingApproaches • Branch-levelcoverage: • Hasthegoaloffollowingeachtrue/falsebranchofaprogram. bool isLetter(char value) { if (value >= 65 && value <= 90) { return true; } // A-Z 3. if (value >= 97 && value <= 122) { return true; } // a-z else { return false; } }

  41. Branch-levelCoverage,example • Firstwetestthe‘true’condition(branch)forthefirstifstatement. • Thisallowsustocover • lines1&2. • Nextwetestthe‘false’branchfollowedbya‘true’branchonthesecondif statement. • Thiscoverslines3&4. TRUE FALSE TRUE

  42. Branch-levelCoverage,example • Finallywetestthe‘false’branchofthesecondifstatement. • Thiscoverslines5&6. FALSE FALSE • Conclusions: • Eventhoughbranch-levelcoverageisnotconcerned with testingindividuallines,ithasthataddedbenefit. • Branch-levelcoverageisamorerobustalternativetostatement-levelcoverage,butaddstothenumberoftestcases.

  43. White-BoxTestingApproaches • Condition-levelcoverage: • Hasthegoaloftestingbothtrueandfalseconditionsfor allbranchconditions. • Thisisespeciallyapplicabletocompound‘if’statements. Example: bool isLetter(char value) { COND #1 COND #2 <= 90) • We can see that there are four distinct conditionals in this code. • Each one must be tested true and false at least once. 1. 2. if (value >= 65 && value { return true; } // A-Z COND #3 COND #4 <= 122) 3. 4. 6. 7. if (value >= 97 && value { return true; } // a-z else { return false; } }

  44. Condition-levelCoverage,example bool isLetter(char value) { TRUE TRUE if (value >= 65 && value <= 90) { return true; } // A-Z DON’T CARE DON’T CARE 3. if (value >= 97 && value <= 122) 4. { return true; } // a-z else { return false; } } • First,wetestbothtrueconditionsonthefirst‘if’ statementanddon’tcareaboutthesecondifstatement. • Next,wetesttrue/falsecombinationsonboth‘if’ statements.Notetheseconditionsarecalculatedtotestas-yet-untestedconditions. bool isLetter(char value) { TRUE FALSE if (value >= 65 && value <= 90) { return true; } // A-Z TRUE FALSE 3. if (value >= 97 && value <= 122) 4. { return true; } // a-z else { return false; } } =Truecond.onlycovered =Falsecond.onlycovered =Bothconditionscovered

  45. Condition-levelCoverage,example bool isLetter(char value) { • Finally,wemustdesigntestcasesthatcovertheremainingconditions. • Wecanaccomplishthis withtwomoretestcases. • Conclusions: • Decision-level coverageisverythoroughandrobust. • Eliminatessurpriseerrorsduetoshort-circuitingthatcanescapemanytestingapproaches. 4. 6. 7. { return else { return true; } // a-z false; } } bool isLetter(char value) { FALSE DON’T CARE if (value >= 65 && value <= 90) { return true; } // A-Z FALSE TRUE 3. if (value >= 97 && value <= 122) 4. { return true; } // a-z else { return false; } } =Truecond.onlycovered =Falsecond.onlycovered =Bothconditionscovered

  46. IntegrationTestMethods • SoftwareIntegrationTesting:“…[the]testinginwhichsoftwarecomponentsarecombinedandtestedtoevaluatetheinteractionbetweenthem…” • –IEEEStandardComputerDictionary • Integrationtestingcandetectinconsistenciesinparalleldesign(assumptionsthatdifferacrossmultiplemodulesbecauseeachwaswrittenbyadifferentengineer). • Incontrasttounittesting,wherewemighthaveavoided(or‘stubbed’)callstoothermodules,integration testingwillfocusontheinterfacebetweensoftwaremodules.

  47. IntegrationTestMethods • Therearetwomainapproachestosoftwareintegrationtesting: • Incremental: • Onepieceofthepuzzleatatime,followingageneral order. • » Top-Down • » Bottom-Up • Non-Incremental: • Nospecifiedordertointegration,ifatall. • » ‘Big-Bang’

  48. IncrementalTesting • Top-DownIntegration: • Startsatthetopoftheprogramhierarchyandtravelsdownitsbranches. • Twoapproachestotop-downintegration: • breadth-first • depth-first

  49. IncrementalTesting • Breadth-firstintegration: • Followsacrossthehierarchy‘lefttoright,’completingcoverageofeachlevelbeforemovingdownto • thenextone. • Thisintegrateseachofthecoreunitsbeforeincludingthelessgeneralmodules. • Thiscanrequiretheuseofstubsforlower • modulesnotyetincluded.

More Related