Software Reliability Modelling

1. Software Reliability Modelling Engin Deveci

2. Agenda • What is Software Reliability? • Software Failure Mechanisms • Hardware vs Software • Measuring Software Reliability • Software Reliability Models • Statistical Testing • Conclusion

3. What is Software Reliability • the probability of failure-free software operation for a specified period of time in a specified environment • Probability of the product working “correctly” over a given period of time. • Informally denotes a product’s trustworthiness or dependability.

4. What is Software Reliability • Software Reliability is an important attribute of software quality, together with • functionality, • usability, • performance, • serviceability, • capability, • installability, • maintainability, • documentation.

5. What is Software Reliability • Software Reliability Modeling • Prediction Analysis • Reliability Measurement • Defect Classification • Trend Analysis • Field Data Analysis • Software Metrics • Software Testing and Reliability • Fault-Tolerance • Fault Trees • Software Reliability Simulation • Software Reliability Tools From Handbook of Software Reliability Engineering, Edited by Michael R. Lyu, 1996

6. What is Software Reliability • Software Reliability is hard to achieve, because the complexity of software tends to be high. • While the complexity of software is inversely related to software reliability, it is directly related to other important factors in software quality, especially functionality, capability.

7. Software Failure Mechanisms • Failure cause: Software defects are mainly design defects. • Wear-out: Software does not have energy related wear-out phase. Errors can occur without warning. • Repairable system concept: Periodic restarts can help fix software problems. • Time dependency and life cycle: Software reliability is not a function of operational time. • Environmental factors: Do not affect Software reliability, except it might affect program inputs. • Reliability prediction: Software reliability can not be predicted from any physical basis, since it depends completely on human factors in design.

8. Software Failure Mechanisms • Redundancy: Can not improve Software reliability if identical software components are used. • Interfaces: Software interfaces are purely conceptual other than visual. • Failure rate motivators: Usually not predictable from analyses of separate statements. • Built with standard components: Well-understood and extensively-tested standard parts will help improve maintainability and reliability. But in software industry, we have not observed this trend. Code reuse has been around for some time, but to a very limited extent. Strictly speaking there are no standard parts for software, except some standardized logic structures.

9. Software Failure Mechanisms

10. Software Failure Mechanisms

11. Measuring Software Reliability Don’t define what you won’t collect.. Don’t collect what you won’t analyse.. Don’t analyse what you won’t use..

12. Measuring Software Reliability • Measuring software reliability remains a difficult problem because we don't have a good understanding of the nature of software • Even the most obvious product metrics such as software size have not uniform definition.

13. Measuring Software Reliability • Current practices of software reliability measurement can be divided into four categories: • Product metrics • Project management metrics • Process metrics • Fault and failure metrics

14. Measuring Software Reliability • Different categories of software products have different reliability requirements: • level of reliability required for a software product should be specified in the SRS document. • A good reliability measure should be observer independent, • so that different people can agree on the reliability.

15. Measuring Software Reliability • LOC, KLOC, SLOC, KSLOC • McCabe's Complexity Metric • Test coverage metrics • ISO-9000 Quality Management Standards • MTBF • Once a failure occurs, the next failure is expected after 100 hours of clock time (not running time).

16. Measuring Software Reliability • Failure Classes • Transient: • Transient failures occur only for certain inputs. • Permanent: • Permanent failures occur for all input values. • Recoverable: • When recoverable failures occur the system recovers with or without operator intervention. • Unrecoverable: • the system may have to be restarted. • Cosmetic: • May cause minor irritations. Do not lead to incorrect results. • Eg. mouse button has to be clicked twice instead of once to invoke a GUI function.

17. Measuring Software Reliability • Errors do not cause failures at the same frequency and severity. • measuring latent errors alone not enough • The failure rate is observer-dependent • No simple relationship observed between system reliability and the number of latent software defects. • Removing errors from parts of software which are rarely used makes little difference to the perceived reliability. • removing 60% defects from least used parts would lead to only about 3% improvement to product reliability. • Reliability improvements from correction of a single error depends on whether the error belongs to the core or the non-core part of the program. • The perceived reliability depends to a large extent upon how the product is used. In technical terms on its operation profile.

18. Software Reliability Models • Software reliability models have emerged as people try to understand the characteristics of how and why software fails, and try to quantify software reliability • Over 200 models have been developed since the early 1970s, but how to quantify software reliability still remains largely unsolved • There is no single model that can be used in all situations. No model is complete or even representative.

19. Software Reliability Models • Most software models contain  the following parts: • assumptions, • factors, • a mathematical function • relates the reliability with the factors. • is usually higher order exponential or logarithmic.

20. Software Reliability Models • Software modeling techniques can be divided into two subcategories: • prediction modeling • estimation modeling. • Both kinds of modeling techniques are based on observing and accumulating failure data and analyzing with statistical inference.

21. Software Reliability Models

22. Software Reliability Models • There are two main types of uncertainty which render any reliability measurement inaccurate: • Type 1 uncertainty: • our lack of knowledge about how the system will be used, i.e. • its operational profile • Type 2 uncertainty: • reflects our lack of knowledge about the effect of fault removal. • When we fix a fault we are not sure if the corrections are complete and successful and no other faults are introduced • Even if the faults are fixed properly we do not know how much will be the improvement to interfailure time.

23. Software Reliability Models • Step Function Model • The simplest reliability growth model: • a step function model • The basic assumption: • reliability increases by a constant amount each time an error is detected and repaired. • Assumes: • all errors contribute equally to reliability growth • highly unrealistic: • we already know that different errors contribute differently to reliability growth.

24. Software Reliability Models • Jelinski and Moranda Model • Realizes each time an error is repaired reliability does not increase by a constant amount. • Reliability improvement due to fixing of an error is assumed to be proportional to the number of errors present in the system at that time.

25. Software Reliability Models • Littlewood and Verall’s Model • Assumes different fault have different sizes, thereby contributing unequally to failures. • Allows for negative reliability growth • Large sized faults tends to be detected and fixed earlier • As number of errors is driven down with the progress in test, so is the average error size, causing a law of diminishing return in debugging

26. Software Reliability Models • Variations exists • LNHPP (Littlewood non homogeneous Poisson process) model • Goel – Okumoto (G-O) Imperfect debugging model • GONHPP • Musa – Okumoto (M-O) Logarithmic Poisson Execution Time model

27. Software Reliability Models • Applicability of models: • There is no universally applicable reliability growth model. • Reliability growth is not independent of application. • Fit observed data to several growth models. • Take the one that best fits the data.

28. Software Reliability Models

29. Software Reliability Models

30. Software Reliability Models

31. Software Reliability Models • Observed failure intensity can be computed in a straightforward manner from the tables of failure time or grouped data (e.g. Musa et al. 1987).

32. Software Reliability Models • Example: (136 failures total): • Failure Times (CPU seconds): 3, 33, 146, 227, 342, 351, 353,444, 556, 571, 709, 759, 836 ..., 88682. • Data are grouped into sets of 5 and the observed intensity, cumulative failure distribution and mean failure times are computed, tabulated and plotted.

33. Software Reliability Models

34. Software Reliability Models • Two common models are the "basic execution time model“ and the "logarithmic Poisson execution time model" (e.g. Musa et al. 1987).

35. Software Reliability Models • Basic Execution Time Model • Failure intensity λ(τ) with debugging time τ: • where λ0 is the initial intensity and ν0 is the total expected number of failures (faults).

36. Software Reliability Models • where μ(τ) is the mean number of failures experienced by time τ.

37. Software Reliability Models • In this case the Logarithmic-Poisson Model fits somewhat better than the Basic Execution Time Model. • In some other projects BE model fits better than LP model.

38. Software Reliability Models • Additional expected number of failures, Δμ, that must be experienced to reach a failure intensity objective • where λP is the present failure intensity, and λF is the failure intensity objective. The additional execution time, Δτ, required to reach the failure intensity objective is

39. Software Reliability Models • After fitting a model describing the failure process we can estimate its parameters, and the quantities such as the total number of faults in the code, future failure intensity and additional time required to achieve a failure intensity objective.

40. Statistical Testing • The objective is to determine reliability rather than discover errors. • Uses data different from defect testing.

41. Statistical Testing • Different users have different operational profile: • i.e. they use the system in different ways • formally, operational profile: • probability distribution of input • Divide the input data into a number of input classes: • e.g. create, edit, print, file operations, etc. • Assign a probability value to each input class: • a probability for an input value from that class to be selected.

42. Statistical Testing • Determine the operational profile of the software: • This can be determined by analyzing the usage pattern. • Manually select or automatically generate a set of test data: • corresponding to the operational profile. • Apply test cases to the program: • record execution time between each failure • it may not be appropriate to use raw execution time • After a statistically significant number of failures have been observed: • reliability can be computed.

43. Statistical Testing • Relies on using large test data set. • Assumes that only a small percentage of test inputs: • likely to cause system failure. • It is straight forward to generate tests corresponding to the most common inputs: • but a statistically significant percentage of unlikely inputs should also be included. • Creating these may be difficult: • especially if test generators are used.

44. Conclusions • Software reliability is a key part in software quality • Software reliability improvement is hard • There are no generic models. • Measurement is very important for finding the correct model. • Statistical testing should be used but it is not easy again… • Software Reliability Modelling is not as simple as described here. 

45. Thank you. Q&A