The Safety Risk of Requirements Incompleteness

1. The Safety Risk of Requirements Incompleteness Jeffrey Howard Patrick Anderson Safeware Engineering Corporation

2. Many incidents and accidents have been linked to flaws in real-time embedded system software Software-related errors are most often requirements errors, particularly incompleteness A specification is incomplete if required behavior is omitted or subject to more than one interpretation Requirements Incompleteness Safeware Engineering Corporation

3. Professor Nancy Leveson has compiled over 60 completeness criteria to address this problem, covering: Human-Computer Interface Trigger Events Robustness Nondeterminism Values and Timing Data Age Feedback And More Validated at JPL and used at Safeware SpecTRM-RL (SpecTRM Requirements Language) enforces these criteria Completeness Criteria Safeware Engineering Corporation

4. Today’s Example Accident • The importance of the criteria is easily demonstrated when they are ignored • No one wants their embarrassing stories told in a conference session • Everything you see here is false • Everything you see here is true • The ElectroShear 2000 Accident Safeware Engineering Corporation

5. ElectroShear 2000 Schematic Safeware Engineering Corporation

6. Shearing pen, where shearing is done Entry and exit gates Gate position sensors Gate actuators Gate locks Four mechanical arms mounted with electric trimmers Three sheep detection sensors Digital camera Weight plate Thermal sensor Trimmer head sensors Wool sensor Skin flush-fit sensor ElectroShear 2000 Shearing Pen Safeware Engineering Corporation

7. The system begins with entry gate open and exit gate closed Workers load a sheep and close the entry gate At least two of the three sheep detection sensors agree on the sheep’s presence The system shears, adjusting trimmer position using the skin flush-fit sensor The wool detection sensor is ignored - the software detects its own completion After shearing, the exit gate opens Collect wool and repeat Normal Operation Safeware Engineering Corporation

8. A technician replaced the trimmer blades in a pen, then greased the entry gate While manually moving the gate, he lowered it to the point of closing it The system exited standby mode and began a shearing cycle The technician was caught in the pen and sheared The system behaved erratically during shearing, and three of the four mechanical arms were damaged The Accident Safeware Engineering Corporation

9. Technician’s Statement • “My next work order was pen #22. The guys working with it had complained that the entrance gate was moving slowly and making some noise. As long as I was there, I was supposed to replace the trimmer heads. They were overdue. I got there and the guys unloaded the sheep they were putting into the pen. They put the pen into standby, so I lifted the exit gate, disconnected the weight plate, and went in to replace the trimmer heads. After that, I sprayed some grease on the gate tracks and worked it by hand a little to get the grease spread out. The machine just went crazy on me. It was a close shave.” Safeware Engineering Corporation

10. The Investigation • ElectroShear’s documentation jumbled requirements and design • Accident investigators used SpecTRM-RL to explore the system’s behavior • SpecTRM-RL uses text attributes and AND/OR tables to represent software behavior Safeware Engineering Corporation

11. SpecTRM-RL Safeware Engineering Corporation

12. SpecTRM-RL (2) Safeware Engineering Corporation

13. Gates do not require frequent maintenance Maintenance procedures require the gates to stay open during maintenance Designers didn’t anticipate entrance gate closings during standby mode Entrance gate closing during standby mode moves the pen into loaded mode Why did the system leave Standby Mode? Safeware Engineering Corporation

14. Shearing Pen Mode Logic Safeware Engineering Corporation

15. Criterion: Nondeterminism • “The behavior of the state machine should be deterministic (only one possible transition out of a state is applicable at any time.” • Automated tools can check this Safeware Engineering Corporation

16. The system classified the technician as a sheep Two of the three sensors must agree Digital Camera Thermal Sensor Weight Plate The camera mistook the human on all fours as a sheep The software still had obsolete input data queued from the disconnected weight plate Was the technician a ram? Safeware Engineering Corporation

17. Weight Plate Input Safeware Engineering Corporation

18. Criterion: Data Age • “All inputs used in specifying output events must be properly limited in the time they can be used (data age).” • In SpecTRM-RL, all inputs have an Obsolete value Safeware Engineering Corporation

19. If the exit gate is open, the shearing cycle shouldn’t start During the accident, it was open No escape for the technician When the system went into standby mode, exit gate position sensors were ignored The system came out of standby mode with an incorrect system model Why was the exit gate open? Safeware Engineering Corporation

20. Exit Gate Position Logic Safeware Engineering Corporation

21. Criterion: State Completeness • “The internal software model of the process must be updated to reflect the actual process state at initial startup and after temporary shutdown.” • SpecTRM-RL requires states to have an Unknown state value Safeware Engineering Corporation

22. The wool sensor didn’t detect wool being sheared That didn’t stop the shearing cycle System engineers provided a wool sensor to detect the end of shearing The software keeps track of shearing completion as progress along the planned shearing path The software ignores the sensor, because it’s easier to detect the end of shearing as running out of planned shearing path What about the wool sensor? Safeware Engineering Corporation

23. Criterion: Input Variable Completeness • “All information from the sensors should be used somewhere in the specification.” • SpecTRM-RL has an “Appears In:” attribute to identify orphaned inputs Safeware Engineering Corporation

24. Mechanical shearing arm motion became increasingly erratic By the end of the accident, three of the four arms were damaged by the controller’s commands The shearing arm fine-adjustment sensor doesn’t handle struggling humans well The data bus was flooded with commands and telemetry Why were the arms flailing? Safeware Engineering Corporation

25. “For the largest interval in which both input and output loads are assumed and specified, the absorption rate of the output environment must equal or exceed the input arrival rate.” SpecTRM-RL’s attributes address timing behavior Criterion: Environmental Capacity Safeware Engineering Corporation

26. An operator finally noticed the calamity The operator issued a stop command to the shearing pen The shearing pen didn’t stop The designers didn’t anticipate high communication load The stop command is just another order on the bus The operator had no way to know the order was lost Why couldn’t the operator help? Safeware Engineering Corporation

27. This problem actually touches on a number of criteria Inadequate display of state to operators Inability to preempt lower priority tasks Lack of feedback For the moment, focus on the lack of feedback to the operators SpecTRM attributes on outputs make feedback paths easy to check Criterion: Output Feedback Safeware Engineering Corporation

28. When the operators realized the system wouldn’t shut down, they commanded the gate open It didn’t open Keeping gates closed during shearing is a safety feature The command that closes the gate isn’t reversible. No notice was given to the operator. Why didn’t the entry gate open? Safeware Engineering Corporation

29. Criterion: Reversibility • “Output commands should usually be reversible.” • SpecTRM-RL outputs have an attribute linking to the output that reverses their command Safeware Engineering Corporation

30. There was no operator error in this accident. There were no component failures in this accident. Even the software didn’t “fail.” It met its requirements, such as they were. The Electroshear 2000 was found to be unsafe. The culprit cited was the shearing pen control software. Software problems stemmed from incomplete requirements. Investigation Findings Safeware Engineering Corporation

31. The ElectroShear accident demonstrates several completeness critera Nondeterminism Data Age State Completeness Input Variable Completeness Environmental Capacity Output Feedback Reversibility Consideration of these criteria could have prevented and/or reduced the severity of the accident Completeness Criteria (2) Safeware Engineering Corporation

32. Summary • The example may be fanciful, but the problems illustrated are quite real • The completeness criteria were compiled from decades of research, accident and incident reports, and specification review • SpecTRM-RL builds the criteria into a state of the art, analyzable, and executable requirements language Safeware Engineering Corporation

33. Discussion And/Or Questions Safeware Engineering Corporation

34. The End Safeware Engineering Corporation