Specification and Reasoning in SE Projects Using a Web IDE

1. Specification and Reasoning in SE Projects Using a Web IDE Charles T. Cook (Clemson) Svetlana V. Drachova-Strang (Limestone College) Yu-Shan Sun (Clemson) Murali Sitaraman (Clemson) Jeffrey C. Carver (Alabama) Joseph E. Hollingsworth (IU Southeast) This research is funded in part by NSF grants CCF-0811748, CCF-1161916, DUE-1022191, and DUE-1022941.

2. Part I: Overview

3. About Clemson • School of Computing has about 600 undergrads and 200 grads • Clemson University has 17,000 students • Located in a town (also Clemson) with an official population of about 13,000 in South Carolina • Has a football stadium with capacity 85,000

4. This talk • This talk is about undergrad courses, though the ideas have been used in both undergrad and grad courses for many years

5. Goals of the SE Projects • Role of formal specifications as contracts in team software development and integration • Role of formal specifications in reasoning about software correctness • Other uses • Specification-based test case design

6. Intro. and Advanced Projects • Introductory projects involve only use of and reasoning with existing library components • 2 weeks of lectures • Advanced projects involve development and reasoning about new components • 3-5 weeks of lectures • Other variations

7. Sample Courses and Institutions • Sample introductory projects • Alabama (Software Engineering) • Cleveland State (Software Engineering) • Sample advanced projects • Clemson (Software Engineering) • Denison (Independent study projects) • Other variations • Depauw (Theory), NC State (Data structures), Southern Wesleyan (Data Structures), Ramapo College (Programming Languages),…

8. Clemson University Details • Two-course sequence • Sophomore-Level, CP SC 215: Software Development Foundations • Junior/Senior-Level: CP SC 372: Software Engineering • Experimentation and assessment • Piloting (2007/2008) • Institutionalization (2009 to present)

9. Soft. Dev. Foundations Course • Intro to Java, object-based computing, software engineering, design patterns, etc. • 10 weeks (includes usual materials and projects) • Intro to formal specifications and reasoning (interspersed with above topics) • 4 weeks • Uses RESOLVE-style specifications in a Java context

10. Software Engineering Course • Software life cycle, process models, requirements analysis and design • 8 weeks (includes usual materials and projects) • Transition from informal to formal • 1 week • Specification-based component development and quality assurance • 5 weeks • Uses RESOLVE

11. Clemson University Key Points • Specification/reasoning included in syllabi for two required courses for majors (2009) • Graduating student learning outcome modified to include “development of bug-free software according to specifications” (2012)

12. Part II: Example Projects

13. RESOLVE • An integrated specification and programming language for verified, component-based software development • A Verifying compiler • A github project • A freely-available web interface to use the verifying compiler: www.cs.clemson.edu/group/resolve

14. Sample Intro Assignments • Generate VCs and prove the given Queue Remove_Last operation. • Make each of the following changes and explain what is unprovable. • Comment out the first Dequeue operation. • Change the maintaining clause (loop invariant) to #Q = <E> o Q. • Change the decreasing clause (termination progress metric) to |T|.

15. Sample Advanced Assignments • Implement and verify: • Extension operations on Queues to Insert_After and Remove_After • Sequence_Template using Queue_Template with extensions • Multiple implementations of Queue_Template satisfying given internal contracts • Assignments to put it all together • Involve a dozen components and teams of 3 students

16. Sample Components • Stack_Template • Queue_Template • Preemptable_Queue_Template • Sequence_Template • List_Template • Search_Store_Template • Map_Template • Prioritizer_Template • …

17. Part III: A Web IDE Demo

18. Getting Started • www.cs.clemson.edu/group/resolve • Tab: Web IDE • Google: • RESOLVE web IDE • RESOLVE verifier • Clemson RESOLVE

19. Sample Intro Assignments • Generate VCs and prove the given Queue Remove_Last operation. • Make each of the following changes and explain what is unprovable. • Comment out the first Dequeue operation. • Change the maintaining clause (loop invariant) to #Q = <E> o Q. • Change the decreasing clause (termination progress metric) to |T|.

20. Elements of the Assignment • Queue_Template concept specification • A specification of Remove_Last Operation • An annotated implementation of Remove_Last operation

21. Mathematical Modeling • Concepts provide mathematical models for programming objects • To write formal specifications, we need to model the state mathematically • Some objects we use in programming, such as Integers and Reals, have implicit models • For others, such as stacks, queues, lists, etc., we need to conceive explicit mathematical models

22. Mathematical Modeling of Queues Concept Queue_Template(type Entry; Max_Length: Integer); uses String_Theory; Type Family Queue is modeled by … Operation Enqueue… Operation Dequeue… … end Queue_Template;

23. Concept Queue_Template(type Entry; Max_Length: Integer); uses String_Theory; Type Family Queue is modeled by Str(Entry); exemplar Q; constraints |Q| <= Max_Length; initialization ensures Q = ; … end Queue_Template; Mathematical Modeling of Queues

24. Operation Remove_Last (updates Q: Queue; replaces E: Entry); requires |Q| /= 0; ensures #Q = Q o <E>; A Specification of Remove_Last

25. Procedure Remove_Last (updates Q: Queue; replaces E: Entry); Var T: Queue; Dequeue (E, Q); While (Length(Q) /= 0) do Enqueue(E,T); Dequeue(E,Q); end; Q :=: T; end Remove_Last; An Implementation of Remove_Last

26. Assume #Q = <10, 20, 30, 40> Dequeue (E, Q); While (Length(Q) /= 0) T E Q Iter. 1  10 <20, 30, 40> Iter. 2 <10> 20 <30, 40> Iter. 3 <10, 20> 30 <40> Iter. 4 <10, 20, 30> 40  do Enqueue(E,T); Dequeue(E,Q); end; Understanding Loop Invariant

27. Assume #Q = <10, 20, 30, 40> Dequeue (E, Q); While (Length(Q) /= 0) T E Q Iter. 1  10 <20, 30, 40> Iter. 2 <10> 20 <30, 40> Iter. 3 <10, 20> 30 <40> Iter. 4 <10, 20, 30> 40  Loop maintains this invariant: #Q = T o <E> o Q Understanding Loop Invariant

28. Procedure Remove_Last (updates Q: Queue; replaces E: Entry); Var T: Queue; Dequeue (E, Q); While (Length(Q) /= 0) maintaining #Q = T o <E> o Q; decreasing |Q|; do Enqueue(E,T); Dequeue(E,Q); end; … An Implementation of Remove_Last

29. Verification • Press Verify Button • Generate and prove automatically a series of verification conditions (VCs) • Students understand why the VCs arise • Understand connections between contracts, code, and proofs • Can prove VCs

30. Impact of Incorrect Code • The first statement, call to Dequeue, is removed. • Leads to one unprovable VC (Verification Condition) • VC 0_1: • Base Case of the Invariant of While Statement in Procedure Remove_Last: Remove_Last_Realiz.rb(8)  • Goal: Q = ((empty_string o <E>) o Q) • Givens: …

31. Part IV: Assessment

32. Assessment Summary • For details • See the paper • See Drachova Ph. D. dissertation (2013, Clemson), available at website • A quick summary • Likert items to assess perceived benefits of web IDE (avg. 4.0/5.0) • Project grades (avg. from 80%); about the same as non-formal parts • RCI-item based analysis

33. What reasoning skills are necessary?Reasoning Concept Inventory http://www.cs.clemson.edu/resolve/teaching/inventory.html

34. Clemson RCI-Based Evaluation • 3.4.3: Precise specifications • 4.2: Design by contract • 4.3.1: Internal contracts

35. SE Project Benefits on RCI-Based Learning Outcomes • RCI Topics: 3.4.2, 3.4.3: Precise specifications • Sample size: 24 students, Spring ‘12

36. Summary • Students can practice formal specification and reasoning principles effectively using the RESOLVE web IDE and its verifier • There are some benefits for even a minimal intro (3 lectures plus project) • Significant benefits with longer-term exposure • IDE and materials online • www.cs.clemson.edu/group/resolve