Issues

1. Experiences Gained from Experimentation in Software Coursesat the University of MarylandVic BasiliUniversity of Maryland

2. Issues • Courses available for experimentation • Benefits and Issues with classroom experiments • Training in experimentation

3. Software Engineering Courses • There are two major software engineering classes • CMSC 435 - A software engineering project course taught by various people in various ways (graduate/undergraduate credit available) • CMSC 735 – A software engineering graduate course in modeling` measurement, and process improvement • And one HCI course • CSMC 435 – A course in Computer Human Interfaces that also teaches and uses experimental methods as part of the curriculum. • There are no research methods courses in the CS department, nor is there likely to be in the near future • Many other courses have projects but they are not used to run experiments

4. CMSC 435 • Students work in teams (3 or more) to use the technology to build some kind of software product • Have been running experiments since 1975, beginning with Bob Rieter’s thesis • Early methodology analysis (top down design, chief programmer teams, …) to specific technology experiments (reading vs. testing, requirements reading, OO reading) • Experiment team is usually two to five people, of various levels of experience, interacting and reviewing the design and implementation, and one or two people analyzing the results.

5. CMSC 735 • Course is directly about model building and measurement – and learning and experimentation • Graduate students work on building models rather than software systems • Run experiments here also but mostly in the small • study effects of technique, • play with experimental methods (observation protocols, grounded theory, …)

6. Running Classroom Experiments • Goals for experimenting with the class: • Teaches students in the class • The study of SE follows an experimental paradigm • The concepts behind experimentation • That we do not know the effects of a process and the effects of context variables and are learning • How to learn more about the reffects • Gives graduate students a chance to run an experiment • Gives experimental software engineers • a set of data points for analysis for mostly novice subjects • a chance to debug protocols and improve the process definition • More confidence and credibility when running of a similar experiment with a company

7. Running Classroom Experiments • Problems with classroom experiments • Limiting the context of the study • Selecting process that is relevant to both students and professionals • Dealing with the learning curve • Dealing with student/classroom conflicts • Subject experience • Multiple motivations • Limited time frame, nature of the problem • Design options • Cheating

8. Running Classroom Experiments • Controlled experiments have limits when done in classroom or training situations • You must train everyone (effects the control group issue) • You cannot teach a new technique and not expect people to use it, especially when it is in comparison to a non-technique • You face a variety of threats to validity • There is high risk – spend the time and do not get sufficient information/analysis • This effects the design • Consider the PBR experimental style design (partial factorial) as opposed to the reading vs. testing design • You need to supplement the approach with other evaluation techniques • e.g., a mix of quantitative and qualitative analysis, observations, grounded theory

9. Teaching CMSC735 • Besides models and metrics, cover aspects of the experimental discipline • Experimental Paradigms: • Scientific, engineering, empirical, analytic methods • Experimental Classifications • Level of variable relationship • Experience of Subjects • Experimental Setting • Type of Study • Types of Analysis • Factors Jeopardizing Validity • internal validity, external validity • Building laboratory Manuals • Picking a dissertation topic in SE

10. The Experimental Discipline • Experimental Classifications • Level of variable relationship • Descriptive, Correlational, Cause-effect: • Experience of Subjects • Expert -> Novice • Experimental Setting • in vivo, in vitro • Type of Study • experimental (controlled experiment, quasi-experimental design, case study, field study • Types of Analysis • Quantitative Analysis, Qualitative Analysis

11. Writing an Experimental SE DissertationDefinitions RESEARCH: Diligent search or inquiry; scientific investigation and study to discover facts. SCIENCE: Systematic knowledge of natural or physical phenomena; Facts ascertained by observation, experiment, and introduction; Ordered arrangement of facts known under classes or heads; Theoretical knowledge as distinguished from practical; Knowledge of principle and rules of invention, construction, mechanism, etc., As distinguished from art. THEORY/MODEL: A system for explaining a set of phenomena by specifying constructs and the laws that relate these constructs to each other.

12. Writing an Experimental SE DissertationDefinitions Fact: information obtained through direct observation Hypothesis: educated guess, precedes an experiment Experiment: operation carried out (sometimes under controlled conditions) to discover unknown effect/law, test/establish hypothesis, illustrate a known law Theory: possible explanation based upon many facts/reason Law: description/observation of behavior used for prediction based upon facts and reason Model: simplified representation of a system/phenomenon can be a theory or a law Paradigm: conceptual filter, how we perceive/interpret Truth: what really is

13. RESEARCH APPROACHES ESTABLISHED FIELD: Easier to answer questions Areas better defined More consensus on the importance of an area Standard methods of study METHODOLOGICAL APPROACHES: ANALYSIS: Build a theory Derive properties Show boundary conditions and limits EXPERIMENTATION: Formulate hypotheses Deduce empirical consequences Test the hypotheses by collecting data

14. QUESTIONS for EVALUATING RESEARCH IS THERE NEW KNOWLEDGE? Were the methods used to obtain the knowledge scientifically sound? ARE THE RESULTS SIGNIFICANT? Do they improve our ability to describe, predict, control or explain? PICKING A TOPIC: Build on prior theories Fill in gaps in theories Create new theories that explain better than old Disprove a commonly held “proven” theory CHARACTERISTICS: Can be neatly packaged Focused Consistent methodology

15. QUESTIONS for EVALUATING RESEARCH What would make the following dissertation research? • Building a descriptive model/theory • Building a predictive model/theory • Improving an existing model/theory • Verifying properties of a model/theory • Implementing/automating a model/theory

16. THINKING ABOUT THE RESEARCH PROCESS THEORY AND RESEARCH PERSPECTIVES

17. THINKING ABOUT THE RESEARCH PROCESS PROCEDURAL PERSEPECTIVE

18. SAMPLE EXPERIEMTNAL DISSERTATION CHAPTER 1:Introduction: A. General statement of the problem B. Statement of the hypotheses, objectives, or questions C. Definitions of terms (assumptions/limitations/significance) CHAPTER 2:Review of the Literature: A. Review of previous research B. Pertinent opinion C. Summary of the state-of-the-art (tie it all together) CHAPTER 3: Method: A. Description of the subjects (how chosen) B. Research design and procedures (overview of statistical procedures) C. Description of measures employed

19. SAMPLE EXPERIMENTAL DISSERTATION CHAPTER 4: Findings: A. Description of finding pertinent to each hypothesis, objective, or question B. Other findings CHAPTER 5: Summary and Discussion: A. Summary of research problem, method, and finding B. Conclusions C. Implications D. Suggestions for further research