Dr Paul Vickers School of Informatics, Northumbria University auralisation

1. Compilers with EARsUsing External Auditory Representations for Program Comprehension and Debugging Dr Paul VickersSchool of Informatics, Northumbria University www.auralisation.org Paul Vickers

2. Or debugging with music… • Vickers, P. & Alty, J. L. (2005). The Well-tempered Compiler: The Aesthetics of Program Auralization. Aesthetic Computing. P. Fishwick MIT Press. • Vickers, P. & Alty, J. L. (2003). “Siren Songs and Swan Songs: Debugging with Music.” Communications of the ACM 46(7): 86-92. • Vickers, P. & Alty, J. L. (2002). “When Bugs Sing” Interacting with Computers 14(6): 793-819. Paul Vickers

3. Whoah! Say what? Musical approach helps programmers catch bugs • New Scientist (2359, 7/9/02) • http://www.newscientist.com/news/news.jsp?id=ns99992757 Programming tool makes bugs sing • Technology Research News (7/8/02) • http://www.trnmag.com/Page_One/2002/Page_One_080702.html Paul Vickers

4. In the news • 13 April 2004 Interview on CBS radio! See acfnews summary and transcript of the Osgood File in which it appeared. Full mp3 of interview available at www.auralisation.org. • British Council story • Article in Science News Ross, D. Music programmes to find bugs IEE Review, 2003, 19. • Henderson, T. Sound of music to bring out the bugs The Journal, Newcastle upon Tyne, 2002, 22. • Survey fever BBC Music Magazine, 2002, 4. • Zimmermann, R. Schräge Töne im Programm: Die Erfindung eines Doktoranden macht Software-Fehler hörbar SonntagsZeitung, Zurich, 2002, 127. Paul Vickers

5. Rewind • Software visualisation uses external visual representations (EVR) • Coordination of multiple EVRs– e.g. CRUSADE project at Sussex • Program comprehension & debugging • The use of external auditory representations (EAR) is called software auralisation • Often an auditory icon approach • Vickers & Alty applied a structured musical framework Paul Vickers

6. Program auralisation • Mapping of program data and events to sound • LogoMedia • Sonnet • ADSL (auditory domain specification language) • LISTEN • Jackson & Francioni (1992) parallel programs • Brown & Herschberger • Largely untested (empirically) Paul Vickers

7. Why music? • Gives multiple audio streams in parallel • Structured and organised according to specific rules (“organised sound” -- Varese) • Time-ordered. Visual techniques are spatial. • Spatial sound placement enhances the music • Gives sense of continuity/continuous flow • How can music’s grammars be used to carry real information? Vickers, P. and Alty, J.L., Using Music to Communicate Computing Information.Interacting with Computers, 2002. 14(5): pp. 434-455. Paul Vickers

8. Musical framework • Parallel & structured • Grammar allows separate musical components to work with rather than against each other • Time-ordered • Visual techniques primarily spatial • Programs operate in temporal domain • Audio well suited to monitoring state changes over time Paul Vickers

9. cAitlIn system • Computer Audio Interface to Locate Incorrect Nonsense • Uses musical motifs to render Pascal language constructs at run time • Motif design is hierarchic as is organisation of constructs Paul Vickers

10. Hierarchic motif design Vickers, P. and Alty, J.L., Musical Program Auralisation: A Structured Approach to Motif Design.Interacting with Computers, 2002. 14(5): pp. 456-484. Paul Vickers

11. Design of motifs • Points of interest • Construct Entry • Evaluation of condition • Execution of body • Exit from construct • Boolean True=major, False=minor • Drones Paul Vickers

12. Selection tunes • Metaphorically asking & answering a question. • Tune rises on entry and descends on exit Paul Vickers

13. Iteration tunes • Major/Minor chord to show when condition evaluated • Heard at start of WHILE loops and at end of REPEAT loops • FOR loops increase/decrease the pitch each time round the loop Paul Vickers

14. Variations • All selections use same basic tune, modified for individual constructs • All iterations use another basic tune, again modified Paul Vickers

15. Auditory parentheses • As () bracket a thought in a sentence, so the auditory parentheses serve to open and close constructs Paul Vickers

16. Other statements • All other statements are currently represented by a drum beat, one per statement. Paul Vickers

17. Advantages • Over Auditory Icons • symbolic mappings = poor system aural ecology • Syntactic design  expect retention • Displays both flow and state • An execution-specific program structure diagram Paul Vickers

18. Demonstration Paul Vickers

19. IF (true) a := 10 ; IF a = 10 THEN Writeln (‘A is 10’) ; Paul Vickers

20. IF…ELSE (false) a := 1 ; IF a = 10 THEN Writeln (‘A is 10’) ELSE Writeln (‘A is not 10’) ; Paul Vickers

21. CASE…ELSE match a := 3 ; CASE a OF 1 : Writeln (‘A is 1’) ; 2 : Writeln (‘A is 2’) ; 3 : Writeln (‘A is 3’) ; ELSE Writeln (‘No match’) ; END ; Paul Vickers

22. WHILE 2 iters cntr := 1; WHILE cntr <= 2 DO cntr := cntr + 1 ; Paul Vickers

23. REPEAT 3 iterations cntr := 1 ; REPEAT cntr := cntr + 1 ; UNTIL cntr > 3 ; Paul Vickers

24. FOR…TO 6 iters FOR cntr := 1 TO 6 DO Writeln (cntr) ; Paul Vickers

25. Sequence & nesting • Clue to nesting is in the drone • If you hear a continuous note then you are inside a construct. • Each construct has a drone. • For nested constructs you hear multiple drones simultaneously • Drone pitch increases as nesting deepens Paul Vickers

26. Example program Program Exemplar VAR counter : Integer ; BEGIN counter := 1 ; WHILE counter <= 2 DO BEGIN IF counter MOD 2 = 0 THEN Writeln (‘Counter is even’) ; counter := counter + 1 ; END ; END. Paul Vickers

27. Score Paul Vickers

28. Score Paul Vickers

29. Structure Diagram Paul Vickers

30. Experiment 1 • Listening test to assess comprehensibility of motifs • 22 subjects, 60 auralisations • Recognition rate better than by chance and with very little training • Hierarchic organisation allowed class level identification & abstraction Vickers, P. and Alty, J.L., Musical Program Auralisation: A Structured Approach to Motif Design.Interacting with Computers, 2002. 14(5): pp. 456-484. Paul Vickers

31. Experiment 2 – Bug location • Questions • Do subjects locate more bugs with the additional auralisation information than without? • Does the musical experience of subjects affect their ability to make use of program auralisations? • 22 subjects, 8 programs, 4 auralised, 4 normal • Given: spec, outline design, input, output, & expected output • Then, ten minutes to locate bug with either • Above documentation, or • Above documentation + auralisation Vickers, P. and Alty, J.L., When Bugs Sing.Interacting with Computers, 2002, 14(6): 793-819. Paul Vickers

32. Experiment 2 – Bug location • Experimental design (within subjects, repeated measures) • Group A had 1, 3, 5, & 7 without auralisation and 2, 4, 6, & 8 with auralisation • Group B had 1, 3, 5, & 7 with auralisation and 2, 4, 6, & 8 without Paul Vickers

33. Experiment 2 – Bug location • Observe a significant difference in debugging scores: t(21)=2.10, p<0.05 • Bug location time not affected:t(21)=1.98, p=0.06 • Auralisations led to increase in subjects perceived workload (measured with NASA TLX): t(21)=2.12, p=0.046 • Musical knowledge/experience had no effect: regression model 0.16 <= p <= 1.71 Paul Vickers

34. Experiment 2 – Bug location • Best candidates were programs • where output contained no clues to bug location • with a high complexity, complex boolean expressions Paul Vickers

35. Limitations • No comparison with visual task • Cognitive latency of motifs? • No longitudinal results • Restricted to program flow Paul Vickers

36. Enter Gamelan • EPSRC bid to investigate Auditory External Representations for an OOP language (prob. Java, though C# has an obvious appeal…) • (c.f. EPSRC CRUSADE project at COGS in Sussex - du Boulay et al) • Comparing EAR with EVR in a variety of program comprehension & debugging scenarios • Software stethoscope? Paul Vickers

37. Interesting issues • EAREVR: how to compare? • Scalability • Parallelism (multi-threading) • Knowing when an identifier goes out of scope • Music flow may help here • Method calling/called-by paths • Control flow & Data flow • Data structures Paul Vickers

38. We can envisage • Attaching auralisation tags to data structures with predefined/preassigned mappings for different types of data: • an integer would sound one way, an array another allowing data and data structures to be inspected at will, their auralizations not being indirect via program flow (as was the case with Alty’s bubble-sort auralization where the auralization of the list was done via a mapping of the algorithm flow/state). • Could listen to program flow and at any time ask for a variable/data structure update auralisation. This might require suspension of execution during the update so as not to miss any program events. Paul Vickers

39. Passive & active auralisation • Two basic types of EAR approaches could be developed for data representation: • program flow/real-time in which data are auralised as they are modified during execution (passive auralisation) and • asynchronous, where auralisations can be requested at any time during execution by button presses/mouse clicks (active auralisation). There are many other possibilities that can be explored Paul Vickers

40. Coordination • Of particular interest will be to explore how multiple EARs are coordinated by users • How EARs can be used in combination with EVRs to provide enhanced multi-modal support of program comprehension and debugging Paul Vickers

41. The power of multi-modality? Visualisation Auralisation Paul Vickers

42. Sonification aesthetics • Chaotic attractor functions • Mayer-Kress, G., Bargar, R. & Choi, I. (1994). Musical Structures in Data from Chaotic Attractors. Auditory Display. G. Kramer. Reading, MA: Addison-Wesley. XVIII: 341-368. • Earthquake • Marty Quinn’s Seismic Sonata • Pacific ocean buoys • Bob Sturm’s Music from the Ocean Paul Vickers

43. Fusion with ‘real music’ • e.g. Ian Boddy: Continuum • Intriguing overlap between sonification/auralisation and generative music & electro-acoustic music Paul Vickers

44. Indicative publications • Vickers, P. & Alty, J. L. (2005). The Well-tempered Compiler: The Aesthetics of Program Auralization. Aesthetic Computing. P. Fishwick MIT Press. • Vickers, P. (2004). External Auditory Representations of Programs: Past, Present, and Future - An Aesthetic Perspective, in Proc ICAD 2004 - The Tenth Meeting of the International Conference on Auditory Display, Sydney, 6-9 July. • Vickers, P. & Alty, J. L. (2003). “Siren Songs and Swan Songs: Debugging with Music.” Communications of the ACM 46(7): 86-92. • Vickers, P. & Alty, J. L. (2002). “When Bugs Sing.” Interacting with Computers 14(6): 793-819. • Vickers, P. & Alty, J. L. (2002). “Musical Program Auralisation: A Structured Approach to Motif Design.” Interacting with Computers 14(5): 457-485. • Vickers, P. & Alty, J. L. (2002). “Using Music to Communicate Computing Information.” Interacting with Computers 14(5): 435-456. Paul Vickers

45. www.auralisation.org paul.vickers@northumbria.ac.uk Paul Vickers

46. cAitlIn musical grammar • Uses motif classes and variations to render Pascal constructs • Allows • abstraction • nesting (tonality & harmony preserved across motifs) • Uses grammatical rules (e.g. a tonal music system) Paul Vickers

47. cAitlIn musical grammar cont • Renders information not directly communicated by source code or other visualisation techniques • e.g. construct persistence (nesting) •  a new grammar, conveys • extra meaning • same meaning but from different perspective Paul Vickers

48. Theoretical framework • Musical grammars as communication medium • Mapping of syntactic features of Pascal to music • Construct hierarchy  motif hierarchy • Motifs organised around tonal musical framework • Use of surface & deep characteristics • Meter & rhythm more useful than harmony • Percussion adds emphasis • Drones maintain context Paul Vickers

49. Real grammar? • Translation or transliteration? • Have we simply rendered written Pascal as musical Pascal (e.g. written vs spoken English) • Like writing “I want to go to bed” as “Ich will gehen zum Bett” (transliteration) • Is there a new grammar? • “Ich will zum Bett gehen” (translation) Paul Vickers

50. More auralisation? • Is further auralisation needed? • Perturbations in flow pinpoint general region of bug. • Non-flow bugs very hard to detect anyway & require variable inspection • Inspection difficult to render in audio Paul Vickers