Music and shape: Exploring cross-modal representation

2. Music and shape:Exploring cross-modal representation Examples, a model and analyses Professor Adam Ockelford University of Roehampton 11th March 2010

3. Music and shape How can the two relate? One set of dynamic relationships can be created through movement: • more or less freely associated with sound (eg, dance, gestures in performance) • creating, causing or controlling sound (eg, playing instruments, conducting)

4. Music and shape Another set of relationships can arise through the visual representation of sound – the focus of this presentation • Seven examples • Attempt to construct common theoretical framework • Indicative analyses • Discussion

5. Children’s ‘picture scores’ 7 year old’s representation of a rhythm (after Bamberger, 1982)

6. Children’s ‘picture scores’ 7 year old’s representation of a rhythm (after Bamberger, 1982)

7. Blind children’s drawings on ‘German Film’ After Welch (1981)

8. Blind children’s drawings on ‘German Film’

9. Conventional Westernmusic notation Any features in common with children’s informal notations?

10. Braille music Key differences with print?

11. Guitar chord tablature What processes are at work here? Similarities and differences with previous examples?

12. Graphic notation Excerpt from Stockhausen, Electronic Study No. 2 (1954)

13. Score produced through synaesthetic response to music Jamie Roberts’s score of Jean Michel Jarre’s Oxygene (Track 4)

14. How does it all work? A common underlying principle or principles? Or fundamentally different processes at work?

15. Perceptual domains • ‘Domains’ = way of modelling the fact that one sensation can have different ‘states’ • Touch: eg, temperature and pressure. • Vision: eg, hue, saturation and brightness.

16. Perceptual domains In music: • Pitch: (chroma, octave), harmony, tonality • Perceived time: onset, duration • Timbre • Loudness • Perceived location of sound source

17. Different categories of domain Can be conceived as being single dimensional, bidimensional or multidimensional and as being linear or cyclical or both, or complex

18. Examples … Pitch Duration

19. ‘Values’ • Each state in which a domain is perceived to exist can be conceived of as a ‘value’. • May be simple or complex (like the domains to which they pertain) • Example:

20. Relationships within domains • Musical structure is not to do with individual values, but the relationships between them within each domain - particularly pitch and perceived time • We can assume that the relationships between values normally pass listeners by as series of qualitative experiences • However, they may be conceptualised (culture specifically?) as differences or ratios, or they may reflect the complexity of the values to which they pertain

21. Examples of intra-domain relationships ‘Primary’ relationships

22. Examples of intra-domain relationships ‘Secondary’ relationships

23. Relationships between domains • Two main types • ‘Regular’ (isomorphic in the domain of ‘shape’) • ‘Irregular’ (through association)

24. Regular inter-domain relationships Guiding principle: to map values systematically between domains, a relationship must involve a function that is not specific to the perceptual domains concerned.

25. Regular inter-domain relationships For example, differences are domain-specific, so single values of difference cannot be mapped systematically between domains.

26. Regular inter-domain relationships Whereas, ratios are abstract, are not bound by the context in which they occur. Hence they permit regular mapping between domains.

27. Regular inter-domain relationships Ratios may occur between differences (as secondary relationships). Hence the regular inter-domain relationships would be tertiary.

28. ‘Irregular’ inter-domain relationships Guiding principle: formed through association … either through repetition or indirect connection

29. Irregular inter-domain relationships Different types: (a) arbitrary (repetition) (b) indirect (indirect connection) (c) synaesthetic (indirect connection)

30. Arbitrary inter-domain relationships Any relationship between a sound and its visual representation (shape) is possible. For example: accent staccato fermata > •

31. Arbitrary inter-domain relationships Example (‘staccato’) – relationship created through repetition (or verbal explanation = proxy repetition):

32. Indirect inter-domain relationships Example (‘note cluster’) – relationship created through action / object as common visual and auditory source Can be reinforced through repetition

33. Synaesthetic inter-domain relationships Example – green corresponds toEbmajor (and vice versa) Again, can be reinforced (for others) through repetition

34. From theory to analysis Children’s ‘picture score’ Simplest interpretation: arbitrary relationships – given shape (circle) maps onto note

35. Children’s ‘picture score’ More advanced interpretation: regular mapping of distance and duration; sequence and temporal order

36. Score on German Film of pitch glide Coordinated regular mapping of vertical and horizontal distances with pitch and perceived time (implied tertiary relationships)

37. Print score Arbitrary and semi-regular mappings combine (the latter reinforcing the former)

38. Braille score Only arbitrary mappings survive transcription into Braille (which makes using this medium more challenging)

39. Guitar tablature Combines both indirect and regular representations of pitch

40. Stockhausen score: comprises regular representations of pitch, perceived time and loudness

41. Jamie’s score: comprises regular and indirect representations of pitch, perceived time and timbre

42. Conclusions • Evidence from children’s untutored representations of music suggests that sophisticated inter-domain mapping between sound and shape occurs early and intuitively • This view is reinforced by blind children’s representations of changing pitch – which are also sophisticated, and exist in the absence of any visual model to guide them

43. Conclusions • The precise nature of inter-domain mapping may vary from individual to individual, and according to cultural convention. • However, it appears to be relatively easy to grasp regularmappings (consider, for example, widespread use of horizontal dimension for time and vertical dimension for pitch) or arbitrarysymbols (as in standard notations)

44. Conclusions • Some inter-domain mappings may be hard-wired through synaesthesia. • These seem to vary from individual to individual. • However, such representations can be learnt and appreciated by others

45. Conclusions • What next? • More exhaustive search of different forms of symbolic visual representation of sound to check broad applicability of the model • Use the model to generate new forms of sound / shape installations • More research into how mappings aries and are learnt, and into sonic/visual synaesthesia

46. Adam Ockelford a.ockelford@roehampton.ac.uk 07818-456 472