1 / 16

On cross-modal sensory streams...

On cross-modal sensory streams. Invited Panel Presentation at SIGGRAPH '98, Orlando, Florida. The v OIC e by Peter B.L. Meijer. Philips Research Laboratories Peter.B.L.Meijer@philips.com. The v OIC e: “Seeing with your Ears”. Basic facts:

neve-finley
Download Presentation

On cross-modal sensory streams...

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. On cross-modal sensory streams... Invited Panel Presentation at SIGGRAPH '98, Orlando, Florida • The vOICe • by • Peter B.L. Meijer Philips Research Laboratories Peter.B.L.Meijer@philips.com

  2. The vOICe: “Seeing with your Ears” • Basic facts: • Blind people cannot see...but usually they can hear • Multimedia computing allowsaudio-visual transformations

  3. Video Audio MentalVideo The vOICe Human brain So translate video to audio!? The vOICe: “Seeing with your Ears” • Basic facts: • Blind people cannot see...but usually they can hear • Multimedia computing allowsaudio-visual transformations

  4. The vOICe implementations Java applet Hardware prototype Windows 95/98/NT http://www.seeingwithsound.com/

  5. The vOICe image-to-sound mapping Frequency Time Sum of sinusoids Amplitude

  6. The vOICe image-to-sound mapping Frequency Spectrographic synthesis Time

  7. The vOICe image-to-sound mapping Could this really work? Check: map sounds back to images Theory Yes!(atleast technically)

  8. The vOICe image reconstruction The vOICe “Soundscape” Spectrogram

  9. The vOICe image reconstruction “Soundscape”

  10. The vOICe and human speech Pronouncing one image per second

  11. The vOICe mental imagery? Representation may dramatically affect recognition!

  12. The vOICe limitations • Issues: • Frequency-Time uncertainty • Physiology / Perception: • just-noticeable difference (JND) • critical bands (human cochlea)

  13. The vOICe limitations • Issues: • Frequency-Time uncertainty • Physiology / Perception: • just-noticeable difference (JND) • critical bands (human cochlea) • auditory streaming/segregation • neural pathways & bandwidth • neural processing & plasticity

  14. The vOICe limitations • Psychology / Education: • sounds (too) unpleasant • minimum required results • acceptable training effort • motivation for learning • training programs • Issues: • Frequency-Time uncertainty • Physiology / Perception: • just-noticeable difference (JND) • critical bands (human cochlea) • auditory streaming/segregation • neural pathways & bandwidth • neural processing & plasticity

  15. Conclusions • The vOICe approach • needs more research • is cheap technology • is non-invasive

  16. Conclusions • The vOICe approach • needs more research • is cheap technology • is non-invasive • is available Now! http://www.seeingwithsound.com/

More Related