MMI

1. MMI Technion - Israel Summer 2002 Man Machine Interface Ohad Stein Sefi Epel

2. Technion - Israel Summer 2002 Introduction • It is known for years that stress, press, fear, happiness and many other feelings, can be measured by several techniques. • Based on this knowledge, we would like to build an interface, that will enable people to communicate without using any muscle, and without making any physical effort. • Our interface, will enable communication by being a “mind-reader”. Detection of voltage deflections measured from subjects arms, will enable us to estimate the subject’s will.

3. How can we do it? Technion - Israel Summer 2002 • There are several ways to “speak” with the machine: • One way, is to measure the resistance of the skin, to the passage of a very small electric current. • The main disadvantage of this technique is that it is hard to calibrate the system. The measured values changes between different subjects, and between different times with the same subject. • Our way was to measure the change in DC potential across neurones of the autonomic nervous system connected to the sensori-motor strip of the cortex. • In this way – we will detect a signal only after causing it.

4. Part 1: Experimental background Technion - Israel Summer 2002 • In order to gain some basic knowledge about the MMI abilities we have done some basic experiments, in several different ways. • On the following slides, 3 basic experiments are shown. • The first and second experiments have a strong emotional effect. We did them in order to have a basic indication about the ability of building a MMI. • The third experiment do not have any emotional aspect. The subject should try to control the voltage deflections measured from his arm. • The third experiment is actually the prototype for our project.

5. Pictures experiments • Exp #1: “Pictures Experiment” – several pictures where shown to the subject, changes on the skin voltage ratio where measured according to the pictures emotional effect. • Be aware that the fall of the voltage ratio is a short time after the “strange/unusual” picture is shown.

6. 5 10 15 20 25 30 Seconds Technion - Israel Summer 2002

7. 5 10 15 20 25 30 Seconds Technion - Israel Summer 2002

8. Technion - Israel Summer 2002 Voice experiment • Exp #2:“Voice Experiment”– several words were announced to the subject, different voltage ratio was measured according to the words emotional effect. • The words that did not affect the subject were words like “silence”, ”beautiful”, ”sunrise”, and the words that did affect the subject were “war”, “accident” and etc. • Note that in this experiment the result can be affected by screaming, changing tone, speaking quietly and etc.

9. GOOD GOOD GOOD GOOD 5 10 15 20 25 30 Seconds Technion - Israel Summer 2002 The person heard 5 different good/bad words. Here are the results: BAD!!!

10. 5 10 15 20 25 30 Seconds Technion - Israel Summer 2002 The person heard 5 different good/bad words. Here are the results: BAD!!! GOOD GOOD GOOD GOOD

11. Technion - Israel Summer 2002 Letters experiment • Exp #3:“Letters Experiment”– several letters were shown to the subject, one by one . Different voltage ratio was measured according to the picture emotional effect. • Exp #3 is a much more complicated experiment. The emotional effect is dependent on the assistant decision and concentration.

12. Technion - Israel Summer 2002 • Based on Exp# 3 – we will try to prove the ability of the man to write down a word without using a pen, pencil, keyboard or anything else. Just by measuring the voltage ratio on his arm ! ! ! • The red letter shown in the following graphs, is the letter that the subject was waiting to be shown.

13. M M Q Q K K F F T T C C 6 6 11 11 17 17 22 22 27 27 33 33 Second Second Technion - Israel Summer 2002 0

14. B B O O L L S S W W U U 0 0 6 6 11 11 16 16 22 22 27 27 33 33 Second Second Technion - Israel Summer 2002

15. The experiments success • As expected, not all the experiments succeeded. • There were some reasons that caused it: • The training session was not long enough. 2. The session training was not led by Bio-feedback experts. 3. Not always the system was calibrated. 4. The low-frequency signal is not reduced by the system. • We had about 70% successful experiments.An experiment is characterized “successful”, if a simple algorithm can detect which simbol the subject chose.

16. T Q K M C F 0 5 11 16 22 28 33 Second Technion - Israel Summer 2002 In this slide, you can see the importance of reducing the low-frequency signal.

17. B O L S W U 0 6 11 16 22 27 33 Second Technion - Israel Summer 2002 Here the calibration system is missing, you see the “initializing” effect.

18. M Q K F T C 0 6 11 17 22 27 33 Second Technion - Israel Summer 2002 In this slide it is hard to detect the effect.

19. 5 10 15 20 25 30 Seconds Technion - Israel Summer 2002 In this slide there isn’t any effect.

20. Technion - Israel Summer 2002 Part 1 Summary • Basic experiments indicate that there is a linkage between the emotions and the voltage deflections measured from our skin. • Emotions can be evoked by pictures and sounds. • Emotions can be controlled by the subject decision.

21. Technion - Israel Summer 2002 Part 2 – Bio-Feedback • The meaning of“Bio-Feedback” is that man can “learn” his reflexes, can control them, and can use them as a way of therapy. • Every hospital uses, nowadays, biofeedback experts in the pain clinics, in order to help the people help themselves. • The technique is based on a long training session, led by experts, in order to learn the “patient emotionalism”, and the special biofeedback way that can be useful for him.

22. Technion - Israel Summer 2002 • There could be some differences between patients: • In the recording site. 2. In the detection way. 3. In the psychological ways of the training. 4. In the time of training session which is needed.

23. Technion - Israel Summer 2002 • In order to improve our biofeedback abilities, and after consulting with some experts from the academy and from the medicine area, we have built 3 different train tools for the training. • A balloon. The balloon radius is affected by the measured voltage ratio, of the subject skin. The subject should try to increase the balloon radius.

24. The balloon radius changes , due to voltage ratio change

25. The balloon radius changes , due to voltage ratio change

26. The balloon radius changes , due to voltage ratio change

27. Technion - Israel Summer 2002 2,3.A beeper and a colored column. The column height and the beeper tone are the feedback in this tool.

32. Technion - Israel Summer 2002 Part 3 – Visual Basic Real-Time implementation • In this part of the project we have build an real-time user-interface, for our implementation. • Stage #1 – a matrix is shown to the subject. 5 lines of 5 letters each. • Each line become bold for 5 seconds. • The subject choose one of the 25 letters, therefore he will “like” one line more than the others (the one that includes the chosen letter…) • The following slides will show the interface and the effect of the “chosen line” to our mind…

33. The user interface Technion - Israel Summer 2002

34. Line 5 Line 1 Line 4 Line 3 Line 2 3 6 9 12 15 18 Seconds Technion - Israel Summer 2002

35. Line 5 Line 1 Line 4 Line 3 Line 2 3 6 9 12 15 18 Seconds Technion - Israel Summer 2002

36. Line 5 Line 1 Line 4 Line 3 Line 2 3 6 9 12 15 18 Seconds Technion - Israel Summer 2002

37. How does it work? Technion - Israel Summer 2002 • We can see, that the voltage ratio measured from the subject skin, is affected from the bold line. • There are some stages on the interface process: • Calibration. On the first 15 seconds the system calibrates.

38. Technion - Israel Summer 2002 2.After calibrating, the system starts with bolding each line in a random order for 5 seconds. The screen appearance is shown in the following slide.

39. Technion - Israel Summer 2002 We can see here the bolded line. Line #3

40. Technion - Israel Summer 2002 • 3. After choosing a line, by the changes of the voltage ratio on the subject arm, we stop bolding lines and start bolding letters on the chosen line. Now, the subject will be able to choose the wanted letter and print it in the right place. The screen will look as followed.

41. Technion - Israel Summer 2002 Bolding letters one by one, in random order.

42. Technion - Israel Summer 2002 The chosen letters are written here.

43. Technion - Israel Summer 2002 Summary • The main target was to be able to write a whole word in the real-time implementation. We have not completely reached this target. • In accuracy of 70% we are able to write one chosen letter. • The real-time implementation functionality is not completed. • We can point at some areas for improvement: 1. A much longer training session. 2. Try to use other Bio-feedback tools. 3. The algorithm should solve some problems that our algorithm did not solve, like reducing the low-frequency signal.