VUT Brno meeting data collection for M4

1. VUT Brno meeting data collection for M4 Petr Jenderka, Stanislav Sumec, Igor Potůček, Petr Motlíček, Jan Černocký, Pavel Zemčík Brno University of Technology, Czech Republic M4 meeting Delft

2. Goals • Provide M4 and follow-ups with more data, complementary to ICSI and IDIAP • Different speakers, different accent (heavily accented CZEnglish) • Setup of a secondary M4 meeting room: low-cost, mobile, built from off-the-shelve products, easy to install and operate. • Plans for meeting recordings • Plans for annotations. M4 meeting Delft

3. The rooms • IDIAP room very valuable but expensive, difficult to assemble/setup and impossible to bring outside IDIAP Need for cheaper, lighter, and “buyable” room • Our meeting room: • recordings of up to 4 persons • 4 lapel mikes, no mike arrays • one camera with hyperbolic mirror allowing for almost 360 degree view. • audio/video recording on one notebook. M4 meeting Delft

4. Setup M4 meeting Delft

5. 4 people around a mirror M4 meeting Delft

6. M4 meeting Delft

7. Original wrapped image M4 meeting Delft

8. Hardware • 4 lapel microphones Sennheiser MKE 2 P-C (same as in IDIAP) • 2 mixers Behringer EURORACK UB1204-PRO providing gain control and phantom power supply (actually 90% of mixers are not used, but they’re cheaper than special mike pre-amps with phantom feeders…) • two Hi-Fi PC-MCIA sound-cards VX-pocket (we know them from the SpeeCon project) M4 meeting Delft

9. Hardware (2) • digital camera SONY CLIP 345 • hyperbolic mirror by NEOVISION (Prague) • FireWire interface • The camera provides 2 additional (global) audio channels. • The cost: 4.2 kEUR w/o the notebook M4 meeting Delft

10. Recording • a home-made (S. Sumec) tool, based on Direct-show libraries, under Windows. • post-processing tool (I. Potúček) for unwrapping video to panoramic format of view. • All audio and video streams are stored in real time: two audio wav files with four audio channels, one DV-compressed video avi file with video and stereo audio stream. • Plan for M4: >30 hours of meetings M4 meeting Delft

11. Unwrapped image M4 meeting Delft

12. Annotations – major meeting events • annotations using the „annotation editor“ by Staňa Sumec (audio, video, hotkeys, XML). • This annotation will be done internally by pre- and post-grads at UPGM. • Plan: >20 hours M4 meeting Delft

13. Annotations - speech • ETCA Transcriber with ICSI MultiChannel extension. • An annotator hired for speech annotations, just beginning to be operational. • Plan: >10 hours of speech annotations. • Both annotations: quality check protocols? (quite an issue in our DB projects SpeechDat-E and SpeeCon!) M4 meeting Delft

14. Current work and future • Recording, annotations, integration to the Media Server mmm.idiap.ch • Still quite some issues to solve (position of the mirror, lighting in the room – what can be done on the resulting video?) • Extensions – PowerPoint and Acrobat Reader slide/change timing grabber, T0 must be synced with the recording software • … and research with the data (phoneme recognition, mouth area detection, speaker activity detection) – where does the simplified setup degrade performances? M4 meeting Delft

15. Questions, comments ?THE DEMO Please have a look during the break :-) M4 meeting Delft