Virtual Cinematography Theory and Practice for Automatic Real-Time Camera Control and Directing

1. Virtual CinematographyTheory and Practice for Automatic Real-Time Camera Control and Directing Liwei He Microsoft Research http://research.microsoft.com/users/lhe

2. Motivation • There are 3 elements in computer graphics • lights, scene objects, and • the camera (focus of this talk) • Camera control is hard (7 DOFs) • position (3), direction (3), field of view (1) • We may learn from cinematography

3. Roles in cinematography • Low-level responsibilities • Cameraman • Position and move the camera • Film editor • Keep the film to proper length • Decide shot transitions • High-level responsibilities • Script writer and director • Story-telling

4. Camera control level 1 • Input: camera position and direction • Output: camera transformation • Application: low-level graphics library routines (Direct3D)

5. Camera control level 2 • Input: show both A and B, follow C, etc. • Output: camera position and direction • Application: 3D game, computer animation package

6. Camera control level 3 • Input: high-level user directions • show a conversation • show a car chase scene • Output: a sequence of level 2 camera spec • Applications: graphical chat, tele-conferencing, Virtual Reality games

7. Camera control level 3 (cont.) • Encodes cinematographic expertise • an interesting problem in itself • Provides an interface that is • real-time • visually entertaining • more informative

9. Microsoft V-Chat Lunar Island BARG07011 DGerste309 Eyepeep Feldegast GrenDayGod Eyepeep says "really" Feldegast says "yup" BARG07011 has joined the conversation Eyepeep says "im still older" Feldegast says "lol" DGeste309 does something silly. Eyepeep BARG07011 Feldegast

10. Principles of Cinematography

11. Film structure Film Scenes ... ... Shots

12. Camera distance Close up Close shot Medium shot Full shot Long shot

13. The line of interest A B

14. External camera A B

15. External camera A B

16. Internal camera A B

17. Apex camera A B

18. Moving cameras A B pan(A) track(B)

19. Some rules in film editing • Don't cross the line of interest • Avoid jump cuts • Let the actor lead • Break movement

20. Don’t cross the line of interest 1 2

25. Some rules in film editing • Don't cross the line of interest • Avoid jump cuts • Let the actor lead • Break movement

26. Avoid jump cut

27. Some rules in film editing • Don't cross the line of interest • Avoid jump cuts • Let the actor lead • Break movement

28. Two-person conversation A B 3 4 1 2

29. 1 2 A B A B 4 3 B A

30. Three-person conversation 4 B C 1 A 2 3

31. 2 1 C C A B 3 4 A B A B

34. Bad

35. Good

36. Three spaces The universe space

37. Three spaces The eye space (z-axis is the look at direction)

38. Three spaces The screen space (screen is ctan(fov/2) away from eye point)

39. Standard look-at transformation Given eye position Pfrom, a look at direction Want a rotation matrix R and a translation to transform a vector from universe space to eye space

40. Standard look-at transformation But this is not unique

41. Standard look-at transformation But this is not unique

42. Standard look-at transformation But this is not unique

43. Standard look-at transformation We will need to specify an up vector , usually [0,1,0]

44. Standard look-at transformation In eye space Z-axis is X-axis is Y-axis is Rotation matrix Translation is -Pfrom

45. Internal close-up of actor A

46. Look at [xat,yat] transformation Given eye position Pfrom, look at point Pat, and Pat in screen space [xat, yat] Want In universe space

47. Look at [xat,yat] transformation In screen space: H = [xat, yat, ctan(fov/2)] Solve from the following equations:

48. External of actor A and actor B

49. Look at [xA,yA] and [xB,yB] Given PA,PB in universe space and eye space, and distance of eye position Pfrom and Pa. Want Pfrom and

50. Look at [xA,yA] and [xB,yB] • Use numerical method: • Set Pfrom to 0 • Solve R using the Look at [xat,yat] method • Transform E by inverse of R, getting a new Pfrom approximation • Goto 2