SPACE SP lit A pplication archite C tur E Enabling predictable SW integration

1. SPACESPlit Application architeCturEEnabling predictable SW integration Bas Engel Royal Philips / Consumer Electronics Division CELF Embedded Linux Conference 2007 April 17, 2007

2. Introduction

3. SW in TV Software size evolution Software partitioning • Total SW size follows Moore’s law • Philips part decreasing (not just relative) • 3rd party involvement increasing

4. Some rationales on 3rd party SW • Cost, however • balance NRE and royalties vs. own development cost • TTM • No time to build up competence in own team • “Off the shelf solution available” • Critical competence available at supplier side Somebody else makes a business out of supplying common features

5. Traditional approach: integrate process Philips 3rd party MIPS (Linux 2.x)

6. Integration nightmare • Integration has it’s advantages • Optimized resource usage • Better viewing experience due to integral architecture approach • Integration pitfall • High effort due to increased complexity • Longer TTM as there is no little change • System validation requires global big bang approach • No independent lifecycle, no distributedintegration cycle

7. The challenge • We have to create an environment in which 3rd parties can integrate their software • Without deep system knowledge • We have to enable predictable integration • Preventing spaghetti SW • We must cater for sharing critical resources • AVG Platform • General purpose infrastructure Create a Win-Win atmosphere with 3rd parties

8. Design objectives • Fast and predictable integration of system extensions • Avoid an extensive (re)validation cycle • Enable convincing module test opportunities • Enable PC based testing • Orthogonal 3rd party SW integration • Leverage standard Linux infrastructure • Multi client usage of platform resources • Manage independently deployable building blocks • Limited building block correlation • Cater for extensions without the need to know all the details • Enhanced execution architecture • Independent application lifecycle • Enable true Multi window architecture • For Video and Graphics • Preserving application orthogonality

9. The mental model: SW bolt-ons Feat1 Feat2 Feat3 Feat4 process TV Middleware TV Platform MIPS (Linux 2.x)

10. SPACE

11. Introducing SPACE Application Feat1 FeatN TVMiddleware Platform API TV platform ApplicationManager Linux 2.x Have orthogonal applications • The resources in the system are explicitly and centrally managed • The client applications are system context unaware • The lifecycle, focus and visual layout of the client applications is centrally managed

12. Application Manager • Application lifecycle management • Starting, stopping applications based on remote control keys • Knows the system requirements of all applications • Focus handling • Determines the active window, reacts to user request • Manages the application requirements to the AV resources • Layout management • Determines how the applications are presented

13. Managing resources • There are implicitly managed resources by the kernel • TCP, flash, USB • Memory allocation • There are explicitly managed resources • AV platform resources • Memory and CPU resources • The explicitly managed resources • Have a statically defined execution behavior • Can by design limit the parallelism in the system • Require an explicitly resource controlled system

14. Resource Controlled System • Application Manager (amApp) • Knows the resource dependencies of any client application • Statically by design • The platform application (plfApp) • Has a static model of resource sharing • By design resources are multi/single client • Every application • Must be resource aware • Meaning the resource is or is not reserved for them

15. Multi Client Management otherApp tvApp SetFrequency (tuner) SetFrequency (tuner)   amApp plfApp ResourceOwner (tvApp) Linux 2.x

16. Resource Groups • There are different resource groups offered by the platform • Front-end, demux, decode, viewing mode, source selection • To allow multiple applications to access different parts of the system • Every resource group has a static set of interfaces • The Application Manager is the main Resource controller • It designates resource groups to clients • By informing the platform application whom to grant access

17. The platform API • The Philips platform API • Leverages as much as possible the current APIs • Enables the resource sharing model • Is supplier independent • AV Interface definition • Analog interfaces: Philips’ Analog TV API • Digital interfaces: NXP’s Digital TV API • AV Interface instance • Simple header files and libraries • Tool to generate proxy/stub code • Graphics interface definition • DirectFB ? ? ?

18. General Connection Management Concept • Connection Management is split up in • Destination setup • Output: FullScreen, PIP • Source setup • Input: HDMI, Tuner • Connection Management is distributed • Application Manager is responsible for • Destination setup • Client application lifecycle • Client applications are responsible for • Source setup • Client applications are destination unaware • Application Manager is source unaware

19. Example 3 SetSource(tuner) 6 Program Freq, PID, … 2 eventDestination(tvApp) 5 eventOnSourceSelected() tvApp 4 SelectTuner amApp plfApp Linux 2.x 1 SetDestinationFullScreen(tvApp)

20. Border Windows and Video Windows Border • Border windows are linked to video windows • Every application controlling video creates a border window • Application manager controls visibility of such a couple • Visible border windows are equal to the visible video windows • There can be multiple border windows for 1 video window • The border window determines the position of the video window • DirectFB support this via the concept of Input Only Windows • Geometry (positioning) is still available to allow focus management • No client buffer is required, no scaling is done Video

21. Graphics Connection Management tvApp App2 amApp Select App1 Draw configure DirectFB infrastructure source destination

22. Window Management responsibilities amApp start/kill Layout Size focus tvApp App1 App2 tvApp create draw App1 App2

23. Traditional DirectFB Window manager • Pluggable WM API • Move, Resize, Raise, Lower, Hide, Show Windows • Process Input, Handle Focus and Grabbing, Dispatch Events • Full Cursor implementation including Show, Hide, Reshape, Move • Composition of the screen or parts, e.g. triggered by • Flip() on a window surface • Move, Raise, Show etc. • 2D composition • Background, Windows, Cursor • Default WM module • Just follow application requests • Basic focus handling App App DirectFB API WM Module SHM

24. SaWMan • Shared Application & Window Manager • New DirectFB window manager • On its own like default window manager implementation • Default is maintained to preserve backwards compatibility • Required was a more controlled behavior • Add/Remove windows, can modify initial configuration • Configuration requests (move, resize, opacity), modify or reject • Filter input events before being processed by SaWMan • Completely overrule window layout, enable focus borders etc. • Application Manager can hook into most of the API flow Application Manager App App App SaWMan API DirectFB API SHM RPC to AM SaWMan Module

25. Window Management Focus mgt Layers Z-order Clients Child abducted San Jose Child abducted San Jose Message high Application med Border low Video HW

26. DirectFB layer mapping on HW • If we need fast rendering (>10FPS) and no HW acceleration available • We must use direct surface drawing • To achieve >25 FPS without SW scaling overhead • DirectFB must allow Layers to be mixed across HW Surfaces • Some Layers can have SW scaling, others may not Legend Graphics Versatile Layers Z-order Surface Layers Z-order Surface Performance Video GFX1 GFX1 Message Message high high GFX1 GFX1 Application Application med med GFX1 GFX0 Border Border low low Video0 Video0 Video Video HW HW Option 1 Option 2

27. Window Association • Windows can be associated to another • Meaning identical position and size • Focus only to associated group • Focus only to top window, not associated window • Example use-case: teletext + menu • Teletext and menu of teletext separate windows CreateAssociative Draw Client Draw window Flip time

28. Pixel Alignment • plfApp can crop video • Eg to remove DNM border artifacts • plfApp can scale video within border window • Eg to set the scaling to 4:3 or 16:9 • Any client application can indicate ‘pixel alignment’ • To assure that the graphics is pixel accurately positioned on top of the video • DirectFB caters for the alignment • plfApp configures the scale/crop factors • Other application is pan/zoom • Any application can use the SW scaling Crop & Scale Crop value VideoAligned Crop & Scale

29. Road to success

30. Planned extensions to DirectFB Published • SawMan • Shared Application Window Manager • Enhanced SW up/down scaling for windows • With dynamic layer reconfiguration and color key conversion • FusionDale & FusionIPC • Applied Fusion for Event handling and IPC • Memory usage optimization • Shared Memory Heap • Window Association • Link windows together for size and location • Window mapping on HW layers • Including layer specific configurations Pixel alignment • Extended Key handling • Sending keys to windows and return key not used • Audio Nodes • General ID for audio resources • Multi processor SPACE • Enable multiple control processors to host client applications To be published Plan

31. Main conclusion • The integration nightmare can become manageable • Using an open standard based multi application approach • DirectFB, Linux • With a clear orthogonal view • No dependencies between applications • Adding the necessary components for success • SPACE, SaWMan • SPACE • Is the architectural concept for multi application systems • Coping with the resource constraints in CE devices • SaWMan • Enables application lifecycle management, focus handling,and window management • Is fully available via the DirectFB mainline

32. Demo: SPACE