CS533 Concepts of Operating Systems

1. CS533 Concepts of Operating Systems The Performance of Micro-Kernel Based Systems

2. Micro-kernels and Binary Compatibility • Emulation libraries • Trampoline mechanism • Single server architecture • Multi-server architecture • IPC overhead proportional to number of servers (independent protection domains) CS533 - Concepts of Operating Systems

3. Micro-kernels must optimize IPC • Liedtke argues Mach’s overhead is due to poor implementation! • Optimized IPC implementation in L3 • Architectural level • System Calls, Messages, Direct Transfer, Strict Process Orientation, Control Blocks. • Algorithmic level • Thread Identifier, Virtual Queues, Timeouts/Wakeups, Lazy Scheduling, Direct Process Switch, Short Messages. • Interface level • Unnecessary Copies, Parameter passing. • Coding level • Cache Misses, TLB Misses, Segment Registers, General Registers, Jumps and Checks, Process Switch. CS533 - Concepts of Operating Systems

4. L3 IPC Performance vs Mach IPC CS533 - Concepts of Operating Systems

5. But Is That Enough? • What is the impact on overall system performance? • Haertig et al explore performance and extensibility of L4-based Linux OS vs Mach-based Linux and native Linux CS533 - Concepts of Operating Systems

6. L4Linux – a micro-kernel based Linux • Fully binary compliant with Linux/X86 • No changes to the architecture-independent parts of Linux • No Linux-specific modifications to the L4 kernel CS533 - Concepts of Operating Systems

7. L4Linux – Design & Implementation • Linux implemented as a single Linux server in a μ-kernel task • μ-kernel tasks used for Linux user processes • A single L4 thread in the Linux server handles system calls and page faults. This thread is multiplexed (treated as a virtual CPU) • On booting, the Linux server requests memory from its pager, which maps physical memory into the server’s address space • The Linux server then acts as the pager for the user processes it creates • L4 converts user-process page faults into an RPC to the Linux server, which maps pages from its address space to the user process

8. Interrupt Handling • Linux top halves are implemented as one server thread per interrupt source • L4 converts a hardware interrupt to a message to the appropriate thread • Linux bottom halves all execute in a single high priority thread • Linux interrupt threads have a higher priority than the main thread • avoids concurrent execution of Linux code on a uniprocessor

9. System Calls • System calls implemented as IPC between user process and the Linux server • Modified libc.so or libc.a avoid trap instructions and use L4 IPC instead to call the Linux server • User-level exception handler (trampoline) emulates the native system call ‘trap’ instruction for binary compatibility • L4 redirects trap to emulation library which then used L4 IPC to call the Linux server

10. Signals • Each user process has a separate signal-handler thread • Linux server’s delivers a signal by sending a message to the user process’s signal-handler thread • The signal-handler causes the user process’s main thread to save it’s state and enter Linux by manipulating the main thread’s SP and PC

11. Scheduling • All thread scheduling is done by the L4 kernel • The Linux server’s schedule() routine is only used for multiplexing the Linux server’s Main thread across concurrent Linux system calls

12. Experiment • What is the penalty of using L4Linux? • Compare L4Linux to native Linux • Does the performance of the underlying micro-kernel matter? • Compare L4Linux to MkLinux • Does co-location improve performance? • Compare L4Linux to an in-kernel version of MkLinux CS533 - Concepts of Operating Systems

13. Microbenchmarks • measured system call overhead on shortest system call “getpid()” Linux 223 cycles L4Linux 526 cycles L4Linux(trampoline) 753 cycles MkLinux (in kernel) 2050 cycles MkLinux (user) 14710 cycles CS533 - Concepts of Operating Systems

14. Microbenchmarks (cont.) • Measures specific system calls to determine basic performance. CS533 - Concepts of Operating Systems

15. Macrobenchmarks • measured time to recompile Linux server CS533 - Concepts of Operating Systems

16. Macrobenchmarks (cont.) • Next use a commercial test suite to simulate a system under full load. CS533 - Concepts of Operating Systems

17. Performance Analysis • L4Linux is, on average 8.3% slower than native Linux. Only 6.8% slower at maximum load. • MkLinux: 49% slower on average, 60% at maximum. • Co-located MkLinux: 29% slower on average, 37% at maximum CS533 - Concepts of Operating Systems

18. Conclusion? • Can hardware-based protection be made to work efficiently enough? • Did these experiments explore the cost of “fine grained” protection? CS533 - Concepts of Operating Systems

19. Spare Slides CS533 - Concepts of Operating Systems

20. The IPC Dilemma • IPC is very import in μ-kernel design • Increases modularity, flexibility, security and scalability. • Past implementations have been inefficient. • Message transfer takes 50 - 500μs. CS533 - Concepts of Operating Systems

21. The L3 (μ-kernel based) OS • A task consists of: • Threads • Communicate via messages that consist of strings and/or memory objects. • Dataspaces • Memory objects. • Address space • Where dataspaces are mapped. CS533 - Concepts of Operating Systems

22. Redesign Principles • IPC performance is the Master. • All design decisions require a performance discussion. • If something performs poorly, look for new techniques. • Synergetic effects have to be taken into considerations. • The design has to cover all levels from architecture down to coding. • The design has to be made on a concrete basis. • The design has to aim at a concrete performance goal. CS533 - Concepts of Operating Systems

23. Achievable Performance • A simple scenario • Thread A sends a null message to thread B • Minimum of 172 cycles • Will aim at 350 cycles (7 μs) • Will actually achieve 250 cycles (5 μs) CS533 - Concepts of Operating Systems

24. Levels of the redesign • Architectural • System Calls, Messages, Direct Transfer, Strict Process Orientation, Control Blocks. • Algorithmic • Thread Identifier, Virtual Queues, Timeouts/Wakeups, Lazy Scheduling, Direct Process Switch, Short Messages. • Interface • Unnecessary Copies, Parameter passing. • Coding • Cache Misses, TLB Misses, Segment Registers, General Registers, Jumps and Checks, Process Switch. CS533 - Concepts of Operating Systems

25. Architectural Level • System Calls • Expensive! So, require as few as possible. • Implement two calls: • Call • Reply & Receive Next • Combines sending an outgoing message with waiting for an incoming message. • Schedulers can handle replies the same as requests. CS533 - Concepts of Operating Systems

26. A Complex Message Messages • Complex Messages: • Direct String, Indirect Strings (optional) • Memory Objects • Used to combine sends if no reply is needed. • Can transfer values directly from sender’s variable to receiver’s variables. CS533 - Concepts of Operating Systems

27. User A Kernel User B Direct Transfer • Each address space has a fixed kernel accessible part. • Messages transferred via the kernel part • User A space -> Kernel -> User B space • Requires 2 copies. • Larger Messages lead to higher costs CS533 - Concepts of Operating Systems

28. Shared User Level memory (LRPC, SRC RPC) • Security can be penetrated. • Cannot check message’s legality. • Long messages -> address space becoming a critical resource. • Explicit opening of communication channels. • Not application friendly. CS533 - Concepts of Operating Systems

29. User A Kernel User B Temporary Mapping • L3 uses a Communication Window • Only kernel accessible, and exists per address space. • Target region is temporarily mapped there. • Then the message is copied to the communication window and ends up in the correct place in the target address space. CS533 - Concepts of Operating Systems

30. Temporary Mapping • Must be fast! • 2 level page table only requires one word to be copied. • pdir A -> pdir B • TLB must be clean of entries relating to the use of the communication window by other operations. • One thread • TLB is always “window clean”. • Multiple threads • Interrupts – TLB is flushed • Thread switch – Invalidate Communication window entries. CS533 - Concepts of Operating Systems

31. Strict Process Orientation • Kernel mode handled in same way as User mode • One kernel stack per thread • May lead to a large number of stacks • Minor problem if stacks are objects in virtual memory CS533 - Concepts of Operating Systems

32. User area Kernel area tcb Kernel stack Thread Control Blocks (tcb’s) • Hold kernel, hardware, and thread-specific data. • Stored in a virtual array in shared kernel space. CS533 - Concepts of Operating Systems

33. Tcb Benefits • Fast tcb access • Saves 3 TLB misses per IPC • Threads can be locked by unmapping the tcb • Helps make thread persistent • IPC independent from memory management CS533 - Concepts of Operating Systems

34. Algorithmic Level • Thread ID’s • L3 uses a 64 bit unique identifier (uid) containing the thread number. • Tcb address is easily obtained • anding the lower 32 bits with a bit mask and adding the tcb base address. • Virtual Queues • Busy queue, present queue, polling-me queue. • Unmapping the tcb includes removal from queues • Prevents page faults from parsing/adding/deleting from the queues. CS533 - Concepts of Operating Systems

35. Algorithmic Level • Timeouts and Wakeups • Operation fails if message transfer has not started t ms after invoking it. • Kept in n unordered wakeup lists. • A new thread’s tcb is linked into the list τ mod n. • Thread with wakeups far away are kept in a long time wakeup list and reinserted into the normal lists when time approaches. • Scheduler will only have to check k/n entries per clock interrupt. • Usually costs less the 4% of ipc time. CS533 - Concepts of Operating Systems

36. Algorithmic Level • Lazy Scheduling • Only a thread state variable is changed (ready/waiting). • Deletion from queues happens when queues are parsed. • Reduces delete operations. • Reduces insert operations when a thread needs to be inserted that hasn’t been deleted yet. CS533 - Concepts of Operating Systems

37. Algorithmic Level • Short messages via registers • Register transfers are fast • 50-80% of messages ≥ 8 bytes • Up to 8 byte messages can be transferred by registers with a decent performance gain. • May not pay off for other processors. CS533 - Concepts of Operating Systems

38. Interface Level • Unnecessary Copies • Message objects grouped by types • Send/receive buffers structured in the same way • Use same variable for sending and receiving • Avoid unnecessary copies • Parameter Passing • Use registers whenever possible. • Far more efficient • Give compilers better opportunities to optimize code. CS533 - Concepts of Operating Systems

39. Code Level • Cache Misses • Cache line fill sequence should match the usual data access sequence. • TLB Misses • Try and pack in one page: • Ipc related kernel code • Processor internal tables • Start/end of Larger tables • Most heavily used entries CS533 - Concepts of Operating Systems

40. Coding Level • Registers • Segment register loading is expensive. • One flat segment coving the complete address space. • On entry, kernel checks if registers contain the flat descriptor. • Guarantees they contain it when returning to user level. • Jumps and Check • Basic code blocks should be arranged so that as few jumps are taken as possible. • Process switch • Save/restore of stack pointer and address space only invoked when really necessary. CS533 - Concepts of Operating Systems

41. L4 Slides CS533 - Concepts of Operating Systems

42. Introduction • μ-kernels have reputation for being too slow,inflexible • Can 2nd generation μ-kernel (L4) overcome limitations? • Experiment: • Port Linux to run on L4 (Mach 3.0) • Compared to native Linux, MkLinux (Linux on 1st gen Mach derived μ-kernel) CS533 - Concepts of Operating Systems

43. Introduction (cont.) • Test speed of standard OS personality on top of fast μ-kernel: Linux implemented on L4 • Test extensibility of system: • pipe-based communication implemented directly on μ-kernel • mapping-related OS extensions implemented as user tasks • user-level real-time memory management implemented • Test if L4 abstractions independent of platform CS533 - Concepts of Operating Systems

44. L4 Essentials • Based on threads and address spaces • Recursive construction of address spaces by user-level servers • Initial address space σ0 represents physical memory • Basic operations: granting, mapping, and unmapping. • Owner of address space can grant or map page to another address space • All address spaces maintained by user-level servers (pagers) CS533 - Concepts of Operating Systems

45. L4Linux – Design & Implementation • Fully binary compliant with Linux/X86 • Restricted modifications to architecture-dependent part of Linux • No Linux-specific modifications to L4 kernel CS533 - Concepts of Operating Systems

46. L4Linux – Design & Implementation • Address Spaces • Initial address space σ0 represents physical memory • Basic operations: granting, mapping, and unmapping. • L4 uses “flexpages”: logical memory ranging from one physical page up to a complete address space. • An invoker can only map and unmap pages that have been mapped into its own address space CS533 - Concepts of Operating Systems

47. L4Linux – Design & Implementation CS533 - Concepts of Operating Systems

48. L4Linux – Design & Implementation • Address Spaces (cont.) • I/O ports are parts of address spaces. • Hardware interrupts are handled by user-level processes. The L4 kernel will send a message via IPC. CS533 - Concepts of Operating Systems

49. L4Linux – Design & Implementation • The Linux server • L4Linux will use a single-server approach. • A single Linux server will run on top of L4, multiplexing a single thread for system calls and page faults. • The Linux server maps physical memory into its address space, and acts as the pager for any user processes it creates. • The Server cannot directly access the hardware page tables, and must maintain logical pages in its own address space. CS533 - Concepts of Operating Systems

50. L4Linux – Design & Implementation • Interrupt Handling • All interrupt handlers are mapped to messages. • The Linux server contains threads that do nothing but wait for interrupt messages. • Interrupt threads have a higher priority than the main thread. CS533 - Concepts of Operating Systems