DTrace for Oracle Linux

2. DTrace for Oracle Linux Avi MillerPrincipal Program Manager, Oracle Linux

3. Overview • Setting the stage… • State of the work. • What can you do with it… (and how)? • A look ahead…

4. Setting the stage… What is DTrace? • DTrace is a runtime observer across the software stack. • Static tracing using instrumentation compiled into kernel and applications. • Dynamic tracing using probes inserted at runtime. • ‘D’ scripts execute when probes fire (under predicate control). • Decision to record events can be deferred using speculative tracing. • DTrace detects illegal memory accesses and reports an error rather than crashing the system!

5. Setting the stage… Why do we want DTrace for Oracle Linux? • A plethora of tracing tools exists for Linux, for varying use cases: • Inconsistent syntax, scripting languages, and output formats. • Lack of integration of both kernel and application tracing in a single tool. • DTrace provides an integrated solution. • Administrators and developers “know” DTrace from Oracle Solaris. • Existing ‘D’ scripts can be used. • Customers ask for it!

6. Setting the stage… What is DTrace for Oracle Linux? • It is… DTrace! • Full(1) port of the Oracle Solaris implementation. • Tailored to the Linux kernel. • Brings new features to Linux (page fault notifiers, CTF, …). • Source code for the DTrace core and providers is available at oss.oracle.com. (1) Standard disclaimers apply.

7. State of the work

8. State of the work Work in progress; not feature complete quite yet. • Project start: summer/fall 2010 • First public release: 0.1.0 (October 2011) • Current release: 0.3.1 (October 2012) • Based on Oracle Linux Unbreakable Enterprise Kernel 2.6.39 (UEK2) • Available as technology preview kernel • 64-bit only (x86_64)

9. High level overview Modular framework • Comprises several packages: • Kernel RPM: UEK2 with patches that augment the tracing infrastructure • Kernel modules: DTrace core and standard providers • Compact Type Format (CTF) library for DTrace • DTrace user space utility, incl. translators • Supports standard extensions: • Custom providers (provider-to-framework API) • Custom consumers (ioctl()-based API)

10. Kernel Oracle Linux Unbreakable Enterprise Kernel modifications • Writable system call table. • Tracing specific fields in the task_struct structure. • Notifier support for page faults. • Generic iterator over kallsyms data, and handle compiled-in modules. • Kernel data type and external variable data stored as CTF data in a separate kernel module. • SDT probe locations linked into the kernel. • Very minimal implementation of cyclics.

11. Probes Two flavours • Direct call probes • Direct calls to the dtrace_probe() function. • Either from regular execution flow or timer-based. • Trap-based probes • Single byte instruction inserted to trigger a trap. • Statically Defined Tracing (SDT) • Function Boundary Tracing (FBT) • …

12. Probes Direct call probes (provider:module:function:name) • dtrace:::BEGIN, dtrace:::END, dtrace:::ERROR • Triggered by the DTrace core. • profile:::tick-<n> where <n> is an integer followed by a length of time specifier (ns, ms, s, …) or “hz” • Triggered by a high resolution timer. • syscall::<syscall>:entry, syscall::<syscall>:return for each system call • Triggered upon entering the kernel for system call, and right before returning from the system call.

13. Probes Statically Defined Tracing (SDT) probes (provider:module:function:name) • proc:::<name> where <name> is one of: • create, lwp-create, start, lwp-start, exec, exec-failure, exec-success, exit, lwp-exit • signal-send, signal-discard, signal-handle, signal-clear • sched:::<name> where <name> is one of: • change-pri, enqueue, dequeue, tick • on-cpu, off-cpu, remain-cpu, preempt, surrender, sleep, wakeup • io:::<name> where <name> is one of: • start, wait-start, wait-done, done

14. Dare to compare? Utterly useless comparison

15. What can you do with it… (and how)?

16. High level workflow a.k.a. What we would like to do Select probes that we are interested in Process events and associated actions 1 2 3 4 Listen for probe fire events on selected probes Generate output

17. What happens when a probe fires? Linear tracing Kernel DTrace core Consumer • Probe reached: • Direct call • Trap • Timer-based: • Direct call • Process event: • Predicate eval • Record event • Execute script • Loop: • Retrieve buffer • Process events • Show output

18. What happens when a probe fires? Aggregation: f(f(x1), f(x2)) ≡ f(x1, x2) Kernel DTrace core Consumer • Probe reached: • Direct call • Trap • Timer-based: • Direct call • Process event: • Predicate eval • Aggregate data • Loop: • Retrieve buffer • Aggregate data • Show output

19. What happens when a probe fires? Speculative tracing Kernel (t1) DTrace core (t1) Point Probe fires Predicate eval Store event Kernel (t2) DTrace core (t2) Consumer (t2) Probe fires Commit/Discard Retrieve buffer Process events

20. Basic example(Aggregation, global variable, predicate) • syscalls.d syscall::: { @[probefunc] = count(); } tick-1s /i++ >= 10/ { exit(0); }

21. Cross-probe example(Thread variable, formatted output) • rtime.d syscall::read:entry { self->t = timestamp; } syscall::read:return /self->t != 0/ { printf(“%s spent %d nsecs in read()\n”); self->t = 0; }

22. Fancy example(Multi-probe action, xlators, formatted output) • io1.d io:::done, io:::start, io:::wait-done, io:::wait-start { printf(“%8s %10s: %d %16s (%s size %d @ sect %d)\n”, args[1]->dev_statname, probename, timestamp & 1000000000, execname, args[0]->b_flags & B_READ ? “R” : args[0]->b_flags & B_WRITE ? “W” : “?”, args[0]->b_bcount, args[0]->b_blkno); }

23. Workflow matters Little details make all the difference Enable probes that we are interested in Process events until done: retrieve, process, output CPU 2 CPU 1 1 2 3 4 … Sleep Commence recording of probe events Stop recording events and disable probes CPU n

24. Output is not time-sequential Why? • DTrace consumer retrieves CPU buffers every N ns. • CPU buffers are retrieved sequentially. • Not necessarily by CPU ID! • Within a CPU buffer, events are time-sequential. • Across multiple CPUs, there are no guarantees. • Solution: use timestamps!

25. Details that matter a.k.a. Good things to keep in mind • When a probe fires, all actions associated with that probe will: • Execute in the order they appear in the script. • Execute on the CPU that triggered the probe. • Kernel implementation details matter (e.g. IO completion is typically not associated with the task that started it.) • If you do not print aggregates, they will be printed automatically after the END probe fires. • If you want fancy output, consider post-processing.

26. Tracing incurs overhead There is no such thing as free tracing… • Cost of calling an probe that has not been enabled: ~75 ns • Cost of calling a probe with empty action: ~95 ns • Cost of triggering an SDT probe: ~3600 ns (38x direct probe) • SDT probe is not called unless enabled • SDT probe uses a trap, which is relatively expensive • Tracing = data collection!

27. A look ahead…

28. A look ahead… Features for future releases • Function boundary tracing • Probe at entry and exit of any function • More SDT probes • Cover more Oracle Linux subsystems • Userspace probing • Statically Defined Tracing of applications • Function Boundary Tracing of applications • Arbitrary address probes

29. A look ahead… Coming of age… • Release as a supported feature. • Integration into the Oracle Linux Unbreakable Enterprise Kernel. • New DTrace features. • Stability and correctness remains the top priority!

30. Graphic Section Divider

31. NEW: Oracle Linux Curriculum Footprint Oracle Linux Training from Oracle University Oracle Linux System Administration Instructor-led and Live virtual Unix/Linux Essentials Instructor-led and Live virtual This Oracle Linux System Administration course teaches you all the essential system administration skills and includes key information specific to Oracle Linux: Unbreakable Enterprise Kernel, Ksplice, ULN, and other key features Visit: oracle.com/education/linux

32. Resources Join our communities YouTube.com/ oraclelinuxchannel Blogs.oracle.com/linux Oracle Linux Experts Group Facebook.com/OracleLinux @ORCL_Linux Visit Oracle.com/linux Download for FREE edelivery.oracle.com/linux

33. The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract.It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.