1 / 29

Speculative Execution in a Distributed File System

Speculative Execution in a Distributed File System. Ed Nightingale Peter Chen Jason Flinn University of Michigan. Motivation. Why are distributed file systems slow(er)? Sync n/w messages provide consistency Sync disk writes provide safety Sacrifice guarantees for speed

levia
Download Presentation

Speculative Execution in a Distributed File System

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Speculative Execution in a Distributed File System Ed Nightingale Peter Chen Jason Flinn University of Michigan

  2. Motivation • Why are distributed file systems slow(er)? • Sync n/w messages provide consistency • Sync disk writes provide safety • Sacrifice guarantees for speed • Can DFS can be safe, consistent and fast? • Yes! With OS support for speculative execution

  3. Big Idea: Slow Way Big Idea: Speculator Client Server 1) Checkpoint RPC Req RPC Req 2) Speculate! Block! RPC Resp RPC Resp 3) Correct? No: restore process & re-execute Yes: discard ckpt. RPC Req RPC Resp • Guarantees without blocking I/O!

  4. Conditions for Success • Operations are highly predictable • Conflicts are rare • Checkpoints are cheaper than network I/O • 52 µs for small process • Computers have resources to spare • Need memory and CPU cycles for speculation

  5. Outline • Motivation • Implementing speculation • Multi-process speculation • Using Speculator • Evaluation

  6. Spec Undo log Implementing Speculation 1) System call 2) Create speculation Time Process Checkpoint

  7. Spec Undo log Speculation Success 1) System call 2) Create speculation 3) Commit speculation Time Process Checkpoint

  8. Spec Undo log Speculation Failure 2)Create speculation 1) System call 3)Fail speculation Time Process Process Checkpoint

  9. Ensuring Correctness • Spec processes often affect external state • Three ways to ensure correct execution • Block • Buffer • Propagate speculations (dependencies)

  10. Systems Calls • Block calls that externalize state • Allow read-only calls (e.g. getpid) • Allow calls that modify only task state (e.g. dup2) • File system calls -- need to dig deeper • Mark file systems that support Speculator Call sys_getpid() getpid reboot Block until specs resolved mkdir Allow only if fs supports Speculator

  11. Spec(stat) Spec (mkdir) Undo log Output Commits 1) sys_stat 2) sys_mkdir 3) Commit speculation Time Process “stat worked” Checkpoint Checkpoint “mkdir worked”

  12. Multi-Process Speculation • Processes often cooperate • Example: “make” forks children to compile, link, etc. • Would block if speculation limited to one task • Allow kernel objects to have speculative state • Examples: inodes, signals, pipes, Unix sockets, etc. • Propagate dependencies among objects • Objects rolled back to prior states when specs fail

  13. Spec 1 Spec 1 Spec 2 Multi-Process Speculation Checkpoint Checkpoint Checkpoint Checkpoint Checkpoint pid 8000 pid 8001 Chown-1 Chown-1 Write-1 Write-1 inode 3456

  14. Multi-Process Speculation • What we handle: • DFS objects, RAMFS, Ext3, Pipes & FIFOs • Unix Sockets, Signals, Fork & Exit • What we don’t (i.e. we block) • System V IPC • Multi-process write-shared memory

  15. Outline • Motivation • Implementing speculation • Multi-process speculation • Using Speculator • Evaluation

  16. Example: NFSv3 Linux Client 1 Server Client 2 Modify B Write Commit Open B Getattr

  17. Example: SpecNFS Client 1 Server Client 2 Write+Commit Modify B speculate Getattr Open B speculate Getattr Open B speculate

  18. Problem: Mutating Operations Client 1 1. cat foo > bar Client 2 2. cat bar • bar depends on cat foo • What does client 2 view in bar?

  19. Solution: Mutating Operations • Server determines speculation success/failure • State at server never speculative • Send server hypothesis speculation based on • List of speculations an operation depends on • Requires server to track failed speculations • Requires in-order processing of messages

  20. Group Commit • Previously sequential ops now concurrent • Sync ops usually committed to disk • Speculator makes group commit possible Client Client Server Server write commit write commit

  21. Putting it all Together: SpecNFS • Apply Speculator to an existing file system • Modified NFSv3 in Linux 2.4 kernel • Same RPCs issued (but many now asynchronous) • SpecNFS has same consistency, safety as NFS • Getattr, lookup, access speculate if data in cache • Create, mkdir, commit, etc. always speculate

  22. Putting it all Together: BlueFS • Design a new file system for Speculator • Single copy semantics • Synchronous I/O • Each file, directory, etc. has version number • Incremented on each mutating op (e.g. on write) • Checked prior to all operations. • Many ops speculate and check version async

  23. Outline • Motivation • Implementing speculation • Multi-process speculation • Using Speculator • Evaluation

  24. Apache Benchmark • SpecNFS up to 14 times faster

  25. The Cost of Rollback • All files out of date SpecNFS up to 11x faster

  26. Conclusion • Speculator greatly improves performance of existing distributed file systems • Speculator enables new file systems to be safe, consistent and fast

  27. Group Commit & Sharing State

  28. Apache Benchmark

  29. Related Work • Chang & Gibson, Fraser & Chang • Speculative pre-fetching • Time Warp • Virtual Time: distributed simulations • Hardware branch prediction • Transactional file systems

More Related