Live Wide-Area Migration of Virtual Machines Including Local Persistent State

1. Robert Bradford, Evangelos Kotsovinos, Anja Feldmann, Harald Schiöberg Presented by Kit Cischke Live Wide-Area Migration of Virtual Machines Including Local Persistent State

2. Differences in Contributions • The first paper discussed transferring run-time memory of a VM in a LAN. • Cool. • This paper expands on that to transfer the VM’s image, its persistent state and on-going network connections over a WAN as well. • By combining pre-copying, write-throttling and a block-driver, we can achieve this.

3. Introduction • In this project, the authors want to extend live VM migration to include: • Persistent state (file systems used by the VM) • Open network connections • Why? • Many apps running on a VM need that storage, and NAS systems may not be available in the new location. • Moving across a WAN will almost certainly involve an IP change, and we don’t want to (overly) disrupt TCP connections. • Contribution • A system enables live migration of VMs that use local storage and open network connections without severely disrupting their live services.

4. Highlights • Some highlights of this work: • Built upon the Xen Live Migration facility as part of XenoServer. • Enables: • Live migration • Consistency • Minimal Service Disruption • Transparency • Utilizes: • Pre-copying, write-throttling and IP tunneling.

5. System Design - Environment • Both the source and destination run Xen, with the VM running XenLinux. • Uses blocktap to export block devices into the migrated VM. • Block devices are file-backed meaning the contents of the block device are stored in an ordinary file on the file system of the VM.

6. System Design - Architecture • The initialization stage starts things off by prepping the migration. • The bulk transfer stage pre-copies the disk image of the VM to the destination while the VM continues to run. • Xen transfer is then initiated, which performs incremental migration, again without stopping the VM. • While the transfers are occurring, all disk writes are intercepted as deltas that will be forwarded to the destination. • Deltas include the data written, the location written and the size of the data. • The deltas are recorded into a queue that will be transferred later. • If write activity is too high and too many deltas are being generated, write-throttling is engaged to slow down the VM. • In parallel with Xen transfer, the deltas are applied to the destination VM. • At some point, the source VM is paused, the destination is started and a temporary network redirect is created to handle the potential IP changes.

7. Implementation - Initialization • Authentication, authorization and access control are handled by XenoServer. • The migration client forks, creating a listener process that signals the block-driver to enter record mode. • In record mode, the driver copies the writes to the listener process, which transfers them to the destination. • The other half of the migration client begins the bulk transfer. • At the destination, there is also a fork in the daemon. One receives the bulk transfer, the other receives the deltas.

8. Implementation – Bulk Xfer • The VM’s disk image is transferred from the source to the destination. • XenoServers platform supports copy-on-write along with immutable template disk images, so we can just transfer the changes, rather than the whole image.

9. Implentation – Xen Migration • The system here relies on the built-in migration mechanism of Xen. • Xen logs dirty memory pages and copies them to the destination without stopping the source VM. • Eventually, we will have to pause the source and copy the remaining memory pages. • Then we start up the migrated VM.

10. Implementation - Intercepting • The blkfront device driver communicates with the dedicated storage VM via ring buffer. • The blktap framework intercepts requests, but does it in user space. • Once a disk request makes it to the backend, it is both committed to the disk and sent to the migration client. • The client then packages the write up as a delta.

11. Implementation - Application • After the bulk transfer, and in parallel with the Xen transfer, the deltas are transferred and applied to the migrated VM by the migration daemon in the storage VM at the destination. • If the delta queue becomes empty and the Xen migration is finished, I/O requests are put on hold until the application of the current crop of deltas is finished. • The authors found that delta application was normally finished before Xen migration, adding zero time to the overall migration.

12. Implementation – Write Throttling • If the VM attempts to complete more writes than a given threshold value, future write attempts are delayed by the block driver. • This process repeats, with the delay and threshold doubling each time. • Experimentally, 16384 is a suitable threshold with a delay of 2048 μs also being good. • Enforcement is separated from policy for extensibility.

13. Implementation – WAN Redirection • If the IP of the VM changes, we use IP tunneling and Dynamic DNS to prevent dropped network connections. • Just before the source VM is paused, an IP tunnel is created from the source to destination using iproute2. • Once the destination VM is capable of responding to requests at its new IP, Dynamic DNS forwards the requests to the new IP. • Packets that arrive during the final stage of migration are simply dropped. • Once no connections exist that use the old IP, the tunnel is torn down. • Practically, this works because: • The source server only needs to cooperate for a short time, most network connections are short-lived and if nothing else, it’s no worse than what you get if the VM doesn’t even try.

14. Evaluation - Metrics • Want to evaluate the disruption of the system, as perceived by users. • Spoiler: Results look good. • Want to show the system handles diabolical workloads, defined in this paper as being heavy disk accessors, rather than heavy memory accessors. • Downtime: Time between pausing the VM on the source and resuming on the destination. • Disruption Time: Time during which clients observe a reduction in service responsiveness. • Additional Disruption: Difference between disruption time and downtime. • Migration time: Time from migration request to running VM at destination. • Number of Deltas and Delta Rate: How many file system changes and how often they occur.

15. Eval – Workload Overview • Web server serving static content, serving dynamic web application and video streaming. • Chosen for realistic usage scenarios and because they neatly trifurcate the spectrum of disk I/O patterns: • Dynamic app generates lots of bursty writes • Static workload generates a medium amount of constant writes • Streaming video causes few writes, but is very sensitive to disruption.

16. Eval – Experimental Setup • Three hosts • Dual Xeon 3.2 GHz, 4 GB DDR RAM, mirrored RAID array of U320 SCSI disks. • The migrated VM was provided with 512 MB RAM and a single CPU. • All hosts were connected by a 100 Mbps switched Ethernet networks. • The migrated VM was running Debian on a 1GB ext3 disk. • Host C is the client. • To emulate WAN transfers, traffic shaping was used to limit the bandwidth to 5Mbps with 100 ms of latency. • Representative of host in London and U.S. east coast.

17. Results – LAN Migration • Measured disruption is 1.04 seconds and “practically unnoticeable by a human user.” • Few deltas by the log files for web server. • If you’re shooting for a “5 9’s” uptime, you still get 289 migrations a year.

18. Results – LAN Migration • phpBB with a randomly posting script. • Disruption is 3.09 seconds due to more deltas. • HTTP throughput is almost not affected and total migration time is shorter. • 5 9’s uptime still lets us migrate 98 migrations.

19. Results – LAN Migration • Streamed a large video file, viewed by a human on host C. • Disruption is 3.04 seconds and alleviated by the buffer of the video player. • No packets are lost, but there is lots of retransmission.

20. Comparison to Freeze-and-Copy • Clearly, freeze and copy provides a much worse disruption than the live migration.

21. Results – WAN Migration Longer migration time leads to more deltas. The tunneling let the connections persist. 68 total seconds of disruption, which is a lot, but much less than freeze-and-copy.

22. Results – Diabolical Workload Ran the Bonnie benchmark as a diabolical process, generating lots of disk writes. Needed to throttle twice. Initially, the bulk transfer is severely impeded and fixed. The overall migration takes 811 s, and without throttling, the transfer would have taken 3 days.

23. Results – I/O Overhead • The overhead of intercepting deltas is pretty low and only noticeable during the migration.

24. Conclusions • Demonstrated a VM migration scheme that includes persistent state, maintains open network connections and therefore works over a WAN without major disruption. • It can handle high I/O workloads too. • Works much better than freeze-and-copy. • Future work includes batching of deltas, data compression and “better support for sparse files.”