Dive Into VM Live Migration

Size: px

Start display at page:

Download "Dive Into VM Live Migration"

Dayna Atkinson
8 years ago
Views:

1 Dive Into VM Live Migration OpenStack Liberty Summit 2015 Vancouver Michał Dulko Michał Jastrzębski Paweł Koniszewski

2 Why Bother?

3 Use Cases o Imminent host failure o Maintenance mode o Optimal resource placement

4 Imminent Host Failure o Cooling issues o Storage problems o Networking problems o Your datacenter was struck by a flood

5 Maintenance Mode o Firmware upgrades o Hardware upgrades o Kernel upgrades

6 Optimal Resource Placement o Reduce costs o Move VMs closer to their storage to lessen network latency o Stack more VMs on hosts to save power o Increase resiliency o Noisy neighbour separation o Spread VMs across more hosts

Stack more VMs on hosts to save power o Increase

7 General Flow

8 Assumptions o Live o Consistent o Transparent o Minimal service disruption

9 Migrations in OpenStack Non-live migration (cold migration) o nova migrate <server> True live migration (shared storage or volume-based) o nova live-migration <server> [<host>] Block live migration o nova live-migration --block-migrate <server> [<host>]

volume-based) o nova live-migration <server> [<host>] Block

10 Compatibility Migration type Local storage Volumes Shared storage Block LM True LM Block LM with read-only devices True LM with read-only devices

11 Live Migration Process o Pre-Migration o Reservation o Iterative pre-copy o Stop and copy o Commitment

12 Pre-migration Pre-migration Reservation Compute node A Compute node B Iterative pre-copy Stop and copy VM A Active Commitment Active VM on physical host A, host B selected by scheduler or preselected.

copy VM A Active Commitment Active VM on physical

13 Reservation Pre-migration Reservation Compute node A Compute node A Compute node B Compute node B Iterative pre-copy Stop and copy VM A VM A Active VM A VM A Reserved Commitment ACTIV E Confirm availability of resources on host B; reserve a new VM.

and copy VM A VM A Active VM A VM A Reserved Commitment ACTIV

14 Iterative pre-copy Pre-migration Reservation Iterative pre-copy Stop and copy Compute node A VM A Active Copying Compute node B VM A Paused Commitment Memory is transferred from A to B and next dirtied pages are iteratively copied.

Compute node B VM A Paused Commitment Memory is

15 Stop and copy Pre-migration Reservation Iterative pre-copy Stop and copy Commitment Compute node A Compute node A VM A PAUSE D VM A Paused Copy Compute node B Compute node B VM A PAUSE D VM A Paused Suspend VM and copy remaining pages and CPU state.

PAUSE D VM A Paused Copy Compute node B Compute node B VM A

16 Commitment Pre-migration Reservation Compute node A Compute node A Compute node B Compute node B Iterative pre-copy Stop and copy Commitment VM A PAUSE D VM A PAUSE D VM A Active Host B becomes primary host for VM A.

Iterative pre-copy Stop and copy Commitment VM A

17 Performance & reliability

18 Pitfalls o OpenStack does not allow triggering any operations on VM during LM o VMs with intensive memory workload are hard to migrate o LM generates heavy load on network o Migrations between CN with different CPUs o Memory oversubscription

workload are hard to migrate o LM generates heavy load on

19 Interacting With Live Migration o OpenStack disallow any operation on ongoing LM o You can use virsh instead to interact

20 Diagnosis o Information about ongoing LM virsh domjobinfo <domain> Time elapsed ms Data processed GiB Data remaining GiB Data total GiB Constant pages Normal pages Normal data GiB Expected downtime 1023 ms

137 GiB Data remaining 4.600 GiB Data total 16.

21 Forcing Migration Finish o Cancel on-going LM virsh domjobabort <domain> o Pause VM during LM virsh suspend <domain>

22 Tuning Maximum Downtime o QEMU virsh qemu-monitor-command --hmp <domain> migrate_set_downtime <time (sec)> o libvirt virsh migrate-setmaxdowntime <domain> <time (sec)>

23 Auto Converge o nova.conf setting live_migration_flag += VIR_MIGRATE_AUTO_CONVERGE

24 Tunneled Migration o nova.conf setting live_migration_flag += VIR_MIGRATE_TUNNELLED libvirt libvirt Hypervisor Hypervisor Source Host Destination Host

25 Tunneled Migration o nova.conf setting live_migration_flag -= VIR_MIGRATE_TUNNELLED libvirt libvirt Hypervisor Hypervisor Source Host Destination Host

26 Tuning Bandwidth o libvirt virsh migrate-setspeed <domain> <speed (MiB/s)> o nova.conf settings live_migration_bandwidth = <speed (MiB/s)>

27 XBZRLE Compression o nova.conf settings live_migration_flag += VIR_MIGRATE_COMPRESSED Sent Page Cache Updated Page Delta Compression Source Host Delta Received Pages Delta Apply Delta Destination Host Updated Page

28 LM On Dedicated Network o nova.conf o live_migration_uri = qemu+tcp://%s/system Compute node A Management Network Compute node B VM A Active VM A Paused

29 LM On Dedicated Network o o nova.conf o live_migration_uri = qemu+tcp://%s-lm/system Set up your DNS to resolve hostnames with -lm suffix to IPs in your dedicated network. Compute node A Management Network Compute node B VM A Active LM Network VM A Paused

30 Different CPUs Between Compute Nodes o CPU instruction set of source node needs to be a subset of CPU instruction set of destination node Compute Node A MMX AVX Live Migration Passed Failed Compute Node B MMX SSE2 AVX

31 Different CPUs Between Compute Nodes o This can be skipped by explicitly setting VM CPU model in nova.conf: o cpu_mode = custom o virt_type = kvm or virt_type = qemu o And then you can set cpu_model o List of supported named CPUs is in libvirt/cpu_map.xml

32 Memory Oversubscription o LM to specific host does not use memory oversubscription o ram_allocation_ratio Compute Node A 2 GB RAM 2 GB nova-conductor 2 GB Reported RAM = available - reserved 2 GB nova-scheduler ram_allocation_ratio = GB

33 Memory Oversubscription o Skip it by o reserved_host_memory_mb=-2048 Compute Node A 2 GB RAM 4 GB nova-conductor 4 GB Reported RAM = available - reserved 4 GB nova-scheduler ram_allocation_ratio = GB

34 Secure Live Migration

35 Why Security Matters? o Everything can be sniffed! o Migrated machines can contain sensitive data o Legal issues with unencrypted data transfer

36 Encryption o Hypervisor native encryption o QEMU doesn t support it o libvirt tunneled transport o live_migration_uri = qemu+ssh://%s/system o live_migration_flag += VIR_MIGRATE_TUNNELLED o Uses only one core o IPSec tunnel between hosts

37 Transfer rate [GBps] Memory Access Is Critical Intel(R) Xeon(R) CPU E v2 Intel(R) Xeon(R) CPU E v QEMU+SSH QEMU+TCP

38 Future Of Live Migration

39 Multithreaded Compression o Compress every page sent during LM o zlib used for compression o Configurable: o Number of threads o Comperession ratio

40 Post-copy Live Migration o Move workload immediately to destination host Compute node A Compute node B Copying VM A Paused VM A Active

41 Post-copy Live Migration o Cheap solution to finish live migration in a finite time o VM needs to be rebooted in case of failure o Heavy performance impact

42 Active LM Monitoring In OpenStack o Track memory transfer progress o Detect possible problems and take actions

43 Actions On Ongoing Live Migration o Pause VM o Abort LM o See progress o Change configuration on the fly: o Maximum tolerable VM down time o Transfer bandwith

44 Your voice matters! o Mailing lists: o openstack-dev@lists.openstack.org o openstack-operators@lists.openstack.org o Win The Enterprise group: o pawel.koniszewski@intel.com (IRC: pkoniszewski) o michal.jastrzebski@intel.com (IRC: inc0) o michal.dulko@intel.com (IRC: dulek)

45 Q&A (& disclaimers) Intel technologies features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at intel.com. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries Intel Corporation.

Utilization-based Scheduling in OpenStack* Compute (Nova)

Utilization-based Scheduling in OpenStack* Compute (Nova) September 2015 Authors: Reviewers: Lianhao Lu, Yingxin Cheng Malini Bhandaru, Will Auld Intel technologies features and benefits depend on system