Yaya48/QEMU-S5L8950X
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Import QEMU upstream snapshot d2e570c

Browse Source 
Upstream: https://gitlab.com/qemu-project/qemu.git

Upstream-Commit: d2e570cc0f97b936902a5b1b86b73c0f5998b475
This commit is contained in:
Yaya48 committed 2026-08-31 02:15:30 +02:00
commit cf256aa081
11315 files changed
+3598369
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/devel/migration/CPR.rst
+436
View File
@@ -0,0 +1,436 @@
CheckPoint and Restart (CPR)
============================
CPR is the umbrella name for a set of migration modes in which the
VM is migrated to a new QEMU instance on the same host. It is
intended for use when the goal is to update host software components
that run the VM, such as QEMU or even the host kernel. At this time,
the cpr-reboot, cpr-transfer, and cpr-exec modes are available.
Because QEMU is restarted on the same host, with access to the same
local devices, CPR is allowed in certain cases where normal migration
would be blocked. However, the user must not modify the contents of
guest block devices between quitting old QEMU and starting new QEMU.
CPR unconditionally stops VM execution before memory is saved, and
thus does not depend on any form of dirty page tracking.
cpr-reboot mode
---------------
In this mode, QEMU stops the VM, and writes VM state to the migration
URI, which will typically be a file. After quitting QEMU, the user
resumes by running QEMU with the ``-incoming`` option. Because the
old and new QEMU instances are not active concurrently, the URI cannot
be a type that streams data from one instance to the other.
Guest RAM can be saved in place if backed by shared memory, or can be
copied to a file. The former is more efficient and is therefore
preferred.
After state and memory are saved, the user may update userland host
software before restarting QEMU and resuming the VM. Further, if
the RAM is backed by persistent shared memory, such as a DAX device,
then the user may reboot to a new host kernel before restarting QEMU.
This mode supports VFIO devices provided the user first puts the
guest in the suspended runstate, such as by issuing the
``guest-suspend-ram`` command to the QEMU guest agent. The agent
must be pre-installed in the guest, and the guest must support
suspend to RAM. Beware that suspension can take a few seconds, so
the user should poll to see the suspended state before proceeding
with the CPR operation.
Usage
^^^^^
It is recommended that guest RAM be backed with some type of shared
memory, such as ``memory-backend-file,share=on``, and that the
``x-ignore-shared`` capability be set. This combination allows memory
to be saved in place. Otherwise, after QEMU stops the VM, all guest
RAM is copied to the migration URI.
Outgoing:
* Set the migration mode parameter to ``cpr-reboot``.
* Set the ``x-ignore-shared`` capability if desired.
* Issue the ``migrate`` command. It is recommended the URI be a
``file`` type, but one can use other types such as ``exec``,
provided the command captures all the data from the outgoing side,
and provides all the data to the incoming side.
* Quit when QEMU reaches the postmigrate state.
Incoming:
* Start QEMU with the ``-incoming defer`` option.
* Set the migration mode parameter to ``cpr-reboot``.
* Set the ``x-ignore-shared`` capability if desired.
* Issue the ``migrate-incoming`` command.
* If the VM was running when the outgoing ``migrate`` command was
issued, then QEMU automatically resumes VM execution.
Example 1
^^^^^^^^^
::
# qemu-kvm -monitor stdio
-object memory-backend-file,id=ram0,size=4G,mem-path=/dev/dax0.0,align=2M,share=on -m 4G
...
(qemu) info status
VM status: running
(qemu) migrate_set_parameter mode cpr-reboot
(qemu) migrate_set_capability x-ignore-shared on
(qemu) migrate -d file:vm.state
(qemu) info status
VM status: paused (postmigrate)
(qemu) quit
### optionally update kernel and reboot
# systemctl kexec
kexec_core: Starting new kernel
...
# qemu-kvm ... -incoming defer
(qemu) info status
VM status: paused (inmigrate)
(qemu) migrate_set_parameter mode cpr-reboot
(qemu) migrate_set_capability x-ignore-shared on
(qemu) migrate_incoming file:vm.state
(qemu) info status
VM status: running
Example 2: VFIO
^^^^^^^^^^^^^^^
::
# qemu-kvm -monitor stdio
-object memory-backend-file,id=ram0,size=4G,mem-path=/dev/dax0.0,align=2M,share=on -m 4G
-device vfio-pci, ...
-chardev socket,id=qga0,path=qga.sock,server=on,wait=off
-device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0
...
(qemu) info status
VM status: running
# echo '{"execute":"guest-suspend-ram"}' | ncat --send-only -U qga.sock
(qemu) info status
VM status: paused (suspended)
(qemu) migrate_set_parameter mode cpr-reboot
(qemu) migrate_set_capability x-ignore-shared on
(qemu) migrate -d file:vm.state
(qemu) info status
VM status: paused (postmigrate)
(qemu) quit
### optionally update kernel and reboot
# systemctl kexec
kexec_core: Starting new kernel
...
# qemu-kvm ... -incoming defer
(qemu) info status
VM status: paused (inmigrate)
(qemu) migrate_set_parameter mode cpr-reboot
(qemu) migrate_set_capability x-ignore-shared on
(qemu) migrate_incoming file:vm.state
(qemu) info status
VM status: paused (suspended)
(qemu) system_wakeup
(qemu) info status
VM status: running
Caveats
^^^^^^^
cpr-reboot mode may not be used with postcopy, background-snapshot,
or COLO.
cpr-transfer mode
-----------------
This mode allows the user to transfer a guest to a new QEMU instance
on the same host with minimal guest pause time, by preserving guest
RAM in place, albeit with new virtual addresses in new QEMU. Devices
and their pinned memory pages are also preserved for VFIO and IOMMUFD.
The user starts new QEMU on the same host as old QEMU, with command-
line arguments to create the same machine, plus the ``-incoming``
option for the main migration channel, like normal live migration.
In addition, the user adds a second -incoming option with channel
type ``cpr``. This CPR channel must support file descriptor transfer
with SCM_RIGHTS, i.e. it must be a UNIX domain socket.
To initiate CPR, the user issues a migrate command to old QEMU,
adding a second migration channel of type ``cpr`` in the channels
argument. Old QEMU stops the VM, saves state to the migration
channels, and enters the postmigrate state. Execution resumes in
new QEMU.
New QEMU reads the CPR channel before opening a monitor, hence
the CPR channel cannot be specified in the list of channels for a
migrate-incoming command. It may only be specified on the command
line.
Usage
^^^^^
Memory backend objects must have the ``share=on`` attribute.
The VM must be started with the ``-machine aux-ram-share=on``
option. This causes implicit RAM blocks (those not described by
a memory-backend object) to be allocated by mmap'ing a memfd.
Examples include VGA and ROM.
Outgoing:
* Set the migration mode parameter to ``cpr-transfer``.
* Issue the ``migrate`` command, containing a main channel and
a cpr channel.
Incoming:
* Start new QEMU with two ``-incoming`` options.
* If the VM was running when the outgoing ``migrate`` command was
issued, then QEMU automatically resumes VM execution.
Caveats
^^^^^^^
cpr-transfer mode may not be used with postcopy, background-snapshot,
or COLO.
memory-backend-epc is not supported.
The main incoming migration channel address cannot be a file type.
If the main incoming channel address is an inet socket, then the port
cannot be 0 (meaning dynamically choose a port).
When using ``-incoming defer``, you must issue the migrate command to
old QEMU before issuing any monitor commands to new QEMU, because new
QEMU blocks waiting to read from the cpr channel before starting its
monitor, and old QEMU does not write to the channel until the migrate
command is issued. However, new QEMU does not open and read the
main migration channel until you issue the migrate incoming command.
Example 1: incoming channel
^^^^^^^^^^^^^^^^^^^^^^^^^^^
In these examples, we simply restart the same version of QEMU, but
in a real scenario one would start new QEMU on the incoming side.
Note that new QEMU does not print the monitor prompt until old QEMU
has issued the migrate command. The outgoing side uses QMP because
HMP cannot specify a CPR channel. Some QMP responses are omitted for
brevity.
::
Outgoing: Incoming:
# qemu-kvm -qmp stdio
-object memory-backend-file,id=ram0,size=4G,
mem-path=/dev/shm/ram0,share=on -m 4G
-machine memory-backend=ram0
-machine aux-ram-share=on
...
# qemu-kvm -monitor stdio
-incoming tcp:0:44444
-incoming '{"channel-type": "cpr",
"addr": { "transport": "socket",
"type": "unix", "path": "cpr.sock"}}'
...
{"execute":"qmp_capabilities"}
{"execute": "query-status"}
{"return": {"status": "running",
"running": true}}
{"execute":"migrate-set-parameters",
"arguments":{"mode":"cpr-transfer"}}
{"execute": "migrate", "arguments": { "channels": [
{"channel-type": "main",
"addr": { "transport": "socket", "type": "inet",
"host": "0", "port": "44444" }},
{"channel-type": "cpr",
"addr": { "transport": "socket", "type": "unix",
"path": "cpr.sock" }}]}}
QEMU 10.0.50 monitor
(qemu) info status
VM status: running
{"execute": "query-status"}
{"return": {"status": "postmigrate",
"running": false}}
Example 2: incoming defer
^^^^^^^^^^^^^^^^^^^^^^^^^
This example uses ``-incoming defer`` to hot plug a device before
accepting the main migration channel. Again note you must issue the
migrate command to old QEMU before you can issue any monitor
commands to new QEMU.
::
Outgoing: Incoming:
# qemu-kvm -monitor stdio
-object memory-backend-file,id=ram0,size=4G,
mem-path=/dev/shm/ram0,share=on -m 4G
-machine memory-backend=ram0
-machine aux-ram-share=on
...
# qemu-kvm -monitor stdio
-incoming defer
-incoming '{"channel-type": "cpr",
"addr": { "transport": "socket",
"type": "unix", "path": "cpr.sock"}}'
...
{"execute":"qmp_capabilities"}
{"execute": "device_add",
"arguments": {"driver": "pcie-root-port"}}
{"execute":"migrate-set-parameters",
"arguments":{"mode":"cpr-transfer"}}
{"execute": "migrate", "arguments": { "channels": [
{"channel-type": "main",
"addr": { "transport": "socket", "type": "inet",
"host": "0", "port": "44444" }},
{"channel-type": "cpr",
"addr": { "transport": "socket", "type": "unix",
"path": "cpr.sock" }}]}}
QEMU 10.0.50 monitor
(qemu) info status
VM status: paused (inmigrate)
(qemu) device_add pcie-root-port
(qemu) migrate_incoming tcp:0:44444
(qemu) info status
VM status: running
{"execute": "query-status"}
{"return": {"status": "postmigrate",
"running": false}}
Futures
^^^^^^^
cpr-transfer mode is based on a capability to transfer open file
descriptors from old to new QEMU. In the future, descriptors for
vhost, and char devices could be transferred,
preserving those devices and their kernel state without interruption,
even if they do not explicitly support live migration.
cpr-exec mode
-------------
In this mode, QEMU stops the VM, writes VM state to the migration
URI, and directly exec's a new version of QEMU on the same host,
replacing the original process while retaining its PID. Guest RAM is
preserved in place, albeit with new virtual addresses. The user
completes the migration by specifying the ``-incoming`` option, and
by issuing the ``migrate-incoming`` command if necessary; see details
below.
This mode supports VFIO/IOMMUFD devices by preserving device
descriptors and hence kernel state across the exec, even for devices
that do not support live migration.
Because the old and new QEMU instances are not active concurrently,
the URI cannot be a type that streams data from one instance to the
other.
This mode does not require a channel of type ``cpr``. The information
that is passed over that channel for cpr-transfer mode is instead
serialized to a memfd, the number of the fd is saved in the
QEMU_CPR_EXEC_STATE environment variable during the exec of new QEMU.
and new QEMU mmaps the memfd.
Usage
^^^^^
Arguments for the new QEMU process are taken from the
@cpr-exec-command parameter. The first argument should be the
path of a new QEMU binary, or a prefix command that exec's the
new QEMU binary, and the arguments should include the ''-incoming''
option.
Memory backend objects must have the ``share=on`` attribute.
The VM must be started with the ``-machine aux-ram-share=on`` option.
Outgoing:
* Set the migration mode parameter to ``cpr-exec``.
* Set the ``cpr-exec-command`` parameter.
* Issue the ``migrate`` command. It is recommended that the URI be
a ``file`` type, but one can use other types such as ``exec``,
provided the command captures all the data from the outgoing side,
and provides all the data to the incoming side.
Incoming:
* You do not need to explicitly start new QEMU. It is started as
a side effect of the migrate command above.
* If the VM was running when the outgoing ``migrate`` command was
issued, then QEMU automatically resumes VM execution.
Example 1: incoming URI
^^^^^^^^^^^^^^^^^^^^^^^
In these examples, we simply restart the same version of QEMU, but in
a real scenario one would set a new QEMU binary path in
cpr-exec-command.
::
# qemu-kvm -monitor stdio
-object memory-backend-memfd,id=ram0,size=4G
-machine memory-backend=ram0
-machine aux-ram-share=on
...
QEMU 10.2.50 monitor - type 'help' for more information
(qemu) info status
VM status: running
(qemu) migrate_set_parameter mode cpr-exec
(qemu) migrate_set_parameter cpr-exec-command qemu-kvm ... -incoming file:vm.state
(qemu) migrate -d file:vm.state
(qemu) QEMU 10.2.50 monitor - type 'help' for more information
(qemu) info status
VM status: running
Example 2: incoming defer
^^^^^^^^^^^^^^^^^^^^^^^^^
::
# qemu-kvm -monitor stdio
-object memory-backend-memfd,id=ram0,size=4G
-machine memory-backend=ram0
-machine aux-ram-share=on
...
QEMU 10.2.50 monitor - type 'help' for more information
(qemu) info status
VM status: running
(qemu) migrate_set_parameter mode cpr-exec
(qemu) migrate_set_parameter cpr-exec-command qemu-kvm ... -incoming defer
(qemu) migrate -d file:vm.state
(qemu) QEMU 10.2.50 monitor - type 'help' for more information
(qemu) info status
status: paused (inmigrate)
(qemu) migrate_incoming file:vm.state
(qemu) info status
VM status: running
Caveats
^^^^^^^
cpr-exec mode may not be used with postcopy, background-snapshot,
or COLO.
cpr-exec mode requires permission to use the exec system call, which
is denied by certain sandbox options, such as spawn.
The guest pause time increases for large guest RAM backed by small pages.
docs/devel/migration/best-practices.rst
+48
View File
@@ -0,0 +1,48 @@
==============
Best practices
==============
Debugging
=========
The migration stream can be analyzed thanks to ``scripts/analyze-migration.py``.
Example usage:
.. code-block:: shell
$ qemu-system-x86_64 -display none -monitor stdio
(qemu) migrate "exec:cat > mig"
(qemu) q
$ ./scripts/analyze-migration.py -f mig
{
"ram (3)": {
"section sizes": {
"pc.ram": "0x0000000008000000",
...
See also ``analyze-migration.py -h`` help for more options.
Firmware
========
Migration migrates the copies of RAM and ROM, and thus when running
on the destination it includes the firmware from the source. Even after
resetting a VM, the old firmware is used. Only once QEMU has been restarted
is the new firmware in use.
- Changes in firmware size can cause changes in the required RAMBlock size
to hold the firmware and thus migration can fail. In practice it's best
to pad firmware images to convenient powers of 2 with plenty of space
for growth.
- Care should be taken with device emulation code so that newer
emulation code can work with older firmware to allow forward migration.
- Care should be taken with newer firmware so that backward migration
to older systems with older device emulation code will work.
In some cases it may be best to tie specific firmware versions to specific
versioned machine types to cut down on the combinations that will need
support. This is also useful when newer versions of firmware outgrow
the padding.
docs/devel/migration/compatibility.rst
+516
View File
@@ -0,0 +1,516 @@
Backwards compatibility
=======================
How backwards compatibility works
---------------------------------
When we do migration, we have two QEMU processes: the source and the
target. There are two cases, they are the same version or they are
different versions. The easy case is when they are the same version.
The difficult one is when they are different versions.
There are two things that are different, but they have very similar
names and sometimes get confused:
- QEMU version
- machine type version
Let's start with a practical example, we start with:
- qemu-system-x86_64 (v5.2), from now on qemu-5.2.
- qemu-system-x86_64 (v5.1), from now on qemu-5.1.
Related to this are the "latest" machine types defined on each of
them:
- pc-q35-5.2 (newer one in qemu-5.2) from now on pc-5.2
- pc-q35-5.1 (newer one in qemu-5.1) from now on pc-5.1
First of all, migration is only supposed to work if you use the same
machine type in both source and destination. The QEMU hardware
configuration needs to be the same also on source and destination.
Most aspects of the backend configuration can be changed at will,
except for a few cases where the backend features influence frontend
device feature exposure. But that is not relevant for this section.
I am going to list the number of combinations that we can have. Let's
start with the trivial ones, QEMU is the same on source and
destination:
1 - qemu-5.2 -M pc-5.2 -> migrates to -> qemu-5.2 -M pc-5.2
This is the latest QEMU with the latest machine type.
This have to work, and if it doesn't work it is a bug.
2 - qemu-5.1 -M pc-5.1 -> migrates to -> qemu-5.1 -M pc-5.1
Exactly the same case than the previous one, but for 5.1.
Nothing to see here either.
This are the easiest ones, we will not talk more about them in this
section.
Now we start with the more interesting cases. Consider the case where
we have the same QEMU version in both sides (qemu-5.2) but we are using
the latest machine type for that version (pc-5.2) but one of an older
QEMU version, in this case pc-5.1.
3 - qemu-5.2 -M pc-5.1 -> migrates to -> qemu-5.2 -M pc-5.1
It needs to use the definition of pc-5.1 and the devices as they
were configured on 5.1, but this should be easy in the sense that
both sides are the same QEMU and both sides have exactly the same
idea of what the pc-5.1 machine is.
4 - qemu-5.1 -M pc-5.2 -> migrates to -> qemu-5.1 -M pc-5.2
This combination is not possible as the qemu-5.1 doesn't understand
pc-5.2 machine type. So nothing to worry here.
Now it comes the interesting ones, when both QEMU processes are
different. Notice also that the machine type needs to be pc-5.1,
because we have the limitation than qemu-5.1 doesn't know pc-5.2. So
the possible cases are:
5 - qemu-5.2 -M pc-5.1 -> migrates to -> qemu-5.1 -M pc-5.1
This migration is known as newer to older. We need to make sure
when we are developing 5.2 we need to take care about not to break
migration to qemu-5.1. Notice that we can't make updates to
qemu-5.1 to understand whatever qemu-5.2 decides to change, so it is
in qemu-5.2 side to make the relevant changes.
6 - qemu-5.1 -M pc-5.1 -> migrates to -> qemu-5.2 -M pc-5.1
This migration is known as older to newer. We need to make sure
than we are able to receive migrations from qemu-5.1. The problem is
similar to the previous one.
If qemu-5.1 and qemu-5.2 were the same, there will not be any
compatibility problems. But the reason that we create qemu-5.2 is to
get new features, devices, defaults, etc.
If we get a device that has a new feature, or change a default value,
we have a problem when we try to migrate between different QEMU
versions.
So we need a way to tell qemu-5.2 that when we are using machine type
pc-5.1, it needs to **not** use the feature, to be able to migrate to
real qemu-5.1.
And the equivalent part when migrating from qemu-5.1 to qemu-5.2.
qemu-5.2 has to expect that it is not going to get data for the new
feature, because qemu-5.1 doesn't know about it.
How do we tell QEMU about these device feature changes? In
hw/core/machine.c:hw_compat_X_Y arrays.
If we change a default value, we need to put back the old value on
that array. And the device, during initialization needs to look at
that array to see what value it needs to get for that feature. And
what are we going to put in that array, the value of a property.
To create a property for a device, we need to use one of the
DEFINE_PROP_*() macros. See include/hw/core/qdev-properties.h to find the
macros that exist. With it, we set the default value for that
property, and that is what it is going to get in the latest released
version. But if we want a different value for a previous version, we
can change that in the hw_compat_X_Y arrays.
hw_compat_X_Y is an array of registers that have the format:
- name_device
- name_property
- value
Let's see a practical example.
In qemu-5.2 virtio-blk-device got multi queue support. This is a
change that is not backward compatible. In qemu-5.1 it has one
queue. In qemu-5.2 it has the same number of queues as the number of
cpus in the system.
When we are doing migration, if we migrate from a device that has 4
queues to a device that have only one queue, we don't know where to
put the extra information for the other 3 queues, and we fail
migration.
Similar problem when we migrate from qemu-5.1 that has only one queue
to qemu-5.2, we only sent information for one queue, but destination
has 4, and we have 3 queues that are not properly initialized and
anything can happen.
So, how can we address this problem. Easy, just convince qemu-5.2
that when it is running pc-5.1, it needs to set the number of queues
for virtio-blk-devices to 1.
That way we fix the cases 5 and 6.
5 - qemu-5.2 -M pc-5.1 -> migrates to -> qemu-5.1 -M pc-5.1
qemu-5.2 -M pc-5.1 sets number of queues to be 1.
qemu-5.1 -M pc-5.1 expects number of queues to be 1.
correct. migration works.
6 - qemu-5.1 -M pc-5.1 -> migrates to -> qemu-5.2 -M pc-5.1
qemu-5.1 -M pc-5.1 sets number of queues to be 1.
qemu-5.2 -M pc-5.1 expects number of queues to be 1.
correct. migration works.
And now the other interesting case, case 3. In this case we have:
3 - qemu-5.2 -M pc-5.1 -> migrates to -> qemu-5.2 -M pc-5.1
Here we have the same QEMU in both sides. So it doesn't matter a
lot if we have set the number of queues to 1 or not, because
they are the same.
WRONG!
Think what happens if we do one of this double migrations:
A -> migrates -> B -> migrates -> C
where:
A: qemu-5.1 -M pc-5.1
B: qemu-5.2 -M pc-5.1
C: qemu-5.2 -M pc-5.1
migration A -> B is case 6, so number of queues needs to be 1.
migration B -> C is case 3, so we don't care. But actually we
care because we haven't started the guest in qemu-5.2, it came
migrated from qemu-5.1. So to be in the safe place, we need to
always use number of queues 1 when we are using pc-5.1.
Now, how was this done in reality? The following commit shows how it
was done::
commit 9445e1e15e66c19e42bea942ba810db28052cd05
Author: Stefan Hajnoczi <[email protected]>
Date: Tue Aug 18 15:33:47 2020 +0100
virtio-blk-pci: default num_queues to -smp N
The relevant parts for migration are::
@@ -1281,7 +1284,8 @@ static const Property virtio_blk_properties[] = {
#endif
DEFINE_PROP_BIT("request-merging", VirtIOBlock, conf.request_merging, 0,
true),
- DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues, 1),
+ DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues,
+ VIRTIO_BLK_AUTO_NUM_QUEUES),
DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size, 256),
It changes the default value of num_queues. But it fishes it for old
machine types to have the right value::
@@ -31,6 +31,7 @@
GlobalProperty hw_compat_5_1[] = {
...
+ { "virtio-blk-device", "num-queues", "1"},
...
};
A device with different features on both sides
----------------------------------------------
Let's assume that we are using the same QEMU binary on both sides,
just to make the things easier. But we have a device that has
different features on both sides of the migration. That can be
because the devices are different, because the kernel driver of both
devices have different features, whatever.
How can we get this to work with migration. The way to do that is
"theoretically" easy. You have to get the features that the device
has in the source of the migration. The features that the device has
on the target of the migration, you get the intersection of the
features of both sides, and that is the way that you should launch
QEMU.
Notice that this is not completely related to QEMU. The most
important thing here is that this should be handled by the managing
application that launches QEMU. If QEMU is configured correctly, the
migration will succeed.
That said, actually doing it is complicated. Almost all devices are
bad at being able to be launched with only some features enabled.
With one big exception: cpus.
You can read the documentation for QEMU x86 cpu models here:
https://qemu-project.gitlab.io/qemu/system/qemu-cpu-models.html
See when they talk about migration they recommend that one chooses the
newest cpu model that is supported for all cpus.
Let's say that we have:
Host A:
Device X has the feature Y
Host B:
Device X has not the feature Y
If we try to migrate without any care from host A to host B, it will
fail because when migration tries to load the feature Y on
destination, it will find that the hardware is not there.
Doing this would be the equivalent of doing with cpus:
Host A:
$ qemu-system-x86_64 -cpu host
Host B:
$ qemu-system-x86_64 -cpu host
When both hosts have different cpu features this is guaranteed to
fail. Especially if Host B has less features than host A. If host A
has less features than host B, sometimes it works. Important word of
last sentence is "sometimes".
So, forgetting about cpu models and continuing with the -cpu host
example, let's see that the differences of the cpus is that Host A and
B have the following features:
Features: 'pcid' 'stibp' 'taa-no'
Host A: X X
Host B: X
And we want to migrate between them, the way configure both QEMU cpu
will be:
Host A:
$ qemu-system-x86_64 -cpu host,pcid=off,stibp=off
Host B:
$ qemu-system-x86_64 -cpu host,taa-no=off
And you would be able to migrate between them. It is responsibility
of the management application or of the user to make sure that the
configuration is correct. QEMU doesn't know how to look at this kind
of features in general.
Notice that we don't recommend to use -cpu host for migration. It is
used in this example because it makes the example simpler.
Other devices have worse control about individual features. If they
want to be able to migrate between hosts that show different features,
the device needs a way to configure which ones it is going to use.
In this section we have considered that we are using the same QEMU
binary in both sides of the migration. If we use different QEMU
versions process, then we need to have into account all other
differences and the examples become even more complicated.
How to mitigate when we have a backward compatibility error
-----------------------------------------------------------
We broke migration for old machine types continuously during
development. But as soon as we find that there is a problem, we fix
it. The problem is what happens when we detect after we have done a
release that something has gone wrong.
Let see how it worked with one example.
After the release of qemu-8.0 we found a problem when doing migration
of the machine type pc-7.2.
- $ qemu-7.2 -M pc-7.2 -> qemu-7.2 -M pc-7.2
This migration works
- $ qemu-8.0 -M pc-7.2 -> qemu-8.0 -M pc-7.2
This migration works
- $ qemu-8.0 -M pc-7.2 -> qemu-7.2 -M pc-7.2
This migration fails
- $ qemu-7.2 -M pc-7.2 -> qemu-8.0 -M pc-7.2
This migration fails
So clearly something fails when migration between qemu-7.2 and
qemu-8.0 with machine type pc-7.2. The error messages, and git bisect
pointed to this commit.
In qemu-8.0 we got this commit::
commit 010746ae1db7f52700cb2e2c46eb94f299cfa0d2
Author: Jonathan Cameron <[email protected]>
Date: Thu Mar 2 13:37:02 2023 +0000
hw/pci/aer: Implement PCI_ERR_UNCOR_MASK register
The relevant bits of the commit for our example are this ones::
--- a/hw/pci/pcie_aer.c
+++ b/hw/pci/pcie_aer.c
@@ -112,6 +112,10 @@ int pcie_aer_init(PCIDevice *dev,
pci_set_long(dev->w1cmask + offset + PCI_ERR_UNCOR_STATUS,
PCI_ERR_UNC_SUPPORTED);
+ pci_set_long(dev->config + offset + PCI_ERR_UNCOR_MASK,
+ PCI_ERR_UNC_MASK_DEFAULT);
+ pci_set_long(dev->wmask + offset + PCI_ERR_UNCOR_MASK,
+ PCI_ERR_UNC_SUPPORTED);
pci_set_long(dev->config + offset + PCI_ERR_UNCOR_SEVER,
PCI_ERR_UNC_SEVERITY_DEFAULT);
The patch changes how we configure PCI space for AER. But QEMU fails
when the PCI space configuration is different between source and
destination.
The following commit shows how this got fixed::
commit 5ed3dabe57dd9f4c007404345e5f5bf0e347317f
Author: Leonardo Bras <[email protected]>
Date: Tue May 2 21:27:02 2023 -0300
hw/pci: Disable PCI_ERR_UNCOR_MASK register for machine type < 8.0
[...]
The relevant parts of the fix in QEMU are as follow:
First, we create a new property for the device to be able to configure
the old behaviour or the new behaviour::
diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index 8a87ccc8b0..5153ad63d6 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -79,6 +79,8 @@ static const Property pci_props[] = {
DEFINE_PROP_STRING("failover_pair_id", PCIDevice,
failover_pair_id),
DEFINE_PROP_UINT32("acpi-index", PCIDevice, acpi_index, 0),
+ DEFINE_PROP_BIT("x-pcie-err-unc-mask", PCIDevice, cap_present,
+ QEMU_PCIE_ERR_UNC_MASK_BITNR, true),
};
Notice that we enable the feature for new machine types.
Now we see how the fix is done. This is going to depend on what kind
of breakage happens, but in this case it is quite simple::
diff --git a/hw/pci/pcie_aer.c b/hw/pci/pcie_aer.c
index 103667c368..374d593ead 100644
--- a/hw/pci/pcie_aer.c
+++ b/hw/pci/pcie_aer.c
@@ -112,10 +112,13 @@ int pcie_aer_init(PCIDevice *dev, uint8_t cap_ver,
uint16_t offset,
pci_set_long(dev->w1cmask + offset + PCI_ERR_UNCOR_STATUS,
PCI_ERR_UNC_SUPPORTED);
- pci_set_long(dev->config + offset + PCI_ERR_UNCOR_MASK,
- PCI_ERR_UNC_MASK_DEFAULT);
- pci_set_long(dev->wmask + offset + PCI_ERR_UNCOR_MASK,
- PCI_ERR_UNC_SUPPORTED);
+
+ if (dev->cap_present & QEMU_PCIE_ERR_UNC_MASK) {
+ pci_set_long(dev->config + offset + PCI_ERR_UNCOR_MASK,
+ PCI_ERR_UNC_MASK_DEFAULT);
+ pci_set_long(dev->wmask + offset + PCI_ERR_UNCOR_MASK,
+ PCI_ERR_UNC_SUPPORTED);
+ }
pci_set_long(dev->config + offset + PCI_ERR_UNCOR_SEVER,
PCI_ERR_UNC_SEVERITY_DEFAULT);
I.e. If the property bit is enabled, we configure it as we did for
qemu-8.0. If the property bit is not set, we configure it as it was in 7.2.
And now, everything that is missing is disabling the feature for old
machine types::
diff --git a/hw/core/machine.c b/hw/core/machine.c
index 47a34841a5..07f763eb2e 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -48,6 +48,7 @@ GlobalProperty hw_compat_7_2[] = {
{ "e1000e", "migrate-timadj", "off" },
{ "virtio-mem", "x-early-migration", "false" },
{ "migration", "x-preempt-pre-7-2", "true" },
+ { TYPE_PCI_DEVICE, "x-pcie-err-unc-mask", "off" },
};
const size_t hw_compat_7_2_len = G_N_ELEMENTS(hw_compat_7_2);
And now, when qemu-8.0.1 is released with this fix, all combinations
are going to work as supposed.
- $ qemu-7.2 -M pc-7.2 -> qemu-7.2 -M pc-7.2 (works)
- $ qemu-8.0.1 -M pc-7.2 -> qemu-8.0.1 -M pc-7.2 (works)
- $ qemu-8.0.1 -M pc-7.2 -> qemu-7.2 -M pc-7.2 (works)
- $ qemu-7.2 -M pc-7.2 -> qemu-8.0.1 -M pc-7.2 (works)
So the normality has been restored and everything is ok, no?
Not really, now our matrix is much bigger. We started with the easy
cases, migration from the same version to the same version always
works:
- $ qemu-7.2 -M pc-7.2 -> qemu-7.2 -M pc-7.2
- $ qemu-8.0 -M pc-7.2 -> qemu-8.0 -M pc-7.2
- $ qemu-8.0.1 -M pc-7.2 -> qemu-8.0.1 -M pc-7.2
Now the interesting ones. When the QEMU processes versions are
different. For the 1st set, their fail and we can do nothing, both
versions are released and we can't change anything.
- $ qemu-7.2 -M pc-7.2 -> qemu-8.0 -M pc-7.2
- $ qemu-8.0 -M pc-7.2 -> qemu-7.2 -M pc-7.2
This two are the ones that work. The whole point of making the
change in qemu-8.0.1 release was to fix this issue:
- $ qemu-7.2 -M pc-7.2 -> qemu-8.0.1 -M pc-7.2
- $ qemu-8.0.1 -M pc-7.2 -> qemu-7.2 -M pc-7.2
But now we found that qemu-8.0 neither can migrate to qemu-7.2 not
qemu-8.0.1.
- $ qemu-8.0 -M pc-7.2 -> qemu-8.0.1 -M pc-7.2
- $ qemu-8.0.1 -M pc-7.2 -> qemu-8.0 -M pc-7.2
So, if we start a pc-7.2 machine in qemu-8.0 we can't migrate it to
anything except to qemu-8.0.
Can we do better?
Yeap. If we know that we are going to do this migration:
- $ qemu-8.0 -M pc-7.2 -> qemu-8.0.1 -M pc-7.2
We can launch the appropriate devices with::
--device...,x-pci-e-err-unc-mask=on
And now we can receive a migration from 8.0. And from now on, we can
do that migration to new machine types if we remember to enable that
property for pc-7.2. Notice that we need to remember, it is not
enough to know that the source of the migration is qemu-8.0. Think of
this example:
$ qemu-8.0 -M pc-7.2 -> qemu-8.0.1 -M pc-7.2 -> qemu-8.2 -M pc-7.2
In the second migration, the source is not qemu-8.0, but we still have
that "problem" and have that property enabled. Notice that we need to
continue having this mark/property until we have this machine
rebooted. But it is not a normal reboot (that don't reload QEMU) we
need the machine to poweroff/poweron on a fixed QEMU. And from now
on we can use the proper real machine.
docs/devel/migration/dirty-limit.rst
+71
View File
@@ -0,0 +1,71 @@
Dirty limit
===========
The dirty limit, short for dirty page rate upper limit, is a new capability
introduced in the 8.1 QEMU release that uses a new algorithm based on the KVM
dirty ring to throttle down the guest during live migration.
The algorithm framework is as follows:
::
------------------------------------------------------------------------------
main --------------> throttle thread ------------> PREPARE(1) <--------
thread \ | |
\ | |
\ V |
-\ CALCULATE(2) |
\ | |
\ | |
\ V |
\ SET PENALTY(3) -----
-\ |
\ |
\ V
-> virtual CPU thread -------> ACCEPT PENALTY(4)
------------------------------------------------------------------------------
When the qmp command qmp_set_vcpu_dirty_limit is called for the first time,
the QEMU main thread starts the throttle thread. The throttle thread, once
launched, executes the loop, which consists of three steps:
- PREPARE (1)
The entire work of PREPARE (1) is preparation for the second stage,
CALCULATE(2), as the name implies. It involves preparing the dirty
page rate value and the corresponding upper limit of the VM:
The dirty page rate is calculated via the KVM dirty ring mechanism,
which tells QEMU how many dirty pages a virtual CPU has had since the
last KVM_EXIT_DIRTY_RING_FULL exception; The dirty page rate upper
limit is specified by caller, therefore fetch it directly.
- CALCULATE (2)
Calculate a suitable sleep period for each virtual CPU, which will be
used to determine the penalty for the target virtual CPU. The
computation must be done carefully in order to reduce the dirty page
rate progressively down to the upper limit without oscillation. To
achieve this, two strategies are provided: the first is to add or
subtract sleep time based on the ratio of the current dirty page rate
to the limit, which is used when the current dirty page rate is far
from the limit; the second is to add or subtract a fixed time when
the current dirty page rate is close to the limit.
- SET PENALTY (3)
Set the sleep time for each virtual CPU that should be penalized based
on the results of the calculation supplied by step CALCULATE (2).
After completing the three above stages, the throttle thread loops back
to step PREPARE (1) until the dirty limit is reached.
On the other hand, each virtual CPU thread reads the sleep duration and
sleeps in the path of the KVM_EXIT_DIRTY_RING_FULL exception handler, that
is ACCEPT PENALTY (4). Virtual CPUs tied with writing processes will
obviously exit to the path and get penalized, whereas virtual CPUs involved
with read processes will not.
In summary, thanks to the KVM dirty ring technology, the dirty limit
algorithm will restrict virtual CPUs as needed to keep their dirty page
rate inside the limit. This leads to more steady reading performance during
live migration and can aid in improving large guest responsiveness.
docs/devel/migration/fast-snapshot-load.rst
+82
View File
@@ -0,0 +1,82 @@
==================
Fast Snapshot Load
==================
Overview
========
Fast snapshot load is an extension of the postcopy migration feature
to disk loads.
Unlike a usual snapshot load, which requires all VM data (RAM as well
as device states) to be loaded into host RAM from the snapshot file
for the guest to run, fast snapshot load uses postcopy infrastructure
to load in only the required device states and allows RAM pages to
be loaded after the guest starts execution. The idea is to start the
guest and serve its page faults on the go, reducing the perceived
resume time for large snapshots.
Architecture
============
This feature combines postcopy migration and mapped-ram capabilities
to load RAM pages on demand. It is done by catching guest faults using
Linux ``userfaultfd`` and loading the page by calculating the offset
of its location in the snapshot file using mapped-ram capabilities.
Fault Thread
------------
The fault thread uses Linux ``userfaultfd`` to catch page faults caused
by guest and directly load the page from the snapshot file. It is
very similar to network postcopy fault thread, with primary difference
being it loads pages directly by reading from the snapshot file.
Eager Thread
------------
Eager thread iterates over all pages in RAM and loads each page not
yet loaded by fault thread. It is required as unlike network postcopy
where majority of RAM has already been loaded via precopy, here entire
RAM is waiting to be loaded. If there is no eager loading thread each
page will only be loaded when it is required by guest. In case there
are some background pages that are never/rarely accessed by guest,
the system will be locked in migration state indefinitely.
Synchronization
---------------
In order to make sure both of these threads do not load the same page
twice potentially overwriting and corrupting user RAM, a bitmap is
used (``RAMBlock->pending_bmap``) which tracks the pages claimed to
be loaded by threads. This prevents race condition when one thread
is loading the page and other one tries to do the same.
Usage
=====
Simply enable ``mapped-ram`` and ``postcopy-ram`` capabilities on
the destination:
.. code-block:: text
migrate_set_capability mapped-ram on
migrate_set_capability postcopy-ram on
Use a ``file:`` URI for migration:
.. code-block:: text
migrate_incoming file:/path/to/snapshot/file
Limitations
===========
- Multifd
Fast snapshot load is currently incompatible with ``multifd``
capability. While ``mapped-ram`` allows for parallel disk I/O,
coupling it with ``postcopy`` capability requires additional
infrastructure.
- Host OS support
Because this feautre essentially depends on ``userfaultfd``
to trap page faults, it is supported only on Linux hosts.
- vhost-user
Fast snapshot load does not currently support ``vhost-user``
backends.
docs/devel/migration/features.rst
+19
View File
@@ -0,0 +1,19 @@
Migration features
==================
Migration has plenty of features to support different use cases.
.. toctree::
:maxdepth: 2
postcopy
dirty-limit
vfio
virtio
mapped-ram
fast-snapshot-load
CPR
qpl-compression
uadk-compression
qatzip-compression
xbzrle
docs/devel/migration/index.rst
+13
View File
@@ -0,0 +1,13 @@
Migration
=========
This is the main entry for QEMU migration documentations. It explains how
QEMU live migration works.
.. toctree::
:maxdepth: 2
main
features
compatibility
best-practices
docs/devel/migration/main.rst
+618
View File
@@ -0,0 +1,618 @@
.. _migration:
===================
Migration framework
===================
QEMU has code to load/save the state of the guest that it is running.
These are two complementary operations. Saving the state just does
that, saves the state for each device that the guest is running.
Restoring a guest is just the opposite operation: we need to load the
state of each device.
For this to work, QEMU has to be launched with the same arguments the
two times. I.e. it can only restore the state in one guest that has
the same devices that the one it was saved (this last requirement can
be relaxed a bit, but for now we can consider that configuration has
to be exactly the same).
Once that we are able to save/restore a guest, a new functionality is
requested: migration. This means that QEMU is able to start in one
machine and being "migrated" to another machine. I.e. being moved to
another machine.
Next was the "live migration" functionality. This is important
because some guests run with a lot of state (specially RAM), and it
can take a while to move all state from one machine to another. Live
migration allows the guest to continue running while the state is
transferred. Only while the last part of the state is transferred has
the guest to be stopped. Typically the time that the guest is
unresponsive during live migration is the low hundred of milliseconds
(notice that this depends on a lot of things).
.. contents::
Transports
==========
The migration stream is normally just a byte stream that can be passed
over any transport.
- tcp migration: do the migration using tcp sockets
- unix migration: do the migration using unix sockets
- exec migration: do the migration using the stdin/stdout through a process.
- fd migration: do the migration using a file descriptor that is
passed to QEMU. QEMU doesn't care how this file descriptor is opened.
- file migration: do the migration using a file that is passed to QEMU
by path. A file offset option is supported to allow a management
application to add its own metadata to the start of the file without
QEMU interference. Note that QEMU does not flush cached file
data/metadata at the end of migration.
The file migration also supports using a file that has already been
opened. A set of file descriptors is passed to QEMU via an "fdset"
(see add-fd QMP command documentation). This method allows a
management application to have control over the migration file
opening operation. There are, however, strict requirements to this
interface if the multifd capability is enabled:
- the fdset must contain two file descriptors that are not
duplicates between themselves;
- if the direct-io capability is to be used, exactly one of the
file descriptors must have the O_DIRECT flag set;
- the file must be opened with WRONLY on the migration source side
and RDONLY on the migration destination side.
- rdma migration: support is included for migration using RDMA, which
transports the page data using ``RDMA``, where the hardware takes
care of transporting the pages, and the load on the CPU is much
lower. While the internals of RDMA migration are a bit different,
this isn't really visible outside the RAM migration code.
All these migration protocols use the same infrastructure to
save/restore state devices. This infrastructure is shared with the
savevm/loadvm functionality.
Common infrastructure
=====================
The files, sockets or fd's that carry the migration stream are abstracted by
the ``QEMUFile`` type (see ``migration/qemu-file.h``). In most cases this
is connected to a subtype of ``QIOChannel`` (see ``io/``).
Saving the state of one device
==============================
For most devices, the state is saved in a single call to the migration
infrastructure; these are *non-iterative* devices. The data for these
devices is sent at the end of precopy migration, when the CPUs are paused.
There are also *iterative* devices, which contain a very large amount of
data (e.g. RAM or large tables). See the iterative device section below.
General advice for device developers
------------------------------------
- The migration state saved should reflect the device being modelled rather
than the way your implementation works. That way if you change the implementation
later the migration stream will stay compatible. That model may include
internal state that's not directly visible in a register.
- When saving a migration stream the device code may walk and check
the state of the device. These checks might fail in various ways (e.g.
discovering internal state is corrupt or that the guest has done something bad).
Consider carefully before asserting/aborting at this point, since the
normal response from users is that *migration broke their VM* since it had
apparently been running fine until then. In these error cases, the device
should log a message indicating the cause of error, and should consider
putting the device into an error state, allowing the rest of the VM to
continue execution.
- The migration might happen at an inconvenient point,
e.g. right in the middle of the guest reprogramming the device, during
guest reboot or shutdown or while the device is waiting for external IO.
It's strongly preferred that migrations do not fail in this situation,
since in the cloud environment migrations might happen automatically to
VMs that the administrator doesn't directly control.
- If you do need to fail a migration, ensure that sufficient information
is logged to identify what went wrong.
- The destination should treat an incoming migration stream as hostile
(which we do to varying degrees in the existing code). Check that offsets
into buffers and the like can't cause overruns. Fail the incoming migration
in the case of a corrupted stream like this.
- Take care with internal device state or behaviour that might become
migration version dependent. For example, the order of PCI capabilities
is required to stay constant across migration. Another example would
be that a special case handled by subsections (see below) might become
much more common if a default behaviour is changed.
- The state of the source should not be changed or destroyed by the
outgoing migration. Migrations timing out or being failed by
higher levels of management, or failures of the destination host are
not unusual, and in that case the VM is restarted on the source.
Note that the management layer can validly revert the migration
even though the QEMU level of migration has succeeded as long as it
does it before starting execution on the destination.
- Buses and devices should be able to explicitly specify addresses when
instantiated, and management tools should use those. For example,
when hot adding USB devices it's important to specify the ports
and addresses, since implicit ordering based on the command line order
may be different on the destination. This can result in the
device state being loaded into the wrong device.
VMState
-------
Most device data can be described using the ``VMSTATE`` macros (mostly defined
in ``include/migration/vmstate.h``).
An example (from hw/input/pckbd.c)
.. code:: c
static const VMStateDescription vmstate_kbd = {
.name = "pckbd",
.version_id = 3,
.minimum_version_id = 3,
.fields = (const VMStateField[]) {
VMSTATE_UINT8(write_cmd, KBDState),
VMSTATE_UINT8(status, KBDState),
VMSTATE_UINT8(mode, KBDState),
VMSTATE_UINT8(pending, KBDState),
VMSTATE_END_OF_LIST()
}
};
We are declaring the state with name "pckbd". The ``version_id`` is
3, and there are 4 uint8_t fields in the KBDState structure. We
registered this ``VMSTATEDescription`` with one of the following
functions. The first one will generate a device ``instance_id``
different for each registration. Use the second one if you already
have an id that is different for each instance of the device:
.. code:: c
vmstate_register_any(NULL, &vmstate_kbd, s);
vmstate_register(NULL, instance_id, &vmstate_kbd, s);
For devices that are ``qdev`` based, we can register the device in the class
init function:
.. code:: c
dc->vmsd = &vmstate_kbd_isa;
The VMState macros take care of ensuring that the device data section
is formatted portably (normally big endian) and make some compile time checks
against the types of the fields in the structures.
VMState macros can include other VMStateDescriptions to store substructures
(see ``VMSTATE_STRUCT_``), arrays (``VMSTATE_ARRAY_``) and variable length
arrays (``VMSTATE_VARRAY_``). Various other macros exist for special
cases.
Note that the format on the wire is still very raw; i.e. a VMSTATE_UINT32
ends up with a 4 byte bigendian representation on the wire; in the future
it might be possible to use a more structured format.
Legacy way
----------
This way is going to disappear as soon as all current users are ported to VMSTATE;
although converting existing code can be tricky, and thus 'soon' is relative.
Each device has to register two functions, one to save the state and
another to load the state back.
.. code:: c
int register_savevm_live(const char *idstr,
int instance_id,
int version_id,
SaveVMHandlers *ops,
void *opaque);
Two functions in the ``ops`` structure are the ``save_state``
and ``load_state`` functions. Notice that ``load_state`` receives a version_id
parameter to know what state format is receiving. ``save_state`` doesn't
have a version_id parameter because it always uses the latest version.
Note that because the VMState macros still save the data in a raw
format, in many cases it's possible to replace legacy code
with a carefully constructed VMState description that matches the
byte layout of the existing code.
Changing migration data structures
----------------------------------
When we migrate a device, we save/load the state as a series
of fields. Sometimes, due to bugs or new functionality, we need to
change the state to store more/different information. Changing the migration
state saved for a device can break migration compatibility unless
care is taken to use the appropriate techniques. In general QEMU tries
to maintain forward migration compatibility (i.e. migrating from
QEMU n->n+1) and there are users who benefit from backward compatibility
as well.
Subsections
-----------
The most common structure change is adding new data, e.g. when adding
a newer form of device, or adding that state that you previously
forgot to migrate. This is best solved using a subsection.
A subsection is "like" a device vmstate, but with a particularity, it
has a Boolean function that tells if that values are needed to be sent
or not. If this functions returns false, the subsection is not sent.
Subsections have a unique name, that is looked for on the receiving
side.
On the receiving side, if we found a subsection for a device that we
don't understand, we just fail the migration. If we understand all
the subsections, then we load the state with success. There's no check
that a subsection is loaded, so a newer QEMU that knows about a subsection
can (with care) load a stream from an older QEMU that didn't send
the subsection.
If the new data is only needed in a rare case, then the subsection
can be made conditional on that case and the migration will still
succeed to older QEMUs in most cases. This is OK for data that's
critical, but in some use cases it's preferred that the migration
should succeed even with the data missing. To support this the
subsection can be connected to a device property and from there
to a versioned machine type.
The 'pre_load' and 'post_load' functions on subsections are only
called if the subsection is loaded.
One important note is that the outer post_load() function is called "after"
loading all subsections, because a newer subsection could change the same
value that it uses. A flag, and the combination of outer pre_load and
post_load can be used to detect whether a subsection was loaded, and to
fall back on default behaviour when the subsection isn't present.
Example:
.. code:: c
static bool ide_drive_pio_state_needed(void *opaque)
{
IDEState *s = opaque;
return ((s->status & DRQ_STAT) != 0)
|| (s->bus->error_status & BM_STATUS_PIO_RETRY);
}
const VMStateDescription vmstate_ide_drive_pio_state = {
.name = "ide_drive/pio_state",
.version_id = 1,
.minimum_version_id = 1,
.pre_save = ide_drive_pio_pre_save,
.post_load = ide_drive_pio_post_load,
.needed = ide_drive_pio_state_needed,
.fields = (const VMStateField[]) {
VMSTATE_INT32(req_nb_sectors, IDEState),
VMSTATE_VARRAY_INT32(io_buffer, IDEState, io_buffer_total_len, 1,
vmstate_info_uint8, uint8_t),
VMSTATE_INT32(cur_io_buffer_offset, IDEState),
VMSTATE_INT32(cur_io_buffer_len, IDEState),
VMSTATE_UINT8(end_transfer_fn_idx, IDEState),
VMSTATE_INT32(elementary_transfer_size, IDEState),
VMSTATE_INT32(packet_transfer_size, IDEState),
VMSTATE_END_OF_LIST()
}
};
const VMStateDescription vmstate_ide_drive = {
.name = "ide_drive",
.version_id = 3,
.minimum_version_id = 0,
.post_load = ide_drive_post_load,
.fields = (const VMStateField[]) {
.... several fields ....
VMSTATE_END_OF_LIST()
},
.subsections = (const VMStateDescription * const []) {
&vmstate_ide_drive_pio_state,
NULL
}
};
Here we have a subsection for the pio state. We only need to
save/send this state when we are in the middle of a pio operation
(that is what ``ide_drive_pio_state_needed()`` checks). If DRQ_STAT is
not enabled, the values on that fields are garbage and don't need to
be sent.
Connecting subsections to properties
------------------------------------
Using a condition function that checks a 'property' to determine whether
to send a subsection allows backward migration compatibility when
new subsections are added, especially when combined with versioned
machine types.
For example:
a) Add a new property using ``DEFINE_PROP_BOOL`` - e.g. support-foo and
default it to true.
b) Add an entry to the ``hw_compat_`` for the previous version that sets
the property to false.
c) Add a static bool support_foo function that tests the property.
d) Add a subsection with a .needed set to the support_foo function
e) (potentially) Add an outer pre_load that sets up a default value
for 'foo' to be used if the subsection isn't loaded.
Now that subsection will not be generated when using an older
machine type and the migration stream will be accepted by older
QEMU versions.
Not sending existing elements
-----------------------------
Sometimes members of the VMState are no longer needed:
- removing them will break migration compatibility
- making them version dependent and bumping the version will break backward migration
compatibility.
Adding a dummy field into the migration stream is normally the best way to preserve
compatibility.
If the field really does need to be removed then:
a) Add a new property/compatibility/function in the same way for subsections above.
b) replace the VMSTATE macro with the _TEST version of the macro, e.g.:
``VMSTATE_UINT32(foo, barstruct)``
becomes
``VMSTATE_UINT32_TEST(foo, barstruct, pre_version_baz)``
Sometime in the future when we no longer care about the ancient versions these can be killed off.
Note that for backward compatibility it's important to fill in the structure with
data that the destination will understand.
Any difference in the predicates on the source and destination will end up
with different fields being enabled and data being loaded into the wrong
fields; for this reason conditional fields like this are very fragile.
Versions
--------
Version numbers are intended for major incompatible changes to the
migration of a device, and using them breaks backward-migration
compatibility; in general most changes can be made by adding Subsections
(see above) or _TEST macros (see above) which won't break compatibility.
Each version is associated with a series of fields saved. The ``save_state`` always saves
the state as the newer version. But ``load_state`` sometimes is able to
load state from an older version.
You can see that there are two version fields:
- ``version_id``: the maximum version_id supported by VMState for that device.
- ``minimum_version_id``: the minimum version_id that VMState is able to understand
for that device.
VMState is able to read versions from minimum_version_id to version_id.
There are *_V* forms of many ``VMSTATE_`` macros to load fields for version dependent fields,
e.g.
.. code:: c
VMSTATE_UINT16_V(ip_id, Slirp, 2),
only loads that field for versions 2 and newer.
Saving state will always create a section with the 'version_id' value
and thus can't be loaded by any older QEMU.
Massaging functions
-------------------
Sometimes, it is not enough to be able to save the state directly
from one structure, we need to fill the correct values there. One
example is when we are using kvm. Before saving the cpu state, we
need to ask kvm to copy to QEMU the state that it is using. And the
opposite when we are loading the state, we need a way to tell kvm to
load the state for the cpu that we have just loaded from the QEMUFile.
The functions to do that are inside a vmstate definition, and are called:
- ``int (*pre_load)(void *opaque);``
This function is called before we load the state of one device.
- ``int (*post_load)(void *opaque, int version_id);``
This function is called after we load the state of one device.
- ``int (*pre_save)(void *opaque);``
This function is called before we save the state of one device.
- ``void (*post_save)(void *opaque);``
This function is called after we save the state of one device
(even upon failure, unless the call to pre_save returned an error).
Following are the errp variants of these functions.
- ``bool (*pre_load_errp)(void *opaque, Error **errp);``
This function is called before we load the state of one device.
- ``bool (*post_load_errp)(void *opaque, int version_id, Error **errp);``
This function is called after we load the state of one device.
- ``bool (*pre_save_errp)(void *opaque, Error **errp);``
This function is called before we save the state of one device.
New impls should preferentally use 'errp' variants of these
methods and existing impls incrementally converted.
The variants without 'errp' are intended to be removed
once all usage is converted.
Example: You can look at hpet.c, that uses the first three functions
to massage the state that is transferred.
The ``VMSTATE_WITH_TMP`` macro may be useful when the migration
data doesn't match the stored device data well; it allows an
intermediate temporary structure to be populated with migration
data and then transferred to the main structure.
If you use memory or portio_list API functions that update memory layout outside
initialization (i.e., in response to a guest action), this is a strong
indication that you need to call these functions in a ``post_load`` callback.
Examples of such API functions are:
- memory_region_add_subregion()
- memory_region_del_subregion()
- memory_region_set_readonly()
- memory_region_set_nonvolatile()
- memory_region_set_enabled()
- memory_region_set_address()
- memory_region_set_alias_offset()
- portio_list_set_address()
- portio_list_set_enabled()
Since the order of device save/restore is not defined, you must
avoid accessing or changing any other device's state in one of these
callbacks. (For instance, don't do anything that calls ``update_irq()``
in a ``post_load`` hook.) Otherwise, restore will not be deterministic,
and this will break execution record/replay.
Iterative device migration
--------------------------
Some devices, such as RAM or certain platform devices,
have large amounts of data that would mean that the CPUs would be
paused for too long if they were sent in one section. For these
devices an *iterative* approach is taken.
The iterative devices generally don't use VMState macros
(although it may be possible in some cases) and instead use
qemu_put_*/qemu_get_* macros to read/write data to the stream. Specialist
versions exist for high bandwidth IO.
An iterative device must provide:
- A ``save_setup`` function that initialises the data structures and
transmits a first section containing information on the device. In the
case of RAM this transmits a list of RAMBlocks and sizes.
- A ``load_setup`` function that initialises the data structures on the
destination.
- A ``save_query_pending`` function that indicates how much more
data we must save.
- A ``save_live_iterate`` function should send a chunk of data until
the point that stream bandwidth limits tell it to stop. Each call
generates one section.
- A ``save_complete`` function that must transmit the last section for
the device containing any remaining data.
- A ``load_state`` function used to load sections generated by
any of the save functions that generate sections.
- ``cleanup`` functions for both save and load that are called
at the end of migration.
Note that the contents of the sections for iterative migration tend
to be open-coded by the devices; care should be taken in parsing
the results and structuring the stream to make them easy to validate.
Device ordering
---------------
There are cases in which the ordering of device loading matters; for
example in some systems where a device may assert an interrupt during loading,
if the interrupt controller is loaded later then it might lose the state.
Some ordering is implicitly provided by the order in which the machine
definition creates devices, however this is somewhat fragile.
The ``MigrationPriority`` enum provides a means of explicitly enforcing
ordering. Numerically higher priorities are loaded earlier.
The priority is set by setting the ``priority`` field of the top level
``VMStateDescription`` for the device.
Stream structure
================
The stream tries to be word and endian agnostic, allowing migration between hosts
of different characteristics running the same VM.
- Header
- Magic
- Version
- VM configuration section
- Machine type
- Target page bits
- List of sections
Each section contains a device, or one iteration of a device save.
- section type
- section id
- ID string (First section of each device)
- instance id (First section of each device)
- version id (First section of each device)
- <device data>
- Footer mark
- EOF mark
- VM Description structure
Consisting of a JSON description of the contents for analysis only
The ``device data`` in each section consists of the data produced
by the code described above. For non-iterative devices they have a single
section; iterative devices have an initial and last section and a set
of parts in between.
Note that there is very little checking by the common code of the integrity
of the ``device data`` contents, that's up to the devices themselves.
The ``footer mark`` provides a little bit of protection for the case where
the receiving side reads more or less data than expected.
The ``ID string`` is normally unique, having been formed from a bus name
and device address, PCI devices and storage devices hung off PCI controllers
fit this pattern well. Some devices are fixed single instances (e.g. "pc-ram").
Others (especially either older devices or system devices which for
some reason don't have a bus concept) make use of the ``instance id``
for otherwise identically named devices.
Return path
-----------
Only a unidirectional stream is required for normal migration, however a
``return path`` can be created when bidirectional communication is desired.
This is primarily used by postcopy, but is also used to return a success
flag to the source at the end of migration.
``qemu_file_get_return_path(QEMUFile* fwdpath)`` gives the QEMUFile* for the return
path.
Source side
Forward path - written by migration thread
Return path - opened by main thread, read by return-path thread
Destination side
Forward path - read by main thread
Return path - opened by main thread, written by main thread AND postcopy
thread (protected by rp_mutex)
docs/devel/migration/mapped-ram.rst
+142
View File
@@ -0,0 +1,142 @@
Mapped-ram
==========
Mapped-ram is a new stream format for the RAM section designed to
supplement the existing ``file:`` migration and make it compatible
with ``multifd``. This enables parallel migration of a guest's RAM to
a file.
The core of the feature is to ensure that RAM pages are mapped
directly to offsets in the resulting migration file. This enables the
``multifd`` threads to write exclusively to those offsets even if the
guest is constantly dirtying pages (i.e. live migration). Another
benefit is that the resulting file will have a bounded size, since
pages which are dirtied multiple times will always go to a fixed
location in the file, rather than constantly being added to a
sequential stream. Having the pages at fixed offsets also allows the
usage of O_DIRECT for save/restore of the migration stream as the
pages are ensured to be written respecting O_DIRECT alignment
restrictions.
Usage
-----
On both source and destination, enable the ``multifd`` and
``mapped-ram`` capabilities:
``migrate_set_capability multifd on``
``migrate_set_capability mapped-ram on``
Use a ``file:`` URL for migration:
``migrate file:/path/to/migration/file``
Mapped-ram migration is best done non-live, i.e. by stopping the VM on
the source side before migrating.
For best performance enable the ``direct-io`` parameter as well:
``migrate_set_parameter direct-io on``
Use-cases
---------
The mapped-ram feature was designed for use cases where the migration
stream will be directed to a file in the filesystem and not
immediately restored on the destination VM\ [#alternatives]_. These could be
thought of as snapshots. We can further categorize them into live and
non-live.
- Non-live snapshot
If the use case requires a VM to be stopped before taking a snapshot,
that's the ideal scenario for mapped-ram migration. Not having to
track dirty pages, the migration will write the RAM pages to the disk
as fast as it can.
Note: if a snapshot is taken of a running VM, but the VM will be
stopped after the snapshot by the admin, then consider stopping it
right before the snapshot to take benefit of the performance gains
mentioned above.
- Live snapshot
If the use case requires that the VM keeps running during and after
the snapshot operation, then mapped-ram migration can still be used,
but will be less performant. Other strategies such as
background-snapshot should be evaluated as well. One benefit of
mapped-ram in this scenario is portability since background-snapshot
depends on async dirty tracking (KVM_GET_DIRTY_LOG) which is not
supported outside of Linux.
.. [#alternatives] While this same effect could be obtained with the usage of
snapshots or the ``file:`` migration alone, mapped-ram provides
a performance increase for VMs with larger RAM sizes (10s to
100s of GiBs), specially if the VM has been stopped beforehand.
RAM section format
------------------
Instead of having a sequential stream of pages that follow the
RAMBlock headers, the dirty pages for a RAMBlock follow its header
instead. This ensures that each RAM page has a fixed offset in the
resulting migration file.
A bitmap is introduced to track which pages have been written in the
migration file. Pages are written at a fixed location for every
ramblock. Zero pages are ignored as they'd be zero in the destination
migration as well.
::
Without mapped-ram: With mapped-ram:
--------------------- --------------------------------
| ramblock 1 header | | ramblock 1 header |
--------------------- --------------------------------
| ramblock 2 header | | ramblock 1 mapped-ram header |
--------------------- --------------------------------
| ... | | padding to next 1MB boundary |
--------------------- | ... |
| ramblock n header | --------------------------------
--------------------- | ramblock 1 pages |
| RAM_SAVE_FLAG_EOS | | ... |
--------------------- --------------------------------
| stream of pages | | ramblock 2 header |
| (iter 1) | --------------------------------
| ... | | ramblock 2 mapped-ram header |
--------------------- --------------------------------
| RAM_SAVE_FLAG_EOS | | padding to next 1MB boundary |
--------------------- | ... |
| stream of pages | --------------------------------
| (iter 2) | | ramblock 2 pages |
| ... | | ... |
--------------------- --------------------------------
| ... | | ... |
--------------------- --------------------------------
| RAM_SAVE_FLAG_EOS |
--------------------------------
| ... |
--------------------------------
where:
- ramblock header: the generic information for a ramblock, such as
idstr, used_len, etc.
- ramblock mapped-ram header: the information added by this feature:
bitmap of pages written, bitmap size and offset of pages in the
migration file.
Restrictions
------------
Since pages are written to their relative offsets and out of order
(due to the memory dirtying patterns), streaming channels such as
sockets are not supported. A seekable channel such as a file is
required. This can be verified in the QIOChannel by the presence of
the QIO_CHANNEL_FEATURE_SEEKABLE.
The improvements brought by this feature apply only to guest physical
RAM. Other types of memory such as VRAM are migrated as part of device
states.
docs/devel/migration/postcopy.rst
+312
View File
@@ -0,0 +1,312 @@
========
Postcopy
========
.. contents::
'Postcopy' migration is a way to deal with migrations that refuse to converge
(or take too long to converge) its plus side is that there is an upper bound on
the amount of migration traffic and time it takes, the down side is that during
the postcopy phase, a failure of *either* side causes the guest to be lost.
In postcopy the destination CPUs are started before all the memory has been
transferred, and accesses to pages that are yet to be transferred cause
a fault that's translated by QEMU into a request to the source QEMU.
Postcopy can be combined with precopy (i.e. normal migration) so that if precopy
doesn't finish in a given time the switch is made to postcopy.
Enabling postcopy
=================
To enable postcopy, issue this command on the monitor (both source and
destination) prior to the start of migration:
``migrate_set_capability postcopy-ram on``
The normal commands are then used to start a migration, which is still
started in precopy mode. Issuing:
``migrate_start_postcopy``
will now cause the transition from precopy to postcopy.
It can be issued immediately after migration is started or any
time later on. Issuing it after the end of a migration is harmless.
Postcopy internals
==================
State machine
-------------
Postcopy moves through a series of states (see postcopy_state) from
ADVISE->DISCARD->LISTEN->RUNNING->END
- Advise
Set at the start of migration if postcopy is enabled, even
if it hasn't had the start command; here the destination
checks that its OS has the support needed for postcopy, and performs
setup to ensure the RAM mappings are suitable for later postcopy.
The destination will fail early in migration at this point if the
required OS support is not present.
(Triggered by reception of POSTCOPY_ADVISE command)
- Discard
Entered on receipt of the first 'discard' command; prior to
the first Discard being performed, hugepages are switched off
(using madvise) to ensure that no new huge pages are created
during the postcopy phase, and to cause any huge pages that
have discards on them to be broken.
- Listen
The first command in the package, POSTCOPY_LISTEN, switches
the destination state to Listen, and starts a new thread
(the 'listen thread') which takes over the job of receiving
pages off the migration stream, while the main thread carries
on processing the blob. With this thread able to process page
reception, the destination now 'sensitises' the RAM to detect
any access to missing pages (on Linux using the 'userfault'
system).
- Running
POSTCOPY_RUN causes the destination to synchronise all
state and start the CPUs and IO devices running. The main
thread now finishes processing the migration package and
now carries on as it would for normal precopy migration
(although it can't do the cleanup it would do as it
finishes a normal migration).
- End
The listen thread can now quit, and perform the cleanup of migration
state, the migration is now complete.
Device transfer
---------------
Loading of device data may cause the device emulation to access guest RAM
that may trigger faults that have to be resolved by the source, as such
the migration stream has to be able to respond with page data *during* the
device load, and hence the device data has to be read from the stream completely
before the device load begins to free the stream up. This is achieved by
'packaging' the device data into a blob that's read in one go.
Source behaviour
----------------
Until postcopy is entered the migration stream is identical to normal
precopy, except for the addition of a 'postcopy advise' command at
the beginning, to tell the destination that postcopy might happen.
When postcopy starts the source sends the page discard data and then
forms the 'package' containing:
- Command: 'postcopy listen'
- The device state
A series of sections, identical to the precopy streams device state stream
containing everything except postcopiable devices (i.e. RAM)
- Command: 'postcopy run'
The 'package' is sent as the data part of a Command: ``CMD_PACKAGED``, and the
contents are formatted in the same way as the main migration stream.
During postcopy the source scans the list of dirty pages and sends them
to the destination without being requested (in much the same way as precopy),
however when a page request is received from the destination, the dirty page
scanning restarts from the requested location. This causes requested pages
to be sent quickly, and also causes pages directly after the requested page
to be sent quickly in the hope that those pages are likely to be used
by the destination soon.
Destination behaviour
---------------------
Initially the destination looks the same as precopy, with a single thread
reading the migration stream; the 'postcopy advise' and 'discard' commands
are processed to change the way RAM is managed, but don't affect the stream
processing.
::
------------------------------------------------------------------------------
1 2 3 4 5 6 7
main -----DISCARD-CMD_PACKAGED ( LISTEN DEVICE DEVICE DEVICE RUN )
thread | |
| (page request)
| \___
v \
listen thread: --- page -- page -- page -- page -- page --
a b c
------------------------------------------------------------------------------
- On receipt of ``CMD_PACKAGED`` (1)
All the data associated with the package - the ( ... ) section in the diagram -
is read into memory, and the main thread recurses into qemu_loadvm_state_main
to process the contents of the package (2) which contains commands (3,6) and
devices (4...)
- On receipt of 'postcopy listen' - 3 -(i.e. the 1st command in the package)
a new thread (a) is started that takes over servicing the migration stream,
while the main thread carries on loading the package. It loads normal
background page data (b) but if during a device load a fault happens (5)
the returned page (c) is loaded by the listen thread allowing the main
threads device load to carry on.
- The last thing in the ``CMD_PACKAGED`` is a 'RUN' command (6)
letting the destination CPUs start running. At the end of the
``CMD_PACKAGED`` (7) the main thread returns to normal running behaviour and
is no longer used by migration, while the listen thread carries on servicing
page data until the end of migration.
Source side page bitmap
-----------------------
The 'migration bitmap' in postcopy is basically the same as in the precopy,
where each of the bit to indicate that page is 'dirty' - i.e. needs
sending. During the precopy phase this is updated as the CPU dirties
pages, however during postcopy the CPUs are stopped and nothing should
dirty anything any more. Instead, dirty bits are cleared when the relevant
pages are sent during postcopy.
Postcopy features
=================
Postcopy recovery
-----------------
Comparing to precopy, postcopy is special on error handlings. When any
error happens (in this case, mostly network errors), QEMU cannot easily
fail a migration because VM data resides in both source and destination
QEMU instances. On the other hand, when issue happens QEMU on both sides
will go into a paused state. It'll need a recovery phase to continue a
paused postcopy migration.
The recovery phase normally contains a few steps:
- When network issue occurs, both QEMU will go into **POSTCOPY_PAUSED**
migration state.
- When the network is recovered (or a new network is provided), the admin
can setup the new channel for migration using QMP command
'migrate-recover' on destination node, preparing for a resume.
- On source host, the admin can continue the interrupted postcopy
migration using QMP command 'migrate' with resume=true flag set.
Source QEMU will go into **POSTCOPY_RECOVER_SETUP** state trying to
re-establish the channels.
- When both sides of QEMU successfully reconnect using a new or fixed up
channel, they will go into **POSTCOPY_RECOVER** state, some handshake
procedure will be needed to properly synchronize the VM states between
the two QEMUs to continue the postcopy migration. For example, there
can be pages sent right during the window when the network is
interrupted, then the handshake will guarantee pages lost in-flight
will be resent again.
- After a proper handshake synchronization, QEMU will continue the
postcopy migration on both sides and go back to **POSTCOPY_ACTIVE**
state. Postcopy migration will continue.
During a paused postcopy migration, the VM can logically still continue
running, and it will not be impacted from any page access to pages that
were already migrated to destination VM before the interruption happens.
However, if any of the missing pages got accessed on destination VM, the VM
thread will be halted waiting for the page to be migrated, it means it can
be halted until the recovery is complete.
The impact of accessing missing pages can be relevant to different
configurations of the guest. For example, when with async page fault
enabled, logically the guest can proactively schedule out the threads
accessing missing pages.
Postcopy with hugepages
-----------------------
Postcopy now works with hugetlbfs backed memory:
a) The linux kernel on the destination must support userfault on hugepages.
b) The huge-page configuration on the source and destination VMs must be
identical; i.e. RAMBlocks on both sides must use the same page size.
c) Note that ``-mem-path /dev/hugepages`` will fall back to allocating normal
RAM if it doesn't have enough hugepages, triggering (b) to fail.
Using ``-mem-prealloc`` enforces the allocation using hugepages.
d) Care should be taken with the size of hugepage used; postcopy with 2MB
hugepages works well, however 1GB hugepages are likely to be problematic
since it takes ~1 second to transfer a 1GB hugepage across a 10Gbps link,
and until the full page is transferred the destination thread is blocked.
Postcopy with shared memory
---------------------------
Postcopy migration with shared memory needs explicit support from the other
processes that share memory and from QEMU. There are restrictions on the type of
memory that userfault can support shared.
The Linux kernel userfault support works on ``/dev/shm`` memory and on ``hugetlbfs``
(although the kernel doesn't provide an equivalent to ``madvise(MADV_DONTNEED)``
for hugetlbfs which may be a problem in some configurations).
The vhost-user code in QEMU supports clients that have Postcopy support,
and the ``vhost-user-bridge`` (in ``tests/``) and the DPDK package have changes
to support postcopy.
The client needs to open a userfaultfd and register the areas
of memory that it maps with userfault. The client must then pass the
userfaultfd back to QEMU together with a mapping table that allows
fault addresses in the clients address space to be converted back to
RAMBlock/offsets. The client's userfaultfd is added to the postcopy
fault-thread and page requests are made on behalf of the client by QEMU.
QEMU performs 'wake' operations on the client's userfaultfd to allow it
to continue after a page has arrived.
.. note::
There are two future improvements that would be nice:
a) Some way to make QEMU ignorant of the addresses in the clients
address space
b) Avoiding the need for QEMU to perform ufd-wake calls after the
pages have arrived
Retro-fitting postcopy to existing clients is possible:
a) A mechanism is needed for the registration with userfault as above,
and the registration needs to be coordinated with the phases of
postcopy. In vhost-user extra messages are added to the existing
control channel.
b) Any thread that can block due to guest memory accesses must be
identified and the implication understood; for example if the
guest memory access is made while holding a lock then all other
threads waiting for that lock will also be blocked.
Postcopy preemption mode
------------------------
Postcopy preempt is a new capability introduced in 8.0 QEMU release, it
allows urgent pages (those got page fault requested from destination QEMU
explicitly) to be sent in a separate preempt channel, rather than queued in
the background migration channel. Anyone who cares about latencies of page
faults during a postcopy migration should enable this feature. By default,
it's not enabled.
Postcopy blocktime statistics
-----------------------------
Blocktime is a postcopy live migration metric, intended to show how
long the vCPU was in state of interruptible sleep due to pagefault.
That metric is calculated both for all vCPUs as overlapped value, and
separately for each vCPU. These values are calculated on destination
side. To enable postcopy blocktime calculation, enter following
command on destination monitor:
``migrate_set_capability postcopy-blocktime on``
Postcopy blocktime can be retrieved by query-migrate qmp command.
postcopy-blocktime value of qmp command will show overlapped blocking
time for all vCPU, postcopy-vcpu-blocktime will show list of blocking
time per vCPU.
docs/devel/migration/qatzip-compression.rst
+165
View File
@@ -0,0 +1,165 @@
==================
QATzip Compression
==================
In scenarios with limited network bandwidth, the ``QATzip`` solution can help
users save a lot of host CPU resources by accelerating compression and
decompression through the Intel QuickAssist Technology(``QAT``) hardware.
The following test was conducted using 8 multifd channels and 10Gbps network
bandwidth. The results show that, compared to zstd, ``QATzip`` significantly
saves CPU resources on the sender and reduces migration time. Compared to the
uncompressed solution, ``QATzip`` greatly improves the dirty page processing
capability, indicated by the Pages per Second metric, and also reduces the
total migration time.
::
VM Configuration: 16 vCPU and 64G memory
VM Workload: all vCPUs are idle and 54G memory is filled with Silesia data.
QAT Devices: 4
|-----------|--------|---------|----------|----------|------|------|
|8 Channels |Total |down |throughput|pages per | send | recv |
| |time(ms)|time(ms) |(mbps) |second | cpu %| cpu% |
|-----------|--------|---------|----------|----------|------|------|
|qatzip | 16630| 28| 10467| 2940235| 160| 360|
|-----------|--------|---------|----------|----------|------|------|
|zstd | 20165| 24| 8579| 2391465| 810| 340|
|-----------|--------|---------|----------|----------|------|------|
|none | 46063| 40| 10848| 330240| 45| 85|
|-----------|--------|---------|----------|----------|------|------|
QATzip Compression Framework
============================
``QATzip`` is a user space library which builds on top of the Intel QuickAssist
Technology to provide extended accelerated compression and decompression
services.
For more ``QATzip`` introduction, please refer to `QATzip Introduction
<https://github.com/intel/QATzip?tab=readme-ov-file#introductionl>`_
::
+----------------+
| MultiFd Thread |
+-------+--------+
|
| compress/decompress
+-------+--------+
| QATzip library |
+-------+--------+
|
+-------+--------+
| QAT library |
+-------+--------+
| user space
--------+---------------------
| kernel space
+------+-------+
| QAT Driver |
+------+-------+
|
+------+-------+
| QAT Devices |
+--------------+
QATzip Installation
-------------------
The ``QATzip`` installation package has been integrated into some Linux
distributions and can be installed directly. For example, the Ubuntu Server
24.04 LTS system can be installed using below command
.. code-block:: shell
#apt search qatzip
libqatzip-dev/noble 1.2.0-0ubuntu3 amd64
Intel QuickAssist user space library development files
libqatzip3/noble 1.2.0-0ubuntu3 amd64
Intel QuickAssist user space library
qatzip/noble,now 1.2.0-0ubuntu3 amd64 [installed]
Compression user-space tool for Intel QuickAssist Technology
#sudo apt install libqatzip-dev libqatzip3 qatzip
If your system does not support the ``QATzip`` installation package, you can
use the source code to build and install, please refer to `QATzip source code installation
<https://github.com/intel/QATzip?tab=readme-ov-file#build-intel-quickassist-technology-driver>`_
QAT Hardware Deployment
-----------------------
``QAT`` supports physical functions(PFs) and virtual functions(VFs) for
deployment, and users can configure ``QAT`` resources for migration according
to actual needs. For more details about ``QAT`` deployment, please refer to
`Intel QuickAssist Technology Documentation
<https://intel.github.io/quickassist/index.html>`_
For more ``QAT`` hardware introduction, please refer to `intel-quick-assist-technology-overview
<https://www.intel.com/content/www/us/en/architecture-and-technology/intel-quick-assist-technology-overview.html>`_
How To Use QATzip Compression
=============================
1 - Install ``QATzip`` library
2 - Build ``QEMU`` with ``--enable-qatzip`` parameter
E.g. configure --target-list=x86_64-softmmu --enable-kvm ``--enable-qatzip``
3 - Set ``migrate_set_parameter multifd-compression qatzip``
4 - Set ``migrate_set_parameter multifd-qatzip-level comp_level``, the default
comp_level value is 1, and it supports levels from 1 to 9
QAT Memory Requirements
=======================
The user needs to reserve system memory for the QAT memory management to
allocate DMA memory. The size of the reserved system memory depends on the
number of devices used for migration and the number of multifd channels.
Because memory usage depends on QAT configuration, please refer to `QAT Memory
Driver Queries
<https://intel.github.io/quickassist/PG/infrastructure_debugability.html?highlight=memory>`_
for memory usage calculation.
.. list-table:: An example of a PF used for migration
:header-rows: 1
* - Number of channels
- Sender memory usage
- Receiver memory usage
* - 2
- 10M
- 10M
* - 4
- 12M
- 14M
* - 8
- 16M
- 20M
How To Choose Between QATzip and QPL
====================================
Starting from 4th Gen Intel Xeon Scalable processors, codenamed Sapphire Rapids
processor(``SPR``), multiple built-in accelerators are supported including
``QAT`` and ``IAA``. The former can accelerate ``QATzip`` and the latter is
used to accelerate ``QPL``.
Here are some suggestions:
1 - If the live migration scenario is limited by network bandwidth and ``QAT``
hardware resources exceed ``IAA``, use the ``QATzip`` method, which can save a
lot of host CPU resources for compression.
2 - If the system cannot support shared virtual memory (SVM) technology, use
the ``QATzip`` method because ``QPL`` performance is not good without SVM
support.
3 - For other scenarios, use the ``QPL`` method first.
docs/devel/migration/qpl-compression.rst
+260
View File
@@ -0,0 +1,260 @@
===============
QPL Compression
===============
The Intel Query Processing Library (Intel ``QPL``) is an open-source library to
provide compression and decompression features and it is based on deflate
compression algorithm (RFC 1951).
The ``QPL`` compression relies on Intel In-Memory Analytics Accelerator(``IAA``)
and Shared Virtual Memory(``SVM``) technology, they are new features supported
from Intel 4th Gen Intel Xeon Scalable processors, codenamed Sapphire Rapids
processor(``SPR``).
For more ``QPL`` introduction, please refer to `QPL Introduction
<https://intel.github.io/qpl/documentation/introduction_docs/introduction.html>`_
QPL Compression Framework
=========================
::
+----------------+ +------------------+
| MultiFD Thread | |accel-config tool |
+-------+--------+ +--------+---------+
| |
| |
|compress/decompress |
+-------+--------+ | Setup IAA
| QPL library | | Resources
+-------+---+----+ |
| | |
| +-------------+-------+
| Open IAA |
| Devices +-----+-----+
| |idxd driver|
| +-----+-----+
| |
| |
| +-----+-----+
+-----------+IAA Devices|
Submit jobs +-----------+
via enqcmd
QPL Build And Installation
--------------------------
.. code-block:: shell
$git clone --recursive https://github.com/intel/qpl.git qpl
$mkdir qpl/build
$cd qpl/build
$cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr -DQPL_LIBRARY_TYPE=SHARED ..
$sudo cmake --build . --target install
For more details about ``QPL`` installation, please refer to `QPL Installation
<https://intel.github.io/qpl/documentation/get_started_docs/installation.html>`_
IAA Device Management
---------------------
The number of ``IAA`` devices will vary depending on the Xeon product model.
On a ``SPR`` server, there can be a maximum of 8 ``IAA`` devices, with up to
4 devices per socket.
By default, all ``IAA`` devices are disabled and need to be configured and
enabled by users manually.
Check the number of devices through the following command
.. code-block:: shell
#lspci -d 8086:0cfe
6a:02.0 System peripheral: Intel Corporation Device 0cfe
6f:02.0 System peripheral: Intel Corporation Device 0cfe
74:02.0 System peripheral: Intel Corporation Device 0cfe
79:02.0 System peripheral: Intel Corporation Device 0cfe
e7:02.0 System peripheral: Intel Corporation Device 0cfe
ec:02.0 System peripheral: Intel Corporation Device 0cfe
f1:02.0 System peripheral: Intel Corporation Device 0cfe
f6:02.0 System peripheral: Intel Corporation Device 0cfe
IAA Device Configuration And Enabling
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The ``accel-config`` tool is used to enable ``IAA`` devices and configure
``IAA`` hardware resources(work queues and engines). One ``IAA`` device
has 8 work queues and 8 processing engines, multiple engines can be assigned
to a work queue via ``group`` attribute.
For ``accel-config`` installation, please refer to `accel-config installation
<https://github.com/intel/idxd-config>`_
One example of configuring and enabling an ``IAA`` device.
.. code-block:: shell
#accel-config config-engine iax1/engine1.0 -g 0
#accel-config config-engine iax1/engine1.1 -g 0
#accel-config config-engine iax1/engine1.2 -g 0
#accel-config config-engine iax1/engine1.3 -g 0
#accel-config config-engine iax1/engine1.4 -g 0
#accel-config config-engine iax1/engine1.5 -g 0
#accel-config config-engine iax1/engine1.6 -g 0
#accel-config config-engine iax1/engine1.7 -g 0
#accel-config config-wq iax1/wq1.0 -g 0 -s 128 -p 10 -b 1 -t 128 -m shared -y user -n app1 -d user
#accel-config enable-device iax1
#accel-config enable-wq iax1/wq1.0
.. note::
IAX is an early name for IAA
- The ``IAA`` device index is 1, use ``ls -lh /sys/bus/dsa/devices/iax*``
command to query the ``IAA`` device index.
- 8 engines and 1 work queue are configured in group 0, so all compression jobs
submitted to this work queue can be processed by all engines at the same time.
- Set work queue attributes including the work mode, work queue size and so on.
- Enable the ``IAA1`` device and work queue 1.0
.. note::
Set work queue mode to shared mode, since ``QPL`` library only supports
shared mode
For more detailed configuration, please refer to `IAA Configuration Samples
<https://github.com/intel/idxd-config/tree/stable/Documentation/accfg>`_
IAA Unit Test
^^^^^^^^^^^^^
- Enabling ``IAA`` devices for Xeon platform, please refer to `IAA User Guide
<https://www.intel.com/content/www/us/en/content-details/780887/intel-in-memory-analytics-accelerator-intel-iaa.html>`_
- ``IAA`` device driver is Intel Data Accelerator Driver (idxd), it is
recommended that the minimum version of Linux kernel is 5.18.
- Add ``"intel_iommu=on,sm_on"`` parameter to kernel command line
for ``SVM`` feature enabling.
Here is an easy way to verify ``IAA`` device driver and ``SVM`` with `iaa_test
<https://github.com/intel/idxd-config/tree/stable/test>`_
.. code-block:: shell
#./test/iaa_test
[ info] alloc wq 0 shared size 128 addr 0x7f26cebe5000 batch sz 0xfffffffe xfer sz 0x80000000
[ info] test noop: tflags 0x1 num_desc 1
[ info] preparing descriptor for noop
[ info] Submitted all noop jobs
[ info] verifying task result for 0x16f7e20
[ info] test with op 0 passed
IAA Resources Allocation For Migration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
There is no ``IAA`` resource configuration parameters for migration and
``accel-config`` tool configuration cannot directly specify the ``IAA``
resources used for migration.
The multifd migration with ``QPL`` compression method will use all work
queues that are enabled and shared mode.
.. note::
Accessing IAA resources requires ``sudo`` command or ``root`` privileges
by default. Administrators can modify the IAA device node ownership
so that QEMU can use IAA with specified user permissions.
For example
#chown -R qemu /dev/iax
Shared Virtual Memory(SVM) Introduction
=======================================
An ability for an accelerator I/O device to operate in the same virtual
memory space of applications on host processors. It also implies the
ability to operate from pageable memory, avoiding functional requirements
to pin memory for DMA operations.
When using ``SVM`` technology, users do not need to reserve memory for the
``IAA`` device and perform pin memory operation. The ``IAA`` device can
directly access data using the virtual address of the process.
For more ``SVM`` technology, please refer to
`Shared Virtual Addressing (SVA) with ENQCMD
<https://docs.kernel.org/next/x86/sva.html>`_
How To Use QPL Compression In Migration
=======================================
1 - Installation of ``QPL`` library and ``accel-config`` library if using IAA
2 - Configure and enable ``IAA`` devices and work queues via ``accel-config``
3 - Build ``QEMU`` with ``--enable-qpl`` parameter
E.g. configure --target-list=x86_64-softmmu --enable-kvm ``--enable-qpl``
4 - Enable ``QPL`` compression during migration
Set ``migrate_set_parameter multifd-compression qpl`` when migrating, the
``QPL`` compression does not support configuring the compression level, it
only supports one compression level.
The Difference Between QPL And ZLIB
===================================
Although both ``QPL`` and ``ZLIB`` are based on the deflate compression
algorithm, and ``QPL`` can support the header and tail of ``ZLIB``, ``QPL``
is still not fully compatible with the ``ZLIB`` compression in the migration.
``QPL`` only supports 4K history buffer, and ``ZLIB`` is 32K by default.
``ZLIB`` compresses data that ``QPL`` may not decompress correctly and
vice versa.
``QPL`` does not support the ``Z_SYNC_FLUSH`` operation in ``ZLIB`` streaming
compression, current ``ZLIB`` implementation uses ``Z_SYNC_FLUSH``, so each
``multifd`` thread has a ``ZLIB`` streaming context, and all page compression
and decompression are based on this stream. ``QPL`` cannot decompress such data
and vice versa.
The introduction for ``Z_SYNC_FLUSH``, please refer to `Zlib Manual
<https://www.zlib.net/manual.html>`_
The Best Practices
==================
When user enables the IAA device for ``QPL`` compression, it is recommended
to add ``-mem-prealloc`` parameter to the destination boot parameters. This
parameter can avoid the occurrence of I/O page fault and reduce the overhead
of IAA compression and decompression.
The example of booting with ``-mem-prealloc`` parameter
.. code-block:: shell
$qemu-system-x86_64 --enable-kvm -cpu host --mem-prealloc ...
An example about I/O page fault measurement of destination without
``-mem-prealloc``, the ``svm_prq`` indicates the number of I/O page fault
occurrences and processing time.
.. code-block:: shell
#echo 1 > /sys/kernel/debug/iommu/intel/dmar_perf_latency
#echo 2 > /sys/kernel/debug/iommu/intel/dmar_perf_latency
#echo 3 > /sys/kernel/debug/iommu/intel/dmar_perf_latency
#echo 4 > /sys/kernel/debug/iommu/intel/dmar_perf_latency
#cat /sys/kernel/debug/iommu/intel/dmar_perf_latency
IOMMU: dmar18 Register Base Address: c87fc000
<0.1us 0.1us-1us 1us-10us 10us-100us 100us-1ms 1ms-10ms >=10ms min(us) max(us) average(us)
inv_iotlb 0 286 123 0 0 0 0 0 1 0
inv_devtlb 0 276 133 0 0 0 0 0 2 0
inv_iec 0 0 0 0 0 0 0 0 0 0
svm_prq 0 0 25206 364 395 0 0 1 556 9
docs/devel/migration/uadk-compression.rst
+144
View File
@@ -0,0 +1,144 @@
=========================================================
User Space Accelerator Development Kit (UADK) Compression
=========================================================
UADK is a general-purpose user space accelerator framework that uses shared
virtual addressing (SVA) to provide a unified programming interface for
hardware acceleration of cryptographic and compression algorithms.
UADK includes Unified/User-space-access-intended Accelerator Framework (UACCE),
which enables hardware accelerators from different vendors that support SVA to
adapt to UADK.
Currently, HiSilicon Kunpeng hardware accelerators have been registered with
UACCE. Through the UADK framework, users can run cryptographic and compression
algorithms using hardware accelerators instead of CPUs, freeing up CPU
computing power and improving computing performance.
https://github.com/Linaro/uadk/tree/master/docs
UADK Framework
==============
UADK consists of UACCE, vendors' drivers, and an algorithm layer. UADK requires
the hardware accelerator to support SVA, and the operating system to support
IOMMU and SVA. Hardware accelerators from different vendors are registered as
different character devices with UACCE by using kernel-mode drivers of the
vendors. A user can access the hardware accelerators by performing user-mode
operations on the character devices.
::
+----------------------------------+
| apps |
+----+------------------------+----+
| |
| |
+-------+--------+ +-------+-------+
| scheduler | | alg libraries |
+-------+--------+ +-------+-------+
| |
| |
| |
| +--------+------+
| | vendor drivers|
| +-+-------------+
| |
| |
+--+------------------+--+
| libwd |
User +----+-------------+-----+
--------------------------------------------------
Kernel +--+-----+ +------+
| uacce | | smmu |
+---+----+ +------+
|
+---+------------------+
| vendor kernel driver |
+----------------------+
--------------------------------------------------
+----------------------+
| HW Accelerators |
+----------------------+
UADK Installation
-----------------
Build UADK
^^^^^^^^^^
.. code-block:: shell
git clone https://github.com/Linaro/uadk.git
cd uadk
mkdir build
./autogen.sh
./configure --prefix=$PWD/build
make
make install
Without --prefix, UADK will be installed to /usr/local/lib by default.
If get error:"cannot find -lnuma", please install the libnuma-dev
Run pkg-config libwd to ensure env is setup correctly
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* export PKG_CONFIG_PATH=$PWD/build/lib/pkgconfig
* pkg-config libwd --cflags --libs
-I/usr/local/include -L/usr/local/lib -lwd
* export PKG_CONFIG_PATH is required on demand.
Not required if UADK is installed to /usr/local/lib
UADK Host Kernel Requirements
-----------------------------
User needs to make sure that ``UACCE`` is already supported in Linux kernel.
The kernel version should be at least v5.9 with SVA (Shared Virtual
Addressing) enabled.
Kernel Configuration
^^^^^^^^^^^^^^^^^^^^
``UACCE`` could be built as module or built-in.
Here's an example to enable UACCE with hardware accelerator in HiSilicon
Kunpeng platform.
* CONFIG_IOMMU_SVA_LIB=y
* CONFIG_ARM_SMMU=y
* CONFIG_ARM_SMMU_V3=y
* CONFIG_ARM_SMMU_V3_SVA=y
* CONFIG_PCI_PASID=y
* CONFIG_UACCE=y
* CONFIG_CRYPTO_DEV_HISI_QM=y
* CONFIG_CRYPTO_DEV_HISI_ZIP=y
Make sure all these above kernel configurations are selected.
Accelerator dev node permissions
--------------------------------
Hardware accelerators (eg: HiSilicon Kunpeng Zip accelerator) gets registered to
UADK and char devices are created in dev directory. In order to access resources
on hardware accelerator devices, write permission should be provided to user.
.. code-block:: shell
$ sudo chmod 777 /dev/hisi_zip-*
How To Use UADK Compression In QEMU Migration
---------------------------------------------
* Make sure UADK is installed as above
* Build ``QEMU`` with ``--enable-uadk`` parameter
E.g. configure --target-list=aarch64-softmmu --enable-kvm ``--enable-uadk``
* Enable ``UADK`` compression during migration
Set ``migrate_set_parameter multifd-compression uadk``
Since UADK uses Shared Virtual Addressing(SVA) and device access virtual memory
directly it is possible that SMMUv3 may encounter page faults while walking the
IO page tables. This may impact the performance. In order to mitigate this,
please make sure to specify ``-mem-prealloc`` parameter to the destination VM
boot parameters.
Though both UADK and ZLIB are based on the deflate compression algorithm, UADK
is not fully compatible with ZLIB. Hence, please make sure to use ``uadk`` on
both source and destination during migration.
docs/devel/migration/vfio.rst
+274
View File
@@ -0,0 +1,274 @@
=====================
VFIO device migration
=====================
Migration of virtual machine involves saving the state for each device that
the guest is running on source host and restoring this saved state on the
destination host. This document details how saving and restoring of VFIO
devices is done in QEMU.
Migration of VFIO devices consists of two phases: the optional pre-copy phase,
and the stop-and-copy phase. The pre-copy phase is iterative and allows to
accommodate VFIO devices that have a large amount of data that needs to be
transferred. The iterative pre-copy phase of migration allows for the guest to
continue whilst the VFIO device state is transferred to the destination, this
helps to reduce the total downtime of the VM. VFIO devices opt-in to pre-copy
support by reporting the VFIO_MIGRATION_PRE_COPY flag in the
VFIO_DEVICE_FEATURE_MIGRATION ioctl.
When pre-copy is supported, it's possible to further reduce downtime by
enabling "switchover-ack" migration capability.
VFIO migration uAPI defines "initial bytes" as part of its pre-copy data stream
and recommends that the initial bytes are sent and loaded in the destination
before stopping the source VM. Enabling this migration capability will
guarantee that and thus, can potentially reduce downtime even further.
For example, in mlx5 devices, the initial bytes hold metadata used for time
consuming pre-allocations of resources on the destination. Although init bytes
may be small in size and sending them may take little time, loading them in the
destination can take a significant amount of time. Switchover-ack guarantees
that this pre-allocation doesn't happen during downtime.
Initial bytes was originally defined to be monotonically decreasing, however
there are cases where a new chunk of initial bytes should be transferred during
precopy, e.g., due to a device reconfiguration, etc. The
VFIO_PRECOPY_INFO_REINIT feature addresses this and when supported, allows to
report a new initial bytes value regardless of any previously reported values.
In this case, a new switchover ACK will be requested to make sure the new
initial bytes are loaded in the destination before switching over.
To support migration of multiple devices that might do P2P transactions between
themselves, VFIO migration uAPI defines an intermediate P2P quiescent state.
While in the P2P quiescent state, P2P DMA transactions cannot be initiated by
the device, but the device can respond to incoming ones. Additionally, all
outstanding P2P transactions are guaranteed to have been completed by the time
the device enters this state.
All the devices that support P2P migration are first transitioned to the P2P
quiescent state and only then are they stopped or started. This makes migration
safe P2P-wise, since starting and stopping the devices is not done atomically
for all the devices together.
Thus, multiple VFIO devices migration is allowed only if all the devices
support P2P migration. Single VFIO device migration is allowed regardless of
P2P migration support.
A detailed description of the UAPI for VFIO device migration can be found in
the comment for the ``vfio_device_mig_state`` structure in the header file
linux-headers/linux/vfio.h.
VFIO implements the device hooks for the iterative approach as follows:
* A ``save_setup`` function that sets up migration on the source.
* A ``load_setup`` function that sets the VFIO device on the destination in
_RESUMING state.
* A ``save_query_pending`` function that reports the remaining data that
the vendor driver has yet to save for the VFIO device.
* An ``is_active_iterate`` function that indicates ``save_live_iterate`` is
active only when the VFIO device is in pre-copy states.
* A ``save_live_iterate`` function that reads the VFIO device's data from the
vendor driver during iterative pre-copy phase.
* A ``switchover_start`` function that in the multifd mode starts a thread that
reassembles the multifd received data and loads it in-order into the device.
In the non-multifd mode this function is a NOP.
* A ``save_state`` function to save the device config space if it is present
in the non-multifd mode.
In the multifd mode it just emits either a dummy EOS marker.
* A ``save_complete`` function that sets the VFIO device in _STOP_COPY
state and iteratively copies the data for the VFIO device until the
vendor driver indicates that no data remains. In the multifd mode it
just emits a dummy EOS marker.
* A ``save_complete_precopy_thread`` function that in the multifd mode
provides thread handler performing multifd device state transfer.
It sets the VFIO device to _STOP_COPY state, iteratively reads the data
from the VFIO device and queues it for multifd transmission until the vendor
driver indicates that no data remains.
After that, it saves the device config space and queues it for multifd
transfer too.
In the non-multifd mode this thread is a NOP.
* A ``load_state`` function that loads the config section and the data
sections that are generated by the save functions above.
* A ``load_state_buffer`` function that loads the device state and the device
config that arrived via multifd channels.
It's used only in the multifd mode.
* ``cleanup`` functions for both save and load that perform any migration
related cleanup.
The VFIO migration code uses a VM state change handler to change the VFIO
device state when the VM state changes from running to not-running, and
vice versa.
Similarly, a migration state change handler is used to trigger a transition of
the VFIO device state when certain changes of the migration state occur. For
example, the VFIO device state is transitioned back to _RUNNING in case a
migration failed or was canceled.
System memory dirty pages tracking
----------------------------------
A ``log_global_start`` and ``log_global_stop`` memory listener callback informs
the VFIO dirty tracking module to start and stop dirty page tracking. A
``log_sync`` memory listener callback queries the dirty page bitmap from the
dirty tracking module and marks system memory pages which were DMA-ed by the
VFIO device as dirty. The dirty page bitmap is queried per container.
Currently there are two ways dirty page tracking can be done:
(1) Device dirty tracking:
In this method the device is responsible to log and report its DMAs. This
method can be used only if the device is capable of tracking its DMAs.
Discovering device capability, starting and stopping dirty tracking, and
syncing the dirty bitmaps from the device are done using the DMA logging uAPI.
More info about the uAPI can be found in the comments of the
``vfio_device_feature_dma_logging_control`` and
``vfio_device_feature_dma_logging_report`` structures in the header file
linux-headers/linux/vfio.h.
(2) VFIO IOMMU module:
In this method dirty tracking is done by IOMMU. However, there is currently no
IOMMU support for dirty page tracking. For this reason, all pages are
perpetually marked dirty, unless the device driver pins pages through external
APIs in which case only those pinned pages are perpetually marked dirty.
If the above two methods are not supported, all pages are perpetually marked
dirty by QEMU.
By default, dirty pages are tracked during pre-copy as well as stop-and-copy
phase. So, a page marked as dirty will be copied to the destination in both
phases. Copying dirty pages in pre-copy phase helps QEMU to predict if it can
achieve its downtime tolerances. If QEMU during pre-copy phase keeps finding
dirty pages continuously, then it understands that even in stop-and-copy phase,
it is likely to find dirty pages and can predict the downtime accordingly.
QEMU also provides a per device opt-out option ``pre-copy-dirty-page-tracking``
which disables querying the dirty bitmap during pre-copy phase. If it is set to
off, all dirty pages will be copied to the destination in stop-and-copy phase
only.
System memory dirty pages tracking when vIOMMU is enabled
---------------------------------------------------------
With vIOMMU, an IO virtual address range can get unmapped while in pre-copy
phase of migration. In that case, the unmap ioctl returns any dirty pages in
that range and QEMU reports corresponding guest physical pages dirty. During
stop-and-copy phase, an IOMMU notifier is used to get a callback for mapped
pages and then dirty pages bitmap is fetched from VFIO IOMMU modules for those
mapped ranges. If device dirty tracking is enabled with vIOMMU, live migration
will be blocked.
Flow of state changes during Live migration
===========================================
Below is the state change flow during live migration for a VFIO device that
supports both precopy and P2P migration. The flow for devices that don't
support it is similar, except that the relevant states for precopy and P2P are
skipped.
The values in the parentheses represent the VM state, the migration state, and
the VFIO device state, respectively.
Live migration save path
------------------------
::
QEMU normal running state
(RUNNING, _NONE, _RUNNING)
|
migrate_init spawns migration_thread
Migration thread then calls each device's .save_setup()
(RUNNING, _SETUP, _PRE_COPY)
|
(RUNNING, _ACTIVE, _PRE_COPY)
If device is active, get pending_bytes by .state_pending_{estimate,exact}()
If total pending_bytes >= threshold_size, call .save_live_iterate()
Data of VFIO device for pre-copy phase is copied
Iterate till total pending bytes converge and are less than threshold
|
On migration completion, the vCPUs and the VFIO device are stopped
The VFIO device is first put in P2P quiescent state
(FINISH_MIGRATE, _ACTIVE, _PRE_COPY_P2P)
|
Then the VFIO device is put in _STOP_COPY state
(FINISH_MIGRATE, _ACTIVE, _STOP_COPY)
.save_complete() is called for each active device
For the VFIO device: in the non-multifd mode iterate in
.save_complete() until
pending data is 0
In the multifd mode this iteration is done in
.save_complete_precopy_thread() instead.
|
(POSTMIGRATE, _COMPLETED, _STOP_COPY)
Migraton thread schedules cleanup bottom half and exits
|
.save_cleanup() is called
(POSTMIGRATE, _COMPLETED, _STOP)
Live migration resume path
--------------------------
::
Incoming migration calls .load_setup() for each device
(RESTORE_VM, _ACTIVE, _STOP)
|
For each device, .load_state() is called for that device section data
transmitted via the main migration channel.
For data transmitted via multifd channels .load_state_buffer() is called
instead.
(RESTORE_VM, _ACTIVE, _RESUMING)
|
At the end, .load_cleanup() is called for each device and vCPUs are started
The VFIO device is first put in P2P quiescent state
(RUNNING, _ACTIVE, _RUNNING_P2P)
|
(RUNNING, _NONE, _RUNNING)
Postcopy
========
Postcopy migration is currently not supported for VFIO devices.
Multifd
=======
Starting from QEMU version 10.0 there's a possibility to transfer VFIO device
_STOP_COPY state via multifd channels. This helps reduce downtime - especially
with multiple VFIO devices or with devices having a large migration state.
As an additional benefit, setting the VFIO device to _STOP_COPY state and
saving its config space is also parallelized (run in a separate thread) in
such migration mode.
The multifd VFIO device state transfer is controlled by
"x-migration-multifd-transfer" VFIO device property. This property defaults to
AUTO, which means that VFIO device state transfer via multifd channels is
attempted in configurations that otherwise support it.
Since the target QEMU needs to load device state buffers in-order it needs to
queue incoming buffers until they can be loaded into the device.
This means that a malicious QEMU source could theoretically cause the target
QEMU to allocate unlimited amounts of memory for such buffers-in-flight.
The "x-migration-max-queued-buffers-size" property allows capping the total size
of these VFIO device state buffers queued at the destination.
Because a malicious QEMU source causing OOM on the target is not expected to be
a realistic threat in most of VFIO live migration use cases and the right value
depends on the particular setup by default this queued buffers size limit is
disabled by setting it to UINT64_MAX.
Some host platforms (like ARM64) require that VFIO device config is loaded only
after all iterables were loaded, during non-iterables loading phase.
Such interlocking is controlled by "x-migration-load-config-after-iter" VFIO
device property, which in its default setting (AUTO) does so only on platforms
that actually require it.
docs/devel/migration/virtio.rst
+115
View File
@@ -0,0 +1,115 @@
=======================
Virtio device migration
=======================
Copyright 2015 IBM Corp.
This work is licensed under the terms of the GNU GPL, version 2 or later. See
the COPYING file in the top-level directory.
Saving and restoring the state of virtio devices is a bit of a twisty maze,
for several reasons:
- state is distributed between several parts:
- virtio core, for common fields like features, number of queues, ...
- virtio transport (pci, ccw, ...), for the different proxy devices and
transport specific state (msix vectors, indicators, ...)
- virtio device (net, blk, ...), for the different device types and their
state (mac address, request queue, ...)
- most fields are saved via the stream interface; subsequently, subsections
have been added to make cross-version migration possible
This file attempts to document the current procedure and point out some
caveats.
Save state procedure
====================
::
virtio core virtio transport virtio device
----------- ---------------- -------------
save() function registered
via VMState wrapper on
device class
virtio_save() <----------
------> save_config()
- save proxy device
- save transport-specific
device fields
- save common device
fields
- save common virtqueue
fields
------> save_queue()
- save transport-specific
virtqueue fields
------> save_device()
- save device-specific
fields
- save subsections
- device endianness,
if changed from
default endianness
- 64 bit features, if
any high feature bit
is set
- virtio-1 virtqueue
fields, if VERSION_1
is set
Load state procedure
====================
::
virtio core virtio transport virtio device
----------- ---------------- -------------
load() function registered
via VMState wrapper on
device class
virtio_load() <----------
------> load_config()
- load proxy device
- load transport-specific
device fields
- load common device
fields
- load common virtqueue
fields
------> load_queue()
- load transport-specific
virtqueue fields
- notify guest
------> load_device()
- load device-specific
fields
- load subsections
- device endianness
- 64 bit features
- virtio-1 virtqueue
fields
- sanitize endianness
- sanitize features
- virtqueue index sanity
check
- feature-dependent setup
Implications of this setup
==========================
Devices need to be careful in their state processing during load: The
load_device() procedure is invoked by the core before subsections have
been loaded. Any code that depends on information transmitted in subsections
therefore has to be invoked in the device's load() function _after_
virtio_load() returned (like e.g. code depending on features).
Any extension of the state being migrated should be done in subsections
added to the core for compatibility reasons. If transport or device specific
state is added, core needs to invoke a callback from the new subsection.
docs/devel/migration/xbzrle.rst
+160
View File
@@ -0,0 +1,160 @@
XBZRLE (Xor Based Zero Run Length Encoding)
===========================================
Using XBZRLE (Xor Based Zero Run Length Encoding) allows for the reduction
of VM downtime and the total live-migration time of Virtual machines.
It is particularly useful for virtual machines running memory write intensive
workloads that are typical of large enterprise applications such as SAP ERP
Systems, and generally speaking for any application that uses a sparse memory
update pattern.
Instead of sending the changed guest memory page this solution will send a
compressed version of the updates, thus reducing the amount of data sent during
live migration.
In order to be able to calculate the update, the previous memory pages need to
be stored on the source. Those pages are stored in a dedicated cache
(hash table) and are accessed by their address.
The larger the cache size the better the chances are that the page has already
been stored in the cache.
A small cache size will result in high cache miss rate.
Cache size can be changed before and during migration.
Format
------
The compression format performs a XOR between the previous and current content
of the page, where zero represents an unchanged value.
The page data delta is represented by zero and non zero runs.
A zero run is represented by its length (in bytes).
A non zero run is represented by its length (in bytes) and the new data.
The run length is encoded using ULEB128 (http://en.wikipedia.org/wiki/LEB128)
There can be more than one valid encoding, the sender may send a longer
encoding for the benefit of reducing computation cost.
::
page = zrun nzrun
| zrun nzrun page
zrun = length
nzrun = length byte...
length = uleb128 encoded integer
On the sender side XBZRLE is used as a compact delta encoding of page updates,
retrieving the old page content from the cache (default size of 64MB). The
receiving side uses the existing page's content and XBZRLE to decode the new
page's content.
This work was originally based on research results published
VEE 2011: Evaluation of Delta Compression Techniques for Efficient Live
Migration of Large Virtual Machines by Benoit, Svard, Tordsson and Elmroth.
Additionally the delta encoder XBRLE was improved further using the XBZRLE
instead.
XBZRLE has a sustained bandwidth of 2-2.5 GB/s for typical workloads making it
ideal for in-line, real-time encoding such as is needed for live-migration.
Example:
old buffer:
.. code:: batch
1001 zeros
05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 68 00 00 6b 00 6d
3074 zeros
new buffer:
.. code:: batch
1001 zeros
01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 68 00 00 67 00 69
3074 zeros
encoded buffer:
.. code:: batch
encoded length 24
e9 07 0f 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 03 01 67 01 01 69
Cache update strategy
---------------------
Keeping the hot pages in the cache is effective for decreasing cache
misses. XBZRLE uses a counter as the age of each page. The counter will
increase after each ram dirty bitmap sync. When a cache conflict is
detected, XBZRLE will only evict pages in the cache that are older than
a threshold.
Usage
-----
1. Verify the destination QEMU version is able to decode the new format::
(qemu) info migrate_capabilities
xbzrle: off
...
2. Activate xbzrle on both source and destination::
(qemu) migrate_set_capability xbzrle on
3. Set the XBZRLE cache size - the cache size is in MBytes and should be a
power of 2. The cache default value is 64 MBytes (on source only)::
(qemu) migrate_set_parameter xbzrle-cache-size 256m
4. Start outgoing migration::
(qemu) migrate -d tcp:destination.host:4444
(qemu) info migrate
capabilities: xbzrle: on
Migration status: active
transferred ram: A kbytes
remaining ram: B kbytes
total ram: C kbytes
total time: D milliseconds
duplicate: E pages
normal: F pages
normal bytes: G kbytes
cache size: H bytes
xbzrle transferred: I kbytes
xbzrle pages: J pages
xbzrle cache miss: K pages
xbzrle cache miss rate: L
xbzrle encoding rate: M
xbzrle overflow: N
xbzrle cache miss: the number of cache misses to date - high cache-miss rate
indicates that the cache size is set too low.
xbzrle overflow: the number of overflows in the decoding which where the delta
could not be compressed. This can happen if the changes in the pages are too
large or there are many short changes; for example, changing every second byte
(half a page).
Testing: Testing indicated that live migration with XBZRLE was completed in 110
seconds, whereas without it would not be able to complete.
A simple synthetic memory r/w load generator:
.. code-block:: c
#include <stdlib.h>
#include <stdio.h>
int main()
{
char *buf = (char *) calloc(4096, 4096);
while (1) {
int i;
for (i = 0; i < 4096 * 4; i++) {
buf[i * 4096 / 4]++;
}
printf(".");
}
}