Import QEMU upstream snapshot d2e570c
Upstream: https://gitlab.com/qemu-project/qemu.git Upstream-Commit: d2e570cc0f97b936902a5b1b86b73c0f5998b475
This commit is contained in:
Vendored
+14
@@ -0,0 +1,14 @@
|
||||
{% extends "!footer.html" %}
|
||||
{% block extrafooter %}
|
||||
|
||||
<!-- Empty para to force a blank line after "Built with Sphinx ..." -->
|
||||
<p></p>
|
||||
|
||||
<p>This documentation is for QEMU version {{ version }}.</p>
|
||||
|
||||
{% trans path=pathto('about/license') %}
|
||||
<p><a href="{{ path }}">QEMU and this manual are released under the
|
||||
GNU General Public License, version 2.</a></p>
|
||||
{% endtrans %}
|
||||
{{ super() }}
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,173 @@
|
||||
.. _Supported-build-platforms:
|
||||
|
||||
Supported build platforms
|
||||
=========================
|
||||
|
||||
QEMU aims to support building and executing on multiple host OS
|
||||
platforms. This appendix outlines which platforms are the major build
|
||||
targets. These platforms are used as the basis for deciding upon the
|
||||
minimum required versions of 3rd party software QEMU depends on. The
|
||||
supported platforms are the targets for automated testing performed by
|
||||
the project when patches are submitted for review, and tested before and
|
||||
after merge.
|
||||
|
||||
If a platform is not listed here, it does not imply that QEMU won't
|
||||
work. If an unlisted platform has comparable software versions to a
|
||||
listed platform, there is every expectation that it will work. Bug
|
||||
reports are welcome for problems encountered on unlisted platforms
|
||||
unless they are clearly older vintage than what is described here.
|
||||
|
||||
Note that when considering software versions shipped in distros as
|
||||
support targets, QEMU considers only the version number, and assumes the
|
||||
features in that distro match the upstream release with the same
|
||||
version. In other words, if a distro backports extra features to the
|
||||
software in their distro, QEMU upstream code will not add explicit
|
||||
support for those backports, unless the feature is auto-detectable in a
|
||||
manner that works for the upstream releases too.
|
||||
|
||||
The `Repology`_ site is a useful resource to identify
|
||||
currently shipped versions of software in various operating systems,
|
||||
though it does not cover all distros listed below.
|
||||
|
||||
You can find how to install build dependencies for different systems on the
|
||||
:ref:`setup-build-env` page.
|
||||
|
||||
Supported host architectures
|
||||
----------------------------
|
||||
|
||||
Those hosts are officially supported, with various accelerators:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
|
||||
* - CPU Architecture
|
||||
- Accelerators
|
||||
* - Arm
|
||||
- hvf, kvm, tcg, whpx, xen
|
||||
* - PPC
|
||||
- kvm, tcg
|
||||
* - RISC-V
|
||||
- kvm, tcg
|
||||
* - s390x
|
||||
- kvm, tcg
|
||||
* - SPARC
|
||||
- tcg
|
||||
* - x86
|
||||
- hvf, mshv, kvm, nvmm, tcg, whpx, xen
|
||||
|
||||
Other host architectures are not supported. It is possible to build QEMU system
|
||||
emulation on an unsupported host architecture using the configure
|
||||
``--enable-tcg-interpreter`` option to enable the TCI support, but note that
|
||||
this is very slow and is not recommended for normal use. QEMU user emulation
|
||||
requires host-specific support for signal handling, therefore TCI won't help
|
||||
on unsupported host architectures.
|
||||
|
||||
Non-supported architectures may be removed in the future following the
|
||||
:ref:`deprecation process<Deprecated features>`.
|
||||
|
||||
Linux OS, macOS, FreeBSD, NetBSD, OpenBSD
|
||||
-----------------------------------------
|
||||
|
||||
The project aims to support the most recent major version at all times for
|
||||
up to five years after its initial release. Support
|
||||
for the previous major version will be dropped 2 years after the new major
|
||||
version is released or when the vendor itself drops support, whichever comes
|
||||
first. In this context, third-party efforts to extend the lifetime of a distro
|
||||
are not considered, even when they are endorsed by the vendor (eg. Debian LTS);
|
||||
the same is true of repositories that contain packages backported from later
|
||||
releases (e.g. Debian backports). Within each major release, only the most
|
||||
recent minor release is considered.
|
||||
|
||||
For the purposes of identifying supported software versions available on Linux,
|
||||
the project will look at CentOS, Debian, Fedora, openSUSE, RHEL, SLES and
|
||||
Ubuntu LTS. Other distros will be assumed to ship similar software versions.
|
||||
|
||||
For FreeBSD and OpenBSD, decisions will be made based on the contents of the
|
||||
respective ports repository, while NetBSD will use the pkgsrc repository.
|
||||
|
||||
For macOS, `Homebrew`_ will be used, although `MacPorts`_ is expected to carry
|
||||
similar versions.
|
||||
|
||||
Some build dependencies may follow less conservative rules:
|
||||
|
||||
Python runtime
|
||||
Distributions with long-term support often provide multiple versions
|
||||
of the Python runtime. While QEMU will initially aim to support the
|
||||
distribution's default runtime, it may later increase its minimum version
|
||||
to any newer python that is available as an option from the vendor.
|
||||
In this case, it will be necessary to use the ``--python`` command line
|
||||
option of the ``configure`` script to point QEMU to a supported
|
||||
version of the Python runtime.
|
||||
|
||||
As of QEMU |version|, the minimum supported version of Python is 3.9.
|
||||
|
||||
Python build dependencies
|
||||
Some of QEMU's build dependencies are written in Python. Usually these
|
||||
are only packaged by distributions for the default Python runtime.
|
||||
If QEMU bumps its minimum Python version and a non-default runtime is
|
||||
required, it may be necessary to fetch python modules from the Python
|
||||
Package Index (PyPI) via ``pip``, in order to build QEMU.
|
||||
|
||||
Rust build dependencies
|
||||
QEMU is generally conservative in adding new Rust dependencies, and all
|
||||
of them are included in the distributed tarballs. One exception is the
|
||||
bindgen tool, which is too big to package and distribute. The minimum
|
||||
supported version of bindgen is 0.60.x. For distributions that do not
|
||||
include bindgen or have an older version, it is recommended to install
|
||||
a newer version using ``cargo install --locked bindgen-cli``.
|
||||
|
||||
QEMU requires Rust 1.83.0. This is available on all supported platforms
|
||||
except for the ``mips64el`` architecture on Debian bookworm. For all other
|
||||
architectures, Debian bookworm provides a new-enough Rust compiler
|
||||
in the ``rustc-web`` package.
|
||||
|
||||
For Ubuntu 22.04 ("Jammy") and 24.04 ("Noble") updated versions of
|
||||
Rust are available through packages such as ``rustc-1.83`` package;
|
||||
the path to ``rustc`` and ``rustdoc`` has to be provided manually to
|
||||
the configure script.
|
||||
|
||||
Some distros prefer to avoid vendored crate sources, and instead use
|
||||
local sources from e.g. ``/usr/share/cargo/registry``. QEMU includes a
|
||||
script, ``scripts/update-cargo-wraps.py``, that automatically
|
||||
performs this task. The script is meant to be invoked after unpacking
|
||||
the QEMU tarball. QEMU also includes ``rust/Cargo.toml`` and
|
||||
``rust/Cargo.lock`` files that can be used to compute QEMU's build
|
||||
dependencies, e.g. using ``cargo2rpm -p rust/Cargo.toml buildrequires``.
|
||||
|
||||
Optional build dependencies
|
||||
Build components whose absence does not affect the ability to build QEMU
|
||||
may not be available in distros, or may be too old for our requirements.
|
||||
Many of these, such as additional modules for the functional testing
|
||||
framework or various linters, are written in Python and therefore can
|
||||
also be installed using ``pip``. Cross compilers are another example
|
||||
of optional build-time dependency; in this case it is possible to
|
||||
download them from repositories such as EPEL, to use container-based
|
||||
cross compilation using ``docker`` or ``podman``, or to use pre-built
|
||||
binaries distributed with QEMU.
|
||||
|
||||
Windows
|
||||
-------
|
||||
|
||||
The project aims to support the two most recent versions of Windows that are
|
||||
still supported by the vendor. The minimum Windows API that is currently
|
||||
targeted is "Windows 8", so theoretically the QEMU binaries can still be run
|
||||
on older versions of Windows, too. However, such old versions of Windows are
|
||||
not tested anymore, so it is recommended to use one of the latest versions of
|
||||
Windows instead.
|
||||
|
||||
The project supports building QEMU with current versions of the MinGW
|
||||
toolchain, either hosted on Linux (Debian/Fedora) or via `MSYS2`_ on Windows.
|
||||
A more recent Windows version is always preferred as it is less likely to have
|
||||
problems with building via MSYS2. The building process of QEMU involves some
|
||||
Python scripts that call os.symlink() which needs special attention for the
|
||||
build process to successfully complete. On newer versions of Windows 10,
|
||||
unprivileged accounts can create symlinks if Developer Mode is enabled.
|
||||
When Developer Mode is not available/enabled, the SeCreateSymbolicLinkPrivilege
|
||||
privilege is required, or the process must be run as an administrator.
|
||||
|
||||
Only 64-bit Windows is supported.
|
||||
|
||||
.. _Homebrew: https://brew.sh/
|
||||
.. _MacPorts: https://www.macports.org/
|
||||
.. _MSYS2: https://www.msys2.org/
|
||||
.. _Repology: https://repology.org/
|
||||
@@ -0,0 +1,479 @@
|
||||
.. _Deprecated features:
|
||||
|
||||
Deprecated features
|
||||
===================
|
||||
|
||||
In general features are intended to be supported indefinitely once
|
||||
introduced into QEMU. In the event that a feature needs to be removed,
|
||||
it will be listed in this section. The feature will remain functional for the
|
||||
release in which it was deprecated and one further release. After these two
|
||||
releases, the feature is liable to be removed. Deprecated features may also
|
||||
generate warnings on the console when QEMU starts up, or if activated via a
|
||||
monitor command, however, this is not a mandatory requirement.
|
||||
|
||||
As a special exception to this general timeframe, rather than have an
|
||||
indefinite lifetime, versioned machine types are only intended to be
|
||||
supported for a period of 6 years, equivalent to 18 QEMU releases. All
|
||||
versioned machine types will be automatically marked deprecated after an
|
||||
initial 3 years (9 QEMU releases) has passed, and will then be deleted after
|
||||
a further 3 year period has passed. It is recommended that a deprecated
|
||||
machine type is only used for incoming migrations and restore of saved state,
|
||||
for pre-existing VM deployments. They should be scheduled for updating to a
|
||||
newer machine type during an appropriate service window. Newly deployed VMs
|
||||
should exclusively use a non-deprecated machine type, with use of the most
|
||||
recent version highly recommended. Non-versioned machine types follow the
|
||||
general feature deprecation policy.
|
||||
|
||||
What follows is a list of all features currently marked as
|
||||
deprecated.
|
||||
|
||||
System emulator command line arguments
|
||||
--------------------------------------
|
||||
|
||||
Short-form boolean options (since 6.0)
|
||||
''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
Boolean options such as ``share=on``/``share=off`` could be written
|
||||
in short form as ``share`` and ``noshare``. This is now deprecated
|
||||
and will cause a warning.
|
||||
|
||||
``delay`` option for socket character devices (since 6.0)
|
||||
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
The replacement for the ``nodelay`` short-form boolean option is ``nodelay=on``
|
||||
rather than ``delay=off``.
|
||||
|
||||
Plugin argument passing through ``arg=<string>`` (since 6.1)
|
||||
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
Passing TCG plugins arguments through ``arg=`` is redundant is makes the
|
||||
command-line less readable, especially when the argument itself consist of a
|
||||
name and a value, e.g. ``-plugin plugin_name,arg="arg_name=arg_value"``.
|
||||
Therefore, the usage of ``arg`` is redundant. Single-word arguments are treated
|
||||
as short-form boolean values, and passed to plugins as ``arg_name=on``.
|
||||
However, short-form booleans are deprecated and full explicit ``arg_name=on``
|
||||
form is preferred.
|
||||
|
||||
``debug-threads`` option for ``-name`` (since 11.0)
|
||||
'''''''''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
The ``debug-threads`` option of the ``-name`` argument is now
|
||||
ignored. Thread naming is unconditionally enabled for all platforms
|
||||
where it is supported.
|
||||
|
||||
``-mon`` option (since 11.1)
|
||||
''''''''''''''''''''''''''''
|
||||
|
||||
The ``-mon`` option was the generic mechanism for creating monitor
|
||||
objects if the convenience ``-qmp`` or ``-monitor`` options were not
|
||||
flexible enough. The monitor objects have been converted to QOM, so
|
||||
``-mon mode=readline`` is replaced by ``-object monitor-hmp`` and
|
||||
``-mon mode=control`` is replaced by ``-object monitor-qmp``. The
|
||||
short convenience options are not deprecated, only ``-mon``.
|
||||
|
||||
QEMU Machine Protocol (QMP) commands
|
||||
------------------------------------
|
||||
|
||||
``blockdev-open-tray``, ``blockdev-close-tray`` argument ``device`` (since 2.8)
|
||||
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
Use argument ``id`` instead.
|
||||
|
||||
``eject`` argument ``device`` (since 2.8)
|
||||
'''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
Use argument ``id`` instead.
|
||||
|
||||
``blockdev-change-medium`` argument ``device`` (since 2.8)
|
||||
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
Use argument ``id`` instead.
|
||||
|
||||
``block_set_io_throttle`` argument ``device`` (since 2.8)
|
||||
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
Use argument ``id`` instead.
|
||||
|
||||
``blockdev-add`` empty string argument ``backing`` (since 2.10)
|
||||
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
Use argument value ``null`` instead.
|
||||
|
||||
``block-commit`` arguments ``base`` and ``top`` (since 3.1)
|
||||
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
Use arguments ``base-node`` and ``top-node`` instead.
|
||||
|
||||
``nbd-server-add`` and ``nbd-server-remove`` (since 5.2)
|
||||
''''''''''''''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
Use the more generic commands ``block-export-add`` and ``block-export-del``
|
||||
instead. As part of this deprecation, where ``nbd-server-add`` used a
|
||||
single ``bitmap``, the new ``block-export-add`` uses a list of ``bitmaps``.
|
||||
|
||||
``query-qmp-schema`` return value member ``values`` (since 6.2)
|
||||
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
Member ``values`` in return value elements with meta-type ``enum`` is
|
||||
deprecated. Use ``members`` instead.
|
||||
|
||||
``drive-backup`` (since 6.2)
|
||||
''''''''''''''''''''''''''''
|
||||
|
||||
Use ``blockdev-backup`` in combination with ``blockdev-add`` instead.
|
||||
This change primarily separates the creation/opening process of the backup
|
||||
target with explicit, separate steps. ``blockdev-backup`` uses mostly the
|
||||
same arguments as ``drive-backup``, except the ``format`` and ``mode``
|
||||
options are removed in favor of using explicit ``blockdev-create`` and
|
||||
``blockdev-add`` calls. See :doc:`/interop/live-block-operations` for
|
||||
details.
|
||||
|
||||
``block-job-pause`` (since 10.1)
|
||||
''''''''''''''''''''''''''''''''
|
||||
|
||||
Use ``job-pause`` instead. The only difference is that ``job-pause``
|
||||
always reports GenericError on failure when ``block-job-pause`` reports
|
||||
DeviceNotActive when block-job is not found.
|
||||
|
||||
``block-job-resume`` (since 10.1)
|
||||
'''''''''''''''''''''''''''''''''
|
||||
|
||||
Use ``job-resume`` instead. The only difference is that ``job-resume``
|
||||
always reports GenericError on failure when ``block-job-resume`` reports
|
||||
DeviceNotActive when block-job is not found.
|
||||
|
||||
``block-job-complete`` (since 10.1)
|
||||
'''''''''''''''''''''''''''''''''''
|
||||
|
||||
Use ``job-complete`` instead. The only difference is that ``job-complete``
|
||||
always reports GenericError on failure when ``block-job-complete`` reports
|
||||
DeviceNotActive when block-job is not found.
|
||||
|
||||
``block-job-dismiss`` (since 10.1)
|
||||
''''''''''''''''''''''''''''''''''
|
||||
|
||||
Use ``job-dismiss`` instead.
|
||||
|
||||
``block-job-finalize`` (since 10.1)
|
||||
'''''''''''''''''''''''''''''''''''
|
||||
|
||||
Use ``job-finalize`` instead.
|
||||
|
||||
``query-kvm`` (since 11.0)
|
||||
''''''''''''''''''''''''''
|
||||
|
||||
Use ``query-accelerators`` instead.
|
||||
|
||||
Human Machine Protocol (HMP) commands
|
||||
-------------------------------------
|
||||
|
||||
``wavcapture`` (since 10.2)
|
||||
''''''''''''''''''''''''''''
|
||||
|
||||
The ``wavcapture`` command is deprecated and will be removed in a future release.
|
||||
|
||||
Use ``-audiodev wav`` or your host audio system to capture audio.
|
||||
|
||||
``stopcapture`` (since 10.2)
|
||||
''''''''''''''''''''''''''''
|
||||
|
||||
The ``stopcapture`` command is deprecated and will be removed in a future release.
|
||||
|
||||
``info`` argument ``capture`` (since 10.2)
|
||||
''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
The ``info capture`` command is deprecated and will be removed in a future release.
|
||||
|
||||
Host Architectures
|
||||
------------------
|
||||
|
||||
TCG Plugin support not enabled by default with TCI (since 9.2)
|
||||
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
While the TCG interpreter can interpret the TCG ops used by plugins it
|
||||
is going to be so much slower it wouldn't make sense for any serious
|
||||
instrumentation. Due to implementation differences there will also be
|
||||
anomalies in things like memory instrumentation.
|
||||
|
||||
System emulator CPUs
|
||||
--------------------
|
||||
|
||||
``power5+`` and ``power7+`` CPU names (since 9.0)
|
||||
'''''''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
The character "+" in device (and thus also CPU) names is not allowed
|
||||
in the QEMU object model anymore. ``power5+``, ``power5+_v2.1``,
|
||||
``power7+`` and ``power7+_v2.1`` are currently still supported via
|
||||
an alias, but for consistency these will get removed in a future
|
||||
release, too. Use ``power5p_v2.1`` and ``power7p_v2.1`` instead.
|
||||
|
||||
``Sun-UltraSparc-IIIi+`` and ``Sun-UltraSparc-IV+`` CPU names (since 9.1)
|
||||
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
The character "+" in device (and thus also CPU) names is not allowed
|
||||
in the QEMU object model anymore. ``Sun-UltraSparc-IIIi+`` and
|
||||
``Sun-UltraSparc-IV+`` are currently still supported via a workaround,
|
||||
but for consistency these will get removed in a future release, too.
|
||||
Use ``Sun-UltraSparc-IIIi-plus`` and ``Sun-UltraSparc-IV-plus`` instead.
|
||||
|
||||
System emulator machines
|
||||
------------------------
|
||||
|
||||
Versioned machine types (aarch64, arm, i386, m68k, ppc64, s390x, x86_64)
|
||||
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
In accordance with our versioned machine type deprecation policy, all machine
|
||||
types with version |VER_MACHINE_DEPRECATION_VERSION|, or older, have been
|
||||
deprecated.
|
||||
|
||||
Arm ``virt`` machine ``dtb-kaslr-seed`` property (since 7.1)
|
||||
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
The ``dtb-kaslr-seed`` property on the ``virt`` board has been
|
||||
deprecated; use the new name ``dtb-randomness`` instead. The new name
|
||||
better reflects the way this property affects all random data within
|
||||
the device tree blob, not just the ``kaslr-seed`` node.
|
||||
|
||||
RISC-V Shakti machine (since 11.1)
|
||||
''''''''''''''''''''''''''''''''''
|
||||
|
||||
The RISC-V ``shakti_c`` machine hasn't had meaningful contributions since 2021
|
||||
and is currently unmaintained. The machine is scheduled to be removed as it
|
||||
appears to have no users.
|
||||
|
||||
``memory-encryption`` machine property (since 11.1)
|
||||
'''''''''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
Use ``confidential-guest-support`` instead. The ``memory-encryption`` object
|
||||
was an early implementation of memory encryption support in QEMU, but it has
|
||||
been superseded by the more comprehensive ``confidential-guest-support``
|
||||
object.
|
||||
|
||||
Backend options
|
||||
---------------
|
||||
|
||||
Using non-persistent backing file with pmem=on (since 6.1)
|
||||
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
This option is used when ``memory-backend-file`` is consumed by emulated NVDIMM
|
||||
device. However enabling ``memory-backend-file.pmem`` option, when backing file
|
||||
is (a) not DAX capable or (b) not on a filesystem that support direct mapping
|
||||
of persistent memory, is not safe and may lead to data loss or corruption in case
|
||||
of host crash.
|
||||
Options are:
|
||||
|
||||
- modify VM configuration to set ``pmem=off`` to continue using fake NVDIMM
|
||||
(without persistence guaranties) with backing file on non DAX storage
|
||||
- move backing file to NVDIMM storage and keep ``pmem=on``
|
||||
(to have NVDIMM with persistence guaranties).
|
||||
|
||||
Using an external DH (Diffie-Hellman) parameters file (since 10.2)
|
||||
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
Loading of external Diffie-Hellman parameters from a 'dh-params.pem'
|
||||
file is deprecated and will be removed with no replacement in a
|
||||
future release. Where no 'dh-params.pem' file is provided, the DH
|
||||
parameters will be automatically negotiated in accordance with
|
||||
RFC7919.
|
||||
|
||||
Devices
|
||||
-------
|
||||
|
||||
``virtio-crypto`` and cryptodev backends (since 11.2)
|
||||
'''''''''''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
The ``virtio-crypto`` device emulation is quite complex code with a
|
||||
number of known flaws. It has never been migratable, so it is
|
||||
unlikely to be used in any serious virtualization setting. Modern
|
||||
ISAs provide on-CPU cryptography instructions (e.g. AES-NI/VAES,
|
||||
armv8 crypto extensions), and the Linux kernel deprecated AF_ALG
|
||||
and dropped its off-CPU accelerator support in Linux 7.2, removing
|
||||
the primary userspace path for off-CPU crypto acceleration. The
|
||||
time for this sort of off-load has passed.
|
||||
|
||||
As this is the only device that uses the cryptodev backends these will
|
||||
be removed at the same time the ``virtio-crypto`` device is.
|
||||
|
||||
Device options
|
||||
--------------
|
||||
|
||||
Emulated device options
|
||||
'''''''''''''''''''''''
|
||||
|
||||
``-device nvme-ns,eui64-default=on|off`` (since 7.1)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
In QEMU versions 6.1, 6.2 and 7.0, the ``nvme-ns`` generates an EUI-64
|
||||
identifier that is not globally unique. If an EUI-64 identifier is required, the
|
||||
user must set it explicitly using the ``nvme-ns`` device parameter ``eui64``.
|
||||
|
||||
``-device nvme,use-intel-id=on|off`` (since 7.1)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The ``nvme`` device originally used a PCI Vendor/Device Identifier combination
|
||||
from Intel that was not properly allocated. Since version 5.2, the controller
|
||||
has used a properly allocated identifier. Deprecate the ``use-intel-id``
|
||||
machine compatibility parameter.
|
||||
|
||||
``-device cxl-type3,memdev=xxxx`` (since 8.0)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The ``cxl-type3`` device initially only used a single memory backend. With
|
||||
the addition of volatile memory support, it is now necessary to distinguish
|
||||
between persistent and volatile memory backends. As such, memdev is deprecated
|
||||
in favor of persistent-memdev.
|
||||
|
||||
|
||||
RISC-V CPU properties which start with capital 'Z' (since 8.2)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
All RISC-V CPU properties which start with capital 'Z' are being deprecated
|
||||
starting in 8.2. The reason is that they were wrongly added with capital 'Z'
|
||||
in the past. CPU properties were later added with lower-case names, which
|
||||
is the format we want to use from now on.
|
||||
|
||||
Users which try to use these deprecated properties will receive a warning
|
||||
recommending to switch to their stable counterparts:
|
||||
|
||||
- "Zifencei" should be replaced with "zifencei"
|
||||
- "Zicsr" should be replaced with "zicsr"
|
||||
- "Zihintntl" should be replaced with "zihintntl"
|
||||
- "Zihintpause" should be replaced with "zihintpause"
|
||||
- "Zawrs" should be replaced with "zawrs"
|
||||
- "Zfa" should be replaced with "zfa"
|
||||
- "Zfh" should be replaced with "zfh"
|
||||
- "Zfhmin" should be replaced with "zfhmin"
|
||||
- "Zve32f" should be replaced with "zve32f"
|
||||
- "Zve64f" should be replaced with "zve64f"
|
||||
- "Zve64d" should be replaced with "zve64d"
|
||||
|
||||
Block device options
|
||||
''''''''''''''''''''
|
||||
|
||||
``"backing": ""`` (since 2.12)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
In order to prevent QEMU from automatically opening an image's backing
|
||||
chain, use ``"backing": null`` instead.
|
||||
|
||||
``rbd`` keyvalue pair encoded filenames: ``""`` (since 3.1)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Options for ``rbd`` should be specified according to its runtime options,
|
||||
like other block drivers. Legacy parsing of keyvalue pair encoded
|
||||
filenames is useful to open images with the old format for backing files;
|
||||
These image files should be updated to use the current format.
|
||||
|
||||
Example of legacy encoding::
|
||||
|
||||
json:{"file.driver":"rbd", "file.filename":"rbd:rbd/name"}
|
||||
|
||||
The above, converted to the current supported format::
|
||||
|
||||
json:{"file.driver":"rbd", "file.pool":"rbd", "file.image":"name"}
|
||||
|
||||
``iscsi,password=xxx`` (since 8.0)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Specifying the iSCSI password in plain text on the command line using the
|
||||
``password`` option is insecure. The ``password-secret`` option should be
|
||||
used instead, to refer to a ``--object secret...`` instance that provides
|
||||
a password via a file, or encrypted.
|
||||
|
||||
|
||||
Character device options
|
||||
''''''''''''''''''''''''
|
||||
|
||||
Backend ``memory`` (since 9.0)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
``memory`` is a deprecated synonym for ``ringbuf``.
|
||||
|
||||
|
||||
CPU device properties
|
||||
'''''''''''''''''''''
|
||||
|
||||
``pmu-num=n`` on RISC-V CPUs (since 8.2)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
In order to support more flexible counter configurations this has been replaced
|
||||
by a ``pmu-mask`` property. If set of counters is continuous then the mask can
|
||||
be calculated with ``((2 ^ n) - 1) << 3``. The least significant three bits
|
||||
must be left clear.
|
||||
|
||||
|
||||
``pcommit`` on x86 (since 9.1)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The PCOMMIT instruction was never included in any physical processor.
|
||||
It was implemented as a no-op instruction in TCG up to QEMU 9.0, but
|
||||
only with ``-cpu max`` (which does not guarantee migration compatibility
|
||||
across versions).
|
||||
|
||||
linux-user mode CPUs
|
||||
--------------------
|
||||
|
||||
OABI and NWFPE support for Arm CPUs
|
||||
'''''''''''''''''''''''''''''''''''
|
||||
|
||||
Linux for 32-bit Arm has had two major ABIs: the original OABI and the
|
||||
more modern EABI. OABI support was marked as obsolete in GCC 4.7 and
|
||||
dropped in GCC 4.8 (released in 2013). In the Linux kernel,
|
||||
compatibility handling for OABI (OABI_COMPAT) is not generally enabled
|
||||
by default and is not compatible with building a Thumb2
|
||||
kernel. Distros dropped OABI support fifteen years or more ago.
|
||||
|
||||
The original floating-point coprocessor for 32-bit Arm was the
|
||||
FPA11. This was not present in many CPUs but did get baked into the
|
||||
OABI for how to pass floating point arguments, and so the Linux kernel
|
||||
has support for emulating it via the config option FPE_NWFPE; QEMU
|
||||
follows that. FPA11 support was also removed from GCC in GCC 4.8.
|
||||
|
||||
QEMU's NWFPE code is old and untested and not thread-safe; the OABI
|
||||
ABI is long-obsolete. We are therefore deprecating both OABI support
|
||||
and NWFPE emulation, and they will be removed in a future QEMU
|
||||
release.
|
||||
|
||||
|
||||
Backwards compatibility
|
||||
-----------------------
|
||||
|
||||
Runnability guarantee of CPU models (since 4.1)
|
||||
'''''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
Previous versions of QEMU never changed existing CPU models in
|
||||
ways that introduced additional host software or hardware
|
||||
requirements to the VM. This allowed management software to
|
||||
safely change the machine type of an existing VM without
|
||||
introducing new requirements ("runnability guarantee"). This
|
||||
prevented CPU models from being updated to include CPU
|
||||
vulnerability mitigations, leaving guests vulnerable in the
|
||||
default configuration.
|
||||
|
||||
The CPU model runnability guarantee won't apply anymore to
|
||||
existing CPU models. Management software that needs runnability
|
||||
guarantees must resolve the CPU model aliases using the
|
||||
``alias-of`` field returned by the ``query-cpu-definitions`` QMP
|
||||
command.
|
||||
|
||||
While those guarantees are kept, the return value of
|
||||
``query-cpu-definitions`` will have existing CPU model aliases
|
||||
point to a version that doesn't break runnability guarantees
|
||||
(specifically, version 1 of those CPU models). In future QEMU
|
||||
versions, aliases will point to newer CPU model versions
|
||||
depending on the machine type, so management software must
|
||||
resolve CPU model aliases before starting a virtual machine.
|
||||
|
||||
x86 "isapc" board use of modern x86 CPUs (since 10.2)
|
||||
'''''''''''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
The "isapc" board represents a historical x86 ISA PC and is intended for
|
||||
older 32-bit x86 CPU models, defaulting to a 486 CPU model. Previously it
|
||||
was possible (but non-sensical) to specify a more modern x86 CPU, including
|
||||
``-cpu host`` or ``-cpu max`` even if the features were incompatible with many
|
||||
of the intended guest OSs.
|
||||
|
||||
If the user requests a modern x86 CPU model (i.e. not one of ``486``,
|
||||
``athlon``, ``kvm32``, ``pentium``, ``pentium2``, ``pentium3``or ``qemu32``)
|
||||
a warning will be displayed until a future QEMU version when such CPUs will
|
||||
be rejected.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,29 @@
|
||||
----------
|
||||
About QEMU
|
||||
----------
|
||||
|
||||
QEMU is a generic and open source machine emulator and virtualizer.
|
||||
|
||||
QEMU can be used in several different ways. The most common is for
|
||||
:ref:`System Emulation`, where it provides a virtual model of an
|
||||
entire machine (CPU, memory and emulated devices) to run a guest OS.
|
||||
In this mode the CPU may be fully emulated, or it may work with a
|
||||
hypervisor such as KVM, Xen or Hypervisor.Framework to allow the
|
||||
guest to run directly on the host CPU.
|
||||
|
||||
The second supported way to use QEMU is :ref:`User Mode Emulation`,
|
||||
where QEMU can launch processes compiled for one CPU on another CPU.
|
||||
In this mode the CPU is always emulated.
|
||||
|
||||
QEMU also provides a number of standalone :ref:`command line
|
||||
utilities<Tools>`, such as the ``qemu-img`` disk image utility that
|
||||
allows you to create, convert and modify disk images.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
build-platforms
|
||||
emulation
|
||||
deprecated
|
||||
removed-features
|
||||
license
|
||||
@@ -0,0 +1,11 @@
|
||||
.. _License:
|
||||
|
||||
License
|
||||
=======
|
||||
|
||||
QEMU is a trademark of Fabrice Bellard.
|
||||
|
||||
QEMU is released under the `GNU General Public
|
||||
License <https://www.gnu.org/licenses/gpl-2.0.txt>`__, version 2. Parts
|
||||
of QEMU have specific licenses, see file
|
||||
`LICENSE <https://gitlab.com/qemu-project/qemu/-/raw/master/LICENSE>`__.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,247 @@
|
||||
Block replication
|
||||
----------------------------------------
|
||||
Copyright Fujitsu, Corp. 2016
|
||||
Copyright (c) 2016 Intel Corporation
|
||||
Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD.
|
||||
|
||||
This work is licensed under the terms of the GNU GPL, version 2 or later.
|
||||
See the COPYING file in the top-level directory.
|
||||
|
||||
Block replication is used for continuous checkpoints. It is designed
|
||||
for COLO (COarse-grain LOck-stepping) where the Secondary VM is running.
|
||||
It can also be applied for FT/HA (Fault-tolerance/High Assurance) scenario,
|
||||
where the Secondary VM is not running.
|
||||
|
||||
This document gives an overview of block replication's design.
|
||||
|
||||
== Background ==
|
||||
High availability solutions such as micro checkpoint and COLO will do
|
||||
consecutive checkpoints. The VM state of the Primary and Secondary VM is
|
||||
identical right after a VM checkpoint, but becomes different as the VM
|
||||
executes till the next checkpoint. To support disk contents checkpoint,
|
||||
the modified disk contents in the Secondary VM must be buffered, and are
|
||||
only dropped at next checkpoint time. To reduce the network transportation
|
||||
effort during a vmstate checkpoint, the disk modification operations of
|
||||
the Primary disk are asynchronously forwarded to the Secondary node.
|
||||
|
||||
== Workflow ==
|
||||
The following is the image of block replication workflow:
|
||||
|
||||
+----------------------+ +------------------------+
|
||||
|Primary Write Requests| |Secondary Write Requests|
|
||||
+----------------------+ +------------------------+
|
||||
| |
|
||||
| (4)
|
||||
| V
|
||||
| /-------------\
|
||||
| Copy and Forward | |
|
||||
|---------(1)----------+ | Disk Buffer |
|
||||
| | | |
|
||||
| (3) \-------------/
|
||||
| speculative ^
|
||||
| write through (2)
|
||||
| | |
|
||||
V V |
|
||||
+--------------+ +----------------+
|
||||
| Primary Disk | | Secondary Disk |
|
||||
+--------------+ +----------------+
|
||||
|
||||
1) Primary write requests will be copied and forwarded to Secondary
|
||||
QEMU.
|
||||
2) Before Primary write requests are written to Secondary disk, the
|
||||
original sector content will be read from Secondary disk and
|
||||
buffered in the Disk buffer, but it will not overwrite the existing
|
||||
sector content (it could be from either "Secondary Write Requests" or
|
||||
previous COW of "Primary Write Requests") in the Disk buffer.
|
||||
3) Primary write requests will be written to Secondary disk.
|
||||
4) Secondary write requests will be buffered in the Disk buffer and it
|
||||
will overwrite the existing sector content in the buffer.
|
||||
|
||||
== Architecture ==
|
||||
We are going to implement block replication from many basic
|
||||
blocks that are already in QEMU.
|
||||
|
||||
virtio-blk ||
|
||||
^ || .----------
|
||||
| || | Secondary
|
||||
1 Quorum || '----------
|
||||
/ \ || virtio-blk
|
||||
/ \ || ^
|
||||
Primary 2 filter |
|
||||
disk ^ 7 Quorum
|
||||
| /
|
||||
3 NBD -------> 3 NBD /
|
||||
client || server 2 filter
|
||||
|| ^ ^
|
||||
--------. || | |
|
||||
Primary | || Secondary disk <--------- hidden-disk 5 <--------- active-disk 4
|
||||
--------' || | backing ^ backing
|
||||
|| | |
|
||||
|| | |
|
||||
|| '-------------------------'
|
||||
|| blockdev-backup sync=none 6
|
||||
|
||||
1) The disk on the primary is represented by a block device with two
|
||||
children, providing replication between a primary disk and the host that
|
||||
runs the secondary VM. The read pattern (fifo) for quorum can be extended
|
||||
to make the primary always read from the local disk instead of going through
|
||||
NBD.
|
||||
|
||||
2) The new block filter (the name is replication) will control the block
|
||||
replication.
|
||||
|
||||
3) The secondary disk receives writes from the primary VM through QEMU's
|
||||
embedded NBD server (speculative write-through).
|
||||
|
||||
4) The disk on the secondary is represented by a custom block device
|
||||
(called active-disk). It should start as an empty disk, and the format
|
||||
should support bdrv_make_empty() and backing file.
|
||||
|
||||
5) The hidden-disk is created automatically. It buffers the original content
|
||||
that is modified by the primary VM. It should also start as an empty disk,
|
||||
and the driver supports bdrv_make_empty() and backing file.
|
||||
|
||||
6) The blockdev-backup job (sync=none) is run to allow hidden-disk to buffer
|
||||
any state that would otherwise be lost by the speculative write-through
|
||||
of the NBD server into the secondary disk. So before block replication,
|
||||
the primary disk and secondary disk should contain the same data.
|
||||
|
||||
7) The secondary also has a quorum node, so after secondary failover it
|
||||
can become the new primary and continue replication.
|
||||
|
||||
|
||||
== Failure Handling ==
|
||||
There are 7 internal errors when block replication is running:
|
||||
1. I/O error on primary disk
|
||||
2. Forwarding primary write requests failed
|
||||
3. Backup failed
|
||||
4. I/O error on secondary disk
|
||||
5. I/O error on active disk
|
||||
6. Making active disk or hidden disk empty failed
|
||||
7. Doing failover failed
|
||||
In case 1 and 5, we just report the error to the disk layer. In case 2, 3,
|
||||
4 and 6, we just report block replication's error to FT/HA manager (which
|
||||
decides when to do a new checkpoint, when to do failover).
|
||||
In case 7, if active commit failed, we use replication failover failed state
|
||||
in Secondary's write operation (what decides which target to write).
|
||||
|
||||
== New block driver interface ==
|
||||
We add four block driver interfaces to control block replication:
|
||||
a. replication_start_all()
|
||||
Start block replication, called in migration/checkpoint thread.
|
||||
We must call block_replication_start_all() in secondary QEMU before
|
||||
calling block_replication_start_all() in primary QEMU. The caller
|
||||
must hold the I/O mutex lock if it is in migration/checkpoint
|
||||
thread.
|
||||
b. replication_do_checkpoint_all()
|
||||
This interface is called after all VM state is transferred to
|
||||
Secondary QEMU. The Disk buffer will be dropped in this interface.
|
||||
The caller must hold the I/O mutex lock if it is in migration/checkpoint
|
||||
thread.
|
||||
c. replication_get_error_all()
|
||||
This interface is called to check if error happened in replication.
|
||||
The caller must hold the I/O mutex lock if it is in migration/checkpoint
|
||||
thread.
|
||||
d. replication_stop_all()
|
||||
It is called on failover. We will flush the Disk buffer into
|
||||
Secondary Disk and stop block replication. The vm should be stopped
|
||||
before calling it if you use this API to shutdown the guest, or other
|
||||
things except failover. The caller must hold the I/O mutex lock if it is
|
||||
in migration/checkpoint thread.
|
||||
|
||||
== Usage ==
|
||||
Primary:
|
||||
-drive if=xxx,driver=quorum,read-pattern=fifo,id=colo1,vote-threshold=1,\
|
||||
children.0.file.filename=1.raw,\
|
||||
children.0.driver=raw
|
||||
|
||||
Run qmp command in primary qemu:
|
||||
{ "execute": "human-monitor-command",
|
||||
"arguments": {
|
||||
"command-line": "drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=xxxx,file.port=xxxx,file.export=colo1,node-name=nbd_client1"
|
||||
}
|
||||
}
|
||||
{ "execute": "x-blockdev-change",
|
||||
"arguments": {
|
||||
"parent": "colo1",
|
||||
"node": "nbd_client1"
|
||||
}
|
||||
}
|
||||
Note:
|
||||
1. There should be only one NBD Client for each primary disk.
|
||||
2. host is the secondary physical machine's hostname or IP
|
||||
3. Each disk must have its own export name.
|
||||
4. It is all a single argument to -drive and you should ignore the
|
||||
leading whitespace.
|
||||
5. The qmp command line must be run after running qmp command line in
|
||||
secondary qemu.
|
||||
6. After primary failover we need remove children.1 (replication driver).
|
||||
|
||||
Secondary:
|
||||
-drive if=none,driver=raw,file.filename=1.raw,id=colo1 \
|
||||
-drive if=none,id=childs1,driver=replication,mode=secondary,top-id=top-disk1
|
||||
file.file.filename=active_disk.qcow2,\
|
||||
file.driver=qcow2,\
|
||||
file.backing.file.filename=hidden_disk.qcow2,\
|
||||
file.backing.driver=qcow2,\
|
||||
file.backing.backing=colo1
|
||||
-drive if=xxx,driver=quorum,read-pattern=fifo,id=top-disk1,\
|
||||
vote-threshold=1,children.0=childs1
|
||||
|
||||
Then run qmp command in secondary qemu:
|
||||
{ "execute": "nbd-server-start",
|
||||
"arguments": {
|
||||
"addr": {
|
||||
"type": "inet",
|
||||
"data": {
|
||||
"host": "xxx",
|
||||
"port": "xxx"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
{ "execute": "nbd-server-add",
|
||||
"arguments": {
|
||||
"device": "colo1",
|
||||
"writable": true
|
||||
}
|
||||
}
|
||||
|
||||
Note:
|
||||
1. The export name in secondary QEMU command line is the secondary
|
||||
disk's id.
|
||||
2. The export name for the same disk must be the same
|
||||
3. The qmp command nbd-server-start and nbd-server-add must be run
|
||||
before running the qmp command migrate on primary QEMU
|
||||
4. Active disk, hidden disk and nbd target's length should be the
|
||||
same.
|
||||
5. It is better to put active disk and hidden disk in ramdisk.
|
||||
6. It is all a single argument to -drive, and you should ignore
|
||||
the leading whitespace.
|
||||
|
||||
After Failover:
|
||||
Primary:
|
||||
The secondary host is down, so we should run the following qmp command
|
||||
to remove the nbd child from the quorum:
|
||||
{ "execute": "x-blockdev-change",
|
||||
"arguments": {
|
||||
"parent": "colo1",
|
||||
"child": "children.1"
|
||||
}
|
||||
}
|
||||
{ "execute": "human-monitor-command",
|
||||
"arguments": {
|
||||
"command-line": "drive_del xxxx"
|
||||
}
|
||||
}
|
||||
Note: there is no qmp command to remove the blockdev now
|
||||
|
||||
Secondary:
|
||||
The primary host is down, so we should do the following thing:
|
||||
{ "execute": "nbd-server-stop" }
|
||||
|
||||
Promote Secondary to Primary:
|
||||
see COLO-FT.txt
|
||||
|
||||
TODO:
|
||||
1. Shared disk
|
||||
@@ -0,0 +1,89 @@
|
||||
BYPASS IOMMU PROPERTY
|
||||
=====================
|
||||
|
||||
Description
|
||||
===========
|
||||
Traditionally, there is a global switch to enable/disable vIOMMU. All
|
||||
devices in the system can only support go through vIOMMU or not, which
|
||||
is not flexible. We introduce this bypass iommu property to support
|
||||
coexist of devices go through vIOMMU and devices not. This is useful to
|
||||
passthrough devices with no-iommu mode and devices go through vIOMMU in
|
||||
the same virtual machine.
|
||||
|
||||
PCI host bridges have a bypass_iommu property. This property is used to
|
||||
determine whether the devices attached on the PCI host bridge will bypass
|
||||
virtual iommu. The bypass_iommu property is valid only when there is a
|
||||
virtual iommu in the system, it is implemented to allow some devices to
|
||||
bypass vIOMMU. When bypass_iommu property is not set for a host bridge,
|
||||
the attached devices will go through vIOMMU by default.
|
||||
|
||||
Usage
|
||||
=====
|
||||
The bypass iommu feature support PXB host bridge and default main host
|
||||
bridge, we add a bypass_iommu property for PXB and default_bus_bypass_iommu
|
||||
for machine. Note that default_bus_bypass_iommu is available only when
|
||||
the 'q35' machine type on x86 architecture and the 'virt' machine type
|
||||
on AArch64. Other machine types do not support bypass iommu for default
|
||||
root bus.
|
||||
|
||||
1. The following is the bypass iommu options:
|
||||
(1) PCI expander bridge
|
||||
qemu -device pxb-pcie,bus_nr=0x10,addr=0x1,bypass_iommu=true
|
||||
(2) Arm default host bridge
|
||||
qemu -machine virt,iommu=smmuv3,default_bus_bypass_iommu=true
|
||||
(3) X86 default root bus bypass iommu:
|
||||
qemu -machine q35,default_bus_bypass_iommu=true
|
||||
|
||||
2. Here is the detailed qemu command line for 'virt' machine with PXB on
|
||||
AArch64:
|
||||
|
||||
qemu-system-aarch64 \
|
||||
-machine virt,kernel_irqchip=on,iommu=smmuv3,default_bus_bypass_iommu=true \
|
||||
-device pxb-pcie,bus_nr=0x10,id=pci.10,bus=pcie.0,addr=0x3.0x1 \
|
||||
-device pxb-pcie,bus_nr=0x20,id=pci.20,bus=pcie.0,addr=0x3.0x2,bypass_iommu=true \
|
||||
|
||||
And we got:
|
||||
- a default host bridge which bypass SMMUv3
|
||||
- a pxb host bridge which go through SMMUv3
|
||||
- a pxb host bridge which bypass SMMUv3
|
||||
|
||||
3. Here is the detailed qemu command line for 'q35' machine with PXB on
|
||||
x86 architecture:
|
||||
|
||||
qemu-system-x86_64 \
|
||||
-machine q35,accel=kvm,default_bus_bypass_iommu=true \
|
||||
-device pxb-pcie,bus_nr=0x10,id=pci.10,bus=pcie.0,addr=0x3 \
|
||||
-device pxb-pcie,bus_nr=0x20,id=pci.20,bus=pcie.0,addr=0x4,bypass_iommu=true \
|
||||
-device intel-iommu \
|
||||
|
||||
And we got:
|
||||
- a default host bridge which bypass iommu
|
||||
- a pxb host bridge which go through iommu
|
||||
- a pxb host bridge which bypass iommu
|
||||
|
||||
Limitations
|
||||
===========
|
||||
There might be potential security risk when devices bypass iommu, because
|
||||
devices might send malicious dma request to virtual machine if there is no
|
||||
iommu isolation. So it would be necessary to only bypass iommu for trusted
|
||||
device.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
The bypass iommu feature includes:
|
||||
- Address space
|
||||
Add bypass iommu property check of PCI Host and do not get iommu address
|
||||
space for devices bypass iommu.
|
||||
- Arm SMMUv3 support
|
||||
We traverse all PCI root bus and get bus number ranges, then build explicit
|
||||
RID mapping for devices which do not bypass iommu.
|
||||
- X86 IOMMU support
|
||||
To support Intel iommu, we traverse all PCI host bridge and get information
|
||||
of devices which do not bypass iommu, then fill the DMAR drhd struct with
|
||||
explicit device scope info. To support AMD iommu, add check of bypass iommu
|
||||
when traverse the PCI hsot bridge.
|
||||
- Machine and PXB options
|
||||
We add bypass iommu options in machine option for default root bus, and add
|
||||
option for PXB also. Note that the default value of bypass iommu is false,
|
||||
so that the devices will by default go through iommu if there exist one.
|
||||
|
||||
@@ -0,0 +1,217 @@
|
||||
COLO-proxy
|
||||
----------
|
||||
Copyright (c) 2016 Intel Corporation
|
||||
Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD.
|
||||
Copyright (c) 2016 Fujitsu, Corp.
|
||||
|
||||
This work is licensed under the terms of the GNU GPL, version 2 or later.
|
||||
See the COPYING file in the top-level directory.
|
||||
|
||||
This document gives an overview of COLO proxy's design.
|
||||
|
||||
== Background ==
|
||||
COLO-proxy is a part of COLO project. It is used
|
||||
to compare the network package to help COLO decide
|
||||
whether to do checkpoint. With COLO-proxy's help,
|
||||
COLO greatly improves the performance.
|
||||
|
||||
The filter-redirector, filter-mirror, colo-compare
|
||||
and filter-rewriter compose the COLO-proxy.
|
||||
|
||||
== Architecture ==
|
||||
|
||||
COLO-Proxy is based on qemu netfilter and it's a plugin for qemu netfilter
|
||||
(except colo-compare). It keep Secondary VM connect normally to
|
||||
client and compare packets sent by PVM with sent by SVM.
|
||||
If the packet difference, notify COLO-frame to do checkpoint and send
|
||||
all primary packet has queued. Otherwise just send the queued primary
|
||||
packet and drop the queued secondary packet.
|
||||
|
||||
Below is a COLO proxy ascii figure:
|
||||
|
||||
Primary qemu Secondary qemu
|
||||
+--------------------------------------------------------------+ +----------------------------------------------------------------+
|
||||
| +----------------------------------------------------------+ | | +-----------------------------------------------------------+ |
|
||||
| | | | | | | |
|
||||
| | guest | | | | guest | |
|
||||
| | | | | | | |
|
||||
| +-------^--------------------------+-----------------------+ | | +---------------------+--------+----------------------------+ |
|
||||
| | | | | ^ | |
|
||||
| | | | | | | |
|
||||
| | +------------------------------------------------------+ | | | |
|
||||
|netfilter| | | | | | netfilter | | |
|
||||
| +----------+ +----------------------------+ | | | +-----------------------------------------------------------+ |
|
||||
| | | | | | out | | | | | | filter execute order | |
|
||||
| | | | +-----------------------------+ | | | | | | +-------------------> | |
|
||||
| | | | | | | | | | | | | | TCP | |
|
||||
| | +-----+--+-+ +-----v----+ +-----v----+ |pri +----+----+sec| | | | +------------+ +---+----+---v+rewriter++ +------------+ | |
|
||||
| | | | | | | | |in | |in | | | | | | | | | | | | |
|
||||
| | | filter | | filter | | filter +------> colo <------+ +--------> filter +--> adjust | adjust +--> filter | | |
|
||||
| | | mirror | |redirector| |redirector| | | compare | | | | | | redirector | | ack | seq | | redirector | | |
|
||||
| | | | | | | | | | | | | | | | | | | | | | | |
|
||||
| | +----^-----+ +----+-----+ +----------+ | +---------+ | | | | +------------+ +--------+--------------+ +---+--------+ | |
|
||||
| | | tx | rx rx | | | | | tx all | rx | |
|
||||
| | | | | | | | +-----------------------------------------------------------+ |
|
||||
| | | +--------------+ | | | | | |
|
||||
| | | filter execute order | | | | | | |
|
||||
| | | +----------------> | | | +--------------------------------------------------------+ |
|
||||
| +-----------------------------------------+ | | |
|
||||
| | | | | |
|
||||
+--------------------------------------------------------------+ +----------------------------------------------------------------+
|
||||
|guest receive | guest send
|
||||
| |
|
||||
+--------+----------------------------v------------------------+
|
||||
| | NOTE: filter direction is rx/tx/all
|
||||
| tap | rx:receive packets sent to the netdev
|
||||
| | tx:receive packets sent by the netdev
|
||||
+--------------------------------------------------------------+
|
||||
|
||||
1.Guest receive packet route:
|
||||
|
||||
Primary:
|
||||
|
||||
Tap --> Mirror Client Filter
|
||||
Mirror client will send packet to guest,at the
|
||||
same time, copy and forward packet to secondary
|
||||
mirror server.
|
||||
|
||||
Secondary:
|
||||
|
||||
Mirror Server Filter --> TCP Rewriter
|
||||
If receive packet is TCP packet,we will adjust ack
|
||||
and update TCP checksum, then send to secondary
|
||||
guest. Otherwise directly send to guest.
|
||||
|
||||
2.Guest send packet route:
|
||||
|
||||
Primary:
|
||||
|
||||
Guest --> Redirect Server Filter
|
||||
Redirect server filter receive primary guest packet
|
||||
but do nothing, just pass to next filter.
|
||||
|
||||
Redirect Server Filter --> COLO-Compare
|
||||
COLO-compare receive primary guest packet then
|
||||
waiting secondary redirect packet to compare it.
|
||||
If packet same,send queued primary packet and clear
|
||||
queued secondary packet, Otherwise send primary packet
|
||||
and do checkpoint.
|
||||
|
||||
COLO-Compare --> Another Redirector Filter
|
||||
The redirector get packet from colo-compare by use
|
||||
chardev socket.
|
||||
|
||||
Redirector Filter --> Tap
|
||||
Send the packet.
|
||||
|
||||
Secondary:
|
||||
|
||||
Guest --> TCP Rewriter Filter
|
||||
If the packet is TCP packet,we will adjust seq
|
||||
and update TCP checksum. Then send it to
|
||||
redirect client filter. Otherwise directly send to
|
||||
redirect client filter.
|
||||
|
||||
Redirect Client Filter --> Redirect Server Filter
|
||||
Forward packet to primary.
|
||||
|
||||
== Components introduction ==
|
||||
|
||||
Filter-mirror is a netfilter plugin.
|
||||
It gives qemu the ability to mirror
|
||||
packets to a chardev.
|
||||
|
||||
Filter-redirector is a netfilter plugin.
|
||||
It gives qemu the ability to redirect net packet.
|
||||
Redirector can redirect filter's net packet to outdev,
|
||||
and redirect indev's packet to filter.
|
||||
|
||||
filter
|
||||
+
|
||||
redirector |
|
||||
+--------------+
|
||||
| | |
|
||||
| | |
|
||||
| | |
|
||||
indev +---------+ +----------> outdev
|
||||
| | |
|
||||
| | |
|
||||
| | |
|
||||
+--------------+
|
||||
|
|
||||
v
|
||||
filter
|
||||
|
||||
COLO-compare, we do packet comparing job.
|
||||
Packets coming from the primary char indev will be sent to outdev.
|
||||
Packets coming from the secondary char dev will be dropped after comparing.
|
||||
COLO-compare needs two input chardevs and one output chardev:
|
||||
primary_in=chardev1-id (source: primary send packet)
|
||||
secondary_in=chardev2-id (source: secondary send packet)
|
||||
outdev=chardev3-id
|
||||
|
||||
Filter-rewriter will rewrite some of secondary packet to make
|
||||
secondary guest's tcp connection established successfully.
|
||||
In this module we will rewrite tcp packet's ack to the secondary
|
||||
from primary,and rewrite tcp packet's seq to the primary from
|
||||
secondary.
|
||||
|
||||
== Usage ==
|
||||
|
||||
Here is an example using demonstration IP and port addresses to more
|
||||
clearly describe the usage.
|
||||
|
||||
Primary(ip:3.3.3.3):
|
||||
-netdev tap,id=hn0,vhost=off
|
||||
-device e1000,id=e0,netdev=hn0,mac=52:a4:00:12:78:66
|
||||
-chardev socket,id=mirror0,host=3.3.3.3,port=9003,server=on,wait=off
|
||||
-chardev socket,id=compare1,host=3.3.3.3,port=9004,server=on,wait=off
|
||||
-chardev socket,id=compare0,host=3.3.3.3,port=9001,server=on,wait=off
|
||||
-chardev socket,id=compare0-0,host=3.3.3.3,port=9001
|
||||
-chardev socket,id=compare_out,host=3.3.3.3,port=9005,server=on,wait=off
|
||||
-chardev socket,id=compare_out0,host=3.3.3.3,port=9005
|
||||
-object iothread,id=iothread1
|
||||
-object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0
|
||||
-object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out
|
||||
-object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0
|
||||
-object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0,iothread=iothread1
|
||||
|
||||
Secondary(ip:3.3.3.8):
|
||||
-netdev tap,id=hn0,vhost=off
|
||||
-device e1000,netdev=hn0,mac=52:a4:00:12:78:66
|
||||
-chardev socket,id=red0,host=3.3.3.3,port=9003
|
||||
-chardev socket,id=red1,host=3.3.3.3,port=9004
|
||||
-object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0
|
||||
-object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1
|
||||
-object filter-rewriter,id=f3,netdev=hn0,queue=all
|
||||
|
||||
If you want to use virtio-net-pci or other driver with vnet_header:
|
||||
|
||||
Primary(ip:3.3.3.3):
|
||||
-netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
|
||||
-device e1000,id=e0,netdev=hn0,mac=52:a4:00:12:78:66
|
||||
-chardev socket,id=mirror0,host=3.3.3.3,port=9003,server=on,wait=off
|
||||
-chardev socket,id=compare1,host=3.3.3.3,port=9004,server=on,wait=off
|
||||
-chardev socket,id=compare0,host=3.3.3.3,port=9001,server=on,wait=off
|
||||
-chardev socket,id=compare0-0,host=3.3.3.3,port=9001
|
||||
-chardev socket,id=compare_out,host=3.3.3.3,port=9005,server=on,wait=off
|
||||
-chardev socket,id=compare_out0,host=3.3.3.3,port=9005
|
||||
-object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0,vnet_hdr_support
|
||||
-object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out,vnet_hdr_support
|
||||
-object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0,vnet_hdr_support
|
||||
-object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0,vnet_hdr_support
|
||||
|
||||
Secondary(ip:3.3.3.8):
|
||||
-netdev tap,id=hn0,vhost=off
|
||||
-device e1000,netdev=hn0,mac=52:a4:00:12:78:66
|
||||
-chardev socket,id=red0,host=3.3.3.3,port=9003
|
||||
-chardev socket,id=red1,host=3.3.3.3,port=9004
|
||||
-object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0,vnet_hdr_support
|
||||
-object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1,vnet_hdr_support
|
||||
-object filter-rewriter,id=f3,netdev=hn0,queue=all,vnet_hdr_support
|
||||
|
||||
Note:
|
||||
a.COLO-proxy must work with COLO-frame and Block-replication.
|
||||
b.Primary COLO must be started firstly, because COLO-proxy needs
|
||||
chardev socket server running before secondary started.
|
||||
c.Filter-rewriter only rewrite tcp packet.
|
||||
+352
@@ -0,0 +1,352 @@
|
||||
# QEMU documentation build configuration file, created by
|
||||
# sphinx-quickstart on Thu Jan 31 16:40:14 2019.
|
||||
#
|
||||
# This config file can be used in one of two ways:
|
||||
# (1) as a common config file which is included by the conf.py
|
||||
# for each of QEMU's manuals: in this case sphinx-build is run multiple
|
||||
# times, once per subdirectory.
|
||||
# (2) as a top level conf file which will result in building all
|
||||
# the manuals into a single document: in this case sphinx-build is
|
||||
# run once, on the top-level docs directory.
|
||||
#
|
||||
# QEMU's makefiles take option (1), which allows us to install
|
||||
# only the ones the user cares about (in particular we don't want
|
||||
# to ship the 'devel' manual to end-users).
|
||||
# Third-party sites such as readthedocs.org will take option (2).
|
||||
#
|
||||
#
|
||||
# This file is execfile()d with the current directory set to its
|
||||
# containing dir.
|
||||
#
|
||||
# Note that not all possible configuration values are present in this
|
||||
# autogenerated file.
|
||||
#
|
||||
# All configuration values have a default; values that are commented out
|
||||
# serve to show the default.
|
||||
|
||||
import os
|
||||
import sys
|
||||
import sphinx
|
||||
from sphinx.errors import ConfigError
|
||||
|
||||
# The per-manual conf.py will set qemu_docdir for a single-manual build;
|
||||
# otherwise set it here if this is an entire-manual-set build.
|
||||
# This is always the absolute path of the docs/ directory in the source tree.
|
||||
try:
|
||||
qemu_docdir
|
||||
except NameError:
|
||||
qemu_docdir = os.path.abspath(".")
|
||||
|
||||
# If extensions (or modules to document with autodoc) are in another directory,
|
||||
# add these directories to sys.path here. If the directory is relative to the
|
||||
# documentation root, use an absolute path starting from qemu_docdir.
|
||||
#
|
||||
# Our extensions are in docs/sphinx; the qapidoc extension requires
|
||||
# the QAPI modules from scripts/.
|
||||
sys.path.insert(0, os.path.join(qemu_docdir, "sphinx"))
|
||||
sys.path.insert(0, os.path.join(qemu_docdir, "../scripts"))
|
||||
|
||||
|
||||
# -- General configuration ------------------------------------------------
|
||||
|
||||
# If your documentation needs a minimal Sphinx version, state it here.
|
||||
#
|
||||
# 3.4.3 is the oldest version of Sphinx that ships on a platform we
|
||||
# pledge build support for.
|
||||
needs_sphinx = '3.4.3'
|
||||
|
||||
# Add any Sphinx extension module names here, as strings. They can be
|
||||
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
|
||||
# ones.
|
||||
extensions = [
|
||||
'depfile',
|
||||
'hxtool',
|
||||
'kerneldoc',
|
||||
'qapi_domain',
|
||||
'qapidoc',
|
||||
'qmp_lexer',
|
||||
]
|
||||
|
||||
if sphinx.version_info[:3] > (4, 0, 0):
|
||||
tags.add('sphinx4')
|
||||
extensions += ['dbusdoc']
|
||||
else:
|
||||
extensions += ['fakedbusdoc']
|
||||
|
||||
# Add any paths that contain templates here, relative to this directory.
|
||||
templates_path = [os.path.join(qemu_docdir, '_templates')]
|
||||
|
||||
# The suffix(es) of source filenames.
|
||||
# You can specify multiple suffix as a list of string:
|
||||
#
|
||||
# source_suffix = ['.rst', '.md']
|
||||
source_suffix = '.rst'
|
||||
|
||||
# The master toctree document.
|
||||
master_doc = 'index'
|
||||
|
||||
# Interpret `single-backticks` to be a cross-reference to any kind of
|
||||
# referenceable object. Unresolvable or ambiguous references will emit a
|
||||
# warning at build time.
|
||||
default_role = 'any'
|
||||
|
||||
# General information about the project.
|
||||
project = u'QEMU'
|
||||
copyright = u'2026, The QEMU Project Developers'
|
||||
author = u'The QEMU Project Developers'
|
||||
|
||||
# The version info for the project you're documenting, acts as replacement for
|
||||
# |version| and |release|, also used in various other places throughout the
|
||||
# built documents.
|
||||
|
||||
# Extract this information from the VERSION file, for the benefit of
|
||||
# standalone Sphinx runs as used by readthedocs.org. Builds run from
|
||||
# the Makefile will pass version and release on the sphinx-build
|
||||
# command line, which override this.
|
||||
try:
|
||||
extracted_version = None
|
||||
with open(os.path.join(qemu_docdir, '../VERSION')) as f:
|
||||
extracted_version = f.readline().strip()
|
||||
except:
|
||||
pass
|
||||
finally:
|
||||
if extracted_version:
|
||||
version = release = extracted_version
|
||||
else:
|
||||
version = release = "unknown version"
|
||||
|
||||
bits = version.split(".")
|
||||
|
||||
major = int(bits[0])
|
||||
minor = int(bits[1])
|
||||
micro = int(bits[2])
|
||||
|
||||
# Check for a dev snapshot, so we can adjust to next
|
||||
# predicted release version.
|
||||
#
|
||||
# This assumes we do 3 releases per year, so must bump
|
||||
# major if minor == 2
|
||||
if micro >= 50:
|
||||
micro = 0
|
||||
if minor == 2:
|
||||
major += 1
|
||||
minor = 0
|
||||
else:
|
||||
minor += 1
|
||||
|
||||
# These thresholds must match the constants
|
||||
# MACHINE_VER_DELETION_MAJOR & MACHINE_VER_DEPRECATION_MAJOR
|
||||
# defined in include/hw/core/boards.h and the introductory text in
|
||||
# docs/about/deprecated.rst
|
||||
ver_machine_deprecation_version = "%d.%d.0" % (major - 3, minor)
|
||||
ver_machine_deletion_version = "%d.%d.0" % (major - 6, minor)
|
||||
|
||||
# The language for content autogenerated by Sphinx. Refer to documentation
|
||||
# for a list of supported languages.
|
||||
#
|
||||
# This is also used if you do content translation via gettext catalogs.
|
||||
# Usually you set "language" from the command line for these cases.
|
||||
language = 'en'
|
||||
|
||||
# List of patterns, relative to source directory, that match files and
|
||||
# directories to ignore when looking for source files.
|
||||
# This patterns also effect to html_static_path and html_extra_path
|
||||
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
|
||||
|
||||
# The name of the Pygments (syntax highlighting) style to use.
|
||||
pygments_style = 'sphinx'
|
||||
|
||||
# If true, `todo` and `todoList` produce output, else they produce nothing.
|
||||
todo_include_todos = False
|
||||
|
||||
# Sphinx defaults to warning about use of :option: for options not defined
|
||||
# with "option::" in the document being processed. Turn that off.
|
||||
suppress_warnings = ["ref.option"]
|
||||
|
||||
# The rst_epilog fragment is effectively included in every rST file.
|
||||
# We use it to define substitutions based on build config that
|
||||
# can then be used in the documentation. The fallback if the
|
||||
# environment variable is not set is for the benefit of readthedocs
|
||||
# style document building; our Makefile always sets the variable.
|
||||
confdir = os.getenv('CONFDIR', "/etc/qemu")
|
||||
|
||||
vars = {
|
||||
"CONFDIR": confdir,
|
||||
"VER_MACHINE_DEPRECATION_VERSION": ver_machine_deprecation_version,
|
||||
"VER_MACHINE_DELETION_VERSION": ver_machine_deletion_version,
|
||||
}
|
||||
|
||||
rst_epilog = "".join([
|
||||
".. |" + key + "| replace:: ``" + vars[key] + "``\n"
|
||||
for key in vars.keys()
|
||||
])
|
||||
|
||||
# We slurp in the defs.rst.inc and literally include it into rst_epilog,
|
||||
# because Sphinx's include:: directive doesn't work with absolute paths
|
||||
# and there isn't any one single relative path that will work for all
|
||||
# documents and for both via-make and direct sphinx-build invocation.
|
||||
with open(os.path.join(qemu_docdir, 'defs.rst.inc')) as f:
|
||||
rst_epilog += f.read()
|
||||
|
||||
|
||||
# Normally, the QAPI domain is picky about what field lists you use to
|
||||
# describe a QAPI entity. If you'd like to use arbitrary additional
|
||||
# fields in source documentation, add them here.
|
||||
qapi_allowed_fields = {
|
||||
"see also",
|
||||
}
|
||||
|
||||
# Due to a limitation in Sphinx, we need to know which indices to
|
||||
# generate in advance. Adding a namespace here allows that generation.
|
||||
qapi_namespaces = {
|
||||
"QGA",
|
||||
"QMP",
|
||||
"QSD",
|
||||
}
|
||||
|
||||
# -- Options for HTML output ----------------------------------------------
|
||||
|
||||
# The theme to use for HTML and HTML Help pages. See the documentation for
|
||||
# a list of builtin themes.
|
||||
#
|
||||
try:
|
||||
import sphinx_rtd_theme
|
||||
except ImportError:
|
||||
raise ConfigError(
|
||||
'The Sphinx \'sphinx_rtd_theme\' HTML theme was not found.\n'
|
||||
)
|
||||
|
||||
html_theme = 'sphinx_rtd_theme'
|
||||
|
||||
# Theme options are theme-specific and customize the look and feel of a theme
|
||||
# further. For a list of options available for each theme, see the
|
||||
# documentation.
|
||||
html_theme_options = {
|
||||
"style_nav_header_background": "#802400",
|
||||
"navigation_with_keys": True,
|
||||
}
|
||||
|
||||
html_logo = os.path.join(qemu_docdir, "../ui/icons/qemu_128x128.png")
|
||||
|
||||
html_favicon = os.path.join(qemu_docdir, "../ui/icons/qemu_32x32.png")
|
||||
|
||||
# Add any paths that contain custom static files (such as style sheets) here,
|
||||
# relative to this directory. They are copied after the builtin static files,
|
||||
# so a file named "default.css" will overwrite the builtin "default.css".
|
||||
html_static_path = [os.path.join(qemu_docdir, "sphinx-static")]
|
||||
|
||||
html_css_files = [
|
||||
'theme_overrides.css',
|
||||
]
|
||||
|
||||
html_js_files = [
|
||||
'custom.js',
|
||||
]
|
||||
|
||||
html_context = {
|
||||
"source_url_prefix": "https://gitlab.com/qemu-project/qemu/-/blob/master/docs/",
|
||||
"gitlab_user": "qemu-project",
|
||||
"gitlab_repo": "qemu",
|
||||
"gitlab_version": "master",
|
||||
"conf_py_path": "/docs/", # Path in the checkout to the docs root
|
||||
}
|
||||
|
||||
# Custom sidebar templates, must be a dictionary that maps document names
|
||||
# to template names.
|
||||
#html_sidebars = {}
|
||||
|
||||
# Don't copy the rST source files to the HTML output directory,
|
||||
# and don't put links to the sources into the output HTML.
|
||||
html_copy_source = False
|
||||
|
||||
# -- Options for HTMLHelp output ------------------------------------------
|
||||
|
||||
# Output file base name for HTML help builder.
|
||||
htmlhelp_basename = 'QEMUdoc'
|
||||
|
||||
|
||||
# -- Options for LaTeX output ---------------------------------------------
|
||||
|
||||
latex_elements = {
|
||||
# The paper size ('letterpaper' or 'a4paper').
|
||||
#
|
||||
# 'papersize': 'letterpaper',
|
||||
|
||||
# The font size ('10pt', '11pt' or '12pt').
|
||||
#
|
||||
# 'pointsize': '10pt',
|
||||
|
||||
# Additional stuff for the LaTeX preamble.
|
||||
#
|
||||
# 'preamble': '',
|
||||
|
||||
# Latex figure (float) alignment
|
||||
#
|
||||
# 'figure_align': 'htbp',
|
||||
}
|
||||
|
||||
# Grouping the document tree into LaTeX files. List of tuples
|
||||
# (source start file, target name, title,
|
||||
# author, documentclass [howto, manual, or own class]).
|
||||
latex_documents = [
|
||||
(master_doc, 'QEMU.tex', u'QEMU Documentation',
|
||||
u'The QEMU Project Developers', 'manual'),
|
||||
]
|
||||
|
||||
|
||||
# -- Options for manual page output ---------------------------------------
|
||||
# Individual manual/conf.py can override this to create man pages
|
||||
man_pages = [
|
||||
('interop/qemu-ga', 'qemu-ga',
|
||||
'QEMU Guest Agent',
|
||||
['Michael Roth <[email protected]>'], 8),
|
||||
('interop/qemu-ga-ref', 'qemu-ga-ref',
|
||||
'QEMU Guest Agent Protocol Reference',
|
||||
[], 7),
|
||||
('interop/qemu-qmp-ref', 'qemu-qmp-ref',
|
||||
'QEMU QMP Reference Manual',
|
||||
[], 7),
|
||||
('interop/qemu-storage-daemon-qmp-ref', 'qemu-storage-daemon-qmp-ref',
|
||||
'QEMU Storage Daemon QMP Reference Manual',
|
||||
[], 7),
|
||||
('system/qemu-manpage', 'qemu',
|
||||
'QEMU User Documentation',
|
||||
['Fabrice Bellard'], 1),
|
||||
('system/qemu-block-drivers', 'qemu-block-drivers',
|
||||
'QEMU block drivers reference',
|
||||
['Fabrice Bellard and the QEMU Project developers'], 7),
|
||||
('system/qemu-cpu-models', 'qemu-cpu-models',
|
||||
'QEMU CPU Models',
|
||||
['The QEMU Project developers'], 7),
|
||||
('tools/qemu-img', 'qemu-img',
|
||||
'QEMU disk image utility',
|
||||
['Fabrice Bellard'], 1),
|
||||
('tools/qemu-nbd', 'qemu-nbd',
|
||||
'QEMU Disk Network Block Device Server',
|
||||
['Anthony Liguori <[email protected]>'], 8),
|
||||
('tools/qemu-pr-helper', 'qemu-pr-helper',
|
||||
'QEMU persistent reservation helper',
|
||||
[], 8),
|
||||
('tools/qemu-storage-daemon', 'qemu-storage-daemon',
|
||||
'QEMU storage daemon',
|
||||
[], 1),
|
||||
('tools/qemu-trace-stap', 'qemu-trace-stap',
|
||||
'QEMU SystemTap trace tool',
|
||||
[], 1),
|
||||
('tools/qemu-vnc', 'qemu-vnc',
|
||||
'QEMU standalone VNC server',
|
||||
[], 1),
|
||||
]
|
||||
man_make_section_directory = False
|
||||
|
||||
# We use paths starting from qemu_docdir here so that you can run
|
||||
# sphinx-build from anywhere and the kerneldoc extension can still
|
||||
# find everything.
|
||||
# Since kernel-doc is now a Python script, we should run it with whatever
|
||||
# Python this sphinx is using (rather than letting it find one via env)
|
||||
kerneldoc_bin = [sys.executable, os.path.join(qemu_docdir, '../scripts/kernel-doc.py')]
|
||||
kerneldoc_srctree = os.path.join(qemu_docdir, '..')
|
||||
hxtool_srctree = os.path.join(qemu_docdir, '..')
|
||||
qapidoc_srctree = os.path.join(qemu_docdir, '..')
|
||||
dbusdoc_srctree = os.path.join(qemu_docdir, '..')
|
||||
dbus_index_common_prefix = ["org.qemu."]
|
||||
@@ -0,0 +1,37 @@
|
||||
###########################################################################
|
||||
#
|
||||
# You can pass this file directly to qemu using the -readconfig
|
||||
# command line switch.
|
||||
#
|
||||
# This config file creates a EHCI adapter with companion UHCI
|
||||
# controllers as multifunction device in PCI slot "1d".
|
||||
#
|
||||
# Specify "bus=ehci.0" when creating usb devices to hook them up
|
||||
# there.
|
||||
#
|
||||
|
||||
[device "ehci"]
|
||||
driver = "ich9-usb-ehci1"
|
||||
addr = "1d.7"
|
||||
multifunction = "on"
|
||||
|
||||
[device "uhci-1"]
|
||||
driver = "ich9-usb-uhci1"
|
||||
addr = "1d.0"
|
||||
multifunction = "on"
|
||||
masterbus = "ehci.0"
|
||||
firstport = "0"
|
||||
|
||||
[device "uhci-2"]
|
||||
driver = "ich9-usb-uhci2"
|
||||
addr = "1d.1"
|
||||
multifunction = "on"
|
||||
masterbus = "ehci.0"
|
||||
firstport = "2"
|
||||
|
||||
[device "uhci-3"]
|
||||
driver = "ich9-usb-uhci3"
|
||||
addr = "1d.2"
|
||||
multifunction = "on"
|
||||
masterbus = "ehci.0"
|
||||
firstport = "4"
|
||||
@@ -0,0 +1,283 @@
|
||||
# mach-virt - VirtIO guest (graphical console)
|
||||
# =========================================================
|
||||
#
|
||||
# Usage:
|
||||
#
|
||||
# $ qemu-system-aarch64 \
|
||||
# -nodefaults \
|
||||
# -readconfig mach-virt-graphical.cfg \
|
||||
# -cpu host
|
||||
#
|
||||
# You will probably need to tweak the lines marked as
|
||||
# CHANGE ME before being able to use this configuration!
|
||||
#
|
||||
# The guest will have a selection of VirtIO devices
|
||||
# tailored towards optimal performance with modern guests,
|
||||
# and will be accessed through a graphical console.
|
||||
#
|
||||
# ---------------------------------------------------------
|
||||
#
|
||||
# Using -nodefaults is required to have full control over
|
||||
# the virtual hardware: when it's specified, QEMU will
|
||||
# populate the board with only the builtin peripherals,
|
||||
# such as the PL011 UART, plus a PCI Express Root Bus; the
|
||||
# user will then have to explicitly add further devices.
|
||||
#
|
||||
# The PCI Express Root Bus shows up in the guest as:
|
||||
#
|
||||
# 00:00.0 Host bridge
|
||||
#
|
||||
# This configuration file adds a number of other useful
|
||||
# devices, more specifically:
|
||||
#
|
||||
# 00:01.0 Display controller
|
||||
# 00.1c.* PCI bridge (PCI Express Root Ports)
|
||||
# 01:00.0 SCSI storage controller
|
||||
# 02:00.0 Ethernet controller
|
||||
# 03:00.0 USB controller
|
||||
#
|
||||
# More information about these devices is available below.
|
||||
|
||||
|
||||
# Machine options
|
||||
# =========================================================
|
||||
#
|
||||
# We use the virt machine type and enable KVM acceleration
|
||||
# for better performance.
|
||||
#
|
||||
# Using less than 1 GiB of memory is probably not going to
|
||||
# yield good performance in the guest, and might even lead
|
||||
# to obscure boot issues in some cases.
|
||||
#
|
||||
# Unfortunately, there is no way to configure the CPU model
|
||||
# in this file, so it will have to be provided on the
|
||||
# command line, but we can configure the guest to use the
|
||||
# same GIC version as the host.
|
||||
|
||||
[machine]
|
||||
type = "virt"
|
||||
gic-version = "host"
|
||||
|
||||
[accel]
|
||||
accel = "kvm"
|
||||
|
||||
[memory]
|
||||
size = "1024"
|
||||
|
||||
|
||||
# Firmware configuration
|
||||
# =========================================================
|
||||
#
|
||||
# There are two parts to the firmware: a read-only image
|
||||
# containing the executable code, which is shared between
|
||||
# guests, and a read/write variable store that is owned
|
||||
# by one specific guest, exclusively, and is used to
|
||||
# record information such as the UEFI boot order.
|
||||
#
|
||||
# For any new guest, its permanent, private variable store
|
||||
# should initially be copied from the template file
|
||||
# provided along with the firmware binary.
|
||||
#
|
||||
# Depending on the OS distribution you're using on the
|
||||
# host, the name of the package containing the firmware
|
||||
# binary and variable store template, as well as the paths
|
||||
# to the files themselves, will be different. For example:
|
||||
#
|
||||
# Fedora
|
||||
# edk2-aarch64 (pkg)
|
||||
# /usr/share/edk2/aarch64/QEMU_EFI-pflash.raw (bin)
|
||||
# /usr/share/edk2/aarch64/vars-template-pflash.raw (var)
|
||||
#
|
||||
# RHEL
|
||||
# AAVMF (pkg)
|
||||
# /usr/share/AAVMF/AAVMF_CODE.fd (bin)
|
||||
# /usr/share/AAVMF/AAVMF_VARS.fd (var)
|
||||
#
|
||||
# Debian/Ubuntu
|
||||
# qemu-efi (pkg)
|
||||
# /usr/share/AAVMF/AAVMF_CODE.fd (bin)
|
||||
# /usr/share/AAVMF/AAVMF_VARS.fd (var)
|
||||
|
||||
[drive "uefi-binary"]
|
||||
file = "/usr/share/AAVMF/AAVMF_CODE.fd" # CHANGE ME
|
||||
format = "raw"
|
||||
if = "pflash"
|
||||
unit = "0"
|
||||
readonly = "on"
|
||||
|
||||
[drive "uefi-varstore"]
|
||||
file = "guest_VARS.fd" # CHANGE ME
|
||||
format = "raw"
|
||||
if = "pflash"
|
||||
unit = "1"
|
||||
|
||||
|
||||
# PCI bridge (PCI Express Root Ports)
|
||||
# =========================================================
|
||||
#
|
||||
# We create eight PCI Express Root Ports, and we plug them
|
||||
# all into separate functions of the same slot. Some of
|
||||
# them will be used by devices, the rest will remain
|
||||
# available for hotplug.
|
||||
|
||||
[device "pcie.1"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.0"
|
||||
port = "1"
|
||||
chassis = "1"
|
||||
multifunction = "on"
|
||||
|
||||
[device "pcie.2"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.1"
|
||||
port = "2"
|
||||
chassis = "2"
|
||||
|
||||
[device "pcie.3"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.2"
|
||||
port = "3"
|
||||
chassis = "3"
|
||||
|
||||
[device "pcie.4"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.3"
|
||||
port = "4"
|
||||
chassis = "4"
|
||||
|
||||
[device "pcie.5"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.4"
|
||||
port = "5"
|
||||
chassis = "5"
|
||||
|
||||
[device "pcie.6"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.5"
|
||||
port = "6"
|
||||
chassis = "6"
|
||||
|
||||
[device "pcie.7"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.6"
|
||||
port = "7"
|
||||
chassis = "7"
|
||||
|
||||
[device "pcie.8"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.7"
|
||||
port = "8"
|
||||
chassis = "8"
|
||||
|
||||
|
||||
# SCSI storage controller (and storage)
|
||||
# =========================================================
|
||||
#
|
||||
# We use virtio-scsi here so that we can (hot)plug a large
|
||||
# number of disks without running into issues; a SCSI disk,
|
||||
# backed by a qcow2 disk image on the host's filesystem, is
|
||||
# attached to it.
|
||||
#
|
||||
# We also create an optical disk, mostly for installation
|
||||
# purposes: once the guest OS has been successfully
|
||||
# installed, the guest will no longer boot from optical
|
||||
# media. If you don't want, or no longer want, to have an
|
||||
# optical disk in the guest you can safely comment out
|
||||
# all relevant sections below.
|
||||
|
||||
[device "scsi"]
|
||||
driver = "virtio-scsi-pci"
|
||||
bus = "pcie.1"
|
||||
addr = "00.0"
|
||||
|
||||
[device "scsi-disk"]
|
||||
driver = "scsi-hd"
|
||||
bus = "scsi.0"
|
||||
drive = "disk"
|
||||
bootindex = "1"
|
||||
|
||||
[drive "disk"]
|
||||
file = "guest.qcow2" # CHANGE ME
|
||||
format = "qcow2"
|
||||
if = "none"
|
||||
|
||||
[device "scsi-optical-disk"]
|
||||
driver = "scsi-cd"
|
||||
bus = "scsi.0"
|
||||
drive = "optical-disk"
|
||||
bootindex = "2"
|
||||
|
||||
[drive "optical-disk"]
|
||||
file = "install.iso" # CHANGE ME
|
||||
format = "raw"
|
||||
if = "none"
|
||||
|
||||
|
||||
# Ethernet controller
|
||||
# =========================================================
|
||||
#
|
||||
# We use virtio-net for improved performance over emulated
|
||||
# hardware; on the host side, we take advantage of user
|
||||
# networking so that the QEMU process doesn't require any
|
||||
# additional privileges.
|
||||
|
||||
[netdev "hostnet"]
|
||||
type = "user"
|
||||
|
||||
[device "net"]
|
||||
driver = "virtio-net-pci"
|
||||
netdev = "hostnet"
|
||||
bus = "pcie.2"
|
||||
addr = "00.0"
|
||||
|
||||
|
||||
# USB controller (and input devices)
|
||||
# =========================================================
|
||||
#
|
||||
# We add a virtualization-friendly USB 3.0 controller and
|
||||
# a USB keyboard / USB tablet combo so that graphical
|
||||
# guests can be controlled appropriately.
|
||||
|
||||
[device "usb"]
|
||||
driver = "nec-usb-xhci"
|
||||
bus = "pcie.3"
|
||||
addr = "00.0"
|
||||
|
||||
[device "keyboard"]
|
||||
driver = "usb-kbd"
|
||||
bus = "usb.0"
|
||||
|
||||
[device "tablet"]
|
||||
driver = "usb-tablet"
|
||||
bus = "usb.0"
|
||||
|
||||
|
||||
# Display controller
|
||||
# =========================================================
|
||||
#
|
||||
# We use virtio-gpu because the legacy VGA framebuffer is
|
||||
# very troublesome on aarch64, and virtio-gpu is the only
|
||||
# video device that doesn't implement it.
|
||||
#
|
||||
# If you're running the guest on a remote, potentially
|
||||
# headless host, you will probably want to append something
|
||||
# like
|
||||
#
|
||||
# -display vnc=127.0.0.1:0
|
||||
#
|
||||
# to the command line in order to prevent QEMU from
|
||||
# creating a graphical display window on the host and
|
||||
# enable remote access instead.
|
||||
|
||||
[device "video"]
|
||||
driver = "virtio-gpu"
|
||||
bus = "pcie.0"
|
||||
addr = "01.0"
|
||||
@@ -0,0 +1,245 @@
|
||||
# mach-virt - VirtIO guest (serial console)
|
||||
# =========================================================
|
||||
#
|
||||
# Usage:
|
||||
#
|
||||
# $ qemu-system-aarch64 \
|
||||
# -nodefaults \
|
||||
# -readconfig mach-virt-serial.cfg \
|
||||
# -display none -serial mon:stdio \
|
||||
# -cpu host
|
||||
#
|
||||
# You will probably need to tweak the lines marked as
|
||||
# CHANGE ME before being able to use this configuration!
|
||||
#
|
||||
# The guest will have a selection of VirtIO devices
|
||||
# tailored towards optimal performance with modern guests,
|
||||
# and will be accessed through the serial console.
|
||||
#
|
||||
# ---------------------------------------------------------
|
||||
#
|
||||
# Using -nodefaults is required to have full control over
|
||||
# the virtual hardware: when it's specified, QEMU will
|
||||
# populate the board with only the builtin peripherals,
|
||||
# such as the PL011 UART, plus a PCI Express Root Bus; the
|
||||
# user will then have to explicitly add further devices.
|
||||
#
|
||||
# The PCI Express Root Bus shows up in the guest as:
|
||||
#
|
||||
# 00:00.0 Host bridge
|
||||
#
|
||||
# This configuration file adds a number of other useful
|
||||
# devices, more specifically:
|
||||
#
|
||||
# 00.1c.* PCI bridge (PCI Express Root Ports)
|
||||
# 01:00.0 SCSI storage controller
|
||||
# 02:00.0 Ethernet controller
|
||||
#
|
||||
# More information about these devices is available below.
|
||||
#
|
||||
# We use '-display none' to prevent QEMU from creating a
|
||||
# graphical display window, which would serve no use in
|
||||
# this specific configuration, and '-serial mon:stdio' to
|
||||
# multiplex the guest's serial console and the QEMU monitor
|
||||
# to the host's stdio; use 'Ctrl+A h' to learn how to
|
||||
# switch between the two and more.
|
||||
|
||||
|
||||
# Machine options
|
||||
# =========================================================
|
||||
#
|
||||
# We use the virt machine type and enable KVM acceleration
|
||||
# for better performance.
|
||||
#
|
||||
# Using less than 1 GiB of memory is probably not going to
|
||||
# yield good performance in the guest, and might even lead
|
||||
# to obscure boot issues in some cases.
|
||||
#
|
||||
# Unfortunately, there is no way to configure the CPU model
|
||||
# in this file, so it will have to be provided on the
|
||||
# command line, but we can configure the guest to use the
|
||||
# same GIC version as the host.
|
||||
|
||||
[machine]
|
||||
type = "virt"
|
||||
gic-version = "host"
|
||||
|
||||
[accel]
|
||||
accel = "kvm"
|
||||
|
||||
[memory]
|
||||
size = "1024"
|
||||
|
||||
|
||||
# Firmware configuration
|
||||
# =========================================================
|
||||
#
|
||||
# There are two parts to the firmware: a read-only image
|
||||
# containing the executable code, which is shared between
|
||||
# guests, and a read/write variable store that is owned
|
||||
# by one specific guest, exclusively, and is used to
|
||||
# record information such as the UEFI boot order.
|
||||
#
|
||||
# For any new guest, its permanent, private variable store
|
||||
# should initially be copied from the template file
|
||||
# provided along with the firmware binary.
|
||||
#
|
||||
# Depending on the OS distribution you're using on the
|
||||
# host, the name of the package containing the firmware
|
||||
# binary and variable store template, as well as the paths
|
||||
# to the files themselves, will be different. For example:
|
||||
#
|
||||
# Fedora
|
||||
# edk2-aarch64 (pkg)
|
||||
# /usr/share/edk2/aarch64/QEMU_EFI-pflash.raw (bin)
|
||||
# /usr/share/edk2/aarch64/vars-template-pflash.raw (var)
|
||||
#
|
||||
# RHEL
|
||||
# AAVMF (pkg)
|
||||
# /usr/share/AAVMF/AAVMF_CODE.fd (bin)
|
||||
# /usr/share/AAVMF/AAVMF_VARS.fd (var)
|
||||
#
|
||||
# Debian/Ubuntu
|
||||
# qemu-efi (pkg)
|
||||
# /usr/share/AAVMF/AAVMF_CODE.fd (bin)
|
||||
# /usr/share/AAVMF/AAVMF_VARS.fd (var)
|
||||
|
||||
[drive "uefi-binary"]
|
||||
file = "/usr/share/AAVMF/AAVMF_CODE.fd" # CHANGE ME
|
||||
format = "raw"
|
||||
if = "pflash"
|
||||
unit = "0"
|
||||
readonly = "on"
|
||||
|
||||
[drive "uefi-varstore"]
|
||||
file = "guest_VARS.fd" # CHANGE ME
|
||||
format = "raw"
|
||||
if = "pflash"
|
||||
unit = "1"
|
||||
|
||||
|
||||
# PCI bridge (PCI Express Root Ports)
|
||||
# =========================================================
|
||||
#
|
||||
# We create eight PCI Express Root Ports, and we plug them
|
||||
# all into separate functions of the same slot. Some of
|
||||
# them will be used by devices, the rest will remain
|
||||
# available for hotplug.
|
||||
|
||||
[device "pcie.1"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.0"
|
||||
port = "1"
|
||||
chassis = "1"
|
||||
multifunction = "on"
|
||||
|
||||
[device "pcie.2"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.1"
|
||||
port = "2"
|
||||
chassis = "2"
|
||||
|
||||
[device "pcie.3"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.2"
|
||||
port = "3"
|
||||
chassis = "3"
|
||||
|
||||
[device "pcie.4"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.3"
|
||||
port = "4"
|
||||
chassis = "4"
|
||||
|
||||
[device "pcie.5"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.4"
|
||||
port = "5"
|
||||
chassis = "5"
|
||||
|
||||
[device "pcie.6"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.5"
|
||||
port = "6"
|
||||
chassis = "6"
|
||||
|
||||
[device "pcie.7"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.6"
|
||||
port = "7"
|
||||
chassis = "7"
|
||||
|
||||
[device "pcie.8"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.7"
|
||||
port = "8"
|
||||
chassis = "8"
|
||||
|
||||
|
||||
# SCSI storage controller (and storage)
|
||||
# =========================================================
|
||||
#
|
||||
# We use virtio-scsi here so that we can (hot)plug a large
|
||||
# number of disks without running into issues; a SCSI disk,
|
||||
# backed by a qcow2 disk image on the host's filesystem, is
|
||||
# attached to it.
|
||||
#
|
||||
# We also create an optical disk, mostly for installation
|
||||
# purposes: once the guest OS has been successfully
|
||||
# installed, the guest will no longer boot from optical
|
||||
# media. If you don't want, or no longer want, to have an
|
||||
# optical disk in the guest you can safely comment out
|
||||
# all relevant sections below.
|
||||
|
||||
[device "scsi"]
|
||||
driver = "virtio-scsi-pci"
|
||||
bus = "pcie.1"
|
||||
addr = "00.0"
|
||||
|
||||
[device "scsi-disk"]
|
||||
driver = "scsi-hd"
|
||||
bus = "scsi.0"
|
||||
drive = "disk"
|
||||
bootindex = "1"
|
||||
|
||||
[drive "disk"]
|
||||
file = "guest.qcow2" # CHANGE ME
|
||||
format = "qcow2"
|
||||
if = "none"
|
||||
|
||||
[device "scsi-optical-disk"]
|
||||
driver = "scsi-cd"
|
||||
bus = "scsi.0"
|
||||
drive = "optical-disk"
|
||||
bootindex = "2"
|
||||
|
||||
[drive "optical-disk"]
|
||||
file = "install.iso" # CHANGE ME
|
||||
format = "raw"
|
||||
if = "none"
|
||||
|
||||
|
||||
# Ethernet controller
|
||||
# =========================================================
|
||||
#
|
||||
# We use virtio-net for improved performance over emulated
|
||||
# hardware; on the host side, we take advantage of user
|
||||
# networking so that the QEMU process doesn't require any
|
||||
# additional privileges.
|
||||
|
||||
[netdev "hostnet"]
|
||||
type = "user"
|
||||
|
||||
[device "net"]
|
||||
driver = "virtio-net-pci"
|
||||
netdev = "hostnet"
|
||||
bus = "pcie.2"
|
||||
addr = "00.0"
|
||||
@@ -0,0 +1,294 @@
|
||||
# q35 - Emulated guest (graphical console)
|
||||
# =========================================================
|
||||
#
|
||||
# Usage:
|
||||
#
|
||||
# $ qemu-system-x86_64 \
|
||||
# -nodefaults \
|
||||
# -readconfig q35-emulated.cfg
|
||||
#
|
||||
# You will probably need to tweak the lines marked as
|
||||
# CHANGE ME before being able to use this configuration!
|
||||
#
|
||||
# The guest will have a selection of emulated devices that
|
||||
# closely resembles that of a physical machine, and will be
|
||||
# accessed through a graphical console.
|
||||
#
|
||||
# ---------------------------------------------------------
|
||||
#
|
||||
# Using -nodefaults is required to have full control over
|
||||
# the virtual hardware: when it's specified, QEMU will
|
||||
# populate the board with only the builtin peripherals
|
||||
# plus a small selection of core PCI devices and
|
||||
# controllers; the user will then have to explicitly add
|
||||
# further devices.
|
||||
#
|
||||
# The core PCI devices show up in the guest as:
|
||||
#
|
||||
# 00:00.0 Host bridge
|
||||
# 00:1f.0 ISA bridge / LPC
|
||||
# 00:1f.2 SATA (AHCI) controller
|
||||
# 00:1f.3 SMBus controller
|
||||
#
|
||||
# This configuration file adds a number of devices that
|
||||
# are pretty much guaranteed to be present in every single
|
||||
# physical machine based on q35, more specifically:
|
||||
#
|
||||
# 00:01.0 VGA compatible controller
|
||||
# 00:19.0 Ethernet controller
|
||||
# 00:1a.* USB controller (#2)
|
||||
# 00:1b.0 Audio device
|
||||
# 00:1c.* PCI bridge (PCI Express Root Ports)
|
||||
# 00:1d.* USB Controller (#1)
|
||||
# 00:1e.0 PCI bridge (legacy PCI bridge)
|
||||
#
|
||||
# More information about these devices is available below.
|
||||
|
||||
|
||||
# Machine options
|
||||
# =========================================================
|
||||
#
|
||||
# We use the q35 machine type and enable KVM acceleration
|
||||
# for better performance.
|
||||
#
|
||||
# Using less than 1 GiB of memory is probably not going to
|
||||
# yield good performance in the guest, and might even lead
|
||||
# to obscure boot issues in some cases.
|
||||
#
|
||||
# Unfortunately, there is no way to configure the CPU model
|
||||
# in this file, so it will have to be provided on the
|
||||
# command line.
|
||||
|
||||
[machine]
|
||||
type = "q35"
|
||||
|
||||
[accel]
|
||||
accel = "kvm"
|
||||
|
||||
[memory]
|
||||
size = "1024"
|
||||
|
||||
|
||||
# PCI bridge (PCI Express Root Ports)
|
||||
# =========================================================
|
||||
#
|
||||
# We add four PCI Express Root Ports, all sharing the same
|
||||
# slot on the PCI Express Root Bus. These ports support
|
||||
# hotplug.
|
||||
|
||||
[device "ich9-pcie-port-1"]
|
||||
driver = "ioh3420"
|
||||
multifunction = "on"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.0"
|
||||
port = "1"
|
||||
chassis = "1"
|
||||
|
||||
[device "ich9-pcie-port-2"]
|
||||
driver = "ioh3420"
|
||||
multifunction = "on"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.1"
|
||||
port = "2"
|
||||
chassis = "2"
|
||||
|
||||
[device "ich9-pcie-port-3"]
|
||||
driver = "ioh3420"
|
||||
multifunction = "on"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.2"
|
||||
port = "3"
|
||||
chassis = "3"
|
||||
|
||||
[device "ich9-pcie-port-4"]
|
||||
driver = "ioh3420"
|
||||
multifunction = "on"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.3"
|
||||
port = "4"
|
||||
chassis = "4"
|
||||
|
||||
|
||||
# PCI bridge (legacy PCI bridge)
|
||||
# =========================================================
|
||||
#
|
||||
# This bridge can be used to build an independent topology
|
||||
# for legacy PCI devices. PCI Express devices should be
|
||||
# plugged into PCI Express slots instead, so ideally there
|
||||
# will be no devices connected to this bridge.
|
||||
|
||||
[device "ich9-pci-bridge"]
|
||||
driver = "i82801b11-bridge"
|
||||
bus = "pcie.0"
|
||||
addr = "1e.0"
|
||||
|
||||
|
||||
# SATA storage
|
||||
# =========================================================
|
||||
#
|
||||
# An implicit SATA controller is created automatically for
|
||||
# every single q35 guest; here we create a disk, backed by
|
||||
# a qcow2 disk image on the host's filesystem, and attach
|
||||
# it to that controller so that the guest can use it.
|
||||
#
|
||||
# We also create an optical disk, mostly for installation
|
||||
# purposes: once the guest OS has been successfully
|
||||
# installed, the guest will no longer boot from optical
|
||||
# media. If you don't want, or no longer want, to have an
|
||||
# optical disk in the guest you can safely comment out
|
||||
# all relevant sections below.
|
||||
|
||||
[device "sata-disk"]
|
||||
driver = "ide-hd"
|
||||
bus = "ide.0"
|
||||
drive = "disk"
|
||||
bootindex = "1"
|
||||
|
||||
[drive "disk"]
|
||||
file = "guest.qcow2" # CHANGE ME
|
||||
format = "qcow2"
|
||||
if = "none"
|
||||
|
||||
[device "sata-optical-disk"]
|
||||
driver = "ide-cd"
|
||||
bus = "ide.1"
|
||||
drive = "optical-disk"
|
||||
bootindex = "2"
|
||||
|
||||
[drive "optical-disk"]
|
||||
file = "install.iso" # CHANGE ME
|
||||
format = "raw"
|
||||
if = "none"
|
||||
|
||||
|
||||
# USB controller (#1)
|
||||
# =========================================================
|
||||
#
|
||||
# EHCI controller + UHCI companion controllers.
|
||||
|
||||
[device "ich9-ehci-1"]
|
||||
driver = "ich9-usb-ehci1"
|
||||
multifunction = "on"
|
||||
bus = "pcie.0"
|
||||
addr = "1d.7"
|
||||
|
||||
[device "ich9-uhci-1"]
|
||||
driver = "ich9-usb-uhci1"
|
||||
multifunction = "on"
|
||||
bus = "pcie.0"
|
||||
addr = "1d.0"
|
||||
masterbus = "ich9-ehci-1.0"
|
||||
firstport = "0"
|
||||
|
||||
[device "ich9-uhci-2"]
|
||||
driver = "ich9-usb-uhci2"
|
||||
multifunction = "on"
|
||||
bus = "pcie.0"
|
||||
addr = "1d.1"
|
||||
masterbus = "ich9-ehci-1.0"
|
||||
firstport = "2"
|
||||
|
||||
[device "ich9-uhci-3"]
|
||||
driver = "ich9-usb-uhci3"
|
||||
multifunction = "on"
|
||||
bus = "pcie.0"
|
||||
addr = "1d.2"
|
||||
masterbus = "ich9-ehci-1.0"
|
||||
firstport = "4"
|
||||
|
||||
|
||||
# USB controller (#2)
|
||||
# =========================================================
|
||||
#
|
||||
# EHCI controller + UHCI companion controllers.
|
||||
|
||||
[device "ich9-ehci-2"]
|
||||
driver = "ich9-usb-ehci2"
|
||||
multifunction = "on"
|
||||
bus = "pcie.0"
|
||||
addr = "1a.7"
|
||||
|
||||
[device "ich9-uhci-4"]
|
||||
driver = "ich9-usb-uhci4"
|
||||
multifunction = "on"
|
||||
bus = "pcie.0"
|
||||
addr = "1a.0"
|
||||
masterbus = "ich9-ehci-2.0"
|
||||
firstport = "0"
|
||||
|
||||
[device "ich9-uhci-5"]
|
||||
driver = "ich9-usb-uhci5"
|
||||
multifunction = "on"
|
||||
bus = "pcie.0"
|
||||
addr = "1a.1"
|
||||
masterbus = "ich9-ehci-2.0"
|
||||
firstport = "2"
|
||||
|
||||
[device "ich9-uhci-6"]
|
||||
driver = "ich9-usb-uhci6"
|
||||
multifunction = "on"
|
||||
bus = "pcie.0"
|
||||
addr = "1a.2"
|
||||
masterbus = "ich9-ehci-2.0"
|
||||
firstport = "4"
|
||||
|
||||
|
||||
# Ethernet controller
|
||||
# =========================================================
|
||||
#
|
||||
# We add a Gigabit Ethernet interface to the guest; on the
|
||||
# host side, we take advantage of user networking so that
|
||||
# the QEMU process doesn't require any additional
|
||||
# privileges.
|
||||
|
||||
[netdev "hostnet"]
|
||||
type = "user"
|
||||
|
||||
[device "net"]
|
||||
driver = "e1000"
|
||||
netdev = "hostnet"
|
||||
bus = "pcie.0"
|
||||
addr = "19.0"
|
||||
|
||||
|
||||
# VGA compatible controller
|
||||
# =========================================================
|
||||
#
|
||||
# We use stdvga instead of Cirrus as it supports more video
|
||||
# modes and is closer to what actual hardware looks like.
|
||||
#
|
||||
# If you're running the guest on a remote, potentially
|
||||
# headless host, you will probably want to append something
|
||||
# like
|
||||
#
|
||||
# -display vnc=127.0.0.1:0
|
||||
#
|
||||
# to the command line in order to prevent QEMU from
|
||||
# creating a graphical display window on the host and
|
||||
# enable remote access instead.
|
||||
|
||||
[device "video"]
|
||||
driver = "VGA"
|
||||
bus = "pcie.0"
|
||||
addr = "01.0"
|
||||
|
||||
|
||||
# Audio device
|
||||
# =========================================================
|
||||
#
|
||||
# The sound card is a legacy PCI device that is plugged
|
||||
# directly into the PCI Express Root Bus.
|
||||
|
||||
[device "ich9-hda-audio"]
|
||||
driver = "ich9-intel-hda"
|
||||
bus = "pcie.0"
|
||||
addr = "1b.0"
|
||||
|
||||
[device "ich9-hda-duplex"]
|
||||
driver = "hda-duplex"
|
||||
bus = "ich9-hda-audio.0"
|
||||
cad = "0"
|
||||
audiodev = "audiodev0"
|
||||
|
||||
[audiodev "audiodev0"]
|
||||
driver = "none" # CHANGE ME
|
||||
@@ -0,0 +1,254 @@
|
||||
# q35 - VirtIO guest (graphical console)
|
||||
# =========================================================
|
||||
#
|
||||
# Usage:
|
||||
#
|
||||
# $ qemu-system-x86_64 \
|
||||
# -nodefaults \
|
||||
# -readconfig q35-virtio-graphical.cfg
|
||||
#
|
||||
# You will probably need to tweak the lines marked as
|
||||
# CHANGE ME before being able to use this configuration!
|
||||
#
|
||||
# The guest will have a selection of VirtIO devices
|
||||
# tailored towards optimal performance with modern guests,
|
||||
# and will be accessed through a graphical console.
|
||||
#
|
||||
# ---------------------------------------------------------
|
||||
#
|
||||
# Using -nodefaults is required to have full control over
|
||||
# the virtual hardware: when it's specified, QEMU will
|
||||
# populate the board with only the builtin peripherals
|
||||
# plus a small selection of core PCI devices and
|
||||
# controllers; the user will then have to explicitly add
|
||||
# further devices.
|
||||
#
|
||||
# The core PCI devices show up in the guest as:
|
||||
#
|
||||
# 00:00.0 Host bridge
|
||||
# 00:1f.0 ISA bridge / LPC
|
||||
# 00:1f.2 SATA (AHCI) controller
|
||||
# 00:1f.3 SMBus controller
|
||||
#
|
||||
# This configuration file adds a number of other useful
|
||||
# devices, more specifically:
|
||||
#
|
||||
# 00:01.0 VGA compatible controller
|
||||
# 00:1b.0 Audio device
|
||||
# 00.1c.* PCI bridge (PCI Express Root Ports)
|
||||
# 01:00.0 SCSI storage controller
|
||||
# 02:00.0 Ethernet controller
|
||||
# 03:00.0 USB controller
|
||||
#
|
||||
# More information about these devices is available below.
|
||||
|
||||
|
||||
# Machine options
|
||||
# =========================================================
|
||||
#
|
||||
# We use the q35 machine type and enable KVM acceleration
|
||||
# for better performance.
|
||||
#
|
||||
# Using less than 1 GiB of memory is probably not going to
|
||||
# yield good performance in the guest, and might even lead
|
||||
# to obscure boot issues in some cases.
|
||||
|
||||
[machine]
|
||||
type = "q35"
|
||||
|
||||
[accel]
|
||||
accel = "kvm"
|
||||
|
||||
[memory]
|
||||
size = "1024"
|
||||
|
||||
|
||||
# PCI bridge (PCI Express Root Ports)
|
||||
# =========================================================
|
||||
#
|
||||
# We create eight PCI Express Root Ports, and we plug them
|
||||
# all into separate functions of the same slot. Some of
|
||||
# them will be used by devices, the rest will remain
|
||||
# available for hotplug.
|
||||
|
||||
[device "pcie.1"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.0"
|
||||
port = "1"
|
||||
chassis = "1"
|
||||
multifunction = "on"
|
||||
|
||||
[device "pcie.2"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.1"
|
||||
port = "2"
|
||||
chassis = "2"
|
||||
|
||||
[device "pcie.3"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.2"
|
||||
port = "3"
|
||||
chassis = "3"
|
||||
|
||||
[device "pcie.4"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.3"
|
||||
port = "4"
|
||||
chassis = "4"
|
||||
|
||||
[device "pcie.5"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.4"
|
||||
port = "5"
|
||||
chassis = "5"
|
||||
|
||||
[device "pcie.6"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.5"
|
||||
port = "6"
|
||||
chassis = "6"
|
||||
|
||||
[device "pcie.7"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.6"
|
||||
port = "7"
|
||||
chassis = "7"
|
||||
|
||||
[device "pcie.8"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.7"
|
||||
port = "8"
|
||||
chassis = "8"
|
||||
|
||||
|
||||
# SCSI storage controller (and storage)
|
||||
# =========================================================
|
||||
#
|
||||
# We use virtio-scsi here so that we can (hot)plug a large
|
||||
# number of disks without running into issues; a SCSI disk,
|
||||
# backed by a qcow2 disk image on the host's filesystem, is
|
||||
# attached to it.
|
||||
#
|
||||
# We also create an optical disk, mostly for installation
|
||||
# purposes: once the guest OS has been successfully
|
||||
# installed, the guest will no longer boot from optical
|
||||
# media. If you don't want, or no longer want, to have an
|
||||
# optical disk in the guest you can safely comment out
|
||||
# all relevant sections below.
|
||||
|
||||
[device "scsi"]
|
||||
driver = "virtio-scsi-pci"
|
||||
bus = "pcie.1"
|
||||
addr = "00.0"
|
||||
|
||||
[device "scsi-disk"]
|
||||
driver = "scsi-hd"
|
||||
bus = "scsi.0"
|
||||
drive = "disk"
|
||||
bootindex = "1"
|
||||
|
||||
[drive "disk"]
|
||||
file = "guest.qcow2" # CHANGE ME
|
||||
format = "qcow2"
|
||||
if = "none"
|
||||
|
||||
[device "scsi-optical-disk"]
|
||||
driver = "scsi-cd"
|
||||
bus = "scsi.0"
|
||||
drive = "optical-disk"
|
||||
bootindex = "2"
|
||||
|
||||
[drive "optical-disk"]
|
||||
file = "install.iso" # CHANGE ME
|
||||
format = "raw"
|
||||
if = "none"
|
||||
|
||||
|
||||
# Ethernet controller
|
||||
# =========================================================
|
||||
#
|
||||
# We use virtio-net for improved performance over emulated
|
||||
# hardware; on the host side, we take advantage of user
|
||||
# networking so that the QEMU process doesn't require any
|
||||
# additional privileges.
|
||||
|
||||
[netdev "hostnet"]
|
||||
type = "user"
|
||||
|
||||
[device "net"]
|
||||
driver = "virtio-net-pci"
|
||||
netdev = "hostnet"
|
||||
bus = "pcie.2"
|
||||
addr = "00.0"
|
||||
|
||||
|
||||
# USB controller (and input devices)
|
||||
# =========================================================
|
||||
#
|
||||
# We add a virtualization-friendly USB 3.0 controller and
|
||||
# a USB tablet so that graphical guests can be controlled
|
||||
# appropriately. A USB keyboard is not needed, as q35
|
||||
# guests get a PS/2 one added automatically.
|
||||
|
||||
[device "usb"]
|
||||
driver = "nec-usb-xhci"
|
||||
bus = "pcie.3"
|
||||
addr = "00.0"
|
||||
|
||||
[device "tablet"]
|
||||
driver = "usb-tablet"
|
||||
bus = "usb.0"
|
||||
|
||||
|
||||
# VGA compatible controller
|
||||
# =========================================================
|
||||
#
|
||||
# We plug the QXL video card directly into the PCI Express
|
||||
# Root Bus as it is a legacy PCI device; this way, we can
|
||||
# reduce the number of PCI Express controllers in the
|
||||
# guest.
|
||||
#
|
||||
# If you're running the guest on a remote, potentially
|
||||
# headless host, you will probably want to append something
|
||||
# like
|
||||
#
|
||||
# -display vnc=127.0.0.1:0
|
||||
#
|
||||
# to the command line in order to prevent QEMU from
|
||||
# creating a graphical display window on the host and
|
||||
# enable remote access instead.
|
||||
|
||||
[device "video"]
|
||||
driver = "qxl-vga"
|
||||
bus = "pcie.0"
|
||||
addr = "01.0"
|
||||
|
||||
|
||||
# Audio device
|
||||
# =========================================================
|
||||
#
|
||||
# Like the video card, the sound card is a legacy PCI
|
||||
# device and as such can be plugged directly into the PCI
|
||||
# Express Root Bus.
|
||||
|
||||
[device "sound"]
|
||||
driver = "ich9-intel-hda"
|
||||
bus = "pcie.0"
|
||||
addr = "1b.0"
|
||||
|
||||
[device "duplex"]
|
||||
driver = "hda-duplex"
|
||||
bus = "sound.0"
|
||||
cad = "0"
|
||||
audiodev = "audiodev0"
|
||||
|
||||
[audiodev "audiodev0"]
|
||||
driver = "none" # CHANGE ME
|
||||
@@ -0,0 +1,195 @@
|
||||
# q35 - VirtIO guest (serial console)
|
||||
# =========================================================
|
||||
#
|
||||
# Usage:
|
||||
#
|
||||
# $ qemu-system-x86_64 \
|
||||
# -nodefaults \
|
||||
# -readconfig q35-virtio-serial.cfg \
|
||||
# -display none -serial mon:stdio
|
||||
#
|
||||
# You will probably need to tweak the lines marked as
|
||||
# CHANGE ME before being able to use this configuration!
|
||||
#
|
||||
# The guest will have a selection of VirtIO devices
|
||||
# tailored towards optimal performance with modern guests,
|
||||
# and will be accessed through the serial console.
|
||||
#
|
||||
# ---------------------------------------------------------
|
||||
#
|
||||
# Using -nodefaults is required to have full control over
|
||||
# the virtual hardware: when it's specified, QEMU will
|
||||
# populate the board with only the builtin peripherals
|
||||
# plus a small selection of core PCI devices and
|
||||
# controllers; the user will then have to explicitly add
|
||||
# further devices.
|
||||
#
|
||||
# The core PCI devices show up in the guest as:
|
||||
#
|
||||
# 00:00.0 Host bridge
|
||||
# 00:1f.0 ISA bridge / LPC
|
||||
# 00:1f.2 SATA (AHCI) controller
|
||||
# 00:1f.3 SMBus controller
|
||||
#
|
||||
# This configuration file adds a number of other useful
|
||||
# devices, more specifically:
|
||||
#
|
||||
# 00.1c.* PCI bridge (PCI Express Root Ports)
|
||||
# 01:00.0 SCSI storage controller
|
||||
# 02:00.0 Ethernet controller
|
||||
#
|
||||
# More information about these devices is available below.
|
||||
#
|
||||
# We use '-display none' to prevent QEMU from creating a
|
||||
# graphical display window, which would serve no use in
|
||||
# this specific configuration, and '-serial mon:stdio' to
|
||||
# multiplex the guest's serial console and the QEMU monitor
|
||||
# to the host's stdio; use 'Ctrl+A h' to learn how to
|
||||
# switch between the two and more.
|
||||
|
||||
|
||||
# Machine options
|
||||
# =========================================================
|
||||
#
|
||||
# We use the q35 machine type and enable KVM acceleration
|
||||
# for better performance.
|
||||
#
|
||||
# Using less than 1 GiB of memory is probably not going to
|
||||
# yield good performance in the guest, and might even lead
|
||||
# to obscure boot issues in some cases.
|
||||
|
||||
[machine]
|
||||
type = "q35"
|
||||
|
||||
[accel]
|
||||
accel = "kvm"
|
||||
|
||||
[memory]
|
||||
size = "1024"
|
||||
|
||||
|
||||
# PCI bridge (PCI Express Root Ports)
|
||||
# =========================================================
|
||||
#
|
||||
# We create eight PCI Express Root Ports, and we plug them
|
||||
# all into separate functions of the same slot. Some of
|
||||
# them will be used by devices, the rest will remain
|
||||
# available for hotplug.
|
||||
|
||||
[device "pcie.1"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.0"
|
||||
port = "1"
|
||||
chassis = "1"
|
||||
multifunction = "on"
|
||||
|
||||
[device "pcie.2"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.1"
|
||||
port = "2"
|
||||
chassis = "2"
|
||||
|
||||
[device "pcie.3"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.2"
|
||||
port = "3"
|
||||
chassis = "3"
|
||||
|
||||
[device "pcie.4"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.3"
|
||||
port = "4"
|
||||
chassis = "4"
|
||||
|
||||
[device "pcie.5"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.4"
|
||||
port = "5"
|
||||
chassis = "5"
|
||||
|
||||
[device "pcie.6"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.5"
|
||||
port = "6"
|
||||
chassis = "6"
|
||||
|
||||
[device "pcie.7"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.6"
|
||||
port = "7"
|
||||
chassis = "7"
|
||||
|
||||
[device "pcie.8"]
|
||||
driver = "pcie-root-port"
|
||||
bus = "pcie.0"
|
||||
addr = "1c.7"
|
||||
port = "8"
|
||||
chassis = "8"
|
||||
|
||||
|
||||
# SCSI storage controller (and storage)
|
||||
# =========================================================
|
||||
#
|
||||
# We use virtio-scsi here so that we can (hot)plug a large
|
||||
# number of disks without running into issues; a SCSI disk,
|
||||
# backed by a qcow2 disk image on the host's filesystem, is
|
||||
# attached to it.
|
||||
#
|
||||
# We also create an optical disk, mostly for installation
|
||||
# purposes: once the guest OS has been successfully
|
||||
# installed, the guest will no longer boot from optical
|
||||
# media. If you don't want, or no longer want, to have an
|
||||
# optical disk in the guest you can safely comment out
|
||||
# all relevant sections below.
|
||||
|
||||
[device "scsi"]
|
||||
driver = "virtio-scsi-pci"
|
||||
bus = "pcie.1"
|
||||
addr = "00.0"
|
||||
|
||||
[device "scsi-disk"]
|
||||
driver = "scsi-hd"
|
||||
bus = "scsi.0"
|
||||
drive = "disk"
|
||||
bootindex = "1"
|
||||
|
||||
[drive "disk"]
|
||||
file = "guest.qcow2" # CHANGE ME
|
||||
format = "qcow2"
|
||||
if = "none"
|
||||
|
||||
[device "scsi-optical-disk"]
|
||||
driver = "scsi-cd"
|
||||
bus = "scsi.0"
|
||||
drive = "optical-disk"
|
||||
bootindex = "2"
|
||||
|
||||
[drive "optical-disk"]
|
||||
file = "install.iso" # CHANGE ME
|
||||
format = "raw"
|
||||
if = "none"
|
||||
|
||||
|
||||
# Ethernet controller
|
||||
# =========================================================
|
||||
#
|
||||
# We use virtio-net for improved performance over emulated
|
||||
# hardware; on the host side, we take advantage of user
|
||||
# networking so that the QEMU process doesn't require any
|
||||
# additional privileges.
|
||||
|
||||
[netdev "hostnet"]
|
||||
type = "user"
|
||||
|
||||
[device "net"]
|
||||
driver = "virtio-net-pci"
|
||||
netdev = "hostnet"
|
||||
bus = "pcie.2"
|
||||
addr = "00.0"
|
||||
@@ -0,0 +1,15 @@
|
||||
..
|
||||
Generally useful rST substitution definitions. This is included for
|
||||
all rST files as part of the epilogue by docs/conf.py. conf.py
|
||||
also defines some dynamically generated substitutions like CONFDIR.
|
||||
|
||||
Note that |qemu_system| and |qemu_system_x86| are intended to be
|
||||
used inside a parsed-literal block: the definition must not include
|
||||
extra literal formatting with ``..``: this works in the HTML output
|
||||
but the manpages will end up misrendered with following normal text
|
||||
incorrectly in boldface.
|
||||
|
||||
.. |qemu_system| replace:: qemu-system-x86_64
|
||||
.. |qemu_system_x86| replace:: qemu-system-x86_64
|
||||
.. |I2C| replace:: I\ :sup:`2`\ C
|
||||
.. |I2S| replace:: I\ :sup:`2`\ S
|
||||
@@ -0,0 +1,515 @@
|
||||
.. _atomics-ref:
|
||||
|
||||
=========================
|
||||
Atomic operations in QEMU
|
||||
=========================
|
||||
|
||||
CPUs perform independent memory operations effectively in random order.
|
||||
but this can be a problem for CPU-CPU interaction (including interactions
|
||||
between QEMU and the guest). Multi-threaded programs use various tools
|
||||
to instruct the compiler and the CPU to restrict the order to something
|
||||
that is consistent with the expectations of the programmer.
|
||||
|
||||
The most basic tool is locking. Mutexes, condition variables and
|
||||
semaphores are used in QEMU, and should be the default approach to
|
||||
synchronization. Anything else is considerably harder, but it's
|
||||
also justified more often than one would like;
|
||||
the most performance-critical parts of QEMU in particular require
|
||||
a very low level approach to concurrency, involving memory barriers
|
||||
and atomic operations. The semantics of concurrent memory accesses are governed
|
||||
by the C11 memory model.
|
||||
|
||||
QEMU provides a header, ``qemu/atomic.h``, which wraps C11 atomics to
|
||||
provide better portability and a less verbose syntax. ``qemu/atomic.h``
|
||||
provides macros that fall in three camps:
|
||||
|
||||
- compiler barriers: ``barrier()``;
|
||||
|
||||
- weak atomic access and manual memory barriers: ``qatomic_read()``,
|
||||
``qatomic_set()``, ``smp_rmb()``, ``smp_wmb()``, ``smp_mb()``,
|
||||
``smp_mb_acquire()``, ``smp_mb_release()``, ``smp_read_barrier_depends()``,
|
||||
``smp_mb__before_rmw()``, ``smp_mb__after_rmw()``;
|
||||
|
||||
- sequentially consistent atomic access: everything else.
|
||||
|
||||
In general, use of ``qemu/atomic.h`` should be wrapped with more easily
|
||||
used data structures (e.g. the lock-free singly-linked list operations
|
||||
``QSLIST_INSERT_HEAD_ATOMIC`` and ``QSLIST_MOVE_ATOMIC``) or synchronization
|
||||
primitives (such as RCU, ``QemuEvent`` or ``QemuLockCnt``). Bare use of
|
||||
atomic operations and memory barriers should be limited to inter-thread
|
||||
checking of flags and documented thoroughly.
|
||||
|
||||
|
||||
|
||||
Compiler memory barrier
|
||||
=======================
|
||||
|
||||
``barrier()`` prevents the compiler from moving the memory accesses on
|
||||
either side of it to the other side. The compiler barrier has no direct
|
||||
effect on the CPU, which may then reorder things however it wishes.
|
||||
|
||||
``barrier()`` is mostly used within ``qemu/atomic.h`` itself. On some
|
||||
architectures, CPU guarantees are strong enough that blocking compiler
|
||||
optimizations already ensures the correct order of execution. In this
|
||||
case, ``qemu/atomic.h`` will reduce stronger memory barriers to simple
|
||||
compiler barriers.
|
||||
|
||||
Still, ``barrier()`` can be useful when writing code that can be interrupted
|
||||
by signal handlers.
|
||||
|
||||
|
||||
Sequentially consistent atomic access
|
||||
=====================================
|
||||
|
||||
Most of the operations in the ``qemu/atomic.h`` header ensure *sequential
|
||||
consistency*, where "the result of any execution is the same as if the
|
||||
operations of all the processors were executed in some sequential order,
|
||||
and the operations of each individual processor appear in this sequence
|
||||
in the order specified by its program".
|
||||
|
||||
``qemu/atomic.h`` provides the following set of atomic read-modify-write
|
||||
operations::
|
||||
|
||||
void qatomic_inc(ptr)
|
||||
void qatomic_dec(ptr)
|
||||
void qatomic_add(ptr, val)
|
||||
void qatomic_sub(ptr, val)
|
||||
void qatomic_and(ptr, val)
|
||||
void qatomic_or(ptr, val)
|
||||
|
||||
typeof(*ptr) qatomic_fetch_inc(ptr)
|
||||
typeof(*ptr) qatomic_fetch_dec(ptr)
|
||||
typeof(*ptr) qatomic_fetch_add(ptr, val)
|
||||
typeof(*ptr) qatomic_fetch_sub(ptr, val)
|
||||
typeof(*ptr) qatomic_fetch_and(ptr, val)
|
||||
typeof(*ptr) qatomic_fetch_or(ptr, val)
|
||||
typeof(*ptr) qatomic_fetch_xor(ptr, val)
|
||||
typeof(*ptr) qatomic_fetch_inc_nonzero(ptr)
|
||||
typeof(*ptr) qatomic_xchg(ptr, val)
|
||||
typeof(*ptr) qatomic_cmpxchg(ptr, old, new)
|
||||
|
||||
all of which return the old value of ``*ptr``. These operations are
|
||||
polymorphic; they operate on any type that is as wide as a pointer or
|
||||
smaller.
|
||||
|
||||
Similar operations return the new value of ``*ptr``::
|
||||
|
||||
typeof(*ptr) qatomic_inc_fetch(ptr)
|
||||
typeof(*ptr) qatomic_dec_fetch(ptr)
|
||||
typeof(*ptr) qatomic_add_fetch(ptr, val)
|
||||
typeof(*ptr) qatomic_sub_fetch(ptr, val)
|
||||
typeof(*ptr) qatomic_and_fetch(ptr, val)
|
||||
typeof(*ptr) qatomic_or_fetch(ptr, val)
|
||||
typeof(*ptr) qatomic_xor_fetch(ptr, val)
|
||||
|
||||
``qemu/atomic.h`` also provides an optimized shortcut for
|
||||
``qatomic_set`` followed by ``smp_mb``::
|
||||
|
||||
void qatomic_set_mb(ptr, val)
|
||||
|
||||
|
||||
Weak atomic access and manual memory barriers
|
||||
=============================================
|
||||
|
||||
Compared to sequentially consistent atomic access, programming with
|
||||
weaker consistency models can be considerably more complicated.
|
||||
The only guarantees that you can rely upon in this case are:
|
||||
|
||||
- atomic accesses will not cause data races (and hence undefined behavior);
|
||||
ordinary accesses instead cause data races if they are concurrent with
|
||||
other accesses of which at least one is a write. In order to ensure this,
|
||||
the compiler will not optimize accesses out of existence, create unsolicited
|
||||
accesses, or perform other similar optimizations.
|
||||
|
||||
- acquire operations will appear to happen, with respect to the other
|
||||
components of the system, before all the LOAD or STORE operations
|
||||
specified afterwards.
|
||||
|
||||
- release operations will appear to happen, with respect to the other
|
||||
components of the system, after all the LOAD or STORE operations
|
||||
specified before.
|
||||
|
||||
- release operations will *synchronize with* acquire operations;
|
||||
see :ref:`acqrel` for a detailed explanation.
|
||||
|
||||
When using this model, variables are accessed with:
|
||||
|
||||
- ``qatomic_read()`` and ``qatomic_set()``; these prevent the compiler from
|
||||
optimizing accesses out of existence and creating unsolicited
|
||||
accesses, but do not otherwise impose any ordering on loads and
|
||||
stores: both the compiler and the processor are free to reorder
|
||||
them.
|
||||
|
||||
- ``qatomic_load_acquire()``, which guarantees the LOAD to appear to
|
||||
happen, with respect to the other components of the system,
|
||||
before all the LOAD or STORE operations specified afterwards.
|
||||
Operations coming before ``qatomic_load_acquire()`` can still be
|
||||
reordered after it.
|
||||
|
||||
- ``qatomic_store_release()``, which guarantees the STORE to appear to
|
||||
happen, with respect to the other components of the system,
|
||||
after all the LOAD or STORE operations specified before.
|
||||
Operations coming after ``qatomic_store_release()`` can still be
|
||||
reordered before it.
|
||||
|
||||
Restrictions to the ordering of accesses can also be specified
|
||||
using the memory barrier macros: ``smp_rmb()``, ``smp_wmb()``, ``smp_mb()``,
|
||||
``smp_mb_acquire()``, ``smp_mb_release()``, ``smp_read_barrier_depends()``.
|
||||
|
||||
Memory barriers control the order of references to shared memory.
|
||||
They come in six kinds:
|
||||
|
||||
- ``smp_rmb()`` guarantees that all the LOAD operations specified before
|
||||
the barrier will appear to happen before all the LOAD operations
|
||||
specified after the barrier with respect to the other components of
|
||||
the system.
|
||||
|
||||
In other words, ``smp_rmb()`` puts a partial ordering on loads, but is not
|
||||
required to have any effect on stores.
|
||||
|
||||
- ``smp_wmb()`` guarantees that all the STORE operations specified before
|
||||
the barrier will appear to happen before all the STORE operations
|
||||
specified after the barrier with respect to the other components of
|
||||
the system.
|
||||
|
||||
In other words, ``smp_wmb()`` puts a partial ordering on stores, but is not
|
||||
required to have any effect on loads.
|
||||
|
||||
- ``smp_mb_acquire()`` guarantees that all the LOAD operations specified before
|
||||
the barrier will appear to happen before all the LOAD or STORE operations
|
||||
specified after the barrier with respect to the other components of
|
||||
the system.
|
||||
|
||||
- ``smp_mb_release()`` guarantees that all the STORE operations specified *after*
|
||||
the barrier will appear to happen after all the LOAD or STORE operations
|
||||
specified *before* the barrier with respect to the other components of
|
||||
the system.
|
||||
|
||||
- ``smp_mb()`` guarantees that all the LOAD and STORE operations specified
|
||||
before the barrier will appear to happen before all the LOAD and
|
||||
STORE operations specified after the barrier with respect to the other
|
||||
components of the system.
|
||||
|
||||
``smp_mb()`` puts a partial ordering on both loads and stores. It is
|
||||
stronger than both a read and a write memory barrier; it implies both
|
||||
``smp_mb_acquire()`` and ``smp_mb_release()``, but it also prevents STOREs
|
||||
coming before the barrier from overtaking LOADs coming after the
|
||||
barrier and vice versa.
|
||||
|
||||
- ``smp_read_barrier_depends()`` is a weaker kind of read barrier. On
|
||||
most processors, whenever two loads are performed such that the
|
||||
second depends on the result of the first (e.g., the first load
|
||||
retrieves the address to which the second load will be directed),
|
||||
the processor will guarantee that the first LOAD will appear to happen
|
||||
before the second with respect to the other components of the system.
|
||||
Therefore, unlike ``smp_rmb()`` or ``qatomic_load_acquire()``,
|
||||
``smp_read_barrier_depends()`` can be just a compiler barrier on
|
||||
weakly-ordered architectures such as Arm or PPC\ [#alpha]_.
|
||||
|
||||
Note that the first load really has to have a _data_ dependency and not
|
||||
a control dependency. If the address for the second load is dependent
|
||||
on the first load, but the dependency is through a conditional rather
|
||||
than actually loading the address itself, then it's a _control_
|
||||
dependency and a full read barrier or better is required.
|
||||
|
||||
.. [#alpha] The DEC Alpha is an exception, because ``smp_read_barrier_depends()``
|
||||
needs a processor barrier. On strongly-ordered architectures such
|
||||
as x86 or s390, ``smp_rmb()`` and ``qatomic_load_acquire()`` can
|
||||
also be compiler barriers only.
|
||||
|
||||
Memory barriers and ``qatomic_load_acquire``/``qatomic_store_release`` are
|
||||
mostly used when a data structure has one thread that is always a writer
|
||||
and one thread that is always a reader:
|
||||
|
||||
+----------------------------------+----------------------------------+
|
||||
| thread 1 | thread 2 |
|
||||
+==================================+==================================+
|
||||
| :: | :: |
|
||||
| | |
|
||||
| qatomic_store_release(&a, x); | y = qatomic_load_acquire(&b); |
|
||||
| qatomic_store_release(&b, y); | x = qatomic_load_acquire(&a); |
|
||||
+----------------------------------+----------------------------------+
|
||||
|
||||
In this case, correctness is easy to check for using the "pairing"
|
||||
trick that is explained below.
|
||||
|
||||
Sometimes, a thread is accessing many variables that are otherwise
|
||||
unrelated to each other (for example because, apart from the current
|
||||
thread, exactly one other thread will read or write each of these
|
||||
variables). In this case, it is possible to "hoist" the barriers
|
||||
outside a loop. For example:
|
||||
|
||||
+------------------------------------------+----------------------------------+
|
||||
| before | after |
|
||||
+==========================================+==================================+
|
||||
| :: | :: |
|
||||
| | |
|
||||
| n = 0; | n = 0; |
|
||||
| for (i = 0; i < 10; i++) | for (i = 0; i < 10; i++) |
|
||||
| n += qatomic_load_acquire(&a[i]); | n += qatomic_read(&a[i]); |
|
||||
| | smp_mb_acquire(); |
|
||||
+------------------------------------------+----------------------------------+
|
||||
| :: | :: |
|
||||
| | |
|
||||
| | smp_mb_release(); |
|
||||
| for (i = 0; i < 10; i++) | for (i = 0; i < 10; i++) |
|
||||
| qatomic_store_release(&a[i], false); | qatomic_set(&a[i], false); |
|
||||
+------------------------------------------+----------------------------------+
|
||||
|
||||
Splitting a loop can also be useful to reduce the number of barriers:
|
||||
|
||||
+------------------------------------------+----------------------------------+
|
||||
| before | after |
|
||||
+==========================================+==================================+
|
||||
| :: | :: |
|
||||
| | |
|
||||
| n = 0; | smp_mb_release(); |
|
||||
| for (i = 0; i < 10; i++) { | for (i = 0; i < 10; i++) |
|
||||
| qatomic_store_release(&a[i], false); | qatomic_set(&a[i], false); |
|
||||
| smp_mb(); | smp_mb(); |
|
||||
| n += qatomic_read(&b[i]); | n = 0; |
|
||||
| } | for (i = 0; i < 10; i++) |
|
||||
| | n += qatomic_read(&b[i]); |
|
||||
+------------------------------------------+----------------------------------+
|
||||
|
||||
In this case, a ``smp_mb_release()`` is also replaced with a (possibly cheaper, and clearer
|
||||
as well) ``smp_wmb()``:
|
||||
|
||||
+------------------------------------------+----------------------------------+
|
||||
| before | after |
|
||||
+==========================================+==================================+
|
||||
| :: | :: |
|
||||
| | |
|
||||
| | smp_mb_release(); |
|
||||
| for (i = 0; i < 10; i++) { | for (i = 0; i < 10; i++) |
|
||||
| qatomic_store_release(&a[i], false); | qatomic_set(&a[i], false); |
|
||||
| qatomic_store_release(&b[i], false); | smp_wmb(); |
|
||||
| } | for (i = 0; i < 10; i++) |
|
||||
| | qatomic_set(&b[i], false); |
|
||||
+------------------------------------------+----------------------------------+
|
||||
|
||||
|
||||
.. _acqrel:
|
||||
|
||||
Acquire/release pairing and the *synchronizes-with* relation
|
||||
------------------------------------------------------------
|
||||
|
||||
Atomic operations other than ``qatomic_set()`` and ``qatomic_read()`` have
|
||||
either *acquire* or *release* semantics\ [#rmw]_. This has two effects:
|
||||
|
||||
.. [#rmw] Read-modify-write operations can have both---acquire applies to the
|
||||
read part, and release to the write.
|
||||
|
||||
- within a thread, they are ordered either before subsequent operations
|
||||
(for acquire) or after previous operations (for release).
|
||||
|
||||
- if a release operation in one thread *synchronizes with* an acquire operation
|
||||
in another thread, the ordering constraints propagates from the first to the
|
||||
second thread. That is, everything before the release operation in the
|
||||
first thread is guaranteed to *happen before* everything after the
|
||||
acquire operation in the second thread.
|
||||
|
||||
The concept of acquire and release semantics is not exclusive to atomic
|
||||
operations; almost all higher-level synchronization primitives also have
|
||||
acquire or release semantics. For example:
|
||||
|
||||
- ``pthread_mutex_lock`` has acquire semantics, ``pthread_mutex_unlock`` has
|
||||
release semantics and synchronizes with a ``pthread_mutex_lock`` for the
|
||||
same mutex.
|
||||
|
||||
- ``pthread_cond_signal`` and ``pthread_cond_broadcast`` have release semantics;
|
||||
``pthread_cond_wait`` has both release semantics (synchronizing with
|
||||
``pthread_mutex_lock``) and acquire semantics (synchronizing with
|
||||
``pthread_mutex_unlock`` and signaling of the condition variable).
|
||||
|
||||
- ``pthread_create`` has release semantics and synchronizes with the start
|
||||
of the new thread; ``pthread_join`` has acquire semantics and synchronizes
|
||||
with the exiting of the thread.
|
||||
|
||||
- ``qemu_event_set`` has release semantics, ``qemu_event_wait`` has
|
||||
acquire semantics.
|
||||
|
||||
For example, in the following example there are no atomic accesses, but still
|
||||
thread 2 is relying on the *synchronizes-with* relation between ``pthread_exit``
|
||||
(release) and ``pthread_join`` (acquire):
|
||||
|
||||
+----------------------+-------------------------------+
|
||||
| thread 1 | thread 2 |
|
||||
+======================+===============================+
|
||||
| :: | :: |
|
||||
| | |
|
||||
| *a = 1; | |
|
||||
| pthread_exit(a); | pthread_join(thread1, &a); |
|
||||
| | x = *a; |
|
||||
+----------------------+-------------------------------+
|
||||
|
||||
Synchronization between threads basically descends from this pairing of
|
||||
a release operation and an acquire operation. Therefore, atomic operations
|
||||
other than ``qatomic_set()`` and ``qatomic_read()`` will almost always be
|
||||
paired with another operation of the opposite kind: an acquire operation
|
||||
will pair with a release operation and vice versa. This rule of thumb is
|
||||
extremely useful; in the case of QEMU, however, note that the other
|
||||
operation may actually be in a driver that runs in the guest!
|
||||
|
||||
``smp_read_barrier_depends()``, ``smp_rmb()``, ``smp_mb_acquire()``,
|
||||
``qatomic_load_acquire()`` and ``qatomic_rcu_read()`` all count
|
||||
as acquire operations. ``smp_wmb()``, ``smp_mb_release()``,
|
||||
``qatomic_store_release()`` and ``qatomic_rcu_set()`` all count as release
|
||||
operations. ``smp_mb()`` counts as both acquire and release, therefore
|
||||
it can pair with any other atomic operation. Here is an example:
|
||||
|
||||
+----------------------+------------------------------+
|
||||
| thread 1 | thread 2 |
|
||||
+======================+==============================+
|
||||
| :: | :: |
|
||||
| | |
|
||||
| qatomic_set(&a, 1);| |
|
||||
| smp_wmb(); | |
|
||||
| qatomic_set(&b, 2);| x = qatomic_read(&b); |
|
||||
| | smp_rmb(); |
|
||||
| | y = qatomic_read(&a); |
|
||||
+----------------------+------------------------------+
|
||||
|
||||
Note that a load-store pair only counts if the two operations access the
|
||||
same variable: that is, a store-release on a variable ``x`` *synchronizes
|
||||
with* a load-acquire on a variable ``x``, while a release barrier
|
||||
synchronizes with any acquire operation. The following example shows
|
||||
correct synchronization:
|
||||
|
||||
+--------------------------------+--------------------------------+
|
||||
| thread 1 | thread 2 |
|
||||
+================================+================================+
|
||||
| :: | :: |
|
||||
| | |
|
||||
| qatomic_set(&a, 1); | |
|
||||
| qatomic_store_release(&b, 2);| x = qatomic_load_acquire(&b);|
|
||||
| | y = qatomic_read(&a); |
|
||||
+--------------------------------+--------------------------------+
|
||||
|
||||
Acquire and release semantics of higher-level primitives can also be
|
||||
relied upon for the purpose of establishing the *synchronizes with*
|
||||
relation.
|
||||
|
||||
Note that the "writing" thread is accessing the variables in the
|
||||
opposite order as the "reading" thread. This is expected: stores
|
||||
before a release operation will normally match the loads after
|
||||
the acquire operation, and vice versa. In fact, this happened already
|
||||
in the ``pthread_exit``/``pthread_join`` example above.
|
||||
|
||||
Finally, this more complex example has more than two accesses and data
|
||||
dependency barriers. It also does not use atomic accesses whenever there
|
||||
cannot be a data race:
|
||||
|
||||
+----------------------+------------------------------+
|
||||
| thread 1 | thread 2 |
|
||||
+======================+==============================+
|
||||
| :: | :: |
|
||||
| | |
|
||||
| b[2] = 1; | |
|
||||
| smp_wmb(); | |
|
||||
| x->i = 2; | |
|
||||
| smp_wmb(); | |
|
||||
| qatomic_set(&a, x);| x = qatomic_read(&a); |
|
||||
| | smp_read_barrier_depends(); |
|
||||
| | y = x->i; |
|
||||
| | smp_read_barrier_depends(); |
|
||||
| | z = b[y]; |
|
||||
+----------------------+------------------------------+
|
||||
|
||||
Comparison with Linux kernel primitives
|
||||
=======================================
|
||||
|
||||
Here is a list of differences between Linux kernel atomic operations
|
||||
and memory barriers, and the equivalents in QEMU:
|
||||
|
||||
- atomic operations in Linux are always on a 32-bit int type and
|
||||
use a boxed ``atomic_t`` type; atomic operations in QEMU are polymorphic
|
||||
and use normal C types.
|
||||
|
||||
- Originally, ``atomic_read`` and ``atomic_set`` in Linux gave no guarantee
|
||||
at all. Linux 4.1 updated them to implement volatile
|
||||
semantics via ``ACCESS_ONCE`` (or the more recent ``READ``/``WRITE_ONCE``).
|
||||
|
||||
QEMU's ``qatomic_read`` and ``qatomic_set`` implement C11 atomic relaxed
|
||||
semantics if the compiler supports it, and volatile semantics otherwise.
|
||||
Both semantics prevent the compiler from doing certain transformations;
|
||||
the difference is that atomic accesses are guaranteed to be atomic,
|
||||
while volatile accesses aren't. Thus, in the volatile case we just cross
|
||||
our fingers hoping that the compiler will generate atomic accesses,
|
||||
since we assume the variables passed are machine-word sized and
|
||||
properly aligned.
|
||||
|
||||
No barriers are implied by ``qatomic_read`` and ``qatomic_set`` in either
|
||||
Linux or QEMU.
|
||||
|
||||
- atomic read-modify-write operations in Linux are of three kinds:
|
||||
|
||||
===================== =========================================
|
||||
``atomic_OP`` returns void
|
||||
``atomic_OP_return`` returns new value of the variable
|
||||
``atomic_fetch_OP`` returns the old value of the variable
|
||||
``atomic_cmpxchg`` returns the old value of the variable
|
||||
===================== =========================================
|
||||
|
||||
In QEMU, the second kind is named ``atomic_OP_fetch``.
|
||||
|
||||
- different atomic read-modify-write operations in Linux imply
|
||||
a different set of memory barriers. In QEMU, all of them enforce
|
||||
sequential consistency: there is a single order in which the
|
||||
program sees them happen.
|
||||
|
||||
- however, according to the C11 memory model that QEMU uses, this order
|
||||
does not propagate to other memory accesses on either side of the
|
||||
read-modify-write operation. As far as those are concerned, the
|
||||
operation consist of just a load-acquire followed by a store-release.
|
||||
Stores that precede the RMW operation, and loads that follow it, can
|
||||
still be reordered and will happen *in the middle* of the read-modify-write
|
||||
operation!
|
||||
|
||||
Therefore, the following example is correct in Linux but not in QEMU:
|
||||
|
||||
+----------------------------------+--------------------------------+
|
||||
| Linux (correct) | QEMU (incorrect) |
|
||||
+==================================+================================+
|
||||
| :: | :: |
|
||||
| | |
|
||||
| a = atomic_fetch_add(&x, 2); | a = qatomic_fetch_add(&x, 2);|
|
||||
| b = READ_ONCE(&y); | b = qatomic_read(&y); |
|
||||
+----------------------------------+--------------------------------+
|
||||
|
||||
because the read of ``y`` can be moved (by either the processor or the
|
||||
compiler) before the write of ``x``.
|
||||
|
||||
Fixing this requires a full memory barrier between the write of ``x`` and
|
||||
the read of ``y``. QEMU provides ``smp_mb__before_rmw()`` and
|
||||
``smp_mb__after_rmw()``; they act both as an optimization,
|
||||
avoiding the memory barrier on processors where it is unnecessary,
|
||||
and as a clarification of this corner case of the C11 memory model:
|
||||
|
||||
+--------------------------------+
|
||||
| QEMU (correct) |
|
||||
+================================+
|
||||
| :: |
|
||||
| |
|
||||
| a = qatomic_fetch_add(&x, 2);|
|
||||
| smp_mb__after_rmw(); |
|
||||
| b = qatomic_read(&y); |
|
||||
+--------------------------------+
|
||||
|
||||
In the common case where only one thread writes ``x``, it is also possible
|
||||
to write it like this:
|
||||
|
||||
+--------------------------------+
|
||||
| QEMU (correct) |
|
||||
+================================+
|
||||
| :: |
|
||||
| |
|
||||
| a = qatomic_read(&x); |
|
||||
| qatomic_set_mb(&x, a + 2); |
|
||||
| b = qatomic_read(&y); |
|
||||
+--------------------------------+
|
||||
|
||||
Sources
|
||||
=======
|
||||
|
||||
- ``Documentation/memory-barriers.txt`` from the Linux kernel
|
||||
@@ -0,0 +1,8 @@
|
||||
==================
|
||||
Bitwise operations
|
||||
==================
|
||||
|
||||
The header ``qemu/bitops.h`` provides utility functions for
|
||||
performing bitwise operations.
|
||||
|
||||
.. kernel-doc:: include/qemu/bitops.h
|
||||
@@ -0,0 +1,54 @@
|
||||
=======================
|
||||
block-coroutine-wrapper
|
||||
=======================
|
||||
|
||||
A lot of functions in QEMU block layer (see ``block/*``) can only be
|
||||
called in coroutine context. Such functions are normally marked by the
|
||||
coroutine_fn specifier. Still, sometimes we need to call them from
|
||||
non-coroutine context; for this we need to start a coroutine, run the
|
||||
needed function from it and wait for the coroutine to finish in a
|
||||
BDRV_POLL_WHILE() loop. To run a coroutine we need a function with one
|
||||
void* argument. So for each coroutine_fn function which needs a
|
||||
non-coroutine interface, we should define a structure to pack the
|
||||
parameters, define a separate function to unpack the parameters and
|
||||
call the original function and finally define a new interface function
|
||||
with same list of arguments as original one, which will pack the
|
||||
parameters into a struct, create a coroutine, run it and wait in
|
||||
BDRV_POLL_WHILE() loop. It's boring to create such wrappers by hand,
|
||||
so we have a script to generate them.
|
||||
|
||||
Usage
|
||||
=====
|
||||
|
||||
Assume we have defined the ``coroutine_fn`` function
|
||||
``bdrv_co_foo(<some args>)`` and need a non-coroutine interface for it,
|
||||
called ``bdrv_foo(<same args>)``. In this case the script can help. To
|
||||
trigger the generation:
|
||||
|
||||
1. You need ``bdrv_foo`` declaration somewhere (for example, in
|
||||
``block/coroutines.h``) with the ``co_wrapper`` mark,
|
||||
like this:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
int co_wrapper bdrv_foo(<some args>);
|
||||
|
||||
2. You need to feed this declaration to block-coroutine-wrapper script.
|
||||
For this, add the .h (or .c) file with the declaration to the
|
||||
``input: files(...)`` list of ``block_gen_c`` target declaration in
|
||||
``block/meson.build``
|
||||
|
||||
You are done. During the build, coroutine wrappers will be generated in
|
||||
``<BUILD_DIR>/block/block-gen.c``.
|
||||
|
||||
Links
|
||||
=====
|
||||
|
||||
1. The script location is ``scripts/block-coroutine-wrapper.py``.
|
||||
|
||||
2. Generic place for private ``co_wrapper`` declarations is
|
||||
``block/coroutines.h``, for public declarations:
|
||||
``include/block/block.h``
|
||||
|
||||
3. The core API of generated coroutine wrappers is placed in
|
||||
(not generated) ``block/block-gen.h``
|
||||
@@ -0,0 +1,165 @@
|
||||
|
||||
.. _setup-build-env:
|
||||
|
||||
Setup build environment
|
||||
=======================
|
||||
|
||||
QEMU uses a lot of dependencies on the host system a large number of
|
||||
which are optional. At a minimum we expect to have a system C library
|
||||
(usually glibc but others can work), the glib2 library (used heavily
|
||||
in the code base) and a few other core libraries for interfacing with
|
||||
code modules and system build descriptions.
|
||||
|
||||
We use the ``libvirt-ci`` project to handle the mapping of
|
||||
dependencies to a wide variety output formats including system install
|
||||
scripts. For example:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# THIS FILE WAS AUTO-GENERATED
|
||||
#
|
||||
# $ lcitool buildenvscript debian-13 ./tests/lcitool/projects/qemu-minimal.yml
|
||||
#
|
||||
# https://gitlab.com/libvirt/libvirt-ci
|
||||
|
||||
function install_buildenv() {
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
apt-get update
|
||||
apt-get dist-upgrade -y
|
||||
apt-get install --no-install-recommends -y \
|
||||
bash \
|
||||
bc \
|
||||
bison \
|
||||
bzip2 \
|
||||
ca-certificates \
|
||||
ccache \
|
||||
findutils \
|
||||
flex \
|
||||
gcc \
|
||||
git \
|
||||
libc6-dev \
|
||||
libfdt-dev \
|
||||
libffi-dev \
|
||||
libglib2.0-dev \
|
||||
libpixman-1-dev \
|
||||
locales \
|
||||
make \
|
||||
meson \
|
||||
ninja-build \
|
||||
pkgconf \
|
||||
python3 \
|
||||
python3-venv \
|
||||
sed \
|
||||
tar
|
||||
sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen
|
||||
dpkg-reconfigure locales
|
||||
rm -f /usr/lib*/python3*/EXTERNALLY-MANAGED
|
||||
dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt
|
||||
mkdir -p /usr/libexec/ccache-wrappers
|
||||
ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc
|
||||
ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc
|
||||
}
|
||||
|
||||
export CCACHE_WRAPPERSDIR="/usr/libexec/ccache-wrappers"
|
||||
export LANG="en_US.UTF-8"
|
||||
export MAKE="/usr/bin/make"
|
||||
export NINJA="/usr/bin/ninja"
|
||||
export PYTHON="/usr/bin/python3"
|
||||
|
||||
If you instead select the ``qemu.yml`` project file you will get all
|
||||
the dependencies that the project can use.
|
||||
|
||||
Using you system package manager
|
||||
--------------------------------
|
||||
|
||||
.. note::
|
||||
|
||||
Installing dependencies using your package manager build dependencies may
|
||||
miss out on deps that have been newly introduced in qemu.git. It
|
||||
also misses deps the distribution has decided to exclude.
|
||||
|
||||
Systems with Package Managers
|
||||
+++++++++++++++++++++++++++++
|
||||
|
||||
.. list-table:: Package Manager Commands
|
||||
:widths: 10 50 40
|
||||
:header-rows: 1
|
||||
|
||||
* - System
|
||||
- Command
|
||||
- Notes
|
||||
* - Fedora
|
||||
- ``sudo dnf update && sudo dnf builddep qemu``
|
||||
-
|
||||
* - Debian/Ubuntu
|
||||
- ``sudo apt update && sudo apt build-dep qemu``
|
||||
- Must enable `Sources List
|
||||
<https://wiki.debian.org/SourcesList>`_ first
|
||||
* - MacOS
|
||||
- ``brew update && brew install $(brew deps --include-build qemu)``
|
||||
- Using `Homebrew <https://brew.sh/>`_.
|
||||
|
||||
Windows
|
||||
+++++++
|
||||
|
||||
You first need to install `MSYS2 <https://www.msys2.org/>`_.
|
||||
MSYS2 offers `different environments <https://www.msys2.org/docs/environments/>`_.
|
||||
x86_64 environments are based on GCC, while aarch64 is based on Clang.
|
||||
|
||||
We recommend to use MINGW64 for windows-x86_64 and CLANGARM64 for windows-aarch64
|
||||
(only available on windows-aarch64 hosts).
|
||||
|
||||
Then, you can open a windows shell, and enter msys2 env using:
|
||||
|
||||
::
|
||||
|
||||
c:/msys64/msys2_shell.cmd -defterm -here -no-start -mingw64
|
||||
# Replace -ucrt64 by -clangarm64 or -ucrt64 for other environments.
|
||||
|
||||
MSYS2 package manager does not offer a built-in way to install build
|
||||
dependencies. You can start with this list of packages using pacman:
|
||||
|
||||
Note: Dependencies need to be installed again if you use a different MSYS2
|
||||
environment.
|
||||
|
||||
::
|
||||
|
||||
# update MSYS2 itself, you need to reopen your shell at the end.
|
||||
pacman -Syu
|
||||
pacman -S \
|
||||
base-devel binutils bison diffutils flex git grep make sed \
|
||||
${MINGW_PACKAGE_PREFIX}-toolchain \
|
||||
${MINGW_PACKAGE_PREFIX}-glib2 \
|
||||
${MINGW_PACKAGE_PREFIX}-gtk3 \
|
||||
${MINGW_PACKAGE_PREFIX}-libnfs \
|
||||
${MINGW_PACKAGE_PREFIX}-libssh \
|
||||
${MINGW_PACKAGE_PREFIX}-ninja \
|
||||
${MINGW_PACKAGE_PREFIX}-pixman \
|
||||
${MINGW_PACKAGE_PREFIX}-pkgconf \
|
||||
${MINGW_PACKAGE_PREFIX}-python \
|
||||
${MINGW_PACKAGE_PREFIX}-SDL2 \
|
||||
${MINGW_PACKAGE_PREFIX}-zstd
|
||||
|
||||
If you want to install all dependencies, it's possible to use recipe used to
|
||||
build QEMU in MSYS2 itself.
|
||||
|
||||
::
|
||||
|
||||
pacman -S wget base-devel git
|
||||
wget https://raw.githubusercontent.com/msys2/MINGW-packages/refs/heads/master/mingw-w64-qemu/PKGBUILD
|
||||
# Some packages may be missing for your environment, installation will still
|
||||
# be done though.
|
||||
makepkg --syncdeps --nobuild PKGBUILD || true
|
||||
|
||||
Build on windows-aarch64
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
When trying to cross compile meson for x86_64 using UCRT64 or MINGW64 env,
|
||||
configure will run into an error because the cpu detected is not correct.
|
||||
|
||||
Meson detects x86_64 processes emulated, so you need to manually set the cpu,
|
||||
and force a cross compilation (with empty prefix).
|
||||
|
||||
::
|
||||
|
||||
./configure --cpu=x86_64 --cross-prefix=
|
||||
@@ -0,0 +1,604 @@
|
||||
==================================
|
||||
The QEMU build system architecture
|
||||
==================================
|
||||
|
||||
This document aims to help developers understand the architecture of the
|
||||
QEMU build system. As with projects using GNU autotools, the QEMU build
|
||||
system has two stages; first the developer runs the "configure" script
|
||||
to determine the local build environment characteristics, then they run
|
||||
"make" to build the project. This is about where the similarities with
|
||||
GNU autotools end, so try to forget what you know about them.
|
||||
|
||||
The two general ways to perform a build are as follows:
|
||||
|
||||
- build artifacts outside of QEMU source tree entirely::
|
||||
|
||||
cd ../
|
||||
mkdir build
|
||||
cd build
|
||||
../qemu/configure
|
||||
make
|
||||
|
||||
- build artifacts in a subdir of QEMU source tree::
|
||||
|
||||
mkdir build
|
||||
cd build
|
||||
../configure
|
||||
make
|
||||
|
||||
Most of the actual build process uses Meson under the hood, therefore
|
||||
build artifacts cannot be placed in the source tree itself.
|
||||
|
||||
|
||||
Stage 1: configure
|
||||
==================
|
||||
|
||||
The configure script has five tasks:
|
||||
|
||||
- detect the host architecture
|
||||
|
||||
- list the targets for which to build emulators; the list of
|
||||
targets also affects which firmware binaries and tests to build
|
||||
|
||||
- find the compilers (native and cross) used to build executables,
|
||||
firmware and tests. The results are written as either Makefile
|
||||
fragments (``config-host.mak``) or a Meson machine file
|
||||
(``config-meson.cross``)
|
||||
|
||||
- create a virtual environment in which all Python code runs during
|
||||
the build, and possibly install packages into it from PyPI
|
||||
|
||||
- invoke Meson in the virtual environment, to perform the actual
|
||||
configuration step for the emulator build
|
||||
|
||||
The configure script automatically recognizes command line options for
|
||||
which a same-named Meson option exists; dashes in the command line are
|
||||
replaced with underscores.
|
||||
|
||||
Almost all QEMU developers that need to modify the build system will
|
||||
only be concerned with Meson, and therefore can skip the rest of this
|
||||
section.
|
||||
|
||||
|
||||
Modifying ``configure``
|
||||
-----------------------
|
||||
|
||||
``configure`` is a shell script; it uses ``#!/bin/sh`` and therefore
|
||||
should be compatible with any POSIX shell. It is important to avoid
|
||||
using bash-isms to avoid breaking development platforms where bash is
|
||||
not the default shell implementation.
|
||||
|
||||
The configure script provides a variety of functions to help writing
|
||||
portable shell code and providing consistent behavior across architectures
|
||||
and operating systems:
|
||||
|
||||
``error_exit $MESSAGE $MORE...``
|
||||
Print $MESSAGE to stderr, followed by $MORE... and then exit from the
|
||||
configure script with non-zero status.
|
||||
|
||||
``has $COMMAND``
|
||||
Determine if $COMMAND exists in the current environment, either as a
|
||||
shell builtin, or executable binary, returning 0 on success. The
|
||||
replacement in Meson is ``find_program()``.
|
||||
|
||||
``probe_target_compiler $TARGET``
|
||||
Detect a cross compiler and cross tools for the QEMU target $TARGET (e.g.,
|
||||
``$CPU-softmmu``, ``$CPU-linux-user``, ``$CPU-bsd-user``). If a working
|
||||
compiler is present, return success and set variables ``$target_cc``,
|
||||
``$target_ar``, etc. to non-empty values.
|
||||
|
||||
``write_target_makefile``
|
||||
Write a Makefile fragment to stdout, exposing the result of the most recent
|
||||
``probe_target_compiler`` call as the usual Make variables (``CC``,
|
||||
``AR``, ``LD``, etc.).
|
||||
|
||||
|
||||
Configure does not generally perform tests for compiler options beyond
|
||||
basic checks to detect the host platform and ensure the compiler is
|
||||
functioning. These are performed using a few more helper functions:
|
||||
|
||||
``compile_object $CFLAGS``
|
||||
Attempt to compile a test program with the system C compiler using
|
||||
$CFLAGS. The test program must have been previously written to a file
|
||||
called $TMPC.
|
||||
|
||||
``compile_prog $CFLAGS $LDFLAGS``
|
||||
Attempt to compile a test program with the system C compiler using
|
||||
$CFLAGS and link it with the system linker using $LDFLAGS. The test
|
||||
program must have been previously written to a file called $TMPC.
|
||||
|
||||
``check_define $NAME``
|
||||
Determine if the macro $NAME is defined by the system C compiler.
|
||||
|
||||
``do_compiler $CC $ARGS...``
|
||||
Attempt to run the C compiler $CC, passing it $ARGS... This function
|
||||
does not use flags passed via options such as ``--extra-cflags``, and
|
||||
therefore can be used to check for cross compilers. However, most
|
||||
such checks are done at ``make`` time instead (see for example the
|
||||
``cc-option`` macro in ``pc-bios/option-rom/Makefile``).
|
||||
|
||||
``write_c_skeleton``
|
||||
Write a minimal C program main() function to the temporary file
|
||||
indicated by $TMPC.
|
||||
|
||||
|
||||
Python virtual environments and the build process
|
||||
-------------------------------------------------
|
||||
|
||||
An important step in ``configure`` is to create a Python virtual
|
||||
environment (venv) during the configuration phase. The Python interpreter
|
||||
comes from the ``--python`` command line option, the ``$PYTHON`` variable
|
||||
from the environment, or the system PATH, in this order. The venv resides
|
||||
in the ``pyvenv`` directory in the build tree, and provides consistency
|
||||
in how the build process runs Python code.
|
||||
|
||||
At this stage, ``configure`` also queries the chosen Python interpreter
|
||||
about QEMU's build dependencies. Note that the build process does *not*
|
||||
look for ``meson`` or ``sphinx-build`` binaries in the PATH;
|
||||
likewise, there are no options such as ``--meson`` or ``--sphinx-build``.
|
||||
This avoids a potential mismatch, where Meson and Sphinx binaries on the
|
||||
PATH might operate in a different Python environment than the one chosen
|
||||
by the user during the build process. On the other hand, it introduces
|
||||
a potential source of confusion where the user installs a dependency but
|
||||
``configure`` is not able to find it. When this happens, the dependency
|
||||
was installed in the ``site-packages`` directory of another interpreter,
|
||||
or with the wrong ``pip`` program.
|
||||
|
||||
If a package is available for the chosen interpreter, ``configure``
|
||||
prepares a small script that invokes it from the venv itself\ [#distlib]_.
|
||||
If not, ``configure`` can also optionally install dependencies in the
|
||||
virtual environment with ``pip``, either from wheels in ``python/wheels``
|
||||
or by downloading the package with PyPI. Downloading can be disabled with
|
||||
``--disable-download``; and anyway, it only happens when a ``configure``
|
||||
option (currently, only ``--enable-docs``) is explicitly enabled but
|
||||
the dependencies are not present.
|
||||
|
||||
.. [#distlib] The scripts are created based on the package's metadata,
|
||||
specifically the ``console_script`` entry points. This is the
|
||||
same mechanism that ``pip`` uses when installing a package.
|
||||
Currently, in all cases it would be possible to use ``python -m``
|
||||
instead of an entry point script, which makes this approach a
|
||||
bit overkill. On the other hand, creating the scripts is
|
||||
future proof and it makes the contents of the ``pyvenv/bin``
|
||||
directory more informative. Portability is also not an issue,
|
||||
because the Python Packaging Authority provides a package
|
||||
``distlib.scripts`` to perform this task.
|
||||
|
||||
The required versions of the packages are stored in a configuration file
|
||||
``pythondeps.toml``. The format is custom to QEMU, but it is documented
|
||||
at the top of the file itself and it should be easy to understand. The
|
||||
requirements should make it possible to use the version that is packaged
|
||||
by QEMU's supported distros.
|
||||
|
||||
When dependencies are downloaded, instead, ``configure`` uses a "known
|
||||
good" version that is also listed in ``pythondeps.toml``. In this
|
||||
scenario, ``pythondeps.toml`` behaves like the "lock file" used by
|
||||
``cargo``, ``poetry`` or other dependency management systems.
|
||||
|
||||
|
||||
Bundled Python packages
|
||||
-----------------------
|
||||
|
||||
Python packages that are **mandatory** dependencies to build QEMU,
|
||||
but are not available in all supported distros, are bundled with the
|
||||
QEMU sources. The only one is currently Meson (outdated in Ubuntu
|
||||
22.04 and openSUSE Leap).
|
||||
|
||||
In order to include a new or updated wheel, modify and rerun the
|
||||
``python/scripts/vendor.py`` script. The script embeds the
|
||||
sha256 hash of package sources and checks it. The pypi.org web site
|
||||
provides an easy way to retrieve the sha256 hash of the sources.
|
||||
|
||||
|
||||
Stage 2: Meson
|
||||
==============
|
||||
|
||||
The Meson build system describes the build and install process for:
|
||||
|
||||
1) executables, which include:
|
||||
|
||||
- Tools - ``qemu-img``, ``qemu-nbd``, ``qemu-ga`` (guest agent), etc
|
||||
|
||||
- System emulators - ``qemu-system-$ARCH``
|
||||
|
||||
- Userspace emulators - ``qemu-$ARCH``
|
||||
|
||||
- Unit tests
|
||||
|
||||
2) documentation
|
||||
|
||||
3) ROMs, whether provided as binary blobs in the QEMU distributions
|
||||
or cross compiled under the direction of the configure script
|
||||
|
||||
4) other data files, such as icons or desktop files
|
||||
|
||||
All executables are built by default, except for some ``contrib/``
|
||||
binaries that are known to fail to build on some platforms (for example
|
||||
32-bit or big-endian platforms). Tests are also built by default,
|
||||
though that might change in the future.
|
||||
|
||||
The source code is highly modularized, split across many files to
|
||||
facilitate building of all of these components with as little duplicated
|
||||
compilation as possible. Using the Meson "sourceset" functionality,
|
||||
``meson.build`` files group the source files in rules that are
|
||||
enabled according to the available system libraries and to various
|
||||
configuration symbols. Sourcesets belong to one of four groups:
|
||||
|
||||
Subsystem sourcesets:
|
||||
Various subsystems that are common to both tools and emulators have
|
||||
their own sourceset, for example ``block_ss`` for the block device subsystem,
|
||||
``chardev_ss`` for the character device subsystem, etc. These sourcesets
|
||||
are then turned into static libraries as follows::
|
||||
|
||||
libchardev = static_library('chardev', chardev_ss.sources(),
|
||||
build_by_default: false)
|
||||
|
||||
chardev = declare_dependency(objects: libchardev.extract_all_objects(recursive: false),
|
||||
dependencies: chardev_ss.dependencies())
|
||||
|
||||
Target-independent emulator sourcesets:
|
||||
Various general purpose helper code is compiled only once and
|
||||
the .o files are linked into all output binaries that need it.
|
||||
This includes error handling infrastructure, standard data structures,
|
||||
platform portability wrapper functions, etc.
|
||||
|
||||
Target-independent code lives in the ``common_ss``, ``system_ss`` and
|
||||
``user_ss`` sourcesets. ``common_ss`` is linked into all emulators,
|
||||
``system_ss`` only in system emulators, ``user_ss`` only in user-mode
|
||||
emulators.
|
||||
|
||||
Target-dependent emulator sourcesets:
|
||||
In the target-dependent set lives CPU emulation, some device emulation and
|
||||
much glue code. This sometimes also has to be compiled multiple times,
|
||||
once for each target being built. Target-dependent files are included
|
||||
in the ``specific_ss`` sourceset.
|
||||
|
||||
Each emulator also includes sources for files in the ``hw/`` and ``target/``
|
||||
subdirectories. The subdirectory used for each emulator comes
|
||||
from the target's definition of ``TARGET_BASE_ARCH`` or (if missing)
|
||||
``TARGET_ARCH``, as found in ``configs/targets/*.mak``.
|
||||
|
||||
Each subdirectory in ``hw/`` adds one sourceset to the ``hw_arch`` dictionary,
|
||||
for example::
|
||||
|
||||
arm_ss = ss.source_set()
|
||||
arm_ss.add(files('boot.c'), fdt)
|
||||
...
|
||||
hw_arch += {'arm': arm_ss}
|
||||
|
||||
The sourceset is only used for system emulators.
|
||||
|
||||
Each subdirectory in ``target/`` instead should add one sourceset to each
|
||||
of the ``target_arch`` and ``target_system_arch``, which are used respectively
|
||||
for all emulators and for system emulators only. For example::
|
||||
|
||||
arm_ss = ss.source_set()
|
||||
arm_system_ss = ss.source_set()
|
||||
...
|
||||
target_arch += {'arm': arm_ss}
|
||||
target_system_arch += {'arm': arm_system_ss}
|
||||
|
||||
Module sourcesets:
|
||||
There are two dictionaries for modules: ``modules`` is used for
|
||||
target-independent modules and ``target_modules`` is used for
|
||||
target-dependent modules. When modules are disabled the ``module``
|
||||
source sets are added to ``system_ss`` and the ``target_modules``
|
||||
source sets are added to ``specific_ss``.
|
||||
|
||||
Both dictionaries are nested. One dictionary is created per
|
||||
subdirectory, and these per-subdirectory dictionaries are added to
|
||||
the toplevel dictionaries. For example::
|
||||
|
||||
hw_display_modules = {}
|
||||
qxl_ss = ss.source_set()
|
||||
...
|
||||
hw_display_modules += { 'qxl': qxl_ss }
|
||||
modules += { 'hw-display': hw_display_modules }
|
||||
|
||||
Utility sourcesets:
|
||||
All binaries link with a static library ``libqemuutil.a``. This library
|
||||
is built from several sourcesets; most of them however host generated
|
||||
code, and the only two of general interest are ``util_ss`` and ``stub_ss``.
|
||||
|
||||
The separation between these two is purely for documentation purposes.
|
||||
``util_ss`` contains generic utility files. Even though this code is only
|
||||
linked in some binaries, sometimes it requires hooks only in some of
|
||||
these and depend on other functions that are not fully implemented by
|
||||
all QEMU binaries. ``stub_ss`` links dummy stubs that will only be linked
|
||||
into the binary if the real implementation is not present. In a way,
|
||||
the stubs can be thought of as a portable implementation of the weak
|
||||
symbols concept.
|
||||
|
||||
|
||||
The following files concur in the definition of which files are linked
|
||||
into each emulator:
|
||||
|
||||
``configs/devices/*.mak``
|
||||
The files under ``configs/devices/`` control the boards and devices
|
||||
that are built into each QEMU system emulation targets. They merely contain
|
||||
a list of config variable definitions such as::
|
||||
|
||||
include arm-softmmu.mak
|
||||
CONFIG_XLNX_ZYNQMP_ARM=y
|
||||
CONFIG_XLNX_VERSAL=y
|
||||
|
||||
``*/Kconfig``
|
||||
These files are processed together with ``configs/devices/*.mak`` and
|
||||
describe the dependencies between various features, subsystems and
|
||||
device models. They are described in :ref:`kconfig`
|
||||
|
||||
``configs/targets/*.mak``
|
||||
These files mostly define symbols that appear in the ``*-config-target.h``
|
||||
file for each emulator\ [#cfgtarget]_. However, the ``TARGET_ARCH``
|
||||
and ``TARGET_BASE_ARCH`` will also be used to select the ``hw/`` and
|
||||
``target/`` subdirectories that are compiled into each target.
|
||||
|
||||
.. [#cfgtarget] This header is included by ``qemu/osdep.h`` when
|
||||
compiling files from the target-specific sourcesets.
|
||||
|
||||
These files rarely need changing unless you are adding a completely
|
||||
new target, or enabling new devices or hardware for a particular
|
||||
system/userspace emulation target
|
||||
|
||||
|
||||
Adding checks
|
||||
-------------
|
||||
|
||||
Compiler checks can be as simple as the following::
|
||||
|
||||
config_host_data.set('HAVE_BTRFS_H', cc.has_header('linux/btrfs.h'))
|
||||
|
||||
A more complex task such as adding a new dependency usually
|
||||
comprises the following tasks:
|
||||
|
||||
- Add a Meson build option to meson_options.txt.
|
||||
|
||||
- Add code to perform the actual feature check.
|
||||
|
||||
- Add code to include the feature status in ``config-host.h``
|
||||
|
||||
- Add code to print out the feature status in the configure summary
|
||||
upon completion.
|
||||
|
||||
Taking the probe for SDL2_Image as an example, we have the following
|
||||
in ``meson_options.txt``::
|
||||
|
||||
option('sdl_image', type : 'feature', value : 'auto',
|
||||
description: 'SDL Image support for icons')
|
||||
|
||||
Unless the option was given a non-``auto`` value (on the configure
|
||||
command line), the detection code must be performed only if the
|
||||
dependency will be used::
|
||||
|
||||
sdl_image = not_found
|
||||
if not get_option('sdl_image').auto() or have_system
|
||||
sdl_image = dependency('SDL2_image', required: get_option('sdl_image'),
|
||||
method: 'pkg-config')
|
||||
endif
|
||||
|
||||
This avoids warnings on static builds of user-mode emulators, for example.
|
||||
Most of the libraries used by system-mode emulators are not available for
|
||||
static linking.
|
||||
|
||||
The other supporting code is generally simple::
|
||||
|
||||
# Create config-host.h (if applicable)
|
||||
config_host_data.set('CONFIG_SDL_IMAGE', sdl_image.found())
|
||||
|
||||
# Summary
|
||||
summary_info += {'SDL image support': sdl_image.found()}
|
||||
|
||||
For the configure script to parse the new option, the
|
||||
``scripts/meson-buildoptions.sh`` file must be up-to-date; ``make
|
||||
update-buildoptions`` (or just ``make``) will take care of updating it.
|
||||
|
||||
|
||||
Support scripts
|
||||
---------------
|
||||
|
||||
Meson has a special convention for invoking Python scripts: if their
|
||||
first line is ``#! /usr/bin/env python3`` and the file is *not* executable,
|
||||
find_program() arranges to invoke the script under the same Python
|
||||
interpreter that was used to invoke Meson. This is the most common
|
||||
and preferred way to invoke support scripts from Meson build files,
|
||||
because it automatically uses the value of configure's --python= option.
|
||||
|
||||
In case the script is not written in Python, use a ``#! /usr/bin/env ...``
|
||||
line and make the script executable.
|
||||
|
||||
Scripts written in Python, where it is desirable to make the script
|
||||
executable (for example for test scripts that developers may want to
|
||||
invoke from the command line, such as tests/qapi-schema/test-qapi.py),
|
||||
should be invoked through the ``python`` variable in meson.build. For
|
||||
example::
|
||||
|
||||
test('QAPI schema regression tests', python,
|
||||
args: files('test-qapi.py'),
|
||||
env: test_env, suite: ['qapi-schema', 'qapi-frontend'])
|
||||
|
||||
This is needed to obey the --python= option passed to the configure
|
||||
script, which may point to something other than the first python3
|
||||
binary on the path.
|
||||
|
||||
By the time Meson runs, Python dependencies are available in the virtual
|
||||
environment and should be invoked through the scripts that ``configure``
|
||||
places under ``pyvenv``. One way to do so is as follows, using Meson's
|
||||
``find_program`` function::
|
||||
|
||||
sphinx_build = find_program(
|
||||
fs.parent(python.full_path()) / 'sphinx-build',
|
||||
required: get_option('docs'))
|
||||
|
||||
|
||||
Stage 3: Make
|
||||
=============
|
||||
|
||||
The next step in building QEMU is to invoke make. GNU Make is required
|
||||
to build QEMU, and may be installed as ``gmake`` on some hosts.
|
||||
|
||||
The output of Meson is a ``build.ninja`` file, which is used with the
|
||||
Ninja build tool. However, QEMU's build comprises other components than
|
||||
just the emulators (namely firmware and the tests in ``tests/tcg``) which
|
||||
need different cross compilers. The QEMU Makefile wraps both Ninja and
|
||||
the smaller build systems for firmware and tests; it also takes care of
|
||||
running ``configure`` again when the script changes. Apart from invoking
|
||||
these sub-Makefiles, the resulting build is largely non-recursive.
|
||||
|
||||
Tests, whether defined in ``meson.build`` or not, are also ran by the
|
||||
Makefile with the traditional ``make check`` phony target, while benchmarks
|
||||
are run with ``make bench``. Meson test suites such as ``unit`` can be ran
|
||||
with ``make check-unit``, and ``make check-tcg`` builds and runs "non-Meson"
|
||||
tests for all targets.
|
||||
|
||||
If desired, it is also possible to use ``ninja`` and ``pyvenv/bin/meson test``,
|
||||
respectively to build emulators and run tests defined in meson.build.
|
||||
The main difference is that ``make`` needs the ``-jN`` flag in order to
|
||||
enable parallel builds or tests.
|
||||
|
||||
Useful make targets
|
||||
-------------------
|
||||
|
||||
``help``
|
||||
Print a help message for the most common build targets.
|
||||
|
||||
``print-VAR``
|
||||
Print the value of the variable VAR. Useful for debugging the build
|
||||
system.
|
||||
|
||||
|
||||
Important files for the build system
|
||||
====================================
|
||||
|
||||
Statically defined files
|
||||
------------------------
|
||||
|
||||
The following key files are statically defined in the source tree, with
|
||||
the rules needed to build QEMU. Their behaviour is influenced by a
|
||||
number of dynamically created files listed later.
|
||||
|
||||
``Makefile``
|
||||
The main entry point used when invoking make to build all the components
|
||||
of QEMU. The default 'all' target will naturally result in the build of
|
||||
every component.
|
||||
|
||||
``*/meson.build``
|
||||
The meson.build file in the root directory is the main entry point for the
|
||||
Meson build system, and it coordinates the configuration and build of all
|
||||
executables. Build rules for various subdirectories are included in
|
||||
other meson.build files spread throughout the QEMU source tree.
|
||||
|
||||
``python/scripts/mkvenv.py``
|
||||
A wrapper for the Python ``venv`` and ``distlib.scripts`` packages.
|
||||
It handles creating the virtual environment, creating scripts in
|
||||
``pyvenv/bin``, and calling ``pip`` to install dependencies.
|
||||
|
||||
``tests/Makefile.include``
|
||||
Rules for external test harnesses like the TCG tests.
|
||||
|
||||
``tests/docker/Makefile.include``
|
||||
Rules for Docker tests. Like ``tests/Makefile.include``, this file is
|
||||
included directly by the top level Makefile, anything defined in this
|
||||
file will influence the entire build system.
|
||||
|
||||
``tests/vm/Makefile.include``
|
||||
Rules for VM-based tests. Like ``tests/Makefile.include``, this file is
|
||||
included directly by the top level Makefile, anything defined in this
|
||||
file will influence the entire build system.
|
||||
|
||||
Dynamically created files
|
||||
-------------------------
|
||||
|
||||
The following files are generated at run-time in order to control the
|
||||
behaviour of the Makefiles. This avoids the need for QEMU makefiles to
|
||||
go through any pre-processing as seen with autotools, where configure
|
||||
generates ``Makefile`` from ``Makefile.in``.
|
||||
|
||||
Built by configure:
|
||||
|
||||
``run``
|
||||
Used to run commands / scripts from the git checkout. Sets ``$PATH``
|
||||
to point to locally built binaries, and activates the python venv
|
||||
before running the requested command. Pass the command to run as
|
||||
args, for example::
|
||||
|
||||
$ ./build/run ./script/qmp/qmp-shell-wrap qemu-system-x86_64
|
||||
|
||||
will use the ``python3`` binary and site-packages from the local
|
||||
venv to run ``qmp-shell-wrap`` and spawn the QEMU emulator from
|
||||
the build directory.
|
||||
|
||||
``config-host.mak``
|
||||
When configure has determined the characteristics of the build host it
|
||||
will write the paths to various tools to this file, for use in ``Makefile``
|
||||
and to a smaller extent ``meson.build``.
|
||||
|
||||
``config-host.mak`` is also used as a dependency checking mechanism. If make
|
||||
sees that the modification timestamp on configure is newer than that on
|
||||
``config-host.mak``, then configure will be re-run.
|
||||
|
||||
``config-meson.cross``
|
||||
|
||||
A Meson "cross file" (or native file) used to communicate the paths to
|
||||
the toolchain and other configuration options.
|
||||
|
||||
``config.status``
|
||||
|
||||
A small shell script that will invoke configure again with the same
|
||||
environment variables that were set during the first run. It's used to
|
||||
rerun configure after changes to the source code, but it can also be
|
||||
inspected manually to check the contents of the environment.
|
||||
|
||||
``Makefile.prereqs``
|
||||
|
||||
A set of Makefile dependencies that order the build and execution of
|
||||
firmware and tests after the container images and emulators that they
|
||||
need.
|
||||
|
||||
``pc-bios/*/config.mak``, ``tests/tcg/config-host.mak``, ``tests/tcg/*/config-target.mak``
|
||||
|
||||
Configuration variables used to build the firmware and TCG tests,
|
||||
including paths to cross compilation toolchains.
|
||||
|
||||
``pyvenv``
|
||||
|
||||
A Python virtual environment that is used for all Python code running
|
||||
during the build. Using a virtual environment ensures that even code
|
||||
that is run via ``sphinx-build``, ``meson`` etc. uses the same interpreter
|
||||
and packages.
|
||||
|
||||
Built by Meson:
|
||||
|
||||
``config-host.h``
|
||||
Used by C code to determine the properties of the build environment
|
||||
and the set of enabled features for the entire build.
|
||||
|
||||
``${TARGET-NAME}-config-devices.mak``
|
||||
TARGET-NAME is the name of a system emulator. The file is
|
||||
generated by Meson using files under ``configs/devices`` as input.
|
||||
|
||||
``${TARGET-NAME}-config-target.mak``
|
||||
TARGET-NAME is the name of a system or usermode emulator. The file is
|
||||
generated by Meson using files under ``configs/targets`` as input.
|
||||
|
||||
``$TARGET_NAME-config-target.h``, ``$TARGET_NAME-config-devices.h``
|
||||
Used by C code to determine the properties and enabled
|
||||
features for each target. enabled. They are generated from
|
||||
the contents of the corresponding ``*.mak`` files using Meson's
|
||||
``configure_file()`` function; each target can include them using
|
||||
the ``CONFIG_TARGET`` and ``CONFIG_DEVICES`` macro respectively.
|
||||
|
||||
``build.ninja``
|
||||
The build rules.
|
||||
|
||||
|
||||
Built by Makefile:
|
||||
|
||||
``Makefile.ninja``
|
||||
A Makefile include that bridges to ninja for the actual build. The
|
||||
Makefile is mostly a list of targets that Meson included in build.ninja.
|
||||
|
||||
``Makefile.mtest``
|
||||
The Makefile definitions that let "make check" run tests defined in
|
||||
meson.build. The rules are produced from Meson's JSON description of
|
||||
tests (obtained with "meson introspect --tests") through the script
|
||||
scripts/mtest2make.py.
|
||||
@@ -0,0 +1,538 @@
|
||||
Modelling a clock tree in QEMU
|
||||
==============================
|
||||
|
||||
What are clocks?
|
||||
----------------
|
||||
|
||||
Clocks are QOM objects developed for the purpose of modelling the
|
||||
distribution of clocks in QEMU.
|
||||
|
||||
They allow us to model the clock distribution of a platform and detect
|
||||
configuration errors in the clock tree such as badly configured PLL, clock
|
||||
source selection or disabled clock.
|
||||
|
||||
The object is *Clock* and its QOM name is ``clock`` (in C code, the macro
|
||||
``TYPE_CLOCK``).
|
||||
|
||||
Clocks are typically used with devices where they are used to model inputs
|
||||
and outputs. They are created in a similar way to GPIOs. Inputs and outputs
|
||||
of different devices can be connected together.
|
||||
|
||||
In these cases a Clock object is a child of a Device object, but this
|
||||
is not a requirement. Clocks can be independent of devices. For
|
||||
example it is possible to create a clock outside of any device to
|
||||
model the main clock source of a machine.
|
||||
|
||||
Here is an example of clocks::
|
||||
|
||||
+---------+ +----------------------+ +--------------+
|
||||
| Clock 1 | | Device B | | Device C |
|
||||
| | | +-------+ +-------+ | | +-------+ |
|
||||
| |>>-+-->>|Clock 2| |Clock 3|>>--->>|Clock 6| |
|
||||
+---------+ | | | (in) | | (out) | | | | (in) | |
|
||||
| | +-------+ +-------+ | | +-------+ |
|
||||
| | +-------+ | +--------------+
|
||||
| | |Clock 4|>>
|
||||
| | | (out) | | +--------------+
|
||||
| | +-------+ | | Device D |
|
||||
| | +-------+ | | +-------+ |
|
||||
| | |Clock 5|>>--->>|Clock 7| |
|
||||
| | | (out) | | | | (in) | |
|
||||
| | +-------+ | | +-------+ |
|
||||
| +----------------------+ | |
|
||||
| | +-------+ |
|
||||
+----------------------------->>|Clock 8| |
|
||||
| | (in) | |
|
||||
| +-------+ |
|
||||
+--------------+
|
||||
|
||||
Clocks are defined in the ``include/hw/core/clock.h`` header and device
|
||||
related functions are defined in the ``include/hw/core/qdev-clock.h``
|
||||
header.
|
||||
|
||||
The clock state
|
||||
---------------
|
||||
|
||||
The state of a clock is its period; it is stored as an integer
|
||||
representing it in units of 2 :sup:`-32` ns. The special value of 0 is used to
|
||||
represent the clock being inactive or gated. The clocks do not model
|
||||
the signal itself (pin toggling) or other properties such as the duty
|
||||
cycle.
|
||||
|
||||
All clocks contain this state: outputs as well as inputs. This allows
|
||||
the current period of a clock to be fetched at any time. When a clock
|
||||
is updated, the value is immediately propagated to all connected
|
||||
clocks in the tree.
|
||||
|
||||
To ease interaction with clocks, helpers with a unit suffix are defined for
|
||||
every clock state setter or getter. The suffixes are:
|
||||
|
||||
- ``_ns`` for handling periods in nanoseconds
|
||||
- ``_hz`` for handling frequencies in hertz
|
||||
|
||||
The 0 period value is converted to 0 in hertz and vice versa. 0 always means
|
||||
that the clock is disabled.
|
||||
|
||||
Adding a new clock
|
||||
------------------
|
||||
|
||||
Adding clocks to a device must be done during the init method of the Device
|
||||
instance.
|
||||
|
||||
To add an input clock to a device, the function ``qdev_init_clock_in()``
|
||||
must be used. It takes the name, a callback, an opaque parameter
|
||||
for the callback and a mask of events when the callback should be
|
||||
called (this will be explained in a following section).
|
||||
Output is simpler; only the name is required. Typically::
|
||||
|
||||
qdev_init_clock_in(DEVICE(dev), "clk_in", clk_in_callback, dev, ClockUpdate);
|
||||
qdev_init_clock_out(DEVICE(dev), "clk_out");
|
||||
|
||||
Both functions return the created Clock pointer, which should be saved in the
|
||||
device's state structure for further use.
|
||||
|
||||
These objects will be automatically deleted by the QOM reference mechanism.
|
||||
|
||||
Note that it is possible to create a static array describing clock inputs and
|
||||
outputs. The function ``qdev_init_clocks()`` must be called with the array as
|
||||
parameter to initialize the clocks: it has the same behaviour as calling the
|
||||
``qdev_init_clock_in/out()`` for each clock in the array. To ease the array
|
||||
construction, some macros are defined in ``include/hw/core/qdev-clock.h``.
|
||||
As an example, the following creates 2 clocks to a device: one input and one
|
||||
output.
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
/* device structure containing pointers to the clock objects */
|
||||
typedef struct MyDeviceState {
|
||||
DeviceState parent_obj;
|
||||
Clock *clk_in;
|
||||
Clock *clk_out;
|
||||
} MyDeviceState;
|
||||
|
||||
/*
|
||||
* callback for the input clock (see "Callback on input clock
|
||||
* change" section below for more information).
|
||||
*/
|
||||
static void clk_in_callback(void *opaque, ClockEvent event);
|
||||
|
||||
/*
|
||||
* static array describing clocks:
|
||||
* + a clock input named "clk_in", whose pointer is stored in
|
||||
* the clk_in field of a MyDeviceState structure with callback
|
||||
* clk_in_callback.
|
||||
* + a clock output named "clk_out" whose pointer is stored in
|
||||
* the clk_out field of a MyDeviceState structure.
|
||||
*/
|
||||
static const ClockPortInitArray mydev_clocks = {
|
||||
QDEV_CLOCK_IN(MyDeviceState, clk_in, clk_in_callback, ClockUpdate),
|
||||
QDEV_CLOCK_OUT(MyDeviceState, clk_out),
|
||||
QDEV_CLOCK_END
|
||||
};
|
||||
|
||||
/* device initialization function */
|
||||
static void mydev_init(Object *obj)
|
||||
{
|
||||
/* cast to MyDeviceState */
|
||||
MyDeviceState *mydev = MYDEVICE(obj);
|
||||
/* create and fill the pointer fields in the MyDeviceState */
|
||||
qdev_init_clocks(mydev, mydev_clocks);
|
||||
[...]
|
||||
}
|
||||
|
||||
An alternative way to create a clock is to simply call
|
||||
``object_new(TYPE_CLOCK)``. In that case the clock will neither be an
|
||||
input nor an output of a device. After the whole QOM hierarchy of the
|
||||
clock has been set ``clock_setup_canonical_path()`` should be called.
|
||||
|
||||
At creation, the period of the clock is 0: the clock is disabled. You can
|
||||
change it using ``clock_set_ns()`` or ``clock_set_hz()``.
|
||||
|
||||
Note that if you are creating a clock with a fixed period which will never
|
||||
change (for example the main clock source of a board), then you'll have
|
||||
nothing else to do. This value will be propagated to other clocks when
|
||||
connecting the clocks together and devices will fetch the right value during
|
||||
the first reset.
|
||||
|
||||
Clock callbacks
|
||||
---------------
|
||||
|
||||
You can give a clock a callback function in several ways:
|
||||
|
||||
* by passing it as an argument to ``qdev_init_clock_in()``
|
||||
* as an argument to the ``QDEV_CLOCK_IN()`` macro initializing an
|
||||
array to be passed to ``qdev_init_clocks()``
|
||||
* by directly calling the ``clock_set_callback()`` function
|
||||
|
||||
The callback function must be of this type:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
typedef void ClockCallback(void *opaque, ClockEvent event);
|
||||
|
||||
The ``opaque`` argument is the pointer passed to ``qdev_init_clock_in()``
|
||||
or ``clock_set_callback()``; for ``qdev_init_clocks()`` it is the
|
||||
``dev`` device pointer.
|
||||
|
||||
The ``event`` argument specifies why the callback has been called.
|
||||
When you register the callback you specify a mask of ClockEvent values
|
||||
that you are interested in. The callback will only be called for those
|
||||
events.
|
||||
|
||||
The events currently supported are:
|
||||
|
||||
* ``ClockPreUpdate`` : called when the input clock's period is about to
|
||||
update. This is useful if the device needs to do some action for
|
||||
which it needs to know the old value of the clock period. During
|
||||
this callback, Clock API functions like ``clock_get()`` or
|
||||
``clock_ticks_to_ns()`` will use the old period.
|
||||
* ``ClockUpdate`` : called after the input clock's period has changed.
|
||||
During this callback, Clock API functions like ``clock_ticks_to_ns()``
|
||||
will use the new period.
|
||||
|
||||
Note that a clock only has one callback: it is not possible to register
|
||||
different functions for different events. You must register a single
|
||||
callback which listens for all of the events you are interested in,
|
||||
and use the ``event`` argument to identify which event has happened.
|
||||
|
||||
Retrieving clocks from a device
|
||||
-------------------------------
|
||||
|
||||
``qdev_get_clock_in()`` and ``dev_get_clock_out()`` are available to
|
||||
get the clock inputs or outputs of a device. For example:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
Clock *clk = qdev_get_clock_in(DEVICE(mydev), "clk_in");
|
||||
|
||||
or:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
Clock *clk = qdev_get_clock_out(DEVICE(mydev), "clk_out");
|
||||
|
||||
Connecting two clocks together
|
||||
------------------------------
|
||||
|
||||
To connect two clocks together, use the ``clock_set_source()`` function.
|
||||
Given two clocks ``clk1``, and ``clk2``, ``clock_set_source(clk2, clk1);``
|
||||
configures ``clk2`` to follow the ``clk1`` period changes. Every time ``clk1``
|
||||
is updated, ``clk2`` will be updated too.
|
||||
|
||||
When connecting clock between devices, prefer using the
|
||||
``qdev_connect_clock_in()`` function to set the source of an input
|
||||
device clock. For example, to connect the input clock ``clk2`` of
|
||||
``devB`` to the output clock ``clk1`` of ``devA``, do:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
qdev_connect_clock_in(devB, "clk2", qdev_get_clock_out(devA, "clk1"))
|
||||
|
||||
We used ``qdev_get_clock_out()`` above, but any clock can drive an
|
||||
input clock, even another input clock. The following diagram shows
|
||||
some examples of connections. Note also that a clock can drive several
|
||||
other clocks.
|
||||
|
||||
::
|
||||
|
||||
+------------+ +--------------------------------------------------+
|
||||
| Device A | | Device B |
|
||||
| | | +---------------------+ |
|
||||
| | | | Device C | |
|
||||
| +-------+ | | +-------+ | +-------+ +-------+ | +-------+ |
|
||||
| |Clock 1|>>-->>|Clock 2|>>+-->>|Clock 3| |Clock 5|>>>>|Clock 6|>>
|
||||
| | (out) | | | | (in) | | | | (in) | | (out) | | | (out) | |
|
||||
| +-------+ | | +-------+ | | +-------+ +-------+ | +-------+ |
|
||||
+------------+ | | +---------------------+ |
|
||||
| | |
|
||||
| | +--------------+ |
|
||||
| | | Device D | |
|
||||
| | | +-------+ | |
|
||||
| +-->>|Clock 4| | |
|
||||
| | | (in) | | |
|
||||
| | +-------+ | |
|
||||
| +--------------+ |
|
||||
+--------------------------------------------------+
|
||||
|
||||
In the above example, when *Clock 1* is updated by *Device A*, three
|
||||
clocks get the new clock period value: *Clock 2*, *Clock 3* and *Clock 4*.
|
||||
|
||||
It is not possible to disconnect a clock or to change the clock connection
|
||||
after it is connected.
|
||||
|
||||
Clock multiplier and divider settings
|
||||
-------------------------------------
|
||||
|
||||
By default, when clocks are connected together, the child
|
||||
clocks run with the same period as their source (parent) clock.
|
||||
The Clock API supports a built-in period multiplier/divider
|
||||
mechanism so you can configure a clock to make its children
|
||||
run at a different period from its own. If you call the
|
||||
``clock_set_mul_div()`` function you can specify the clock's
|
||||
multiplier and divider values. The children of that clock
|
||||
will all run with a period of ``parent_period * multiplier / divider``.
|
||||
For instance, if the clock has a frequency of 8MHz and you set its
|
||||
multiplier to 2 and its divider to 3, the child clocks will run
|
||||
at 12MHz.
|
||||
|
||||
You can change the multiplier and divider of a clock at runtime,
|
||||
so you can use this to model clock controller devices which
|
||||
have guest-programmable frequency multipliers or dividers.
|
||||
|
||||
Similarly to ``clock_set()``, ``clock_set_mul_div()`` returns ``true`` if
|
||||
the clock state was modified; that is, if the multiplier or the diviser
|
||||
or both were changed by the call.
|
||||
|
||||
Note that ``clock_set_mul_div()`` does not automatically call
|
||||
``clock_propagate()``. If you make a runtime change to the
|
||||
multiplier or divider you must call clock_propagate() yourself.
|
||||
|
||||
Unconnected input clocks
|
||||
------------------------
|
||||
|
||||
A newly created input clock is disabled (period of 0). This means the
|
||||
clock will be considered as disabled until the period is updated. If
|
||||
the clock remains unconnected it will always keep its initial value
|
||||
of 0. If this is not the desired behaviour, ``clock_set()``,
|
||||
``clock_set_ns()`` or ``clock_set_hz()`` should be called on the Clock
|
||||
object during device instance init. For example:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
clk = qdev_init_clock_in(DEVICE(dev), "clk-in", clk_in_callback,
|
||||
dev, ClockUpdate);
|
||||
/* set initial value to 10ns / 100MHz */
|
||||
clock_set_ns(clk, 10);
|
||||
|
||||
To enforce that the clock is wired up by the board code, you can
|
||||
call ``clock_has_source()`` in your device's realize method:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
if (!clock_has_source(s->clk)) {
|
||||
error_setg(errp, "MyDevice: clk input must be connected");
|
||||
return;
|
||||
}
|
||||
|
||||
Note that this only checks that the clock has been wired up; it is
|
||||
still possible that the output clock connected to it is disabled
|
||||
or has not yet been configured, in which case the period will be
|
||||
zero. You should use the clock callback to find out when the clock
|
||||
period changes.
|
||||
|
||||
Fetching clock frequency/period
|
||||
-------------------------------
|
||||
|
||||
To get the current state of a clock, use the functions ``clock_get()``
|
||||
or ``clock_get_hz()``.
|
||||
|
||||
``clock_get()`` returns the period of the clock in its fully precise
|
||||
internal representation, as an unsigned 64-bit integer in units of
|
||||
2^-32 nanoseconds. (For many purposes ``clock_ticks_to_ns()`` will
|
||||
be more convenient; see the section below on expiry deadlines.)
|
||||
|
||||
``clock_get_hz()`` returns the frequency of the clock, rounded to the
|
||||
next lowest integer. This implies some inaccuracy due to the rounding,
|
||||
so be cautious about using it in calculations.
|
||||
|
||||
It is also possible to register a callback on clock frequency changes.
|
||||
Here is an example, which assumes that ``clock_callback`` has been
|
||||
specified as the callback for the ``ClockUpdate`` event:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
void clock_callback(void *opaque, ClockEvent event) {
|
||||
MyDeviceState *s = (MyDeviceState *) opaque;
|
||||
/*
|
||||
* 'opaque' is the argument passed to qdev_init_clock_in();
|
||||
* usually this will be the device state pointer.
|
||||
*/
|
||||
|
||||
/* do something with the new period */
|
||||
fprintf(stdout, "device new period is %" PRIu64 "* 2^-32 ns\n",
|
||||
clock_get(dev->my_clk_input));
|
||||
}
|
||||
|
||||
If you are only interested in the frequency for displaying it to
|
||||
humans (for instance in debugging), use ``clock_display_freq()``,
|
||||
which returns a prettified string-representation, e.g. "33.3 MHz".
|
||||
The caller must free the string with g_free() after use.
|
||||
|
||||
It's also possible to retrieve the clock period from a QTest by
|
||||
accessing QOM property ``qtest-clock-period`` using a QMP command.
|
||||
This property is only present when the device is being run under
|
||||
the ``qtest`` accelerator; it is not available when QEMU is
|
||||
being run normally.
|
||||
|
||||
Calculating expiry deadlines
|
||||
----------------------------
|
||||
|
||||
A commonly required operation for a clock is to calculate how long
|
||||
it will take for the clock to tick N times; this can then be used
|
||||
to set a timer expiry deadline. Use the function ``clock_ticks_to_ns()``,
|
||||
which takes an unsigned 64-bit count of ticks and returns the length
|
||||
of time in nanoseconds required for the clock to tick that many times.
|
||||
|
||||
It is important not to try to calculate expiry deadlines using a
|
||||
shortcut like multiplying a "period of clock in nanoseconds" value
|
||||
by the tick count, because clocks can have periods which are not a
|
||||
whole number of nanoseconds, and the accumulated error in the
|
||||
multiplication can be significant.
|
||||
|
||||
For a clock with a very long period and a large number of ticks,
|
||||
the result of this function could in theory be too large to fit in
|
||||
a 64-bit value. To avoid overflow in this case, ``clock_ticks_to_ns()``
|
||||
saturates the result to INT64_MAX (because this is the largest valid
|
||||
input to the QEMUTimer APIs). Since INT64_MAX nanoseconds is almost
|
||||
300 years, anything with an expiry later than that is in the "will
|
||||
never happen" category. Callers of ``clock_ticks_to_ns()`` should
|
||||
therefore generally not special-case the possibility of a saturated
|
||||
result but just allow the timer to be set to that far-future value.
|
||||
(If you are performing further calculations on the returned value
|
||||
rather than simply passing it to a QEMUTimer function like
|
||||
``timer_mod_ns()`` then you should be careful to avoid overflow
|
||||
in those calculations, of course.)
|
||||
|
||||
Obtaining tick counts
|
||||
---------------------
|
||||
|
||||
For calculations where you need to know the number of ticks in
|
||||
a given duration, use ``clock_ns_to_ticks()``. This function handles
|
||||
possible non-whole-number-of-nanoseconds periods and avoids
|
||||
potential rounding errors. It will return '0' if the clock is stopped
|
||||
(i.e. it has period zero). If the inputs imply a tick count that
|
||||
overflows a 64-bit value (a very long duration for a clock with a
|
||||
very short period) the output value is truncated, so effectively
|
||||
the 64-bit output wraps around.
|
||||
|
||||
Changing a clock period
|
||||
-----------------------
|
||||
|
||||
A device can change its outputs using the ``clock_update()``,
|
||||
``clock_update_ns()`` or ``clock_update_hz()`` function. It will trigger
|
||||
updates on every connected input.
|
||||
|
||||
For example, let's say that we have an output clock *clkout* and we
|
||||
have a pointer to it in the device state because we did the following
|
||||
in init phase:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
dev->clkout = qdev_init_clock_out(DEVICE(dev), "clkout");
|
||||
|
||||
Then at any time (apart from the cases listed below), it is possible to
|
||||
change the clock value by doing:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
clock_update_hz(dev->clkout, 1000 * 1000 * 1000); /* 1GHz */
|
||||
|
||||
Because updating a clock may trigger any side effects through
|
||||
connected clocks and their callbacks, this operation must be done
|
||||
while holding the qemu io lock.
|
||||
|
||||
For the same reason, one can update clocks only when it is allowed to have
|
||||
side effects on other objects. In consequence, it is forbidden:
|
||||
|
||||
* during migration,
|
||||
* and in the enter phase of reset.
|
||||
|
||||
Note that calling ``clock_update[_ns|_hz]()`` is equivalent to calling
|
||||
``clock_set[_ns|_hz]()`` (with the same arguments) then
|
||||
``clock_propagate()`` on the clock. Thus, setting the clock value can
|
||||
be separated from triggering the side-effects. This is often required
|
||||
to factorize code to handle reset and migration in devices.
|
||||
|
||||
Aliasing clocks
|
||||
---------------
|
||||
|
||||
Sometimes, one needs to forward, or inherit, a clock from another
|
||||
device. Typically, when doing device composition, a device might
|
||||
expose a sub-device's clock without interfering with it. The function
|
||||
``qdev_alias_clock()`` can be used to achieve this behaviour. Note
|
||||
that it is possible to expose the clock under a different name.
|
||||
``qdev_alias_clock()`` works for both input and output clocks.
|
||||
|
||||
For example, if device B is a child of device A,
|
||||
``device_a_instance_init()`` may do something like this:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
void device_a_instance_init(Object *obj)
|
||||
{
|
||||
AState *A = DEVICE_A(obj);
|
||||
BState *B;
|
||||
/* create object B as child of A */
|
||||
[...]
|
||||
qdev_alias_clock(B, "clk", A, "b_clk");
|
||||
/*
|
||||
* Now A has a clock "b_clk" which is an alias to
|
||||
* the clock "clk" of its child B.
|
||||
*/
|
||||
}
|
||||
|
||||
This function does not return any clock object. The new clock has the
|
||||
same direction (input or output) as the original one. This function
|
||||
only adds a link to the existing clock. In the above example, object B
|
||||
remains the only object allowed to use the clock and device A must not
|
||||
try to change the clock period or set a callback to the clock. This
|
||||
diagram describes the example with an input clock::
|
||||
|
||||
+--------------------------+
|
||||
| Device A |
|
||||
| +--------------+ |
|
||||
| | Device B | |
|
||||
| | +-------+ | |
|
||||
>>"b_clk">>>| "clk" | | |
|
||||
| (in) | | (in) | | |
|
||||
| | +-------+ | |
|
||||
| +--------------+ |
|
||||
+--------------------------+
|
||||
|
||||
Migration
|
||||
---------
|
||||
|
||||
Clock state is not migrated automatically. Every device must handle its
|
||||
clock migration. Alias clocks must not be migrated.
|
||||
|
||||
To ensure clock states are restored correctly during migration, there
|
||||
are two solutions.
|
||||
|
||||
Clock states can be migrated by adding an entry into the device
|
||||
vmstate description. You should use the ``VMSTATE_CLOCK`` macro for this.
|
||||
This is typically used to migrate an input clock state. For example:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
MyDeviceState {
|
||||
DeviceState parent_obj;
|
||||
[...] /* some fields */
|
||||
Clock *clk;
|
||||
};
|
||||
|
||||
VMStateDescription my_device_vmstate = {
|
||||
.name = "my_device",
|
||||
.fields = (const VMStateField[]) {
|
||||
[...], /* other migrated fields */
|
||||
VMSTATE_CLOCK(clk, MyDeviceState),
|
||||
VMSTATE_END_OF_LIST()
|
||||
}
|
||||
};
|
||||
|
||||
The second solution is to restore the clock state using information already
|
||||
at our disposal. This can be used to restore output clock states using the
|
||||
device state. The functions ``clock_set[_ns|_hz]()`` can be used during the
|
||||
``post_load()`` migration callback.
|
||||
|
||||
When adding clock support to an existing device, if you care about
|
||||
migration compatibility you will need to be careful, as simply adding
|
||||
a ``VMSTATE_CLOCK()`` line will break compatibility. Instead, you can
|
||||
put the ``VMSTATE_CLOCK()`` line into a vmstate subsection with a
|
||||
suitable ``needed`` function, and use ``clock_set()`` in a
|
||||
``pre_load()`` function to set the default value that will be used if
|
||||
the source virtual machine in the migration does not send the clock
|
||||
state.
|
||||
|
||||
Care should be taken not to use ``clock_update[_ns|_hz]()`` or
|
||||
``clock_propagate()`` during the whole migration procedure because it
|
||||
will trigger side effects to other devices in an unknown state.
|
||||
@@ -0,0 +1,62 @@
|
||||
.. _code_of_conduct:
|
||||
|
||||
Code of Conduct
|
||||
===============
|
||||
|
||||
The QEMU community is made up of a mixture of professionals and
|
||||
volunteers from all over the world. Diversity is one of our strengths,
|
||||
but it can also lead to communication issues and unhappiness.
|
||||
To that end, we have a few ground rules that we ask people to adhere to.
|
||||
|
||||
* Be welcoming. We are committed to making participation in this project
|
||||
a harassment-free experience for everyone, regardless of level of
|
||||
experience, gender, gender identity and expression, sexual orientation,
|
||||
disability, personal appearance, body size, race, ethnicity, age, religion,
|
||||
or nationality.
|
||||
|
||||
* Be respectful. Not all of us will agree all the time. Disagreements, both
|
||||
social and technical, happen all the time and the QEMU community is no
|
||||
exception. When we disagree, we try to understand why. It is important that
|
||||
we resolve disagreements and differing views constructively. Members of the
|
||||
QEMU community should be respectful when dealing with other contributors as
|
||||
well as with people outside the QEMU community and with users of QEMU.
|
||||
|
||||
Harassment and other exclusionary behavior are not acceptable. A community
|
||||
where people feel uncomfortable or threatened is neither welcoming nor
|
||||
respectful. Examples of unacceptable behavior by participants include:
|
||||
|
||||
* The use of sexualized language or imagery
|
||||
|
||||
* Personal attacks
|
||||
|
||||
* Trolling or insulting/derogatory comments
|
||||
|
||||
* Public or private harassment
|
||||
|
||||
* Publishing other's private information, such as physical or electronic
|
||||
addresses, without explicit permission
|
||||
|
||||
This isn't an exhaustive list of things that you can't do. Rather, take
|
||||
it in the spirit in which it's intended: a guide to make it easier to
|
||||
be excellent to each other.
|
||||
|
||||
This code of conduct applies to all spaces managed by the QEMU project.
|
||||
This includes IRC, the mailing lists, the issue tracker, community
|
||||
events, and any other forums created by the project team which the
|
||||
community uses for communication. This code of conduct also applies
|
||||
outside these spaces, when an individual acts as a representative or a
|
||||
member of the project or its community.
|
||||
|
||||
By adopting this code of conduct, project maintainers commit themselves
|
||||
to fairly and consistently applying these principles to every aspect of
|
||||
managing this project. If you believe someone is violating the code of
|
||||
conduct, please read the :ref:`conflict-resolution` document for
|
||||
information about how to proceed.
|
||||
|
||||
Sources
|
||||
-------
|
||||
|
||||
This document is based on the `Fedora Code of Conduct
|
||||
<http://web.archive.org/web/20210429132536/https://docs.fedoraproject.org/en-US/project/code-of-conduct/>`__
|
||||
(as of April 2021) and the `Contributor Covenant version 1.3.0
|
||||
<https://www.contributor-covenant.org/version/1/3/0/code-of-conduct/>`__.
|
||||
@@ -0,0 +1,349 @@
|
||||
.. _code-provenance:
|
||||
|
||||
Code provenance
|
||||
===============
|
||||
|
||||
Certifying patch submissions
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The QEMU community **mandates** all contributors to certify provenance of
|
||||
patch submissions they make to the project. To put it another way,
|
||||
contributors must indicate that they are legally permitted to contribute to
|
||||
the project.
|
||||
|
||||
Certification is achieved with a low overhead by adding a single line to the
|
||||
bottom of every git commit::
|
||||
|
||||
Signed-off-by: YOUR NAME <YOUR@EMAIL>
|
||||
|
||||
The addition of this line asserts that the author of the patch is contributing
|
||||
in accordance with the clauses specified in the
|
||||
`Developer's Certificate of Origin <https://developercertificate.org>`__:
|
||||
|
||||
.. _dco:
|
||||
|
||||
Developer's Certificate of Origin 1.1
|
||||
|
||||
By making a contribution to this project, I certify that:
|
||||
|
||||
(a) The contribution was created in whole or in part by me and I
|
||||
have the right to submit it under the open source license
|
||||
indicated in the file; or
|
||||
|
||||
(b) The contribution is based upon previous work that, to the best
|
||||
of my knowledge, is covered under an appropriate open source
|
||||
license and I have the right under that license to submit that
|
||||
work with modifications, whether created in whole or in part
|
||||
by me, under the same open source license (unless I am
|
||||
permitted to submit under a different license), as indicated
|
||||
in the file; or
|
||||
|
||||
(c) The contribution was provided directly to me by some other
|
||||
person who certified (a), (b) or (c) and I have not modified
|
||||
it.
|
||||
|
||||
(d) I understand and agree that this project and the contribution
|
||||
are public and that a record of the contribution (including all
|
||||
personal information I submit with it, including my sign-off) is
|
||||
maintained indefinitely and may be redistributed consistent with
|
||||
this project or the open source license(s) involved.
|
||||
|
||||
The name used with "Signed-off-by" does not need to be your legal name, nor
|
||||
birth name, nor appear on any government ID. It is the identity you choose to
|
||||
be known by in the community, but should not be anonymous, nor misrepresent
|
||||
whom you are.
|
||||
|
||||
It is generally expected that the name and email addresses used in one of the
|
||||
``Signed-off-by`` lines, matches that of the git commit ``Author`` field.
|
||||
It's okay if you subscribe or contribute to the list via more than one
|
||||
address, but using multiple addresses in one commit just confuses
|
||||
things.
|
||||
|
||||
If the person sending the mail is not one of the patch authors, they are
|
||||
nonetheless expected to add their own ``Signed-off-by`` to comply with the
|
||||
DCO clause (c).
|
||||
|
||||
Multiple authorship
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
It is not uncommon for a patch to have contributions from multiple authors. In
|
||||
this scenario, git commits will usually be expected to have a ``Signed-off-by``
|
||||
line for each contributor involved in creation of the patch. Some edge cases:
|
||||
|
||||
* The non-primary author's contributions were so trivial that they can be
|
||||
considered not subject to copyright. In this case the secondary authors
|
||||
need not include a ``Signed-off-by``.
|
||||
|
||||
This case most commonly applies where QEMU reviewers give short snippets
|
||||
of code as suggested fixes to a patch. The reviewers don't need to have
|
||||
their own ``Signed-off-by`` added unless their code suggestion was
|
||||
unusually large, but it is common to add ``Suggested-by`` as a credit
|
||||
for non-trivial code.
|
||||
|
||||
* Both contributors work for the same employer and the employer requires
|
||||
copyright assignment.
|
||||
|
||||
It can be said that in this case a ``Signed-off-by`` is indicating that
|
||||
the person has permission to contribute from their employer who is the
|
||||
copyright holder. It is nonetheless still preferable to include a
|
||||
``Signed-off-by`` for each contributor, as in some countries employees are
|
||||
not able to assign copyright to their employer, and it also covers any
|
||||
time invested outside working hours.
|
||||
|
||||
When multiple ``Signed-off-by`` tags are present, they should be strictly kept
|
||||
in order of authorship, from oldest to newest.
|
||||
|
||||
Other commit tags
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
While the ``Signed-off-by`` tag is mandatory, there are a number of other tags
|
||||
that are commonly used during QEMU development:
|
||||
|
||||
* **Reviewed-by**: when a QEMU community member reviews a patch on the
|
||||
mailing list, if they consider the patch acceptable, they should send an
|
||||
email reply containing a ``Reviewed-by`` tag. Subsystem maintainers who
|
||||
review a patch should add this even if they are also adding their
|
||||
``Signed-off-by`` to the same commit.
|
||||
|
||||
* **Acked-by**: when a QEMU subsystem maintainer approves a patch that
|
||||
touches their subsystem, but intends to allow a different maintainer to
|
||||
queue it and send a pull request, they would send a mail containing a
|
||||
``Acked-by`` tag. Where a patch touches multiple subsystems, ``Acked-by``
|
||||
only implies review of the maintainers' own areas of responsibility. If a
|
||||
maintainer wants to indicate they have done a full review they should use
|
||||
a ``Reviewed-by`` tag.
|
||||
|
||||
* **Tested-by**: when a QEMU community member has functionally tested the
|
||||
behaviour of the patch in some manner, they should send an email reply
|
||||
containing a ``Tested-by`` tag.
|
||||
|
||||
* **Reported-by**: when a QEMU community member reports a problem via the
|
||||
mailing list, or some other informal channel that is not the issue tracker,
|
||||
it is good practice to credit them by including a ``Reported-by`` tag on
|
||||
any patch fixing the issue. When the problem is reported via the GitLab
|
||||
issue tracker, however, it is sufficient to just include a link to the
|
||||
issue.
|
||||
|
||||
* **Suggested-by**: when a reviewer or other 3rd party makes non-trivial
|
||||
suggestions for how to change a patch, it is good practice to credit them
|
||||
by including a ``Suggested-by`` tag.
|
||||
|
||||
Subsystem maintainer requirements
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
When a subsystem maintainer accepts a patch from a contributor, in addition to
|
||||
the normal code review points, they are expected to validate the presence of
|
||||
suitable ``Signed-off-by`` tags.
|
||||
|
||||
At the time they queue the patch in their subsystem tree, the maintainer
|
||||
**must** also then add their own ``Signed-off-by`` to indicate that they have
|
||||
done the aforementioned validation. This is in addition to any of their own
|
||||
``Reviewed-by`` tags the subsystem maintainer may wish to include.
|
||||
|
||||
When the maintainer modifies the patch after pulling into their tree, they
|
||||
should record their contribution. This is typically done via a note in the
|
||||
commit message, just prior to the maintainer's ``Signed-off-by``::
|
||||
|
||||
Signed-off-by: Cory Contributor <[email protected]>
|
||||
[Comment rephrased for clarity]
|
||||
Signed-off-by: Mary Maintainer <[email protected]>
|
||||
|
||||
|
||||
Tools for adding ``Signed-off-by``
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
There are a variety of ways tools can support adding ``Signed-off-by`` tags
|
||||
for patches, avoiding the need for contributors to manually type in this
|
||||
repetitive text each time.
|
||||
|
||||
git commands
|
||||
^^^^^^^^^^^^
|
||||
|
||||
When creating, or amending, a commit the ``-s`` flag to ``git commit`` will
|
||||
append a suitable line matching the configured git author details.
|
||||
|
||||
If preparing patches using the ``git format-patch`` tool, the ``-s`` flag can
|
||||
be used to append a suitable line in the emails it creates, without modifying
|
||||
the local commits. Alternatively to modify all the local commits on a branch::
|
||||
|
||||
git rebase master -x 'git commit --amend --no-edit -s'
|
||||
|
||||
emacs
|
||||
^^^^^
|
||||
|
||||
In the file ``$HOME/.emacs.d/abbrev_defs`` add:
|
||||
|
||||
.. code:: elisp
|
||||
|
||||
(define-abbrev-table 'global-abbrev-table
|
||||
'(
|
||||
("8rev" "Reviewed-by: YOUR NAME <[email protected]>" nil 1)
|
||||
("8ack" "Acked-by: YOUR NAME <[email protected]>" nil 1)
|
||||
("8test" "Tested-by: YOUR NAME <[email protected]>" nil 1)
|
||||
("8sob" "Signed-off-by: YOUR NAME <[email protected]>" nil 1)
|
||||
))
|
||||
|
||||
with this change, if you type (for example) ``8rev`` followed by ``<space>``
|
||||
or ``<enter>`` it will expand to the whole phrase.
|
||||
|
||||
vim
|
||||
^^^
|
||||
|
||||
In the file ``$HOME/.vimrc`` add::
|
||||
|
||||
iabbrev 8rev Reviewed-by: YOUR NAME <[email protected]>
|
||||
iabbrev 8ack Acked-by: YOUR NAME <[email protected]>
|
||||
iabbrev 8test Tested-by: YOUR NAME <[email protected]>
|
||||
iabbrev 8sob Signed-off-by: YOUR NAME <[email protected]>
|
||||
|
||||
with this change, if you type (for example) ``8rev`` followed by ``<space>``
|
||||
or ``<enter>`` it will expand to the whole phrase.
|
||||
|
||||
Re-starting abandoned work
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
For a variety of reasons there are some patches that get submitted to QEMU but
|
||||
never merged. An unrelated contributor may decide (months or years later) to
|
||||
continue working from the abandoned patch and re-submit it with extra changes.
|
||||
|
||||
The general principles when picking up abandoned work are:
|
||||
|
||||
* Continue to credit the original author for their work, by maintaining their
|
||||
original ``Signed-off-by``
|
||||
* Indicate where the original patch was obtained from (mailing list, bug
|
||||
tracker, author's git repo, etc) when sending it for review
|
||||
* Acknowledge the extra work of the new contributor by including their
|
||||
``Signed-off-by`` in the patch in addition to the original author's
|
||||
* Indicate who is responsible for what parts of the patch. This is typically
|
||||
done via a note in the commit message, just prior to the new contributor's
|
||||
``Signed-off-by``::
|
||||
|
||||
Signed-off-by: Some Person <[email protected]>
|
||||
[Rebased and added support for 'foo']
|
||||
Signed-off-by: New Person <[email protected]>
|
||||
|
||||
In complicated cases, or if otherwise unsure, ask for advice on the project
|
||||
mailing list.
|
||||
|
||||
It is also recommended to attempt to contact the original author to let them
|
||||
know you are interested in taking over their work, in case they still intended
|
||||
to return to the work, or had any suggestions about the best way to continue.
|
||||
|
||||
Inclusion of generated files
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Files in patches contributed to QEMU are generally expected to be provided
|
||||
only in the preferred format for making modifications. The implication of
|
||||
this is that the output of code generators or compilers is usually not
|
||||
appropriate to contribute to QEMU.
|
||||
|
||||
For reasons of practicality there are some exceptions to this rule, where
|
||||
generated code is permitted, provided it is also accompanied by the
|
||||
corresponding preferred source format. This is done where it is impractical
|
||||
to expect those building QEMU to run the code generation or compilation
|
||||
process. A non-exhaustive list of examples is:
|
||||
|
||||
* Images: where an bitmap image is created from a vector file it is common
|
||||
to include the rendered bitmaps at desired resolution(s), since subtle
|
||||
changes in the rasterization process / tools may affect quality. The
|
||||
original vector file is expected to accompany any generated bitmaps.
|
||||
|
||||
* Firmware: QEMU includes pre-compiled binary ROMs for a variety of guest
|
||||
firmwares. When such binary ROMs are contributed, the corresponding source
|
||||
must also be provided, either directly, or through a git submodule link.
|
||||
|
||||
* Dockerfiles: the majority of the dockerfiles are automatically generated
|
||||
from a canonical list of build dependencies maintained in tree, together
|
||||
with the libvirt-ci git submodule link. The generated dockerfiles are
|
||||
included in tree because it is desirable to be able to directly build
|
||||
container images from a clean git checkout.
|
||||
|
||||
* eBPF: QEMU includes some generated eBPF machine code, since the required
|
||||
eBPF compilation tools are not broadly available on all targeted OS
|
||||
distributions. The corresponding eBPF C code for the binary is also
|
||||
provided. This is a time-limited exception until the eBPF toolchain is
|
||||
sufficiently broadly available in distros.
|
||||
|
||||
In all cases above, the existence of generated files must be acknowledged
|
||||
and justified in the commit that introduces them.
|
||||
|
||||
Tools which perform changes to existing code with deterministic algorithmic
|
||||
manipulation, driven by user specified inputs, are not generally considered
|
||||
to be "generators".
|
||||
|
||||
For instance, using Coccinelle to convert code from one pattern to another
|
||||
pattern, or fixing documentation typos with a spell checker, or transforming
|
||||
code using sed / awk / etc, are not considered to be acts of code
|
||||
generation. Where an automated manipulation is performed on code, however,
|
||||
this should be declared in the commit message.
|
||||
|
||||
At times contributors may use or create scripts/tools to generate an initial
|
||||
boilerplate code template which is then filled in to produce the final patch.
|
||||
The output of such a tool would still be considered the "preferred format",
|
||||
since it is intended to be a foundation for further human authored changes.
|
||||
Such tools are acceptable to use, provided there is clearly defined copyright
|
||||
and licensing for their output. Note in particular the caveats applying to AI
|
||||
content generators below.
|
||||
|
||||
Use of AI-generated content
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
TL;DR:
|
||||
|
||||
**Current QEMU project policy is to DECLINE any contributions which are
|
||||
believed to include or derive from AI generated content. This includes
|
||||
ChatGPT, Claude, Copilot, Llama and similar tools.**
|
||||
|
||||
**This policy does not apply to other uses of AI, such as researching APIs
|
||||
or algorithms, static analysis, or debugging, provided their output is not
|
||||
included in contributions.**
|
||||
|
||||
The increasing prevalence of AI-assisted software development results in a
|
||||
number of difficult legal questions and risks for software projects, including
|
||||
QEMU. Of particular concern is content generated by `Large Language Models
|
||||
<https://en.wikipedia.org/wiki/Large_language_model>`__ (LLMs).
|
||||
|
||||
The QEMU community requires that contributors certify their patch submissions
|
||||
are made in accordance with the rules of the `Developer's Certificate of
|
||||
Origin (DCO) <dco>`.
|
||||
|
||||
To satisfy the DCO, the patch contributor has to fully understand the
|
||||
copyright and license status of content they are contributing to QEMU. With AI
|
||||
content generators, the copyright and license status of the output is
|
||||
ill-defined with no generally accepted, settled legal foundation.
|
||||
|
||||
Where the training material is known, it is common for it to include large
|
||||
volumes of material under restrictive licensing/copyright terms. Even where
|
||||
the training material is all known to be under open source licenses, it is
|
||||
likely to be under a variety of terms, not all of which will be compatible
|
||||
with QEMU's licensing requirements.
|
||||
|
||||
How contributors could comply with DCO terms (b) or (c) for the output of AI
|
||||
content generators commonly available today is unclear. The QEMU project is
|
||||
not willing or able to accept the legal risks of non-compliance.
|
||||
|
||||
The QEMU project thus requires that contributors refrain from using AI content
|
||||
generators on patches intended to be submitted to the project, and will
|
||||
decline any contribution if use of AI is either known or suspected.
|
||||
|
||||
Examples of tools impacted by this policy includes GitHub's CoPilot, OpenAI's
|
||||
ChatGPT, Anthropic's Claude, and Meta's Code Llama, and code/content
|
||||
generation agents which are built on top of such tools.
|
||||
|
||||
This policy may evolve as AI tools mature and the legal situation is
|
||||
clarified.
|
||||
|
||||
Exceptions
|
||||
^^^^^^^^^^
|
||||
|
||||
The QEMU project welcomes discussion on any exceptions to this policy,
|
||||
or more general revisions. This can be done by contacting the qemu-devel
|
||||
mailing list with details of a proposed tool, model, usage scenario, etc.
|
||||
that is beneficial to QEMU, while still mitigating issues around compliance
|
||||
with the DCO. After discussion, any exception will be listed below.
|
||||
|
||||
Exceptions do not remove the need for authors to comply with all other
|
||||
requirements for contribution. In particular, the "Signed-off-by"
|
||||
label in a patch submission is a statement that the author takes
|
||||
responsibility for the entire contents of the patch, including any parts
|
||||
that were generated or assisted by AI tools or other tools.
|
||||
@@ -0,0 +1,218 @@
|
||||
========
|
||||
Codebase
|
||||
========
|
||||
|
||||
This section presents the various parts of QEMU and how the codebase is
|
||||
organized.
|
||||
|
||||
Beyond giving succinct descriptions, the goal is to offer links to various
|
||||
parts of the documentation/codebase.
|
||||
|
||||
Those two videos are an excellent introduction to QEMU codebase:
|
||||
|
||||
- `QEMU Dev Starter guide - General overview <https://www.youtube.com/watch?v=OCBLTMKLGAk>`_
|
||||
- `QEMU Dev Starter guide - System mode <https://www.youtube.com/watch?v=jrZ56K3Sl_k>`_
|
||||
|
||||
Subsystems
|
||||
----------
|
||||
|
||||
An exhaustive list of subsystems and associated files can be found in the
|
||||
`MAINTAINERS <https://gitlab.com/qemu-project/qemu/-/blob/master/MAINTAINERS>`_
|
||||
file.
|
||||
|
||||
Some of the main QEMU subsystems are:
|
||||
|
||||
- `Accelerators<Accelerators>`
|
||||
- Block devices and `disk images<disk images>` support
|
||||
- `CI<ci>` and `Tests<testing>`
|
||||
- `Devices<device-emulation>` & Board models
|
||||
- `Documentation <documentation-root>`
|
||||
- `GDB support<GDB usage>`
|
||||
- :ref:`Migration<migration>`
|
||||
- `Monitor<QEMU monitor>`
|
||||
- :ref:`QOM (QEMU Object Model)<qom>`
|
||||
- `System mode<System emulation>`
|
||||
- :ref:`TCG (Tiny Code Generator)<tcg>`
|
||||
- `User mode<user-mode>` (`Linux<linux-user-mode>` & `BSD<bsd-user-mode>`)
|
||||
- User Interfaces
|
||||
|
||||
More documentation on QEMU subsystems can be found on :ref:`internal-subsystem`
|
||||
page.
|
||||
|
||||
The Grand tour
|
||||
--------------
|
||||
|
||||
We present briefly here what every folder in the top directory of the codebase
|
||||
contains. Hop on!
|
||||
|
||||
The folder name links here will take you to that folder in our gitlab
|
||||
repository. Other links will take you to more detailed documentation for that
|
||||
subsystem, where we have it. Unfortunately not every subsystem has documentation
|
||||
yet, so sometimes the source code is all you have.
|
||||
|
||||
* `accel <https://gitlab.com/qemu-project/qemu/-/tree/master/accel>`_:
|
||||
Infrastructure and architecture agnostic code related to the various
|
||||
`accelerators <Accelerators>` supported by QEMU
|
||||
(TCG, KVM, hvf, whpx, xen, nvmm, mshv).
|
||||
Contains interfaces for operations that will be implemented per
|
||||
`target <https://gitlab.com/qemu-project/qemu/-/tree/master/target>`_.
|
||||
* `audio <https://gitlab.com/qemu-project/qemu/-/tree/master/audio>`_:
|
||||
Audio (host) support.
|
||||
* `authz <https://gitlab.com/qemu-project/qemu/-/tree/master/authz>`_:
|
||||
`QEMU Authorization framework<client authorization>`.
|
||||
* `backends <https://gitlab.com/qemu-project/qemu/-/tree/master/backends>`_:
|
||||
Various backends that are used to access resources on the host (e.g. for
|
||||
random number generation, memory backing or cryptographic functions).
|
||||
* `block <https://gitlab.com/qemu-project/qemu/-/tree/master/block>`_:
|
||||
Block devices and `image formats<disk images>` implementation.
|
||||
* `bsd-user <https://gitlab.com/qemu-project/qemu/-/tree/master/bsd-user>`_:
|
||||
`BSD User mode<bsd-user-mode>`.
|
||||
* build: Where the code built goes by default. You can tell the QEMU build
|
||||
system to put the built code anywhere else you like.
|
||||
* `chardev <https://gitlab.com/qemu-project/qemu/-/tree/master/chardev>`_:
|
||||
Various backends used by char devices.
|
||||
* `common-user <https://gitlab.com/qemu-project/qemu/-/tree/master/common-user>`_:
|
||||
User-mode assembly code for dealing with signals occurring during syscalls.
|
||||
* `configs <https://gitlab.com/qemu-project/qemu/-/tree/master/configs>`_:
|
||||
Makefiles defining configurations to build QEMU.
|
||||
* `contrib <https://gitlab.com/qemu-project/qemu/-/tree/master/contrib>`_:
|
||||
Community contributed devices/plugins/tools.
|
||||
* `crypto <https://gitlab.com/qemu-project/qemu/-/tree/master/crypto>`_:
|
||||
Cryptographic algorithms used in QEMU.
|
||||
* `disas <https://gitlab.com/qemu-project/qemu/-/tree/master/disas>`_:
|
||||
Disassembly functions used by QEMU target code.
|
||||
* `docs <https://gitlab.com/qemu-project/qemu/-/tree/master/docs>`_:
|
||||
QEMU Documentation.
|
||||
* `dump <https://gitlab.com/qemu-project/qemu/-/tree/master/dump>`_:
|
||||
Code to dump memory of a running VM.
|
||||
* `ebpf <https://gitlab.com/qemu-project/qemu/-/tree/master/ebpf>`_:
|
||||
eBPF program support in QEMU. `virtio-net RSS<ebpf-rss>` uses it.
|
||||
* `fpu <https://gitlab.com/qemu-project/qemu/-/tree/master/fpu>`_:
|
||||
Floating-point software emulation.
|
||||
* `fsdev <https://gitlab.com/qemu-project/qemu/-/tree/master/fsdev>`_:
|
||||
`VirtFS <https://www.linux-kvm.org/page/VirtFS>`_ support.
|
||||
* `gdbstub <https://gitlab.com/qemu-project/qemu/-/tree/master/gdbstub>`_:
|
||||
`GDB <GDB usage>` support.
|
||||
* `host <https://gitlab.com/qemu-project/qemu/-/tree/master/host>`_:
|
||||
Various architecture specific header files (crypto, atomic, memory
|
||||
operations).
|
||||
* `linux-headers <https://gitlab.com/qemu-project/qemu/-/tree/master/linux-headers>`_:
|
||||
A subset of headers imported from Linux kernel and used for implementing
|
||||
KVM support and user-mode.
|
||||
* `linux-user <https://gitlab.com/qemu-project/qemu/-/tree/master/linux-user>`_:
|
||||
`User mode <user-mode>` implementation. Contains one folder per target
|
||||
architecture.
|
||||
* `.gitlab-ci.d <https://gitlab.com/qemu-project/qemu/-/tree/master/.gitlab-ci.d>`_:
|
||||
`CI <ci>` yaml and scripts.
|
||||
* `include <https://gitlab.com/qemu-project/qemu/-/tree/master/include>`_:
|
||||
All headers associated to different subsystems in QEMU. The hierarchy used
|
||||
mirrors source code organization and naming.
|
||||
* `hw <https://gitlab.com/qemu-project/qemu/-/tree/master/hw>`_:
|
||||
`Devices <device-emulation>` and boards emulation. Devices are categorized by
|
||||
type/protocol/architecture and located in associated subfolder.
|
||||
* `io <https://gitlab.com/qemu-project/qemu/-/tree/master/io>`_:
|
||||
QEMU `I/O channels <https://lists.gnu.org/archive/html/qemu-devel/2015-11/msg04208.html>`_.
|
||||
* `libdecnumber <https://gitlab.com/qemu-project/qemu/-/tree/master/libdecnumber>`_:
|
||||
Import of gcc library, used to implement decimal number arithmetic.
|
||||
* `migration <https://gitlab.com/qemu-project/qemu/-/tree/master/migration>`__:
|
||||
:ref:`Migration framework <migration>`.
|
||||
* `monitor <https://gitlab.com/qemu-project/qemu/-/tree/master/monitor>`_:
|
||||
`Monitor <QEMU monitor>` implementation (HMP & QMP).
|
||||
* `nbd <https://gitlab.com/qemu-project/qemu/-/tree/master/nbd>`_:
|
||||
QEMU NBD (Network Block Device) server.
|
||||
* `net <https://gitlab.com/qemu-project/qemu/-/tree/master/net>`_:
|
||||
Network (host) support.
|
||||
* `pc-bios <https://gitlab.com/qemu-project/qemu/-/tree/master/pc-bios>`_:
|
||||
Contains pre-built firmware binaries and boot images, ready to use in
|
||||
QEMU without compilation.
|
||||
* `plugins <https://gitlab.com/qemu-project/qemu/-/tree/master/plugins>`_:
|
||||
:ref:`TCG plugins <tcg-plugins>` core implementation. Plugins can be found in
|
||||
`tests <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/tcg/plugins>`__
|
||||
and `contrib <https://gitlab.com/qemu-project/qemu/-/tree/master/contrib/plugins>`__
|
||||
folders.
|
||||
* `po <https://gitlab.com/qemu-project/qemu/-/tree/master/po>`_:
|
||||
Translation files.
|
||||
* `python <https://gitlab.com/qemu-project/qemu/-/tree/master/python>`_:
|
||||
Python part of our build/test system.
|
||||
* `qapi <https://gitlab.com/qemu-project/qemu/-/tree/master/qapi>`_:
|
||||
`QAPI <qapi>` implementation.
|
||||
* `qobject <https://gitlab.com/qemu-project/qemu/-/tree/master/qobject>`_:
|
||||
QEMU Object implementation.
|
||||
* `qga <https://gitlab.com/qemu-project/qemu/-/tree/master/qga>`_:
|
||||
QEMU `Guest agent <qemu-ga>` implementation.
|
||||
* `qom <https://gitlab.com/qemu-project/qemu/-/tree/master/qom>`_:
|
||||
QEMU :ref:`Object model <qom>` implementation, with monitor associated commands.
|
||||
* `replay <https://gitlab.com/qemu-project/qemu/-/tree/master/replay>`_:
|
||||
QEMU :ref:`Record/replay <replay>` implementation.
|
||||
* `roms <https://gitlab.com/qemu-project/qemu/-/tree/master/roms>`_:
|
||||
Contains source code for various firmware and ROMs, which can be compiled if
|
||||
custom or updated versions are needed.
|
||||
* `rust <https://gitlab.com/qemu-project/qemu/-/tree/master/rust>`_:
|
||||
Rust integration in QEMU. It contains the new interfaces defined and
|
||||
associated devices using it.
|
||||
* `scripts <https://gitlab.com/qemu-project/qemu/-/tree/master/scripts>`_:
|
||||
Collection of scripts used in build and test systems, and various
|
||||
tools for QEMU codebase and execution traces.
|
||||
* `scsi <https://gitlab.com/qemu-project/qemu/-/tree/master/scsi>`_:
|
||||
Code related to SCSI support, used by SCSI devices.
|
||||
* `semihosting <https://gitlab.com/qemu-project/qemu/-/tree/master/semihosting>`_:
|
||||
QEMU `Semihosting <Semihosting>` implementation.
|
||||
* `stats <https://gitlab.com/qemu-project/qemu/-/tree/master/stats>`_:
|
||||
`Monitor <QEMU monitor>` stats commands implementation.
|
||||
* `storage-daemon <https://gitlab.com/qemu-project/qemu/-/tree/master/storage-daemon>`_:
|
||||
QEMU `Storage daemon <storage-daemon>` implementation.
|
||||
* `stubs <https://gitlab.com/qemu-project/qemu/-/tree/master/stubs>`_:
|
||||
Various stubs (empty functions) used to compile QEMU with specific
|
||||
configurations.
|
||||
* `subprojects <https://gitlab.com/qemu-project/qemu/-/tree/master/subprojects>`_:
|
||||
QEMU submodules used by QEMU build system.
|
||||
* `system <https://gitlab.com/qemu-project/qemu/-/tree/master/system>`_:
|
||||
QEMU `system mode <System emulation>` implementation (cpu, mmu, boot support).
|
||||
* `target <https://gitlab.com/qemu-project/qemu/-/tree/master/target>`_:
|
||||
Contains code for all target architectures supported (one subfolder
|
||||
per arch). For every architecture, you can find accelerator specific
|
||||
implementations.
|
||||
* `tcg <https://gitlab.com/qemu-project/qemu/-/tree/master/tcg>`_:
|
||||
:ref:`TCG <tcg>` related code.
|
||||
Contains one subfolder per host supported architecture.
|
||||
* `tests <https://gitlab.com/qemu-project/qemu/-/tree/master/tests>`_:
|
||||
QEMU `test <testing>` suite
|
||||
|
||||
- `data <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/data>`_:
|
||||
Data for various tests.
|
||||
- `decode <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/decode>`_:
|
||||
Testsuite for :ref:`decodetree <decodetree>` implementation.
|
||||
- `docker <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/docker>`_:
|
||||
Code and scripts to create `containers <container-ref>` used in `CI <ci>`.
|
||||
- `fp <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/fp>`_:
|
||||
QEMU testsuite for soft float implementation.
|
||||
- `functional <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/functional>`_:
|
||||
`Functional tests <checkfunctional-ref>` (full VM boot).
|
||||
- `lcitool <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/lcitool>`_:
|
||||
Generate dockerfiles for CI containers.
|
||||
- `migration <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/migration>`_:
|
||||
Test scripts and data for :ref:`Migration framework <migration>`.
|
||||
- `multiboot <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/multiboot>`_:
|
||||
Test multiboot functionality for x86_64/i386.
|
||||
- `qapi-schema <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/qapi-schema>`_:
|
||||
Test scripts and data for `QAPI <qapi-tests>`.
|
||||
- `qemu-iotests <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/qemu-iotests>`_:
|
||||
`Disk image and block tests <qemu-iotests>`.
|
||||
- `qtest <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/qtest>`_:
|
||||
`Device emulation testing <qtest>`.
|
||||
- `tcg <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/tcg>`__:
|
||||
`TCG related tests <checktcg-ref>`. Contains code per architecture
|
||||
(subfolder) and multiarch tests as well.
|
||||
- `tsan <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/tsan>`_:
|
||||
`Suppressions <tsan-suppressions>` for thread sanitizer.
|
||||
- `uefi-test-tools <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/uefi-test-tools>`_:
|
||||
Test tool for UEFI support.
|
||||
- `unit <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/unit>`_:
|
||||
QEMU `Unit tests <unit-tests>`.
|
||||
* `trace <https://gitlab.com/qemu-project/qemu/-/tree/master/trace>`_:
|
||||
:ref:`Tracing framework <tracing>`. Used to print information associated to various
|
||||
events during execution.
|
||||
* `ui <https://gitlab.com/qemu-project/qemu/-/tree/master/ui>`_:
|
||||
QEMU User interfaces.
|
||||
* `util <https://gitlab.com/qemu-project/qemu/-/tree/master/util>`_:
|
||||
Utility code used by other parts of QEMU.
|
||||
@@ -0,0 +1,80 @@
|
||||
.. _conflict-resolution:
|
||||
|
||||
Conflict Resolution Policy
|
||||
==========================
|
||||
|
||||
Conflicts in the community can take many forms, from someone having a
|
||||
bad day and using harsh and hurtful language on the mailing list to more
|
||||
serious code of conduct violations (including sexist/racist statements
|
||||
or threats of violence), and everything in between.
|
||||
|
||||
For the vast majority of issues, we aim to empower individuals to first
|
||||
resolve conflicts themselves, asking for help when needed, and only
|
||||
after that fails to escalate further. This approach gives people more
|
||||
control over the outcome of their dispute.
|
||||
|
||||
How we resolve conflicts
|
||||
------------------------
|
||||
|
||||
If you are experiencing conflict, please consider first addressing the
|
||||
perceived conflict directly with other involved parties, preferably through
|
||||
a real-time medium such as IRC. You could also try to get a third-party (e.g.
|
||||
a mutual friend, and/or someone with background on the issue, but not
|
||||
involved in the conflict) to intercede or mediate.
|
||||
|
||||
If this fails or if you do not feel comfortable proceeding this way, or
|
||||
if the problem requires immediate escalation, report the issue to the QEMU
|
||||
leadership committee by sending an email to [email protected], providing
|
||||
references to the misconduct.
|
||||
For very urgent topics, you can also inform one or more members through IRC.
|
||||
The up-to-date list of members is `available on the QEMU wiki
|
||||
<https://wiki.qemu.org/Conservancy>`__.
|
||||
|
||||
Your report will be treated confidentially by the leadership committee and
|
||||
not be published without your agreement. The QEMU leadership committee will
|
||||
then do its best to review the incident in a timely manner, and will either
|
||||
seek further information, or will make a determination on next steps.
|
||||
|
||||
Remedies
|
||||
--------
|
||||
|
||||
Escalating an issue to the QEMU leadership committee may result in actions
|
||||
impacting one or more involved parties. In the event the leadership
|
||||
committee has to intervene, here are some of the ways they might respond:
|
||||
|
||||
1. Take no action. For example, if the leadership committee determines
|
||||
the complaint has not been substantiated or is being made in bad faith,
|
||||
or if it is deemed to be outside its purview.
|
||||
|
||||
2. A private reprimand, explaining the consequences of continued behavior,
|
||||
to one or more involved individuals.
|
||||
|
||||
3. A private reprimand and request for a private or public apology
|
||||
|
||||
4. A public reprimand and request for a public apology
|
||||
|
||||
5. A public reprimand plus a mandatory cooling off period. The cooling
|
||||
off period may require, for example, one or more of the following:
|
||||
abstaining from maintainer duties; not interacting with people involved,
|
||||
including unsolicited interaction with those enforcing the guidelines
|
||||
and interaction on social media; being denied participation to in-person
|
||||
events. The cooling off period is voluntary but may escalate to a
|
||||
temporary ban in order to enforce it.
|
||||
|
||||
6. A temporary or permanent ban from some or all current and future QEMU
|
||||
spaces (mailing lists, IRC, wiki, etc.), possibly including in-person
|
||||
events.
|
||||
|
||||
In the event of severe harassment, the leadership committee may advise that
|
||||
the matter be escalated to the relevant local law enforcement agency. It
|
||||
is however not the role of the leadership committee to initiate contact
|
||||
with law enforcement on behalf of any of the community members involved
|
||||
in an incident.
|
||||
|
||||
Sources
|
||||
-------
|
||||
|
||||
This document was developed based on the `Drupal Conflict Resolution
|
||||
Policy and Process <https://www.drupal.org/conflict-resolution>`__
|
||||
and the `Mozilla Consequence Ladder
|
||||
<https://github.com/mozilla/diversity/blob/master/code-of-conduct-enforcement/consequence-ladder.md>`__
|
||||
@@ -0,0 +1,139 @@
|
||||
.. _cfi:
|
||||
|
||||
============================
|
||||
Control-Flow Integrity (CFI)
|
||||
============================
|
||||
|
||||
This document describes the current control-flow integrity (CFI) mechanism in
|
||||
QEMU. How it can be enabled, its benefits and deficiencies, and how it affects
|
||||
new and existing code in QEMU
|
||||
|
||||
Basics
|
||||
------
|
||||
|
||||
CFI is a hardening technique that focusing on guaranteeing that indirect
|
||||
function calls have not been altered by an attacker.
|
||||
The type used in QEMU is a forward-edge control-flow integrity that ensures
|
||||
function calls performed through function pointers, always call a "compatible"
|
||||
function. A compatible function is a function with the same signature of the
|
||||
function pointer declared in the source code.
|
||||
|
||||
This type of CFI is entirely compiler-based and relies on the compiler knowing
|
||||
the signature of every function and every function pointer used in the code.
|
||||
As of now, the only compiler that provides support for CFI is Clang.
|
||||
|
||||
CFI is best used on production binaries, to protect against unknown attack
|
||||
vectors.
|
||||
|
||||
In case of a CFI violation (i.e. call to a non-compatible function) QEMU will
|
||||
terminate abruptly, to stop the possible attack.
|
||||
|
||||
Building with CFI
|
||||
-----------------
|
||||
|
||||
NOTE: CFI requires the use of link-time optimization. Therefore, when CFI is
|
||||
selected, LTO will be automatically enabled.
|
||||
|
||||
To build with CFI, the minimum requirement is Clang 6+. If you
|
||||
are planning to also enable fuzzing, then Clang 11+ is needed (more on this
|
||||
later).
|
||||
|
||||
Given the use of LTO, a version of AR that supports LLVM IR is required.
|
||||
The easies way of doing this is by selecting the AR provided by LLVM::
|
||||
|
||||
AR=llvm-ar-9 CC=clang-9 CXX=clang++-9 /path/to/configure --enable-cfi
|
||||
|
||||
CFI is enabled on every binary produced.
|
||||
|
||||
If desired, an additional flag to increase the verbosity of the output in case
|
||||
of a CFI violation is offered (``--enable-debug-cfi``).
|
||||
|
||||
Using QEMU built with CFI
|
||||
-------------------------
|
||||
|
||||
A binary with CFI will work exactly like a standard binary. In case of a CFI
|
||||
violation, the binary will terminate with an illegal instruction signal.
|
||||
|
||||
Incompatible code with CFI
|
||||
--------------------------
|
||||
|
||||
As mentioned above, CFI is entirely compiler-based and therefore relies on
|
||||
compile-time knowledge of the code. This means that, while generally supported
|
||||
for most code, some specific use pattern can break CFI compatibility, and
|
||||
create false-positives. The two main patterns that can cause issues are:
|
||||
|
||||
* Just-in-time compiled code: since such code is created at runtime, the jump
|
||||
to the buffer containing JIT code will fail.
|
||||
|
||||
* Libraries loaded dynamically, e.g. with dlopen/dlsym, since the library was
|
||||
not known at compile time.
|
||||
|
||||
Current areas of QEMU that are not entirely compatible with CFI are:
|
||||
|
||||
1. TCG, since the idea of TCG is to pre-compile groups of instructions at
|
||||
runtime to speed-up interpretation, quite similarly to a JIT compiler
|
||||
|
||||
2. TCI, where the interpreter has to interpret the generic *call* operation
|
||||
|
||||
3. Plugins, since a plugin is implemented as an external library
|
||||
|
||||
4. Modules, since they are implemented as an external library
|
||||
|
||||
5. Directly calling signal handlers from the QEMU source code, since the
|
||||
signal handler may have been provided by an external library or even plugged
|
||||
at runtime.
|
||||
|
||||
Disabling CFI for a specific function
|
||||
-------------------------------------
|
||||
|
||||
If you are working on function that is performing a call using an
|
||||
incompatible way, as described before, you can selectively disable CFI checks
|
||||
for such function by using the decorator ``QEMU_DISABLE_CFI`` at function
|
||||
definition, and add an explanation on why the function is not compatible
|
||||
with CFI. An example of the use of ``QEMU_DISABLE_CFI`` is provided here::
|
||||
|
||||
/*
|
||||
* Disable CFI checks.
|
||||
* TCG creates binary blobs at runtime, with the transformed code.
|
||||
* A TB is a blob of binary code, created at runtime and called with an
|
||||
* indirect function call. Since such function did not exist at compile time,
|
||||
* the CFI runtime has no way to verify its signature and would fail.
|
||||
* TCG is not considered a security-sensitive part of QEMU so this does not
|
||||
* affect the impact of CFI in environment with high security requirements
|
||||
*/
|
||||
QEMU_DISABLE_CFI
|
||||
static inline tcg_target_ulong cpu_tb_exec(CPUState *cpu, TranslationBlock *itb)
|
||||
|
||||
NOTE: CFI needs to be disabled at the **caller** function, (i.e. a compatible
|
||||
cfi function that calls a non-compatible one), since the check is performed
|
||||
when the function call is performed.
|
||||
|
||||
CFI and fuzzing
|
||||
---------------
|
||||
|
||||
There is generally no advantage of using CFI and fuzzing together, because
|
||||
they target different environments (production for CFI, debug for fuzzing).
|
||||
|
||||
CFI could be used in conjunction with fuzzing to identify a broader set of
|
||||
bugs that may not end immediately in a segmentation fault or triggering
|
||||
an assertion. However, other sanitizers such as address and ub sanitizers
|
||||
can identify such bugs in a more precise way than CFI.
|
||||
|
||||
There is, however, an interesting use case in using CFI in conjunction with
|
||||
fuzzing, that is to make sure that CFI is not triggering any false positive
|
||||
in remote-but-possible parts of the code.
|
||||
|
||||
CFI can be enabled with fuzzing, but with some caveats:
|
||||
1. Fuzzing relies on the linker performing function wrapping at link-time.
|
||||
The standard BFD linker does not support function wrapping when LTO is
|
||||
also enabled. The workaround is to use LLVM's lld linker.
|
||||
2. Fuzzing also relies on a custom linker script, which is only supported by
|
||||
lld with version 11+.
|
||||
|
||||
In other words, to compile with fuzzing and CFI, clang 11+ is required, and
|
||||
lld needs to be used as a linker::
|
||||
|
||||
AR=llvm-ar-11 CC=clang-11 CXX=clang++-11 /path/to/configure --enable-cfi \
|
||||
-enable-fuzzing --extra-ldflags="-fuse-ld=lld"
|
||||
|
||||
and then, compile the fuzzers as usual.
|
||||
@@ -0,0 +1,10 @@
|
||||
.. _crypto-ref:
|
||||
|
||||
====================
|
||||
Cryptography in QEMU
|
||||
====================
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
luks-detached-header
|
||||
@@ -0,0 +1,272 @@
|
||||
.. _decodetree:
|
||||
|
||||
========================
|
||||
Decodetree Specification
|
||||
========================
|
||||
|
||||
A *decodetree* is built from instruction *patterns*. A pattern may
|
||||
represent a single architectural instruction or a group of same, depending
|
||||
on what is convenient for further processing.
|
||||
|
||||
Each pattern has both *fixedbits* and *fixedmask*, the combination of which
|
||||
describes the condition under which the pattern is matched::
|
||||
|
||||
(insn & fixedmask) == fixedbits
|
||||
|
||||
Each pattern may have *fields*, which are extracted from the insn and
|
||||
passed along to the translator. Examples of such are registers,
|
||||
immediates, and sub-opcodes.
|
||||
|
||||
In support of patterns, one may declare *fields*, *argument sets*, and
|
||||
*formats*, each of which may be re-used to simplify further definitions.
|
||||
|
||||
Fields
|
||||
======
|
||||
|
||||
Syntax::
|
||||
|
||||
field_def := '%' identifier ( field )* ( !function=identifier )?
|
||||
field := unnamed_field | named_field
|
||||
unnamed_field := number ':' ( 's' ) number
|
||||
named_field := identifier ':' ( 's' ) number
|
||||
|
||||
For *unnamed_field*, the first number is the least-significant bit position
|
||||
of the field and the second number is the length of the field. If the 's' is
|
||||
present, the field is considered signed.
|
||||
|
||||
A *named_field* refers to some other field in the instruction pattern
|
||||
or format. Regardless of the length of the other field where it is
|
||||
defined, it will be inserted into this field with the specified
|
||||
signedness and bit width.
|
||||
|
||||
Field definitions that involve loops (i.e. where a field is defined
|
||||
directly or indirectly in terms of itself) are errors.
|
||||
|
||||
A format can include fields that refer to named fields that are
|
||||
defined in the instruction pattern(s) that use the format.
|
||||
Conversely, an instruction pattern can include fields that refer to
|
||||
named fields that are defined in the format it uses. However you
|
||||
cannot currently do both at once (i.e. pattern P uses format F; F has
|
||||
a field A that refers to a named field B that is defined in P, and P
|
||||
has a field C that refers to a named field D that is defined in F).
|
||||
|
||||
If multiple ``fields`` are present, they are concatenated.
|
||||
In this way one can define disjoint fields.
|
||||
|
||||
If ``!function`` is specified, the concatenated result is passed through the
|
||||
named function, taking and returning an integral value.
|
||||
|
||||
One may use ``!function`` with zero ``fields``. This case is called
|
||||
a *parameter*, and the named function is only passed the ``DisasContext``
|
||||
and returns an integral value extracted from there.
|
||||
|
||||
A field with no ``fields`` and no ``!function`` is in error.
|
||||
|
||||
Field examples:
|
||||
|
||||
+-----------------------------+----------------------------------------------+
|
||||
| Input | Generated code |
|
||||
+=============================+==============================================+
|
||||
| ``%disp 0:s16`` | :: |
|
||||
| | |
|
||||
| | sextract(i, 0, 16) |
|
||||
+-----------------------------+----------------------------------------------+
|
||||
| ``%imm9 16:6 10:3`` | :: |
|
||||
| | |
|
||||
| | extract(i, 16, 6) << 3 | extract(i, 10, 3) |
|
||||
+-----------------------------+----------------------------------------------+
|
||||
| ``%disp12 0:s1 1:1 2:10`` | :: |
|
||||
| | |
|
||||
| | sextract(i, 0, 1) << 11 | |
|
||||
| | extract(i, 1, 1) << 10 | |
|
||||
| | extract(i, 2, 10) |
|
||||
+-----------------------------+----------------------------------------------+
|
||||
| ``%shimm8 5:s8 13:1 | :: |
|
||||
| !function=expand_shimm8`` | |
|
||||
| | expand_shimm8(sextract(i, 5, 8) << 1 | |
|
||||
| | extract(i, 13, 1)) |
|
||||
+-----------------------------+----------------------------------------------+
|
||||
| ``%sz_imm 10:2 sz:3 | :: |
|
||||
| !function=expand_sz_imm`` | |
|
||||
| | expand_sz_imm(extract(i, 10, 2) << 3 | |
|
||||
| | extract(a->sz, 0, 3)) |
|
||||
+-----------------------------+----------------------------------------------+
|
||||
|
||||
Argument Sets
|
||||
=============
|
||||
|
||||
Syntax::
|
||||
|
||||
args_def := '&' identifier ( args_elt )+ ( !extern )?
|
||||
args_elt := identifier (':' identifier)?
|
||||
|
||||
Each *args_elt* defines an argument within the argument set.
|
||||
If the form of the *args_elt* contains a colon, the first
|
||||
identifier is the argument name and the second identifier is
|
||||
the argument type. If the colon is missing, the argument
|
||||
type will be ``int``.
|
||||
|
||||
Each argument set will be rendered as a C structure "arg_$name"
|
||||
with each of the fields being one of the member arguments.
|
||||
|
||||
If ``!extern`` is specified, the backing structure is assumed
|
||||
to have been already declared, typically via a second decoder.
|
||||
|
||||
Argument sets are useful when one wants to define helper functions
|
||||
for the translator functions that can perform operations on a common
|
||||
set of arguments. This can ensure, for instance, that the ``AND``
|
||||
pattern and the ``OR`` pattern put their operands into the same named
|
||||
structure, so that a common ``gen_logic_insn`` may be able to handle
|
||||
the operations common between the two.
|
||||
|
||||
Argument set examples::
|
||||
|
||||
®3 ra rb rc
|
||||
&loadstore reg base offset
|
||||
&longldst reg base offset:int64_t
|
||||
|
||||
|
||||
Formats
|
||||
=======
|
||||
|
||||
Syntax::
|
||||
|
||||
fmt_def := '@' identifier ( fmt_elt )+
|
||||
fmt_elt := fixedbit_elt | field_elt | field_ref | args_ref
|
||||
fixedbit_elt := [01.-]+
|
||||
field_elt := identifier ':' 's'? number
|
||||
field_ref := '%' identifier | identifier '=' '%' identifier
|
||||
args_ref := '&' identifier
|
||||
|
||||
Defining a format is a handy way to avoid replicating groups of fields
|
||||
across many instruction patterns.
|
||||
|
||||
A *fixedbit_elt* describes a contiguous sequence of bits that must
|
||||
be 1, 0, or don't care. The difference between '.' and '-'
|
||||
is that '.' means that the bit will be covered with a field or a
|
||||
final 0 or 1 from the pattern, and '-' means that the bit is really
|
||||
ignored by the cpu and will not be specified.
|
||||
|
||||
A *field_elt* describes a simple field only given a width; the position of
|
||||
the field is implied by its position with respect to other *fixedbit_elt*
|
||||
and *field_elt*.
|
||||
|
||||
If any *fixedbit_elt* or *field_elt* appear, then all bits must be defined.
|
||||
Padding with a *fixedbit_elt* of all '.' is an easy way to accomplish that.
|
||||
|
||||
A *field_ref* incorporates a field by reference. This is the only way to
|
||||
add a complex field to a format. A field may be renamed in the process
|
||||
via assignment to another identifier. This is intended to allow the
|
||||
same argument set be used with disjoint named fields.
|
||||
|
||||
A single *args_ref* may specify an argument set to use for the format.
|
||||
The set of fields in the format must be a subset of the arguments in
|
||||
the argument set. If an argument set is not specified, one will be
|
||||
inferred from the set of fields.
|
||||
|
||||
It is recommended, but not required, that all *field_ref* and *args_ref*
|
||||
appear at the end of the line, not interleaving with *fixedbit_elf* or
|
||||
*field_elt*.
|
||||
|
||||
Format examples::
|
||||
|
||||
@opr ...... ra:5 rb:5 ... 0 ....... rc:5
|
||||
@opi ...... ra:5 lit:8 1 ....... rc:5
|
||||
|
||||
Patterns
|
||||
========
|
||||
|
||||
Syntax::
|
||||
|
||||
pat_def := identifier ( pat_elt )+
|
||||
pat_elt := fixedbit_elt | field_elt | field_ref | args_ref | fmt_ref | const_elt
|
||||
fmt_ref := '@' identifier
|
||||
const_elt := identifier '=' number
|
||||
|
||||
The *fixedbit_elt* and *field_elt* specifiers are unchanged from formats.
|
||||
A pattern that does not specify a named format will have one inferred
|
||||
from a referenced argument set (if present) and the set of fields.
|
||||
|
||||
A *const_elt* allows a argument to be set to a constant value. This may
|
||||
come in handy when fields overlap between patterns and one has to
|
||||
include the values in the *fixedbit_elt* instead.
|
||||
|
||||
The decoder will call a translator function for each pattern matched.
|
||||
|
||||
Pattern examples::
|
||||
|
||||
addl_r 010000 ..... ..... .... 0000000 ..... @opr
|
||||
addl_i 010000 ..... ..... .... 0000000 ..... @opi
|
||||
|
||||
which will, in part, invoke::
|
||||
|
||||
trans_addl_r(ctx, &arg_opr, insn)
|
||||
|
||||
and::
|
||||
|
||||
trans_addl_i(ctx, &arg_opi, insn)
|
||||
|
||||
Pattern Groups
|
||||
==============
|
||||
|
||||
Syntax::
|
||||
|
||||
group := overlap_group | no_overlap_group
|
||||
overlap_group := '{' ( pat_def | group )+ '}'
|
||||
no_overlap_group := '[' ( pat_def | group )+ ']'
|
||||
|
||||
A *group* begins with a lone open-brace or open-bracket, with all
|
||||
subsequent lines indented two spaces, and ending with a lone
|
||||
close-brace or close-bracket. Groups may be nested, increasing the
|
||||
required indentation of the lines within the nested group to two
|
||||
spaces per nesting level.
|
||||
|
||||
Patterns within overlap groups are allowed to overlap. Conflicts are
|
||||
resolved by selecting the patterns in order. If all of the fixedbits
|
||||
for a pattern match, its translate function will be called. If the
|
||||
translate function returns false, then subsequent patterns within the
|
||||
group will be matched.
|
||||
|
||||
Patterns within no-overlap groups are not allowed to overlap, just
|
||||
the same as ungrouped patterns. Thus no-overlap groups are intended
|
||||
to be nested inside overlap groups.
|
||||
|
||||
The following example from PA-RISC shows specialization of the *or*
|
||||
instruction::
|
||||
|
||||
{
|
||||
{
|
||||
nop 000010 ----- ----- 0000 001001 0 00000
|
||||
copy 000010 00000 r1:5 0000 001001 0 rt:5
|
||||
}
|
||||
or 000010 rt2:5 r1:5 cf:4 001001 0 rt:5
|
||||
}
|
||||
|
||||
When the *cf* field is zero, the instruction has no side effects,
|
||||
and may be specialized. When the *rt* field is zero, the output
|
||||
is discarded and so the instruction has no effect. When the *rt2*
|
||||
field is zero, the operation is ``reg[r1] | 0`` and so encodes
|
||||
the canonical register copy operation.
|
||||
|
||||
The output from the generator might look like::
|
||||
|
||||
switch (insn & 0xfc000fe0) {
|
||||
case 0x08000240:
|
||||
/* 000010.. ........ ....0010 010..... */
|
||||
if ((insn & 0x0000f000) == 0x00000000) {
|
||||
/* 000010.. ........ 00000010 010..... */
|
||||
if ((insn & 0x0000001f) == 0x00000000) {
|
||||
/* 000010.. ........ 00000010 01000000 */
|
||||
extract_decode_Fmt_0(&u.f_decode0, insn);
|
||||
if (trans_nop(ctx, &u.f_decode0)) return true;
|
||||
}
|
||||
if ((insn & 0x03e00000) == 0x00000000) {
|
||||
/* 00001000 000..... 00000010 010..... */
|
||||
extract_decode_Fmt_1(&u.f_decode1, insn);
|
||||
if (trans_copy(ctx, &u.f_decode1)) return true;
|
||||
}
|
||||
}
|
||||
extract_decode_Fmt_2(&u.f_decode2, insn);
|
||||
if (trans_or(ctx, &u.f_decode2)) return true;
|
||||
return false;
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
|
||||
==================
|
||||
QEMU Documentation
|
||||
==================
|
||||
|
||||
QEMU's documentation is written in reStructuredText format and
|
||||
built using the Sphinx documentation generator. We generate both
|
||||
the HTML manual and the manpages from the some documentation sources.
|
||||
|
||||
hxtool and .hx files
|
||||
--------------------
|
||||
|
||||
The documentation for QEMU command line options and Human Monitor Protocol
|
||||
(HMP) commands is written in files with the ``.hx`` suffix. These
|
||||
are processed in two ways:
|
||||
|
||||
* ``scripts/hxtool`` creates C header files from them, which are included
|
||||
in QEMU to do things like handle the ``--help`` option output
|
||||
* a Sphinx extension in ``docs/sphinx/hxtool.py`` generates rST output
|
||||
to be included in the HTML or manpage documentation
|
||||
|
||||
The syntax of these ``.hx`` files is simple. It is broadly an
|
||||
alternation of C code put into the C output and rST format text
|
||||
put into the documentation. A few special directives are recognised;
|
||||
these are all-caps and must be at the beginning of the line.
|
||||
|
||||
``HXCOMM`` is the comment marker. The line, including any arbitrary
|
||||
text after the marker, is discarded and appears neither in the C output
|
||||
nor the documentation output.
|
||||
|
||||
``SRST`` starts a reStructuredText section. Following lines
|
||||
are put into the documentation verbatim, and discarded from the C output.
|
||||
The alternative form ``SRST()`` is used to define a label which can be
|
||||
referenced from elsewhere in the rST documentation. The label will take
|
||||
the form ``<DOCNAME-HXFILE-LABEL>``, where ``DOCNAME`` is the name of the
|
||||
top level rST file, ``HXFILE`` is the filename of the .hx file without
|
||||
the ``.hx`` extension, and ``LABEL`` is the text provided within the
|
||||
``SRST()`` directive. For example,
|
||||
``<system/invocation-qemu-options-initrd>``.
|
||||
|
||||
``ERST`` ends the documentation section started with ``SRST``,
|
||||
and switches back to a C code section.
|
||||
|
||||
``DEFHEADING()`` defines a heading that should appear in both the
|
||||
``--help`` output and in the documentation. This directive should
|
||||
be in the C code block. If there is a string inside the brackets,
|
||||
this is the heading to use. If this string is empty, it produces
|
||||
a blank line in the ``--help`` output and is ignored for the rST
|
||||
output.
|
||||
|
||||
``ARCHHEADING()`` is a variant of ``DEFHEADING()`` which produces
|
||||
the heading only if the specified guest architecture was compiled
|
||||
into QEMU. This should be avoided in new documentation.
|
||||
|
||||
Within C code sections, you should check the comments at the top
|
||||
of the file to see what the expected usage is, because this
|
||||
varies between files. For instance in ``qemu-options.hx`` we use
|
||||
the ``DEF()`` macro to define each option and specify its ``--help``
|
||||
text, but in ``hmp-commands.hx`` the C code sections are elements
|
||||
of an array of structs of type ``HMPCommand`` which define the
|
||||
name, behaviour and help text for each monitor command.
|
||||
|
||||
In the file ``qemu-options.hx``, do not try to explicitly define a
|
||||
reStructuredText label within a documentation section. This file
|
||||
is included into two separate Sphinx documents, and some
|
||||
versions of Sphinx will complain about the duplicate label
|
||||
that results. Use the ``SRST()`` directive documented above, to
|
||||
emit an unambiguous label.
|
||||
@@ -0,0 +1,127 @@
|
||||
.. _ebpf-rss:
|
||||
|
||||
===========================
|
||||
eBPF RSS virtio-net support
|
||||
===========================
|
||||
|
||||
RSS(Receive Side Scaling) is used to distribute network packets to guest virtqueues
|
||||
by calculating packet hash. Usually every queue is processed then by a specific guest CPU core.
|
||||
|
||||
For now there are 2 RSS implementations in qemu:
|
||||
- 'in-qemu' RSS (functions if qemu receives network packets, i.e. vhost=off)
|
||||
- eBPF RSS (can function with also with vhost=on)
|
||||
|
||||
eBPF support (CONFIG_EBPF) is enabled by 'configure' script.
|
||||
To enable eBPF RSS support use './configure --enable-bpf'.
|
||||
|
||||
If steering BPF is not set for kernel's TUN module, the TUN uses automatic selection
|
||||
of rx virtqueue based on lookup table built according to calculated symmetric hash
|
||||
of transmitted packets.
|
||||
If steering BPF is set for TUN the BPF code calculates the hash of packet header and
|
||||
returns the virtqueue number to place the packet to.
|
||||
|
||||
Simplified decision formula:
|
||||
|
||||
.. code:: C
|
||||
|
||||
queue_index = indirection_table[hash(<packet data>)%<indirection_table size>]
|
||||
|
||||
|
||||
Not for all packets, the hash can/should be calculated.
|
||||
|
||||
Note: currently, eBPF RSS does not support hash reporting.
|
||||
|
||||
eBPF RSS turned on by different combinations of vhost-net, vitrio-net and tap configurations:
|
||||
|
||||
- eBPF is used:
|
||||
|
||||
tap,vhost=off & virtio-net-pci,rss=on,hash=off
|
||||
|
||||
- eBPF is used:
|
||||
|
||||
tap,vhost=on & virtio-net-pci,rss=on,hash=off
|
||||
|
||||
- 'in-qemu' RSS is used:
|
||||
|
||||
tap,vhost=off & virtio-net-pci,rss=on,hash=on
|
||||
|
||||
- eBPF is used, hash population feature is not reported to the guest:
|
||||
|
||||
tap,vhost=on & virtio-net-pci,rss=on,hash=on
|
||||
|
||||
If CONFIG_EBPF is not set then only 'in-qemu' RSS is supported.
|
||||
Also 'in-qemu' RSS, as a fallback, is used if the eBPF program failed to load or set to TUN.
|
||||
|
||||
RSS eBPF program
|
||||
----------------
|
||||
|
||||
RSS program located in ebpf/rss.bpf.skeleton.h generated by bpftool.
|
||||
So the program is part of the qemu binary.
|
||||
Initially, the eBPF program was compiled by clang and source code located at tools/ebpf/rss.bpf.c.
|
||||
Prerequisites to recompile the eBPF program (regenerate ebpf/rss.bpf.skeleton.h):
|
||||
|
||||
llvm, clang, kernel source tree, bpftool
|
||||
Adjust Makefile.ebpf to reflect the location of the kernel source tree
|
||||
|
||||
$ cd tools/ebpf
|
||||
$ make -f Makefile.ebpf
|
||||
|
||||
Current eBPF RSS implementation uses 'bounded loops' with 'backward jump instructions' which present in the last kernels.
|
||||
Overall eBPF RSS works on kernels 5.8+.
|
||||
|
||||
eBPF RSS implementation
|
||||
-----------------------
|
||||
|
||||
eBPF RSS loading functionality located in ebpf/ebpf_rss.c and ebpf/ebpf_rss.h.
|
||||
|
||||
The ``struct EBPFRSSContext`` structure that holds 4 file descriptors:
|
||||
|
||||
- ctx - pointer of the libbpf context.
|
||||
- program_fd - file descriptor of the eBPF RSS program.
|
||||
- map_configuration - file descriptor of the 'configuration' map. This map contains one element of 'struct EBPFRSSConfig'. This configuration determines eBPF program behavior.
|
||||
- map_toeplitz_key - file descriptor of the 'Toeplitz key' map. One element of the 40byte key prepared for the hashing algorithm.
|
||||
- map_indirections_table - 128 elements of queue indexes.
|
||||
|
||||
``struct EBPFRSSConfig`` fields:
|
||||
|
||||
- redirect - "boolean" value, should the hash be calculated, on false - ``default_queue`` would be used as the final decision.
|
||||
- populate_hash - for now, not used. eBPF RSS doesn't support hash reporting.
|
||||
- hash_types - binary mask of different hash types. See ``VIRTIO_NET_RSS_HASH_TYPE_*`` defines. If for packet hash should not be calculated - ``default_queue`` would be used.
|
||||
- indirections_len - length of the indirections table, maximum 128.
|
||||
- default_queue - the queue index that used for packet that shouldn't be hashed. For some packets, the hash can't be calculated(g.e ARP).
|
||||
|
||||
Functions:
|
||||
|
||||
- ``ebpf_rss_init()`` - sets ctx to NULL, which indicates that EBPFRSSContext is not loaded.
|
||||
- ``ebpf_rss_load()`` - creates 3 maps and loads eBPF program from the rss.bpf.skeleton.h. Returns 'true' on success. After that, program_fd can be used to set steering for TAP.
|
||||
- ``ebpf_rss_set_all()`` - sets values for eBPF maps. ``indirections_table`` length is in EBPFRSSConfig. ``toeplitz_key`` is VIRTIO_NET_RSS_MAX_KEY_SIZE aka 40 bytes array.
|
||||
- ``ebpf_rss_unload()`` - close all file descriptors and set ctx to NULL.
|
||||
|
||||
Simplified eBPF RSS workflow:
|
||||
|
||||
.. code:: C
|
||||
|
||||
struct EBPFRSSConfig config;
|
||||
config.redirect = 1;
|
||||
config.hash_types = VIRTIO_NET_RSS_HASH_TYPE_UDPv4 | VIRTIO_NET_RSS_HASH_TYPE_TCPv4;
|
||||
config.indirections_len = VIRTIO_NET_RSS_MAX_TABLE_LEN;
|
||||
config.default_queue = 0;
|
||||
|
||||
uint16_t table[VIRTIO_NET_RSS_MAX_TABLE_LEN] = {...};
|
||||
uint8_t key[VIRTIO_NET_RSS_MAX_KEY_SIZE] = {...};
|
||||
|
||||
struct EBPFRSSContext ctx;
|
||||
ebpf_rss_init(&ctx);
|
||||
ebpf_rss_load(&ctx);
|
||||
ebpf_rss_set_all(&ctx, &config, table, key);
|
||||
if (net_client->info->set_steering_ebpf != NULL) {
|
||||
net_client->info->set_steering_ebpf(net_client, ctx->program_fd);
|
||||
}
|
||||
...
|
||||
ebpf_unload(&ctx);
|
||||
|
||||
|
||||
NetClientState SetSteeringEBPF()
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
For now, ``set_steering_ebpf()`` method supported by Linux TAP NetClientState. The method requires an eBPF program file descriptor as an argument.
|
||||
@@ -0,0 +1,55 @@
|
||||
.. SPDX-License-Identifier: GPL-2.0-or-later
|
||||
|
||||
Hexagon L2 Vectored Interrupt Controller
|
||||
========================================
|
||||
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
+-------------+ +----------------------+
|
||||
| l2vic | | hexagon core |
|
||||
| | | |
|
||||
IRQ in ---->| | | |
|
||||
IRQ in ---->| VID0 -|----------------->| irq2 |
|
||||
... ---->| | | | |
|
||||
IRQ in ---->| | | v |
|
||||
| ... | | <int steering> |
|
||||
| | | / | | \ |
|
||||
IRQ in ---->| | | t0 t1 t2 t3 ...|
|
||||
IRQ in ---->| VIDN -| | |
|
||||
... ---->| | | |
|
||||
IRQ in ---->| | | Global SREG File |
|
||||
| | | |
|
||||
| State | | |
|
||||
| [ ] <--|==================|==> [ VID ] |
|
||||
| [ ] <--|==================|==> [ VID1 ] |
|
||||
| | | |
|
||||
+-------------+ +----------------------+
|
||||
|
||||
L2VIC/Core Integration
|
||||
----------------------
|
||||
|
||||
* hexagon core supports 8 external interrupt sources
|
||||
* l2vic supports 1024 input interrupts mapped among 4 output interrupts
|
||||
* l2vic has four output signals: { VID0, VID1, VID2, VID3 }
|
||||
* l2vic device has a bank of registers per-VID that can be used to query
|
||||
the status or assert new interrupts.
|
||||
* Interrupts are 'steered' to threads based on { thread priority, 'EX' state,
|
||||
thread interrupt mask, thread interrupt enable, global interrupt enable,
|
||||
etc. }.
|
||||
* Any hardware thread could conceivably handle any input interrupt, dependent
|
||||
on state.
|
||||
* The system register transfer instruction can read the VID0-VID3 values from
|
||||
the l2vic when reading from hexagon core system registers "VID" and "VID1".
|
||||
* When l2vic VID0 has multiple active interrupts, it pulses the VID0 output
|
||||
IRQ and stores the IRQ number for the VID0 register field. Only after this
|
||||
interrupt is cleared can the l2vic pulse the VID0 output IRQ again and provide
|
||||
the next interrupt number on the VID0 register.
|
||||
* The ``ciad`` instruction clears the l2vic input interrupt and un-disables the
|
||||
core interrupt. If some/an l2vic VID0 interrupt is pending when this occurs,
|
||||
the next interrupt should fire and any subsequent reads of the VID register
|
||||
should reflect the newly raised interrupt.
|
||||
* In QEMU, on an external interrupt or an unmasked-pending interrupt,
|
||||
all vCPUs are triggered (has_work==true) and each will grab the IO lock
|
||||
while considering the steering logic to determine whether they're the thread
|
||||
that must handle the interrupt.
|
||||
@@ -0,0 +1,112 @@
|
||||
.. SPDX-License-Identifier: GPL-2.0-or-later
|
||||
|
||||
.. _Hexagon-System-arch:
|
||||
|
||||
Hexagon System Architecture
|
||||
===========================
|
||||
|
||||
The hexagon architecture has some unique elements which are described here.
|
||||
|
||||
Interrupts
|
||||
----------
|
||||
When interrupts arrive at a Hexagon DSP core, they are priority-steered to
|
||||
be handled by an eligible hardware thread with the lowest priority.
|
||||
|
||||
Memory
|
||||
------
|
||||
Each hardware thread has an ``SSR.ASID`` field that contains its Address
|
||||
Space Identifier. This value is catenated with a 32-bit virtual address -
|
||||
the MMU can then resolve this extended virtual address to a physical address.
|
||||
|
||||
TLBs
|
||||
----
|
||||
The format of a TLB entry is shown below.
|
||||
|
||||
.. note::
|
||||
The Small Core DSPs have a different TLB format which is not yet
|
||||
supported.
|
||||
|
||||
.. admonition:: Diagram
|
||||
|
||||
.. code:: text
|
||||
|
||||
6 5 4 3
|
||||
3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2
|
||||
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
||||
|v|g|x|A|A| | |
|
||||
|a|l|P|1|0| ASID | Virtual Page |
|
||||
|l|b| | | | | |
|
||||
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
||||
|
||||
3 2 1 0
|
||||
1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
|
||||
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
||||
| | | | | | | |
|
||||
|x|w|r|u|Cacheab| Physical Page |S|
|
||||
| | | | | | | |
|
||||
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
||||
|
||||
|
||||
* ASID: the address-space identifier
|
||||
* A1, A0: the behavior of these cache line attributes are not modeled by QEMU.
|
||||
* xP: the extra-physical bit is the most significant physical address bit.
|
||||
* S: the S bit and the LSBs of the physical page indicate the page size
|
||||
* val: this is the 'valid' bit, when set it indicates that page matching
|
||||
should consider this entry.
|
||||
|
||||
.. list-table:: Page sizes
|
||||
:widths: 25 25 50
|
||||
:header-rows: 1
|
||||
|
||||
* - S-bit
|
||||
- Phys page LSBs
|
||||
- Page size
|
||||
* - 1
|
||||
- N/A
|
||||
- 4kb
|
||||
* - 0
|
||||
- 0b1
|
||||
- 16kb
|
||||
* - 0
|
||||
- 0b10
|
||||
- 64kb
|
||||
* - 0
|
||||
- 0b100
|
||||
- 256kb
|
||||
* - 0
|
||||
- 0b1000
|
||||
- 1MB
|
||||
* - 0
|
||||
- 0b10000
|
||||
- 4MB
|
||||
* - 0
|
||||
- 0b100000
|
||||
- 16MB
|
||||
|
||||
* glb: if the global bit is set, the ASID is not considered when matching
|
||||
TLBs.
|
||||
* Cacheab: the cacheability attributes of TLBs are not modeled, these bits
|
||||
are ignored.
|
||||
* RWX: read-, write-, execute-, enable bits. Indicates if user programs
|
||||
are permitted to read/write/execute the given page.
|
||||
* U: indicates if user programs can access this page.
|
||||
|
||||
Scheduler
|
||||
---------
|
||||
The Hexagon system architecture has a feature to assist the guest OS
|
||||
task scheduler. The guest OS can enable this feature by setting
|
||||
``SCHEDCFG.EN``. The ``BESTWAIT`` register is programmed by the guest OS
|
||||
to indicate the priority of the highest priority task waiting to run on a
|
||||
hardware thread. The reschedule interrupt is triggered when any hardware
|
||||
thread's priority in ``STID.PRIO`` is worse than the ``BESTWAIT``. When
|
||||
it is triggered, the ``BESTWAIT.PRIO`` value is reset to 0x1ff.
|
||||
|
||||
HVX Coprocessor
|
||||
---------------
|
||||
The Supervisor Status Register field ``SSR.XA`` binds a DSP hardware thread
|
||||
to one of the eight possible HVX contexts. The guest OS is responsible for
|
||||
managing this resource.
|
||||
|
||||
.. seealso::
|
||||
|
||||
``target/hexagon/README`` in the QEMU source tree for more info about Hexagon.
|
||||
@@ -0,0 +1,19 @@
|
||||
Internal QEMU APIs
|
||||
------------------
|
||||
|
||||
Details about how QEMU's various internal APIs. Most of these are
|
||||
generated from in-code annotations to function prototypes.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
bitops
|
||||
loads-stores
|
||||
lockcnt
|
||||
memory
|
||||
modules
|
||||
pci
|
||||
qom-api
|
||||
qdev-api
|
||||
ui
|
||||
zoned-storage
|
||||
@@ -0,0 +1,16 @@
|
||||
QEMU Build System
|
||||
-----------------
|
||||
|
||||
Details about how QEMU's build system works. You will need to understand
|
||||
some of the basics if you are adding new files and targets to the build.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 3
|
||||
|
||||
build-system
|
||||
build-environment
|
||||
kconfig
|
||||
docs
|
||||
qapi-code-gen
|
||||
qapi-domain
|
||||
control-flow-integrity
|
||||
@@ -0,0 +1,32 @@
|
||||
.. _internal-subsystem:
|
||||
|
||||
Internal Subsystem Information
|
||||
------------------------------
|
||||
|
||||
Details about QEMU's various subsystems including how to add features to them.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
qom
|
||||
atomics
|
||||
rcu
|
||||
block-coroutine-wrapper
|
||||
clocks
|
||||
ebpf_rss
|
||||
hexagon-sys
|
||||
hexagon-l2vic
|
||||
migration/index
|
||||
multi-process
|
||||
reset
|
||||
s390-cpu-topology
|
||||
s390-dasd-ipl
|
||||
ssi
|
||||
tracing
|
||||
uefi-vars
|
||||
vfio-iommufd
|
||||
vfio-mdpy
|
||||
writing-monitor-commands
|
||||
virtio-backends
|
||||
crypto
|
||||
multiple-iothreads
|
||||
@@ -0,0 +1,21 @@
|
||||
.. _development_process:
|
||||
|
||||
QEMU Community Processes
|
||||
------------------------
|
||||
|
||||
Notes about how to interact with the community and how and where to submit patches.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
code-of-conduct
|
||||
conflict-resolution
|
||||
maintainers
|
||||
style
|
||||
submitting-a-patch
|
||||
code-provenance
|
||||
trivial-patches
|
||||
stable-process
|
||||
submitting-a-pull-request
|
||||
secure-coding-practices
|
||||
rust
|
||||
@@ -0,0 +1,19 @@
|
||||
.. _tcg:
|
||||
|
||||
TCG Emulation
|
||||
-------------
|
||||
|
||||
Details about QEMU's Tiny Code Generator and the infrastructure
|
||||
associated with emulation. You do not need to worry about this if you
|
||||
are only implementing things for HW accelerated hypervisors.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
tcg
|
||||
tcg-ops
|
||||
decodetree
|
||||
multi-thread-tcg
|
||||
tcg-icount
|
||||
tcg-plugins
|
||||
replay
|
||||
@@ -0,0 +1,38 @@
|
||||
---------------------
|
||||
Developer Information
|
||||
---------------------
|
||||
|
||||
This section of the manual documents various parts of the internals of
|
||||
QEMU. You only need to read it if you are interested in reading or
|
||||
modifying QEMU's source code.
|
||||
|
||||
QEMU is a large and mature project with a number of complex subsystems
|
||||
that can be overwhelming to understand. The development documentation
|
||||
is not comprehensive but hopefully presents enough to get you started.
|
||||
If there are areas that are unclear please reach out either via the
|
||||
IRC channel or mailing list and hopefully we can improve the
|
||||
documentation for future developers.
|
||||
|
||||
All developers will want to familiarise themselves with
|
||||
:ref:`development_process` and how the community interacts. Please pay
|
||||
particular attention to the :ref:`coding-style` and
|
||||
:ref:`submitting-a-patch` sections to avoid common pitfalls.
|
||||
|
||||
If you wish to implement a new hardware model you will want to read
|
||||
through the :ref:`qom` documentation to understand how QEMU's object
|
||||
model works.
|
||||
|
||||
Those wishing to enhance or add new CPU emulation capabilities will
|
||||
want to read our :ref:`tcg` documentation, especially the overview of
|
||||
the :ref:`tcg_internals`.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
index-process
|
||||
index-build
|
||||
testing/index
|
||||
index-api
|
||||
index-internals
|
||||
index-tcg
|
||||
codebase
|
||||
@@ -0,0 +1,329 @@
|
||||
.. _kconfig:
|
||||
|
||||
================
|
||||
QEMU and Kconfig
|
||||
================
|
||||
|
||||
QEMU is a very versatile emulator; it can be built for a variety of
|
||||
targets, where each target can emulate various boards and at the same
|
||||
time different targets can share large amounts of code. For example,
|
||||
a POWER and an x86 board can run the same code to emulate a PCI network
|
||||
card, even though the boards use different PCI host bridges, and they
|
||||
can run the same code to emulate a SCSI disk while using different
|
||||
SCSI adapters. Arm, s390 and x86 boards can all present a virtio-blk
|
||||
disk to their guests, but with three different virtio guest interfaces.
|
||||
|
||||
Each QEMU target enables a subset of the boards, devices and buses that
|
||||
are included in QEMU's source code. As a result, each QEMU executable
|
||||
only links a small subset of the files that form QEMU's source code;
|
||||
anything that is not needed to support a particular target is culled.
|
||||
|
||||
QEMU uses a simple domain-specific language to describe the dependencies
|
||||
between components. This is useful for two reasons:
|
||||
|
||||
* new targets and boards can be added without knowing in detail the
|
||||
architecture of the hardware emulation subsystems. Boards only have
|
||||
to list the components they need, and the compiled executable will
|
||||
include all the required dependencies and all the devices that the
|
||||
user can add to that board;
|
||||
|
||||
* users can easily build reduced versions of QEMU that support only a subset
|
||||
of boards or devices. For example, by default most targets will include
|
||||
all emulated PCI devices that QEMU supports, but the build process is
|
||||
configurable and it is easy to drop unnecessary (or otherwise unwanted)
|
||||
code to make a leaner binary.
|
||||
|
||||
This domain-specific language is based on the Kconfig language that
|
||||
originated in the Linux kernel, though it was heavily simplified and
|
||||
the handling of dependencies is stricter in QEMU.
|
||||
|
||||
Unlike Linux, there is no user interface to edit the configuration, which
|
||||
is instead specified in per-target files under the ``configs/``
|
||||
directory of the QEMU source tree. This is because, unlike Linux,
|
||||
configuration and dependencies can be treated as a black box when building
|
||||
QEMU; the default configuration that QEMU ships with should be okay in
|
||||
almost all cases.
|
||||
|
||||
The Kconfig language
|
||||
--------------------
|
||||
|
||||
Kconfig defines configurable components in files named ``hw/*/Kconfig``.
|
||||
Note that configurable components are _not_ visible in C code as preprocessor
|
||||
symbols; they are only visible in the Makefile. Each configurable component
|
||||
defines a Makefile variable whose name starts with ``CONFIG_``.
|
||||
|
||||
All elements have boolean (true/false) type; truth is written as ``y``, while
|
||||
falsehood is written ``n``. They are defined in a Kconfig
|
||||
stanza like the following::
|
||||
|
||||
config ARM_VIRT
|
||||
bool
|
||||
imply PCI_DEVICES
|
||||
select A15MPCORE
|
||||
select ACPI
|
||||
select ARM_SMMUV3
|
||||
|
||||
The ``config`` keyword introduces a new configuration element. In the example
|
||||
above, Makefiles will have access to a variable named ``CONFIG_ARM_VIRT``,
|
||||
with value ``y`` or ``n`` (respectively for boolean true and false).
|
||||
|
||||
Boolean expressions can be used within the language, whenever ``<expr>``
|
||||
is written in the remainder of this section. The ``&&``, ``||`` and
|
||||
``!`` operators respectively denote conjunction (AND), disjunction (OR)
|
||||
and negation (NOT).
|
||||
|
||||
The ``bool`` data type declaration is optional, but it is suggested to
|
||||
include it for clarity and future-proofing. After ``bool`` the following
|
||||
directives can be included:
|
||||
|
||||
**dependencies**: ``depends on <expr>``
|
||||
|
||||
This defines a dependency for this configurable element. Dependencies
|
||||
evaluate an expression and force the value of the variable to false
|
||||
if the expression is false.
|
||||
|
||||
**reverse dependencies**: ``select <symbol> [if <expr>]``
|
||||
|
||||
While ``depends on`` can force a symbol to false, reverse dependencies can
|
||||
be used to force another symbol to true. In the following example,
|
||||
``CONFIG_BAZ`` will be true whenever ``CONFIG_FOO`` is true::
|
||||
|
||||
config FOO
|
||||
select BAZ
|
||||
|
||||
The optional expression will prevent ``select`` from having any effect
|
||||
unless it is true.
|
||||
|
||||
Note that unlike Linux's Kconfig implementation, QEMU will detect
|
||||
contradictions between ``depends on`` and ``select`` statements and prevent
|
||||
you from building such a configuration.
|
||||
|
||||
**default value**: ``default <value> [if <expr>]``
|
||||
|
||||
Default values are assigned to the config symbol if no other value was
|
||||
set by the user via ``configs/*.mak`` files, and only if
|
||||
``select`` or ``depends on`` directives do not force the value to true
|
||||
or false respectively. ``<value>`` can be ``y`` or ``n``; it cannot
|
||||
be an arbitrary Boolean expression. However, a condition for applying
|
||||
the default value can be added with ``if``.
|
||||
|
||||
A configuration element can have any number of default values (usually,
|
||||
if more than one default is present, they will have different
|
||||
conditions). If multiple default values satisfy their condition,
|
||||
only the first defined one is active.
|
||||
|
||||
**reverse default** (weak reverse dependency): ``imply <symbol> [if <expr>]``
|
||||
|
||||
This is similar to ``select`` as it applies a lower limit of ``y``
|
||||
to another symbol. However, the lower limit is only a default
|
||||
and the "implied" symbol's value may still be set to ``n`` from a
|
||||
``configs/*.mak`` files. The following two examples are
|
||||
equivalent::
|
||||
|
||||
config FOO
|
||||
bool
|
||||
imply BAZ
|
||||
|
||||
config BAZ
|
||||
bool
|
||||
default y if FOO
|
||||
|
||||
The next section explains where to use ``imply`` or ``default y``.
|
||||
|
||||
Guidelines for writing Kconfig files
|
||||
------------------------------------
|
||||
|
||||
Configurable elements in QEMU fall under five broad groups. Each group
|
||||
declares its dependencies in different ways:
|
||||
|
||||
**subsystems**, of which **buses** are a special case
|
||||
|
||||
Example::
|
||||
|
||||
config SCSI
|
||||
bool
|
||||
|
||||
Subsystems always default to false (they have no ``default`` directive)
|
||||
and are never visible in ``configs/*.mak`` files. It's
|
||||
up to other symbols to ``select`` whatever subsystems they require.
|
||||
|
||||
They sometimes have ``select`` directives to bring in other required
|
||||
subsystems or buses. For example, ``AUX`` (the DisplayPort auxiliary
|
||||
channel "bus") selects ``I2C`` because it can act as an I2C master too.
|
||||
|
||||
**devices**
|
||||
|
||||
Example::
|
||||
|
||||
config MEGASAS_SCSI_PCI
|
||||
bool
|
||||
default y if PCI_DEVICES
|
||||
depends on PCI
|
||||
select SCSI
|
||||
|
||||
Devices are the most complex of the five. They can have a variety
|
||||
of directives that cooperate so that a default configuration includes
|
||||
all the devices that can be accessed from QEMU.
|
||||
|
||||
Devices *depend on* the bus that they lie on, for example a PCI
|
||||
device would specify ``depends on PCI``. An MMIO device will likely
|
||||
have no ``depends on`` directive. Devices also *select* the buses
|
||||
that the device provides, for example a SCSI adapter would specify
|
||||
``select SCSI``. Finally, devices are usually ``default y`` if and
|
||||
only if they have at least one ``depends on``; the default could be
|
||||
conditional on a device group.
|
||||
|
||||
Devices also select any optional subsystem that they use; for example
|
||||
a video card might specify ``select EDID`` if it needs to build EDID
|
||||
information and publish it to the guest.
|
||||
|
||||
**device groups**
|
||||
|
||||
Example::
|
||||
|
||||
config PCI_DEVICES
|
||||
bool
|
||||
|
||||
Device groups provide a convenient mechanism to enable/disable many
|
||||
devices in one go. This is useful when a set of devices is likely to
|
||||
be enabled/disabled by several targets. Device groups usually need
|
||||
no directive and are not used in the Makefile either; they only appear
|
||||
as conditions for ``default y`` directives.
|
||||
|
||||
QEMU currently has three device groups, ``PCI_DEVICES``, ``I2C_DEVICES``,
|
||||
and ``TEST_DEVICES``. PCI devices usually have a ``default y if
|
||||
PCI_DEVICES`` directive rather than just ``default y``. This lets
|
||||
some boards (notably s390) easily support a subset of PCI devices,
|
||||
for example only VFIO (passthrough) and virtio-pci devices.
|
||||
``I2C_DEVICES`` is similar to ``PCI_DEVICES``. It contains i2c devices
|
||||
that users might reasonably want to plug in to an i2c bus on any
|
||||
board (and not ones which are very board-specific or that need
|
||||
to be wired up in a way that can't be done on the command line).
|
||||
``TEST_DEVICES`` instead is used for devices that are rarely used on
|
||||
production virtual machines, but provide useful hooks to test QEMU
|
||||
or KVM.
|
||||
|
||||
**boards**
|
||||
|
||||
Example::
|
||||
|
||||
config SUN4M
|
||||
bool
|
||||
default y
|
||||
depends on SPARC && !SPARC64
|
||||
imply TCX
|
||||
imply CG3
|
||||
select CS4231
|
||||
select ECCMEMCTL
|
||||
select EMPTY_SLOT
|
||||
select ESCC
|
||||
select ESP
|
||||
select FDC
|
||||
select SLAVIO
|
||||
select LANCE
|
||||
select M48T59
|
||||
select STP2000
|
||||
|
||||
Boards specify their constituent devices using ``imply`` and ``select``
|
||||
directives. A device should be listed under ``select`` if the board
|
||||
cannot be started at all without it. It should be listed under
|
||||
``imply`` if (depending on the QEMU command line) the board may or
|
||||
may not be started without it. Boards default to true, but also
|
||||
have a ``depends on`` clause to limit them to the appropriate targets.
|
||||
For some targets, not all boards may be supported by hardware
|
||||
virtualization, in which case they also depend on the ``TCG`` symbol,
|
||||
Other symbols that are commonly used as dependencies for boards
|
||||
include libraries (such as ``FDT``) or ``TARGET_BIG_ENDIAN``
|
||||
(possibly negated).
|
||||
|
||||
Boards are listed for convenience in the ``configs/*.mak``
|
||||
for the target they apply to.
|
||||
|
||||
**internal elements**
|
||||
|
||||
Example::
|
||||
|
||||
config ECCMEMCTL
|
||||
bool
|
||||
select ECC
|
||||
|
||||
Internal elements group code that is useful in several boards or
|
||||
devices. They are usually enabled with ``select`` and in turn select
|
||||
other elements; they are never visible in ``configs/*.mak``
|
||||
files, and often not even in the Makefile.
|
||||
|
||||
Writing and modifying default configurations
|
||||
--------------------------------------------
|
||||
|
||||
In addition to the Kconfig files under hw/, each target also includes
|
||||
a file called ``configs/TARGETNAME-softmmu.mak``. These files
|
||||
initialize some Kconfig variables to non-default values and provide the
|
||||
starting point to turn on devices and subsystems.
|
||||
|
||||
A file in ``configs/`` looks like the following example::
|
||||
|
||||
# Default configuration for alpha-softmmu
|
||||
|
||||
# Uncomment the following lines to disable these optional devices:
|
||||
#
|
||||
#CONFIG_PCI_DEVICES=n
|
||||
#CONFIG_TEST_DEVICES=n
|
||||
|
||||
# Boards:
|
||||
#
|
||||
CONFIG_DP264=y
|
||||
|
||||
The first part, consisting of commented-out ``=n`` assignments, tells
|
||||
the user which devices or device groups are implied by the boards.
|
||||
The second part, consisting of ``=y`` assignments, tells the user which
|
||||
boards are supported by the target. The user will typically modify
|
||||
the default configuration by uncommenting lines in the first group,
|
||||
or commenting out lines in the second group.
|
||||
|
||||
It is also possible to run QEMU's configure script with the
|
||||
``--without-default-devices`` option. When this is done, everything defaults
|
||||
to ``n`` unless it is ``select``\ ed or explicitly switched on in the
|
||||
``.mak`` files. In other words, ``default`` and ``imply`` directives
|
||||
are disabled. When QEMU is built with this option, the user will probably
|
||||
want to change some lines in the first group, for example like this::
|
||||
|
||||
CONFIG_PCI_DEVICES=y
|
||||
#CONFIG_TEST_DEVICES=n
|
||||
|
||||
and/or pick a subset of the devices in those device groups. Without
|
||||
further modifications to ``configs/devices/``, a system emulator built
|
||||
without default devices might not do much more than start an empty
|
||||
machine, and even then only if ``--nodefaults`` is specified on the
|
||||
command line. Starting a VM *without* ``--nodefaults`` is allowed to
|
||||
fail, but should never abort. Failures in ``make check`` with
|
||||
``--without-default-devices`` are considered bugs in the test code:
|
||||
the tests should either use ``--nodefaults``, and should be skipped
|
||||
if a necessary device is not present in the build. Such failures
|
||||
should not be worked around with ``select`` directives.
|
||||
|
||||
Right now there is no single place that lists all the optional devices
|
||||
for ``CONFIG_PCI_DEVICES`` and ``CONFIG_TEST_DEVICES``. In the future,
|
||||
we expect that ``.mak`` files will be automatically generated, so that
|
||||
they will include all these symbols and some help text on what they do.
|
||||
|
||||
``Kconfig.host``
|
||||
----------------
|
||||
|
||||
In some special cases, a configurable element depends on host features
|
||||
that are detected by QEMU's configure or ``meson.build`` scripts; for
|
||||
example some devices depend on the availability of KVM or on the presence
|
||||
of a library on the host.
|
||||
|
||||
These symbols should be listed in ``Kconfig.host`` like this::
|
||||
|
||||
config TPM
|
||||
bool
|
||||
|
||||
and also listed as follows in the top-level meson.build's host_kconfig
|
||||
variable::
|
||||
|
||||
host_kconfig = \
|
||||
(have_tpm ? ['CONFIG_TPM=y'] : []) + \
|
||||
(host_os == 'linux' ? ['CONFIG_LINUX=y'] : []) + \
|
||||
(have_ivshmem ? ['CONFIG_IVSHMEM=y'] : []) + \
|
||||
...
|
||||
@@ -0,0 +1,541 @@
|
||||
..
|
||||
Copyright (c) 2017 Linaro Limited
|
||||
Written by Peter Maydell
|
||||
|
||||
===================
|
||||
Load and Store APIs
|
||||
===================
|
||||
|
||||
QEMU internally has multiple families of functions for performing
|
||||
loads and stores. This document attempts to enumerate them all
|
||||
and indicate when to use them. It does not provide detailed
|
||||
documentation of each API -- for that you should look at the
|
||||
documentation comments in the relevant header files.
|
||||
|
||||
|
||||
``ld*_p and st*_p``
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
These functions operate on a host pointer, and should be used
|
||||
when you already have a pointer into host memory (corresponding
|
||||
to guest ram or a local buffer). They deal with doing accesses
|
||||
with the desired endianness and with correctly handling
|
||||
potentially unaligned pointer values.
|
||||
|
||||
Function names follow the pattern:
|
||||
|
||||
load: ``ld{sign}{size}_{endian}_p(ptr)``
|
||||
|
||||
store: ``st{size}_{endian}_p(ptr, val)``
|
||||
|
||||
``sign``
|
||||
- (empty) : for 32 or 64 bit sizes
|
||||
- ``u`` : unsigned
|
||||
- ``s`` : signed
|
||||
|
||||
``size``
|
||||
- ``b`` : 8 bits
|
||||
- ``w`` : 16 bits
|
||||
- ``24`` : 24 bits
|
||||
- ``l`` : 32 bits
|
||||
- ``q`` : 64 bits
|
||||
|
||||
``endian``
|
||||
- ``he`` : host endian
|
||||
- ``be`` : big endian
|
||||
- ``le`` : little endian
|
||||
|
||||
The ``_{endian}`` infix is omitted for target-endian accesses.
|
||||
|
||||
The target endian accessors are only available to source
|
||||
files which are built per-target.
|
||||
|
||||
There are also functions which take the size as an argument:
|
||||
|
||||
load: ``ldn_{endian}_p(ptr, sz)``
|
||||
|
||||
which performs an unsigned load of ``sz`` bytes from ``ptr``
|
||||
as an ``{endian}`` order value and returns it in a uint64_t.
|
||||
|
||||
store: ``stn_{endian}_p(ptr, sz, val)``
|
||||
|
||||
which stores ``val`` to ``ptr`` as an ``{endian}`` order value
|
||||
of size ``sz`` bytes.
|
||||
|
||||
|
||||
Regexes for git grep:
|
||||
- ``\<ld[us]\?[bwlq]\(_[hbl]e\)\?_p\>``
|
||||
- ``\<st[bwlq]\(_[hbl]e\)\?_p\>``
|
||||
- ``\<st24\(_[hbl]e\)\?_p\>``
|
||||
- ``\<ldn\(_[hbl]e\)\?_p\>``
|
||||
- ``\<stn\(_[hbl]e\)\?_p\>``
|
||||
|
||||
``cpu_{ld,st}*_mmu``
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
These functions operate on a guest virtual address, plus a context
|
||||
known as a "mmu index" which controls how that virtual address is
|
||||
translated, plus a ``MemOp`` which contains alignment requirements
|
||||
among other things. The ``MemOp`` and mmu index are combined into
|
||||
a single argument of type ``MemOpIdx``.
|
||||
|
||||
The meaning of the indexes are target specific, but specifying a
|
||||
particular index might be necessary if, for instance, the helper
|
||||
requires a "always as non-privileged" access rather than the
|
||||
default access for the current state of the guest CPU.
|
||||
|
||||
These functions may cause a guest CPU exception to be taken
|
||||
(e.g. for an alignment fault or MMU fault) which will result in
|
||||
guest CPU state being updated and control longjmp'ing out of the
|
||||
function call. They should therefore only be used in code that is
|
||||
implementing emulation of the guest CPU.
|
||||
|
||||
The ``retaddr`` parameter is used to control unwinding of the
|
||||
guest CPU state in case of a guest CPU exception. This is passed
|
||||
to ``cpu_restore_state()``. Therefore the value should either be 0,
|
||||
to indicate that the guest CPU state is already synchronized, or
|
||||
the result of ``GETPC()`` from the top level ``HELPER(foo)``
|
||||
function, which is a return address into the generated code\ [#gpc]_.
|
||||
|
||||
.. [#gpc] Note that ``GETPC()`` should be used with great care: calling
|
||||
it in other functions that are *not* the top level
|
||||
``HELPER(foo)`` will cause unexpected behavior. Instead, the
|
||||
value of ``GETPC()`` should be read from the helper and passed
|
||||
if needed to the functions that the helper calls.
|
||||
|
||||
Function names follow the pattern:
|
||||
|
||||
load: ``cpu_ld{size}{end}_mmu(env, ptr, oi, retaddr)``
|
||||
|
||||
store: ``cpu_st{size}{end}_mmu(env, ptr, val, oi, retaddr)``
|
||||
|
||||
``size``
|
||||
- ``b`` : 8 bits
|
||||
- ``w`` : 16 bits
|
||||
- ``l`` : 32 bits
|
||||
- ``q`` : 64 bits
|
||||
|
||||
``end``
|
||||
- (empty) : for target endian, or 8 bit sizes
|
||||
- ``_be`` : big endian
|
||||
- ``_le`` : little endian
|
||||
|
||||
Regexes for git grep:
|
||||
- ``\<cpu_ld[bwlq]\(_[bl]e\)\?_mmu\>``
|
||||
- ``\<cpu_st[bwlq]\(_[bl]e\)\?_mmu\>``
|
||||
|
||||
|
||||
``cpu_{ld,st}*_mmuidx_ra``
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
These functions work like the ``cpu_{ld,st}_mmu`` functions except
|
||||
that the ``mmuidx`` parameter is not combined with a ``MemOp``,
|
||||
and therefore there is no required alignment supplied or enforced.
|
||||
|
||||
Function names follow the pattern:
|
||||
|
||||
load: ``cpu_ld{sign}{size}{end}_mmuidx_ra(env, ptr, mmuidx, retaddr)``
|
||||
|
||||
store: ``cpu_st{size}{end}_mmuidx_ra(env, ptr, val, mmuidx, retaddr)``
|
||||
|
||||
``sign``
|
||||
- (empty) : for 32 or 64 bit sizes
|
||||
- ``u`` : unsigned
|
||||
- ``s`` : signed
|
||||
|
||||
``size``
|
||||
- ``b`` : 8 bits
|
||||
- ``w`` : 16 bits
|
||||
- ``l`` : 32 bits
|
||||
- ``q`` : 64 bits
|
||||
|
||||
``end``
|
||||
- (empty) : for target endian, or 8 bit sizes
|
||||
- ``_be`` : big endian
|
||||
- ``_le`` : little endian
|
||||
|
||||
Regexes for git grep:
|
||||
- ``\<cpu_ld[us]\?[bwlq]\(_[bl]e\)\?_mmuidx_ra\>``
|
||||
- ``\<cpu_st[bwlq]\(_[bl]e\)\?_mmuidx_ra\>``
|
||||
|
||||
``cpu_{ld,st}*_data_ra``
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
These functions work like the ``cpu_{ld,st}_mmuidx_ra`` functions
|
||||
except that the ``mmuidx`` parameter is taken from the current mode
|
||||
of the guest CPU, as determined by ``cpu_mmu_index(env, false)``.
|
||||
|
||||
These are generally the preferred way to do accesses by guest
|
||||
virtual address from helper functions, unless the access should
|
||||
be performed with a context other than the default, or alignment
|
||||
should be enforced for the access.
|
||||
|
||||
Function names follow the pattern:
|
||||
|
||||
load: ``cpu_ld{sign}{size}{end}_data_ra(env, ptr, ra)``
|
||||
|
||||
store: ``cpu_st{size}{end}_data_ra(env, ptr, val, ra)``
|
||||
|
||||
``sign``
|
||||
- (empty) : for 32 or 64 bit sizes
|
||||
- ``u`` : unsigned
|
||||
- ``s`` : signed
|
||||
|
||||
``size``
|
||||
- ``b`` : 8 bits
|
||||
- ``w`` : 16 bits
|
||||
- ``l`` : 32 bits
|
||||
- ``q`` : 64 bits
|
||||
|
||||
``end``
|
||||
- (empty) : for target endian, or 8 bit sizes
|
||||
- ``_be`` : big endian
|
||||
- ``_le`` : little endian
|
||||
|
||||
Regexes for git grep:
|
||||
- ``\<cpu_ld[us]\?[bwlq]\(_[bl]e\)\?_data_ra\>``
|
||||
- ``\<cpu_st[bwlq]\(_[bl]e\)\?_data_ra\>``
|
||||
|
||||
``cpu_{ld,st}*_data``
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
These functions work like the ``cpu_{ld,st}_data_ra`` functions
|
||||
except that the ``retaddr`` parameter is 0, and thus does not
|
||||
unwind guest CPU state.
|
||||
|
||||
This means they must only be used from helper functions where the
|
||||
translator has saved all necessary CPU state. These functions are
|
||||
the right choice for calls made from hooks like the CPU ``do_interrupt``
|
||||
hook or when you know for certain that the translator had to save all
|
||||
the CPU state anyway.
|
||||
|
||||
Function names follow the pattern:
|
||||
|
||||
load: ``cpu_ld{sign}{size}{end}_data(env, ptr)``
|
||||
|
||||
store: ``cpu_st{size}{end}_data(env, ptr, val)``
|
||||
|
||||
``sign``
|
||||
- (empty) : for 32 or 64 bit sizes
|
||||
- ``u`` : unsigned
|
||||
- ``s`` : signed
|
||||
|
||||
``size``
|
||||
- ``b`` : 8 bits
|
||||
- ``w`` : 16 bits
|
||||
- ``l`` : 32 bits
|
||||
- ``q`` : 64 bits
|
||||
|
||||
``end``
|
||||
- (empty) : for target endian, or 8 bit sizes
|
||||
- ``_be`` : big endian
|
||||
- ``_le`` : little endian
|
||||
|
||||
Regexes for git grep:
|
||||
- ``\<cpu_ld[us]\?[bwlq]\(_[bl]e\)\?_data\>``
|
||||
- ``\<cpu_st[bwlq]\(_[bl]e\)\?_data\+\>``
|
||||
|
||||
``cpu_ld*_code_mmu``
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
These functions work like the ``cpu_{ld,st}*_mmu`` functions
|
||||
except that they perform a read for instruction execution.
|
||||
Any guest CPU exception that is raised will indicate an instruction
|
||||
execution fault rather than a data read fault.
|
||||
|
||||
In general these functions should not be used directly during translation.
|
||||
There are wrapper functions that are to be used which also take care of
|
||||
plugins for tracing.
|
||||
|
||||
Function names follow the pattern:
|
||||
|
||||
load: ``cpu_ld{sign}{size}_code_mmu(env, addr, oi, retaddr)``
|
||||
|
||||
``sign``
|
||||
- (empty) : for 32 or 64 bit sizes
|
||||
- ``u`` : unsigned
|
||||
- ``s`` : signed
|
||||
|
||||
``size``
|
||||
- ``b`` : 8 bits
|
||||
- ``w`` : 16 bits
|
||||
- ``l`` : 32 bits
|
||||
- ``q`` : 64 bits
|
||||
|
||||
Regexes for git grep:
|
||||
- ``\<cpu_ld[us]\?[bwlq]_code_mmu\>``
|
||||
|
||||
``translator_ld*``
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
These functions are a wrapper for ``cpu_ld*_code_mmu`` which also perform
|
||||
any actions required by any tracing plugins. They are only to be
|
||||
called during the translator callback ``translate_insn``.
|
||||
|
||||
There is a set of functions ending in ``_swap`` which, if the parameter
|
||||
is true, returns the value in the endianness that is the reverse of
|
||||
the guest native endianness, as determined by ``TARGET_BIG_ENDIAN``.
|
||||
|
||||
Function names follow the pattern:
|
||||
|
||||
load: ``translator_ld{sign}{size}(env, ptr)``
|
||||
|
||||
swap: ``translator_ld{sign}{size}_swap(env, ptr, swap)``
|
||||
|
||||
``sign``
|
||||
- (empty) : for 32 or 64 bit sizes
|
||||
- ``u`` : unsigned
|
||||
- ``s`` : signed
|
||||
|
||||
``size``
|
||||
- ``b`` : 8 bits
|
||||
- ``w`` : 16 bits
|
||||
- ``l`` : 32 bits
|
||||
- ``q`` : 64 bits
|
||||
|
||||
Regexes for git grep:
|
||||
- ``\<translator_ld[us]\?[bwlq]\(_swap\)\?\>``
|
||||
|
||||
``helper_{ld,st}*_mmu``
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
These functions are intended primarily to be called by the code
|
||||
generated by the TCG backend. Like the ``cpu_{ld,st}_mmu`` functions
|
||||
they perform accesses by guest virtual address, with a given ``MemOpIdx``.
|
||||
|
||||
They differ from ``cpu_{ld,st}_mmu`` in that they take the endianness
|
||||
of the operation only from the MemOpIdx, and loads extend the return
|
||||
value to the size of a host general register (``tcg_target_ulong``).
|
||||
|
||||
load: ``helper_ld{sign}{size}_mmu(env, addr, opindex, retaddr)``
|
||||
|
||||
store: ``helper_{size}_mmu(env, addr, val, opindex, retaddr)``
|
||||
|
||||
``sign``
|
||||
- (empty) : for 32 or 64 bit sizes
|
||||
- ``u`` : unsigned
|
||||
- ``s`` : signed
|
||||
|
||||
``size``
|
||||
- ``b`` : 8 bits
|
||||
- ``w`` : 16 bits
|
||||
- ``l`` : 32 bits
|
||||
- ``q`` : 64 bits
|
||||
|
||||
Regexes for git grep:
|
||||
- ``\<helper_ld[us]\?[bwlq]_mmu\>``
|
||||
- ``\<helper_st[bwlq]_mmu\>``
|
||||
|
||||
``address_space_*``
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
These functions are the primary ones to use when emulating CPU
|
||||
or device memory accesses. They take an AddressSpace, which is the
|
||||
way QEMU defines the view of memory that a device or CPU has.
|
||||
(They generally correspond to being the "master" end of a hardware bus
|
||||
or bus fabric.)
|
||||
|
||||
Each CPU has an AddressSpace. Some kinds of CPU have more than
|
||||
one AddressSpace (for instance Arm guest CPUs have an AddressSpace
|
||||
for the Secure world and one for NonSecure if they implement TrustZone).
|
||||
Devices which can do DMA-type operations should generally have an
|
||||
AddressSpace. There is also a "system address space" which typically
|
||||
has all the devices and memory that all CPUs can see. (Some older
|
||||
device models use the "system address space" rather than properly
|
||||
modelling that they have an AddressSpace of their own.)
|
||||
|
||||
Functions are provided for doing byte-buffer reads and writes,
|
||||
and also for doing one-data-item loads and stores.
|
||||
|
||||
In all cases the caller provides a MemTxAttrs to specify bus
|
||||
transaction attributes, and can check whether the memory transaction
|
||||
succeeded using a MemTxResult return code.
|
||||
|
||||
``address_space_read(address_space, addr, attrs, buf, len)``
|
||||
|
||||
``address_space_write(address_space, addr, attrs, buf, len)``
|
||||
|
||||
``address_space_rw(address_space, addr, attrs, buf, len, is_write)``
|
||||
|
||||
``address_space_ld{sign}{size}_{endian}(address_space, addr, attrs, txresult)``
|
||||
|
||||
``address_space_st{size}_{endian}(address_space, addr, val, attrs, txresult)``
|
||||
|
||||
``sign``
|
||||
- (empty) : for 32 or 64 bit sizes
|
||||
- ``u`` : unsigned
|
||||
|
||||
(No signed load operations are provided.)
|
||||
|
||||
``size``
|
||||
- ``b`` : 8 bits
|
||||
- ``w`` : 16 bits
|
||||
- ``l`` : 32 bits
|
||||
- ``q`` : 64 bits
|
||||
|
||||
``endian``
|
||||
- ``le`` : little endian
|
||||
- ``be`` : big endian
|
||||
|
||||
The ``_{endian}`` suffix is omitted for byte accesses.
|
||||
|
||||
Regexes for git grep:
|
||||
- ``\<address_space_\(read\|write\|rw\)\>``
|
||||
- ``\<address_space_ldu\?[bwql]\(_[lb]e\)\?\>``
|
||||
- ``\<address_space_st[bwql]\(_[lb]e\)\?\>``
|
||||
|
||||
``address_space_write_rom``
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
This function performs a write by physical address like
|
||||
``address_space_write``, except that if the write is to a ROM then
|
||||
the ROM contents will be modified, even though a write by the guest
|
||||
CPU to the ROM would be ignored. This is used for non-guest writes
|
||||
like writes from the gdb debug stub or initial loading of ROM contents.
|
||||
|
||||
Note that portions of the write which attempt to write data to a
|
||||
device will be silently ignored -- only real RAM and ROM will
|
||||
be written to.
|
||||
|
||||
Regexes for git grep:
|
||||
- ``address_space_write_rom``
|
||||
|
||||
``{ld,st}*_phys``
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
These are functions which are identical to
|
||||
``address_space_{ld,st}*``, except that they always pass
|
||||
``MEMTXATTRS_UNSPECIFIED`` for the transaction attributes, and ignore
|
||||
whether the transaction succeeded or failed.
|
||||
|
||||
The fact that they ignore whether the transaction succeeded means
|
||||
they should not be used in new code, unless you know for certain
|
||||
that your code will only be used in a context where the CPU or
|
||||
device doing the access has no way to report such an error.
|
||||
|
||||
``load: ld{sign}{size}_{endian}_phys``
|
||||
|
||||
``store: st{size}_{endian}_phys``
|
||||
|
||||
``sign``
|
||||
- (empty) : for 32 or 64 bit sizes
|
||||
- ``u`` : unsigned
|
||||
|
||||
(No signed load operations are provided.)
|
||||
|
||||
``size``
|
||||
- ``b`` : 8 bits
|
||||
- ``w`` : 16 bits
|
||||
- ``l`` : 32 bits
|
||||
- ``q`` : 64 bits
|
||||
|
||||
``endian``
|
||||
- ``le`` : little endian
|
||||
- ``be`` : big endian
|
||||
|
||||
The ``_{endian}_`` infix is omitted for byte accesses.
|
||||
|
||||
Regexes for git grep:
|
||||
- ``\<ldu\?[bwlq]\(_[bl]e\)\?_phys\>``
|
||||
- ``\<st[bwlq]\(_[bl]e\)\?_phys\>``
|
||||
|
||||
``physical_memory_*``
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
These are convenience functions which are identical to
|
||||
``address_space_*`` but operate specifically on the legacy global
|
||||
``&address_space_memory`` address space (which might not be used by all
|
||||
machines), always pass a ``MEMTXATTRS_UNSPECIFIED`` set of memory attributes
|
||||
and ignore whether the memory transaction succeeded or failed. Expected
|
||||
users are hardware device models. For new code they are better avoided:
|
||||
|
||||
* there is likely to be behaviour you need to model correctly for a
|
||||
failed read or write operation
|
||||
* a device should usually perform operations on its own AddressSpace
|
||||
rather than using the system address space
|
||||
* some machines do not use this global address space at all
|
||||
|
||||
``physical_memory_read``
|
||||
|
||||
``physical_memory_write``
|
||||
|
||||
Regexes for git grep:
|
||||
- ``\<physical_memory_\(read\|write\)\>``
|
||||
|
||||
``cpu_memory_rw_debug``
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Access CPU memory by virtual address for debug purposes.
|
||||
|
||||
This function is intended for use by the GDB stub and similar code.
|
||||
It takes a virtual address, converts it to a physical address via
|
||||
an MMU lookup using the current settings of the specified CPU,
|
||||
and then performs the access (using ``address_space_rw`` for
|
||||
reads or ``address_space_write_rom`` for writes).
|
||||
This means that if the access is a write to a ROM then this
|
||||
function will modify the contents (whereas a normal guest CPU access
|
||||
would ignore the write attempt).
|
||||
|
||||
``cpu_memory_rw_debug``
|
||||
|
||||
``dma_memory_*``
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
These behave like ``address_space_*``, except that they perform a DMA
|
||||
barrier operation first.
|
||||
|
||||
**TODO**: We should provide guidance on when you need the DMA
|
||||
barrier operation and when it's OK to use ``address_space_*``, and
|
||||
make sure our existing code is doing things correctly.
|
||||
|
||||
``dma_memory_read``
|
||||
|
||||
``dma_memory_write``
|
||||
|
||||
``dma_memory_rw``
|
||||
|
||||
Regexes for git grep:
|
||||
- ``\<dma_memory_\(read\|write\|rw\)\>``
|
||||
- ``\<ldu\?[bwlq]\(_[bl]e\)\?_dma\>``
|
||||
- ``\<st[bwlq]\(_[bl]e\)\?_dma\>``
|
||||
|
||||
``pci_dma_*`` and ``{ld,st}*_pci_dma``
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
These functions are specifically for PCI device models which need to
|
||||
perform accesses where the PCI device is a bus master. You pass them a
|
||||
``PCIDevice *`` and they will do ``dma_memory_*`` operations on the
|
||||
correct address space for that device.
|
||||
|
||||
``pci_dma_read``
|
||||
|
||||
``pci_dma_write``
|
||||
|
||||
``pci_dma_rw``
|
||||
|
||||
``load: ld{sign}{size}_{endian}_pci_dma``
|
||||
|
||||
``store: st{size}_{endian}_pci_dma``
|
||||
|
||||
``sign``
|
||||
- (empty) : for 32 or 64 bit sizes
|
||||
- ``u`` : unsigned
|
||||
|
||||
(No signed load operations are provided.)
|
||||
|
||||
``size``
|
||||
- ``b`` : 8 bits
|
||||
- ``w`` : 16 bits
|
||||
- ``l`` : 32 bits
|
||||
- ``q`` : 64 bits
|
||||
|
||||
``endian``
|
||||
- ``le`` : little endian
|
||||
- ``be`` : big endian
|
||||
|
||||
The ``_{endian}_`` infix is omitted for byte accesses.
|
||||
|
||||
Regexes for git grep:
|
||||
- ``\<pci_dma_\(read\|write\|rw\)\>``
|
||||
- ``\<ldu\?[bwlq]\(_[bl]e\)\?_pci_dma\>``
|
||||
- ``\<st[bwlq]\(_[bl]e\)\?_pci_dma\>``
|
||||
@@ -0,0 +1,278 @@
|
||||
Locked Counters (aka ``QemuLockCnt``)
|
||||
=====================================
|
||||
|
||||
QEMU often uses reference counts to track data structures that are being
|
||||
accessed and should not be freed. For example, a loop that invoke
|
||||
callbacks like this is not safe::
|
||||
|
||||
QLIST_FOREACH_SAFE(ioh, &io_handlers, next, pioh) {
|
||||
if (ioh->revents & G_IO_OUT) {
|
||||
ioh->fd_write(ioh->opaque);
|
||||
}
|
||||
}
|
||||
|
||||
``QLIST_FOREACH_SAFE`` protects against deletion of the current node (``ioh``)
|
||||
by stashing away its ``next`` pointer. However, ``ioh->fd_write`` could
|
||||
actually delete the next node from the list. The simplest way to
|
||||
avoid this is to mark the node as deleted, and remove it from the
|
||||
list in the above loop::
|
||||
|
||||
QLIST_FOREACH_SAFE(ioh, &io_handlers, next, pioh) {
|
||||
if (ioh->deleted) {
|
||||
QLIST_REMOVE(ioh, next);
|
||||
g_free(ioh);
|
||||
} else {
|
||||
if (ioh->revents & G_IO_OUT) {
|
||||
ioh->fd_write(ioh->opaque);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
If however this loop must also be reentrant, i.e. it is possible that
|
||||
``ioh->fd_write`` invokes the loop again, some kind of counting is needed::
|
||||
|
||||
walking_handlers++;
|
||||
QLIST_FOREACH_SAFE(ioh, &io_handlers, next, pioh) {
|
||||
if (ioh->deleted) {
|
||||
if (walking_handlers == 1) {
|
||||
QLIST_REMOVE(ioh, next);
|
||||
g_free(ioh);
|
||||
}
|
||||
} else {
|
||||
if (ioh->revents & G_IO_OUT) {
|
||||
ioh->fd_write(ioh->opaque);
|
||||
}
|
||||
}
|
||||
}
|
||||
walking_handlers--;
|
||||
|
||||
One may think of using the RCU primitives, ``rcu_read_lock()`` and
|
||||
``rcu_read_unlock()``; effectively, the RCU nesting count would take
|
||||
the place of the walking_handlers global variable. Indeed,
|
||||
reference counting and RCU have similar purposes, but their usage in
|
||||
general is complementary:
|
||||
|
||||
- reference counting is fine-grained and limited to a single data
|
||||
structure; RCU delays reclamation of *all* RCU-protected data
|
||||
structures;
|
||||
|
||||
- reference counting works even in the presence of code that keeps
|
||||
a reference for a long time; RCU critical sections in principle
|
||||
should be kept short;
|
||||
|
||||
- reference counting is often applied to code that is not thread-safe
|
||||
but is reentrant; in fact, usage of reference counting in QEMU predates
|
||||
the introduction of threads by many years. RCU is generally used to
|
||||
protect readers from other threads freeing memory after concurrent
|
||||
modifications to a data structure.
|
||||
|
||||
- reclaiming data can be done by a separate thread in the case of RCU;
|
||||
this can improve performance, but also delay reclamation undesirably.
|
||||
With reference counting, reclamation is deterministic.
|
||||
|
||||
This file documents ``QemuLockCnt``, an abstraction for using reference
|
||||
counting in code that has to be both thread-safe and reentrant.
|
||||
|
||||
|
||||
``QemuLockCnt`` concepts
|
||||
------------------------
|
||||
|
||||
A ``QemuLockCnt`` comprises both a counter and a mutex; it has primitives
|
||||
to increment and decrement the counter, and to take and release the
|
||||
mutex. The counter notes how many visits to the data structures are
|
||||
taking place (the visits could be from different threads, or there could
|
||||
be multiple reentrant visits from the same thread). The basic rules
|
||||
governing the counter/mutex pair then are the following:
|
||||
|
||||
- Data protected by the QemuLockCnt must not be freed unless the
|
||||
counter is zero and the mutex is taken.
|
||||
|
||||
- A new visit cannot be started while the counter is zero and the
|
||||
mutex is taken.
|
||||
|
||||
Most of the time, the mutex protects all writes to the data structure,
|
||||
not just frees, though there could be cases where this is not necessary.
|
||||
|
||||
Reads, instead, can be done without taking the mutex, as long as the
|
||||
readers and writers use the same macros that are used for RCU, for
|
||||
example ``qatomic_rcu_read``, ``qatomic_rcu_set``, ``QLIST_FOREACH_RCU``,
|
||||
etc. This is because the reads are done outside a lock and a set
|
||||
or ``QLIST_INSERT_HEAD``
|
||||
can happen concurrently with the read. The RCU API ensures that the
|
||||
processor and the compiler see all required memory barriers.
|
||||
|
||||
This could be implemented simply by protecting the counter with the
|
||||
mutex, for example::
|
||||
|
||||
// (1)
|
||||
qemu_mutex_lock(&walking_handlers_mutex);
|
||||
walking_handlers++;
|
||||
qemu_mutex_unlock(&walking_handlers_mutex);
|
||||
|
||||
...
|
||||
|
||||
// (2)
|
||||
qemu_mutex_lock(&walking_handlers_mutex);
|
||||
if (--walking_handlers == 0) {
|
||||
QLIST_FOREACH_SAFE(ioh, &io_handlers, next, pioh) {
|
||||
if (ioh->deleted) {
|
||||
QLIST_REMOVE(ioh, next);
|
||||
g_free(ioh);
|
||||
}
|
||||
}
|
||||
}
|
||||
qemu_mutex_unlock(&walking_handlers_mutex);
|
||||
|
||||
Here, no frees can happen in the code represented by the ellipsis.
|
||||
If another thread is executing critical section (2), that part of
|
||||
the code cannot be entered, because the thread will not be able
|
||||
to increment the ``walking_handlers`` variable. And of course
|
||||
during the visit any other thread will see a nonzero value for
|
||||
``walking_handlers``, as in the single-threaded code.
|
||||
|
||||
Note that it is possible for multiple concurrent accesses to delay
|
||||
the cleanup arbitrarily; in other words, for the ``walking_handlers``
|
||||
counter to never become zero. For this reason, this technique is
|
||||
more easily applicable if concurrent access to the structure is rare.
|
||||
|
||||
However, critical sections are easy to forget since you have to do
|
||||
them for each modification of the counter. ``QemuLockCnt`` ensures that
|
||||
all modifications of the counter take the lock appropriately, and it
|
||||
can also be more efficient in two ways:
|
||||
|
||||
- it avoids taking the lock for many operations (for example
|
||||
incrementing the counter while it is non-zero);
|
||||
|
||||
- on some platforms, one can implement ``QemuLockCnt`` to hold the lock
|
||||
and the mutex in a single word, making the fast path no more expensive
|
||||
than simply managing a counter using atomic operations (see
|
||||
:doc:`atomics`). This can be very helpful if concurrent access to
|
||||
the data structure is expected to be rare.
|
||||
|
||||
|
||||
Using the same mutex for frees and writes can still incur some small
|
||||
inefficiencies; for example, a visit can never start if the counter is
|
||||
zero and the mutex is taken -- even if the mutex is taken by a write,
|
||||
which in principle need not block a visit of the data structure.
|
||||
However, these are usually not a problem if any of the following
|
||||
assumptions are valid:
|
||||
|
||||
- concurrent access is possible but rare
|
||||
|
||||
- writes are rare
|
||||
|
||||
- writes are frequent, but this kind of write (e.g. appending to a
|
||||
list) has a very small critical section.
|
||||
|
||||
For example, QEMU uses ``QemuLockCnt`` to manage an ``AioContext``'s list of
|
||||
bottom halves and file descriptor handlers. Modifications to the list
|
||||
of file descriptor handlers are rare. Creation of a new bottom half is
|
||||
frequent and can happen on a fast path; however: 1) it is almost never
|
||||
concurrent with a visit to the list of bottom halves; 2) it only has
|
||||
three instructions in the critical path, two assignments and a ``smp_wmb()``.
|
||||
|
||||
|
||||
``QemuLockCnt`` API
|
||||
-------------------
|
||||
|
||||
.. kernel-doc:: include/qemu/lockcnt.h
|
||||
|
||||
|
||||
``QemuLockCnt`` usage
|
||||
---------------------
|
||||
|
||||
This section explains the typical usage patterns for ``QemuLockCnt`` functions.
|
||||
|
||||
Setting a variable to a non-NULL value can be done between
|
||||
``qemu_lockcnt_lock`` and ``qemu_lockcnt_unlock``::
|
||||
|
||||
qemu_lockcnt_lock(&xyz_lockcnt);
|
||||
if (!xyz) {
|
||||
new_xyz = g_new(XYZ, 1);
|
||||
...
|
||||
qatomic_rcu_set(&xyz, new_xyz);
|
||||
}
|
||||
qemu_lockcnt_unlock(&xyz_lockcnt);
|
||||
|
||||
Accessing the value can be done between ``qemu_lockcnt_inc`` and
|
||||
``qemu_lockcnt_dec``::
|
||||
|
||||
qemu_lockcnt_inc(&xyz_lockcnt);
|
||||
if (xyz) {
|
||||
XYZ *p = qatomic_rcu_read(&xyz);
|
||||
...
|
||||
/* Accesses can now be done through "p". */
|
||||
}
|
||||
qemu_lockcnt_dec(&xyz_lockcnt);
|
||||
|
||||
Freeing the object can similarly use ``qemu_lockcnt_lock`` and
|
||||
``qemu_lockcnt_unlock``, but you also need to ensure that the count
|
||||
is zero (i.e. there is no concurrent visit). Because ``qemu_lockcnt_inc``
|
||||
takes the ``QemuLockCnt``'s lock, the count cannot become non-zero while
|
||||
the object is being freed. Freeing an object looks like this::
|
||||
|
||||
qemu_lockcnt_lock(&xyz_lockcnt);
|
||||
if (!qemu_lockcnt_count(&xyz_lockcnt)) {
|
||||
g_free(xyz);
|
||||
xyz = NULL;
|
||||
}
|
||||
qemu_lockcnt_unlock(&xyz_lockcnt);
|
||||
|
||||
If an object has to be freed right after a visit, you can combine
|
||||
the decrement, the locking and the check on count as follows::
|
||||
|
||||
qemu_lockcnt_inc(&xyz_lockcnt);
|
||||
if (xyz) {
|
||||
XYZ *p = qatomic_rcu_read(&xyz);
|
||||
...
|
||||
/* Accesses can now be done through "p". */
|
||||
}
|
||||
if (qemu_lockcnt_dec_and_lock(&xyz_lockcnt)) {
|
||||
g_free(xyz);
|
||||
xyz = NULL;
|
||||
qemu_lockcnt_unlock(&xyz_lockcnt);
|
||||
}
|
||||
|
||||
``QemuLockCnt`` can also be used to access a list as follows::
|
||||
|
||||
qemu_lockcnt_inc(&io_handlers_lockcnt);
|
||||
QLIST_FOREACH_RCU(ioh, &io_handlers, pioh) {
|
||||
if (ioh->revents & G_IO_OUT) {
|
||||
ioh->fd_write(ioh->opaque);
|
||||
}
|
||||
}
|
||||
|
||||
if (qemu_lockcnt_dec_and_lock(&io_handlers_lockcnt)) {
|
||||
QLIST_FOREACH_SAFE(ioh, &io_handlers, next, pioh) {
|
||||
if (ioh->deleted) {
|
||||
QLIST_REMOVE(ioh, next);
|
||||
g_free(ioh);
|
||||
}
|
||||
}
|
||||
qemu_lockcnt_unlock(&io_handlers_lockcnt);
|
||||
}
|
||||
|
||||
Again, the RCU primitives are used because new items can be added to the
|
||||
list during the walk. ``QLIST_FOREACH_RCU`` ensures that the processor and
|
||||
the compiler see the appropriate memory barriers.
|
||||
|
||||
An alternative pattern uses ``qemu_lockcnt_dec_if_lock``::
|
||||
|
||||
qemu_lockcnt_inc(&io_handlers_lockcnt);
|
||||
QLIST_FOREACH_SAFE_RCU(ioh, &io_handlers, next, pioh) {
|
||||
if (ioh->deleted) {
|
||||
if (qemu_lockcnt_dec_if_lock(&io_handlers_lockcnt)) {
|
||||
QLIST_REMOVE(ioh, next);
|
||||
g_free(ioh);
|
||||
qemu_lockcnt_inc_and_unlock(&io_handlers_lockcnt);
|
||||
}
|
||||
} else {
|
||||
if (ioh->revents & G_IO_OUT) {
|
||||
ioh->fd_write(ioh->opaque);
|
||||
}
|
||||
}
|
||||
}
|
||||
qemu_lockcnt_dec(&io_handlers_lockcnt);
|
||||
|
||||
Here you can use ``qemu_lockcnt_dec`` instead of ``qemu_lockcnt_dec_and_lock``,
|
||||
because there is no special task to do if the count goes from 1 to 0.
|
||||
@@ -0,0 +1,182 @@
|
||||
================================
|
||||
LUKS volume with detached header
|
||||
================================
|
||||
|
||||
Introduction
|
||||
============
|
||||
|
||||
This document gives an overview of the design of LUKS volume with detached
|
||||
header and how to use it.
|
||||
|
||||
Background
|
||||
==========
|
||||
|
||||
The LUKS format has ability to store the header in a separate volume from
|
||||
the payload. We could extend the LUKS driver in QEMU to support this use
|
||||
case.
|
||||
|
||||
Normally a LUKS volume has a layout:
|
||||
|
||||
::
|
||||
|
||||
+-----------------------------------------------+
|
||||
| | | |
|
||||
disk | header | key material | disk payload data |
|
||||
| | | |
|
||||
+-----------------------------------------------+
|
||||
|
||||
With a detached LUKS header, you need 2 disks so getting:
|
||||
|
||||
::
|
||||
|
||||
+--------------------------+
|
||||
disk1 | header | key material |
|
||||
+--------------------------+
|
||||
+---------------------+
|
||||
disk2 | disk payload data |
|
||||
+---------------------+
|
||||
|
||||
There are a variety of benefits to doing this:
|
||||
|
||||
* Secrecy - the disk2 cannot be identified as containing LUKS
|
||||
volume since there's no header
|
||||
* Control - if access to the disk1 is restricted, then even
|
||||
if someone has access to disk2 they can't unlock
|
||||
it. Might be useful if you have disks on NFS but
|
||||
want to restrict which host can launch a VM
|
||||
instance from it, by dynamically providing access
|
||||
to the header to a designated host
|
||||
* Flexibility - your application data volume may be a given
|
||||
size and it is inconvenient to resize it to
|
||||
add encryption.You can store the LUKS header
|
||||
separately and use the existing storage
|
||||
volume for payload
|
||||
* Recovery - corruption of a bit in the header may make the
|
||||
entire payload inaccessible. It might be
|
||||
convenient to take backups of the header. If
|
||||
your primary disk header becomes corrupt, you
|
||||
can unlock the data still by pointing to the
|
||||
backup detached header
|
||||
|
||||
Architecture
|
||||
============
|
||||
|
||||
Take the qcow2 encryption, for example. The architecture of the
|
||||
LUKS volume with detached header is shown in the diagram below.
|
||||
|
||||
There are two children of the root node: a file and a header.
|
||||
Data from the disk payload is stored in the file node. The
|
||||
LUKS header and key material are located in the header node,
|
||||
as previously mentioned.
|
||||
|
||||
::
|
||||
|
||||
+-----------------------------+
|
||||
Root node | foo[luks] |
|
||||
+-----------------------------+
|
||||
| |
|
||||
file | header |
|
||||
| |
|
||||
+---------------------+ +------------------+
|
||||
Child node |payload-format[qcow2]| |header-format[raw]|
|
||||
+---------------------+ +------------------+
|
||||
| |
|
||||
file | file |
|
||||
| |
|
||||
+----------------------+ +---------------------+
|
||||
Child node |payload-protocol[file]| |header-protocol[file]|
|
||||
+----------------------+ +---------------------+
|
||||
| |
|
||||
| |
|
||||
| |
|
||||
Host storage Host storage
|
||||
|
||||
Usage
|
||||
=====
|
||||
|
||||
Create a LUKS disk with a detached header using qemu-img
|
||||
--------------------------------------------------------
|
||||
|
||||
Shell commandline::
|
||||
|
||||
# qemu-img create --object secret,id=sec0,data=abc123 -f luks \
|
||||
-o cipher-alg=aes-256,cipher-mode=xts -o key-secret=sec0 \
|
||||
-o detached-header=true test-header.img
|
||||
# qemu-img create -f qcow2 test-payload.qcow2 200G
|
||||
# qemu-img info 'json:{"driver":"luks","file":{"filename": \
|
||||
"test-payload.img"},"header":{"filename":"test-header.img"}}'
|
||||
|
||||
Set up a VM's LUKS volume with a detached header
|
||||
------------------------------------------------
|
||||
|
||||
Qemu commandline::
|
||||
|
||||
# qemu-system-x86_64 ... \
|
||||
-object '{"qom-type":"secret","id":"libvirt-3-format-secret", \
|
||||
"data":"abc123"}' \
|
||||
-blockdev '{"driver":"file","filename":"/path/to/test-header.img", \
|
||||
"node-name":"libvirt-1-storage"}' \
|
||||
-blockdev '{"node-name":"libvirt-1-format","read-only":false, \
|
||||
"driver":"raw","file":"libvirt-1-storage"}' \
|
||||
-blockdev '{"driver":"file","filename":"/path/to/test-payload.qcow2", \
|
||||
"node-name":"libvirt-2-storage"}' \
|
||||
-blockdev '{"node-name":"libvirt-2-format","read-only":false, \
|
||||
"driver":"qcow2","file":"libvirt-2-storage"}' \
|
||||
-blockdev '{"node-name":"libvirt-3-format","driver":"luks", \
|
||||
"file":"libvirt-2-format","header":"libvirt-1-format","key-secret": \
|
||||
"libvirt-3-format-secret"}' \
|
||||
-device '{"driver":"virtio-blk-pci","bus":XXX,"addr":YYY,"drive": \
|
||||
"libvirt-3-format","id":"virtio-disk1"}'
|
||||
|
||||
Add LUKS volume to a VM with a detached header
|
||||
----------------------------------------------
|
||||
|
||||
1. object-add the secret for decrypting the cipher stored in
|
||||
LUKS header above::
|
||||
|
||||
# virsh qemu-monitor-command vm '{"execute":"object-add", \
|
||||
"arguments":{"qom-type":"secret", "id": \
|
||||
"libvirt-4-format-secret", "data":"abc123"}}'
|
||||
|
||||
2. block-add the protocol node for LUKS header::
|
||||
|
||||
# virsh qemu-monitor-command vm '{"execute":"blockdev-add", \
|
||||
"arguments":{"node-name":"libvirt-1-storage", "driver":"file", \
|
||||
"filename": "/path/to/test-header.img" }}'
|
||||
|
||||
3. block-add the raw-drived node for LUKS header::
|
||||
|
||||
# virsh qemu-monitor-command vm '{"execute":"blockdev-add", \
|
||||
"arguments":{"node-name":"libvirt-1-format", "driver":"raw", \
|
||||
"file":"libvirt-1-storage"}}'
|
||||
|
||||
4. block-add the protocol node for disk payload image::
|
||||
|
||||
# virsh qemu-monitor-command vm '{"execute":"blockdev-add", \
|
||||
"arguments":{"node-name":"libvirt-2-storage", "driver":"file", \
|
||||
"filename":"/path/to/test-payload.qcow2"}}'
|
||||
|
||||
5. block-add the qcow2-drived format node for disk payload data::
|
||||
|
||||
# virsh qemu-monitor-command vm '{"execute":"blockdev-add", \
|
||||
"arguments":{"node-name":"libvirt-2-format", "driver":"qcow2", \
|
||||
"file":"libvirt-2-storage"}}'
|
||||
|
||||
6. block-add the luks-drived format node to link the qcow2 disk
|
||||
with the LUKS header by specifying the field "header"::
|
||||
|
||||
# virsh qemu-monitor-command vm '{"execute":"blockdev-add", \
|
||||
"arguments":{"node-name":"libvirt-3-format", "driver":"luks", \
|
||||
"file":"libvirt-2-format", "header":"libvirt-1-format", \
|
||||
"key-secret":"libvirt-2-format-secret"}}'
|
||||
|
||||
7. hot-plug the virtio-blk device finally::
|
||||
|
||||
# virsh qemu-monitor-command vm '{"execute":"device_add", \
|
||||
"arguments": {"driver":"virtio-blk-pci", \
|
||||
"drive": "libvirt-3-format", "id":"virtio-disk2"}}
|
||||
|
||||
TODO
|
||||
====
|
||||
|
||||
1. Support the shared detached LUKS header within the VM.
|
||||
@@ -0,0 +1,107 @@
|
||||
.. _maintainers:
|
||||
|
||||
The Role of Maintainers
|
||||
=======================
|
||||
|
||||
Maintainers are a critical part of the project's contributor ecosystem.
|
||||
They come from a wide range of backgrounds from unpaid hobbyists
|
||||
working in their spare time to employees who work on the project as
|
||||
part of their job. Maintainer activities include:
|
||||
|
||||
- reviewing patches and suggesting changes
|
||||
- collecting patches and preparing pull requests
|
||||
- tending to the long term health of their area
|
||||
- participating in other project activities
|
||||
|
||||
They are also human and subject to the same pressures as everyone else
|
||||
including overload and burnout. Like everyone else they are subject
|
||||
to project's :ref:`code_of_conduct` and should also be exemplars of
|
||||
excellent community collaborators.
|
||||
|
||||
The MAINTAINERS file
|
||||
--------------------
|
||||
|
||||
The `MAINTAINERS
|
||||
<https://gitlab.com/qemu-project/qemu/-/blob/master/MAINTAINERS>`__
|
||||
file contains the canonical list of who is a maintainer. The file
|
||||
is machine readable so an appropriately configured git (see
|
||||
:ref:`cc_the_relevant_maintainer`) can automatically Cc them on
|
||||
patches that touch their area of code.
|
||||
|
||||
The file also describes the status of the area of code to give an idea
|
||||
of how actively that section is maintained.
|
||||
|
||||
.. list-table:: Meaning of support status in MAINTAINERS
|
||||
:widths: 25 75
|
||||
:header-rows: 1
|
||||
|
||||
* - Status
|
||||
- Meaning
|
||||
* - Supported
|
||||
- Someone is actually paid to look after this.
|
||||
* - Maintained
|
||||
- Someone actually looks after it.
|
||||
* - Odd Fixes
|
||||
- It has a maintainer but they don't have time to do
|
||||
much other than throw the odd patch in.
|
||||
* - Orphan
|
||||
- No current maintainer.
|
||||
* - Obsolete
|
||||
- Old obsolete code, should use something else.
|
||||
|
||||
Please bear in mind that even if someone is paid to support something
|
||||
it does not mean they are paid to support you. This is open source and
|
||||
the code comes with no warranty and the project makes no guarantees
|
||||
about dealing with bugs or features requests.
|
||||
|
||||
|
||||
|
||||
Becoming a reviewer
|
||||
-------------------
|
||||
|
||||
Most maintainers start by becoming subsystem reviewers. While anyone
|
||||
is welcome to review code on the mailing list getting added to the
|
||||
MAINTAINERS file with a line like::
|
||||
|
||||
R: Random Hacker <[email protected]>
|
||||
|
||||
marks you as a 'designated reviewer' - expected to provide regular
|
||||
spontaneous feedback. This will ensure that patches touching a given
|
||||
subsystem will automatically be CC'd to you.
|
||||
|
||||
Becoming a maintainer
|
||||
---------------------
|
||||
|
||||
Maintainers are volunteers who put themselves forward or have been
|
||||
asked by others to keep an eye on an area of code. They have generally
|
||||
demonstrated to the community, usually via contributions and code
|
||||
reviews, that they have a good understanding of the subsystem. They
|
||||
are also trusted to make a positive contribution to the project and
|
||||
work well with the other contributors.
|
||||
|
||||
The process is simple - simply send a patch to the list that updates
|
||||
the ``MAINTAINERS`` file. Sometimes this is done as part of a larger
|
||||
series when a new sub-system is being added to the code base. This can
|
||||
also be done by a retiring maintainer who nominates their replacement
|
||||
after discussion with other contributors.
|
||||
|
||||
Once the patch is reviewed and merged the only other step is to make
|
||||
sure your GPG key is signed.
|
||||
|
||||
.. _maintainer_keys:
|
||||
|
||||
Maintainer GPG Keys
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
GPG is used to sign pull requests so they can be identified as really
|
||||
coming from the maintainer. If your key is not already signed by
|
||||
members of the QEMU community, you should make arrangements to attend
|
||||
a `KeySigningParty <https://wiki.qemu.org/KeySigningParty>`__ (for
|
||||
example at KVM Forum) or make alternative arrangements to have your
|
||||
key signed by an attendee. Key signing requires meeting another
|
||||
community member **in person**\ [#2020]_ so please make appropriate
|
||||
arrangements.
|
||||
|
||||
.. [#2020] In recent pandemic times we have had to exercise some
|
||||
flexibility here. Maintainers still need to sign their pull
|
||||
requests though.
|
||||
@@ -0,0 +1,366 @@
|
||||
==============
|
||||
The memory API
|
||||
==============
|
||||
|
||||
The memory API models the memory and I/O buses and controllers of a QEMU
|
||||
machine. It attempts to allow modelling of:
|
||||
|
||||
- ordinary RAM
|
||||
- memory-mapped I/O (MMIO)
|
||||
- memory controllers that can dynamically reroute physical memory regions
|
||||
to different destinations
|
||||
|
||||
The memory model provides support for
|
||||
|
||||
- tracking RAM changes by the guest
|
||||
- setting up coalesced memory for kvm
|
||||
- setting up ioeventfd regions for kvm
|
||||
|
||||
Memory is modelled as an acyclic graph of MemoryRegion objects. Sinks
|
||||
(leaves) are RAM and MMIO regions, while other nodes represent
|
||||
buses, memory controllers, and memory regions that have been rerouted.
|
||||
|
||||
In addition to MemoryRegion objects, the memory API provides AddressSpace
|
||||
objects for every root and possibly for intermediate MemoryRegions too.
|
||||
These represent memory as seen from the CPU or a device's viewpoint.
|
||||
|
||||
Types of regions
|
||||
----------------
|
||||
|
||||
There are multiple types of memory regions (all represented by a single C type
|
||||
MemoryRegion):
|
||||
|
||||
- RAM: a RAM region is simply a range of host memory that can be made available
|
||||
to the guest.
|
||||
You typically initialize these with memory_region_init_ram(). Some special
|
||||
purposes require the variants memory_region_init_resizeable_ram(),
|
||||
memory_region_init_ram_from_file(), or memory_region_init_ram_ptr().
|
||||
|
||||
- MMIO: a range of guest memory that is implemented by host callbacks;
|
||||
each read or write causes a callback to be called on the host.
|
||||
You initialize these with memory_region_init_io(), passing it a
|
||||
MemoryRegionOps structure describing the callbacks.
|
||||
|
||||
- ROM: a ROM memory region works like RAM for reads (directly accessing
|
||||
a region of host memory), and forbids writes. You initialize these with
|
||||
memory_region_init_rom().
|
||||
|
||||
- ROM device: a ROM device memory region works like RAM for reads
|
||||
(directly accessing a region of host memory), but like MMIO for
|
||||
writes (invoking a callback). You initialize these with
|
||||
memory_region_init_rom_device().
|
||||
|
||||
- IOMMU region: an IOMMU region translates addresses of accesses made to it
|
||||
and forwards them to some other target memory region. As the name suggests,
|
||||
these are only needed for modelling an IOMMU, not for simple devices.
|
||||
You initialize these with memory_region_init_iommu().
|
||||
|
||||
- container: a container simply includes other memory regions, each at
|
||||
a different offset. Containers are useful for grouping several regions
|
||||
into one unit. For example, a PCI BAR may be composed of a RAM region
|
||||
and an MMIO region.
|
||||
|
||||
A container's subregions are usually non-overlapping. In some cases it is
|
||||
useful to have overlapping regions; for example a memory controller that
|
||||
can overlay a subregion of RAM with MMIO or ROM, or a PCI controller
|
||||
that does not prevent card from claiming overlapping BARs.
|
||||
|
||||
You initialize a pure container with memory_region_init().
|
||||
|
||||
- alias: a subsection of another region. Aliases allow a region to be
|
||||
split apart into discontiguous regions. Examples of uses are memory
|
||||
banks used when the guest address space is smaller than the amount
|
||||
of RAM addressed, or a memory controller that splits main memory to
|
||||
expose a "PCI hole". You can also create aliases to avoid trying to
|
||||
add the original region to multiple parents via
|
||||
`memory_region_add_subregion`.
|
||||
|
||||
Aliases may point to any type of region, including other aliases,
|
||||
but an alias may not point back to itself, directly or indirectly.
|
||||
You initialize these with memory_region_init_alias().
|
||||
|
||||
- reservation region: a reservation region is primarily for debugging.
|
||||
It claims I/O space that is not supposed to be handled by QEMU itself.
|
||||
The typical use is to track parts of the address space which will be
|
||||
handled by the host kernel when KVM is enabled. You initialize these
|
||||
by passing a NULL callback parameter to memory_region_init_io().
|
||||
|
||||
It is valid to add subregions to a region which is not a pure container
|
||||
(that is, to an MMIO, RAM or ROM region). This means that the region
|
||||
will act like a container, except that any addresses within the container's
|
||||
region which are not claimed by any subregion are handled by the
|
||||
container itself (ie by its MMIO callbacks or RAM backing). However
|
||||
it is generally possible to achieve the same effect with a pure container
|
||||
one of whose subregions is a low priority "background" region covering
|
||||
the whole address range; this is often clearer and is preferred.
|
||||
Subregions cannot be added to an alias region.
|
||||
|
||||
Migration
|
||||
---------
|
||||
|
||||
Where the memory region is backed by host memory (RAM, ROM and
|
||||
ROM device memory region types), this host memory needs to be
|
||||
copied to the destination on migration. These APIs which allocate
|
||||
the host memory for you will also register the memory so it is
|
||||
migrated:
|
||||
|
||||
- memory_region_init_ram()
|
||||
- memory_region_init_rom()
|
||||
- memory_region_init_rom_device()
|
||||
|
||||
For most devices and boards this is the correct thing. If you
|
||||
have a special case where you need to manage the migration of
|
||||
the backing memory yourself, you can call the function
|
||||
memory_region_init_ram_flags_nomigrate()
|
||||
which only initializes the MemoryRegion and leaves handling
|
||||
migration to the caller.
|
||||
|
||||
The functions:
|
||||
|
||||
- memory_region_init_resizeable_ram()
|
||||
- memory_region_init_ram_from_file()
|
||||
- memory_region_init_ram_from_fd()
|
||||
- memory_region_init_ram_ptr()
|
||||
- memory_region_init_ram_device_ptr()
|
||||
|
||||
are for special cases only, and so they do not automatically
|
||||
register the backing memory for migration; the caller must
|
||||
manage migration if necessary.
|
||||
|
||||
Region names
|
||||
------------
|
||||
|
||||
Regions are assigned names by the constructor. For most regions these are
|
||||
only used for debugging purposes, but RAM regions also use the name to identify
|
||||
live migration sections. This means that RAM region names need to have ABI
|
||||
stability.
|
||||
|
||||
Region lifecycle
|
||||
----------------
|
||||
|
||||
A region is created by one of the memory_region_init*() functions and
|
||||
attached to an object, which acts as its owner or parent. QEMU ensures
|
||||
that the owner object remains alive as long as the region is visible to
|
||||
the guest, or as long as the region is in use by a virtual CPU or another
|
||||
device. For example, the owner object will not die between an
|
||||
address_space_map operation and the corresponding address_space_unmap.
|
||||
|
||||
After creation, a region can be added to an address space or a
|
||||
container with memory_region_add_subregion(), and removed using
|
||||
memory_region_del_subregion().
|
||||
|
||||
Various region attributes (read-only, dirty logging, coalesced mmio,
|
||||
ioeventfd) can be changed during the region lifecycle. They take effect
|
||||
as soon as the region is made visible. This can be immediately, later,
|
||||
or never.
|
||||
|
||||
Destruction of a memory region happens automatically when the owner object
|
||||
dies. When there are multiple memory regions under the same owner object,
|
||||
the memory API will guarantee all memory regions will be properly detached
|
||||
and finalized one by one. The order in which memory regions will be
|
||||
finalized is not guaranteed.
|
||||
|
||||
If however the memory region is part of a dynamically allocated data
|
||||
structure, you should free the memory region in the instance_finalize
|
||||
callback. For an example see VFIOMSIXInfo and VFIOQuirk in
|
||||
hw/vfio/pci.c.
|
||||
|
||||
You must not destroy a memory region as long as it may be in use by a
|
||||
device or CPU. In order to do this, as a general rule do not create or
|
||||
destroy memory regions dynamically during a device's lifetime, and never
|
||||
call object_unparent().
|
||||
|
||||
If you break this rule, the following situation can happen:
|
||||
|
||||
- the memory region's owner had a reference taken via memory_region_ref
|
||||
(for example by address_space_map)
|
||||
|
||||
- the region is unparented, and has no owner anymore
|
||||
|
||||
- when address_space_unmap is called, the reference to the memory region's
|
||||
owner is leaked.
|
||||
|
||||
|
||||
There is an exception to the above rule: it is okay to call
|
||||
object_unparent at any time for an alias or a container region. It is
|
||||
therefore also okay to create or destroy alias and container regions
|
||||
dynamically during a device's lifetime.
|
||||
|
||||
This exceptional usage is valid because aliases and containers only help
|
||||
QEMU building the guest's memory map; they are never accessed directly.
|
||||
memory_region_ref and memory_region_unref are never called on aliases
|
||||
or containers, and the above situation then cannot happen. Exploiting
|
||||
this exception is rarely necessary, and therefore it is discouraged,
|
||||
but nevertheless it is used in a few places.
|
||||
|
||||
For regions that "have no owner" (NULL is passed at creation time), the
|
||||
machine object is actually used as the owner.
|
||||
|
||||
|
||||
Overlapping regions and priority
|
||||
--------------------------------
|
||||
Usually, regions may not overlap each other; a memory address decodes into
|
||||
exactly one target. In some cases it is useful to allow regions to overlap,
|
||||
and sometimes to control which of an overlapping regions is visible to the
|
||||
guest. This is done with memory_region_add_subregion_overlap(), which
|
||||
allows the region to overlap any other region in the same container, and
|
||||
specifies a priority that allows the core to decide which of two regions at
|
||||
the same address are visible (highest wins).
|
||||
Priority values are signed, and the default value is zero. This means that
|
||||
you can use memory_region_add_subregion_overlap() both to specify a region
|
||||
that must sit 'above' any others (with a positive priority) and also a
|
||||
background region that sits 'below' others (with a negative priority).
|
||||
|
||||
If the higher priority region in an overlap is a container or alias, then
|
||||
the lower priority region will appear in any "holes" that the higher priority
|
||||
region has left by not mapping subregions to that area of its address range.
|
||||
(This applies recursively -- if the subregions are themselves containers or
|
||||
aliases that leave holes then the lower priority region will appear in these
|
||||
holes too.)
|
||||
|
||||
For example, suppose we have a container A of size 0x8000 with two subregions
|
||||
B and C. B is a container mapped at 0x2000, size 0x4000, priority 2; C is
|
||||
an MMIO region mapped at 0x0, size 0x6000, priority 1. B currently has two
|
||||
of its own subregions: D of size 0x1000 at offset 0 and E of size 0x1000 at
|
||||
offset 0x2000. As a diagram::
|
||||
|
||||
0 1000 2000 3000 4000 5000 6000 7000 8000
|
||||
|------|------|------|------|------|------|------|------|
|
||||
A: [ ]
|
||||
C: [CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC]
|
||||
B: [ ]
|
||||
D: [DDDDD]
|
||||
E: [EEEEE]
|
||||
|
||||
The regions that will be seen within this address range then are::
|
||||
|
||||
[CCCCCCCCCCCC][DDDDD][CCCCC][EEEEE][CCCCC]
|
||||
|
||||
Since B has higher priority than C, its subregions appear in the flat map
|
||||
even where they overlap with C. In ranges where B has not mapped anything
|
||||
C's region appears.
|
||||
|
||||
If B had provided its own MMIO operations (ie it was not a pure container)
|
||||
then these would be used for any addresses in its range not handled by
|
||||
D or E, and the result would be::
|
||||
|
||||
[CCCCCCCCCCCC][DDDDD][BBBBB][EEEEE][BBBBB]
|
||||
|
||||
Priority values are local to a container, because the priorities of two
|
||||
regions are only compared when they are both children of the same container.
|
||||
This means that the device in charge of the container (typically modelling
|
||||
a bus or a memory controller) can use them to manage the interaction of
|
||||
its child regions without any side effects on other parts of the system.
|
||||
In the example above, the priorities of D and E are unimportant because
|
||||
they do not overlap each other. It is the relative priority of B and C
|
||||
that causes D and E to appear on top of C: D and E's priorities are never
|
||||
compared against the priority of C.
|
||||
|
||||
Visibility
|
||||
----------
|
||||
The memory core uses the following rules to select a memory region when the
|
||||
guest accesses an address:
|
||||
|
||||
- all direct subregions of the root region are matched against the address, in
|
||||
descending priority order
|
||||
|
||||
- if the address lies outside the region offset/size, the subregion is
|
||||
discarded
|
||||
- if the subregion is a leaf (RAM or MMIO), the search terminates, returning
|
||||
this leaf region
|
||||
- if the subregion is a container, the same algorithm is used within the
|
||||
subregion (after the address is adjusted by the subregion offset)
|
||||
- if the subregion is an alias, the search is continued at the alias target
|
||||
(after the address is adjusted by the subregion offset and alias offset)
|
||||
- if a recursive search within a container or alias subregion does not
|
||||
find a match (because of a "hole" in the container's coverage of its
|
||||
address range), then if this is a container with its own MMIO or RAM
|
||||
backing the search terminates, returning the container itself. Otherwise
|
||||
we continue with the next subregion in priority order
|
||||
|
||||
- if none of the subregions match the address then the search terminates
|
||||
with no match found
|
||||
|
||||
Example memory map
|
||||
------------------
|
||||
|
||||
::
|
||||
|
||||
system_memory: container@0-2^48-1
|
||||
|
|
||||
+---- lomem: alias@0-0xdfffffff ---> #ram (0-0xdfffffff)
|
||||
|
|
||||
+---- himem: alias@0x100000000-0x11fffffff ---> #ram (0xe0000000-0xffffffff)
|
||||
|
|
||||
+---- vga-window: alias@0xa0000-0xbffff ---> #pci (0xa0000-0xbffff)
|
||||
| (prio 1)
|
||||
|
|
||||
+---- pci-hole: alias@0xe0000000-0xffffffff ---> #pci (0xe0000000-0xffffffff)
|
||||
|
||||
pci (0-2^32-1)
|
||||
|
|
||||
+--- vga-area: container@0xa0000-0xbffff
|
||||
| |
|
||||
| +--- alias@0x00000-0x7fff ---> #vram (0x010000-0x017fff)
|
||||
| |
|
||||
| +--- alias@0x08000-0xffff ---> #vram (0x020000-0x027fff)
|
||||
|
|
||||
+---- vram: ram@0xe1000000-0xe1ffffff
|
||||
|
|
||||
+---- vga-mmio: mmio@0xe2000000-0xe200ffff
|
||||
|
||||
ram: ram@0x00000000-0xffffffff
|
||||
|
||||
This is a (simplified) PC memory map. The 4GB RAM block is mapped into the
|
||||
system address space via two aliases: "lomem" is a 1:1 mapping of the first
|
||||
3.5GB; "himem" maps the last 0.5GB at address 4GB. This leaves 0.5GB for the
|
||||
so-called PCI hole, that allows a 32-bit PCI bus to exist in a system with
|
||||
4GB of memory.
|
||||
|
||||
The memory controller diverts addresses in the range 640K-768K to the PCI
|
||||
address space. This is modelled using the "vga-window" alias, mapped at a
|
||||
higher priority so it obscures the RAM at the same addresses. The vga window
|
||||
can be removed by programming the memory controller; this is modelled by
|
||||
removing the alias and exposing the RAM underneath.
|
||||
|
||||
The pci address space is not a direct child of the system address space, since
|
||||
we only want parts of it to be visible (we accomplish this using aliases).
|
||||
It has two subregions: vga-area models the legacy vga window and is occupied
|
||||
by two 32K memory banks pointing at two sections of the framebuffer.
|
||||
In addition the vram is mapped as a BAR at address e1000000, and an additional
|
||||
BAR containing MMIO registers is mapped after it.
|
||||
|
||||
Note that if the guest maps a BAR outside the PCI hole, it would not be
|
||||
visible as the pci-hole alias clips it to a 0.5GB range.
|
||||
|
||||
MMIO Operations
|
||||
---------------
|
||||
|
||||
MMIO regions are provided with ->read() and ->write() callbacks,
|
||||
which are sufficient for most devices. Some devices change behaviour
|
||||
based on the attributes used for the memory transaction, or need
|
||||
to be able to respond that the access should provoke a bus error
|
||||
rather than completing successfully; those devices can use the
|
||||
->read_with_attrs() and ->write_with_attrs() callbacks instead.
|
||||
|
||||
In addition various constraints can be supplied to control how these
|
||||
callbacks are called:
|
||||
|
||||
- .valid.min_access_size, .valid.max_access_size define the access sizes
|
||||
(in bytes) which the device accepts; accesses outside this range will
|
||||
have device and bus specific behaviour (ignored, or machine check)
|
||||
- .valid.unaligned specifies that the *device being modelled* supports
|
||||
unaligned accesses; if false, unaligned accesses will invoke the
|
||||
appropriate bus or CPU specific behaviour.
|
||||
- .impl.min_access_size, .impl.max_access_size define the access sizes
|
||||
(in bytes) supported by the *implementation*; other access sizes will be
|
||||
emulated using the ones available. For example a 4-byte write will be
|
||||
emulated using four 1-byte writes, if .impl.max_access_size = 1.
|
||||
- .impl.unaligned specifies that the *implementation* supports unaligned
|
||||
accesses; if false, unaligned accesses will be emulated by two aligned
|
||||
accesses.
|
||||
|
||||
API Reference
|
||||
-------------
|
||||
|
||||
.. kernel-doc:: include/system/memory.h
|
||||
@@ -0,0 +1,436 @@
|
||||
CheckPoint and Restart (CPR)
|
||||
============================
|
||||
|
||||
CPR is the umbrella name for a set of migration modes in which the
|
||||
VM is migrated to a new QEMU instance on the same host. It is
|
||||
intended for use when the goal is to update host software components
|
||||
that run the VM, such as QEMU or even the host kernel. At this time,
|
||||
the cpr-reboot, cpr-transfer, and cpr-exec modes are available.
|
||||
|
||||
Because QEMU is restarted on the same host, with access to the same
|
||||
local devices, CPR is allowed in certain cases where normal migration
|
||||
would be blocked. However, the user must not modify the contents of
|
||||
guest block devices between quitting old QEMU and starting new QEMU.
|
||||
|
||||
CPR unconditionally stops VM execution before memory is saved, and
|
||||
thus does not depend on any form of dirty page tracking.
|
||||
|
||||
cpr-reboot mode
|
||||
---------------
|
||||
|
||||
In this mode, QEMU stops the VM, and writes VM state to the migration
|
||||
URI, which will typically be a file. After quitting QEMU, the user
|
||||
resumes by running QEMU with the ``-incoming`` option. Because the
|
||||
old and new QEMU instances are not active concurrently, the URI cannot
|
||||
be a type that streams data from one instance to the other.
|
||||
|
||||
Guest RAM can be saved in place if backed by shared memory, or can be
|
||||
copied to a file. The former is more efficient and is therefore
|
||||
preferred.
|
||||
|
||||
After state and memory are saved, the user may update userland host
|
||||
software before restarting QEMU and resuming the VM. Further, if
|
||||
the RAM is backed by persistent shared memory, such as a DAX device,
|
||||
then the user may reboot to a new host kernel before restarting QEMU.
|
||||
|
||||
This mode supports VFIO devices provided the user first puts the
|
||||
guest in the suspended runstate, such as by issuing the
|
||||
``guest-suspend-ram`` command to the QEMU guest agent. The agent
|
||||
must be pre-installed in the guest, and the guest must support
|
||||
suspend to RAM. Beware that suspension can take a few seconds, so
|
||||
the user should poll to see the suspended state before proceeding
|
||||
with the CPR operation.
|
||||
|
||||
Usage
|
||||
^^^^^
|
||||
|
||||
It is recommended that guest RAM be backed with some type of shared
|
||||
memory, such as ``memory-backend-file,share=on``, and that the
|
||||
``x-ignore-shared`` capability be set. This combination allows memory
|
||||
to be saved in place. Otherwise, after QEMU stops the VM, all guest
|
||||
RAM is copied to the migration URI.
|
||||
|
||||
Outgoing:
|
||||
* Set the migration mode parameter to ``cpr-reboot``.
|
||||
* Set the ``x-ignore-shared`` capability if desired.
|
||||
* Issue the ``migrate`` command. It is recommended the URI be a
|
||||
``file`` type, but one can use other types such as ``exec``,
|
||||
provided the command captures all the data from the outgoing side,
|
||||
and provides all the data to the incoming side.
|
||||
* Quit when QEMU reaches the postmigrate state.
|
||||
|
||||
Incoming:
|
||||
* Start QEMU with the ``-incoming defer`` option.
|
||||
* Set the migration mode parameter to ``cpr-reboot``.
|
||||
* Set the ``x-ignore-shared`` capability if desired.
|
||||
* Issue the ``migrate-incoming`` command.
|
||||
* If the VM was running when the outgoing ``migrate`` command was
|
||||
issued, then QEMU automatically resumes VM execution.
|
||||
|
||||
Example 1
|
||||
^^^^^^^^^
|
||||
::
|
||||
|
||||
# qemu-kvm -monitor stdio
|
||||
-object memory-backend-file,id=ram0,size=4G,mem-path=/dev/dax0.0,align=2M,share=on -m 4G
|
||||
...
|
||||
|
||||
(qemu) info status
|
||||
VM status: running
|
||||
(qemu) migrate_set_parameter mode cpr-reboot
|
||||
(qemu) migrate_set_capability x-ignore-shared on
|
||||
(qemu) migrate -d file:vm.state
|
||||
(qemu) info status
|
||||
VM status: paused (postmigrate)
|
||||
(qemu) quit
|
||||
|
||||
### optionally update kernel and reboot
|
||||
# systemctl kexec
|
||||
kexec_core: Starting new kernel
|
||||
...
|
||||
|
||||
# qemu-kvm ... -incoming defer
|
||||
(qemu) info status
|
||||
VM status: paused (inmigrate)
|
||||
(qemu) migrate_set_parameter mode cpr-reboot
|
||||
(qemu) migrate_set_capability x-ignore-shared on
|
||||
(qemu) migrate_incoming file:vm.state
|
||||
(qemu) info status
|
||||
VM status: running
|
||||
|
||||
Example 2: VFIO
|
||||
^^^^^^^^^^^^^^^
|
||||
::
|
||||
|
||||
# qemu-kvm -monitor stdio
|
||||
-object memory-backend-file,id=ram0,size=4G,mem-path=/dev/dax0.0,align=2M,share=on -m 4G
|
||||
-device vfio-pci, ...
|
||||
-chardev socket,id=qga0,path=qga.sock,server=on,wait=off
|
||||
-device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0
|
||||
...
|
||||
|
||||
(qemu) info status
|
||||
VM status: running
|
||||
|
||||
# echo '{"execute":"guest-suspend-ram"}' | ncat --send-only -U qga.sock
|
||||
|
||||
(qemu) info status
|
||||
VM status: paused (suspended)
|
||||
(qemu) migrate_set_parameter mode cpr-reboot
|
||||
(qemu) migrate_set_capability x-ignore-shared on
|
||||
(qemu) migrate -d file:vm.state
|
||||
(qemu) info status
|
||||
VM status: paused (postmigrate)
|
||||
(qemu) quit
|
||||
|
||||
### optionally update kernel and reboot
|
||||
# systemctl kexec
|
||||
kexec_core: Starting new kernel
|
||||
...
|
||||
|
||||
# qemu-kvm ... -incoming defer
|
||||
(qemu) info status
|
||||
VM status: paused (inmigrate)
|
||||
(qemu) migrate_set_parameter mode cpr-reboot
|
||||
(qemu) migrate_set_capability x-ignore-shared on
|
||||
(qemu) migrate_incoming file:vm.state
|
||||
(qemu) info status
|
||||
VM status: paused (suspended)
|
||||
(qemu) system_wakeup
|
||||
(qemu) info status
|
||||
VM status: running
|
||||
|
||||
Caveats
|
||||
^^^^^^^
|
||||
|
||||
cpr-reboot mode may not be used with postcopy, background-snapshot,
|
||||
or COLO.
|
||||
|
||||
cpr-transfer mode
|
||||
-----------------
|
||||
|
||||
This mode allows the user to transfer a guest to a new QEMU instance
|
||||
on the same host with minimal guest pause time, by preserving guest
|
||||
RAM in place, albeit with new virtual addresses in new QEMU. Devices
|
||||
and their pinned memory pages are also preserved for VFIO and IOMMUFD.
|
||||
|
||||
The user starts new QEMU on the same host as old QEMU, with command-
|
||||
line arguments to create the same machine, plus the ``-incoming``
|
||||
option for the main migration channel, like normal live migration.
|
||||
In addition, the user adds a second -incoming option with channel
|
||||
type ``cpr``. This CPR channel must support file descriptor transfer
|
||||
with SCM_RIGHTS, i.e. it must be a UNIX domain socket.
|
||||
|
||||
To initiate CPR, the user issues a migrate command to old QEMU,
|
||||
adding a second migration channel of type ``cpr`` in the channels
|
||||
argument. Old QEMU stops the VM, saves state to the migration
|
||||
channels, and enters the postmigrate state. Execution resumes in
|
||||
new QEMU.
|
||||
|
||||
New QEMU reads the CPR channel before opening a monitor, hence
|
||||
the CPR channel cannot be specified in the list of channels for a
|
||||
migrate-incoming command. It may only be specified on the command
|
||||
line.
|
||||
|
||||
Usage
|
||||
^^^^^
|
||||
|
||||
Memory backend objects must have the ``share=on`` attribute.
|
||||
|
||||
The VM must be started with the ``-machine aux-ram-share=on``
|
||||
option. This causes implicit RAM blocks (those not described by
|
||||
a memory-backend object) to be allocated by mmap'ing a memfd.
|
||||
Examples include VGA and ROM.
|
||||
|
||||
Outgoing:
|
||||
* Set the migration mode parameter to ``cpr-transfer``.
|
||||
* Issue the ``migrate`` command, containing a main channel and
|
||||
a cpr channel.
|
||||
|
||||
Incoming:
|
||||
* Start new QEMU with two ``-incoming`` options.
|
||||
* If the VM was running when the outgoing ``migrate`` command was
|
||||
issued, then QEMU automatically resumes VM execution.
|
||||
|
||||
Caveats
|
||||
^^^^^^^
|
||||
|
||||
cpr-transfer mode may not be used with postcopy, background-snapshot,
|
||||
or COLO.
|
||||
|
||||
memory-backend-epc is not supported.
|
||||
|
||||
The main incoming migration channel address cannot be a file type.
|
||||
|
||||
If the main incoming channel address is an inet socket, then the port
|
||||
cannot be 0 (meaning dynamically choose a port).
|
||||
|
||||
When using ``-incoming defer``, you must issue the migrate command to
|
||||
old QEMU before issuing any monitor commands to new QEMU, because new
|
||||
QEMU blocks waiting to read from the cpr channel before starting its
|
||||
monitor, and old QEMU does not write to the channel until the migrate
|
||||
command is issued. However, new QEMU does not open and read the
|
||||
main migration channel until you issue the migrate incoming command.
|
||||
|
||||
Example 1: incoming channel
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
In these examples, we simply restart the same version of QEMU, but
|
||||
in a real scenario one would start new QEMU on the incoming side.
|
||||
Note that new QEMU does not print the monitor prompt until old QEMU
|
||||
has issued the migrate command. The outgoing side uses QMP because
|
||||
HMP cannot specify a CPR channel. Some QMP responses are omitted for
|
||||
brevity.
|
||||
|
||||
::
|
||||
|
||||
Outgoing: Incoming:
|
||||
|
||||
# qemu-kvm -qmp stdio
|
||||
-object memory-backend-file,id=ram0,size=4G,
|
||||
mem-path=/dev/shm/ram0,share=on -m 4G
|
||||
-machine memory-backend=ram0
|
||||
-machine aux-ram-share=on
|
||||
...
|
||||
# qemu-kvm -monitor stdio
|
||||
-incoming tcp:0:44444
|
||||
-incoming '{"channel-type": "cpr",
|
||||
"addr": { "transport": "socket",
|
||||
"type": "unix", "path": "cpr.sock"}}'
|
||||
...
|
||||
{"execute":"qmp_capabilities"}
|
||||
|
||||
{"execute": "query-status"}
|
||||
{"return": {"status": "running",
|
||||
"running": true}}
|
||||
|
||||
{"execute":"migrate-set-parameters",
|
||||
"arguments":{"mode":"cpr-transfer"}}
|
||||
|
||||
{"execute": "migrate", "arguments": { "channels": [
|
||||
{"channel-type": "main",
|
||||
"addr": { "transport": "socket", "type": "inet",
|
||||
"host": "0", "port": "44444" }},
|
||||
{"channel-type": "cpr",
|
||||
"addr": { "transport": "socket", "type": "unix",
|
||||
"path": "cpr.sock" }}]}}
|
||||
|
||||
QEMU 10.0.50 monitor
|
||||
(qemu) info status
|
||||
VM status: running
|
||||
|
||||
{"execute": "query-status"}
|
||||
{"return": {"status": "postmigrate",
|
||||
"running": false}}
|
||||
|
||||
Example 2: incoming defer
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
This example uses ``-incoming defer`` to hot plug a device before
|
||||
accepting the main migration channel. Again note you must issue the
|
||||
migrate command to old QEMU before you can issue any monitor
|
||||
commands to new QEMU.
|
||||
|
||||
|
||||
::
|
||||
|
||||
Outgoing: Incoming:
|
||||
|
||||
# qemu-kvm -monitor stdio
|
||||
-object memory-backend-file,id=ram0,size=4G,
|
||||
mem-path=/dev/shm/ram0,share=on -m 4G
|
||||
-machine memory-backend=ram0
|
||||
-machine aux-ram-share=on
|
||||
...
|
||||
# qemu-kvm -monitor stdio
|
||||
-incoming defer
|
||||
-incoming '{"channel-type": "cpr",
|
||||
"addr": { "transport": "socket",
|
||||
"type": "unix", "path": "cpr.sock"}}'
|
||||
...
|
||||
{"execute":"qmp_capabilities"}
|
||||
|
||||
{"execute": "device_add",
|
||||
"arguments": {"driver": "pcie-root-port"}}
|
||||
|
||||
{"execute":"migrate-set-parameters",
|
||||
"arguments":{"mode":"cpr-transfer"}}
|
||||
|
||||
{"execute": "migrate", "arguments": { "channels": [
|
||||
{"channel-type": "main",
|
||||
"addr": { "transport": "socket", "type": "inet",
|
||||
"host": "0", "port": "44444" }},
|
||||
{"channel-type": "cpr",
|
||||
"addr": { "transport": "socket", "type": "unix",
|
||||
"path": "cpr.sock" }}]}}
|
||||
|
||||
QEMU 10.0.50 monitor
|
||||
(qemu) info status
|
||||
VM status: paused (inmigrate)
|
||||
(qemu) device_add pcie-root-port
|
||||
(qemu) migrate_incoming tcp:0:44444
|
||||
(qemu) info status
|
||||
VM status: running
|
||||
|
||||
{"execute": "query-status"}
|
||||
{"return": {"status": "postmigrate",
|
||||
"running": false}}
|
||||
|
||||
Futures
|
||||
^^^^^^^
|
||||
|
||||
cpr-transfer mode is based on a capability to transfer open file
|
||||
descriptors from old to new QEMU. In the future, descriptors for
|
||||
vhost, and char devices could be transferred,
|
||||
preserving those devices and their kernel state without interruption,
|
||||
even if they do not explicitly support live migration.
|
||||
|
||||
cpr-exec mode
|
||||
-------------
|
||||
|
||||
In this mode, QEMU stops the VM, writes VM state to the migration
|
||||
URI, and directly exec's a new version of QEMU on the same host,
|
||||
replacing the original process while retaining its PID. Guest RAM is
|
||||
preserved in place, albeit with new virtual addresses. The user
|
||||
completes the migration by specifying the ``-incoming`` option, and
|
||||
by issuing the ``migrate-incoming`` command if necessary; see details
|
||||
below.
|
||||
|
||||
This mode supports VFIO/IOMMUFD devices by preserving device
|
||||
descriptors and hence kernel state across the exec, even for devices
|
||||
that do not support live migration.
|
||||
|
||||
Because the old and new QEMU instances are not active concurrently,
|
||||
the URI cannot be a type that streams data from one instance to the
|
||||
other.
|
||||
|
||||
This mode does not require a channel of type ``cpr``. The information
|
||||
that is passed over that channel for cpr-transfer mode is instead
|
||||
serialized to a memfd, the number of the fd is saved in the
|
||||
QEMU_CPR_EXEC_STATE environment variable during the exec of new QEMU.
|
||||
and new QEMU mmaps the memfd.
|
||||
|
||||
Usage
|
||||
^^^^^
|
||||
|
||||
Arguments for the new QEMU process are taken from the
|
||||
@cpr-exec-command parameter. The first argument should be the
|
||||
path of a new QEMU binary, or a prefix command that exec's the
|
||||
new QEMU binary, and the arguments should include the ''-incoming''
|
||||
option.
|
||||
|
||||
Memory backend objects must have the ``share=on`` attribute.
|
||||
The VM must be started with the ``-machine aux-ram-share=on`` option.
|
||||
|
||||
Outgoing:
|
||||
* Set the migration mode parameter to ``cpr-exec``.
|
||||
* Set the ``cpr-exec-command`` parameter.
|
||||
* Issue the ``migrate`` command. It is recommended that the URI be
|
||||
a ``file`` type, but one can use other types such as ``exec``,
|
||||
provided the command captures all the data from the outgoing side,
|
||||
and provides all the data to the incoming side.
|
||||
|
||||
Incoming:
|
||||
* You do not need to explicitly start new QEMU. It is started as
|
||||
a side effect of the migrate command above.
|
||||
* If the VM was running when the outgoing ``migrate`` command was
|
||||
issued, then QEMU automatically resumes VM execution.
|
||||
|
||||
Example 1: incoming URI
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
In these examples, we simply restart the same version of QEMU, but in
|
||||
a real scenario one would set a new QEMU binary path in
|
||||
cpr-exec-command.
|
||||
|
||||
::
|
||||
|
||||
# qemu-kvm -monitor stdio
|
||||
-object memory-backend-memfd,id=ram0,size=4G
|
||||
-machine memory-backend=ram0
|
||||
-machine aux-ram-share=on
|
||||
...
|
||||
|
||||
QEMU 10.2.50 monitor - type 'help' for more information
|
||||
(qemu) info status
|
||||
VM status: running
|
||||
(qemu) migrate_set_parameter mode cpr-exec
|
||||
(qemu) migrate_set_parameter cpr-exec-command qemu-kvm ... -incoming file:vm.state
|
||||
(qemu) migrate -d file:vm.state
|
||||
(qemu) QEMU 10.2.50 monitor - type 'help' for more information
|
||||
(qemu) info status
|
||||
VM status: running
|
||||
|
||||
Example 2: incoming defer
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
::
|
||||
|
||||
# qemu-kvm -monitor stdio
|
||||
-object memory-backend-memfd,id=ram0,size=4G
|
||||
-machine memory-backend=ram0
|
||||
-machine aux-ram-share=on
|
||||
...
|
||||
|
||||
QEMU 10.2.50 monitor - type 'help' for more information
|
||||
(qemu) info status
|
||||
VM status: running
|
||||
(qemu) migrate_set_parameter mode cpr-exec
|
||||
(qemu) migrate_set_parameter cpr-exec-command qemu-kvm ... -incoming defer
|
||||
(qemu) migrate -d file:vm.state
|
||||
(qemu) QEMU 10.2.50 monitor - type 'help' for more information
|
||||
(qemu) info status
|
||||
status: paused (inmigrate)
|
||||
(qemu) migrate_incoming file:vm.state
|
||||
(qemu) info status
|
||||
VM status: running
|
||||
|
||||
Caveats
|
||||
^^^^^^^
|
||||
|
||||
cpr-exec mode may not be used with postcopy, background-snapshot,
|
||||
or COLO.
|
||||
|
||||
cpr-exec mode requires permission to use the exec system call, which
|
||||
is denied by certain sandbox options, such as spawn.
|
||||
|
||||
The guest pause time increases for large guest RAM backed by small pages.
|
||||
@@ -0,0 +1,48 @@
|
||||
==============
|
||||
Best practices
|
||||
==============
|
||||
|
||||
Debugging
|
||||
=========
|
||||
|
||||
The migration stream can be analyzed thanks to ``scripts/analyze-migration.py``.
|
||||
|
||||
Example usage:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ qemu-system-x86_64 -display none -monitor stdio
|
||||
(qemu) migrate "exec:cat > mig"
|
||||
(qemu) q
|
||||
$ ./scripts/analyze-migration.py -f mig
|
||||
{
|
||||
"ram (3)": {
|
||||
"section sizes": {
|
||||
"pc.ram": "0x0000000008000000",
|
||||
...
|
||||
|
||||
See also ``analyze-migration.py -h`` help for more options.
|
||||
|
||||
Firmware
|
||||
========
|
||||
|
||||
Migration migrates the copies of RAM and ROM, and thus when running
|
||||
on the destination it includes the firmware from the source. Even after
|
||||
resetting a VM, the old firmware is used. Only once QEMU has been restarted
|
||||
is the new firmware in use.
|
||||
|
||||
- Changes in firmware size can cause changes in the required RAMBlock size
|
||||
to hold the firmware and thus migration can fail. In practice it's best
|
||||
to pad firmware images to convenient powers of 2 with plenty of space
|
||||
for growth.
|
||||
|
||||
- Care should be taken with device emulation code so that newer
|
||||
emulation code can work with older firmware to allow forward migration.
|
||||
|
||||
- Care should be taken with newer firmware so that backward migration
|
||||
to older systems with older device emulation code will work.
|
||||
|
||||
In some cases it may be best to tie specific firmware versions to specific
|
||||
versioned machine types to cut down on the combinations that will need
|
||||
support. This is also useful when newer versions of firmware outgrow
|
||||
the padding.
|
||||
@@ -0,0 +1,516 @@
|
||||
Backwards compatibility
|
||||
=======================
|
||||
|
||||
How backwards compatibility works
|
||||
---------------------------------
|
||||
|
||||
When we do migration, we have two QEMU processes: the source and the
|
||||
target. There are two cases, they are the same version or they are
|
||||
different versions. The easy case is when they are the same version.
|
||||
The difficult one is when they are different versions.
|
||||
|
||||
There are two things that are different, but they have very similar
|
||||
names and sometimes get confused:
|
||||
|
||||
- QEMU version
|
||||
- machine type version
|
||||
|
||||
Let's start with a practical example, we start with:
|
||||
|
||||
- qemu-system-x86_64 (v5.2), from now on qemu-5.2.
|
||||
- qemu-system-x86_64 (v5.1), from now on qemu-5.1.
|
||||
|
||||
Related to this are the "latest" machine types defined on each of
|
||||
them:
|
||||
|
||||
- pc-q35-5.2 (newer one in qemu-5.2) from now on pc-5.2
|
||||
- pc-q35-5.1 (newer one in qemu-5.1) from now on pc-5.1
|
||||
|
||||
First of all, migration is only supposed to work if you use the same
|
||||
machine type in both source and destination. The QEMU hardware
|
||||
configuration needs to be the same also on source and destination.
|
||||
Most aspects of the backend configuration can be changed at will,
|
||||
except for a few cases where the backend features influence frontend
|
||||
device feature exposure. But that is not relevant for this section.
|
||||
|
||||
I am going to list the number of combinations that we can have. Let's
|
||||
start with the trivial ones, QEMU is the same on source and
|
||||
destination:
|
||||
|
||||
1 - qemu-5.2 -M pc-5.2 -> migrates to -> qemu-5.2 -M pc-5.2
|
||||
|
||||
This is the latest QEMU with the latest machine type.
|
||||
This have to work, and if it doesn't work it is a bug.
|
||||
|
||||
2 - qemu-5.1 -M pc-5.1 -> migrates to -> qemu-5.1 -M pc-5.1
|
||||
|
||||
Exactly the same case than the previous one, but for 5.1.
|
||||
Nothing to see here either.
|
||||
|
||||
This are the easiest ones, we will not talk more about them in this
|
||||
section.
|
||||
|
||||
Now we start with the more interesting cases. Consider the case where
|
||||
we have the same QEMU version in both sides (qemu-5.2) but we are using
|
||||
the latest machine type for that version (pc-5.2) but one of an older
|
||||
QEMU version, in this case pc-5.1.
|
||||
|
||||
3 - qemu-5.2 -M pc-5.1 -> migrates to -> qemu-5.2 -M pc-5.1
|
||||
|
||||
It needs to use the definition of pc-5.1 and the devices as they
|
||||
were configured on 5.1, but this should be easy in the sense that
|
||||
both sides are the same QEMU and both sides have exactly the same
|
||||
idea of what the pc-5.1 machine is.
|
||||
|
||||
4 - qemu-5.1 -M pc-5.2 -> migrates to -> qemu-5.1 -M pc-5.2
|
||||
|
||||
This combination is not possible as the qemu-5.1 doesn't understand
|
||||
pc-5.2 machine type. So nothing to worry here.
|
||||
|
||||
Now it comes the interesting ones, when both QEMU processes are
|
||||
different. Notice also that the machine type needs to be pc-5.1,
|
||||
because we have the limitation than qemu-5.1 doesn't know pc-5.2. So
|
||||
the possible cases are:
|
||||
|
||||
5 - qemu-5.2 -M pc-5.1 -> migrates to -> qemu-5.1 -M pc-5.1
|
||||
|
||||
This migration is known as newer to older. We need to make sure
|
||||
when we are developing 5.2 we need to take care about not to break
|
||||
migration to qemu-5.1. Notice that we can't make updates to
|
||||
qemu-5.1 to understand whatever qemu-5.2 decides to change, so it is
|
||||
in qemu-5.2 side to make the relevant changes.
|
||||
|
||||
6 - qemu-5.1 -M pc-5.1 -> migrates to -> qemu-5.2 -M pc-5.1
|
||||
|
||||
This migration is known as older to newer. We need to make sure
|
||||
than we are able to receive migrations from qemu-5.1. The problem is
|
||||
similar to the previous one.
|
||||
|
||||
If qemu-5.1 and qemu-5.2 were the same, there will not be any
|
||||
compatibility problems. But the reason that we create qemu-5.2 is to
|
||||
get new features, devices, defaults, etc.
|
||||
|
||||
If we get a device that has a new feature, or change a default value,
|
||||
we have a problem when we try to migrate between different QEMU
|
||||
versions.
|
||||
|
||||
So we need a way to tell qemu-5.2 that when we are using machine type
|
||||
pc-5.1, it needs to **not** use the feature, to be able to migrate to
|
||||
real qemu-5.1.
|
||||
|
||||
And the equivalent part when migrating from qemu-5.1 to qemu-5.2.
|
||||
qemu-5.2 has to expect that it is not going to get data for the new
|
||||
feature, because qemu-5.1 doesn't know about it.
|
||||
|
||||
How do we tell QEMU about these device feature changes? In
|
||||
hw/core/machine.c:hw_compat_X_Y arrays.
|
||||
|
||||
If we change a default value, we need to put back the old value on
|
||||
that array. And the device, during initialization needs to look at
|
||||
that array to see what value it needs to get for that feature. And
|
||||
what are we going to put in that array, the value of a property.
|
||||
|
||||
To create a property for a device, we need to use one of the
|
||||
DEFINE_PROP_*() macros. See include/hw/core/qdev-properties.h to find the
|
||||
macros that exist. With it, we set the default value for that
|
||||
property, and that is what it is going to get in the latest released
|
||||
version. But if we want a different value for a previous version, we
|
||||
can change that in the hw_compat_X_Y arrays.
|
||||
|
||||
hw_compat_X_Y is an array of registers that have the format:
|
||||
|
||||
- name_device
|
||||
- name_property
|
||||
- value
|
||||
|
||||
Let's see a practical example.
|
||||
|
||||
In qemu-5.2 virtio-blk-device got multi queue support. This is a
|
||||
change that is not backward compatible. In qemu-5.1 it has one
|
||||
queue. In qemu-5.2 it has the same number of queues as the number of
|
||||
cpus in the system.
|
||||
|
||||
When we are doing migration, if we migrate from a device that has 4
|
||||
queues to a device that have only one queue, we don't know where to
|
||||
put the extra information for the other 3 queues, and we fail
|
||||
migration.
|
||||
|
||||
Similar problem when we migrate from qemu-5.1 that has only one queue
|
||||
to qemu-5.2, we only sent information for one queue, but destination
|
||||
has 4, and we have 3 queues that are not properly initialized and
|
||||
anything can happen.
|
||||
|
||||
So, how can we address this problem. Easy, just convince qemu-5.2
|
||||
that when it is running pc-5.1, it needs to set the number of queues
|
||||
for virtio-blk-devices to 1.
|
||||
|
||||
That way we fix the cases 5 and 6.
|
||||
|
||||
5 - qemu-5.2 -M pc-5.1 -> migrates to -> qemu-5.1 -M pc-5.1
|
||||
|
||||
qemu-5.2 -M pc-5.1 sets number of queues to be 1.
|
||||
qemu-5.1 -M pc-5.1 expects number of queues to be 1.
|
||||
|
||||
correct. migration works.
|
||||
|
||||
6 - qemu-5.1 -M pc-5.1 -> migrates to -> qemu-5.2 -M pc-5.1
|
||||
|
||||
qemu-5.1 -M pc-5.1 sets number of queues to be 1.
|
||||
qemu-5.2 -M pc-5.1 expects number of queues to be 1.
|
||||
|
||||
correct. migration works.
|
||||
|
||||
And now the other interesting case, case 3. In this case we have:
|
||||
|
||||
3 - qemu-5.2 -M pc-5.1 -> migrates to -> qemu-5.2 -M pc-5.1
|
||||
|
||||
Here we have the same QEMU in both sides. So it doesn't matter a
|
||||
lot if we have set the number of queues to 1 or not, because
|
||||
they are the same.
|
||||
|
||||
WRONG!
|
||||
|
||||
Think what happens if we do one of this double migrations:
|
||||
|
||||
A -> migrates -> B -> migrates -> C
|
||||
|
||||
where:
|
||||
|
||||
A: qemu-5.1 -M pc-5.1
|
||||
B: qemu-5.2 -M pc-5.1
|
||||
C: qemu-5.2 -M pc-5.1
|
||||
|
||||
migration A -> B is case 6, so number of queues needs to be 1.
|
||||
|
||||
migration B -> C is case 3, so we don't care. But actually we
|
||||
care because we haven't started the guest in qemu-5.2, it came
|
||||
migrated from qemu-5.1. So to be in the safe place, we need to
|
||||
always use number of queues 1 when we are using pc-5.1.
|
||||
|
||||
Now, how was this done in reality? The following commit shows how it
|
||||
was done::
|
||||
|
||||
commit 9445e1e15e66c19e42bea942ba810db28052cd05
|
||||
Author: Stefan Hajnoczi <[email protected]>
|
||||
Date: Tue Aug 18 15:33:47 2020 +0100
|
||||
|
||||
virtio-blk-pci: default num_queues to -smp N
|
||||
|
||||
The relevant parts for migration are::
|
||||
|
||||
@@ -1281,7 +1284,8 @@ static const Property virtio_blk_properties[] = {
|
||||
#endif
|
||||
DEFINE_PROP_BIT("request-merging", VirtIOBlock, conf.request_merging, 0,
|
||||
true),
|
||||
- DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues, 1),
|
||||
+ DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues,
|
||||
+ VIRTIO_BLK_AUTO_NUM_QUEUES),
|
||||
DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size, 256),
|
||||
|
||||
It changes the default value of num_queues. But it fishes it for old
|
||||
machine types to have the right value::
|
||||
|
||||
@@ -31,6 +31,7 @@
|
||||
GlobalProperty hw_compat_5_1[] = {
|
||||
...
|
||||
+ { "virtio-blk-device", "num-queues", "1"},
|
||||
...
|
||||
};
|
||||
|
||||
A device with different features on both sides
|
||||
----------------------------------------------
|
||||
|
||||
Let's assume that we are using the same QEMU binary on both sides,
|
||||
just to make the things easier. But we have a device that has
|
||||
different features on both sides of the migration. That can be
|
||||
because the devices are different, because the kernel driver of both
|
||||
devices have different features, whatever.
|
||||
|
||||
How can we get this to work with migration. The way to do that is
|
||||
"theoretically" easy. You have to get the features that the device
|
||||
has in the source of the migration. The features that the device has
|
||||
on the target of the migration, you get the intersection of the
|
||||
features of both sides, and that is the way that you should launch
|
||||
QEMU.
|
||||
|
||||
Notice that this is not completely related to QEMU. The most
|
||||
important thing here is that this should be handled by the managing
|
||||
application that launches QEMU. If QEMU is configured correctly, the
|
||||
migration will succeed.
|
||||
|
||||
That said, actually doing it is complicated. Almost all devices are
|
||||
bad at being able to be launched with only some features enabled.
|
||||
With one big exception: cpus.
|
||||
|
||||
You can read the documentation for QEMU x86 cpu models here:
|
||||
|
||||
https://qemu-project.gitlab.io/qemu/system/qemu-cpu-models.html
|
||||
|
||||
See when they talk about migration they recommend that one chooses the
|
||||
newest cpu model that is supported for all cpus.
|
||||
|
||||
Let's say that we have:
|
||||
|
||||
Host A:
|
||||
|
||||
Device X has the feature Y
|
||||
|
||||
Host B:
|
||||
|
||||
Device X has not the feature Y
|
||||
|
||||
If we try to migrate without any care from host A to host B, it will
|
||||
fail because when migration tries to load the feature Y on
|
||||
destination, it will find that the hardware is not there.
|
||||
|
||||
Doing this would be the equivalent of doing with cpus:
|
||||
|
||||
Host A:
|
||||
|
||||
$ qemu-system-x86_64 -cpu host
|
||||
|
||||
Host B:
|
||||
|
||||
$ qemu-system-x86_64 -cpu host
|
||||
|
||||
When both hosts have different cpu features this is guaranteed to
|
||||
fail. Especially if Host B has less features than host A. If host A
|
||||
has less features than host B, sometimes it works. Important word of
|
||||
last sentence is "sometimes".
|
||||
|
||||
So, forgetting about cpu models and continuing with the -cpu host
|
||||
example, let's see that the differences of the cpus is that Host A and
|
||||
B have the following features:
|
||||
|
||||
Features: 'pcid' 'stibp' 'taa-no'
|
||||
Host A: X X
|
||||
Host B: X
|
||||
|
||||
And we want to migrate between them, the way configure both QEMU cpu
|
||||
will be:
|
||||
|
||||
Host A:
|
||||
|
||||
$ qemu-system-x86_64 -cpu host,pcid=off,stibp=off
|
||||
|
||||
Host B:
|
||||
|
||||
$ qemu-system-x86_64 -cpu host,taa-no=off
|
||||
|
||||
And you would be able to migrate between them. It is responsibility
|
||||
of the management application or of the user to make sure that the
|
||||
configuration is correct. QEMU doesn't know how to look at this kind
|
||||
of features in general.
|
||||
|
||||
Notice that we don't recommend to use -cpu host for migration. It is
|
||||
used in this example because it makes the example simpler.
|
||||
|
||||
Other devices have worse control about individual features. If they
|
||||
want to be able to migrate between hosts that show different features,
|
||||
the device needs a way to configure which ones it is going to use.
|
||||
|
||||
In this section we have considered that we are using the same QEMU
|
||||
binary in both sides of the migration. If we use different QEMU
|
||||
versions process, then we need to have into account all other
|
||||
differences and the examples become even more complicated.
|
||||
|
||||
How to mitigate when we have a backward compatibility error
|
||||
-----------------------------------------------------------
|
||||
|
||||
We broke migration for old machine types continuously during
|
||||
development. But as soon as we find that there is a problem, we fix
|
||||
it. The problem is what happens when we detect after we have done a
|
||||
release that something has gone wrong.
|
||||
|
||||
Let see how it worked with one example.
|
||||
|
||||
After the release of qemu-8.0 we found a problem when doing migration
|
||||
of the machine type pc-7.2.
|
||||
|
||||
- $ qemu-7.2 -M pc-7.2 -> qemu-7.2 -M pc-7.2
|
||||
|
||||
This migration works
|
||||
|
||||
- $ qemu-8.0 -M pc-7.2 -> qemu-8.0 -M pc-7.2
|
||||
|
||||
This migration works
|
||||
|
||||
- $ qemu-8.0 -M pc-7.2 -> qemu-7.2 -M pc-7.2
|
||||
|
||||
This migration fails
|
||||
|
||||
- $ qemu-7.2 -M pc-7.2 -> qemu-8.0 -M pc-7.2
|
||||
|
||||
This migration fails
|
||||
|
||||
So clearly something fails when migration between qemu-7.2 and
|
||||
qemu-8.0 with machine type pc-7.2. The error messages, and git bisect
|
||||
pointed to this commit.
|
||||
|
||||
In qemu-8.0 we got this commit::
|
||||
|
||||
commit 010746ae1db7f52700cb2e2c46eb94f299cfa0d2
|
||||
Author: Jonathan Cameron <[email protected]>
|
||||
Date: Thu Mar 2 13:37:02 2023 +0000
|
||||
|
||||
hw/pci/aer: Implement PCI_ERR_UNCOR_MASK register
|
||||
|
||||
|
||||
The relevant bits of the commit for our example are this ones::
|
||||
|
||||
--- a/hw/pci/pcie_aer.c
|
||||
+++ b/hw/pci/pcie_aer.c
|
||||
@@ -112,6 +112,10 @@ int pcie_aer_init(PCIDevice *dev,
|
||||
|
||||
pci_set_long(dev->w1cmask + offset + PCI_ERR_UNCOR_STATUS,
|
||||
PCI_ERR_UNC_SUPPORTED);
|
||||
+ pci_set_long(dev->config + offset + PCI_ERR_UNCOR_MASK,
|
||||
+ PCI_ERR_UNC_MASK_DEFAULT);
|
||||
+ pci_set_long(dev->wmask + offset + PCI_ERR_UNCOR_MASK,
|
||||
+ PCI_ERR_UNC_SUPPORTED);
|
||||
|
||||
pci_set_long(dev->config + offset + PCI_ERR_UNCOR_SEVER,
|
||||
PCI_ERR_UNC_SEVERITY_DEFAULT);
|
||||
|
||||
The patch changes how we configure PCI space for AER. But QEMU fails
|
||||
when the PCI space configuration is different between source and
|
||||
destination.
|
||||
|
||||
The following commit shows how this got fixed::
|
||||
|
||||
commit 5ed3dabe57dd9f4c007404345e5f5bf0e347317f
|
||||
Author: Leonardo Bras <[email protected]>
|
||||
Date: Tue May 2 21:27:02 2023 -0300
|
||||
|
||||
hw/pci: Disable PCI_ERR_UNCOR_MASK register for machine type < 8.0
|
||||
|
||||
[...]
|
||||
|
||||
The relevant parts of the fix in QEMU are as follow:
|
||||
|
||||
First, we create a new property for the device to be able to configure
|
||||
the old behaviour or the new behaviour::
|
||||
|
||||
diff --git a/hw/pci/pci.c b/hw/pci/pci.c
|
||||
index 8a87ccc8b0..5153ad63d6 100644
|
||||
--- a/hw/pci/pci.c
|
||||
+++ b/hw/pci/pci.c
|
||||
@@ -79,6 +79,8 @@ static const Property pci_props[] = {
|
||||
DEFINE_PROP_STRING("failover_pair_id", PCIDevice,
|
||||
failover_pair_id),
|
||||
DEFINE_PROP_UINT32("acpi-index", PCIDevice, acpi_index, 0),
|
||||
+ DEFINE_PROP_BIT("x-pcie-err-unc-mask", PCIDevice, cap_present,
|
||||
+ QEMU_PCIE_ERR_UNC_MASK_BITNR, true),
|
||||
};
|
||||
|
||||
Notice that we enable the feature for new machine types.
|
||||
|
||||
Now we see how the fix is done. This is going to depend on what kind
|
||||
of breakage happens, but in this case it is quite simple::
|
||||
|
||||
diff --git a/hw/pci/pcie_aer.c b/hw/pci/pcie_aer.c
|
||||
index 103667c368..374d593ead 100644
|
||||
--- a/hw/pci/pcie_aer.c
|
||||
+++ b/hw/pci/pcie_aer.c
|
||||
@@ -112,10 +112,13 @@ int pcie_aer_init(PCIDevice *dev, uint8_t cap_ver,
|
||||
uint16_t offset,
|
||||
|
||||
pci_set_long(dev->w1cmask + offset + PCI_ERR_UNCOR_STATUS,
|
||||
PCI_ERR_UNC_SUPPORTED);
|
||||
- pci_set_long(dev->config + offset + PCI_ERR_UNCOR_MASK,
|
||||
- PCI_ERR_UNC_MASK_DEFAULT);
|
||||
- pci_set_long(dev->wmask + offset + PCI_ERR_UNCOR_MASK,
|
||||
- PCI_ERR_UNC_SUPPORTED);
|
||||
+
|
||||
+ if (dev->cap_present & QEMU_PCIE_ERR_UNC_MASK) {
|
||||
+ pci_set_long(dev->config + offset + PCI_ERR_UNCOR_MASK,
|
||||
+ PCI_ERR_UNC_MASK_DEFAULT);
|
||||
+ pci_set_long(dev->wmask + offset + PCI_ERR_UNCOR_MASK,
|
||||
+ PCI_ERR_UNC_SUPPORTED);
|
||||
+ }
|
||||
|
||||
pci_set_long(dev->config + offset + PCI_ERR_UNCOR_SEVER,
|
||||
PCI_ERR_UNC_SEVERITY_DEFAULT);
|
||||
|
||||
I.e. If the property bit is enabled, we configure it as we did for
|
||||
qemu-8.0. If the property bit is not set, we configure it as it was in 7.2.
|
||||
|
||||
And now, everything that is missing is disabling the feature for old
|
||||
machine types::
|
||||
|
||||
diff --git a/hw/core/machine.c b/hw/core/machine.c
|
||||
index 47a34841a5..07f763eb2e 100644
|
||||
--- a/hw/core/machine.c
|
||||
+++ b/hw/core/machine.c
|
||||
@@ -48,6 +48,7 @@ GlobalProperty hw_compat_7_2[] = {
|
||||
{ "e1000e", "migrate-timadj", "off" },
|
||||
{ "virtio-mem", "x-early-migration", "false" },
|
||||
{ "migration", "x-preempt-pre-7-2", "true" },
|
||||
+ { TYPE_PCI_DEVICE, "x-pcie-err-unc-mask", "off" },
|
||||
};
|
||||
const size_t hw_compat_7_2_len = G_N_ELEMENTS(hw_compat_7_2);
|
||||
|
||||
And now, when qemu-8.0.1 is released with this fix, all combinations
|
||||
are going to work as supposed.
|
||||
|
||||
- $ qemu-7.2 -M pc-7.2 -> qemu-7.2 -M pc-7.2 (works)
|
||||
- $ qemu-8.0.1 -M pc-7.2 -> qemu-8.0.1 -M pc-7.2 (works)
|
||||
- $ qemu-8.0.1 -M pc-7.2 -> qemu-7.2 -M pc-7.2 (works)
|
||||
- $ qemu-7.2 -M pc-7.2 -> qemu-8.0.1 -M pc-7.2 (works)
|
||||
|
||||
So the normality has been restored and everything is ok, no?
|
||||
|
||||
Not really, now our matrix is much bigger. We started with the easy
|
||||
cases, migration from the same version to the same version always
|
||||
works:
|
||||
|
||||
- $ qemu-7.2 -M pc-7.2 -> qemu-7.2 -M pc-7.2
|
||||
- $ qemu-8.0 -M pc-7.2 -> qemu-8.0 -M pc-7.2
|
||||
- $ qemu-8.0.1 -M pc-7.2 -> qemu-8.0.1 -M pc-7.2
|
||||
|
||||
Now the interesting ones. When the QEMU processes versions are
|
||||
different. For the 1st set, their fail and we can do nothing, both
|
||||
versions are released and we can't change anything.
|
||||
|
||||
- $ qemu-7.2 -M pc-7.2 -> qemu-8.0 -M pc-7.2
|
||||
- $ qemu-8.0 -M pc-7.2 -> qemu-7.2 -M pc-7.2
|
||||
|
||||
This two are the ones that work. The whole point of making the
|
||||
change in qemu-8.0.1 release was to fix this issue:
|
||||
|
||||
- $ qemu-7.2 -M pc-7.2 -> qemu-8.0.1 -M pc-7.2
|
||||
- $ qemu-8.0.1 -M pc-7.2 -> qemu-7.2 -M pc-7.2
|
||||
|
||||
But now we found that qemu-8.0 neither can migrate to qemu-7.2 not
|
||||
qemu-8.0.1.
|
||||
|
||||
- $ qemu-8.0 -M pc-7.2 -> qemu-8.0.1 -M pc-7.2
|
||||
- $ qemu-8.0.1 -M pc-7.2 -> qemu-8.0 -M pc-7.2
|
||||
|
||||
So, if we start a pc-7.2 machine in qemu-8.0 we can't migrate it to
|
||||
anything except to qemu-8.0.
|
||||
|
||||
Can we do better?
|
||||
|
||||
Yeap. If we know that we are going to do this migration:
|
||||
|
||||
- $ qemu-8.0 -M pc-7.2 -> qemu-8.0.1 -M pc-7.2
|
||||
|
||||
We can launch the appropriate devices with::
|
||||
|
||||
--device...,x-pci-e-err-unc-mask=on
|
||||
|
||||
And now we can receive a migration from 8.0. And from now on, we can
|
||||
do that migration to new machine types if we remember to enable that
|
||||
property for pc-7.2. Notice that we need to remember, it is not
|
||||
enough to know that the source of the migration is qemu-8.0. Think of
|
||||
this example:
|
||||
|
||||
$ qemu-8.0 -M pc-7.2 -> qemu-8.0.1 -M pc-7.2 -> qemu-8.2 -M pc-7.2
|
||||
|
||||
In the second migration, the source is not qemu-8.0, but we still have
|
||||
that "problem" and have that property enabled. Notice that we need to
|
||||
continue having this mark/property until we have this machine
|
||||
rebooted. But it is not a normal reboot (that don't reload QEMU) we
|
||||
need the machine to poweroff/poweron on a fixed QEMU. And from now
|
||||
on we can use the proper real machine.
|
||||
@@ -0,0 +1,71 @@
|
||||
Dirty limit
|
||||
===========
|
||||
|
||||
The dirty limit, short for dirty page rate upper limit, is a new capability
|
||||
introduced in the 8.1 QEMU release that uses a new algorithm based on the KVM
|
||||
dirty ring to throttle down the guest during live migration.
|
||||
|
||||
The algorithm framework is as follows:
|
||||
|
||||
::
|
||||
|
||||
------------------------------------------------------------------------------
|
||||
main --------------> throttle thread ------------> PREPARE(1) <--------
|
||||
thread \ | |
|
||||
\ | |
|
||||
\ V |
|
||||
-\ CALCULATE(2) |
|
||||
\ | |
|
||||
\ | |
|
||||
\ V |
|
||||
\ SET PENALTY(3) -----
|
||||
-\ |
|
||||
\ |
|
||||
\ V
|
||||
-> virtual CPU thread -------> ACCEPT PENALTY(4)
|
||||
------------------------------------------------------------------------------
|
||||
|
||||
When the qmp command qmp_set_vcpu_dirty_limit is called for the first time,
|
||||
the QEMU main thread starts the throttle thread. The throttle thread, once
|
||||
launched, executes the loop, which consists of three steps:
|
||||
|
||||
- PREPARE (1)
|
||||
|
||||
The entire work of PREPARE (1) is preparation for the second stage,
|
||||
CALCULATE(2), as the name implies. It involves preparing the dirty
|
||||
page rate value and the corresponding upper limit of the VM:
|
||||
The dirty page rate is calculated via the KVM dirty ring mechanism,
|
||||
which tells QEMU how many dirty pages a virtual CPU has had since the
|
||||
last KVM_EXIT_DIRTY_RING_FULL exception; The dirty page rate upper
|
||||
limit is specified by caller, therefore fetch it directly.
|
||||
|
||||
- CALCULATE (2)
|
||||
|
||||
Calculate a suitable sleep period for each virtual CPU, which will be
|
||||
used to determine the penalty for the target virtual CPU. The
|
||||
computation must be done carefully in order to reduce the dirty page
|
||||
rate progressively down to the upper limit without oscillation. To
|
||||
achieve this, two strategies are provided: the first is to add or
|
||||
subtract sleep time based on the ratio of the current dirty page rate
|
||||
to the limit, which is used when the current dirty page rate is far
|
||||
from the limit; the second is to add or subtract a fixed time when
|
||||
the current dirty page rate is close to the limit.
|
||||
|
||||
- SET PENALTY (3)
|
||||
|
||||
Set the sleep time for each virtual CPU that should be penalized based
|
||||
on the results of the calculation supplied by step CALCULATE (2).
|
||||
|
||||
After completing the three above stages, the throttle thread loops back
|
||||
to step PREPARE (1) until the dirty limit is reached.
|
||||
|
||||
On the other hand, each virtual CPU thread reads the sleep duration and
|
||||
sleeps in the path of the KVM_EXIT_DIRTY_RING_FULL exception handler, that
|
||||
is ACCEPT PENALTY (4). Virtual CPUs tied with writing processes will
|
||||
obviously exit to the path and get penalized, whereas virtual CPUs involved
|
||||
with read processes will not.
|
||||
|
||||
In summary, thanks to the KVM dirty ring technology, the dirty limit
|
||||
algorithm will restrict virtual CPUs as needed to keep their dirty page
|
||||
rate inside the limit. This leads to more steady reading performance during
|
||||
live migration and can aid in improving large guest responsiveness.
|
||||
@@ -0,0 +1,82 @@
|
||||
==================
|
||||
Fast Snapshot Load
|
||||
==================
|
||||
|
||||
Overview
|
||||
========
|
||||
Fast snapshot load is an extension of the postcopy migration feature
|
||||
to disk loads.
|
||||
|
||||
Unlike a usual snapshot load, which requires all VM data (RAM as well
|
||||
as device states) to be loaded into host RAM from the snapshot file
|
||||
for the guest to run, fast snapshot load uses postcopy infrastructure
|
||||
to load in only the required device states and allows RAM pages to
|
||||
be loaded after the guest starts execution. The idea is to start the
|
||||
guest and serve its page faults on the go, reducing the perceived
|
||||
resume time for large snapshots.
|
||||
|
||||
Architecture
|
||||
============
|
||||
This feature combines postcopy migration and mapped-ram capabilities
|
||||
to load RAM pages on demand. It is done by catching guest faults using
|
||||
Linux ``userfaultfd`` and loading the page by calculating the offset
|
||||
of its location in the snapshot file using mapped-ram capabilities.
|
||||
|
||||
Fault Thread
|
||||
------------
|
||||
The fault thread uses Linux ``userfaultfd`` to catch page faults caused
|
||||
by guest and directly load the page from the snapshot file. It is
|
||||
very similar to network postcopy fault thread, with primary difference
|
||||
being it loads pages directly by reading from the snapshot file.
|
||||
|
||||
Eager Thread
|
||||
------------
|
||||
Eager thread iterates over all pages in RAM and loads each page not
|
||||
yet loaded by fault thread. It is required as unlike network postcopy
|
||||
where majority of RAM has already been loaded via precopy, here entire
|
||||
RAM is waiting to be loaded. If there is no eager loading thread each
|
||||
page will only be loaded when it is required by guest. In case there
|
||||
are some background pages that are never/rarely accessed by guest,
|
||||
the system will be locked in migration state indefinitely.
|
||||
|
||||
Synchronization
|
||||
---------------
|
||||
In order to make sure both of these threads do not load the same page
|
||||
twice potentially overwriting and corrupting user RAM, a bitmap is
|
||||
used (``RAMBlock->pending_bmap``) which tracks the pages claimed to
|
||||
be loaded by threads. This prevents race condition when one thread
|
||||
is loading the page and other one tries to do the same.
|
||||
|
||||
Usage
|
||||
=====
|
||||
|
||||
Simply enable ``mapped-ram`` and ``postcopy-ram`` capabilities on
|
||||
the destination:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
migrate_set_capability mapped-ram on
|
||||
migrate_set_capability postcopy-ram on
|
||||
|
||||
Use a ``file:`` URI for migration:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
migrate_incoming file:/path/to/snapshot/file
|
||||
|
||||
Limitations
|
||||
===========
|
||||
|
||||
- Multifd
|
||||
Fast snapshot load is currently incompatible with ``multifd``
|
||||
capability. While ``mapped-ram`` allows for parallel disk I/O,
|
||||
coupling it with ``postcopy`` capability requires additional
|
||||
infrastructure.
|
||||
|
||||
- Host OS support
|
||||
Because this feautre essentially depends on ``userfaultfd``
|
||||
to trap page faults, it is supported only on Linux hosts.
|
||||
|
||||
- vhost-user
|
||||
Fast snapshot load does not currently support ``vhost-user``
|
||||
backends.
|
||||
@@ -0,0 +1,19 @@
|
||||
Migration features
|
||||
==================
|
||||
|
||||
Migration has plenty of features to support different use cases.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
postcopy
|
||||
dirty-limit
|
||||
vfio
|
||||
virtio
|
||||
mapped-ram
|
||||
fast-snapshot-load
|
||||
CPR
|
||||
qpl-compression
|
||||
uadk-compression
|
||||
qatzip-compression
|
||||
xbzrle
|
||||
@@ -0,0 +1,13 @@
|
||||
Migration
|
||||
=========
|
||||
|
||||
This is the main entry for QEMU migration documentations. It explains how
|
||||
QEMU live migration works.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
main
|
||||
features
|
||||
compatibility
|
||||
best-practices
|
||||
@@ -0,0 +1,618 @@
|
||||
.. _migration:
|
||||
|
||||
===================
|
||||
Migration framework
|
||||
===================
|
||||
|
||||
QEMU has code to load/save the state of the guest that it is running.
|
||||
These are two complementary operations. Saving the state just does
|
||||
that, saves the state for each device that the guest is running.
|
||||
Restoring a guest is just the opposite operation: we need to load the
|
||||
state of each device.
|
||||
|
||||
For this to work, QEMU has to be launched with the same arguments the
|
||||
two times. I.e. it can only restore the state in one guest that has
|
||||
the same devices that the one it was saved (this last requirement can
|
||||
be relaxed a bit, but for now we can consider that configuration has
|
||||
to be exactly the same).
|
||||
|
||||
Once that we are able to save/restore a guest, a new functionality is
|
||||
requested: migration. This means that QEMU is able to start in one
|
||||
machine and being "migrated" to another machine. I.e. being moved to
|
||||
another machine.
|
||||
|
||||
Next was the "live migration" functionality. This is important
|
||||
because some guests run with a lot of state (specially RAM), and it
|
||||
can take a while to move all state from one machine to another. Live
|
||||
migration allows the guest to continue running while the state is
|
||||
transferred. Only while the last part of the state is transferred has
|
||||
the guest to be stopped. Typically the time that the guest is
|
||||
unresponsive during live migration is the low hundred of milliseconds
|
||||
(notice that this depends on a lot of things).
|
||||
|
||||
.. contents::
|
||||
|
||||
Transports
|
||||
==========
|
||||
|
||||
The migration stream is normally just a byte stream that can be passed
|
||||
over any transport.
|
||||
|
||||
- tcp migration: do the migration using tcp sockets
|
||||
- unix migration: do the migration using unix sockets
|
||||
- exec migration: do the migration using the stdin/stdout through a process.
|
||||
- fd migration: do the migration using a file descriptor that is
|
||||
passed to QEMU. QEMU doesn't care how this file descriptor is opened.
|
||||
- file migration: do the migration using a file that is passed to QEMU
|
||||
by path. A file offset option is supported to allow a management
|
||||
application to add its own metadata to the start of the file without
|
||||
QEMU interference. Note that QEMU does not flush cached file
|
||||
data/metadata at the end of migration.
|
||||
|
||||
The file migration also supports using a file that has already been
|
||||
opened. A set of file descriptors is passed to QEMU via an "fdset"
|
||||
(see add-fd QMP command documentation). This method allows a
|
||||
management application to have control over the migration file
|
||||
opening operation. There are, however, strict requirements to this
|
||||
interface if the multifd capability is enabled:
|
||||
|
||||
- the fdset must contain two file descriptors that are not
|
||||
duplicates between themselves;
|
||||
- if the direct-io capability is to be used, exactly one of the
|
||||
file descriptors must have the O_DIRECT flag set;
|
||||
- the file must be opened with WRONLY on the migration source side
|
||||
and RDONLY on the migration destination side.
|
||||
|
||||
- rdma migration: support is included for migration using RDMA, which
|
||||
transports the page data using ``RDMA``, where the hardware takes
|
||||
care of transporting the pages, and the load on the CPU is much
|
||||
lower. While the internals of RDMA migration are a bit different,
|
||||
this isn't really visible outside the RAM migration code.
|
||||
|
||||
All these migration protocols use the same infrastructure to
|
||||
save/restore state devices. This infrastructure is shared with the
|
||||
savevm/loadvm functionality.
|
||||
|
||||
Common infrastructure
|
||||
=====================
|
||||
|
||||
The files, sockets or fd's that carry the migration stream are abstracted by
|
||||
the ``QEMUFile`` type (see ``migration/qemu-file.h``). In most cases this
|
||||
is connected to a subtype of ``QIOChannel`` (see ``io/``).
|
||||
|
||||
|
||||
Saving the state of one device
|
||||
==============================
|
||||
|
||||
For most devices, the state is saved in a single call to the migration
|
||||
infrastructure; these are *non-iterative* devices. The data for these
|
||||
devices is sent at the end of precopy migration, when the CPUs are paused.
|
||||
There are also *iterative* devices, which contain a very large amount of
|
||||
data (e.g. RAM or large tables). See the iterative device section below.
|
||||
|
||||
General advice for device developers
|
||||
------------------------------------
|
||||
|
||||
- The migration state saved should reflect the device being modelled rather
|
||||
than the way your implementation works. That way if you change the implementation
|
||||
later the migration stream will stay compatible. That model may include
|
||||
internal state that's not directly visible in a register.
|
||||
|
||||
- When saving a migration stream the device code may walk and check
|
||||
the state of the device. These checks might fail in various ways (e.g.
|
||||
discovering internal state is corrupt or that the guest has done something bad).
|
||||
Consider carefully before asserting/aborting at this point, since the
|
||||
normal response from users is that *migration broke their VM* since it had
|
||||
apparently been running fine until then. In these error cases, the device
|
||||
should log a message indicating the cause of error, and should consider
|
||||
putting the device into an error state, allowing the rest of the VM to
|
||||
continue execution.
|
||||
|
||||
- The migration might happen at an inconvenient point,
|
||||
e.g. right in the middle of the guest reprogramming the device, during
|
||||
guest reboot or shutdown or while the device is waiting for external IO.
|
||||
It's strongly preferred that migrations do not fail in this situation,
|
||||
since in the cloud environment migrations might happen automatically to
|
||||
VMs that the administrator doesn't directly control.
|
||||
|
||||
- If you do need to fail a migration, ensure that sufficient information
|
||||
is logged to identify what went wrong.
|
||||
|
||||
- The destination should treat an incoming migration stream as hostile
|
||||
(which we do to varying degrees in the existing code). Check that offsets
|
||||
into buffers and the like can't cause overruns. Fail the incoming migration
|
||||
in the case of a corrupted stream like this.
|
||||
|
||||
- Take care with internal device state or behaviour that might become
|
||||
migration version dependent. For example, the order of PCI capabilities
|
||||
is required to stay constant across migration. Another example would
|
||||
be that a special case handled by subsections (see below) might become
|
||||
much more common if a default behaviour is changed.
|
||||
|
||||
- The state of the source should not be changed or destroyed by the
|
||||
outgoing migration. Migrations timing out or being failed by
|
||||
higher levels of management, or failures of the destination host are
|
||||
not unusual, and in that case the VM is restarted on the source.
|
||||
Note that the management layer can validly revert the migration
|
||||
even though the QEMU level of migration has succeeded as long as it
|
||||
does it before starting execution on the destination.
|
||||
|
||||
- Buses and devices should be able to explicitly specify addresses when
|
||||
instantiated, and management tools should use those. For example,
|
||||
when hot adding USB devices it's important to specify the ports
|
||||
and addresses, since implicit ordering based on the command line order
|
||||
may be different on the destination. This can result in the
|
||||
device state being loaded into the wrong device.
|
||||
|
||||
VMState
|
||||
-------
|
||||
|
||||
Most device data can be described using the ``VMSTATE`` macros (mostly defined
|
||||
in ``include/migration/vmstate.h``).
|
||||
|
||||
An example (from hw/input/pckbd.c)
|
||||
|
||||
.. code:: c
|
||||
|
||||
static const VMStateDescription vmstate_kbd = {
|
||||
.name = "pckbd",
|
||||
.version_id = 3,
|
||||
.minimum_version_id = 3,
|
||||
.fields = (const VMStateField[]) {
|
||||
VMSTATE_UINT8(write_cmd, KBDState),
|
||||
VMSTATE_UINT8(status, KBDState),
|
||||
VMSTATE_UINT8(mode, KBDState),
|
||||
VMSTATE_UINT8(pending, KBDState),
|
||||
VMSTATE_END_OF_LIST()
|
||||
}
|
||||
};
|
||||
|
||||
We are declaring the state with name "pckbd". The ``version_id`` is
|
||||
3, and there are 4 uint8_t fields in the KBDState structure. We
|
||||
registered this ``VMSTATEDescription`` with one of the following
|
||||
functions. The first one will generate a device ``instance_id``
|
||||
different for each registration. Use the second one if you already
|
||||
have an id that is different for each instance of the device:
|
||||
|
||||
.. code:: c
|
||||
|
||||
vmstate_register_any(NULL, &vmstate_kbd, s);
|
||||
vmstate_register(NULL, instance_id, &vmstate_kbd, s);
|
||||
|
||||
For devices that are ``qdev`` based, we can register the device in the class
|
||||
init function:
|
||||
|
||||
.. code:: c
|
||||
|
||||
dc->vmsd = &vmstate_kbd_isa;
|
||||
|
||||
The VMState macros take care of ensuring that the device data section
|
||||
is formatted portably (normally big endian) and make some compile time checks
|
||||
against the types of the fields in the structures.
|
||||
|
||||
VMState macros can include other VMStateDescriptions to store substructures
|
||||
(see ``VMSTATE_STRUCT_``), arrays (``VMSTATE_ARRAY_``) and variable length
|
||||
arrays (``VMSTATE_VARRAY_``). Various other macros exist for special
|
||||
cases.
|
||||
|
||||
Note that the format on the wire is still very raw; i.e. a VMSTATE_UINT32
|
||||
ends up with a 4 byte bigendian representation on the wire; in the future
|
||||
it might be possible to use a more structured format.
|
||||
|
||||
Legacy way
|
||||
----------
|
||||
|
||||
This way is going to disappear as soon as all current users are ported to VMSTATE;
|
||||
although converting existing code can be tricky, and thus 'soon' is relative.
|
||||
|
||||
Each device has to register two functions, one to save the state and
|
||||
another to load the state back.
|
||||
|
||||
.. code:: c
|
||||
|
||||
int register_savevm_live(const char *idstr,
|
||||
int instance_id,
|
||||
int version_id,
|
||||
SaveVMHandlers *ops,
|
||||
void *opaque);
|
||||
|
||||
Two functions in the ``ops`` structure are the ``save_state``
|
||||
and ``load_state`` functions. Notice that ``load_state`` receives a version_id
|
||||
parameter to know what state format is receiving. ``save_state`` doesn't
|
||||
have a version_id parameter because it always uses the latest version.
|
||||
|
||||
Note that because the VMState macros still save the data in a raw
|
||||
format, in many cases it's possible to replace legacy code
|
||||
with a carefully constructed VMState description that matches the
|
||||
byte layout of the existing code.
|
||||
|
||||
Changing migration data structures
|
||||
----------------------------------
|
||||
|
||||
When we migrate a device, we save/load the state as a series
|
||||
of fields. Sometimes, due to bugs or new functionality, we need to
|
||||
change the state to store more/different information. Changing the migration
|
||||
state saved for a device can break migration compatibility unless
|
||||
care is taken to use the appropriate techniques. In general QEMU tries
|
||||
to maintain forward migration compatibility (i.e. migrating from
|
||||
QEMU n->n+1) and there are users who benefit from backward compatibility
|
||||
as well.
|
||||
|
||||
Subsections
|
||||
-----------
|
||||
|
||||
The most common structure change is adding new data, e.g. when adding
|
||||
a newer form of device, or adding that state that you previously
|
||||
forgot to migrate. This is best solved using a subsection.
|
||||
|
||||
A subsection is "like" a device vmstate, but with a particularity, it
|
||||
has a Boolean function that tells if that values are needed to be sent
|
||||
or not. If this functions returns false, the subsection is not sent.
|
||||
Subsections have a unique name, that is looked for on the receiving
|
||||
side.
|
||||
|
||||
On the receiving side, if we found a subsection for a device that we
|
||||
don't understand, we just fail the migration. If we understand all
|
||||
the subsections, then we load the state with success. There's no check
|
||||
that a subsection is loaded, so a newer QEMU that knows about a subsection
|
||||
can (with care) load a stream from an older QEMU that didn't send
|
||||
the subsection.
|
||||
|
||||
If the new data is only needed in a rare case, then the subsection
|
||||
can be made conditional on that case and the migration will still
|
||||
succeed to older QEMUs in most cases. This is OK for data that's
|
||||
critical, but in some use cases it's preferred that the migration
|
||||
should succeed even with the data missing. To support this the
|
||||
subsection can be connected to a device property and from there
|
||||
to a versioned machine type.
|
||||
|
||||
The 'pre_load' and 'post_load' functions on subsections are only
|
||||
called if the subsection is loaded.
|
||||
|
||||
One important note is that the outer post_load() function is called "after"
|
||||
loading all subsections, because a newer subsection could change the same
|
||||
value that it uses. A flag, and the combination of outer pre_load and
|
||||
post_load can be used to detect whether a subsection was loaded, and to
|
||||
fall back on default behaviour when the subsection isn't present.
|
||||
|
||||
Example:
|
||||
|
||||
.. code:: c
|
||||
|
||||
static bool ide_drive_pio_state_needed(void *opaque)
|
||||
{
|
||||
IDEState *s = opaque;
|
||||
|
||||
return ((s->status & DRQ_STAT) != 0)
|
||||
|| (s->bus->error_status & BM_STATUS_PIO_RETRY);
|
||||
}
|
||||
|
||||
const VMStateDescription vmstate_ide_drive_pio_state = {
|
||||
.name = "ide_drive/pio_state",
|
||||
.version_id = 1,
|
||||
.minimum_version_id = 1,
|
||||
.pre_save = ide_drive_pio_pre_save,
|
||||
.post_load = ide_drive_pio_post_load,
|
||||
.needed = ide_drive_pio_state_needed,
|
||||
.fields = (const VMStateField[]) {
|
||||
VMSTATE_INT32(req_nb_sectors, IDEState),
|
||||
VMSTATE_VARRAY_INT32(io_buffer, IDEState, io_buffer_total_len, 1,
|
||||
vmstate_info_uint8, uint8_t),
|
||||
VMSTATE_INT32(cur_io_buffer_offset, IDEState),
|
||||
VMSTATE_INT32(cur_io_buffer_len, IDEState),
|
||||
VMSTATE_UINT8(end_transfer_fn_idx, IDEState),
|
||||
VMSTATE_INT32(elementary_transfer_size, IDEState),
|
||||
VMSTATE_INT32(packet_transfer_size, IDEState),
|
||||
VMSTATE_END_OF_LIST()
|
||||
}
|
||||
};
|
||||
|
||||
const VMStateDescription vmstate_ide_drive = {
|
||||
.name = "ide_drive",
|
||||
.version_id = 3,
|
||||
.minimum_version_id = 0,
|
||||
.post_load = ide_drive_post_load,
|
||||
.fields = (const VMStateField[]) {
|
||||
.... several fields ....
|
||||
VMSTATE_END_OF_LIST()
|
||||
},
|
||||
.subsections = (const VMStateDescription * const []) {
|
||||
&vmstate_ide_drive_pio_state,
|
||||
NULL
|
||||
}
|
||||
};
|
||||
|
||||
Here we have a subsection for the pio state. We only need to
|
||||
save/send this state when we are in the middle of a pio operation
|
||||
(that is what ``ide_drive_pio_state_needed()`` checks). If DRQ_STAT is
|
||||
not enabled, the values on that fields are garbage and don't need to
|
||||
be sent.
|
||||
|
||||
Connecting subsections to properties
|
||||
------------------------------------
|
||||
|
||||
Using a condition function that checks a 'property' to determine whether
|
||||
to send a subsection allows backward migration compatibility when
|
||||
new subsections are added, especially when combined with versioned
|
||||
machine types.
|
||||
|
||||
For example:
|
||||
|
||||
a) Add a new property using ``DEFINE_PROP_BOOL`` - e.g. support-foo and
|
||||
default it to true.
|
||||
b) Add an entry to the ``hw_compat_`` for the previous version that sets
|
||||
the property to false.
|
||||
c) Add a static bool support_foo function that tests the property.
|
||||
d) Add a subsection with a .needed set to the support_foo function
|
||||
e) (potentially) Add an outer pre_load that sets up a default value
|
||||
for 'foo' to be used if the subsection isn't loaded.
|
||||
|
||||
Now that subsection will not be generated when using an older
|
||||
machine type and the migration stream will be accepted by older
|
||||
QEMU versions.
|
||||
|
||||
Not sending existing elements
|
||||
-----------------------------
|
||||
|
||||
Sometimes members of the VMState are no longer needed:
|
||||
|
||||
- removing them will break migration compatibility
|
||||
|
||||
- making them version dependent and bumping the version will break backward migration
|
||||
compatibility.
|
||||
|
||||
Adding a dummy field into the migration stream is normally the best way to preserve
|
||||
compatibility.
|
||||
|
||||
If the field really does need to be removed then:
|
||||
|
||||
a) Add a new property/compatibility/function in the same way for subsections above.
|
||||
b) replace the VMSTATE macro with the _TEST version of the macro, e.g.:
|
||||
|
||||
``VMSTATE_UINT32(foo, barstruct)``
|
||||
|
||||
becomes
|
||||
|
||||
``VMSTATE_UINT32_TEST(foo, barstruct, pre_version_baz)``
|
||||
|
||||
Sometime in the future when we no longer care about the ancient versions these can be killed off.
|
||||
Note that for backward compatibility it's important to fill in the structure with
|
||||
data that the destination will understand.
|
||||
|
||||
Any difference in the predicates on the source and destination will end up
|
||||
with different fields being enabled and data being loaded into the wrong
|
||||
fields; for this reason conditional fields like this are very fragile.
|
||||
|
||||
Versions
|
||||
--------
|
||||
|
||||
Version numbers are intended for major incompatible changes to the
|
||||
migration of a device, and using them breaks backward-migration
|
||||
compatibility; in general most changes can be made by adding Subsections
|
||||
(see above) or _TEST macros (see above) which won't break compatibility.
|
||||
|
||||
Each version is associated with a series of fields saved. The ``save_state`` always saves
|
||||
the state as the newer version. But ``load_state`` sometimes is able to
|
||||
load state from an older version.
|
||||
|
||||
You can see that there are two version fields:
|
||||
|
||||
- ``version_id``: the maximum version_id supported by VMState for that device.
|
||||
- ``minimum_version_id``: the minimum version_id that VMState is able to understand
|
||||
for that device.
|
||||
|
||||
VMState is able to read versions from minimum_version_id to version_id.
|
||||
|
||||
There are *_V* forms of many ``VMSTATE_`` macros to load fields for version dependent fields,
|
||||
e.g.
|
||||
|
||||
.. code:: c
|
||||
|
||||
VMSTATE_UINT16_V(ip_id, Slirp, 2),
|
||||
|
||||
only loads that field for versions 2 and newer.
|
||||
|
||||
Saving state will always create a section with the 'version_id' value
|
||||
and thus can't be loaded by any older QEMU.
|
||||
|
||||
Massaging functions
|
||||
-------------------
|
||||
|
||||
Sometimes, it is not enough to be able to save the state directly
|
||||
from one structure, we need to fill the correct values there. One
|
||||
example is when we are using kvm. Before saving the cpu state, we
|
||||
need to ask kvm to copy to QEMU the state that it is using. And the
|
||||
opposite when we are loading the state, we need a way to tell kvm to
|
||||
load the state for the cpu that we have just loaded from the QEMUFile.
|
||||
|
||||
The functions to do that are inside a vmstate definition, and are called:
|
||||
|
||||
- ``int (*pre_load)(void *opaque);``
|
||||
|
||||
This function is called before we load the state of one device.
|
||||
|
||||
- ``int (*post_load)(void *opaque, int version_id);``
|
||||
|
||||
This function is called after we load the state of one device.
|
||||
|
||||
- ``int (*pre_save)(void *opaque);``
|
||||
|
||||
This function is called before we save the state of one device.
|
||||
|
||||
- ``void (*post_save)(void *opaque);``
|
||||
|
||||
This function is called after we save the state of one device
|
||||
(even upon failure, unless the call to pre_save returned an error).
|
||||
|
||||
Following are the errp variants of these functions.
|
||||
|
||||
- ``bool (*pre_load_errp)(void *opaque, Error **errp);``
|
||||
|
||||
This function is called before we load the state of one device.
|
||||
|
||||
- ``bool (*post_load_errp)(void *opaque, int version_id, Error **errp);``
|
||||
|
||||
This function is called after we load the state of one device.
|
||||
|
||||
- ``bool (*pre_save_errp)(void *opaque, Error **errp);``
|
||||
|
||||
This function is called before we save the state of one device.
|
||||
|
||||
New impls should preferentally use 'errp' variants of these
|
||||
methods and existing impls incrementally converted.
|
||||
The variants without 'errp' are intended to be removed
|
||||
once all usage is converted.
|
||||
|
||||
Example: You can look at hpet.c, that uses the first three functions
|
||||
to massage the state that is transferred.
|
||||
|
||||
The ``VMSTATE_WITH_TMP`` macro may be useful when the migration
|
||||
data doesn't match the stored device data well; it allows an
|
||||
intermediate temporary structure to be populated with migration
|
||||
data and then transferred to the main structure.
|
||||
|
||||
If you use memory or portio_list API functions that update memory layout outside
|
||||
initialization (i.e., in response to a guest action), this is a strong
|
||||
indication that you need to call these functions in a ``post_load`` callback.
|
||||
Examples of such API functions are:
|
||||
|
||||
- memory_region_add_subregion()
|
||||
- memory_region_del_subregion()
|
||||
- memory_region_set_readonly()
|
||||
- memory_region_set_nonvolatile()
|
||||
- memory_region_set_enabled()
|
||||
- memory_region_set_address()
|
||||
- memory_region_set_alias_offset()
|
||||
- portio_list_set_address()
|
||||
- portio_list_set_enabled()
|
||||
|
||||
Since the order of device save/restore is not defined, you must
|
||||
avoid accessing or changing any other device's state in one of these
|
||||
callbacks. (For instance, don't do anything that calls ``update_irq()``
|
||||
in a ``post_load`` hook.) Otherwise, restore will not be deterministic,
|
||||
and this will break execution record/replay.
|
||||
|
||||
Iterative device migration
|
||||
--------------------------
|
||||
|
||||
Some devices, such as RAM or certain platform devices,
|
||||
have large amounts of data that would mean that the CPUs would be
|
||||
paused for too long if they were sent in one section. For these
|
||||
devices an *iterative* approach is taken.
|
||||
|
||||
The iterative devices generally don't use VMState macros
|
||||
(although it may be possible in some cases) and instead use
|
||||
qemu_put_*/qemu_get_* macros to read/write data to the stream. Specialist
|
||||
versions exist for high bandwidth IO.
|
||||
|
||||
|
||||
An iterative device must provide:
|
||||
|
||||
- A ``save_setup`` function that initialises the data structures and
|
||||
transmits a first section containing information on the device. In the
|
||||
case of RAM this transmits a list of RAMBlocks and sizes.
|
||||
|
||||
- A ``load_setup`` function that initialises the data structures on the
|
||||
destination.
|
||||
|
||||
- A ``save_query_pending`` function that indicates how much more
|
||||
data we must save.
|
||||
|
||||
- A ``save_live_iterate`` function should send a chunk of data until
|
||||
the point that stream bandwidth limits tell it to stop. Each call
|
||||
generates one section.
|
||||
|
||||
- A ``save_complete`` function that must transmit the last section for
|
||||
the device containing any remaining data.
|
||||
|
||||
- A ``load_state`` function used to load sections generated by
|
||||
any of the save functions that generate sections.
|
||||
|
||||
- ``cleanup`` functions for both save and load that are called
|
||||
at the end of migration.
|
||||
|
||||
Note that the contents of the sections for iterative migration tend
|
||||
to be open-coded by the devices; care should be taken in parsing
|
||||
the results and structuring the stream to make them easy to validate.
|
||||
|
||||
Device ordering
|
||||
---------------
|
||||
|
||||
There are cases in which the ordering of device loading matters; for
|
||||
example in some systems where a device may assert an interrupt during loading,
|
||||
if the interrupt controller is loaded later then it might lose the state.
|
||||
|
||||
Some ordering is implicitly provided by the order in which the machine
|
||||
definition creates devices, however this is somewhat fragile.
|
||||
|
||||
The ``MigrationPriority`` enum provides a means of explicitly enforcing
|
||||
ordering. Numerically higher priorities are loaded earlier.
|
||||
The priority is set by setting the ``priority`` field of the top level
|
||||
``VMStateDescription`` for the device.
|
||||
|
||||
Stream structure
|
||||
================
|
||||
|
||||
The stream tries to be word and endian agnostic, allowing migration between hosts
|
||||
of different characteristics running the same VM.
|
||||
|
||||
- Header
|
||||
|
||||
- Magic
|
||||
- Version
|
||||
- VM configuration section
|
||||
|
||||
- Machine type
|
||||
- Target page bits
|
||||
- List of sections
|
||||
Each section contains a device, or one iteration of a device save.
|
||||
|
||||
- section type
|
||||
- section id
|
||||
- ID string (First section of each device)
|
||||
- instance id (First section of each device)
|
||||
- version id (First section of each device)
|
||||
- <device data>
|
||||
- Footer mark
|
||||
- EOF mark
|
||||
- VM Description structure
|
||||
Consisting of a JSON description of the contents for analysis only
|
||||
|
||||
The ``device data`` in each section consists of the data produced
|
||||
by the code described above. For non-iterative devices they have a single
|
||||
section; iterative devices have an initial and last section and a set
|
||||
of parts in between.
|
||||
Note that there is very little checking by the common code of the integrity
|
||||
of the ``device data`` contents, that's up to the devices themselves.
|
||||
The ``footer mark`` provides a little bit of protection for the case where
|
||||
the receiving side reads more or less data than expected.
|
||||
|
||||
The ``ID string`` is normally unique, having been formed from a bus name
|
||||
and device address, PCI devices and storage devices hung off PCI controllers
|
||||
fit this pattern well. Some devices are fixed single instances (e.g. "pc-ram").
|
||||
Others (especially either older devices or system devices which for
|
||||
some reason don't have a bus concept) make use of the ``instance id``
|
||||
for otherwise identically named devices.
|
||||
|
||||
Return path
|
||||
-----------
|
||||
|
||||
Only a unidirectional stream is required for normal migration, however a
|
||||
``return path`` can be created when bidirectional communication is desired.
|
||||
This is primarily used by postcopy, but is also used to return a success
|
||||
flag to the source at the end of migration.
|
||||
|
||||
``qemu_file_get_return_path(QEMUFile* fwdpath)`` gives the QEMUFile* for the return
|
||||
path.
|
||||
|
||||
Source side
|
||||
|
||||
Forward path - written by migration thread
|
||||
Return path - opened by main thread, read by return-path thread
|
||||
|
||||
Destination side
|
||||
|
||||
Forward path - read by main thread
|
||||
Return path - opened by main thread, written by main thread AND postcopy
|
||||
thread (protected by rp_mutex)
|
||||
|
||||
@@ -0,0 +1,142 @@
|
||||
Mapped-ram
|
||||
==========
|
||||
|
||||
Mapped-ram is a new stream format for the RAM section designed to
|
||||
supplement the existing ``file:`` migration and make it compatible
|
||||
with ``multifd``. This enables parallel migration of a guest's RAM to
|
||||
a file.
|
||||
|
||||
The core of the feature is to ensure that RAM pages are mapped
|
||||
directly to offsets in the resulting migration file. This enables the
|
||||
``multifd`` threads to write exclusively to those offsets even if the
|
||||
guest is constantly dirtying pages (i.e. live migration). Another
|
||||
benefit is that the resulting file will have a bounded size, since
|
||||
pages which are dirtied multiple times will always go to a fixed
|
||||
location in the file, rather than constantly being added to a
|
||||
sequential stream. Having the pages at fixed offsets also allows the
|
||||
usage of O_DIRECT for save/restore of the migration stream as the
|
||||
pages are ensured to be written respecting O_DIRECT alignment
|
||||
restrictions.
|
||||
|
||||
Usage
|
||||
-----
|
||||
|
||||
On both source and destination, enable the ``multifd`` and
|
||||
``mapped-ram`` capabilities:
|
||||
|
||||
``migrate_set_capability multifd on``
|
||||
|
||||
``migrate_set_capability mapped-ram on``
|
||||
|
||||
Use a ``file:`` URL for migration:
|
||||
|
||||
``migrate file:/path/to/migration/file``
|
||||
|
||||
Mapped-ram migration is best done non-live, i.e. by stopping the VM on
|
||||
the source side before migrating.
|
||||
|
||||
For best performance enable the ``direct-io`` parameter as well:
|
||||
|
||||
``migrate_set_parameter direct-io on``
|
||||
|
||||
Use-cases
|
||||
---------
|
||||
|
||||
The mapped-ram feature was designed for use cases where the migration
|
||||
stream will be directed to a file in the filesystem and not
|
||||
immediately restored on the destination VM\ [#alternatives]_. These could be
|
||||
thought of as snapshots. We can further categorize them into live and
|
||||
non-live.
|
||||
|
||||
- Non-live snapshot
|
||||
|
||||
If the use case requires a VM to be stopped before taking a snapshot,
|
||||
that's the ideal scenario for mapped-ram migration. Not having to
|
||||
track dirty pages, the migration will write the RAM pages to the disk
|
||||
as fast as it can.
|
||||
|
||||
Note: if a snapshot is taken of a running VM, but the VM will be
|
||||
stopped after the snapshot by the admin, then consider stopping it
|
||||
right before the snapshot to take benefit of the performance gains
|
||||
mentioned above.
|
||||
|
||||
- Live snapshot
|
||||
|
||||
If the use case requires that the VM keeps running during and after
|
||||
the snapshot operation, then mapped-ram migration can still be used,
|
||||
but will be less performant. Other strategies such as
|
||||
background-snapshot should be evaluated as well. One benefit of
|
||||
mapped-ram in this scenario is portability since background-snapshot
|
||||
depends on async dirty tracking (KVM_GET_DIRTY_LOG) which is not
|
||||
supported outside of Linux.
|
||||
|
||||
.. [#alternatives] While this same effect could be obtained with the usage of
|
||||
snapshots or the ``file:`` migration alone, mapped-ram provides
|
||||
a performance increase for VMs with larger RAM sizes (10s to
|
||||
100s of GiBs), specially if the VM has been stopped beforehand.
|
||||
|
||||
RAM section format
|
||||
------------------
|
||||
|
||||
Instead of having a sequential stream of pages that follow the
|
||||
RAMBlock headers, the dirty pages for a RAMBlock follow its header
|
||||
instead. This ensures that each RAM page has a fixed offset in the
|
||||
resulting migration file.
|
||||
|
||||
A bitmap is introduced to track which pages have been written in the
|
||||
migration file. Pages are written at a fixed location for every
|
||||
ramblock. Zero pages are ignored as they'd be zero in the destination
|
||||
migration as well.
|
||||
|
||||
::
|
||||
|
||||
Without mapped-ram: With mapped-ram:
|
||||
|
||||
--------------------- --------------------------------
|
||||
| ramblock 1 header | | ramblock 1 header |
|
||||
--------------------- --------------------------------
|
||||
| ramblock 2 header | | ramblock 1 mapped-ram header |
|
||||
--------------------- --------------------------------
|
||||
| ... | | padding to next 1MB boundary |
|
||||
--------------------- | ... |
|
||||
| ramblock n header | --------------------------------
|
||||
--------------------- | ramblock 1 pages |
|
||||
| RAM_SAVE_FLAG_EOS | | ... |
|
||||
--------------------- --------------------------------
|
||||
| stream of pages | | ramblock 2 header |
|
||||
| (iter 1) | --------------------------------
|
||||
| ... | | ramblock 2 mapped-ram header |
|
||||
--------------------- --------------------------------
|
||||
| RAM_SAVE_FLAG_EOS | | padding to next 1MB boundary |
|
||||
--------------------- | ... |
|
||||
| stream of pages | --------------------------------
|
||||
| (iter 2) | | ramblock 2 pages |
|
||||
| ... | | ... |
|
||||
--------------------- --------------------------------
|
||||
| ... | | ... |
|
||||
--------------------- --------------------------------
|
||||
| RAM_SAVE_FLAG_EOS |
|
||||
--------------------------------
|
||||
| ... |
|
||||
--------------------------------
|
||||
|
||||
where:
|
||||
- ramblock header: the generic information for a ramblock, such as
|
||||
idstr, used_len, etc.
|
||||
|
||||
- ramblock mapped-ram header: the information added by this feature:
|
||||
bitmap of pages written, bitmap size and offset of pages in the
|
||||
migration file.
|
||||
|
||||
Restrictions
|
||||
------------
|
||||
|
||||
Since pages are written to their relative offsets and out of order
|
||||
(due to the memory dirtying patterns), streaming channels such as
|
||||
sockets are not supported. A seekable channel such as a file is
|
||||
required. This can be verified in the QIOChannel by the presence of
|
||||
the QIO_CHANNEL_FEATURE_SEEKABLE.
|
||||
|
||||
The improvements brought by this feature apply only to guest physical
|
||||
RAM. Other types of memory such as VRAM are migrated as part of device
|
||||
states.
|
||||
@@ -0,0 +1,312 @@
|
||||
========
|
||||
Postcopy
|
||||
========
|
||||
|
||||
.. contents::
|
||||
|
||||
'Postcopy' migration is a way to deal with migrations that refuse to converge
|
||||
(or take too long to converge) its plus side is that there is an upper bound on
|
||||
the amount of migration traffic and time it takes, the down side is that during
|
||||
the postcopy phase, a failure of *either* side causes the guest to be lost.
|
||||
|
||||
In postcopy the destination CPUs are started before all the memory has been
|
||||
transferred, and accesses to pages that are yet to be transferred cause
|
||||
a fault that's translated by QEMU into a request to the source QEMU.
|
||||
|
||||
Postcopy can be combined with precopy (i.e. normal migration) so that if precopy
|
||||
doesn't finish in a given time the switch is made to postcopy.
|
||||
|
||||
Enabling postcopy
|
||||
=================
|
||||
|
||||
To enable postcopy, issue this command on the monitor (both source and
|
||||
destination) prior to the start of migration:
|
||||
|
||||
``migrate_set_capability postcopy-ram on``
|
||||
|
||||
The normal commands are then used to start a migration, which is still
|
||||
started in precopy mode. Issuing:
|
||||
|
||||
``migrate_start_postcopy``
|
||||
|
||||
will now cause the transition from precopy to postcopy.
|
||||
It can be issued immediately after migration is started or any
|
||||
time later on. Issuing it after the end of a migration is harmless.
|
||||
|
||||
Postcopy internals
|
||||
==================
|
||||
|
||||
State machine
|
||||
-------------
|
||||
|
||||
Postcopy moves through a series of states (see postcopy_state) from
|
||||
ADVISE->DISCARD->LISTEN->RUNNING->END
|
||||
|
||||
- Advise
|
||||
|
||||
Set at the start of migration if postcopy is enabled, even
|
||||
if it hasn't had the start command; here the destination
|
||||
checks that its OS has the support needed for postcopy, and performs
|
||||
setup to ensure the RAM mappings are suitable for later postcopy.
|
||||
The destination will fail early in migration at this point if the
|
||||
required OS support is not present.
|
||||
(Triggered by reception of POSTCOPY_ADVISE command)
|
||||
|
||||
- Discard
|
||||
|
||||
Entered on receipt of the first 'discard' command; prior to
|
||||
the first Discard being performed, hugepages are switched off
|
||||
(using madvise) to ensure that no new huge pages are created
|
||||
during the postcopy phase, and to cause any huge pages that
|
||||
have discards on them to be broken.
|
||||
|
||||
- Listen
|
||||
|
||||
The first command in the package, POSTCOPY_LISTEN, switches
|
||||
the destination state to Listen, and starts a new thread
|
||||
(the 'listen thread') which takes over the job of receiving
|
||||
pages off the migration stream, while the main thread carries
|
||||
on processing the blob. With this thread able to process page
|
||||
reception, the destination now 'sensitises' the RAM to detect
|
||||
any access to missing pages (on Linux using the 'userfault'
|
||||
system).
|
||||
|
||||
- Running
|
||||
|
||||
POSTCOPY_RUN causes the destination to synchronise all
|
||||
state and start the CPUs and IO devices running. The main
|
||||
thread now finishes processing the migration package and
|
||||
now carries on as it would for normal precopy migration
|
||||
(although it can't do the cleanup it would do as it
|
||||
finishes a normal migration).
|
||||
|
||||
- End
|
||||
|
||||
The listen thread can now quit, and perform the cleanup of migration
|
||||
state, the migration is now complete.
|
||||
|
||||
Device transfer
|
||||
---------------
|
||||
|
||||
Loading of device data may cause the device emulation to access guest RAM
|
||||
that may trigger faults that have to be resolved by the source, as such
|
||||
the migration stream has to be able to respond with page data *during* the
|
||||
device load, and hence the device data has to be read from the stream completely
|
||||
before the device load begins to free the stream up. This is achieved by
|
||||
'packaging' the device data into a blob that's read in one go.
|
||||
|
||||
Source behaviour
|
||||
----------------
|
||||
|
||||
Until postcopy is entered the migration stream is identical to normal
|
||||
precopy, except for the addition of a 'postcopy advise' command at
|
||||
the beginning, to tell the destination that postcopy might happen.
|
||||
When postcopy starts the source sends the page discard data and then
|
||||
forms the 'package' containing:
|
||||
|
||||
- Command: 'postcopy listen'
|
||||
- The device state
|
||||
|
||||
A series of sections, identical to the precopy streams device state stream
|
||||
containing everything except postcopiable devices (i.e. RAM)
|
||||
- Command: 'postcopy run'
|
||||
|
||||
The 'package' is sent as the data part of a Command: ``CMD_PACKAGED``, and the
|
||||
contents are formatted in the same way as the main migration stream.
|
||||
|
||||
During postcopy the source scans the list of dirty pages and sends them
|
||||
to the destination without being requested (in much the same way as precopy),
|
||||
however when a page request is received from the destination, the dirty page
|
||||
scanning restarts from the requested location. This causes requested pages
|
||||
to be sent quickly, and also causes pages directly after the requested page
|
||||
to be sent quickly in the hope that those pages are likely to be used
|
||||
by the destination soon.
|
||||
|
||||
Destination behaviour
|
||||
---------------------
|
||||
|
||||
Initially the destination looks the same as precopy, with a single thread
|
||||
reading the migration stream; the 'postcopy advise' and 'discard' commands
|
||||
are processed to change the way RAM is managed, but don't affect the stream
|
||||
processing.
|
||||
|
||||
::
|
||||
|
||||
------------------------------------------------------------------------------
|
||||
1 2 3 4 5 6 7
|
||||
main -----DISCARD-CMD_PACKAGED ( LISTEN DEVICE DEVICE DEVICE RUN )
|
||||
thread | |
|
||||
| (page request)
|
||||
| \___
|
||||
v \
|
||||
listen thread: --- page -- page -- page -- page -- page --
|
||||
|
||||
a b c
|
||||
------------------------------------------------------------------------------
|
||||
|
||||
- On receipt of ``CMD_PACKAGED`` (1)
|
||||
|
||||
All the data associated with the package - the ( ... ) section in the diagram -
|
||||
is read into memory, and the main thread recurses into qemu_loadvm_state_main
|
||||
to process the contents of the package (2) which contains commands (3,6) and
|
||||
devices (4...)
|
||||
|
||||
- On receipt of 'postcopy listen' - 3 -(i.e. the 1st command in the package)
|
||||
|
||||
a new thread (a) is started that takes over servicing the migration stream,
|
||||
while the main thread carries on loading the package. It loads normal
|
||||
background page data (b) but if during a device load a fault happens (5)
|
||||
the returned page (c) is loaded by the listen thread allowing the main
|
||||
threads device load to carry on.
|
||||
|
||||
- The last thing in the ``CMD_PACKAGED`` is a 'RUN' command (6)
|
||||
|
||||
letting the destination CPUs start running. At the end of the
|
||||
``CMD_PACKAGED`` (7) the main thread returns to normal running behaviour and
|
||||
is no longer used by migration, while the listen thread carries on servicing
|
||||
page data until the end of migration.
|
||||
|
||||
Source side page bitmap
|
||||
-----------------------
|
||||
|
||||
The 'migration bitmap' in postcopy is basically the same as in the precopy,
|
||||
where each of the bit to indicate that page is 'dirty' - i.e. needs
|
||||
sending. During the precopy phase this is updated as the CPU dirties
|
||||
pages, however during postcopy the CPUs are stopped and nothing should
|
||||
dirty anything any more. Instead, dirty bits are cleared when the relevant
|
||||
pages are sent during postcopy.
|
||||
|
||||
Postcopy features
|
||||
=================
|
||||
|
||||
Postcopy recovery
|
||||
-----------------
|
||||
|
||||
Comparing to precopy, postcopy is special on error handlings. When any
|
||||
error happens (in this case, mostly network errors), QEMU cannot easily
|
||||
fail a migration because VM data resides in both source and destination
|
||||
QEMU instances. On the other hand, when issue happens QEMU on both sides
|
||||
will go into a paused state. It'll need a recovery phase to continue a
|
||||
paused postcopy migration.
|
||||
|
||||
The recovery phase normally contains a few steps:
|
||||
|
||||
- When network issue occurs, both QEMU will go into **POSTCOPY_PAUSED**
|
||||
migration state.
|
||||
|
||||
- When the network is recovered (or a new network is provided), the admin
|
||||
can setup the new channel for migration using QMP command
|
||||
'migrate-recover' on destination node, preparing for a resume.
|
||||
|
||||
- On source host, the admin can continue the interrupted postcopy
|
||||
migration using QMP command 'migrate' with resume=true flag set.
|
||||
Source QEMU will go into **POSTCOPY_RECOVER_SETUP** state trying to
|
||||
re-establish the channels.
|
||||
|
||||
- When both sides of QEMU successfully reconnect using a new or fixed up
|
||||
channel, they will go into **POSTCOPY_RECOVER** state, some handshake
|
||||
procedure will be needed to properly synchronize the VM states between
|
||||
the two QEMUs to continue the postcopy migration. For example, there
|
||||
can be pages sent right during the window when the network is
|
||||
interrupted, then the handshake will guarantee pages lost in-flight
|
||||
will be resent again.
|
||||
|
||||
- After a proper handshake synchronization, QEMU will continue the
|
||||
postcopy migration on both sides and go back to **POSTCOPY_ACTIVE**
|
||||
state. Postcopy migration will continue.
|
||||
|
||||
During a paused postcopy migration, the VM can logically still continue
|
||||
running, and it will not be impacted from any page access to pages that
|
||||
were already migrated to destination VM before the interruption happens.
|
||||
However, if any of the missing pages got accessed on destination VM, the VM
|
||||
thread will be halted waiting for the page to be migrated, it means it can
|
||||
be halted until the recovery is complete.
|
||||
|
||||
The impact of accessing missing pages can be relevant to different
|
||||
configurations of the guest. For example, when with async page fault
|
||||
enabled, logically the guest can proactively schedule out the threads
|
||||
accessing missing pages.
|
||||
|
||||
Postcopy with hugepages
|
||||
-----------------------
|
||||
|
||||
Postcopy now works with hugetlbfs backed memory:
|
||||
|
||||
a) The linux kernel on the destination must support userfault on hugepages.
|
||||
b) The huge-page configuration on the source and destination VMs must be
|
||||
identical; i.e. RAMBlocks on both sides must use the same page size.
|
||||
c) Note that ``-mem-path /dev/hugepages`` will fall back to allocating normal
|
||||
RAM if it doesn't have enough hugepages, triggering (b) to fail.
|
||||
Using ``-mem-prealloc`` enforces the allocation using hugepages.
|
||||
d) Care should be taken with the size of hugepage used; postcopy with 2MB
|
||||
hugepages works well, however 1GB hugepages are likely to be problematic
|
||||
since it takes ~1 second to transfer a 1GB hugepage across a 10Gbps link,
|
||||
and until the full page is transferred the destination thread is blocked.
|
||||
|
||||
Postcopy with shared memory
|
||||
---------------------------
|
||||
|
||||
Postcopy migration with shared memory needs explicit support from the other
|
||||
processes that share memory and from QEMU. There are restrictions on the type of
|
||||
memory that userfault can support shared.
|
||||
|
||||
The Linux kernel userfault support works on ``/dev/shm`` memory and on ``hugetlbfs``
|
||||
(although the kernel doesn't provide an equivalent to ``madvise(MADV_DONTNEED)``
|
||||
for hugetlbfs which may be a problem in some configurations).
|
||||
|
||||
The vhost-user code in QEMU supports clients that have Postcopy support,
|
||||
and the ``vhost-user-bridge`` (in ``tests/``) and the DPDK package have changes
|
||||
to support postcopy.
|
||||
|
||||
The client needs to open a userfaultfd and register the areas
|
||||
of memory that it maps with userfault. The client must then pass the
|
||||
userfaultfd back to QEMU together with a mapping table that allows
|
||||
fault addresses in the clients address space to be converted back to
|
||||
RAMBlock/offsets. The client's userfaultfd is added to the postcopy
|
||||
fault-thread and page requests are made on behalf of the client by QEMU.
|
||||
QEMU performs 'wake' operations on the client's userfaultfd to allow it
|
||||
to continue after a page has arrived.
|
||||
|
||||
.. note::
|
||||
There are two future improvements that would be nice:
|
||||
a) Some way to make QEMU ignorant of the addresses in the clients
|
||||
address space
|
||||
b) Avoiding the need for QEMU to perform ufd-wake calls after the
|
||||
pages have arrived
|
||||
|
||||
Retro-fitting postcopy to existing clients is possible:
|
||||
a) A mechanism is needed for the registration with userfault as above,
|
||||
and the registration needs to be coordinated with the phases of
|
||||
postcopy. In vhost-user extra messages are added to the existing
|
||||
control channel.
|
||||
b) Any thread that can block due to guest memory accesses must be
|
||||
identified and the implication understood; for example if the
|
||||
guest memory access is made while holding a lock then all other
|
||||
threads waiting for that lock will also be blocked.
|
||||
|
||||
Postcopy preemption mode
|
||||
------------------------
|
||||
|
||||
Postcopy preempt is a new capability introduced in 8.0 QEMU release, it
|
||||
allows urgent pages (those got page fault requested from destination QEMU
|
||||
explicitly) to be sent in a separate preempt channel, rather than queued in
|
||||
the background migration channel. Anyone who cares about latencies of page
|
||||
faults during a postcopy migration should enable this feature. By default,
|
||||
it's not enabled.
|
||||
|
||||
Postcopy blocktime statistics
|
||||
-----------------------------
|
||||
|
||||
Blocktime is a postcopy live migration metric, intended to show how
|
||||
long the vCPU was in state of interruptible sleep due to pagefault.
|
||||
That metric is calculated both for all vCPUs as overlapped value, and
|
||||
separately for each vCPU. These values are calculated on destination
|
||||
side. To enable postcopy blocktime calculation, enter following
|
||||
command on destination monitor:
|
||||
|
||||
``migrate_set_capability postcopy-blocktime on``
|
||||
|
||||
Postcopy blocktime can be retrieved by query-migrate qmp command.
|
||||
postcopy-blocktime value of qmp command will show overlapped blocking
|
||||
time for all vCPU, postcopy-vcpu-blocktime will show list of blocking
|
||||
time per vCPU.
|
||||
@@ -0,0 +1,165 @@
|
||||
==================
|
||||
QATzip Compression
|
||||
==================
|
||||
In scenarios with limited network bandwidth, the ``QATzip`` solution can help
|
||||
users save a lot of host CPU resources by accelerating compression and
|
||||
decompression through the Intel QuickAssist Technology(``QAT``) hardware.
|
||||
|
||||
|
||||
The following test was conducted using 8 multifd channels and 10Gbps network
|
||||
bandwidth. The results show that, compared to zstd, ``QATzip`` significantly
|
||||
saves CPU resources on the sender and reduces migration time. Compared to the
|
||||
uncompressed solution, ``QATzip`` greatly improves the dirty page processing
|
||||
capability, indicated by the Pages per Second metric, and also reduces the
|
||||
total migration time.
|
||||
|
||||
::
|
||||
|
||||
VM Configuration: 16 vCPU and 64G memory
|
||||
VM Workload: all vCPUs are idle and 54G memory is filled with Silesia data.
|
||||
QAT Devices: 4
|
||||
|-----------|--------|---------|----------|----------|------|------|
|
||||
|8 Channels |Total |down |throughput|pages per | send | recv |
|
||||
| |time(ms)|time(ms) |(mbps) |second | cpu %| cpu% |
|
||||
|-----------|--------|---------|----------|----------|------|------|
|
||||
|qatzip | 16630| 28| 10467| 2940235| 160| 360|
|
||||
|-----------|--------|---------|----------|----------|------|------|
|
||||
|zstd | 20165| 24| 8579| 2391465| 810| 340|
|
||||
|-----------|--------|---------|----------|----------|------|------|
|
||||
|none | 46063| 40| 10848| 330240| 45| 85|
|
||||
|-----------|--------|---------|----------|----------|------|------|
|
||||
|
||||
|
||||
QATzip Compression Framework
|
||||
============================
|
||||
|
||||
``QATzip`` is a user space library which builds on top of the Intel QuickAssist
|
||||
Technology to provide extended accelerated compression and decompression
|
||||
services.
|
||||
|
||||
For more ``QATzip`` introduction, please refer to `QATzip Introduction
|
||||
<https://github.com/intel/QATzip?tab=readme-ov-file#introductionl>`_
|
||||
|
||||
::
|
||||
|
||||
+----------------+
|
||||
| MultiFd Thread |
|
||||
+-------+--------+
|
||||
|
|
||||
| compress/decompress
|
||||
+-------+--------+
|
||||
| QATzip library |
|
||||
+-------+--------+
|
||||
|
|
||||
+-------+--------+
|
||||
| QAT library |
|
||||
+-------+--------+
|
||||
| user space
|
||||
--------+---------------------
|
||||
| kernel space
|
||||
+------+-------+
|
||||
| QAT Driver |
|
||||
+------+-------+
|
||||
|
|
||||
+------+-------+
|
||||
| QAT Devices |
|
||||
+--------------+
|
||||
|
||||
|
||||
QATzip Installation
|
||||
-------------------
|
||||
|
||||
The ``QATzip`` installation package has been integrated into some Linux
|
||||
distributions and can be installed directly. For example, the Ubuntu Server
|
||||
24.04 LTS system can be installed using below command
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
#apt search qatzip
|
||||
libqatzip-dev/noble 1.2.0-0ubuntu3 amd64
|
||||
Intel QuickAssist user space library development files
|
||||
|
||||
libqatzip3/noble 1.2.0-0ubuntu3 amd64
|
||||
Intel QuickAssist user space library
|
||||
|
||||
qatzip/noble,now 1.2.0-0ubuntu3 amd64 [installed]
|
||||
Compression user-space tool for Intel QuickAssist Technology
|
||||
|
||||
#sudo apt install libqatzip-dev libqatzip3 qatzip
|
||||
|
||||
If your system does not support the ``QATzip`` installation package, you can
|
||||
use the source code to build and install, please refer to `QATzip source code installation
|
||||
<https://github.com/intel/QATzip?tab=readme-ov-file#build-intel-quickassist-technology-driver>`_
|
||||
|
||||
QAT Hardware Deployment
|
||||
-----------------------
|
||||
|
||||
``QAT`` supports physical functions(PFs) and virtual functions(VFs) for
|
||||
deployment, and users can configure ``QAT`` resources for migration according
|
||||
to actual needs. For more details about ``QAT`` deployment, please refer to
|
||||
`Intel QuickAssist Technology Documentation
|
||||
<https://intel.github.io/quickassist/index.html>`_
|
||||
|
||||
For more ``QAT`` hardware introduction, please refer to `intel-quick-assist-technology-overview
|
||||
<https://www.intel.com/content/www/us/en/architecture-and-technology/intel-quick-assist-technology-overview.html>`_
|
||||
|
||||
How To Use QATzip Compression
|
||||
=============================
|
||||
|
||||
1 - Install ``QATzip`` library
|
||||
|
||||
2 - Build ``QEMU`` with ``--enable-qatzip`` parameter
|
||||
|
||||
E.g. configure --target-list=x86_64-softmmu --enable-kvm ``--enable-qatzip``
|
||||
|
||||
3 - Set ``migrate_set_parameter multifd-compression qatzip``
|
||||
|
||||
4 - Set ``migrate_set_parameter multifd-qatzip-level comp_level``, the default
|
||||
comp_level value is 1, and it supports levels from 1 to 9
|
||||
|
||||
QAT Memory Requirements
|
||||
=======================
|
||||
|
||||
The user needs to reserve system memory for the QAT memory management to
|
||||
allocate DMA memory. The size of the reserved system memory depends on the
|
||||
number of devices used for migration and the number of multifd channels.
|
||||
|
||||
Because memory usage depends on QAT configuration, please refer to `QAT Memory
|
||||
Driver Queries
|
||||
<https://intel.github.io/quickassist/PG/infrastructure_debugability.html?highlight=memory>`_
|
||||
for memory usage calculation.
|
||||
|
||||
.. list-table:: An example of a PF used for migration
|
||||
:header-rows: 1
|
||||
|
||||
* - Number of channels
|
||||
- Sender memory usage
|
||||
- Receiver memory usage
|
||||
* - 2
|
||||
- 10M
|
||||
- 10M
|
||||
* - 4
|
||||
- 12M
|
||||
- 14M
|
||||
* - 8
|
||||
- 16M
|
||||
- 20M
|
||||
|
||||
How To Choose Between QATzip and QPL
|
||||
====================================
|
||||
Starting from 4th Gen Intel Xeon Scalable processors, codenamed Sapphire Rapids
|
||||
processor(``SPR``), multiple built-in accelerators are supported including
|
||||
``QAT`` and ``IAA``. The former can accelerate ``QATzip`` and the latter is
|
||||
used to accelerate ``QPL``.
|
||||
|
||||
Here are some suggestions:
|
||||
|
||||
1 - If the live migration scenario is limited by network bandwidth and ``QAT``
|
||||
hardware resources exceed ``IAA``, use the ``QATzip`` method, which can save a
|
||||
lot of host CPU resources for compression.
|
||||
|
||||
2 - If the system cannot support shared virtual memory (SVM) technology, use
|
||||
the ``QATzip`` method because ``QPL`` performance is not good without SVM
|
||||
support.
|
||||
|
||||
3 - For other scenarios, use the ``QPL`` method first.
|
||||
@@ -0,0 +1,260 @@
|
||||
===============
|
||||
QPL Compression
|
||||
===============
|
||||
The Intel Query Processing Library (Intel ``QPL``) is an open-source library to
|
||||
provide compression and decompression features and it is based on deflate
|
||||
compression algorithm (RFC 1951).
|
||||
|
||||
The ``QPL`` compression relies on Intel In-Memory Analytics Accelerator(``IAA``)
|
||||
and Shared Virtual Memory(``SVM``) technology, they are new features supported
|
||||
from Intel 4th Gen Intel Xeon Scalable processors, codenamed Sapphire Rapids
|
||||
processor(``SPR``).
|
||||
|
||||
For more ``QPL`` introduction, please refer to `QPL Introduction
|
||||
<https://intel.github.io/qpl/documentation/introduction_docs/introduction.html>`_
|
||||
|
||||
QPL Compression Framework
|
||||
=========================
|
||||
|
||||
::
|
||||
|
||||
+----------------+ +------------------+
|
||||
| MultiFD Thread | |accel-config tool |
|
||||
+-------+--------+ +--------+---------+
|
||||
| |
|
||||
| |
|
||||
|compress/decompress |
|
||||
+-------+--------+ | Setup IAA
|
||||
| QPL library | | Resources
|
||||
+-------+---+----+ |
|
||||
| | |
|
||||
| +-------------+-------+
|
||||
| Open IAA |
|
||||
| Devices +-----+-----+
|
||||
| |idxd driver|
|
||||
| +-----+-----+
|
||||
| |
|
||||
| |
|
||||
| +-----+-----+
|
||||
+-----------+IAA Devices|
|
||||
Submit jobs +-----------+
|
||||
via enqcmd
|
||||
|
||||
|
||||
QPL Build And Installation
|
||||
--------------------------
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$git clone --recursive https://github.com/intel/qpl.git qpl
|
||||
$mkdir qpl/build
|
||||
$cd qpl/build
|
||||
$cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr -DQPL_LIBRARY_TYPE=SHARED ..
|
||||
$sudo cmake --build . --target install
|
||||
|
||||
For more details about ``QPL`` installation, please refer to `QPL Installation
|
||||
<https://intel.github.io/qpl/documentation/get_started_docs/installation.html>`_
|
||||
|
||||
IAA Device Management
|
||||
---------------------
|
||||
|
||||
The number of ``IAA`` devices will vary depending on the Xeon product model.
|
||||
On a ``SPR`` server, there can be a maximum of 8 ``IAA`` devices, with up to
|
||||
4 devices per socket.
|
||||
|
||||
By default, all ``IAA`` devices are disabled and need to be configured and
|
||||
enabled by users manually.
|
||||
|
||||
Check the number of devices through the following command
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
#lspci -d 8086:0cfe
|
||||
6a:02.0 System peripheral: Intel Corporation Device 0cfe
|
||||
6f:02.0 System peripheral: Intel Corporation Device 0cfe
|
||||
74:02.0 System peripheral: Intel Corporation Device 0cfe
|
||||
79:02.0 System peripheral: Intel Corporation Device 0cfe
|
||||
e7:02.0 System peripheral: Intel Corporation Device 0cfe
|
||||
ec:02.0 System peripheral: Intel Corporation Device 0cfe
|
||||
f1:02.0 System peripheral: Intel Corporation Device 0cfe
|
||||
f6:02.0 System peripheral: Intel Corporation Device 0cfe
|
||||
|
||||
IAA Device Configuration And Enabling
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The ``accel-config`` tool is used to enable ``IAA`` devices and configure
|
||||
``IAA`` hardware resources(work queues and engines). One ``IAA`` device
|
||||
has 8 work queues and 8 processing engines, multiple engines can be assigned
|
||||
to a work queue via ``group`` attribute.
|
||||
|
||||
For ``accel-config`` installation, please refer to `accel-config installation
|
||||
<https://github.com/intel/idxd-config>`_
|
||||
|
||||
One example of configuring and enabling an ``IAA`` device.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
#accel-config config-engine iax1/engine1.0 -g 0
|
||||
#accel-config config-engine iax1/engine1.1 -g 0
|
||||
#accel-config config-engine iax1/engine1.2 -g 0
|
||||
#accel-config config-engine iax1/engine1.3 -g 0
|
||||
#accel-config config-engine iax1/engine1.4 -g 0
|
||||
#accel-config config-engine iax1/engine1.5 -g 0
|
||||
#accel-config config-engine iax1/engine1.6 -g 0
|
||||
#accel-config config-engine iax1/engine1.7 -g 0
|
||||
#accel-config config-wq iax1/wq1.0 -g 0 -s 128 -p 10 -b 1 -t 128 -m shared -y user -n app1 -d user
|
||||
#accel-config enable-device iax1
|
||||
#accel-config enable-wq iax1/wq1.0
|
||||
|
||||
.. note::
|
||||
IAX is an early name for IAA
|
||||
|
||||
- The ``IAA`` device index is 1, use ``ls -lh /sys/bus/dsa/devices/iax*``
|
||||
command to query the ``IAA`` device index.
|
||||
|
||||
- 8 engines and 1 work queue are configured in group 0, so all compression jobs
|
||||
submitted to this work queue can be processed by all engines at the same time.
|
||||
|
||||
- Set work queue attributes including the work mode, work queue size and so on.
|
||||
|
||||
- Enable the ``IAA1`` device and work queue 1.0
|
||||
|
||||
.. note::
|
||||
|
||||
Set work queue mode to shared mode, since ``QPL`` library only supports
|
||||
shared mode
|
||||
|
||||
For more detailed configuration, please refer to `IAA Configuration Samples
|
||||
<https://github.com/intel/idxd-config/tree/stable/Documentation/accfg>`_
|
||||
|
||||
IAA Unit Test
|
||||
^^^^^^^^^^^^^
|
||||
|
||||
- Enabling ``IAA`` devices for Xeon platform, please refer to `IAA User Guide
|
||||
<https://www.intel.com/content/www/us/en/content-details/780887/intel-in-memory-analytics-accelerator-intel-iaa.html>`_
|
||||
|
||||
- ``IAA`` device driver is Intel Data Accelerator Driver (idxd), it is
|
||||
recommended that the minimum version of Linux kernel is 5.18.
|
||||
|
||||
- Add ``"intel_iommu=on,sm_on"`` parameter to kernel command line
|
||||
for ``SVM`` feature enabling.
|
||||
|
||||
Here is an easy way to verify ``IAA`` device driver and ``SVM`` with `iaa_test
|
||||
<https://github.com/intel/idxd-config/tree/stable/test>`_
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
#./test/iaa_test
|
||||
[ info] alloc wq 0 shared size 128 addr 0x7f26cebe5000 batch sz 0xfffffffe xfer sz 0x80000000
|
||||
[ info] test noop: tflags 0x1 num_desc 1
|
||||
[ info] preparing descriptor for noop
|
||||
[ info] Submitted all noop jobs
|
||||
[ info] verifying task result for 0x16f7e20
|
||||
[ info] test with op 0 passed
|
||||
|
||||
|
||||
IAA Resources Allocation For Migration
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
There is no ``IAA`` resource configuration parameters for migration and
|
||||
``accel-config`` tool configuration cannot directly specify the ``IAA``
|
||||
resources used for migration.
|
||||
|
||||
The multifd migration with ``QPL`` compression method will use all work
|
||||
queues that are enabled and shared mode.
|
||||
|
||||
.. note::
|
||||
|
||||
Accessing IAA resources requires ``sudo`` command or ``root`` privileges
|
||||
by default. Administrators can modify the IAA device node ownership
|
||||
so that QEMU can use IAA with specified user permissions.
|
||||
|
||||
For example
|
||||
|
||||
#chown -R qemu /dev/iax
|
||||
|
||||
Shared Virtual Memory(SVM) Introduction
|
||||
=======================================
|
||||
|
||||
An ability for an accelerator I/O device to operate in the same virtual
|
||||
memory space of applications on host processors. It also implies the
|
||||
ability to operate from pageable memory, avoiding functional requirements
|
||||
to pin memory for DMA operations.
|
||||
|
||||
When using ``SVM`` technology, users do not need to reserve memory for the
|
||||
``IAA`` device and perform pin memory operation. The ``IAA`` device can
|
||||
directly access data using the virtual address of the process.
|
||||
|
||||
For more ``SVM`` technology, please refer to
|
||||
`Shared Virtual Addressing (SVA) with ENQCMD
|
||||
<https://docs.kernel.org/next/x86/sva.html>`_
|
||||
|
||||
|
||||
How To Use QPL Compression In Migration
|
||||
=======================================
|
||||
|
||||
1 - Installation of ``QPL`` library and ``accel-config`` library if using IAA
|
||||
|
||||
2 - Configure and enable ``IAA`` devices and work queues via ``accel-config``
|
||||
|
||||
3 - Build ``QEMU`` with ``--enable-qpl`` parameter
|
||||
|
||||
E.g. configure --target-list=x86_64-softmmu --enable-kvm ``--enable-qpl``
|
||||
|
||||
4 - Enable ``QPL`` compression during migration
|
||||
|
||||
Set ``migrate_set_parameter multifd-compression qpl`` when migrating, the
|
||||
``QPL`` compression does not support configuring the compression level, it
|
||||
only supports one compression level.
|
||||
|
||||
The Difference Between QPL And ZLIB
|
||||
===================================
|
||||
|
||||
Although both ``QPL`` and ``ZLIB`` are based on the deflate compression
|
||||
algorithm, and ``QPL`` can support the header and tail of ``ZLIB``, ``QPL``
|
||||
is still not fully compatible with the ``ZLIB`` compression in the migration.
|
||||
|
||||
``QPL`` only supports 4K history buffer, and ``ZLIB`` is 32K by default.
|
||||
``ZLIB`` compresses data that ``QPL`` may not decompress correctly and
|
||||
vice versa.
|
||||
|
||||
``QPL`` does not support the ``Z_SYNC_FLUSH`` operation in ``ZLIB`` streaming
|
||||
compression, current ``ZLIB`` implementation uses ``Z_SYNC_FLUSH``, so each
|
||||
``multifd`` thread has a ``ZLIB`` streaming context, and all page compression
|
||||
and decompression are based on this stream. ``QPL`` cannot decompress such data
|
||||
and vice versa.
|
||||
|
||||
The introduction for ``Z_SYNC_FLUSH``, please refer to `Zlib Manual
|
||||
<https://www.zlib.net/manual.html>`_
|
||||
|
||||
The Best Practices
|
||||
==================
|
||||
When user enables the IAA device for ``QPL`` compression, it is recommended
|
||||
to add ``-mem-prealloc`` parameter to the destination boot parameters. This
|
||||
parameter can avoid the occurrence of I/O page fault and reduce the overhead
|
||||
of IAA compression and decompression.
|
||||
|
||||
The example of booting with ``-mem-prealloc`` parameter
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$qemu-system-x86_64 --enable-kvm -cpu host --mem-prealloc ...
|
||||
|
||||
|
||||
An example about I/O page fault measurement of destination without
|
||||
``-mem-prealloc``, the ``svm_prq`` indicates the number of I/O page fault
|
||||
occurrences and processing time.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
#echo 1 > /sys/kernel/debug/iommu/intel/dmar_perf_latency
|
||||
#echo 2 > /sys/kernel/debug/iommu/intel/dmar_perf_latency
|
||||
#echo 3 > /sys/kernel/debug/iommu/intel/dmar_perf_latency
|
||||
#echo 4 > /sys/kernel/debug/iommu/intel/dmar_perf_latency
|
||||
#cat /sys/kernel/debug/iommu/intel/dmar_perf_latency
|
||||
IOMMU: dmar18 Register Base Address: c87fc000
|
||||
<0.1us 0.1us-1us 1us-10us 10us-100us 100us-1ms 1ms-10ms >=10ms min(us) max(us) average(us)
|
||||
inv_iotlb 0 286 123 0 0 0 0 0 1 0
|
||||
inv_devtlb 0 276 133 0 0 0 0 0 2 0
|
||||
inv_iec 0 0 0 0 0 0 0 0 0 0
|
||||
svm_prq 0 0 25206 364 395 0 0 1 556 9
|
||||
@@ -0,0 +1,144 @@
|
||||
=========================================================
|
||||
User Space Accelerator Development Kit (UADK) Compression
|
||||
=========================================================
|
||||
UADK is a general-purpose user space accelerator framework that uses shared
|
||||
virtual addressing (SVA) to provide a unified programming interface for
|
||||
hardware acceleration of cryptographic and compression algorithms.
|
||||
|
||||
UADK includes Unified/User-space-access-intended Accelerator Framework (UACCE),
|
||||
which enables hardware accelerators from different vendors that support SVA to
|
||||
adapt to UADK.
|
||||
|
||||
Currently, HiSilicon Kunpeng hardware accelerators have been registered with
|
||||
UACCE. Through the UADK framework, users can run cryptographic and compression
|
||||
algorithms using hardware accelerators instead of CPUs, freeing up CPU
|
||||
computing power and improving computing performance.
|
||||
|
||||
https://github.com/Linaro/uadk/tree/master/docs
|
||||
|
||||
UADK Framework
|
||||
==============
|
||||
UADK consists of UACCE, vendors' drivers, and an algorithm layer. UADK requires
|
||||
the hardware accelerator to support SVA, and the operating system to support
|
||||
IOMMU and SVA. Hardware accelerators from different vendors are registered as
|
||||
different character devices with UACCE by using kernel-mode drivers of the
|
||||
vendors. A user can access the hardware accelerators by performing user-mode
|
||||
operations on the character devices.
|
||||
|
||||
::
|
||||
|
||||
+----------------------------------+
|
||||
| apps |
|
||||
+----+------------------------+----+
|
||||
| |
|
||||
| |
|
||||
+-------+--------+ +-------+-------+
|
||||
| scheduler | | alg libraries |
|
||||
+-------+--------+ +-------+-------+
|
||||
| |
|
||||
| |
|
||||
| |
|
||||
| +--------+------+
|
||||
| | vendor drivers|
|
||||
| +-+-------------+
|
||||
| |
|
||||
| |
|
||||
+--+------------------+--+
|
||||
| libwd |
|
||||
User +----+-------------+-----+
|
||||
--------------------------------------------------
|
||||
Kernel +--+-----+ +------+
|
||||
| uacce | | smmu |
|
||||
+---+----+ +------+
|
||||
|
|
||||
+---+------------------+
|
||||
| vendor kernel driver |
|
||||
+----------------------+
|
||||
--------------------------------------------------
|
||||
+----------------------+
|
||||
| HW Accelerators |
|
||||
+----------------------+
|
||||
|
||||
UADK Installation
|
||||
-----------------
|
||||
Build UADK
|
||||
^^^^^^^^^^
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
git clone https://github.com/Linaro/uadk.git
|
||||
cd uadk
|
||||
mkdir build
|
||||
./autogen.sh
|
||||
./configure --prefix=$PWD/build
|
||||
make
|
||||
make install
|
||||
|
||||
Without --prefix, UADK will be installed to /usr/local/lib by default.
|
||||
If get error:"cannot find -lnuma", please install the libnuma-dev
|
||||
|
||||
Run pkg-config libwd to ensure env is setup correctly
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
* export PKG_CONFIG_PATH=$PWD/build/lib/pkgconfig
|
||||
* pkg-config libwd --cflags --libs
|
||||
-I/usr/local/include -L/usr/local/lib -lwd
|
||||
|
||||
* export PKG_CONFIG_PATH is required on demand.
|
||||
Not required if UADK is installed to /usr/local/lib
|
||||
|
||||
UADK Host Kernel Requirements
|
||||
-----------------------------
|
||||
User needs to make sure that ``UACCE`` is already supported in Linux kernel.
|
||||
The kernel version should be at least v5.9 with SVA (Shared Virtual
|
||||
Addressing) enabled.
|
||||
|
||||
Kernel Configuration
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
``UACCE`` could be built as module or built-in.
|
||||
|
||||
Here's an example to enable UACCE with hardware accelerator in HiSilicon
|
||||
Kunpeng platform.
|
||||
|
||||
* CONFIG_IOMMU_SVA_LIB=y
|
||||
* CONFIG_ARM_SMMU=y
|
||||
* CONFIG_ARM_SMMU_V3=y
|
||||
* CONFIG_ARM_SMMU_V3_SVA=y
|
||||
* CONFIG_PCI_PASID=y
|
||||
* CONFIG_UACCE=y
|
||||
* CONFIG_CRYPTO_DEV_HISI_QM=y
|
||||
* CONFIG_CRYPTO_DEV_HISI_ZIP=y
|
||||
|
||||
Make sure all these above kernel configurations are selected.
|
||||
|
||||
Accelerator dev node permissions
|
||||
--------------------------------
|
||||
Hardware accelerators (eg: HiSilicon Kunpeng Zip accelerator) gets registered to
|
||||
UADK and char devices are created in dev directory. In order to access resources
|
||||
on hardware accelerator devices, write permission should be provided to user.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ sudo chmod 777 /dev/hisi_zip-*
|
||||
|
||||
How To Use UADK Compression In QEMU Migration
|
||||
---------------------------------------------
|
||||
* Make sure UADK is installed as above
|
||||
* Build ``QEMU`` with ``--enable-uadk`` parameter
|
||||
|
||||
E.g. configure --target-list=aarch64-softmmu --enable-kvm ``--enable-uadk``
|
||||
|
||||
* Enable ``UADK`` compression during migration
|
||||
|
||||
Set ``migrate_set_parameter multifd-compression uadk``
|
||||
|
||||
Since UADK uses Shared Virtual Addressing(SVA) and device access virtual memory
|
||||
directly it is possible that SMMUv3 may encounter page faults while walking the
|
||||
IO page tables. This may impact the performance. In order to mitigate this,
|
||||
please make sure to specify ``-mem-prealloc`` parameter to the destination VM
|
||||
boot parameters.
|
||||
|
||||
Though both UADK and ZLIB are based on the deflate compression algorithm, UADK
|
||||
is not fully compatible with ZLIB. Hence, please make sure to use ``uadk`` on
|
||||
both source and destination during migration.
|
||||
@@ -0,0 +1,274 @@
|
||||
=====================
|
||||
VFIO device migration
|
||||
=====================
|
||||
|
||||
Migration of virtual machine involves saving the state for each device that
|
||||
the guest is running on source host and restoring this saved state on the
|
||||
destination host. This document details how saving and restoring of VFIO
|
||||
devices is done in QEMU.
|
||||
|
||||
Migration of VFIO devices consists of two phases: the optional pre-copy phase,
|
||||
and the stop-and-copy phase. The pre-copy phase is iterative and allows to
|
||||
accommodate VFIO devices that have a large amount of data that needs to be
|
||||
transferred. The iterative pre-copy phase of migration allows for the guest to
|
||||
continue whilst the VFIO device state is transferred to the destination, this
|
||||
helps to reduce the total downtime of the VM. VFIO devices opt-in to pre-copy
|
||||
support by reporting the VFIO_MIGRATION_PRE_COPY flag in the
|
||||
VFIO_DEVICE_FEATURE_MIGRATION ioctl.
|
||||
|
||||
When pre-copy is supported, it's possible to further reduce downtime by
|
||||
enabling "switchover-ack" migration capability.
|
||||
VFIO migration uAPI defines "initial bytes" as part of its pre-copy data stream
|
||||
and recommends that the initial bytes are sent and loaded in the destination
|
||||
before stopping the source VM. Enabling this migration capability will
|
||||
guarantee that and thus, can potentially reduce downtime even further.
|
||||
|
||||
For example, in mlx5 devices, the initial bytes hold metadata used for time
|
||||
consuming pre-allocations of resources on the destination. Although init bytes
|
||||
may be small in size and sending them may take little time, loading them in the
|
||||
destination can take a significant amount of time. Switchover-ack guarantees
|
||||
that this pre-allocation doesn't happen during downtime.
|
||||
|
||||
Initial bytes was originally defined to be monotonically decreasing, however
|
||||
there are cases where a new chunk of initial bytes should be transferred during
|
||||
precopy, e.g., due to a device reconfiguration, etc. The
|
||||
VFIO_PRECOPY_INFO_REINIT feature addresses this and when supported, allows to
|
||||
report a new initial bytes value regardless of any previously reported values.
|
||||
In this case, a new switchover ACK will be requested to make sure the new
|
||||
initial bytes are loaded in the destination before switching over.
|
||||
|
||||
To support migration of multiple devices that might do P2P transactions between
|
||||
themselves, VFIO migration uAPI defines an intermediate P2P quiescent state.
|
||||
While in the P2P quiescent state, P2P DMA transactions cannot be initiated by
|
||||
the device, but the device can respond to incoming ones. Additionally, all
|
||||
outstanding P2P transactions are guaranteed to have been completed by the time
|
||||
the device enters this state.
|
||||
|
||||
All the devices that support P2P migration are first transitioned to the P2P
|
||||
quiescent state and only then are they stopped or started. This makes migration
|
||||
safe P2P-wise, since starting and stopping the devices is not done atomically
|
||||
for all the devices together.
|
||||
|
||||
Thus, multiple VFIO devices migration is allowed only if all the devices
|
||||
support P2P migration. Single VFIO device migration is allowed regardless of
|
||||
P2P migration support.
|
||||
|
||||
A detailed description of the UAPI for VFIO device migration can be found in
|
||||
the comment for the ``vfio_device_mig_state`` structure in the header file
|
||||
linux-headers/linux/vfio.h.
|
||||
|
||||
VFIO implements the device hooks for the iterative approach as follows:
|
||||
|
||||
* A ``save_setup`` function that sets up migration on the source.
|
||||
|
||||
* A ``load_setup`` function that sets the VFIO device on the destination in
|
||||
_RESUMING state.
|
||||
|
||||
* A ``save_query_pending`` function that reports the remaining data that
|
||||
the vendor driver has yet to save for the VFIO device.
|
||||
|
||||
* An ``is_active_iterate`` function that indicates ``save_live_iterate`` is
|
||||
active only when the VFIO device is in pre-copy states.
|
||||
|
||||
* A ``save_live_iterate`` function that reads the VFIO device's data from the
|
||||
vendor driver during iterative pre-copy phase.
|
||||
|
||||
* A ``switchover_start`` function that in the multifd mode starts a thread that
|
||||
reassembles the multifd received data and loads it in-order into the device.
|
||||
In the non-multifd mode this function is a NOP.
|
||||
|
||||
* A ``save_state`` function to save the device config space if it is present
|
||||
in the non-multifd mode.
|
||||
In the multifd mode it just emits either a dummy EOS marker.
|
||||
|
||||
* A ``save_complete`` function that sets the VFIO device in _STOP_COPY
|
||||
state and iteratively copies the data for the VFIO device until the
|
||||
vendor driver indicates that no data remains. In the multifd mode it
|
||||
just emits a dummy EOS marker.
|
||||
|
||||
* A ``save_complete_precopy_thread`` function that in the multifd mode
|
||||
provides thread handler performing multifd device state transfer.
|
||||
It sets the VFIO device to _STOP_COPY state, iteratively reads the data
|
||||
from the VFIO device and queues it for multifd transmission until the vendor
|
||||
driver indicates that no data remains.
|
||||
After that, it saves the device config space and queues it for multifd
|
||||
transfer too.
|
||||
In the non-multifd mode this thread is a NOP.
|
||||
|
||||
* A ``load_state`` function that loads the config section and the data
|
||||
sections that are generated by the save functions above.
|
||||
|
||||
* A ``load_state_buffer`` function that loads the device state and the device
|
||||
config that arrived via multifd channels.
|
||||
It's used only in the multifd mode.
|
||||
|
||||
* ``cleanup`` functions for both save and load that perform any migration
|
||||
related cleanup.
|
||||
|
||||
|
||||
The VFIO migration code uses a VM state change handler to change the VFIO
|
||||
device state when the VM state changes from running to not-running, and
|
||||
vice versa.
|
||||
|
||||
Similarly, a migration state change handler is used to trigger a transition of
|
||||
the VFIO device state when certain changes of the migration state occur. For
|
||||
example, the VFIO device state is transitioned back to _RUNNING in case a
|
||||
migration failed or was canceled.
|
||||
|
||||
System memory dirty pages tracking
|
||||
----------------------------------
|
||||
|
||||
A ``log_global_start`` and ``log_global_stop`` memory listener callback informs
|
||||
the VFIO dirty tracking module to start and stop dirty page tracking. A
|
||||
``log_sync`` memory listener callback queries the dirty page bitmap from the
|
||||
dirty tracking module and marks system memory pages which were DMA-ed by the
|
||||
VFIO device as dirty. The dirty page bitmap is queried per container.
|
||||
|
||||
Currently there are two ways dirty page tracking can be done:
|
||||
(1) Device dirty tracking:
|
||||
In this method the device is responsible to log and report its DMAs. This
|
||||
method can be used only if the device is capable of tracking its DMAs.
|
||||
Discovering device capability, starting and stopping dirty tracking, and
|
||||
syncing the dirty bitmaps from the device are done using the DMA logging uAPI.
|
||||
More info about the uAPI can be found in the comments of the
|
||||
``vfio_device_feature_dma_logging_control`` and
|
||||
``vfio_device_feature_dma_logging_report`` structures in the header file
|
||||
linux-headers/linux/vfio.h.
|
||||
|
||||
(2) VFIO IOMMU module:
|
||||
In this method dirty tracking is done by IOMMU. However, there is currently no
|
||||
IOMMU support for dirty page tracking. For this reason, all pages are
|
||||
perpetually marked dirty, unless the device driver pins pages through external
|
||||
APIs in which case only those pinned pages are perpetually marked dirty.
|
||||
|
||||
If the above two methods are not supported, all pages are perpetually marked
|
||||
dirty by QEMU.
|
||||
|
||||
By default, dirty pages are tracked during pre-copy as well as stop-and-copy
|
||||
phase. So, a page marked as dirty will be copied to the destination in both
|
||||
phases. Copying dirty pages in pre-copy phase helps QEMU to predict if it can
|
||||
achieve its downtime tolerances. If QEMU during pre-copy phase keeps finding
|
||||
dirty pages continuously, then it understands that even in stop-and-copy phase,
|
||||
it is likely to find dirty pages and can predict the downtime accordingly.
|
||||
|
||||
QEMU also provides a per device opt-out option ``pre-copy-dirty-page-tracking``
|
||||
which disables querying the dirty bitmap during pre-copy phase. If it is set to
|
||||
off, all dirty pages will be copied to the destination in stop-and-copy phase
|
||||
only.
|
||||
|
||||
System memory dirty pages tracking when vIOMMU is enabled
|
||||
---------------------------------------------------------
|
||||
|
||||
With vIOMMU, an IO virtual address range can get unmapped while in pre-copy
|
||||
phase of migration. In that case, the unmap ioctl returns any dirty pages in
|
||||
that range and QEMU reports corresponding guest physical pages dirty. During
|
||||
stop-and-copy phase, an IOMMU notifier is used to get a callback for mapped
|
||||
pages and then dirty pages bitmap is fetched from VFIO IOMMU modules for those
|
||||
mapped ranges. If device dirty tracking is enabled with vIOMMU, live migration
|
||||
will be blocked.
|
||||
|
||||
Flow of state changes during Live migration
|
||||
===========================================
|
||||
|
||||
Below is the state change flow during live migration for a VFIO device that
|
||||
supports both precopy and P2P migration. The flow for devices that don't
|
||||
support it is similar, except that the relevant states for precopy and P2P are
|
||||
skipped.
|
||||
The values in the parentheses represent the VM state, the migration state, and
|
||||
the VFIO device state, respectively.
|
||||
|
||||
Live migration save path
|
||||
------------------------
|
||||
|
||||
::
|
||||
|
||||
QEMU normal running state
|
||||
(RUNNING, _NONE, _RUNNING)
|
||||
|
|
||||
migrate_init spawns migration_thread
|
||||
Migration thread then calls each device's .save_setup()
|
||||
(RUNNING, _SETUP, _PRE_COPY)
|
||||
|
|
||||
(RUNNING, _ACTIVE, _PRE_COPY)
|
||||
If device is active, get pending_bytes by .state_pending_{estimate,exact}()
|
||||
If total pending_bytes >= threshold_size, call .save_live_iterate()
|
||||
Data of VFIO device for pre-copy phase is copied
|
||||
Iterate till total pending bytes converge and are less than threshold
|
||||
|
|
||||
On migration completion, the vCPUs and the VFIO device are stopped
|
||||
The VFIO device is first put in P2P quiescent state
|
||||
(FINISH_MIGRATE, _ACTIVE, _PRE_COPY_P2P)
|
||||
|
|
||||
Then the VFIO device is put in _STOP_COPY state
|
||||
(FINISH_MIGRATE, _ACTIVE, _STOP_COPY)
|
||||
.save_complete() is called for each active device
|
||||
For the VFIO device: in the non-multifd mode iterate in
|
||||
.save_complete() until
|
||||
pending data is 0
|
||||
In the multifd mode this iteration is done in
|
||||
.save_complete_precopy_thread() instead.
|
||||
|
|
||||
(POSTMIGRATE, _COMPLETED, _STOP_COPY)
|
||||
Migraton thread schedules cleanup bottom half and exits
|
||||
|
|
||||
.save_cleanup() is called
|
||||
(POSTMIGRATE, _COMPLETED, _STOP)
|
||||
|
||||
Live migration resume path
|
||||
--------------------------
|
||||
|
||||
::
|
||||
|
||||
Incoming migration calls .load_setup() for each device
|
||||
(RESTORE_VM, _ACTIVE, _STOP)
|
||||
|
|
||||
For each device, .load_state() is called for that device section data
|
||||
transmitted via the main migration channel.
|
||||
For data transmitted via multifd channels .load_state_buffer() is called
|
||||
instead.
|
||||
(RESTORE_VM, _ACTIVE, _RESUMING)
|
||||
|
|
||||
At the end, .load_cleanup() is called for each device and vCPUs are started
|
||||
The VFIO device is first put in P2P quiescent state
|
||||
(RUNNING, _ACTIVE, _RUNNING_P2P)
|
||||
|
|
||||
(RUNNING, _NONE, _RUNNING)
|
||||
|
||||
Postcopy
|
||||
========
|
||||
|
||||
Postcopy migration is currently not supported for VFIO devices.
|
||||
|
||||
Multifd
|
||||
=======
|
||||
|
||||
Starting from QEMU version 10.0 there's a possibility to transfer VFIO device
|
||||
_STOP_COPY state via multifd channels. This helps reduce downtime - especially
|
||||
with multiple VFIO devices or with devices having a large migration state.
|
||||
As an additional benefit, setting the VFIO device to _STOP_COPY state and
|
||||
saving its config space is also parallelized (run in a separate thread) in
|
||||
such migration mode.
|
||||
|
||||
The multifd VFIO device state transfer is controlled by
|
||||
"x-migration-multifd-transfer" VFIO device property. This property defaults to
|
||||
AUTO, which means that VFIO device state transfer via multifd channels is
|
||||
attempted in configurations that otherwise support it.
|
||||
|
||||
Since the target QEMU needs to load device state buffers in-order it needs to
|
||||
queue incoming buffers until they can be loaded into the device.
|
||||
This means that a malicious QEMU source could theoretically cause the target
|
||||
QEMU to allocate unlimited amounts of memory for such buffers-in-flight.
|
||||
|
||||
The "x-migration-max-queued-buffers-size" property allows capping the total size
|
||||
of these VFIO device state buffers queued at the destination.
|
||||
|
||||
Because a malicious QEMU source causing OOM on the target is not expected to be
|
||||
a realistic threat in most of VFIO live migration use cases and the right value
|
||||
depends on the particular setup by default this queued buffers size limit is
|
||||
disabled by setting it to UINT64_MAX.
|
||||
|
||||
Some host platforms (like ARM64) require that VFIO device config is loaded only
|
||||
after all iterables were loaded, during non-iterables loading phase.
|
||||
Such interlocking is controlled by "x-migration-load-config-after-iter" VFIO
|
||||
device property, which in its default setting (AUTO) does so only on platforms
|
||||
that actually require it.
|
||||
@@ -0,0 +1,115 @@
|
||||
=======================
|
||||
Virtio device migration
|
||||
=======================
|
||||
|
||||
Copyright 2015 IBM Corp.
|
||||
|
||||
This work is licensed under the terms of the GNU GPL, version 2 or later. See
|
||||
the COPYING file in the top-level directory.
|
||||
|
||||
Saving and restoring the state of virtio devices is a bit of a twisty maze,
|
||||
for several reasons:
|
||||
|
||||
- state is distributed between several parts:
|
||||
|
||||
- virtio core, for common fields like features, number of queues, ...
|
||||
|
||||
- virtio transport (pci, ccw, ...), for the different proxy devices and
|
||||
transport specific state (msix vectors, indicators, ...)
|
||||
|
||||
- virtio device (net, blk, ...), for the different device types and their
|
||||
state (mac address, request queue, ...)
|
||||
|
||||
- most fields are saved via the stream interface; subsequently, subsections
|
||||
have been added to make cross-version migration possible
|
||||
|
||||
This file attempts to document the current procedure and point out some
|
||||
caveats.
|
||||
|
||||
Save state procedure
|
||||
====================
|
||||
|
||||
::
|
||||
|
||||
virtio core virtio transport virtio device
|
||||
----------- ---------------- -------------
|
||||
|
||||
save() function registered
|
||||
via VMState wrapper on
|
||||
device class
|
||||
virtio_save() <----------
|
||||
------> save_config()
|
||||
- save proxy device
|
||||
- save transport-specific
|
||||
device fields
|
||||
- save common device
|
||||
fields
|
||||
- save common virtqueue
|
||||
fields
|
||||
------> save_queue()
|
||||
- save transport-specific
|
||||
virtqueue fields
|
||||
------> save_device()
|
||||
- save device-specific
|
||||
fields
|
||||
- save subsections
|
||||
- device endianness,
|
||||
if changed from
|
||||
default endianness
|
||||
- 64 bit features, if
|
||||
any high feature bit
|
||||
is set
|
||||
- virtio-1 virtqueue
|
||||
fields, if VERSION_1
|
||||
is set
|
||||
|
||||
Load state procedure
|
||||
====================
|
||||
|
||||
::
|
||||
|
||||
virtio core virtio transport virtio device
|
||||
----------- ---------------- -------------
|
||||
|
||||
load() function registered
|
||||
via VMState wrapper on
|
||||
device class
|
||||
virtio_load() <----------
|
||||
------> load_config()
|
||||
- load proxy device
|
||||
- load transport-specific
|
||||
device fields
|
||||
- load common device
|
||||
fields
|
||||
- load common virtqueue
|
||||
fields
|
||||
------> load_queue()
|
||||
- load transport-specific
|
||||
virtqueue fields
|
||||
- notify guest
|
||||
------> load_device()
|
||||
- load device-specific
|
||||
fields
|
||||
- load subsections
|
||||
- device endianness
|
||||
- 64 bit features
|
||||
- virtio-1 virtqueue
|
||||
fields
|
||||
- sanitize endianness
|
||||
- sanitize features
|
||||
- virtqueue index sanity
|
||||
check
|
||||
- feature-dependent setup
|
||||
|
||||
Implications of this setup
|
||||
==========================
|
||||
|
||||
Devices need to be careful in their state processing during load: The
|
||||
load_device() procedure is invoked by the core before subsections have
|
||||
been loaded. Any code that depends on information transmitted in subsections
|
||||
therefore has to be invoked in the device's load() function _after_
|
||||
virtio_load() returned (like e.g. code depending on features).
|
||||
|
||||
Any extension of the state being migrated should be done in subsections
|
||||
added to the core for compatibility reasons. If transport or device specific
|
||||
state is added, core needs to invoke a callback from the new subsection.
|
||||
@@ -0,0 +1,160 @@
|
||||
XBZRLE (Xor Based Zero Run Length Encoding)
|
||||
===========================================
|
||||
|
||||
Using XBZRLE (Xor Based Zero Run Length Encoding) allows for the reduction
|
||||
of VM downtime and the total live-migration time of Virtual machines.
|
||||
It is particularly useful for virtual machines running memory write intensive
|
||||
workloads that are typical of large enterprise applications such as SAP ERP
|
||||
Systems, and generally speaking for any application that uses a sparse memory
|
||||
update pattern.
|
||||
|
||||
Instead of sending the changed guest memory page this solution will send a
|
||||
compressed version of the updates, thus reducing the amount of data sent during
|
||||
live migration.
|
||||
In order to be able to calculate the update, the previous memory pages need to
|
||||
be stored on the source. Those pages are stored in a dedicated cache
|
||||
(hash table) and are accessed by their address.
|
||||
The larger the cache size the better the chances are that the page has already
|
||||
been stored in the cache.
|
||||
A small cache size will result in high cache miss rate.
|
||||
Cache size can be changed before and during migration.
|
||||
|
||||
Format
|
||||
------
|
||||
|
||||
The compression format performs a XOR between the previous and current content
|
||||
of the page, where zero represents an unchanged value.
|
||||
The page data delta is represented by zero and non zero runs.
|
||||
A zero run is represented by its length (in bytes).
|
||||
A non zero run is represented by its length (in bytes) and the new data.
|
||||
The run length is encoded using ULEB128 (http://en.wikipedia.org/wiki/LEB128)
|
||||
|
||||
There can be more than one valid encoding, the sender may send a longer
|
||||
encoding for the benefit of reducing computation cost.
|
||||
|
||||
::
|
||||
|
||||
page = zrun nzrun
|
||||
| zrun nzrun page
|
||||
|
||||
zrun = length
|
||||
|
||||
nzrun = length byte...
|
||||
|
||||
length = uleb128 encoded integer
|
||||
|
||||
On the sender side XBZRLE is used as a compact delta encoding of page updates,
|
||||
retrieving the old page content from the cache (default size of 64MB). The
|
||||
receiving side uses the existing page's content and XBZRLE to decode the new
|
||||
page's content.
|
||||
|
||||
This work was originally based on research results published
|
||||
VEE 2011: Evaluation of Delta Compression Techniques for Efficient Live
|
||||
Migration of Large Virtual Machines by Benoit, Svard, Tordsson and Elmroth.
|
||||
Additionally the delta encoder XBRLE was improved further using the XBZRLE
|
||||
instead.
|
||||
|
||||
XBZRLE has a sustained bandwidth of 2-2.5 GB/s for typical workloads making it
|
||||
ideal for in-line, real-time encoding such as is needed for live-migration.
|
||||
|
||||
Example:
|
||||
|
||||
old buffer:
|
||||
|
||||
.. code:: batch
|
||||
|
||||
1001 zeros
|
||||
05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 68 00 00 6b 00 6d
|
||||
3074 zeros
|
||||
|
||||
new buffer:
|
||||
|
||||
.. code:: batch
|
||||
|
||||
1001 zeros
|
||||
01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 68 00 00 67 00 69
|
||||
3074 zeros
|
||||
|
||||
encoded buffer:
|
||||
|
||||
.. code:: batch
|
||||
|
||||
encoded length 24
|
||||
e9 07 0f 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 03 01 67 01 01 69
|
||||
|
||||
Cache update strategy
|
||||
---------------------
|
||||
|
||||
Keeping the hot pages in the cache is effective for decreasing cache
|
||||
misses. XBZRLE uses a counter as the age of each page. The counter will
|
||||
increase after each ram dirty bitmap sync. When a cache conflict is
|
||||
detected, XBZRLE will only evict pages in the cache that are older than
|
||||
a threshold.
|
||||
|
||||
Usage
|
||||
-----
|
||||
|
||||
1. Verify the destination QEMU version is able to decode the new format::
|
||||
|
||||
(qemu) info migrate_capabilities
|
||||
xbzrle: off
|
||||
...
|
||||
|
||||
2. Activate xbzrle on both source and destination::
|
||||
|
||||
(qemu) migrate_set_capability xbzrle on
|
||||
|
||||
3. Set the XBZRLE cache size - the cache size is in MBytes and should be a
|
||||
power of 2. The cache default value is 64 MBytes (on source only)::
|
||||
|
||||
(qemu) migrate_set_parameter xbzrle-cache-size 256m
|
||||
|
||||
4. Start outgoing migration::
|
||||
|
||||
(qemu) migrate -d tcp:destination.host:4444
|
||||
(qemu) info migrate
|
||||
capabilities: xbzrle: on
|
||||
Migration status: active
|
||||
transferred ram: A kbytes
|
||||
remaining ram: B kbytes
|
||||
total ram: C kbytes
|
||||
total time: D milliseconds
|
||||
duplicate: E pages
|
||||
normal: F pages
|
||||
normal bytes: G kbytes
|
||||
cache size: H bytes
|
||||
xbzrle transferred: I kbytes
|
||||
xbzrle pages: J pages
|
||||
xbzrle cache miss: K pages
|
||||
xbzrle cache miss rate: L
|
||||
xbzrle encoding rate: M
|
||||
xbzrle overflow: N
|
||||
|
||||
xbzrle cache miss: the number of cache misses to date - high cache-miss rate
|
||||
indicates that the cache size is set too low.
|
||||
|
||||
xbzrle overflow: the number of overflows in the decoding which where the delta
|
||||
could not be compressed. This can happen if the changes in the pages are too
|
||||
large or there are many short changes; for example, changing every second byte
|
||||
(half a page).
|
||||
|
||||
Testing: Testing indicated that live migration with XBZRLE was completed in 110
|
||||
seconds, whereas without it would not be able to complete.
|
||||
|
||||
A simple synthetic memory r/w load generator:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
#include <stdlib.h>
|
||||
#include <stdio.h>
|
||||
int main()
|
||||
{
|
||||
char *buf = (char *) calloc(4096, 4096);
|
||||
while (1) {
|
||||
int i;
|
||||
for (i = 0; i < 4096 * 4; i++) {
|
||||
buf[i * 4096 / 4]++;
|
||||
}
|
||||
printf(".");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
============
|
||||
QEMU modules
|
||||
============
|
||||
|
||||
.. kernel-doc:: include/qemu/module.h
|
||||
@@ -0,0 +1,969 @@
|
||||
Multi-process QEMU
|
||||
===================
|
||||
|
||||
.. note::
|
||||
|
||||
This is the design document for multi-process QEMU. It does not
|
||||
necessarily reflect the status of the current implementation, which
|
||||
may lack features or be considerably different from what is described
|
||||
in this document. This document is still useful as a description of
|
||||
the goals and general direction of this feature.
|
||||
|
||||
Please refer to the following wiki for latest details:
|
||||
https://wiki.qemu.org/Features/MultiProcessQEMU
|
||||
|
||||
QEMU is often used as the hypervisor for virtual machines running in the
|
||||
Oracle cloud. Since one of the advantages of cloud computing is the
|
||||
ability to run many VMs from different tenants in the same cloud
|
||||
infrastructure, a guest that compromised its hypervisor could
|
||||
potentially use the hypervisor's access privileges to access data it is
|
||||
not authorized for.
|
||||
|
||||
QEMU can be susceptible to security attacks because it is a large,
|
||||
monolithic program that provides many features to the VMs it services.
|
||||
Many of these features can be configured out of QEMU, but even a reduced
|
||||
configuration QEMU has a large amount of code a guest can potentially
|
||||
attack. Separating QEMU reduces the attack surface by aiding to
|
||||
limit each component in the system to only access the resources that
|
||||
it needs to perform its job.
|
||||
|
||||
QEMU services
|
||||
-------------
|
||||
|
||||
QEMU can be broadly described as providing three main services. One is a
|
||||
VM control point, where VMs can be created, migrated, re-configured, and
|
||||
destroyed. A second is to emulate the CPU instructions within the VM,
|
||||
often accelerated by HW virtualization features such as Intel's VT
|
||||
extensions. Finally, it provides IO services to the VM by emulating HW
|
||||
IO devices, such as disk and network devices.
|
||||
|
||||
A multi-process QEMU
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
A multi-process QEMU involves separating QEMU services into separate
|
||||
host processes. Each of these processes can be given only the privileges
|
||||
it needs to provide its service, e.g., a disk service could be given
|
||||
access only to the disk images it provides, and not be allowed to
|
||||
access other files, or any network devices. An attacker who compromised
|
||||
this service would not be able to use this exploit to access files or
|
||||
devices beyond what the disk service was given access to.
|
||||
|
||||
A QEMU control process would remain, but in multi-process mode, will
|
||||
have no direct interfaces to the VM. During VM execution, it would still
|
||||
provide the user interface to hot-plug devices or live migrate the VM.
|
||||
|
||||
A first step in creating a multi-process QEMU is to separate IO services
|
||||
from the main QEMU program, which would continue to provide CPU
|
||||
emulation. i.e., the control process would also be the CPU emulation
|
||||
process. In a later phase, CPU emulation could be separated from the
|
||||
control process.
|
||||
|
||||
Separating IO services
|
||||
----------------------
|
||||
|
||||
Separating IO services into individual host processes is a good place to
|
||||
begin for a couple of reasons. One is the sheer number of IO devices QEMU
|
||||
can emulate provides a large surface of interfaces which could potentially
|
||||
be exploited, and, indeed, have been a source of exploits in the past.
|
||||
Another is the modular nature of QEMU device emulation code provides
|
||||
interface points where the QEMU functions that perform device emulation
|
||||
can be separated from the QEMU functions that manage the emulation of
|
||||
guest CPU instructions. The devices emulated in the separate process are
|
||||
referred to as remote devices.
|
||||
|
||||
QEMU device emulation
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
QEMU uses an object oriented SW architecture for device emulation code.
|
||||
Configured objects are all compiled into the QEMU binary, then objects
|
||||
are instantiated by name when used by the guest VM. For example, the
|
||||
code to emulate a device named "foo" is always present in QEMU, but its
|
||||
instantiation code is only run when the device is included in the target
|
||||
VM. (e.g., via the QEMU command line as *-device foo*)
|
||||
|
||||
The object model is hierarchical, so device emulation code names its
|
||||
parent object (such as "pci-device" for a PCI device) and QEMU will
|
||||
instantiate a parent object before calling the device's instantiation
|
||||
code.
|
||||
|
||||
Current separation models
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
In order to separate the device emulation code from the CPU emulation
|
||||
code, the device object code must run in a different process. There are
|
||||
a couple of existing QEMU features that can run emulation code
|
||||
separately from the main QEMU process. These are examined below.
|
||||
|
||||
vhost user model
|
||||
^^^^^^^^^^^^^^^^
|
||||
|
||||
Virtio guest device drivers can be connected to vhost user applications
|
||||
in order to perform their IO operations. This model uses special virtio
|
||||
device drivers in the guest and vhost user device objects in QEMU, but
|
||||
once the QEMU vhost user code has configured the vhost user application,
|
||||
mission-mode IO is performed by the application. The vhost user
|
||||
application is a daemon process that can be contacted via a known UNIX
|
||||
domain socket.
|
||||
|
||||
vhost socket
|
||||
''''''''''''
|
||||
|
||||
As mentioned above, one of the tasks of the vhost device object within
|
||||
QEMU is to contact the vhost application and send it configuration
|
||||
information about this device instance. As part of the configuration
|
||||
process, the application can also be sent other file descriptors over
|
||||
the socket, which then can be used by the vhost user application in
|
||||
various ways, some of which are described below.
|
||||
|
||||
vhost MMIO store acceleration
|
||||
'''''''''''''''''''''''''''''
|
||||
|
||||
VMs are often run using HW virtualization features via the KVM kernel
|
||||
driver. This driver allows QEMU to accelerate the emulation of guest CPU
|
||||
instructions by running the guest in a virtual HW mode. When the guest
|
||||
executes instructions that cannot be executed by virtual HW mode,
|
||||
execution returns to the KVM driver so it can inform QEMU to emulate the
|
||||
instructions in SW.
|
||||
|
||||
One of the events that can cause a return to QEMU is when a guest device
|
||||
driver accesses an IO location. QEMU then dispatches the memory
|
||||
operation to the corresponding QEMU device object. In the case of a
|
||||
vhost user device, the memory operation would need to be sent over a
|
||||
socket to the vhost application. This path is accelerated by the QEMU
|
||||
virtio code by setting up an eventfd file descriptor that the vhost
|
||||
application can directly receive MMIO store notifications from the KVM
|
||||
driver, instead of needing them to be sent to the QEMU process first.
|
||||
|
||||
vhost interrupt acceleration
|
||||
''''''''''''''''''''''''''''
|
||||
|
||||
Another optimization used by the vhost application is the ability to
|
||||
directly inject interrupts into the VM via the KVM driver, again,
|
||||
bypassing the need to send the interrupt back to the QEMU process first.
|
||||
The QEMU virtio setup code configures the KVM driver with an eventfd
|
||||
that triggers the device interrupt in the guest when the eventfd is
|
||||
written. This irqfd file descriptor is then passed to the vhost user
|
||||
application program.
|
||||
|
||||
vhost access to guest memory
|
||||
''''''''''''''''''''''''''''
|
||||
|
||||
The vhost application is also allowed to directly access guest memory,
|
||||
instead of needing to send the data as messages to QEMU. This is also
|
||||
done with file descriptors sent to the vhost user application by QEMU.
|
||||
These descriptors can be passed to ``mmap()`` by the vhost application
|
||||
to map the guest address space into the vhost application.
|
||||
|
||||
IOMMUs introduce another level of complexity, since the address given to
|
||||
the guest virtio device to DMA to or from is not a guest physical
|
||||
address. This case is handled by having vhost code within QEMU register
|
||||
as a listener for IOMMU mapping changes. The vhost application maintains
|
||||
a cache of IOMMMU translations: sending translation requests back to
|
||||
QEMU on cache misses, and in turn receiving flush requests from QEMU
|
||||
when mappings are purged.
|
||||
|
||||
applicability to device separation
|
||||
''''''''''''''''''''''''''''''''''
|
||||
|
||||
Much of the vhost model can be re-used by separated device emulation. In
|
||||
particular, the ideas of using a socket between QEMU and the device
|
||||
emulation application, using a file descriptor to inject interrupts into
|
||||
the VM via KVM, and allowing the application to ``mmap()`` the guest
|
||||
should be re used.
|
||||
|
||||
There are, however, some notable differences between how a vhost
|
||||
application works and the needs of separated device emulation. The most
|
||||
basic is that vhost uses custom virtio device drivers which always
|
||||
trigger IO with MMIO stores. A separated device emulation model must
|
||||
work with existing IO device models and guest device drivers. MMIO loads
|
||||
break vhost store acceleration since they are synchronous - guest
|
||||
progress cannot continue until the load has been emulated. By contrast,
|
||||
stores are asynchronous, the guest can continue after the store event
|
||||
has been sent to the vhost application.
|
||||
|
||||
Another difference is that in the vhost user model, a single daemon can
|
||||
support multiple QEMU instances. This is contrary to the security regime
|
||||
desired, in which the emulation application should only be allowed to
|
||||
access the files or devices the VM it's running on behalf of can access.
|
||||
#### qemu-io model
|
||||
|
||||
``qemu-io`` is a test harness used to test changes to the QEMU block backend
|
||||
object code (e.g., the code that implements disk images for disk driver
|
||||
emulation). ``qemu-io`` is not a device emulation application per se, but it
|
||||
does compile the QEMU block objects into a separate binary from the main
|
||||
QEMU one. This could be useful for disk device emulation, since its
|
||||
emulation applications will need to include the QEMU block objects.
|
||||
|
||||
New separation model based on proxy objects
|
||||
-------------------------------------------
|
||||
|
||||
A different model based on proxy objects in the QEMU program
|
||||
communicating with remote emulation programs could provide separation
|
||||
while minimizing the changes needed to the device emulation code. The
|
||||
rest of this section is a discussion of how a proxy object model would
|
||||
work.
|
||||
|
||||
Remote emulation processes
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The remote emulation process will run the QEMU object hierarchy without
|
||||
modification. The device emulation objects will be also be based on the
|
||||
QEMU code, because for anything but the simplest device, it would not be
|
||||
a tractable to re-implement both the object model and the many device
|
||||
backends that QEMU has.
|
||||
|
||||
The processes will communicate with the QEMU process over UNIX domain
|
||||
sockets. The processes can be executed either as standalone processes,
|
||||
or be executed by QEMU. In both cases, the host backends the emulation
|
||||
processes will provide are specified on its command line, as they would
|
||||
be for QEMU. For example:
|
||||
|
||||
::
|
||||
|
||||
disk-proc -blockdev driver=file,node-name=file0,filename=disk-file0 \
|
||||
-blockdev driver=qcow2,node-name=drive0,file=file0
|
||||
|
||||
would indicate process *disk-proc* uses a qcow2 emulated disk named
|
||||
*file0* as its backend.
|
||||
|
||||
Emulation processes may emulate more than one guest controller. A common
|
||||
configuration might be to put all controllers of the same device class
|
||||
(e.g., disk, network, etc.) in a single process, so that all backends of
|
||||
the same type can be managed by a single QMP monitor.
|
||||
|
||||
communication with QEMU
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The first argument to the remote emulation process will be a Unix domain
|
||||
socket that connects with the Proxy object. This is a required argument.
|
||||
|
||||
::
|
||||
|
||||
disk-proc <socket number> <backend list>
|
||||
|
||||
remote process QMP monitor
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Remote emulation processes can be monitored via QMP, similar to QEMU
|
||||
itself. The QMP monitor socket is specified the same as for a QEMU
|
||||
process:
|
||||
|
||||
::
|
||||
|
||||
disk-proc -qmp unix:/tmp/disk-mon,server
|
||||
|
||||
can be monitored over the UNIX socket path */tmp/disk-mon*.
|
||||
|
||||
QEMU command line
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
Each remote device emulated in a remote process on the host is
|
||||
represented as a *-device* of type *pci-proxy-dev*. A socket
|
||||
sub-option to this option specifies the Unix socket that connects
|
||||
to the remote process. An *id* sub-option is required, and it should
|
||||
be the same id as used in the remote process.
|
||||
|
||||
::
|
||||
|
||||
qemu-system-x86_64 ... -device pci-proxy-dev,id=lsi0,socket=3
|
||||
|
||||
can be used to add a device emulated in a remote process
|
||||
|
||||
|
||||
QEMU management of remote processes
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
QEMU is not aware of the type of type of the remote PCI device. It is
|
||||
a pass through device as far as QEMU is concerned.
|
||||
|
||||
communication with emulation process
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
primary channel
|
||||
'''''''''''''''
|
||||
|
||||
The primary channel (referred to as com in the code) is used to bootstrap
|
||||
the remote process. It is also used to pass on device-agnostic commands
|
||||
like reset.
|
||||
|
||||
per-device channels
|
||||
'''''''''''''''''''
|
||||
|
||||
Each remote device communicates with QEMU using a dedicated communication
|
||||
channel. The proxy object sets up this channel using the primary
|
||||
channel during its initialization.
|
||||
|
||||
QEMU device proxy objects
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
QEMU has an object model based on sub-classes inherited from the
|
||||
"object" super-class. The sub-classes that are of interest here are the
|
||||
"device" and "bus" sub-classes whose child sub-classes make up the
|
||||
device tree of a QEMU emulated system.
|
||||
|
||||
The proxy object model will use device proxy objects to replace the
|
||||
device emulation code within the QEMU process. These objects will live
|
||||
in the same place in the object and bus hierarchies as the objects they
|
||||
replace. i.e., the proxy object for an LSI SCSI controller will be a
|
||||
sub-class of the "pci-device" class, and will have the same PCI bus
|
||||
parent and the same SCSI bus child objects as the LSI controller object
|
||||
it replaces.
|
||||
|
||||
It is worth noting that the same proxy object is used to mediate with
|
||||
all types of remote PCI devices.
|
||||
|
||||
object initialization
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The Proxy device objects are initialized in the exact same manner in
|
||||
which any other QEMU device would be initialized.
|
||||
|
||||
In addition, the Proxy objects perform the following two tasks:
|
||||
- Parses the "socket" sub option and connects to the remote process
|
||||
using this channel
|
||||
- Uses the "id" sub-option to connect to the emulated device on the
|
||||
separate process
|
||||
|
||||
class\_init
|
||||
'''''''''''
|
||||
|
||||
The ``class_init()`` method of a proxy object will, in general behave
|
||||
similarly to the object it replaces, including setting any static
|
||||
properties and methods needed by the proxy.
|
||||
|
||||
instance\_init / realize
|
||||
''''''''''''''''''''''''
|
||||
|
||||
The ``instance_init()`` and ``realize()`` functions would only need to
|
||||
perform tasks related to being a proxy, such are registering its own
|
||||
MMIO handlers, or creating a child bus that other proxy devices can be
|
||||
attached to later.
|
||||
|
||||
Other tasks will be device-specific. For example, PCI device objects
|
||||
will initialize the PCI config space in order to make a valid PCI device
|
||||
tree within the QEMU process.
|
||||
|
||||
address space registration
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Most devices are driven by guest device driver accesses to IO addresses
|
||||
or ports. The QEMU device emulation code uses QEMU's memory region
|
||||
function calls (such as ``memory_region_init_io()``) to add callback
|
||||
functions that QEMU will invoke when the guest accesses the device's
|
||||
areas of the IO address space. When a guest driver does access the
|
||||
device, the VM will exit HW virtualization mode and return to QEMU,
|
||||
which will then lookup and execute the corresponding callback function.
|
||||
|
||||
A proxy object would need to mirror the memory region calls the actual
|
||||
device emulator would perform in its initialization code, but with its
|
||||
own callbacks. When invoked by QEMU as a result of a guest IO operation,
|
||||
they will forward the operation to the device emulation process.
|
||||
|
||||
PCI config space
|
||||
^^^^^^^^^^^^^^^^
|
||||
|
||||
PCI devices also have a configuration space that can be accessed by the
|
||||
guest driver. Guest accesses to this space is not handled by the device
|
||||
emulation object, but by its PCI parent object. Much of this space is
|
||||
read-only, but certain registers (especially BAR and MSI-related ones)
|
||||
need to be propagated to the emulation process.
|
||||
|
||||
PCI parent proxy
|
||||
''''''''''''''''
|
||||
|
||||
One way to propagate guest PCI config accesses is to create a
|
||||
"pci-device-proxy" class that can serve as the parent of a PCI device
|
||||
proxy object. This class's parent would be "pci-device" and it would
|
||||
override the PCI parent's ``config_read()`` and ``config_write()``
|
||||
methods with ones that forward these operations to the emulation
|
||||
program.
|
||||
|
||||
interrupt receipt
|
||||
^^^^^^^^^^^^^^^^^
|
||||
|
||||
A proxy for a device that generates interrupts will need to create a
|
||||
socket to receive interrupt indications from the emulation process. An
|
||||
incoming interrupt indication would then be sent up to its bus parent to
|
||||
be injected into the guest. For example, a PCI device object may use
|
||||
``pci_set_irq()``.
|
||||
|
||||
live migration
|
||||
^^^^^^^^^^^^^^
|
||||
|
||||
The proxy will register to save and restore any *vmstate* it needs over
|
||||
a live migration event. The device proxy does not need to manage the
|
||||
remote device's *vmstate*; that will be handled by the remote process
|
||||
proxy (see below).
|
||||
|
||||
QEMU remote device operation
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Generic device operations, such as DMA, will be performed by the remote
|
||||
process proxy by sending messages to the remote process.
|
||||
|
||||
DMA operations
|
||||
^^^^^^^^^^^^^^
|
||||
|
||||
DMA operations would be handled much like vhost applications do. One of
|
||||
the initial messages sent to the emulation process is a guest memory
|
||||
table. Each entry in this table consists of a file descriptor and size
|
||||
that the emulation process can ``mmap()`` to directly access guest
|
||||
memory, similar to ``vhost_user_set_mem_table()``. Note guest memory
|
||||
must be backed by shared file-backed memory, for example, using
|
||||
*-object memory-backend-file,share=on* and setting that memory backend
|
||||
as RAM for the machine.
|
||||
|
||||
IOMMU operations
|
||||
^^^^^^^^^^^^^^^^
|
||||
|
||||
When the emulated system includes an IOMMU, the remote process proxy in
|
||||
QEMU will need to create a socket for IOMMU requests from the emulation
|
||||
process. It will handle those requests with an
|
||||
``address_space_get_iotlb_entry()`` call. In order to handle IOMMU
|
||||
unmaps, the remote process proxy will also register as a listener on the
|
||||
device's DMA address space. When an IOMMU memory region is created
|
||||
within the DMA address space, an IOMMU notifier for unmaps will be added
|
||||
to the memory region that will forward unmaps to the emulation process
|
||||
over the IOMMU socket.
|
||||
|
||||
device hot-plug via QMP
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
An QMP "device\_add" command can add a device emulated by a remote
|
||||
process. It will also have "rid" option to the command, just as the
|
||||
*-device* command line option does. The remote process may either be one
|
||||
started at QEMU startup, or be one added by the "add-process" QMP
|
||||
command described above. In either case, the remote process proxy will
|
||||
forward the new device's JSON description to the corresponding emulation
|
||||
process.
|
||||
|
||||
live migration
|
||||
^^^^^^^^^^^^^^
|
||||
|
||||
The remote process proxy will also register for live migration
|
||||
notifications with ``vmstate_register()``. When called to save state,
|
||||
the proxy will send the remote process a secondary socket file
|
||||
descriptor to save the remote process's device *vmstate* over. The
|
||||
incoming byte stream length and data will be saved as the proxy's
|
||||
*vmstate*. When the proxy is resumed on its new host, this *vmstate*
|
||||
will be extracted, and a secondary socket file descriptor will be sent
|
||||
to the new remote process through which it receives the *vmstate* in
|
||||
order to restore the devices there.
|
||||
|
||||
device emulation in remote process
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The parts of QEMU that the emulation program will need include the
|
||||
object model; the memory emulation objects; the device emulation objects
|
||||
of the targeted device, and any dependent devices; and, the device's
|
||||
backends. It will also need code to setup the machine environment,
|
||||
handle requests from the QEMU process, and route machine-level requests
|
||||
(such as interrupts or IOMMU mappings) back to the QEMU process.
|
||||
|
||||
initialization
|
||||
^^^^^^^^^^^^^^
|
||||
|
||||
The process initialization sequence will follow the same sequence
|
||||
followed by QEMU. It will first initialize the backend objects, then
|
||||
device emulation objects. The JSON descriptions sent by the QEMU process
|
||||
will drive which objects need to be created.
|
||||
|
||||
- address spaces
|
||||
|
||||
Before the device objects are created, the initial address spaces and
|
||||
memory regions must be configured with ``memory_map_init()``. This
|
||||
creates a RAM memory region object (*system\_memory*) and an IO memory
|
||||
region object (*system\_io*).
|
||||
|
||||
- RAM
|
||||
|
||||
RAM memory region creation will follow how ``pc_memory_init()`` creates
|
||||
them, but must use ``memory_region_init_ram_from_fd()`` instead of
|
||||
``memory_region_allocate_system_memory()``. The file descriptors needed
|
||||
will be supplied by the guest memory table from above. Those RAM regions
|
||||
would then be added to the *system\_memory* memory region with
|
||||
``memory_region_add_subregion()``.
|
||||
|
||||
- PCI
|
||||
|
||||
IO initialization will be driven by the JSON descriptions sent from the
|
||||
QEMU process. For a PCI device, a PCI bus will need to be created with
|
||||
``pci_root_bus_new()``, and a PCI memory region will need to be created
|
||||
and added to the *system\_memory* memory region with
|
||||
``memory_region_add_subregion_overlap()``. The overlap version is
|
||||
required for architectures where PCI memory overlaps with RAM memory.
|
||||
|
||||
MMIO handling
|
||||
^^^^^^^^^^^^^
|
||||
|
||||
The device emulation objects will use ``memory_region_init_io()`` to
|
||||
install their MMIO handlers, and ``pci_register_bar()`` to associate
|
||||
those handlers with a PCI BAR, as they do within QEMU currently.
|
||||
|
||||
In order to use ``address_space_rw()`` in the emulation process to
|
||||
handle MMIO requests from QEMU, the PCI physical addresses must be the
|
||||
same in the QEMU process and the device emulation process. In order to
|
||||
accomplish that, guest BAR programming must also be forwarded from QEMU
|
||||
to the emulation process.
|
||||
|
||||
interrupt injection
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
When device emulation wants to inject an interrupt into the VM, the
|
||||
request climbs the device's bus object hierarchy until the point where a
|
||||
bus object knows how to signal the interrupt to the guest. The details
|
||||
depend on the type of interrupt being raised.
|
||||
|
||||
- PCI pin interrupts
|
||||
|
||||
On x86 systems, there is an emulated IOAPIC object attached to the root
|
||||
PCI bus object, and the root PCI object forwards interrupt requests to
|
||||
it. The IOAPIC object, in turn, calls the KVM driver to inject the
|
||||
corresponding interrupt into the VM. The simplest way to handle this in
|
||||
an emulation process would be to setup the root PCI bus driver (via
|
||||
``pci_bus_irqs()``) to send a interrupt request back to the QEMU
|
||||
process, and have the device proxy object reflect it up the PCI tree
|
||||
there.
|
||||
|
||||
- PCI MSI/X interrupts
|
||||
|
||||
PCI MSI/X interrupts are implemented in HW as DMA writes to a
|
||||
CPU-specific PCI address. In QEMU on x86, a KVM APIC object receives
|
||||
these DMA writes, then calls into the KVM driver to inject the interrupt
|
||||
into the VM. A simple emulation process implementation would be to send
|
||||
the MSI DMA address from QEMU as a message at initialization, then
|
||||
install an address space handler at that address which forwards the MSI
|
||||
message back to QEMU.
|
||||
|
||||
DMA operations
|
||||
^^^^^^^^^^^^^^
|
||||
|
||||
When a emulation object wants to DMA into or out of guest memory, it
|
||||
first must use dma\_memory\_map() to convert the DMA address to a local
|
||||
virtual address. The emulation process memory region objects setup above
|
||||
will be used to translate the DMA address to a local virtual address the
|
||||
device emulation code can access.
|
||||
|
||||
IOMMU
|
||||
^^^^^
|
||||
|
||||
When an IOMMU is in use in QEMU, DMA translation uses IOMMU memory
|
||||
regions to translate the DMA address to a guest physical address before
|
||||
that physical address can be translated to a local virtual address. The
|
||||
emulation process will need similar functionality.
|
||||
|
||||
- IOTLB cache
|
||||
|
||||
The emulation process will maintain a cache of recent IOMMU translations
|
||||
(the IOTLB). When the translate() callback of an IOMMU memory region is
|
||||
invoked, the IOTLB cache will be searched for an entry that will map the
|
||||
DMA address to a guest PA. On a cache miss, a message will be sent back
|
||||
to QEMU requesting the corresponding translation entry, which be both be
|
||||
used to return a guest address and be added to the cache.
|
||||
|
||||
- IOTLB purge
|
||||
|
||||
The IOMMU emulation will also need to act on unmap requests from QEMU.
|
||||
These happen when the guest IOMMU driver purges an entry from the
|
||||
guest's translation table.
|
||||
|
||||
live migration
|
||||
^^^^^^^^^^^^^^
|
||||
|
||||
When a remote process receives a live migration indication from QEMU, it
|
||||
will set up a channel using the received file descriptor with
|
||||
``qio_channel_socket_new_fd()``. This channel will be used to create a
|
||||
*QEMUfile* that can be passed to ``qemu_save_device_state()`` to send
|
||||
the process's device state back to QEMU. This method will be reversed on
|
||||
restore - the channel will be passed to ``qemu_loadvm_state()`` to
|
||||
restore the device state.
|
||||
|
||||
Accelerating device emulation
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The messages that are required to be sent between QEMU and the emulation
|
||||
process can add considerable latency to IO operations. The optimizations
|
||||
described below attempt to ameliorate this effect by allowing the
|
||||
emulation process to communicate directly with the kernel KVM driver.
|
||||
The KVM file descriptors created would be passed to the emulation process
|
||||
via initialization messages, much like the guest memory table is done.
|
||||
#### MMIO acceleration
|
||||
|
||||
Vhost user applications can receive guest virtio driver stores directly
|
||||
from KVM. The issue with the eventfd mechanism used by vhost user is
|
||||
that it does not pass any data with the event indication, so it cannot
|
||||
handle guest loads or guest stores that carry store data. This concept
|
||||
could, however, be expanded to cover more cases.
|
||||
|
||||
The expanded idea would require a new type of KVM device:
|
||||
*KVM\_DEV\_TYPE\_USER*. This device has two file descriptors: a master
|
||||
descriptor that QEMU can use for configuration, and a slave descriptor
|
||||
that the emulation process can use to receive MMIO notifications. QEMU
|
||||
would create both descriptors using the KVM driver, and pass the slave
|
||||
descriptor to the emulation process via an initialization message.
|
||||
|
||||
data structures
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
- guest physical range
|
||||
|
||||
The guest physical range structure describes the address range that a
|
||||
device will respond to. It includes the base and length of the range, as
|
||||
well as which bus the range resides on (e.g., on an x86machine, it can
|
||||
specify whether the range refers to memory or IO addresses).
|
||||
|
||||
A device can have multiple physical address ranges it responds to (e.g.,
|
||||
a PCI device can have multiple BARs), so the structure will also include
|
||||
an enumerated identifier to specify which of the device's ranges is
|
||||
being referred to.
|
||||
|
||||
+--------+----------------------------+
|
||||
| Name | Description |
|
||||
+========+============================+
|
||||
| addr | range base address |
|
||||
+--------+----------------------------+
|
||||
| len | range length |
|
||||
+--------+----------------------------+
|
||||
| bus | addr type (memory or IO) |
|
||||
+--------+----------------------------+
|
||||
| id | range ID (e.g., PCI BAR) |
|
||||
+--------+----------------------------+
|
||||
|
||||
- MMIO request structure
|
||||
|
||||
This structure describes an MMIO operation. It includes which guest
|
||||
physical range the MMIO was within, the offset within that range, the
|
||||
MMIO type (e.g., load or store), and its length and data. It also
|
||||
includes a sequence number that can be used to reply to the MMIO, and
|
||||
the CPU that issued the MMIO.
|
||||
|
||||
+----------+------------------------+
|
||||
| Name | Description |
|
||||
+==========+========================+
|
||||
| rid | range MMIO is within |
|
||||
+----------+------------------------+
|
||||
| offset | offset within *rid* |
|
||||
+----------+------------------------+
|
||||
| type | e.g., load or store |
|
||||
+----------+------------------------+
|
||||
| len | MMIO length |
|
||||
+----------+------------------------+
|
||||
| data | store data |
|
||||
+----------+------------------------+
|
||||
| seq | sequence ID |
|
||||
+----------+------------------------+
|
||||
|
||||
- MMIO request queues
|
||||
|
||||
MMIO request queues are FIFO arrays of MMIO request structures. There
|
||||
are two queues: pending queue is for MMIOs that haven't been read by the
|
||||
emulation program, and the sent queue is for MMIOs that haven't been
|
||||
acknowledged. The main use of the second queue is to validate MMIO
|
||||
replies from the emulation program.
|
||||
|
||||
- scoreboard
|
||||
|
||||
Each CPU in the VM is emulated in QEMU by a separate thread, so multiple
|
||||
MMIOs may be waiting to be consumed by an emulation program and multiple
|
||||
threads may be waiting for MMIO replies. The scoreboard would contain a
|
||||
wait queue and sequence number for the per-CPU threads, allowing them to
|
||||
be individually woken when the MMIO reply is received from the emulation
|
||||
program. It also tracks the number of posted MMIO stores to the device
|
||||
that haven't been replied to, in order to satisfy the PCI constraint
|
||||
that a load to a device will not complete until all previous stores to
|
||||
that device have been completed.
|
||||
|
||||
- device shadow memory
|
||||
|
||||
Some MMIO loads do not have device side-effects. These MMIOs can be
|
||||
completed without sending a MMIO request to the emulation program if the
|
||||
emulation program shares a shadow image of the device's memory image
|
||||
with the KVM driver.
|
||||
|
||||
The emulation program will ask the KVM driver to allocate memory for the
|
||||
shadow image, and will then use ``mmap()`` to directly access it. The
|
||||
emulation program can control KVM access to the shadow image by sending
|
||||
KVM an access map telling it which areas of the image have no
|
||||
side-effects (and can be completed immediately), and which require a
|
||||
MMIO request to the emulation program. The access map can also inform
|
||||
the KVM drive which size accesses are allowed to the image.
|
||||
|
||||
master descriptor
|
||||
^^^^^^^^^^^^^^^^^
|
||||
|
||||
The master descriptor is used by QEMU to configure the new KVM device.
|
||||
The descriptor would be returned by the KVM driver when QEMU issues a
|
||||
*KVM\_CREATE\_DEVICE* ``ioctl()`` with a *KVM\_DEV\_TYPE\_USER* type.
|
||||
|
||||
KVM\_DEV\_TYPE\_USER device ops
|
||||
|
||||
|
||||
The *KVM\_DEV\_TYPE\_USER* operations vector will be registered by a
|
||||
``kvm_register_device_ops()`` call when the KVM system in initialized by
|
||||
``kvm_init()``. These device ops are called by the KVM driver when QEMU
|
||||
executes certain ``ioctl()`` operations on its KVM file descriptor. They
|
||||
include:
|
||||
|
||||
- create
|
||||
|
||||
This routine is called when QEMU issues a *KVM\_CREATE\_DEVICE*
|
||||
``ioctl()`` on its per-VM file descriptor. It will allocate and
|
||||
initialize a KVM user device specific data structure, and assign the
|
||||
*kvm\_device* private field to it.
|
||||
|
||||
- ioctl
|
||||
|
||||
This routine is invoked when QEMU issues an ``ioctl()`` on the master
|
||||
descriptor. The ``ioctl()`` commands supported are defined by the KVM
|
||||
device type. *KVM\_DEV\_TYPE\_USER* ones will need several commands:
|
||||
|
||||
*KVM\_DEV\_USER\_SLAVE\_FD* creates the slave file descriptor that will
|
||||
be passed to the device emulation program. Only one slave can be created
|
||||
by each master descriptor. The file operations performed by this
|
||||
descriptor are described below.
|
||||
|
||||
The *KVM\_DEV\_USER\_PA\_RANGE* command configures a guest physical
|
||||
address range that the slave descriptor will receive MMIO notifications
|
||||
for. The range is specified by a guest physical range structure
|
||||
argument. For buses that assign addresses to devices dynamically, this
|
||||
command can be executed while the guest is running, such as the case
|
||||
when a guest changes a device's PCI BAR registers.
|
||||
|
||||
*KVM\_DEV\_USER\_PA\_RANGE* will use ``kvm_io_bus_register_dev()`` to
|
||||
register *kvm\_io\_device\_ops* callbacks to be invoked when the guest
|
||||
performs a MMIO operation within the range. When a range is changed,
|
||||
``kvm_io_bus_unregister_dev()`` is used to remove the previous
|
||||
instantiation.
|
||||
|
||||
*KVM\_DEV\_USER\_TIMEOUT* will configure a timeout value that specifies
|
||||
how long KVM will wait for the emulation process to respond to a MMIO
|
||||
indication.
|
||||
|
||||
- destroy
|
||||
|
||||
This routine is called when the VM instance is destroyed. It will need
|
||||
to destroy the slave descriptor; and free any memory allocated by the
|
||||
driver, as well as the *kvm\_device* structure itself.
|
||||
|
||||
slave descriptor
|
||||
^^^^^^^^^^^^^^^^
|
||||
|
||||
The slave descriptor will have its own file operations vector, which
|
||||
responds to system calls on the descriptor performed by the device
|
||||
emulation program.
|
||||
|
||||
- read
|
||||
|
||||
A read returns any pending MMIO requests from the KVM driver as MMIO
|
||||
request structures. Multiple structures can be returned if there are
|
||||
multiple MMIO operations pending. The MMIO requests are moved from the
|
||||
pending queue to the sent queue, and if there are threads waiting for
|
||||
space in the pending to add new MMIO operations, they will be woken
|
||||
here.
|
||||
|
||||
- write
|
||||
|
||||
A write also consists of a set of MMIO requests. They are compared to
|
||||
the MMIO requests in the sent queue. Matches are removed from the sent
|
||||
queue, and any threads waiting for the reply are woken. If a store is
|
||||
removed, then the number of posted stores in the per-CPU scoreboard is
|
||||
decremented. When the number is zero, and a non side-effect load was
|
||||
waiting for posted stores to complete, the load is continued.
|
||||
|
||||
- ioctl
|
||||
|
||||
There are several ioctl()s that can be performed on the slave
|
||||
descriptor.
|
||||
|
||||
A *KVM\_DEV\_USER\_SHADOW\_SIZE* ``ioctl()`` causes the KVM driver to
|
||||
allocate memory for the shadow image. This memory can later be
|
||||
``mmap()``\ ed by the emulation process to share the emulation's view of
|
||||
device memory with the KVM driver.
|
||||
|
||||
A *KVM\_DEV\_USER\_SHADOW\_CTRL* ``ioctl()`` controls access to the
|
||||
shadow image. It will send the KVM driver a shadow control map, which
|
||||
specifies which areas of the image can complete guest loads without
|
||||
sending the load request to the emulation program. It will also specify
|
||||
the size of load operations that are allowed.
|
||||
|
||||
- poll
|
||||
|
||||
An emulation program will use the ``poll()`` call with a *POLLIN* flag
|
||||
to determine if there are MMIO requests waiting to be read. It will
|
||||
return if the pending MMIO request queue is not empty.
|
||||
|
||||
- mmap
|
||||
|
||||
This call allows the emulation program to directly access the shadow
|
||||
image allocated by the KVM driver. As device emulation updates device
|
||||
memory, changes with no side-effects will be reflected in the shadow,
|
||||
and the KVM driver can satisfy guest loads from the shadow image without
|
||||
needing to wait for the emulation program.
|
||||
|
||||
kvm\_io\_device ops
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Each KVM per-CPU thread can handle MMIO operation on behalf of the guest
|
||||
VM. KVM will use the MMIO's guest physical address to search for a
|
||||
matching *kvm\_io\_device* to see if the MMIO can be handled by the KVM
|
||||
driver instead of exiting back to QEMU. If a match is found, the
|
||||
corresponding callback will be invoked.
|
||||
|
||||
- read
|
||||
|
||||
This callback is invoked when the guest performs a load to the device.
|
||||
Loads with side-effects must be handled synchronously, with the KVM
|
||||
driver putting the QEMU thread to sleep waiting for the emulation
|
||||
process reply before re-starting the guest. Loads that do not have
|
||||
side-effects may be optimized by satisfying them from the shadow image,
|
||||
if there are no outstanding stores to the device by this CPU. PCI memory
|
||||
ordering demands that a load cannot complete before all older stores to
|
||||
the same device have been completed.
|
||||
|
||||
- write
|
||||
|
||||
Stores can be handled asynchronously unless the pending MMIO request
|
||||
queue is full. In this case, the QEMU thread must sleep waiting for
|
||||
space in the queue. Stores will increment the number of posted stores in
|
||||
the per-CPU scoreboard, in order to implement the PCI ordering
|
||||
constraint above.
|
||||
|
||||
interrupt acceleration
|
||||
^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
This performance optimization would work much like a vhost user
|
||||
application does, where the QEMU process sets up *eventfds* that cause
|
||||
the device's corresponding interrupt to be triggered by the KVM driver.
|
||||
These irq file descriptors are sent to the emulation process at
|
||||
initialization, and are used when the emulation code raises a device
|
||||
interrupt.
|
||||
|
||||
intx acceleration
|
||||
'''''''''''''''''
|
||||
|
||||
Traditional PCI pin interrupts are level based, so, in addition to an
|
||||
irq file descriptor, a re-sampling file descriptor needs to be sent to
|
||||
the emulation program. This second file descriptor allows multiple
|
||||
devices sharing an irq to be notified when the interrupt has been
|
||||
acknowledged by the guest, so they can re-trigger the interrupt if their
|
||||
device has not de-asserted its interrupt.
|
||||
|
||||
intx irq descriptor
|
||||
|
||||
|
||||
The irq descriptors are created by the proxy object
|
||||
``using event_notifier_init()`` to create the irq and re-sampling
|
||||
*eventds*, and ``kvm_vm_ioctl(KVM_IRQFD)`` to bind them to an interrupt.
|
||||
The interrupt route can be found with
|
||||
``pci_device_route_intx_to_irq()``.
|
||||
|
||||
intx routing changes
|
||||
|
||||
|
||||
Intx routing can be changed when the guest programs the APIC the device
|
||||
pin is connected to. The proxy object in QEMU will use
|
||||
``pci_device_set_intx_routing_notifier()`` to be informed of any guest
|
||||
changes to the route. This handler will broadly follow the VFIO
|
||||
interrupt logic to change the route: de-assigning the existing irq
|
||||
descriptor from its route, then assigning it the new route. (see
|
||||
``vfio_intx_update()``)
|
||||
|
||||
MSI/X acceleration
|
||||
''''''''''''''''''
|
||||
|
||||
MSI/X interrupts are sent as DMA transactions to the host. The interrupt
|
||||
data contains a vector that is programmed by the guest, A device may have
|
||||
multiple MSI interrupts associated with it, so multiple irq descriptors
|
||||
may need to be sent to the emulation program.
|
||||
|
||||
MSI/X irq descriptor
|
||||
|
||||
|
||||
This case will also follow the VFIO example. For each MSI/X interrupt,
|
||||
an *eventfd* is created, a virtual interrupt is allocated by
|
||||
``kvm_irqchip_add_msi_route()``, and the virtual interrupt is bound to
|
||||
the eventfd with ``kvm_irqchip_add_irqfd_notifier()``.
|
||||
|
||||
MSI/X config space changes
|
||||
|
||||
|
||||
The guest may dynamically update several MSI-related tables in the
|
||||
device's PCI config space. These include per-MSI interrupt enables and
|
||||
vector data. Additionally, MSIX tables exist in device memory space, not
|
||||
config space. Much like the BAR case above, the proxy object must look
|
||||
at guest config space programming to keep the MSI interrupt state
|
||||
consistent between QEMU and the emulation program.
|
||||
|
||||
--------------
|
||||
|
||||
Disaggregated CPU emulation
|
||||
---------------------------
|
||||
|
||||
After IO services have been disaggregated, a second phase would be to
|
||||
separate a process to handle CPU instruction emulation from the main
|
||||
QEMU control function. There are no object separation points for this
|
||||
code, so the first task would be to create one.
|
||||
|
||||
Host access controls
|
||||
--------------------
|
||||
|
||||
Separating QEMU relies on the host OS's access restriction mechanisms to
|
||||
enforce that the differing processes can only access the objects they
|
||||
are entitled to. There are a couple types of mechanisms usually provided
|
||||
by general purpose OSs.
|
||||
|
||||
Discretionary access control
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Discretionary access control allows each user to control who can access
|
||||
their files. In Linux, this type of control is usually too coarse for
|
||||
QEMU separation, since it only provides three separate access controls:
|
||||
one for the same user ID, the second for users IDs with the same group
|
||||
ID, and the third for all other user IDs. Each device instance would
|
||||
need a separate user ID to provide access control, which is likely to be
|
||||
unwieldy for dynamically created VMs.
|
||||
|
||||
Mandatory access control
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Mandatory access control allows the OS to add an additional set of
|
||||
controls on top of discretionary access for the OS to control. It also
|
||||
adds other attributes to processes and files such as types, roles, and
|
||||
categories, and can establish rules for how processes and files can
|
||||
interact.
|
||||
|
||||
Type enforcement
|
||||
^^^^^^^^^^^^^^^^
|
||||
|
||||
Type enforcement assigns a *type* attribute to processes and files, and
|
||||
allows rules to be written on what operations a process with a given
|
||||
type can perform on a file with a given type. QEMU separation could take
|
||||
advantage of type enforcement by running the emulation processes with
|
||||
different types, both from the main QEMU process, and from the emulation
|
||||
processes of different classes of devices.
|
||||
|
||||
For example, guest disk images and disk emulation processes could have
|
||||
types separate from the main QEMU process and non-disk emulation
|
||||
processes, and the type rules could prevent processes other than disk
|
||||
emulation ones from accessing guest disk images. Similarly, network
|
||||
emulation processes can have a type separate from the main QEMU process
|
||||
and non-network emulation process, and only that type can access the
|
||||
host tun/tap device used to provide guest networking.
|
||||
|
||||
Category enforcement
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Category enforcement assigns a set of numbers within a given range to
|
||||
the process or file. The process is granted access to the file if the
|
||||
process's set is a superset of the file's set. This enforcement can be
|
||||
used to separate multiple instances of devices in the same class.
|
||||
|
||||
For example, if there are multiple disk devices provides to a guest,
|
||||
each device emulation process could be provisioned with a separate
|
||||
category. The different device emulation processes would not be able to
|
||||
access each other's backing disk images.
|
||||
|
||||
Alternatively, categories could be used in lieu of the type enforcement
|
||||
scheme described above. In this scenario, different categories would be
|
||||
used to prevent device emulation processes in different classes from
|
||||
accessing resources assigned to other classes.
|
||||
@@ -0,0 +1,369 @@
|
||||
..
|
||||
Copyright (c) 2015-2020 Linaro Ltd.
|
||||
|
||||
This work is licensed under the terms of the GNU GPL, version 2 or
|
||||
later. See the COPYING file in the top-level directory.
|
||||
|
||||
.. _mttcg:
|
||||
|
||||
==================
|
||||
Multi-threaded TCG
|
||||
==================
|
||||
|
||||
This document outlines the design for multi-threaded TCG (a.k.a MTTCG)
|
||||
system-mode emulation. user-mode emulation has always mirrored the
|
||||
thread structure of the translated executable although some of the
|
||||
changes done for MTTCG system emulation have improved the stability of
|
||||
linux-user emulation.
|
||||
|
||||
The original system-mode TCG implementation was single threaded and
|
||||
dealt with multiple CPUs with simple round-robin scheduling. This
|
||||
simplified a lot of things but became increasingly limited as systems
|
||||
being emulated gained additional cores and per-core performance gains
|
||||
for host systems started to level off.
|
||||
|
||||
vCPU Scheduling
|
||||
===============
|
||||
|
||||
We introduce a new running mode where each vCPU will run on its own
|
||||
user-space thread. This is enabled by default for all FE/BE
|
||||
combinations where the host memory model is able to accommodate the
|
||||
guest (TCGCPUOps::guest_default_memory_order & ~TCG_TARGET_DEFAULT_MO is zero)
|
||||
and the guest has had the required work done to support this safely
|
||||
(TCGCPUOps::mttcg_supported).
|
||||
|
||||
System emulation will fall back to the original round robin approach
|
||||
if:
|
||||
|
||||
* forced by --accel tcg,thread=single
|
||||
* enabling --icount mode
|
||||
|
||||
In the general case of running translated code there should be no
|
||||
inter-vCPU dependencies and all vCPUs should be able to run at full
|
||||
speed. Synchronisation will only be required while accessing internal
|
||||
shared data structures or when the emulated architecture requires a
|
||||
coherent representation of the emulated machine state.
|
||||
|
||||
Shared Data Structures
|
||||
======================
|
||||
|
||||
Main Run Loop
|
||||
-------------
|
||||
|
||||
Even when there is no code being generated there are a number of
|
||||
structures associated with the hot-path through the main run-loop.
|
||||
These are associated with looking up the next translation block to
|
||||
execute. These include:
|
||||
|
||||
tb_jmp_cache (per-vCPU, cache of recent jumps)
|
||||
tb_ctx.htable (global hash table, phys address->tb lookup)
|
||||
|
||||
As TB linking only occurs when blocks are in the same page this code
|
||||
is critical to performance as looking up the next TB to execute is the
|
||||
most common reason to exit the generated code.
|
||||
|
||||
DESIGN REQUIREMENT: Make access to lookup structures safe with
|
||||
multiple reader/writer threads. Minimise any lock contention to do it.
|
||||
|
||||
The hot-path avoids using locks where possible. The tb_jmp_cache is
|
||||
updated with atomic accesses to ensure consistent results. The fall
|
||||
back QHT based hash table is also designed for lockless lookups. Locks
|
||||
are only taken when code generation is required or TranslationBlocks
|
||||
have their block-to-block jumps patched.
|
||||
|
||||
Global TCG State
|
||||
----------------
|
||||
|
||||
User-mode emulation
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
We need to protect the entire code generation cycle including any post
|
||||
generation patching of the translated code. This also implies a shared
|
||||
translation buffer which contains code running on all cores. Any
|
||||
execution path that comes to the main run loop will need to hold a
|
||||
mutex for code generation. This also includes times when we need flush
|
||||
code or entries from any shared lookups/caches. Structures held on a
|
||||
per-vCPU basis won't need locking unless other vCPUs will need to
|
||||
modify them.
|
||||
|
||||
DESIGN REQUIREMENT: Add locking around all code generation and TB
|
||||
patching.
|
||||
|
||||
(Current solution)
|
||||
|
||||
Code generation is serialised with mmap_lock().
|
||||
|
||||
!User-mode emulation
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Each vCPU has its own TCG context and associated TCG region, thereby
|
||||
requiring no locking during translation.
|
||||
|
||||
Translation Blocks
|
||||
------------------
|
||||
|
||||
Currently the whole system shares a single code generation buffer
|
||||
which when full will force a flush of all translations and start from
|
||||
scratch again. Some operations also force a full flush of translations
|
||||
including:
|
||||
|
||||
- debugging operations (breakpoint insertion/removal)
|
||||
- some CPU helper functions
|
||||
- linux-user spawning its first thread
|
||||
- operations related to TCG Plugins
|
||||
|
||||
This is done with the async_safe_run_on_cpu() mechanism to ensure all
|
||||
vCPUs are quiescent when changes are being made to shared global
|
||||
structures.
|
||||
|
||||
More granular translation invalidation events are typically due
|
||||
to a change of the state of a physical page:
|
||||
|
||||
- code modification (self modify code, patching code)
|
||||
- page changes (new page mapping in linux-user mode)
|
||||
|
||||
While setting the invalid flag in a TranslationBlock will stop it
|
||||
being used when looked up in the hot-path there are a number of other
|
||||
book-keeping structures that need to be safely cleared.
|
||||
|
||||
Any TranslationBlocks which have been patched to jump directly to the
|
||||
now invalid blocks need the jump patches reversing so they will return
|
||||
to the C code.
|
||||
|
||||
There are a number of look-up caches that need to be properly updated
|
||||
including the:
|
||||
|
||||
- jump lookup cache
|
||||
- the physical-to-tb lookup hash table
|
||||
- the global page table
|
||||
|
||||
The global page table (l1_map) which provides a multi-level look-up
|
||||
for PageDesc structures which contain pointers to the start of a
|
||||
linked list of all Translation Blocks in that page (see page_next).
|
||||
|
||||
Both the jump patching and the page cache involve linked lists that
|
||||
the invalidated TranslationBlock needs to be removed from.
|
||||
|
||||
DESIGN REQUIREMENT: Safely handle invalidation of TBs
|
||||
- safely patch/revert direct jumps
|
||||
- remove central PageDesc lookup entries
|
||||
- ensure lookup caches/hashes are safely updated
|
||||
|
||||
(Current solution)
|
||||
|
||||
The direct jump themselves are updated atomically by the TCG
|
||||
tb_set_jmp_target() code. Modification to the linked lists that allow
|
||||
searching for linked pages are done under the protection of tb->jmp_lock,
|
||||
where tb is the destination block of a jump. Each origin block keeps a
|
||||
pointer to its destinations so that the appropriate lock can be acquired before
|
||||
iterating over a jump list.
|
||||
|
||||
The global page table is a lockless radix tree; cmpxchg is used
|
||||
to atomically insert new elements.
|
||||
|
||||
The lookup caches are updated atomically and the lookup hash uses QHT
|
||||
which is designed for concurrent safe lookup.
|
||||
|
||||
Parallel code generation is supported. QHT is used at insertion time
|
||||
as the synchronization point across threads, thereby ensuring that we only
|
||||
keep track of a single TranslationBlock for each guest code block.
|
||||
|
||||
Memory maps and TLBs
|
||||
--------------------
|
||||
|
||||
The memory handling code is fairly critical to the speed of memory
|
||||
access in the emulated system. The SoftMMU code is designed so the
|
||||
hot-path can be handled entirely within translated code. This is
|
||||
handled with a per-vCPU TLB structure which once populated will allow
|
||||
a series of accesses to the page to occur without exiting the
|
||||
translated code. It is possible to set flags in the TLB address which
|
||||
will ensure the slow-path is taken for each access. This can be done
|
||||
to support:
|
||||
|
||||
- Memory regions (dividing up access to PIO, MMIO and RAM)
|
||||
- Dirty page tracking (for code gen, SMC detection, migration and display)
|
||||
- Virtual TLB (for translating guest address->real address)
|
||||
|
||||
When the TLB tables are updated by a vCPU thread other than their own
|
||||
we need to ensure it is done in a safe way so no inconsistent state is
|
||||
seen by the vCPU thread.
|
||||
|
||||
Some operations require updating a number of vCPUs TLBs at the same
|
||||
time in a synchronised manner.
|
||||
|
||||
DESIGN REQUIREMENTS:
|
||||
|
||||
- TLB Flush All/Page
|
||||
- can be across-vCPUs
|
||||
- cross vCPU TLB flush may need other vCPU brought to halt
|
||||
- change may need to be visible to the calling vCPU immediately
|
||||
- TLB Flag Update
|
||||
- usually cross-vCPU
|
||||
- want change to be visible as soon as possible
|
||||
- TLB Update (update a CPUTLBEntry, via tlb_set_page_with_attrs)
|
||||
- This is a per-vCPU table - by definition can't race
|
||||
- updated by its own thread when the slow-path is forced
|
||||
|
||||
(Current solution)
|
||||
|
||||
A new set of tlb flush operations (tlb_flush_*_all_cpus_synced) force
|
||||
synchronisation by setting the source vCPUs work as "safe work" and
|
||||
exiting the cpu run loop. This ensures that by the time execution
|
||||
restarts all flush operations have completed.
|
||||
|
||||
TLB flag updates are all done atomically and are also protected by the
|
||||
corresponding page lock.
|
||||
|
||||
(Known limitation)
|
||||
|
||||
Not really a limitation but the wait mechanism is overly strict for
|
||||
some architectures which only need flushes completed by a barrier
|
||||
instruction. This could be a future optimisation.
|
||||
|
||||
Emulated hardware state
|
||||
-----------------------
|
||||
|
||||
Currently thanks to KVM work any access to IO memory is automatically protected
|
||||
by the BQL (Big QEMU Lock). Any IO region that doesn't use the BQL is expected
|
||||
to do its own locking.
|
||||
|
||||
However IO memory isn't the only way emulated hardware state can be
|
||||
modified. Some architectures have model specific registers that
|
||||
trigger hardware emulation features. Generally any translation helper
|
||||
that needs to update more than a single vCPUs of state should take the
|
||||
BQL.
|
||||
|
||||
As the BQL, or global iothread mutex is shared across the system we
|
||||
push the use of the lock as far down into the TCG code as possible to
|
||||
minimise contention.
|
||||
|
||||
(Current solution)
|
||||
|
||||
MMIO access automatically serialises hardware emulation by way of the
|
||||
BQL. Currently Arm targets serialise all ARM_CP_IO register accesses
|
||||
and also defer the reset/startup of vCPUs to the vCPU context by way
|
||||
of async_run_on_cpu().
|
||||
|
||||
Updates to interrupt state are also protected by the BQL as they can
|
||||
often be cross vCPU.
|
||||
|
||||
Memory Consistency
|
||||
==================
|
||||
|
||||
Between emulated guests and host systems there are a range of memory
|
||||
consistency models. Even emulating weakly ordered systems on strongly
|
||||
ordered hosts needs to ensure things like store-after-load re-ordering
|
||||
can be prevented when the guest wants to.
|
||||
|
||||
Memory Barriers
|
||||
---------------
|
||||
|
||||
Barriers (sometimes known as fences) provide a mechanism for software
|
||||
to enforce a particular ordering of memory operations from the point
|
||||
of view of external observers (e.g. another processor core). They can
|
||||
apply to any memory operations as well as just loads or stores.
|
||||
|
||||
The Linux kernel has an excellent `write-up
|
||||
<https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/plain/Documentation/memory-barriers.txt>`_
|
||||
on the various forms of memory barrier and the guarantees they can
|
||||
provide.
|
||||
|
||||
Barriers are often wrapped around synchronisation primitives to
|
||||
provide explicit memory ordering semantics. However they can be used
|
||||
by themselves to provide safe lockless access by ensuring for example
|
||||
a change to a signal flag will only be visible once the changes to
|
||||
payload are.
|
||||
|
||||
DESIGN REQUIREMENT: Add a new tcg_memory_barrier op
|
||||
|
||||
This would enforce a strong load/store ordering so all loads/stores
|
||||
complete at the memory barrier. On single-core non-SMP strongly
|
||||
ordered backends this could become a NOP.
|
||||
|
||||
Aside from explicit standalone memory barrier instructions there are
|
||||
also implicit memory ordering semantics which comes with each guest
|
||||
memory access instruction. For example all x86 load/stores come with
|
||||
fairly strong guarantees of sequential consistency whereas Arm has
|
||||
special variants of load/store instructions that imply acquire/release
|
||||
semantics.
|
||||
|
||||
In the case of a strongly ordered guest architecture being emulated on
|
||||
a weakly ordered host the scope for a heavy performance impact is
|
||||
quite high.
|
||||
|
||||
DESIGN REQUIREMENTS: Be efficient with use of memory barriers
|
||||
- host systems with stronger implied guarantees can skip some barriers
|
||||
- merge consecutive barriers to the strongest one
|
||||
|
||||
(Current solution)
|
||||
|
||||
The system currently has a tcg_gen_mb() which will add memory barrier
|
||||
operations if code generation is being done in a parallel context. The
|
||||
tcg_optimize() function attempts to merge barriers up to their
|
||||
strongest form before any load/store operations. The solution was
|
||||
originally developed and tested for linux-user based systems. All
|
||||
backends have been converted to emit fences when required. So far the
|
||||
following front-ends have been updated to emit fences when required:
|
||||
|
||||
- target-i386
|
||||
- target-arm
|
||||
- target-aarch64
|
||||
- target-alpha
|
||||
- target-mips
|
||||
|
||||
Memory Control and Maintenance
|
||||
------------------------------
|
||||
|
||||
This includes a class of instructions for controlling system cache
|
||||
behaviour. While QEMU doesn't model cache behaviour these instructions
|
||||
are often seen when code modification has taken place to ensure the
|
||||
changes take effect.
|
||||
|
||||
Synchronisation Primitives
|
||||
--------------------------
|
||||
|
||||
There are two broad types of synchronisation primitives found in
|
||||
modern ISAs: atomic instructions and exclusive regions.
|
||||
|
||||
The first type offer a simple atomic instruction which will guarantee
|
||||
some sort of test and conditional store will be truly atomic w.r.t.
|
||||
other cores sharing access to the memory. The classic example is the
|
||||
x86 cmpxchg instruction.
|
||||
|
||||
The second type offer a pair of load/store instructions which offer a
|
||||
guarantee that a region of memory has not been touched between the
|
||||
load and store instructions. An example of this is Arm's ldrex/strex
|
||||
pair where the strex instruction will return a flag indicating a
|
||||
successful store only if no other CPU has accessed the memory region
|
||||
since the ldrex.
|
||||
|
||||
Traditionally TCG has generated a series of operations that work
|
||||
because they are within the context of a single translation block so
|
||||
will have completed before another CPU is scheduled. However with
|
||||
the ability to have multiple threads running to emulate multiple CPUs
|
||||
we will need to explicitly expose these semantics.
|
||||
|
||||
DESIGN REQUIREMENTS:
|
||||
- Support classic atomic instructions
|
||||
- Support load/store exclusive (or load link/store conditional) pairs
|
||||
- Generic enough infrastructure to support all guest architectures
|
||||
CURRENT OPEN QUESTIONS:
|
||||
- How problematic is the ABA problem in general?
|
||||
|
||||
(Current solution)
|
||||
|
||||
The TCG provides a number of atomic helpers (tcg_gen_atomic_*) which
|
||||
can be used directly or combined to emulate other instructions like
|
||||
Arm's ldrex/strex instructions. While they are susceptible to the ABA
|
||||
problem so far common guests have not implemented patterns where
|
||||
this may be a problem - typically presenting a locking ABI which
|
||||
assumes cmpxchg like semantics.
|
||||
|
||||
The code also includes a fall-back for cases where multi-threaded TCG
|
||||
ops can't work (e.g. guest atomic width > host atomic width). In this
|
||||
case an EXCP_ATOMIC exit occurs and the instruction is emulated with
|
||||
an exclusive lock which ensures all emulation is serialised.
|
||||
|
||||
While the atomic helpers look good enough for now there may be a need
|
||||
to look at solutions that can more closely model the guest
|
||||
architectures semantics.
|
||||
@@ -0,0 +1,139 @@
|
||||
Using Multiple ``IOThread``\ s
|
||||
==============================
|
||||
|
||||
..
|
||||
Copyright (c) 2014-2017 Red Hat Inc.
|
||||
|
||||
This work is licensed under the terms of the GNU GPL, version 2 or later. See
|
||||
the COPYING file in the top-level directory.
|
||||
|
||||
|
||||
This document explains the ``IOThread`` feature and how to write code that runs
|
||||
outside the BQL.
|
||||
|
||||
The main loop and ``IOThread``\ s
|
||||
---------------------------------
|
||||
QEMU is an event-driven program that can do several things at once using an
|
||||
event loop. The VNC server and the QMP monitor are both processed from the
|
||||
same event loop, which monitors their file descriptors until they become
|
||||
readable and then invokes a callback.
|
||||
|
||||
The default event loop is called the main loop (see ``main-loop.c``). It is
|
||||
possible to create additional event loop threads using
|
||||
``-object iothread,id=my-iothread``.
|
||||
|
||||
Side note: The main loop and ``IOThread`` are both event loops but their code is
|
||||
not shared completely. Sometimes it is useful to remember that although they
|
||||
are conceptually similar they are currently not interchangeable.
|
||||
|
||||
Why ``IOThread``\ s are useful
|
||||
------------------------------
|
||||
``IOThread``\ s allow the user to control the placement of work. The main loop is a
|
||||
scalability bottleneck on hosts with many CPUs. Work can be spread across
|
||||
several ``IOThread``\ s instead of just one main loop. When set up correctly this
|
||||
can improve I/O latency and reduce jitter seen by the guest.
|
||||
|
||||
The main loop is also deeply associated with the BQL, which is a
|
||||
scalability bottleneck in itself. vCPU threads and the main loop use the BQL
|
||||
to serialize execution of QEMU code. This mutex is necessary because a lot of
|
||||
QEMU's code historically was not thread-safe.
|
||||
|
||||
The fact that all I/O processing is done in a single main loop and that the
|
||||
BQL is contended by all vCPU threads and the main loop explain
|
||||
why it is desirable to place work into ``IOThread``\ s.
|
||||
|
||||
The experimental ``virtio-blk`` data-plane implementation has been benchmarked and
|
||||
shows these effects:
|
||||
ftp://public.dhe.ibm.com/linux/pdfs/KVM_Virtualized_IO_Performance_Paper.pdf
|
||||
|
||||
.. _how-to-program:
|
||||
|
||||
How to program for ``IOThread``\ s
|
||||
----------------------------------
|
||||
The main difference between legacy code and new code that can run in an
|
||||
``IOThread`` is dealing explicitly with the event loop object, ``AioContext``
|
||||
(see ``include/qemu/aio.h``). Code that only works in the main loop
|
||||
implicitly uses the main loop's ``AioContext``. Code that supports running
|
||||
in ``IOThread``\ s must be aware of its ``AioContext``.
|
||||
|
||||
AioContext supports the following services:
|
||||
* File descriptor monitoring (read/write/error on POSIX hosts)
|
||||
* Event notifiers (inter-thread signalling)
|
||||
* Timers
|
||||
* Bottom Halves (BH) deferred callbacks
|
||||
|
||||
There are several old APIs that use the main loop AioContext:
|
||||
* LEGACY ``qemu_aio_set_fd_handler()`` - monitor a file descriptor
|
||||
* LEGACY ``qemu_aio_set_event_notifier()`` - monitor an event notifier
|
||||
* LEGACY ``timer_new_ms()`` - create a timer
|
||||
* LEGACY ``qemu_bh_new()`` - create a BH
|
||||
* LEGACY ``qemu_bh_new_guarded()`` - create a BH with a device re-entrancy guard
|
||||
* LEGACY ``qemu_aio_wait()`` - run an event loop iteration
|
||||
|
||||
Since they implicitly work on the main loop they cannot be used in code that
|
||||
runs in an ``IOThread``. They might cause a crash or deadlock if called from an
|
||||
``IOThread`` since the BQL is not held.
|
||||
|
||||
Instead, use the ``AioContext`` functions directly (see ``include/qemu/aio.h``):
|
||||
* ``aio_set_fd_handler()`` - monitor a file descriptor
|
||||
* ``aio_set_event_notifier()`` - monitor an event notifier
|
||||
* ``aio_timer_new()`` - create a timer
|
||||
* ``aio_bh_new()`` - create a BH
|
||||
* ``aio_bh_new_guarded()`` - create a BH with a device re-entrancy guard
|
||||
* ``aio_poll()`` - run an event loop iteration
|
||||
|
||||
The ``qemu_bh_new_guarded``/``aio_bh_new_guarded`` APIs accept a
|
||||
``MemReentrancyGuard``
|
||||
argument, which is used to check for and prevent re-entrancy problems. For
|
||||
BHs associated with devices, the reentrancy-guard is contained in the
|
||||
corresponding ``DeviceState`` and named ``mem_reentrancy_guard``.
|
||||
|
||||
The ``AioContext`` can be obtained from the ``IOThread`` using
|
||||
``iothread_get_aio_context()`` or for the main loop using
|
||||
``qemu_get_aio_context()``. Code that takes an ``AioContext`` argument
|
||||
works both in ``IOThread``\ s or the main loop, depending on which ``AioContext``
|
||||
instance the caller passes in.
|
||||
|
||||
How to synchronize with an ``IOThread``
|
||||
---------------------------------------
|
||||
Variables that can be accessed by multiple threads require some form of
|
||||
synchronization such as ``qemu_mutex_lock()``, ``rcu_read_lock()``, etc.
|
||||
|
||||
``AioContext`` functions like ``aio_set_fd_handler()``,
|
||||
``aio_set_event_notifier()``, ``aio_bh_new()``, and ``aio_timer_new()``
|
||||
are thread-safe. They can be used to trigger activity in an ``IOThread``.
|
||||
|
||||
Side note: the best way to schedule a function call across threads is to call
|
||||
``aio_bh_schedule_oneshot()``.
|
||||
|
||||
The main loop thread can wait synchronously for a condition using
|
||||
``AIO_WAIT_WHILE()``.
|
||||
|
||||
``AioContext`` and the block layer
|
||||
----------------------------------
|
||||
The ``AioContext`` originates from the QEMU block layer, even though nowadays
|
||||
``AioContext`` is a generic event loop that can be used by any QEMU subsystem.
|
||||
|
||||
The block layer has support for ``AioContext`` integrated. Each
|
||||
``BlockDriverState`` is associated with an ``AioContext`` using
|
||||
``bdrv_try_change_aio_context()`` and ``bdrv_get_aio_context()``.
|
||||
This allows block layer code to process I/O inside the
|
||||
right ``AioContext``. Other subsystems may wish to follow a similar approach.
|
||||
|
||||
Block layer code must therefore expect to run in an ``IOThread`` and avoid using
|
||||
old APIs that implicitly use the main loop. See
|
||||
`How to program for IOThreads`_ for information on how to do that.
|
||||
|
||||
Code running in the monitor typically needs to ensure that past
|
||||
requests from the guest are completed. When a block device is running
|
||||
in an ``IOThread``, the ``IOThread`` can also process requests from the guest
|
||||
(via ioeventfd). To achieve both objects, wrap the code between
|
||||
``bdrv_drained_begin()`` and ``bdrv_drained_end()``, thus creating a "drained
|
||||
section".
|
||||
|
||||
Long-running jobs (usually in the form of coroutines) are often scheduled in
|
||||
the ``BlockDriverState``'s ``AioContext``. The functions
|
||||
``bdrv_add``/``remove_aio_context_notifier``, or alternatively
|
||||
``blk_add``/``remove_aio_context_notifier`` if you use ``BlockBackends``,
|
||||
can be used to get a notification whenever ``bdrv_try_change_aio_context()``
|
||||
moves a ``BlockDriverState`` to a different ``AioContext``.
|
||||
@@ -0,0 +1,8 @@
|
||||
=============
|
||||
PCI subsystem
|
||||
=============
|
||||
|
||||
API Reference
|
||||
-------------
|
||||
|
||||
.. kernel-doc:: include/hw/pci/pci.h
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,748 @@
|
||||
======================
|
||||
The Sphinx QAPI Domain
|
||||
======================
|
||||
|
||||
An extension to the `rST syntax
|
||||
<https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html>`_
|
||||
in Sphinx is provided by the QAPI Domain, located in
|
||||
``docs/sphinx/qapi_domain.py``. This extension is analogous to the
|
||||
`Python Domain
|
||||
<https://www.sphinx-doc.org/en/master/usage/domains/python.html>`_
|
||||
included with Sphinx, but provides special directives and roles
|
||||
for annotating and documenting QAPI definitions
|
||||
specifically.
|
||||
|
||||
A `Domain
|
||||
<https://www.sphinx-doc.org/en/master/usage/domains/index.html>`_
|
||||
provides a set of special rST directives and cross-referencing roles to
|
||||
Sphinx for understanding rST markup written to document a specific
|
||||
language. By itself, this QAPI extension is only sufficient to parse rST
|
||||
markup written by hand; the `autodoc
|
||||
<https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html>`_
|
||||
functionality is provided elsewhere, in ``docs/sphinx/qapidoc.py``, by
|
||||
the "Transmogrifier".
|
||||
|
||||
It is not expected that any developer nor documentation writer would
|
||||
never need to write *nor* read these special rST forms. However, in the
|
||||
event that something needs to be debugged, knowing the syntax of the
|
||||
domain is quite handy. This reference may also be useful as a guide for
|
||||
understanding the QAPI Domain extension code itself. Although most of
|
||||
these forms will not be needed for documentation writing purposes,
|
||||
understanding the cross-referencing syntax *will* be helpful when
|
||||
writing rST documentation elsewhere, or for enriching the body of
|
||||
QAPIDoc blocks themselves.
|
||||
|
||||
|
||||
Concepts
|
||||
========
|
||||
|
||||
The QAPI Domain itself provides no mechanisms for reading the QAPI
|
||||
Schema or generating documentation from code that exists. It is merely
|
||||
the rST syntax used to describe things. For instance, the Sphinx Python
|
||||
domain adds syntax like ``:py:func:`` for describing Python functions in
|
||||
documentation, but it's the autodoc module that is responsible for
|
||||
reading Python code and generating such syntax. QAPI is analogous here:
|
||||
qapidoc.py is responsible for reading the QAPI Schema and generating rST
|
||||
syntax, and qapi_domain.py is responsible for translating that special
|
||||
syntax and providing APIs for Sphinx internals.
|
||||
|
||||
In other words:
|
||||
|
||||
qapi_domain.py adds syntax like ``.. qapi:command::`` to Sphinx, and
|
||||
qapidoc.py transforms the documentation in ``qapi/*.json`` into rST
|
||||
using directives defined by the domain.
|
||||
|
||||
Or even shorter:
|
||||
|
||||
``:py:`` is to ``:qapi:`` as *autodoc* is to *qapidoc*.
|
||||
|
||||
|
||||
Info Field Lists
|
||||
================
|
||||
|
||||
`Field lists
|
||||
<https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#field-lists>`_
|
||||
are a standard syntax in reStructuredText. Sphinx `extends that syntax
|
||||
<https://www.sphinx-doc.org/en/master/usage/domains/python.html#info-field-lists>`_
|
||||
to give certain field list entries special meaning and parsing to, for
|
||||
example, add cross-references. The QAPI Domain takes advantage of this
|
||||
field list extension to document things like Arguments, Members, Values,
|
||||
and so on.
|
||||
|
||||
The special parsing and handling of info field lists in Sphinx is provided by
|
||||
three main classes; Field, GroupedField, and TypedField. The behavior
|
||||
and formatting for each configured field list entry in the domain
|
||||
changes depending on which class is used.
|
||||
|
||||
Field:
|
||||
* Creates an ungrouped field: i.e., each entry will create its own
|
||||
section and they will not be combined.
|
||||
* May *optionally* support an argument.
|
||||
* May apply cross-reference roles to *either* the argument *or* the
|
||||
content body, both, or neither.
|
||||
|
||||
This is used primarily for entries which are not expected to be
|
||||
repeated, i.e., items that may only show up at most once. The QAPI
|
||||
domain uses this class for "Errors" section.
|
||||
|
||||
GroupedField:
|
||||
* Creates a grouped field: i.e. multiple adjacent entries will be
|
||||
merged into one section, and the content will form a bulleted list.
|
||||
* *Must* take an argument.
|
||||
* May optionally apply a cross-reference role to the argument, but not
|
||||
the body.
|
||||
* Can be configured to remove the bulleted list if there is only a
|
||||
single entry.
|
||||
* All items will be generated with the form: "argument -- body"
|
||||
|
||||
This is used for entries which are expected to be repeated, but aren't
|
||||
expected to have two arguments, i.e. types without names, or names
|
||||
without types. The QAPI domain uses this class for features, returns,
|
||||
and enum values.
|
||||
|
||||
TypedField:
|
||||
* Creates a grouped, typed field. Multiple adjacent entries will be
|
||||
merged into one section, and the content will form a bulleted list.
|
||||
* *Must* take at least one argument, but supports up to two -
|
||||
nominally, a name and a type.
|
||||
* May optionally apply a cross-reference role to the type or the name
|
||||
argument, but not the body.
|
||||
* Can be configured to remove the bulleted list if there is only a
|
||||
single entry.
|
||||
* All items will be generated with the form "name (type) -- body"
|
||||
|
||||
This is used for entries that are expected to be repeated and will have
|
||||
a name, a type, and a description. The QAPI domain uses this class for
|
||||
arguments, alternatives, and members. Wherever type names are referenced
|
||||
below, They must be a valid, documented type that will be
|
||||
cross-referenced in the HTML output; or one of the built-in JSON types
|
||||
(string, number, int, boolean, null, value, q_empty).
|
||||
|
||||
|
||||
``:feat:``
|
||||
----------
|
||||
|
||||
Document a feature attached to a QAPI definition.
|
||||
|
||||
:availability: This field list is available in the body of Command,
|
||||
Event, Enum, Object and Alternate directives.
|
||||
:syntax: ``:feat name: Lorem ipsum, dolor sit amet...``
|
||||
:type: `sphinx.util.docfields.GroupedField
|
||||
<https://pydoc.dev/sphinx/latest/sphinx.util.docfields.GroupedField.html?private=1>`_
|
||||
|
||||
Example::
|
||||
|
||||
.. qapi:object:: BlockdevOptionsVirtioBlkVhostVdpa
|
||||
:since: 7.2
|
||||
:ifcond: CONFIG_BLKIO
|
||||
|
||||
Driver specific block device options for the virtio-blk-vhost-vdpa
|
||||
backend.
|
||||
|
||||
:memb string path: path to the vhost-vdpa character device.
|
||||
:feat fdset: Member ``path`` supports the special "/dev/fdset/N" path
|
||||
(since 8.1)
|
||||
|
||||
|
||||
``:arg:``
|
||||
---------
|
||||
|
||||
Document an argument to a QAPI command.
|
||||
|
||||
:availability: This field list is only available in the body of the
|
||||
Command directive.
|
||||
:syntax: ``:arg type name: description``
|
||||
:type: `sphinx.util.docfields.TypedField
|
||||
<https://pydoc.dev/sphinx/latest/sphinx.util.docfields.TypedField.html?private=1>`_
|
||||
|
||||
|
||||
Example::
|
||||
|
||||
.. qapi:command:: job-pause
|
||||
:since: 3.0
|
||||
|
||||
Pause an active job.
|
||||
|
||||
This command returns immediately after marking the active job for
|
||||
pausing. Pausing an already paused job is an error.
|
||||
|
||||
The job will pause as soon as possible, which means transitioning
|
||||
into the PAUSED state if it was RUNNING, or into STANDBY if it was
|
||||
READY. The corresponding JOB_STATUS_CHANGE event will be emitted.
|
||||
|
||||
Cancelling a paused job automatically resumes it.
|
||||
|
||||
:arg string id: The job identifier.
|
||||
|
||||
|
||||
``:error:``
|
||||
-----------
|
||||
|
||||
Document the error condition(s) of a QAPI command.
|
||||
|
||||
:availability: This field list is only available in the body of the
|
||||
Command directive.
|
||||
:syntax: ``:error: Lorem ipsum dolor sit amet ...``
|
||||
:type: `sphinx.util.docfields.Field
|
||||
<https://pydoc.dev/sphinx/latest/sphinx.util.docfields.Field.html?private=1>`_
|
||||
|
||||
The format of the :errors: field list description is free-form rST. The
|
||||
alternative spelling ":errors:" is also permitted, but strictly
|
||||
analogous.
|
||||
|
||||
Example::
|
||||
|
||||
.. qapi:command:: block-job-set-speed
|
||||
:since: 1.1
|
||||
|
||||
Set maximum speed for a background block operation.
|
||||
|
||||
This command can only be issued when there is an active block job.
|
||||
|
||||
Throttling can be disabled by setting the speed to 0.
|
||||
|
||||
:arg string device: The job identifier. This used to be a device
|
||||
name (hence the name of the parameter), but since QEMU 2.7 it
|
||||
can have other values.
|
||||
:arg int speed: the maximum speed, in bytes per second, or 0 for
|
||||
unlimited. Defaults to 0.
|
||||
:error:
|
||||
- If no background operation is active on this device,
|
||||
DeviceNotActive
|
||||
|
||||
|
||||
``:return:``
|
||||
-------------
|
||||
|
||||
Document the return type(s) and value(s) of a QAPI command.
|
||||
|
||||
:availability: This field list is only available in the body of the
|
||||
Command directive.
|
||||
:syntax: ``:return type: Lorem ipsum dolor sit amet ...``
|
||||
:type: `sphinx.util.docfields.GroupedField
|
||||
<https://pydoc.dev/sphinx/latest/sphinx.util.docfields.GroupedField.html?private=1>`_
|
||||
|
||||
|
||||
Example::
|
||||
|
||||
.. qapi:command:: query-replay
|
||||
:since: 5.2
|
||||
|
||||
Retrieve the record/replay information. It includes current
|
||||
instruction count which may be used for ``replay-break`` and
|
||||
``replay-seek`` commands.
|
||||
|
||||
:return ReplayInfo: record/replay information.
|
||||
|
||||
.. qmp-example::
|
||||
|
||||
-> { "execute": "query-replay" }
|
||||
<- { "return": {
|
||||
"mode": "play", "filename": "log.rr", "icount": 220414 }
|
||||
}
|
||||
|
||||
|
||||
``:return-nodesc:``
|
||||
-------------------
|
||||
|
||||
Document the return type of a QAPI command, without an accompanying
|
||||
description.
|
||||
|
||||
:availability: This field list is only available in the body of the
|
||||
Command directive.
|
||||
:syntax: ``:return-nodesc: type``
|
||||
:type: `sphinx.util.docfields.Field
|
||||
<https://pydoc.dev/sphinx/latest/sphinx.util.docfields.Field.html?private=1>`_
|
||||
|
||||
|
||||
Example::
|
||||
|
||||
.. qapi:command:: query-replay
|
||||
:since: 5.2
|
||||
|
||||
Retrieve the record/replay information. It includes current
|
||||
instruction count which may be used for ``replay-break`` and
|
||||
``replay-seek`` commands.
|
||||
|
||||
:return-nodesc: ReplayInfo
|
||||
|
||||
.. qmp-example::
|
||||
|
||||
-> { "execute": "query-replay" }
|
||||
<- { "return": {
|
||||
"mode": "play", "filename": "log.rr", "icount": 220414 }
|
||||
}
|
||||
|
||||
``:value:``
|
||||
-----------
|
||||
|
||||
Document a possible value for a QAPI enum.
|
||||
|
||||
:availability: This field list is only available in the body of the Enum
|
||||
directive.
|
||||
:syntax: ``:value name: Lorem ipsum, dolor sit amet ...``
|
||||
:type: `sphinx.util.docfields.GroupedField
|
||||
<https://pydoc.dev/sphinx/latest/sphinx.util.docfields.GroupedField.html?private=1>`_
|
||||
|
||||
Example::
|
||||
|
||||
.. qapi:enum:: QapiErrorClass
|
||||
:since: 1.2
|
||||
|
||||
QEMU error classes
|
||||
|
||||
:value GenericError: this is used for errors that don't require a specific
|
||||
error class. This should be the default case for most errors
|
||||
:value CommandNotFound: the requested command has not been found
|
||||
:value DeviceNotActive: a device has failed to be become active
|
||||
:value DeviceNotFound: the requested device has not been found
|
||||
:value KVMMissingCap: the requested operation can't be fulfilled because a
|
||||
required KVM capability is missing
|
||||
|
||||
|
||||
``:alt:``
|
||||
------------
|
||||
|
||||
Document a possible branch for a QAPI alternate.
|
||||
|
||||
:availability: This field list is only available in the body of the
|
||||
Alternate directive.
|
||||
:syntax: ``:alt type name: Lorem ipsum, dolor sit amet ...``
|
||||
:type: `sphinx.util.docfields.TypedField
|
||||
<https://pydoc.dev/sphinx/latest/sphinx.util.docfields.TypedField.html?private=1>`_
|
||||
|
||||
As a limitation of Sphinx, we must document the "name" of the branch in
|
||||
addition to the type, even though this information is not visible on the
|
||||
wire in the QMP protocol format. This limitation *may* be lifted at a
|
||||
future date.
|
||||
|
||||
Example::
|
||||
|
||||
.. qapi:alternate:: StrOrNull
|
||||
:since: 2.10
|
||||
|
||||
This is a string value or the explicit lack of a string (null
|
||||
pointer in C). Intended for cases when 'optional absent' already
|
||||
has a different meaning.
|
||||
|
||||
:alt string s: the string value
|
||||
:alt null n: no string value
|
||||
|
||||
|
||||
``:memb:``
|
||||
----------
|
||||
|
||||
Document a member of an Event or Object.
|
||||
|
||||
:availability: This field list is available in the body of Event or
|
||||
Object directives.
|
||||
:syntax: ``:memb type name: Lorem ipsum, dolor sit amet ...``
|
||||
:type: `sphinx.util.docfields.TypedField
|
||||
<https://pydoc.dev/sphinx/latest/sphinx.util.docfields.TypedField.html?private=1>`_
|
||||
|
||||
This is fundamentally the same as ``:arg:`` and ``:alt:``, but uses the
|
||||
"Members" phrasing for Events and Objects (Structs and Unions).
|
||||
|
||||
Example::
|
||||
|
||||
.. qapi:event:: JOB_STATUS_CHANGE
|
||||
:since: 3.0
|
||||
|
||||
Emitted when a job transitions to a different status.
|
||||
|
||||
:memb string id: The job identifier
|
||||
:memb JobStatus status: The new job status
|
||||
|
||||
|
||||
Arbitrary field lists
|
||||
---------------------
|
||||
|
||||
Other field list names, while valid rST syntax, are prohibited inside of
|
||||
QAPI directives to help prevent accidental misspellings of info field
|
||||
list names. If you want to add a new arbitrary "non-value-added" field
|
||||
list to QAPI documentation, you must add the field name to the allow
|
||||
list in ``docs/conf.py``
|
||||
|
||||
For example::
|
||||
|
||||
qapi_allowed_fields = {
|
||||
"see also",
|
||||
}
|
||||
|
||||
Will allow you to add arbitrary field lists in QAPI directives::
|
||||
|
||||
.. qapi:command:: x-fake-command
|
||||
|
||||
:see also: Lorem ipsum, dolor sit amet ...
|
||||
|
||||
.. _QAPI-domain-cross-references:
|
||||
|
||||
Cross-references
|
||||
================
|
||||
|
||||
Cross-reference `roles
|
||||
<https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html>`_
|
||||
in the QAPI domain are modeled closely after the `Python
|
||||
cross-referencing syntax
|
||||
<https://www.sphinx-doc.org/en/master/usage/domains/python.html#cross-referencing-python-objects>`_.
|
||||
|
||||
QAPI definitions can be referenced using the standard `any
|
||||
<https://www.sphinx-doc.org/en/master/usage/referencing.html#role-any>`_
|
||||
role cross-reference syntax, such as with ```query-blockstats```. In
|
||||
the event that disambiguation is needed, cross-references can also be
|
||||
written using a number of explicit cross-reference roles:
|
||||
|
||||
* ``:qapi:mod:`block-core``` -- Reference a QAPI module. The link will
|
||||
take you to the beginning of that section in the documentation.
|
||||
* ``:qapi:cmd:`query-block``` -- Reference a QAPI command.
|
||||
* ``:qapi:event:`JOB_STATUS_CHANGE``` -- Reference a QAPI event.
|
||||
* ``:qapi:enum:`QapiErrorClass``` -- Reference a QAPI enum.
|
||||
* ``:qapi:obj:`BlockdevOptionsVirtioBlkVhostVdpa`` -- Reference a QAPI
|
||||
object (struct or union)
|
||||
* ``:qapi:alt:`StrOrNull``` -- Reference a QAPI alternate.
|
||||
* ``:qapi:type:`BlockDirtyInfo``` -- Reference *any* QAPI type; this
|
||||
excludes modules, commands, and events.
|
||||
* ``:qapi:any:`block-job-set-speed``` -- Reference absolutely any QAPI entity.
|
||||
|
||||
Type arguments in info field lists are converted into references as if
|
||||
you had used the ``:qapi:type:`` role. All of the special syntax below
|
||||
applies to both info field lists and standalone explicit
|
||||
cross-references.
|
||||
|
||||
|
||||
Type decorations
|
||||
----------------
|
||||
|
||||
Type names in references can be surrounded by brackets, like
|
||||
``[typename]``, to indicate an array of that type. The cross-reference
|
||||
will apply only to the type name between the brackets. For example;
|
||||
``:qapi:type:`[Qcow2BitmapInfoFlags]``` renders to:
|
||||
:qapi:type:`[QMP:Qcow2BitmapInfoFlags]`
|
||||
|
||||
To indicate an optional argument/member in a field list, the type name
|
||||
can be suffixed with ``?``. The cross-reference will be transformed to
|
||||
"type, Optional" with the link applying only to the type name. For
|
||||
example; ``:qapi:type:`BitmapSyncMode?``` renders to:
|
||||
:qapi:type:`QMP:BitmapSyncMode?`
|
||||
|
||||
|
||||
Namespaces
|
||||
----------
|
||||
|
||||
Mimicking the `Python domain target specification syntax
|
||||
<https://www.sphinx-doc.org/en/master/usage/domains/python.html#target-specification>`_,
|
||||
QAPI allows you to specify the fully qualified path for a data
|
||||
type.
|
||||
|
||||
* A namespace can be explicitly provided;
|
||||
e.g. ``:qapi:type:`QMP:BitmapSyncMode``
|
||||
* A module can be explicitly provided;
|
||||
``:qapi:type:`QMP:block-core.BitmapSyncMode``` will render to:
|
||||
:qapi:type:`QMP:block-core.BitmapSyncMode`
|
||||
* If you don't want to display the "fully qualified" name, it can be
|
||||
prefixed with a tilde; ``:qapi:type:`~QMP:block-core.BitmapSyncMode```
|
||||
will render to: :qapi:type:`~QMP:block-core.BitmapSyncMode`
|
||||
|
||||
|
||||
Target resolution
|
||||
-----------------
|
||||
|
||||
Any cross-reference to a QAPI type, whether using the ```any``` style of
|
||||
reference or the more explicit ```:qapi:any:`target``` syntax, allows
|
||||
for the presence or absence of either the namespace or module
|
||||
information.
|
||||
|
||||
When absent, their value will be inferred from context by the presence
|
||||
of any ``qapi:namespace`` or ``qapi:module`` directives preceding the
|
||||
cross-reference.
|
||||
|
||||
If no results are found when using the inferred values, other
|
||||
namespaces/modules will be searched as a last resort; but any explicitly
|
||||
provided values must always match in order to succeed.
|
||||
|
||||
This allows for efficient cross-referencing with a minimum of syntax in
|
||||
the large majority of cases, but additional context or namespace markup
|
||||
may be required outside of the QAPI reference documents when linking to
|
||||
items that share a name across multiple documented QAPI schema.
|
||||
|
||||
|
||||
Custom link text
|
||||
----------------
|
||||
|
||||
The name of a cross-reference link can be explicitly overridden like
|
||||
`most stock Sphinx references
|
||||
<https://www.sphinx-doc.org/en/master/usage/referencing.html#syntax>`_
|
||||
using the ``custom text <target>`` syntax.
|
||||
|
||||
For example, ``:qapi:cmd:`Merge dirty bitmaps
|
||||
<block-dirty-bitmap-merge>``` will render as: :qapi:cmd:`Merge dirty
|
||||
bitmaps <QMP:block-dirty-bitmap-merge>`
|
||||
|
||||
|
||||
Directives
|
||||
==========
|
||||
|
||||
The QAPI domain adds a number of custom directives for documenting
|
||||
various QAPI/QMP entities. The syntax is plain rST, and follows this
|
||||
general format::
|
||||
|
||||
.. qapi:directive:: argument
|
||||
:option:
|
||||
:another-option: with an argument
|
||||
|
||||
Content body, arbitrary rST is allowed here.
|
||||
|
||||
|
||||
Sphinx standard options
|
||||
-----------------------
|
||||
|
||||
All QAPI directives inherit a number of `standard options
|
||||
<https://www.sphinx-doc.org/en/master/usage/domains/index.html#basic-markup>`_
|
||||
from Sphinx's ObjectDescription class.
|
||||
|
||||
The dashed spellings of the below options were added in Sphinx 7.2, the
|
||||
undashed spellings are currently retained as aliases, but will be
|
||||
removed in a future version.
|
||||
|
||||
* ``:no-index:`` and ``:noindex:`` -- Do not add this item into the
|
||||
Index, and do not make it available for cross-referencing.
|
||||
* ``no-index-entry:`` and ``:noindexentry:`` -- Do not add this item
|
||||
into the Index, but allow it to be cross-referenced.
|
||||
* ``no-contents-entry`` and ``:nocontentsentry:`` -- Exclude this item
|
||||
from the Table of Contents.
|
||||
* ``no-typesetting`` -- Create TOC, Index and cross-referencing
|
||||
entities, but don't actually display the content.
|
||||
|
||||
|
||||
QAPI standard options
|
||||
---------------------
|
||||
|
||||
All QAPI directives -- *except* for namespace and module -- support
|
||||
these common options.
|
||||
|
||||
* ``:namespace: name`` -- This option allows you to override the
|
||||
namespace association of a given definition.
|
||||
* ``:module: modname`` -- Borrowed from the Python domain, this option allows
|
||||
you to override the module association of a given definition.
|
||||
* ``:since: x.y`` -- Allows the documenting of "Since" information, which is
|
||||
displayed in the signature bar.
|
||||
* ``:ifcond: CONDITION`` -- Allows the documenting of conditional availability
|
||||
information, which is displayed in an eyecatch just below the
|
||||
signature bar.
|
||||
* ``:deprecated:`` -- Adds an eyecatch just below the signature bar that
|
||||
advertises that this definition is deprecated and should be avoided.
|
||||
* ``:unstable:`` -- Adds an eyecatch just below the signature bar that
|
||||
advertises that this definition is unstable and should not be used in
|
||||
production code.
|
||||
|
||||
|
||||
qapi:namespace
|
||||
--------------
|
||||
|
||||
The ``qapi:namespace`` directive marks the start of a QAPI namespace. It
|
||||
does not take a content body, nor any options. All subsequent QAPI
|
||||
directives are associated with the most recent namespace. This affects
|
||||
the definition's "fully qualified name", allowing two different
|
||||
namespaces to create an otherwise identically named definition.
|
||||
|
||||
This directive also influences how reference resolution works for any
|
||||
references that do not explicitly specify a namespace, so this directive
|
||||
can be used to nudge references into preferring targets from within that
|
||||
namespace.
|
||||
|
||||
Example::
|
||||
|
||||
.. qapi:namespace:: QMP
|
||||
|
||||
|
||||
This directive has no visible effect.
|
||||
|
||||
|
||||
qapi:module
|
||||
-----------
|
||||
|
||||
The ``qapi:module`` directive marks the start of a QAPI module. It may have
|
||||
a content body, but it can be omitted. All subsequent QAPI directives
|
||||
are associated with the most recent module; this effects their "fully
|
||||
qualified" name, but has no other effect.
|
||||
|
||||
Example::
|
||||
|
||||
.. qapi:module:: block-core
|
||||
|
||||
Welcome to the block-core module!
|
||||
|
||||
Will be rendered as:
|
||||
|
||||
.. qapi:module:: block-core
|
||||
:noindex:
|
||||
|
||||
Welcome to the block-core module!
|
||||
|
||||
|
||||
qapi:command
|
||||
------------
|
||||
|
||||
This directive documents a QMP command. It may use any of the standard
|
||||
Sphinx or QAPI options, and the documentation body may contain
|
||||
``:arg:``, ``:feat:``, ``:error:``, or ``:return:`` info field list
|
||||
entries.
|
||||
|
||||
Example::
|
||||
|
||||
.. qapi:command:: x-fake-command
|
||||
:since: 42.0
|
||||
:unstable:
|
||||
|
||||
This command is fake, so it can't hurt you!
|
||||
|
||||
:arg int foo: Your favorite number.
|
||||
:arg string? bar: Your favorite season.
|
||||
:return [string]: A lovely computer-written poem for you.
|
||||
|
||||
|
||||
Will be rendered as:
|
||||
|
||||
.. qapi:command:: x-fake-command
|
||||
:noindex:
|
||||
:since: 42.0
|
||||
:unstable:
|
||||
|
||||
This command is fake, so it can't hurt you!
|
||||
|
||||
:arg int foo: Your favorite number.
|
||||
:arg string? bar: Your favorite season.
|
||||
:return [string]: A lovely computer-written poem for you.
|
||||
|
||||
|
||||
qapi:event
|
||||
----------
|
||||
|
||||
This directive documents a QMP event. It may use any of the standard
|
||||
Sphinx or QAPI options, and the documentation body may contain
|
||||
``:memb:`` or ``:feat:`` info field list entries.
|
||||
|
||||
Example::
|
||||
|
||||
.. qapi:event:: COMPUTER_IS_RUINED
|
||||
:since: 0.1
|
||||
:deprecated:
|
||||
|
||||
This event is emitted when your computer is *extremely* ruined.
|
||||
|
||||
:memb string reason: Diagnostics as to what caused your computer to
|
||||
be ruined.
|
||||
:feat sadness: When present, the diagnostic message will also
|
||||
explain how sad the computer is as a result of your wrongdoings.
|
||||
|
||||
Will be rendered as:
|
||||
|
||||
.. qapi:event:: COMPUTER_IS_RUINED
|
||||
:noindex:
|
||||
:since: 0.1
|
||||
:deprecated:
|
||||
|
||||
This event is emitted when your computer is *extremely* ruined.
|
||||
|
||||
:memb string reason: Diagnostics as to what caused your computer to
|
||||
be ruined.
|
||||
:feat sadness: When present, the diagnostic message will also explain
|
||||
how sad the computer is as a result of your wrongdoings.
|
||||
|
||||
|
||||
qapi:enum
|
||||
---------
|
||||
|
||||
This directive documents a QAPI enum. It may use any of the standard
|
||||
Sphinx or QAPI options, and the documentation body may contain
|
||||
``:value:`` or ``:feat:`` info field list entries.
|
||||
|
||||
Example::
|
||||
|
||||
.. qapi:enum:: Mood
|
||||
:ifcond: LIB_PERSONALITY
|
||||
|
||||
This enum represents your virtual machine's current mood!
|
||||
|
||||
:value Happy: Your VM is content and well-fed.
|
||||
:value Hungry: Your VM needs food.
|
||||
:value Melancholic: Your VM is experiencing existential angst.
|
||||
:value Petulant: Your VM is throwing a temper tantrum.
|
||||
|
||||
Will be rendered as:
|
||||
|
||||
.. qapi:enum:: Mood
|
||||
:noindex:
|
||||
:ifcond: LIB_PERSONALITY
|
||||
|
||||
This enum represents your virtual machine's current mood!
|
||||
|
||||
:value Happy: Your VM is content and well-fed.
|
||||
:value Hungry: Your VM needs food.
|
||||
:value Melancholic: Your VM is experiencing existential angst.
|
||||
:value Petulant: Your VM is throwing a temper tantrum.
|
||||
|
||||
|
||||
qapi:object
|
||||
-----------
|
||||
|
||||
This directive documents a QAPI structure or union and represents a QMP
|
||||
object. It may use any of the standard Sphinx or QAPI options, and the
|
||||
documentation body may contain ``:memb:`` or ``:feat:`` info field list
|
||||
entries.
|
||||
|
||||
Example::
|
||||
|
||||
.. qapi:object:: BigBlobOfStuff
|
||||
|
||||
This object has a bunch of disparate and unrelated things in it.
|
||||
|
||||
:memb int Birthday: Your birthday, represented in seconds since the
|
||||
UNIX epoch.
|
||||
:memb [string] Fav-Foods: A list of your favorite foods.
|
||||
:memb boolean? Bizarre-Docs: True if the documentation reference
|
||||
should be strange.
|
||||
|
||||
Will be rendered as:
|
||||
|
||||
.. qapi:object:: BigBlobOfStuff
|
||||
:noindex:
|
||||
|
||||
This object has a bunch of disparate and unrelated things in it.
|
||||
|
||||
:memb int Birthday: Your birthday, represented in seconds since the
|
||||
UNIX epoch.
|
||||
:memb [string] Fav-Foods: A list of your favorite foods.
|
||||
:memb boolean? Bizarre-Docs: True if the documentation reference
|
||||
should be strange.
|
||||
|
||||
|
||||
qapi:alternate
|
||||
--------------
|
||||
|
||||
This directive documents a QAPI alternate. It may use any of the
|
||||
standard Sphinx or QAPI options, and the documentation body may contain
|
||||
``:alt:`` or ``:feat:`` info field list entries.
|
||||
|
||||
Example::
|
||||
|
||||
.. qapi:alternate:: ErrorCode
|
||||
|
||||
This alternate represents an Error Code from the VM.
|
||||
|
||||
:alt int ec: An error code, like the type you're used to.
|
||||
:alt string em: An expletive-laced error message, if your
|
||||
computer is feeling particularly cranky and tired of your
|
||||
antics.
|
||||
|
||||
Will be rendered as:
|
||||
|
||||
.. qapi:alternate:: ErrorCode
|
||||
:noindex:
|
||||
|
||||
This alternate represents an Error Code from the VM.
|
||||
|
||||
:alt int ec: An error code, like the type you're used to.
|
||||
:alt string em: An expletive-laced error message, if your
|
||||
computer is feeling particularly cranky and tired of your
|
||||
antics.
|
||||
@@ -0,0 +1,7 @@
|
||||
.. _qdev-api:
|
||||
|
||||
================================
|
||||
QEMU Device (qdev) API Reference
|
||||
================================
|
||||
|
||||
.. kernel-doc:: include/hw/core/qdev.h
|
||||
@@ -0,0 +1,9 @@
|
||||
.. _qom-api:
|
||||
|
||||
=====================================
|
||||
QEMU Object Model (QOM) API Reference
|
||||
=====================================
|
||||
|
||||
This is the complete API documentation for :ref:`qom`.
|
||||
|
||||
.. kernel-doc:: include/qom/object.h
|
||||
@@ -0,0 +1,461 @@
|
||||
.. _qom:
|
||||
|
||||
===========================
|
||||
The QEMU Object Model (QOM)
|
||||
===========================
|
||||
|
||||
.. highlight:: c
|
||||
|
||||
The QEMU Object Model provides a framework for registering user creatable
|
||||
types and instantiating objects from those types. QOM provides the following
|
||||
features:
|
||||
|
||||
- System for dynamically registering types
|
||||
- Support for single-inheritance of types
|
||||
- Multiple inheritance of stateless interfaces
|
||||
- Mapping internal members to publicly exposed properties
|
||||
|
||||
The root object class is TYPE_OBJECT which provides for the basic
|
||||
object methods.
|
||||
|
||||
The QOM tree
|
||||
============
|
||||
|
||||
The QOM tree is a composition tree which represents all of the objects
|
||||
that make up a QEMU "machine". You can view this tree by running
|
||||
``info qom-tree`` in the :ref:`QEMU monitor`. It will contain both
|
||||
objects created by the machine itself as well those created due to
|
||||
user configuration.
|
||||
|
||||
Creating a QOM class
|
||||
====================
|
||||
|
||||
A simple minimal device implementation may look something like below:
|
||||
|
||||
.. code-block:: c
|
||||
:caption: Creating a minimal type
|
||||
|
||||
#include "qdev.h"
|
||||
|
||||
#define TYPE_MY_DEVICE "my-device"
|
||||
|
||||
// No new virtual functions: we can reuse the typedef for the
|
||||
// superclass.
|
||||
typedef DeviceClass MyDeviceClass;
|
||||
typedef struct MyDevice
|
||||
{
|
||||
DeviceState parent_obj;
|
||||
|
||||
int reg0, reg1, reg2;
|
||||
} MyDevice;
|
||||
|
||||
static const TypeInfo my_device_info = {
|
||||
.name = TYPE_MY_DEVICE,
|
||||
.parent = TYPE_DEVICE,
|
||||
.instance_size = sizeof(MyDevice),
|
||||
};
|
||||
|
||||
static void my_device_register_types(void)
|
||||
{
|
||||
type_register_static(&my_device_info);
|
||||
}
|
||||
|
||||
type_init(my_device_register_types)
|
||||
|
||||
In the above example, we create a simple type that is described by #TypeInfo.
|
||||
#TypeInfo describes information about the type including what it inherits
|
||||
from, the instance and class size, and constructor/destructor hooks.
|
||||
|
||||
The TYPE_DEVICE class is the parent class for all modern devices
|
||||
implemented in QEMU and adds some specific methods to handle QEMU
|
||||
device model. This includes managing the lifetime of devices from
|
||||
creation through to when they become visible to the guest and
|
||||
eventually unrealized.
|
||||
|
||||
Alternatively several static types could be registered using helper macro
|
||||
DEFINE_TYPES()
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
static const TypeInfo device_types_info[] = {
|
||||
{
|
||||
.name = TYPE_MY_DEVICE_A,
|
||||
.parent = TYPE_DEVICE,
|
||||
.instance_size = sizeof(MyDeviceA),
|
||||
},
|
||||
{
|
||||
.name = TYPE_MY_DEVICE_B,
|
||||
.parent = TYPE_DEVICE,
|
||||
.instance_size = sizeof(MyDeviceB),
|
||||
},
|
||||
};
|
||||
|
||||
DEFINE_TYPES(device_types_info)
|
||||
|
||||
Every type has an #ObjectClass associated with it. #ObjectClass derivatives
|
||||
are instantiated dynamically but there is only ever one instance for any
|
||||
given type. The #ObjectClass typically holds a table of function pointers
|
||||
for the virtual methods implemented by this type.
|
||||
|
||||
Using object_new(), a new #Object derivative will be instantiated. You can
|
||||
cast an #Object to a subclass (or base-class) type using
|
||||
object_dynamic_cast(). You typically want to define macro wrappers around
|
||||
OBJECT_CHECK() and OBJECT_CLASS_CHECK() to make it easier to convert to a
|
||||
specific type:
|
||||
|
||||
.. code-block:: c
|
||||
:caption: Typecasting macros
|
||||
|
||||
#define MY_DEVICE_GET_CLASS(obj) \
|
||||
OBJECT_GET_CLASS(MyDeviceClass, obj, TYPE_MY_DEVICE)
|
||||
#define MY_DEVICE_CLASS(klass) \
|
||||
OBJECT_CLASS_CHECK(MyDeviceClass, klass, TYPE_MY_DEVICE)
|
||||
#define MY_DEVICE(obj) \
|
||||
OBJECT_CHECK(MyDevice, obj, TYPE_MY_DEVICE)
|
||||
|
||||
In case the ObjectClass implementation can be built as module a
|
||||
module_obj() line must be added to make sure qemu loads the module
|
||||
when the object is needed.
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
module_obj(TYPE_MY_DEVICE);
|
||||
|
||||
Class Initialization
|
||||
--------------------
|
||||
|
||||
Before an object is initialized, the class for the object must be
|
||||
initialized. There is only one class object for all instance objects
|
||||
that is created lazily.
|
||||
|
||||
Classes are initialized by first initializing any parent classes (if
|
||||
necessary). After the parent class object has initialized, it will be
|
||||
copied into the current class object and any additional storage in the
|
||||
class object is zero filled.
|
||||
|
||||
The effect of this is that classes automatically inherit any virtual
|
||||
function pointers that the parent class has already initialized. All
|
||||
other fields will be zero filled.
|
||||
|
||||
Once all of the parent classes have been initialized, #TypeInfo::class_init
|
||||
is called to let the class being instantiated provide default initialize for
|
||||
its virtual functions. Here is how the above example might be modified
|
||||
to introduce an overridden virtual function:
|
||||
|
||||
.. code-block:: c
|
||||
:caption: Overriding a virtual function
|
||||
|
||||
#include "qdev.h"
|
||||
|
||||
void my_device_class_init(ObjectClass *klass, const void *class_data)
|
||||
{
|
||||
DeviceClass *dc = DEVICE_CLASS(klass);
|
||||
dc->reset = my_device_reset;
|
||||
}
|
||||
|
||||
static const TypeInfo my_device_info = {
|
||||
.name = TYPE_MY_DEVICE,
|
||||
.parent = TYPE_DEVICE,
|
||||
.instance_size = sizeof(MyDevice),
|
||||
.class_init = my_device_class_init,
|
||||
};
|
||||
|
||||
Introducing new virtual methods requires a class to define its own
|
||||
struct and to add a .class_size member to the #TypeInfo. Each method
|
||||
will also have a wrapper function to call it easily:
|
||||
|
||||
.. code-block:: c
|
||||
:caption: Defining an abstract class
|
||||
|
||||
#include "qdev.h"
|
||||
|
||||
typedef struct MyDeviceClass
|
||||
{
|
||||
DeviceClass parent_class;
|
||||
|
||||
void (*frobnicate) (MyDevice *obj);
|
||||
} MyDeviceClass;
|
||||
|
||||
static const TypeInfo my_device_info = {
|
||||
.name = TYPE_MY_DEVICE,
|
||||
.parent = TYPE_DEVICE,
|
||||
.instance_size = sizeof(MyDevice),
|
||||
.abstract = true, // or set a default in my_device_class_init
|
||||
.class_size = sizeof(MyDeviceClass),
|
||||
};
|
||||
|
||||
void my_device_frobnicate(MyDevice *obj)
|
||||
{
|
||||
MyDeviceClass *klass = MY_DEVICE_GET_CLASS(obj);
|
||||
|
||||
klass->frobnicate(obj);
|
||||
}
|
||||
|
||||
Interfaces
|
||||
----------
|
||||
|
||||
Interfaces allow a limited form of multiple inheritance. Instances are
|
||||
similar to normal types except for the fact that are only defined by
|
||||
their classes and never carry any state. As a consequence, a pointer to
|
||||
an interface instance should always be of incomplete type in order to be
|
||||
sure it cannot be dereferenced. That is, you should define the
|
||||
'typedef struct SomethingIf SomethingIf' so that you can pass around
|
||||
``SomethingIf *si`` arguments, but not define a ``struct SomethingIf { ... }``.
|
||||
The only things you can validly do with a ``SomethingIf *`` are to pass it as
|
||||
an argument to a method on its corresponding SomethingIfClass, or to
|
||||
dynamically cast it to an object that implements the interface.
|
||||
|
||||
Methods
|
||||
-------
|
||||
|
||||
A *method* is a function within the namespace scope of
|
||||
a class. It usually operates on the object instance by passing it as a
|
||||
strongly-typed first argument.
|
||||
If it does not operate on an object instance, it is dubbed
|
||||
*class method*.
|
||||
|
||||
Methods cannot be overloaded. That is, the #ObjectClass and method name
|
||||
uniquely identity the function to be called; the signature does not vary
|
||||
except for trailing varargs.
|
||||
|
||||
Methods are always *virtual*. Overriding a method in
|
||||
#TypeInfo.class_init of a subclass leads to any user of the class obtained
|
||||
via OBJECT_GET_CLASS() accessing the overridden function.
|
||||
The original function is not automatically invoked. It is the responsibility
|
||||
of the overriding class to determine whether and when to invoke the method
|
||||
being overridden.
|
||||
|
||||
To invoke the method being overridden, the preferred solution is to store
|
||||
the original value in the overriding class before overriding the method.
|
||||
This corresponds to ``{super,base}.method(...)`` in Java and C#
|
||||
respectively; this frees the overriding class from hardcoding its parent
|
||||
class, which someone might choose to change at some point.
|
||||
|
||||
.. code-block:: c
|
||||
:caption: Overriding a virtual method
|
||||
|
||||
typedef struct MyState MyState;
|
||||
|
||||
typedef void (*MyDoSomething)(MyState *obj);
|
||||
|
||||
typedef struct MyClass {
|
||||
ObjectClass parent_class;
|
||||
|
||||
MyDoSomething do_something;
|
||||
} MyClass;
|
||||
|
||||
static void my_do_something(MyState *obj)
|
||||
{
|
||||
// do something
|
||||
}
|
||||
|
||||
static void my_class_init(ObjectClass *oc, const void *data)
|
||||
{
|
||||
MyClass *mc = MY_CLASS(oc);
|
||||
|
||||
mc->do_something = my_do_something;
|
||||
}
|
||||
|
||||
static const TypeInfo my_type_info = {
|
||||
.name = TYPE_MY,
|
||||
.parent = TYPE_OBJECT,
|
||||
.instance_size = sizeof(MyState),
|
||||
.class_size = sizeof(MyClass),
|
||||
.class_init = my_class_init,
|
||||
};
|
||||
|
||||
typedef struct DerivedClass {
|
||||
MyClass parent_class;
|
||||
|
||||
MyDoSomething parent_do_something;
|
||||
} DerivedClass;
|
||||
|
||||
static void derived_do_something(MyState *obj)
|
||||
{
|
||||
DerivedClass *dc = DERIVED_GET_CLASS(obj);
|
||||
|
||||
// do something here
|
||||
dc->parent_do_something(obj);
|
||||
// do something else here
|
||||
}
|
||||
|
||||
static void derived_class_init(ObjectClass *oc, const void *data)
|
||||
{
|
||||
MyClass *mc = MY_CLASS(oc);
|
||||
DerivedClass *dc = DERIVED_CLASS(oc);
|
||||
|
||||
dc->parent_do_something = mc->do_something;
|
||||
mc->do_something = derived_do_something;
|
||||
}
|
||||
|
||||
static const TypeInfo derived_type_info = {
|
||||
.name = TYPE_DERIVED,
|
||||
.parent = TYPE_MY,
|
||||
.class_size = sizeof(DerivedClass),
|
||||
.class_init = derived_class_init,
|
||||
};
|
||||
|
||||
Alternatively, object_class_by_name() can be used to obtain the class and
|
||||
its non-overridden methods for a specific type. This would correspond to
|
||||
``MyClass::method(...)`` in C++.
|
||||
|
||||
One example of such methods is ``DeviceClass.reset``. More examples
|
||||
can be found at :ref:`device-life-cycle`.
|
||||
|
||||
Standard type declaration and definition macros
|
||||
===============================================
|
||||
|
||||
A lot of the code outlined above follows a standard pattern and naming
|
||||
convention. To reduce the amount of boilerplate code that needs to be
|
||||
written for a new type there are two sets of macros to generate the
|
||||
common parts in a standard format.
|
||||
|
||||
A type is declared using the OBJECT_DECLARE macro family. In types
|
||||
which do not require any virtual functions in the class, the
|
||||
OBJECT_DECLARE_SIMPLE_TYPE macro is suitable, and is commonly placed
|
||||
in the header file:
|
||||
|
||||
.. code-block:: c
|
||||
:caption: Declaring a simple type
|
||||
|
||||
OBJECT_DECLARE_SIMPLE_TYPE(MyDevice, MY_DEVICE)
|
||||
|
||||
This is equivalent to the following:
|
||||
|
||||
.. code-block:: c
|
||||
:caption: Expansion from declaring a simple type
|
||||
|
||||
typedef struct MyDevice MyDevice;
|
||||
typedef struct MyDeviceClass MyDeviceClass;
|
||||
|
||||
G_DEFINE_AUTOPTR_CLEANUP_FUNC(MyDeviceClass, object_unref)
|
||||
|
||||
#define MY_DEVICE_GET_CLASS(void *obj) \
|
||||
OBJECT_GET_CLASS(MyDeviceClass, obj, TYPE_MY_DEVICE)
|
||||
#define MY_DEVICE_CLASS(void *klass) \
|
||||
OBJECT_CLASS_CHECK(MyDeviceClass, klass, TYPE_MY_DEVICE)
|
||||
#define MY_DEVICE(void *obj)
|
||||
OBJECT_CHECK(MyDevice, obj, TYPE_MY_DEVICE)
|
||||
|
||||
struct MyDeviceClass {
|
||||
DeviceClass parent_class;
|
||||
};
|
||||
|
||||
The 'struct MyDevice' needs to be declared separately.
|
||||
If the type requires virtual functions to be declared in the class
|
||||
struct, then the alternative OBJECT_DECLARE_TYPE() macro can be
|
||||
used. This does the same as OBJECT_DECLARE_SIMPLE_TYPE(), but without
|
||||
the 'struct MyDeviceClass' definition.
|
||||
|
||||
To implement the type, the OBJECT_DEFINE macro family is available.
|
||||
For the simplest case of a leaf class which doesn't need any of its
|
||||
own virtual functions (i.e. which was declared with OBJECT_DECLARE_SIMPLE_TYPE)
|
||||
the OBJECT_DEFINE_SIMPLE_TYPE macro is suitable:
|
||||
|
||||
.. code-block:: c
|
||||
:caption: Defining a simple type
|
||||
|
||||
OBJECT_DEFINE_SIMPLE_TYPE(MyDevice, my_device, MY_DEVICE, DEVICE)
|
||||
|
||||
This is equivalent to the following:
|
||||
|
||||
.. code-block:: c
|
||||
:caption: Expansion from defining a simple type
|
||||
|
||||
static void my_device_finalize(Object *obj);
|
||||
static void my_device_class_init(ObjectClass *oc, const void *data);
|
||||
static void my_device_init(Object *obj);
|
||||
|
||||
static const TypeInfo my_device_info = {
|
||||
.parent = TYPE_DEVICE,
|
||||
.name = TYPE_MY_DEVICE,
|
||||
.instance_size = sizeof(MyDevice),
|
||||
.instance_init = my_device_init,
|
||||
.instance_finalize = my_device_finalize,
|
||||
.class_init = my_device_class_init,
|
||||
};
|
||||
|
||||
static void
|
||||
my_device_register_types(void)
|
||||
{
|
||||
type_register_static(&my_device_info);
|
||||
}
|
||||
type_init(my_device_register_types);
|
||||
|
||||
This is sufficient to get the type registered with the type
|
||||
system, and the three standard methods now need to be implemented
|
||||
along with any other logic required for the type.
|
||||
|
||||
If the class needs its own virtual methods, or has some other
|
||||
per-class state it needs to store in its own class struct,
|
||||
then you can use the OBJECT_DEFINE_TYPE macro. This does the
|
||||
same thing as OBJECT_DEFINE_SIMPLE_TYPE, but it also sets the
|
||||
class_size of the type to the size of the class struct.
|
||||
|
||||
.. code-block:: c
|
||||
:caption: Defining a type which needs a class struct
|
||||
|
||||
OBJECT_DEFINE_TYPE(MyDevice, my_device, MY_DEVICE, DEVICE)
|
||||
|
||||
If the type needs to implement one or more interfaces, then the
|
||||
OBJECT_DEFINE_SIMPLE_TYPE_WITH_INTERFACES() and
|
||||
OBJECT_DEFINE_TYPE_WITH_INTERFACES() macros can be used instead.
|
||||
These accept an array of interface type names. The difference between
|
||||
them is that the former is for simple leaf classes that don't need
|
||||
a class struct, and the latter is for when you will be defining
|
||||
a class struct.
|
||||
|
||||
.. code-block:: c
|
||||
:caption: Defining a simple type implementing interfaces
|
||||
|
||||
OBJECT_DEFINE_SIMPLE_TYPE_WITH_INTERFACES(MyDevice, my_device,
|
||||
MY_DEVICE, DEVICE,
|
||||
{ TYPE_USER_CREATABLE },
|
||||
{ NULL })
|
||||
|
||||
.. code-block:: c
|
||||
:caption: Defining a type implementing interfaces
|
||||
|
||||
OBJECT_DEFINE_TYPE_WITH_INTERFACES(MyDevice, my_device,
|
||||
MY_DEVICE, DEVICE,
|
||||
{ TYPE_USER_CREATABLE },
|
||||
{ NULL })
|
||||
|
||||
If the type is not intended to be instantiated, then the
|
||||
OBJECT_DEFINE_ABSTRACT_TYPE() macro can be used instead:
|
||||
|
||||
.. code-block:: c
|
||||
:caption: Defining a simple abstract type
|
||||
|
||||
OBJECT_DEFINE_ABSTRACT_TYPE(MyDevice, my_device,
|
||||
MY_DEVICE, DEVICE)
|
||||
|
||||
.. _device-life-cycle:
|
||||
|
||||
Device Life-cycle
|
||||
=================
|
||||
|
||||
As class initialisation cannot fail, devices have two additional
|
||||
methods to handle the creation of dynamic devices. The ``realize``
|
||||
function is called with ``Error **`` pointer which should be set if
|
||||
the device cannot complete its setup. Otherwise on successful
|
||||
completion of the ``realize`` method the device object is added to the
|
||||
QOM tree and made visible to the guest.
|
||||
|
||||
The reverse function is ``unrealize`` and should be where clean-up
|
||||
code lives to tidy up after the system is done with the device.
|
||||
|
||||
All devices can be instantiated by C code, however only some can
|
||||
created dynamically via the command line or monitor.
|
||||
|
||||
Likewise only some can be unplugged after creation and need an
|
||||
explicit ``unrealize`` implementation. This is determined by the
|
||||
``user_creatable`` variable in the root ``DeviceClass`` structure.
|
||||
Devices can only be unplugged if their ``parent_bus`` has a registered
|
||||
``HotplugHandler``.
|
||||
|
||||
API Reference
|
||||
=============
|
||||
|
||||
See the :ref:`QOM API<qom-api>` and :ref:`QDEV API<qdev-api>`
|
||||
documents for the complete API description.
|
||||
@@ -0,0 +1,394 @@
|
||||
Using RCU (Read-Copy-Update) for synchronization
|
||||
================================================
|
||||
|
||||
Read-copy update (RCU) is a synchronization mechanism that is used to
|
||||
protect read-mostly data structures. RCU is very efficient and scalable
|
||||
on the read side (it is wait-free), and thus can make the read paths
|
||||
extremely fast.
|
||||
|
||||
RCU supports concurrency between a single writer and multiple readers,
|
||||
thus it is not used alone. Typically, the write-side will use a lock to
|
||||
serialize multiple updates, but other approaches are possible (e.g.,
|
||||
restricting updates to a single task). In QEMU, when a lock is used,
|
||||
this will often be the "iothread mutex", also known as the "big QEMU
|
||||
lock" (BQL). Also, restricting updates to a single task is done in
|
||||
QEMU using the "bottom half" API.
|
||||
|
||||
RCU is fundamentally a "wait-to-finish" mechanism. The read side marks
|
||||
sections of code with "critical sections", and the update side will wait
|
||||
for the execution of all *currently running* critical sections before
|
||||
proceeding, or before asynchronously executing a callback.
|
||||
|
||||
The key point here is that only the currently running critical sections
|
||||
are waited for; critical sections that are started **after** the beginning
|
||||
of the wait do not extend the wait, despite running concurrently with
|
||||
the updater. This is the reason why RCU is more scalable than,
|
||||
for example, reader-writer locks. It is so much more scalable that
|
||||
the system will have a single instance of the RCU mechanism; a single
|
||||
mechanism can be used for an arbitrary number of "things", without
|
||||
having to worry about things such as contention or deadlocks.
|
||||
|
||||
How is this possible? The basic idea is to split updates in two phases,
|
||||
"removal" and "reclamation". During removal, we ensure that subsequent
|
||||
readers will not be able to get a reference to the old data. After
|
||||
removal has completed, a critical section will not be able to access
|
||||
the old data. Therefore, critical sections that begin after removal
|
||||
do not matter; as soon as all previous critical sections have finished,
|
||||
there cannot be any readers who hold references to the data structure,
|
||||
and these can now be safely reclaimed (e.g., freed or unref'ed).
|
||||
|
||||
Here is a picture::
|
||||
|
||||
thread 1 thread 2 thread 3
|
||||
------------------- ------------------------ -------------------
|
||||
enter RCU crit.sec.
|
||||
| finish removal phase
|
||||
| begin wait
|
||||
| | enter RCU crit.sec.
|
||||
exit RCU crit.sec | |
|
||||
complete wait |
|
||||
begin reclamation phase |
|
||||
exit RCU crit.sec.
|
||||
|
||||
|
||||
Note how thread 3 is still executing its critical section when thread 2
|
||||
starts reclaiming data. This is possible, because the old version of the
|
||||
data structure was not accessible at the time thread 3 began executing
|
||||
that critical section.
|
||||
|
||||
|
||||
RCU API
|
||||
-------
|
||||
|
||||
The core RCU API is small:
|
||||
|
||||
``void rcu_read_lock(void);``
|
||||
Used by a reader to inform the reclaimer that the reader is
|
||||
entering an RCU read-side critical section.
|
||||
|
||||
``void rcu_read_unlock(void);``
|
||||
Used by a reader to inform the reclaimer that the reader is
|
||||
exiting an RCU read-side critical section. Note that RCU
|
||||
read-side critical sections may be nested and/or overlapping.
|
||||
|
||||
``void synchronize_rcu(void);``
|
||||
Blocks until all pre-existing RCU read-side critical sections
|
||||
on all threads have completed. This marks the end of the removal
|
||||
phase and the beginning of reclamation phase.
|
||||
|
||||
Note that it would be valid for another update to come while
|
||||
``synchronize_rcu`` is running. Because of this, it is better that
|
||||
the updater releases any locks it may hold before calling
|
||||
``synchronize_rcu``. If this is not possible (for example, because
|
||||
the updater is protected by the BQL), you can use ``call_rcu``.
|
||||
|
||||
``void call_rcu1(struct rcu_head * head, void (*func)(struct rcu_head *head));``
|
||||
This function invokes ``func(head)`` after all pre-existing RCU
|
||||
read-side critical sections on all threads have completed. This
|
||||
marks the end of the removal phase, with func taking care
|
||||
asynchronously of the reclamation phase.
|
||||
|
||||
The ``foo`` struct needs to have an ``rcu_head`` structure added,
|
||||
perhaps as follows::
|
||||
|
||||
struct foo {
|
||||
struct rcu_head rcu;
|
||||
int a;
|
||||
char b;
|
||||
long c;
|
||||
};
|
||||
|
||||
so that the reclaimer function can fetch the ``struct foo`` address
|
||||
and free it::
|
||||
|
||||
call_rcu1(&foo.rcu, foo_reclaim);
|
||||
|
||||
void foo_reclaim(struct rcu_head *rp)
|
||||
{
|
||||
struct foo *fp = container_of(rp, struct foo, rcu);
|
||||
g_free(fp);
|
||||
}
|
||||
|
||||
``call_rcu1`` is typically used via either the ``call_rcu`` or
|
||||
``g_free_rcu`` macros, which handle the common case where the
|
||||
``rcu_head`` member is the first of the struct.
|
||||
|
||||
``void call_rcu(T *p, void (*func)(T *p), field-name);``
|
||||
If the ``struct rcu_head`` is the first field in the struct, you can
|
||||
use this macro instead of ``call_rcu1``.
|
||||
|
||||
``void g_free_rcu(T *p, field-name);``
|
||||
This is a special-case version of ``call_rcu`` where the callback
|
||||
function is ``g_free``.
|
||||
In the example given in ``call_rcu1``, one could have written simply::
|
||||
|
||||
g_free_rcu(&foo, rcu);
|
||||
|
||||
``typeof(*p) qatomic_rcu_read(p);``
|
||||
``qatomic_rcu_read()`` is similar to ``qatomic_load_acquire()``, but
|
||||
it makes some assumptions on the code that calls it. This allows a
|
||||
more optimized implementation.
|
||||
|
||||
``qatomic_rcu_read`` assumes that whenever a single RCU critical
|
||||
section reads multiple shared data, these reads are either
|
||||
data-dependent or need no ordering. This is almost always the
|
||||
case when using RCU, because read-side critical sections typically
|
||||
navigate one or more pointers (the pointers that are changed on
|
||||
every update) until reaching a data structure of interest,
|
||||
and then read from there.
|
||||
|
||||
RCU read-side critical sections must use ``qatomic_rcu_read()`` to
|
||||
read data, unless concurrent writes are prevented by another
|
||||
synchronization mechanism.
|
||||
|
||||
Furthermore, RCU read-side critical sections should traverse the
|
||||
data structure in a single direction, opposite to the direction
|
||||
in which the updater initializes it.
|
||||
|
||||
``void qatomic_rcu_set(p, typeof(*p) v);``
|
||||
``qatomic_rcu_set()`` is similar to ``qatomic_store_release()``,
|
||||
though it also makes assumptions on the code that calls it in
|
||||
order to allow a more optimized implementation.
|
||||
|
||||
In particular, ``qatomic_rcu_set()`` suffices for synchronization
|
||||
with readers, if the updater never mutates a field within a
|
||||
data item that is already accessible to readers. This is the
|
||||
case when initializing a new copy of the RCU-protected data
|
||||
structure; just ensure that initialization of ``*p`` is carried out
|
||||
before ``qatomic_rcu_set()`` makes the data item visible to readers.
|
||||
If this rule is observed, writes will happen in the opposite
|
||||
order as reads in the RCU read-side critical sections (or if
|
||||
there is just one update), and there will be no need for other
|
||||
synchronization mechanism to coordinate the accesses.
|
||||
|
||||
The following APIs must be used before RCU is used in a thread:
|
||||
|
||||
``void rcu_register_thread(void);``
|
||||
Mark a thread as taking part in the RCU mechanism. Such a thread
|
||||
will have to report quiescent points regularly, either manually
|
||||
or through the ``QemuCond``/``QemuSemaphore``/``QemuEvent`` APIs.
|
||||
|
||||
``void rcu_unregister_thread(void);``
|
||||
Mark a thread as not taking part anymore in the RCU mechanism.
|
||||
It is not a problem if such a thread reports quiescent points,
|
||||
either manually or by using the
|
||||
``QemuCond``/``QemuSemaphore``/``QemuEvent`` APIs.
|
||||
|
||||
Note that these APIs are relatively heavyweight, and should **not** be
|
||||
nested.
|
||||
|
||||
Convenience macros
|
||||
------------------
|
||||
|
||||
Two macros are provided that automatically release the read lock at the
|
||||
end of the scope.
|
||||
|
||||
``RCU_READ_LOCK_GUARD()``
|
||||
Takes the lock and will release it at the end of the block it's
|
||||
used in.
|
||||
|
||||
``WITH_RCU_READ_LOCK_GUARD() { code }``
|
||||
Is used at the head of a block to protect the code within the block.
|
||||
|
||||
Note that a ``goto`` out of the guarded block will also drop the lock.
|
||||
|
||||
Differences with Linux
|
||||
----------------------
|
||||
|
||||
- Waiting on a mutex is possible, though discouraged, within an RCU critical
|
||||
section. This is because spinlocks are rarely (if ever) used in userspace
|
||||
programming; not allowing this would prevent upgrading an RCU read-side
|
||||
critical section to become an updater.
|
||||
|
||||
- ``qatomic_rcu_read`` and ``qatomic_rcu_set`` replace ``rcu_dereference`` and
|
||||
``rcu_assign_pointer``. They take a **pointer** to the variable being accessed.
|
||||
|
||||
- ``call_rcu`` is a macro that has an extra argument (the name of the first
|
||||
field in the struct, which must be a struct ``rcu_head``), and expects the
|
||||
type of the callback's argument to be the type of the first argument.
|
||||
``call_rcu1`` is the same as Linux's ``call_rcu``.
|
||||
|
||||
|
||||
RCU Patterns
|
||||
------------
|
||||
|
||||
Many patterns using read-writer locks translate directly to RCU, with
|
||||
the advantages of higher scalability and deadlock immunity.
|
||||
|
||||
In general, RCU can be used whenever it is possible to create a new
|
||||
"version" of a data structure every time the updater runs. This may
|
||||
sound like a very strict restriction, however:
|
||||
|
||||
- the updater does not mean "everything that writes to a data structure",
|
||||
but rather "everything that involves a reclamation step". See the
|
||||
array example below
|
||||
|
||||
- in some cases, creating a new version of a data structure may actually
|
||||
be very cheap. For example, modifying the "next" pointer of a singly
|
||||
linked list is effectively creating a new version of the list.
|
||||
|
||||
Here are some frequently-used RCU idioms that are worth noting.
|
||||
|
||||
|
||||
RCU list processing
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
TBD (not yet used in QEMU)
|
||||
|
||||
|
||||
RCU reference counting
|
||||
^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Because grace periods are not allowed to complete while there is an RCU
|
||||
read-side critical section in progress, the RCU read-side primitives
|
||||
may be used as a restricted reference-counting mechanism. For example,
|
||||
consider the following code fragment::
|
||||
|
||||
rcu_read_lock();
|
||||
p = qatomic_rcu_read(&foo);
|
||||
/* do something with p. */
|
||||
rcu_read_unlock();
|
||||
|
||||
The RCU read-side critical section ensures that the value of ``p`` remains
|
||||
valid until after the ``rcu_read_unlock()``. In some sense, it is acquiring
|
||||
a reference to ``p`` that is later released when the critical section ends.
|
||||
The write side looks simply like this (with appropriate locking)::
|
||||
|
||||
qemu_mutex_lock(&foo_mutex);
|
||||
old = foo;
|
||||
qatomic_rcu_set(&foo, new);
|
||||
qemu_mutex_unlock(&foo_mutex);
|
||||
synchronize_rcu();
|
||||
free(old);
|
||||
|
||||
If the processing cannot be done purely within the critical section, it
|
||||
is possible to combine this idiom with a "real" reference count::
|
||||
|
||||
rcu_read_lock();
|
||||
p = qatomic_rcu_read(&foo);
|
||||
foo_ref(p);
|
||||
rcu_read_unlock();
|
||||
/* do something with p. */
|
||||
foo_unref(p);
|
||||
|
||||
The write side can be like this::
|
||||
|
||||
qemu_mutex_lock(&foo_mutex);
|
||||
old = foo;
|
||||
qatomic_rcu_set(&foo, new);
|
||||
qemu_mutex_unlock(&foo_mutex);
|
||||
synchronize_rcu();
|
||||
foo_unref(old);
|
||||
|
||||
or with ``call_rcu``::
|
||||
|
||||
qemu_mutex_lock(&foo_mutex);
|
||||
old = foo;
|
||||
qatomic_rcu_set(&foo, new);
|
||||
qemu_mutex_unlock(&foo_mutex);
|
||||
call_rcu(foo_unref, old, rcu);
|
||||
|
||||
In both cases, the write side only performs removal. Reclamation
|
||||
happens when the last reference to a ``foo`` object is dropped.
|
||||
Using ``synchronize_rcu()`` is undesirably expensive, because the
|
||||
last reference may be dropped on the read side. Hence you can
|
||||
use ``call_rcu()`` instead::
|
||||
|
||||
foo_unref(struct foo *p) {
|
||||
if (qatomic_fetch_dec(&p->refcount) == 1) {
|
||||
call_rcu(foo_destroy, p, rcu);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Note that the same idioms would be possible with reader/writer
|
||||
locks::
|
||||
|
||||
read_lock(&foo_rwlock); write_mutex_lock(&foo_rwlock);
|
||||
p = foo; p = foo;
|
||||
/* do something with p. */ foo = new;
|
||||
read_unlock(&foo_rwlock); free(p);
|
||||
write_mutex_unlock(&foo_rwlock);
|
||||
free(p);
|
||||
|
||||
------------------------------------------------------------------
|
||||
|
||||
read_lock(&foo_rwlock); write_mutex_lock(&foo_rwlock);
|
||||
p = foo; old = foo;
|
||||
foo_ref(p); foo = new;
|
||||
read_unlock(&foo_rwlock); foo_unref(old);
|
||||
/* do something with p. */ write_mutex_unlock(&foo_rwlock);
|
||||
read_lock(&foo_rwlock);
|
||||
foo_unref(p);
|
||||
read_unlock(&foo_rwlock);
|
||||
|
||||
``foo_unref`` could use a mechanism such as bottom halves to move deallocation
|
||||
out of the write-side critical section.
|
||||
|
||||
|
||||
RCU resizable arrays
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Resizable arrays can be used with RCU. The expensive RCU synchronization
|
||||
(or ``call_rcu``) only needs to take place when the array is resized.
|
||||
The two items to take care of are:
|
||||
|
||||
- ensuring that the old version of the array is available between removal
|
||||
and reclamation;
|
||||
|
||||
- avoiding mismatches in the read side between the array data and the
|
||||
array size.
|
||||
|
||||
The first problem is avoided simply by not using ``realloc``. Instead,
|
||||
each resize will allocate a new array and copy the old data into it.
|
||||
The second problem would arise if the size and the data pointers were
|
||||
two members of a larger struct::
|
||||
|
||||
struct mystuff {
|
||||
...
|
||||
int data_size;
|
||||
int data_alloc;
|
||||
T *data;
|
||||
...
|
||||
};
|
||||
|
||||
Instead, we store the size of the array with the array itself::
|
||||
|
||||
struct arr {
|
||||
int size;
|
||||
int alloc;
|
||||
T data[];
|
||||
};
|
||||
struct arr *global_array;
|
||||
|
||||
read side:
|
||||
rcu_read_lock();
|
||||
struct arr *array = qatomic_rcu_read(&global_array);
|
||||
x = i < array->size ? array->data[i] : -1;
|
||||
rcu_read_unlock();
|
||||
return x;
|
||||
|
||||
write side (running under a lock):
|
||||
if (global_array->size == global_array->alloc) {
|
||||
/* Creating a new version. */
|
||||
new_array = g_malloc(sizeof(struct arr) +
|
||||
global_array->alloc * 2 * sizeof(T));
|
||||
new_array->size = global_array->size;
|
||||
new_array->alloc = global_array->alloc * 2;
|
||||
memcpy(new_array->data, global_array->data,
|
||||
global_array->alloc * sizeof(T));
|
||||
|
||||
/* Removal phase. */
|
||||
old_array = global_array;
|
||||
qatomic_rcu_set(&global_array, new_array);
|
||||
synchronize_rcu();
|
||||
|
||||
/* Reclamation phase. */
|
||||
free(old_array);
|
||||
}
|
||||
|
||||
|
||||
References
|
||||
----------
|
||||
|
||||
* The `Linux kernel RCU documentation <https://docs.kernel.org/RCU/>`__
|
||||
@@ -0,0 +1,309 @@
|
||||
..
|
||||
Copyright (c) 2022, ISP RAS
|
||||
Written by Pavel Dovgalyuk and Alex Bennée
|
||||
|
||||
=======================
|
||||
Execution Record/Replay
|
||||
=======================
|
||||
|
||||
Core concepts
|
||||
=============
|
||||
|
||||
Record/replay functions are used for the deterministic replay of qemu
|
||||
execution. Execution recording writes a non-deterministic events log, which
|
||||
can be later used for replaying the execution anywhere and for unlimited
|
||||
number of times. Execution replaying reads the log and replays all
|
||||
non-deterministic events including external input, hardware clocks,
|
||||
and interrupts.
|
||||
|
||||
Several parts of QEMU include function calls to make event log recording
|
||||
and replaying.
|
||||
Devices' models that have non-deterministic input from external devices were
|
||||
changed to write every external event into the execution log immediately.
|
||||
E.g. network packets are written into the log when they arrive into the virtual
|
||||
network adapter.
|
||||
|
||||
All non-deterministic events are coming from these devices. But to
|
||||
replay them we need to know at which moments they occur. We specify
|
||||
these moments by counting the number of instructions executed between
|
||||
every pair of consecutive events.
|
||||
|
||||
Academic papers with description of deterministic replay implementation:
|
||||
|
||||
* `Deterministic Replay of System's Execution with Multi-target QEMU Simulator for Dynamic Analysis and Reverse Debugging <https://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html>`_
|
||||
* `Don't panic: reverse debugging of kernel drivers <https://dl.acm.org/citation.cfm?id=2786805.2803179>`_
|
||||
|
||||
Modifications of qemu include:
|
||||
|
||||
* wrappers for clock and time functions to save their return values in the log
|
||||
* saving different asynchronous events (e.g. system shutdown) into the log
|
||||
* synchronization of the bottom halves execution
|
||||
* synchronization of the threads from thread pool
|
||||
* recording/replaying user input (mouse, keyboard, and microphone)
|
||||
* adding internal checkpoints for cpu and io synchronization
|
||||
* network filter for recording and replaying the packets
|
||||
* block driver for making block layer deterministic
|
||||
* serial port input record and replay
|
||||
* recording of random numbers obtained from the external sources
|
||||
|
||||
Instruction counting
|
||||
--------------------
|
||||
|
||||
QEMU should work in icount mode to use record/replay feature. icount was
|
||||
designed to allow deterministic execution in absence of external inputs
|
||||
of the virtual machine. We also use icount to control the occurrence of the
|
||||
non-deterministic events. The number of instructions elapsed from the last event
|
||||
is written to the log while recording the execution. In replay mode we
|
||||
can predict when to inject that event using the instruction counter.
|
||||
|
||||
Locking and thread synchronisation
|
||||
----------------------------------
|
||||
|
||||
Previously the synchronisation of the main thread and the vCPU thread
|
||||
was ensured by the holding of the BQL. However the trend has been to
|
||||
reduce the time the BQL was held across the system including under TCG
|
||||
system emulation. As it is important that batches of events are kept
|
||||
in sequence (e.g. expiring timers and checkpoints in the main thread
|
||||
while instruction checkpoints are written by the vCPU thread) we need
|
||||
another lock to keep things in lock-step. This role is now handled by
|
||||
the replay_mutex_lock. It used to be held only for each event being
|
||||
written but now it is held for a whole execution period. This results
|
||||
in a deterministic ping-pong between the two main threads.
|
||||
|
||||
As the BQL is now a finer grained lock than the replay_lock it is almost
|
||||
certainly a bug, and a source of deadlocks, to take the
|
||||
replay_mutex_lock while the BQL is held. This is enforced by an assert.
|
||||
While the unlocks are usually in the reverse order, this is not
|
||||
necessary; you can drop the replay_lock while holding the BQL, without
|
||||
doing a more complicated unlock_iothread/replay_unlock/lock_iothread
|
||||
sequence.
|
||||
|
||||
Checkpoints
|
||||
-----------
|
||||
|
||||
Replaying the execution of virtual machine is bound by sources of
|
||||
non-determinism. These are inputs from clock and peripheral devices,
|
||||
and QEMU thread scheduling. Thread scheduling affect on processing events
|
||||
from timers, asynchronous input-output, and bottom halves.
|
||||
|
||||
Invocations of timers are coupled with clock reads and changing the state
|
||||
of the virtual machine. Reads produce non-deterministic data taken from
|
||||
host clock. And VM state changes should preserve their order. Their relative
|
||||
order in replay mode must replicate the order of callbacks in record mode.
|
||||
To preserve this order we use checkpoints. When a specific clock is processed
|
||||
in record mode we save to the log special "checkpoint" event.
|
||||
Checkpoints here do not refer to virtual machine snapshots. They are just
|
||||
record/replay events used for synchronization.
|
||||
|
||||
QEMU in replay mode will try to invoke timers processing in random moment
|
||||
of time. That's why we do not process a group of timers until the checkpoint
|
||||
event will be read from the log. Such an event allows synchronizing CPU
|
||||
execution and timer events.
|
||||
|
||||
Two other checkpoints govern the "warping" of the virtual clock.
|
||||
While the virtual machine is idle, the virtual clock increments at
|
||||
1 ns per *real time* nanosecond. This is done by setting up a timer
|
||||
(called the warp timer) on the virtual real time clock, so that the
|
||||
timer fires at the next deadline of the virtual clock; the virtual clock
|
||||
is then incremented (which is called "warping" the virtual clock) as
|
||||
soon as the timer fires or the CPUs need to go out of the idle state.
|
||||
Two functions are used for this purpose; because these actions change
|
||||
virtual machine state and must be deterministic, each of them creates a
|
||||
checkpoint. ``icount_start_warp_timer`` checks if the CPUs are idle and if so
|
||||
starts accounting real time to virtual clock. ``icount_account_warp_timer``
|
||||
is called when the CPUs get an interrupt or when the warp timer fires,
|
||||
and it warps the virtual clock by the amount of real time that has passed
|
||||
since ``icount_start_warp_timer``.
|
||||
|
||||
Virtual devices
|
||||
===============
|
||||
|
||||
Record/replay mechanism, that could be enabled through icount mode, expects
|
||||
the virtual devices to satisfy the following requirement:
|
||||
everything that affects
|
||||
the guest state during execution in icount mode should be deterministic.
|
||||
|
||||
Timers
|
||||
------
|
||||
|
||||
Timers are used to execute callbacks from different subsystems of QEMU
|
||||
at the specified moments of time. There are several kinds of timers:
|
||||
|
||||
* Real time clock. Based on host time and used only for callbacks that
|
||||
do not change the virtual machine state. For this reason real time
|
||||
clock and timers does not affect deterministic replay at all.
|
||||
* Virtual clock. These timers run only during the emulation. In icount
|
||||
mode virtual clock value is calculated using executed instructions counter.
|
||||
That is why it is completely deterministic and does not have to be recorded.
|
||||
* Host clock. This clock is used by device models that simulate real time
|
||||
sources (e.g. real time clock chip). Host clock is the one of the sources
|
||||
of non-determinism. Host clock read operations should be logged to
|
||||
make the execution deterministic.
|
||||
* Virtual real time clock. This clock is similar to real time clock but
|
||||
it is used only for increasing virtual clock while virtual machine is
|
||||
sleeping. Due to its nature it is also non-deterministic as the host clock
|
||||
and has to be logged too.
|
||||
|
||||
All virtual devices should use virtual clock for timers that change the guest
|
||||
state. Virtual clock is deterministic, therefore such timers are deterministic
|
||||
too.
|
||||
|
||||
Virtual devices can also use realtime clock for the events that do not change
|
||||
the guest state directly. When the clock ticking should depend on VM execution
|
||||
speed, use virtual clock with EXTERNAL attribute. It is not deterministic,
|
||||
but its speed depends on the guest execution. This clock is used by
|
||||
the virtual devices (e.g., slirp routing device) that lie outside the
|
||||
replayed guest.
|
||||
|
||||
Block devices
|
||||
-------------
|
||||
|
||||
Block devices record/replay module (``blkreplay``) intercepts calls of
|
||||
bdrv coroutine functions at the top of block drivers stack.
|
||||
|
||||
All block completion operations are added to the queue in the coroutines.
|
||||
When the queue is flushed the information about processed requests
|
||||
is recorded to the log. In replay phase the queue is matched with
|
||||
events read from the log. Therefore block devices requests are processed
|
||||
deterministically.
|
||||
|
||||
Bottom halves
|
||||
-------------
|
||||
|
||||
Bottom half callbacks, that affect the guest state, should be invoked through
|
||||
``replay_bh_schedule_event`` or ``replay_bh_schedule_oneshot_event`` functions.
|
||||
Their invocations are saved in record mode and synchronized with the existing
|
||||
log in replay mode.
|
||||
|
||||
Disk I/O events are completely deterministic in our model, because
|
||||
in both record and replay modes we start virtual machine from the same
|
||||
disk state. But callbacks that virtual disk controller uses for reading and
|
||||
writing the disk may occur at different moments of time in record and replay
|
||||
modes.
|
||||
|
||||
Reading and writing requests are created by CPU thread of QEMU. Later these
|
||||
requests proceed to block layer which creates "bottom halves". Bottom
|
||||
halves consist of callback and its parameters. They are processed when
|
||||
main loop locks the BQL. These locks are not synchronized with
|
||||
replaying process because main loop also processes the events that do not
|
||||
affect the virtual machine state (like user interaction with monitor).
|
||||
|
||||
That is why we had to implement saving and replaying bottom halves callbacks
|
||||
synchronously to the CPU execution. When the callback is about to execute
|
||||
it is added to the queue in the replay module. This queue is written to the
|
||||
log when its callbacks are executed. In replay mode callbacks are not processed
|
||||
until the corresponding event is read from the events log file.
|
||||
|
||||
Sometimes the block layer uses asynchronous callbacks for its internal purposes
|
||||
(like reading or writing VM snapshots or disk image cluster tables). In this
|
||||
case bottom halves are not marked as "replayable" and do not saved
|
||||
into the log.
|
||||
|
||||
Saving/restoring the VM state
|
||||
-----------------------------
|
||||
|
||||
Record/replay relies on VM state save and restore being complete and
|
||||
deterministic.
|
||||
|
||||
All fields in the device state structure (including virtual timers)
|
||||
should be restored by loadvm to the same values they had before savevm.
|
||||
|
||||
Avoid accessing other devices' state, because the order of saving/restoring
|
||||
is not defined. It means that you should not call functions like
|
||||
``update_irq`` in ``post_load`` callback. Save everything explicitly to avoid
|
||||
the dependencies that may make restoring the VM state non-deterministic.
|
||||
|
||||
Stopping the VM
|
||||
---------------
|
||||
|
||||
Stopping the guest should not interfere with its state (with the exception
|
||||
of the network connections, that could be broken by the remote timeouts).
|
||||
VM can be stopped at any moment of replay by the user. Restarting the VM
|
||||
after that stop should not break the replay by the unneeded guest state change.
|
||||
|
||||
Replay log format
|
||||
=================
|
||||
|
||||
Record/replay log consists of the header and the sequence of execution
|
||||
events. The header includes 4-byte replay version id and 8-byte reserved
|
||||
field. Version is updated every time replay log format changes to prevent
|
||||
using replay log created by another build of qemu.
|
||||
|
||||
The sequence of the events describes virtual machine state changes.
|
||||
It includes all non-deterministic inputs of VM, synchronization marks and
|
||||
instruction counts used to correctly inject inputs at replay.
|
||||
|
||||
Synchronization marks (checkpoints) are used for synchronizing qemu threads
|
||||
that perform operations with virtual hardware. These operations may change
|
||||
system's state (e.g., change some register or generate interrupt) and
|
||||
therefore should execute synchronously with CPU thread.
|
||||
|
||||
Every event in the log includes 1-byte event id and optional arguments.
|
||||
When argument is an array, it is stored as 4-byte array length
|
||||
and corresponding number of bytes with data.
|
||||
Here is the list of events that are written into the log:
|
||||
|
||||
- EVENT_INSTRUCTION. Instructions executed since last event. Followed by:
|
||||
|
||||
- 4-byte number of executed instructions.
|
||||
|
||||
- EVENT_INTERRUPT. Used to synchronize interrupt processing.
|
||||
- EVENT_EXCEPTION. Used to synchronize exception handling.
|
||||
- EVENT_ASYNC. This is a group of events. When such an event is generated,
|
||||
it is stored in the queue and processed in icount_account_warp_timer().
|
||||
Every such event has it's own id from the following list:
|
||||
|
||||
- REPLAY_ASYNC_EVENT_BH. Bottom-half callback. This event synchronizes
|
||||
callbacks that affect virtual machine state, but normally called
|
||||
asynchronously. Followed by:
|
||||
|
||||
- 8-byte operation id.
|
||||
|
||||
- REPLAY_ASYNC_EVENT_INPUT. Input device event. Contains
|
||||
parameters of keyboard and mouse input operations
|
||||
(key press/release, mouse pointer movement). Followed by:
|
||||
|
||||
- 9-16 bytes depending of input event.
|
||||
|
||||
- REPLAY_ASYNC_EVENT_INPUT_SYNC. Internal input synchronization event.
|
||||
- REPLAY_ASYNC_EVENT_CHAR_READ. Character (e.g., serial port) device input
|
||||
initiated by the sender. Followed by:
|
||||
|
||||
- 1-byte character device id.
|
||||
- Array with bytes were read.
|
||||
|
||||
- REPLAY_ASYNC_EVENT_BLOCK. Block device operation. Used to synchronize
|
||||
operations with disk and flash drives with CPU. Followed by:
|
||||
|
||||
- 8-byte operation id.
|
||||
|
||||
- REPLAY_ASYNC_EVENT_NET. Incoming network packet. Followed by:
|
||||
|
||||
- 1-byte network adapter id.
|
||||
- 4-byte packet flags.
|
||||
- Array with packet bytes.
|
||||
|
||||
- EVENT_SHUTDOWN. Occurs when user sends shutdown event to qemu,
|
||||
e.g., by closing the window.
|
||||
- EVENT_CHAR_WRITE. Used to synchronize character output operations. Followed by:
|
||||
|
||||
- 4-byte output function return value.
|
||||
- 4-byte offset in the output array.
|
||||
|
||||
- EVENT_CHAR_READ_ALL. Used to synchronize character input operations,
|
||||
initiated by qemu. Followed by:
|
||||
|
||||
- Array with bytes that were read.
|
||||
|
||||
- EVENT_CHAR_READ_ALL_ERROR. Unsuccessful character input operation,
|
||||
initiated by qemu. Followed by:
|
||||
|
||||
- 4-byte error code.
|
||||
|
||||
- EVENT_CLOCK + clock_id. Group of events for host clock read operations. Followed by:
|
||||
|
||||
- 8-byte clock value.
|
||||
|
||||
- EVENT_CHECKPOINT + checkpoint_id. Checkpoint for synchronization of
|
||||
CPU, internal threads, and asynchronous input events.
|
||||
- EVENT_END. Last event in the log.
|
||||
@@ -0,0 +1,366 @@
|
||||
|
||||
=======================================
|
||||
Reset in QEMU: the Resettable interface
|
||||
=======================================
|
||||
|
||||
The reset of qemu objects is handled using the resettable interface declared
|
||||
in ``include/hw/core/resettable.h``.
|
||||
|
||||
This interface allows objects to be grouped (on a tree basis); so that the
|
||||
whole group can be reset consistently. Each individual member object does not
|
||||
have to care about others; in particular, problems of order (which object is
|
||||
reset first) are addressed.
|
||||
|
||||
The main object types which implement this interface are DeviceClass
|
||||
and BusClass.
|
||||
|
||||
Triggering reset
|
||||
----------------
|
||||
|
||||
This section documents the APIs which "users" of a resettable object should use
|
||||
to control it. All resettable control functions must be called while holding
|
||||
the BQL.
|
||||
|
||||
You can apply a reset to an object using ``resettable_assert_reset()``. You need
|
||||
to call ``resettable_release_reset()`` to release the object from reset. To
|
||||
instantly reset an object, without keeping it in reset state, just call
|
||||
``resettable_reset()``. These functions take two parameters: a pointer to the
|
||||
object to reset and a reset type.
|
||||
|
||||
The Resettable interface handles reset types with an enum ``ResetType``:
|
||||
|
||||
``RESET_TYPE_COLD``
|
||||
Cold reset is supported by every resettable object. In QEMU, it means we reset
|
||||
to the initial state corresponding to the start of QEMU; this might differ
|
||||
from what is a real hardware cold reset. It differs from other resets (like
|
||||
warm or bus resets) which may keep certain parts untouched.
|
||||
|
||||
``RESET_TYPE_SNAPSHOT_LOAD``
|
||||
This is called for a reset which is being done to put the system into a
|
||||
clean state prior to loading a snapshot. (This corresponds to a reset
|
||||
with ``SHUTDOWN_CAUSE_SNAPSHOT_LOAD``.) Almost all devices should treat
|
||||
this the same as ``RESET_TYPE_COLD``. The main exception is devices which
|
||||
have some non-deterministic state they want to reinitialize to a different
|
||||
value on each cold reset, such as RNG seed information, and which they
|
||||
must not reinitialize on a snapshot-load reset.
|
||||
|
||||
``RESET_TYPE_WAKEUP``
|
||||
If the machine supports waking up from a suspended state and needs to reset
|
||||
its devices during wake-up (from the ``MachineClass::wakeup()`` method), this
|
||||
reset type should be used for such a request. Devices can utilize this reset
|
||||
type to differentiate the reset requested during machine wake-up from other
|
||||
reset requests. For example, RAM content must not be lost during wake-up, and
|
||||
memory devices like virtio-mem that provide additional RAM must not reset
|
||||
such state during wake-ups, but might do so during cold resets. However, this
|
||||
reset type should not be used for wake-up detection, as not every machine
|
||||
type issues a device reset request during wake-up.
|
||||
|
||||
``RESET_TYPE_S390_CPU_NORMAL``
|
||||
This is only used for S390 CPU objects; it clears interrupts, stops
|
||||
processing, and clears the TLB, but does not touch register contents.
|
||||
|
||||
``RESET_TYPE_S390_CPU_INITIAL``
|
||||
This is only used for S390 CPU objects; it does everything
|
||||
``RESET_TYPE_S390_CPU_NORMAL`` does and also clears the PSW, prefix,
|
||||
FPC, timer and control registers. It does not touch gprs, fprs or acrs.
|
||||
|
||||
Devices which implement reset methods must treat any unknown ``ResetType``
|
||||
as equivalent to ``RESET_TYPE_COLD``; this will reduce the amount of
|
||||
existing code we need to change if we add more types in future.
|
||||
|
||||
Calling ``resettable_reset()`` is equivalent to calling
|
||||
``resettable_assert_reset()`` then ``resettable_release_reset()``. It is
|
||||
possible to interleave multiple calls to these three functions. There may
|
||||
be several reset sources/controllers of a given object. The interface handles
|
||||
everything and the different reset controllers do not need to know anything
|
||||
about each others. The object will leave reset state only when each other
|
||||
controllers end their reset operation. This point is handled internally by
|
||||
maintaining a count of in-progress resets; it is crucial to call
|
||||
``resettable_release_reset()`` one time and only one time per
|
||||
``resettable_assert_reset()`` call.
|
||||
|
||||
For now migration of a device or bus in reset is not supported. Care must be
|
||||
taken not to delay ``resettable_release_reset()`` after its
|
||||
``resettable_assert_reset()`` counterpart.
|
||||
|
||||
Note that, since resettable is an interface, the API takes a simple Object as
|
||||
parameter. Still, it is a programming error to call a resettable function on a
|
||||
non-resettable object and it will trigger a run time assert error. Since most
|
||||
calls to resettable interface are done through base class functions, such an
|
||||
error is not likely to happen.
|
||||
|
||||
For Devices and Buses, the following helper functions exist:
|
||||
|
||||
- ``device_cold_reset()``
|
||||
- ``bus_cold_reset()``
|
||||
|
||||
These are simple wrappers around resettable_reset() function; they only cast the
|
||||
Device or Bus into an Object and pass the cold reset type. When possible
|
||||
prefer to use these functions instead of ``resettable_reset()``.
|
||||
|
||||
Device and bus functions co-exist because there can be semantic differences
|
||||
between resetting a bus and resetting the controller bridge which owns it.
|
||||
For example, consider a SCSI controller. Resetting the controller puts all
|
||||
its registers back to what reset state was as well as reset everything on the
|
||||
SCSI bus, whereas resetting just the SCSI bus only resets everything that's on
|
||||
it but not the controller.
|
||||
|
||||
|
||||
Multi-phase mechanism
|
||||
---------------------
|
||||
|
||||
This section documents the internals of the resettable interface.
|
||||
|
||||
The resettable interface uses a multi-phase system to relieve objects and
|
||||
machines from reset ordering problems. To address this, the reset operation
|
||||
of an object is split into three well defined phases.
|
||||
|
||||
When resetting several objects (for example the whole machine at simulation
|
||||
startup), all first phases of all objects are executed, then all second phases
|
||||
and then all third phases.
|
||||
|
||||
The three phases are:
|
||||
|
||||
1. The **enter** phase is executed when the object enters reset. It resets only
|
||||
local state of the object; it must not do anything that has a side-effect
|
||||
on other objects, such as raising or lowering a qemu_irq line or reading or
|
||||
writing guest memory.
|
||||
|
||||
2. The **hold** phase is executed for entry into reset, once every object in the
|
||||
group which is being reset has had its *enter* phase executed. At this point
|
||||
devices can do actions that affect other objects.
|
||||
|
||||
3. The **exit** phase is executed when the object leaves the reset state.
|
||||
Actions affecting other objects are permitted.
|
||||
|
||||
As said in previous section, the interface maintains a count of reset. This
|
||||
count is used to ensure phases are executed only when required. *enter* and
|
||||
*hold* phases are executed only when asserting reset for the first time
|
||||
(if an object is already in reset state when calling
|
||||
``resettable_assert_reset()`` or ``resettable_reset()``, they are not
|
||||
executed).
|
||||
The *exit* phase is executed only when the last reset operation ends. Therefore
|
||||
the object does not need to care how many of reset controllers it has and how
|
||||
many of them have started a reset.
|
||||
|
||||
DMA capable devices are expected to cancel all outstanding DMA operations
|
||||
during either 'enter' or 'hold' phases. IOMMUs are expected to reset during
|
||||
the 'exit' phase and this sequencing makes sure no outstanding DMA request
|
||||
will fault.
|
||||
|
||||
|
||||
Handling reset in a resettable object
|
||||
-------------------------------------
|
||||
|
||||
This section documents the APIs that an implementation of a resettable object
|
||||
must provide and what functions it has access to. It is intended for people
|
||||
who want to implement or convert a class which has the resettable interface;
|
||||
for example when specializing an existing device or bus.
|
||||
|
||||
Methods to implement
|
||||
....................
|
||||
|
||||
Three methods should be defined or left empty. Each method corresponds to a
|
||||
phase of the reset; they are name ``phases.enter()``, ``phases.hold()`` and
|
||||
``phases.exit()``. They all take the object as parameter. The *enter* method
|
||||
also take the reset type as second parameter.
|
||||
|
||||
When extending an existing class, these methods may need to be extended too.
|
||||
The ``resettable_class_set_parent_phases()`` class function may be used to
|
||||
backup parent class methods.
|
||||
|
||||
Here follows an example to implement reset for a Device which sets an IO while
|
||||
in reset.
|
||||
|
||||
::
|
||||
|
||||
static void mydev_reset_enter(Object *obj, ResetType type)
|
||||
{
|
||||
MyDevClass *myclass = MYDEV_GET_CLASS(obj);
|
||||
MyDevState *mydev = MYDEV(obj);
|
||||
/* call parent class enter phase */
|
||||
if (myclass->parent_phases.enter) {
|
||||
myclass->parent_phases.enter(obj, type);
|
||||
}
|
||||
/* initialize local state only */
|
||||
mydev->var = 0;
|
||||
}
|
||||
|
||||
static void mydev_reset_hold(Object *obj, ResetType type)
|
||||
{
|
||||
MyDevClass *myclass = MYDEV_GET_CLASS(obj);
|
||||
MyDevState *mydev = MYDEV(obj);
|
||||
/* call parent class hold phase */
|
||||
if (myclass->parent_phases.hold) {
|
||||
myclass->parent_phases.hold(obj, type);
|
||||
}
|
||||
/* set an IO */
|
||||
qemu_set_irq(mydev->irq, 1);
|
||||
}
|
||||
|
||||
static void mydev_reset_exit(Object *obj, ResetType type)
|
||||
{
|
||||
MyDevClass *myclass = MYDEV_GET_CLASS(obj);
|
||||
MyDevState *mydev = MYDEV(obj);
|
||||
/* call parent class exit phase */
|
||||
if (myclass->parent_phases.exit) {
|
||||
myclass->parent_phases.exit(obj, type);
|
||||
}
|
||||
/* clear an IO */
|
||||
qemu_set_irq(mydev->irq, 0);
|
||||
}
|
||||
|
||||
typedef struct MyDevClass {
|
||||
MyParentClass parent_class;
|
||||
/* to store eventual parent reset methods */
|
||||
ResettablePhases parent_phases;
|
||||
} MyDevClass;
|
||||
|
||||
static void mydev_class_init(ObjectClass *class, const void *data)
|
||||
{
|
||||
MyDevClass *myclass = MYDEV_CLASS(class);
|
||||
ResettableClass *rc = RESETTABLE_CLASS(class);
|
||||
resettable_class_set_parent_phases(rc,
|
||||
mydev_reset_enter,
|
||||
mydev_reset_hold,
|
||||
mydev_reset_exit,
|
||||
&myclass->parent_phases);
|
||||
}
|
||||
|
||||
In the above example, we override all three phases. It is possible to override
|
||||
only some of them by passing NULL instead of a function pointer to
|
||||
``resettable_class_set_parent_phases()``. For example, the following will
|
||||
only override the *enter* phase and leave *hold* and *exit* untouched::
|
||||
|
||||
resettable_class_set_parent_phases(rc, mydev_reset_enter, NULL, NULL,
|
||||
&myclass->parent_phases);
|
||||
|
||||
This is equivalent to providing a trivial implementation of the hold and exit
|
||||
phases which does nothing but call the parent class's implementation of the
|
||||
phase.
|
||||
|
||||
Polling the reset state
|
||||
.......................
|
||||
|
||||
Resettable interface provides the ``resettable_is_in_reset()`` function.
|
||||
This function returns true if the object parameter is currently under reset.
|
||||
|
||||
An object is under reset from the beginning of the *enter* phase (before
|
||||
either its children or its own enter method is called) to the *exit*
|
||||
phase. During *enter* and *hold* phase only, the function will return that the
|
||||
object is in reset. The state is changed after the *exit* is propagated to
|
||||
its children and just before calling the object's own *exit* method.
|
||||
|
||||
This function may be used if the object behavior has to be adapted
|
||||
while in reset state. For example if a device has an irq input,
|
||||
it will probably need to ignore it while in reset; then it can for
|
||||
example check the reset state at the beginning of the irq callback.
|
||||
|
||||
Note that until migration of the reset state is supported, an object
|
||||
should not be left in reset. So apart from being currently executing
|
||||
one of the reset phases, the only cases when this function will return
|
||||
true is if an external interaction (like changing an io) is made during
|
||||
*hold* or *exit* phase of another object in the same reset group.
|
||||
|
||||
Helpers ``device_is_in_reset()`` and ``bus_is_in_reset()`` are also provided
|
||||
for devices and buses and should be preferred.
|
||||
|
||||
|
||||
Base class handling of reset
|
||||
----------------------------
|
||||
|
||||
This section documents parts of the reset mechanism that you only need to know
|
||||
about if you are extending it to work with a new base class other than
|
||||
DeviceClass or BusClass, or maintaining the existing code in those classes. Most
|
||||
people can ignore it.
|
||||
|
||||
Methods to implement
|
||||
....................
|
||||
|
||||
There are two other methods that need to exist in a class implementing the
|
||||
interface: ``get_state()`` and ``child_foreach()``.
|
||||
|
||||
``get_state()`` is simple. *resettable* is an interface and, as a consequence,
|
||||
does not have any class state structure. But in order to factorize the code, we
|
||||
need one. This method must return a pointer to ``ResettableState`` structure.
|
||||
The structure must be allocated by the base class; preferably it should be
|
||||
located inside the object instance structure.
|
||||
|
||||
``child_foreach()`` is more complex. It should execute the given callback on
|
||||
every reset child of the given resettable object. All children must be
|
||||
resettable too. Additional parameters (a reset type and an opaque pointer) must
|
||||
be passed to the callback too.
|
||||
|
||||
In ``DeviceClass`` and ``BusClass`` the ``ResettableState`` is located in the
|
||||
``DeviceState`` and ``BusState`` structures. ``child_foreach()`` is implemented
|
||||
to follow the bus hierarchy; for a bus, it calls the function on every child
|
||||
device; for a device, it calls the function on every bus child. When we reset
|
||||
the main system bus, we reset the whole machine bus tree.
|
||||
|
||||
Changing a resettable parent
|
||||
............................
|
||||
|
||||
One thing which should be taken care of by the base class is handling reset
|
||||
hierarchy changes.
|
||||
|
||||
The reset hierarchy is supposed to be static and built during machine creation.
|
||||
But there are actually some exceptions. To cope with this, the resettable API
|
||||
provides ``resettable_change_parent()``. This function allows to set, update or
|
||||
remove the parent of a resettable object after machine creation is done. As
|
||||
parameters, it takes the object being moved, the old parent if any and the new
|
||||
parent if any.
|
||||
|
||||
This function can be used at any time when not in a reset operation. During
|
||||
a reset operation it must be used only in *hold* phase. Using it in *enter* or
|
||||
*exit* phase is an error.
|
||||
Also it should not be used during machine creation, although it is harmless to
|
||||
do so: the function is a no-op as long as old and new parent are NULL or not
|
||||
in reset.
|
||||
|
||||
There is currently 2 cases where this function is used:
|
||||
|
||||
1. *device hotplug*; it means a new device is introduced on a live bus.
|
||||
|
||||
2. *hot bus change*; it means an existing live device is added, moved or
|
||||
removed in the bus hierarchy. At the moment, it occurs only in the raspi
|
||||
machines for changing the sdbus used by sd card.
|
||||
|
||||
Reset of the complete system
|
||||
----------------------------
|
||||
|
||||
Reset of the complete system is a little complicated. The typical
|
||||
flow is:
|
||||
|
||||
1. Code which wishes to reset the entire system does so by calling
|
||||
``qemu_system_reset_request()``. This schedules a reset, but the
|
||||
reset will happen asynchronously after the function returns.
|
||||
That makes this safe to call from, for example, device models.
|
||||
|
||||
2. The function which is called to make the reset happen is
|
||||
``qemu_system_reset()``. Generally only core system code should
|
||||
call this directly.
|
||||
|
||||
3. ``qemu_system_reset()`` calls the ``MachineClass::reset`` method of
|
||||
the current machine, if it has one. That method must call
|
||||
``qemu_devices_reset()``. If the machine has no reset method,
|
||||
``qemu_system_reset()`` calls ``qemu_devices_reset()`` directly.
|
||||
|
||||
4. ``qemu_devices_reset()`` performs a reset of the system, using
|
||||
the three-phase mechanism listed above. It resets all objects
|
||||
that were registered with it using ``qemu_register_resettable()``.
|
||||
It also calls all the functions registered with it using
|
||||
``qemu_register_reset()``. Those functions are called during the
|
||||
"hold" phase of this reset.
|
||||
|
||||
5. The most important object that this reset resets is the
|
||||
'sysbus' bus. The sysbus bus is the root of the qbus tree. This
|
||||
means that all devices on the sysbus are reset, and all their
|
||||
child buses, and all the devices on those child buses.
|
||||
|
||||
6. Devices which are not on the qbus tree are *not* automatically
|
||||
reset! (The most obvious example of this is CPU objects, but
|
||||
anything that directly inherits from ``TYPE_OBJECT`` or ``TYPE_DEVICE``
|
||||
rather than from ``TYPE_SYS_BUS_DEVICE`` or some other plugs-into-a-bus
|
||||
type will be in this category.) You need to therefore arrange for these
|
||||
to be reset in some other way (e.g. using ``qemu_register_resettable()``
|
||||
or ``qemu_register_reset()``).
|
||||
@@ -0,0 +1,499 @@
|
||||
.. |msrv| replace:: 1.83.0
|
||||
|
||||
Rust in QEMU
|
||||
============
|
||||
|
||||
Rust in QEMU is a project to enable using the Rust programming language
|
||||
to add new functionality to QEMU.
|
||||
|
||||
Right now, the focus is on making it possible to write devices that inherit
|
||||
from ``SysBusDevice`` in `*safe*`__ Rust. Later, it may become possible
|
||||
to write other kinds of devices (e.g. PCI devices that can do DMA),
|
||||
complete boards, or backends (e.g. block device formats).
|
||||
|
||||
__ https://doc.rust-lang.org/nomicon/meet-safe-and-unsafe.html
|
||||
|
||||
Building the Rust in QEMU code
|
||||
------------------------------
|
||||
|
||||
The Rust in QEMU code is included in the emulators via Meson. Meson
|
||||
invokes rustc directly, building static libraries that are then linked
|
||||
together with the C code. This is completely automatic when you run
|
||||
``make`` or ``ninja``.
|
||||
|
||||
However, Meson is able to consume ``Cargo.toml`` files and tries
|
||||
to be easy to use for people who are accustomed to the more "normal"
|
||||
Cargo-based development workflow. In the case of QEMU, in addition,
|
||||
it is possible to use ``cargo`` for common Rust-specific coding
|
||||
tasks, in particular to invoke ``clippy``, ``rustfmt`` and ``rustdoc``.
|
||||
|
||||
To this end, QEMU includes a ``build.rs`` build script that picks up
|
||||
generated sources from QEMU's build directory and puts it in Cargo's
|
||||
output directory (typically ``target/``). A vanilla invocation
|
||||
of Cargo will complain that it cannot find the generated sources,
|
||||
which can be fixed in different ways:
|
||||
|
||||
* by using Makefile targets, provided by Meson, that run ``clippy`` or
|
||||
``rustdoc``:
|
||||
|
||||
make clippy
|
||||
make rustdoc
|
||||
|
||||
A target for ``rustfmt`` is also declared in ``rust/meson.build``:
|
||||
|
||||
make rustfmt
|
||||
|
||||
* by invoking ``cargo`` through the Meson `development environment`__
|
||||
feature::
|
||||
|
||||
pyvenv/bin/meson devenv -w ../rust cargo clippy --tests
|
||||
pyvenv/bin/meson devenv -w ../rust cargo fmt
|
||||
|
||||
If you are going to use ``cargo`` repeatedly, ``pyvenv/bin/meson devenv``
|
||||
will enter a shell where commands like ``cargo fmt`` just work.
|
||||
|
||||
__ https://mesonbuild.com/Commands.html#devenv
|
||||
|
||||
* by pointing the ``MESON_BUILD_ROOT`` to the top of your QEMU build
|
||||
tree. This third method is useful if you are using ``rust-analyzer``;
|
||||
you can set the environment variable through the
|
||||
``rust-analyzer.cargo.extraEnv`` setting.
|
||||
|
||||
As shown above, you can use the ``--tests`` option as usual to operate on test
|
||||
code. Note however that you cannot *build* or run tests via ``cargo``, because
|
||||
they need support C code from QEMU that Cargo does not know about. Tests can
|
||||
be run via Meson (``pyvenv/bin/meson test``) or ``make``::
|
||||
|
||||
make check-rust
|
||||
|
||||
Note that doctests require all ``.o`` files from the build to be available.
|
||||
|
||||
Supported tools
|
||||
'''''''''''''''
|
||||
|
||||
QEMU supports rustc version 1.83.0 and newer. The following features
|
||||
from relatively new versions of Rust are not used for historical reasons;
|
||||
patches are welcome:
|
||||
|
||||
* associated constants are still explicitly marked ``'static`` (`changed in
|
||||
1.81.0`__)
|
||||
|
||||
* ``&raw`` (stable in 1.82.0).
|
||||
|
||||
* NUL-terminated file names with ``#[track_caller]`` are scheduled for
|
||||
inclusion as ``#![feature(location_file_nul)]``, but it will be a while
|
||||
before QEMU can use them. For now, there is special code in
|
||||
``util/error.c`` to support non-NUL-terminated file names.
|
||||
|
||||
Associated const equality would be nice to have for some users of
|
||||
``callbacks::FnCall``, but is still experimental. Const assertions
|
||||
are used instead.
|
||||
|
||||
__ https://github.com/rust-lang/rust/pull/125258
|
||||
|
||||
QEMU also supports version 0.60.x of bindgen, which is missing option
|
||||
``--generate-cstr``. This option requires version 0.66.x and will
|
||||
be adopted as soon as supporting these older versions is not necessary
|
||||
anymore.
|
||||
|
||||
Writing Rust code in QEMU
|
||||
-------------------------
|
||||
|
||||
QEMU includes several crates:
|
||||
|
||||
* ``common`` provides Rust-only utilities
|
||||
|
||||
* ``bql``, ``chardev``, ``hw/core``, ``migration``, ``qom``, ``system``,
|
||||
``util`` for bindings to respective QEMU C library APIs
|
||||
|
||||
* ``qemu_macros`` defines several procedural macros that are useful when
|
||||
writing C code
|
||||
|
||||
* ``pl011`` (under ``rust/hw/char/pl011``) and ``hpet`` (under ``rust/hw/timer/hpet``)
|
||||
are sample devices that demonstrate Rust binding usage and ``qemu_macros``, and are
|
||||
used to further develop them. These two crates are functional\ [#issues]_ replacements
|
||||
for the ``hw/char/pl011.c`` and ``hw/timer/hpet.c`` files.
|
||||
|
||||
.. [#issues] The ``pl011`` crate is synchronized with ``hw/char/pl011.c``
|
||||
as of commit 3e0f118f82. The ``hpet`` crate is synchronized as of
|
||||
commit 1433e38cc8. Both are lacking tracing functionality.
|
||||
|
||||
This section explains how to work with them.
|
||||
|
||||
Status
|
||||
''''''
|
||||
|
||||
The stability of the modules can be defined as:
|
||||
|
||||
- *complete*: ready for use in new devices; if applicable, the API supports the
|
||||
full functionality available in C
|
||||
|
||||
- *stable*: ready for production use, the API is safe and should not undergo
|
||||
major changes
|
||||
|
||||
- *proof of concept*: the API is subject to change but allows working with safe
|
||||
Rust
|
||||
|
||||
- *initial*: the API is in its initial stages; it requires large amount of
|
||||
unsafe code; it might have soundness or type-safety issues
|
||||
|
||||
The status of the modules is as follows:
|
||||
|
||||
========================== ======================
|
||||
module status
|
||||
========================== ======================
|
||||
``bql::cell`` stable
|
||||
``common::assertions`` stable
|
||||
``common::bitops`` complete
|
||||
``common::callbacks`` complete
|
||||
``common::errno`` complete
|
||||
``common::zeroable`` stable
|
||||
``hwcore::irq`` complete
|
||||
``hwcore::qdev`` stable
|
||||
``hwcore::sysbus`` stable
|
||||
``migration::migratable`` proof of concept
|
||||
``migration::vmstate`` stable
|
||||
``qom`` stable
|
||||
``system::memory`` stable
|
||||
``util::error`` stable
|
||||
``util::log`` proof of concept
|
||||
``util::module`` complete
|
||||
``util::timer`` stable
|
||||
========================== ======================
|
||||
|
||||
.. note::
|
||||
API stability is not a promise, if anything because the C APIs are not a stable
|
||||
interface either. Also, ``unsafe`` interfaces may be replaced by safe interfaces
|
||||
later.
|
||||
|
||||
Naming convention
|
||||
'''''''''''''''''
|
||||
|
||||
C function names usually are prefixed according to the data type that they
|
||||
apply to, for example ``timer_mod`` or ``sysbus_connect_irq``. Furthermore,
|
||||
both function and structs sometimes have a ``qemu_`` or ``QEMU`` prefix.
|
||||
Generally speaking, these are all removed in the corresponding Rust functions:
|
||||
``QEMUTimer`` becomes ``timer::Timer``, ``timer_mod`` becomes ``Timer::modify``,
|
||||
``sysbus_connect_irq`` becomes ``SysBusDeviceMethods::connect_irq``.
|
||||
|
||||
Sometimes however a name appears multiple times in the QOM class hierarchy,
|
||||
and the only difference is in the prefix. An example is ``qdev_realize`` and
|
||||
``sysbus_realize``. In such cases, whenever a name is not unique in
|
||||
the hierarchy, always add the prefix to the classes that are lower in
|
||||
the hierarchy; for the top class, decide on a case by case basis.
|
||||
|
||||
For example:
|
||||
|
||||
========================== =========================================
|
||||
``device_cold_reset()`` ``DeviceMethods::cold_reset()``
|
||||
``pci_device_reset()`` ``PciDeviceMethods::pci_device_reset()``
|
||||
``pci_bridge_reset()`` ``PciBridgeMethods::pci_bridge_reset()``
|
||||
========================== =========================================
|
||||
|
||||
Here, the name is not exactly the same, but nevertheless ``PciDeviceMethods``
|
||||
adds the prefix to avoid confusion, because the functionality of
|
||||
``device_cold_reset()`` and ``pci_device_reset()`` is subtly different.
|
||||
|
||||
In this case, however, no prefix is needed:
|
||||
|
||||
========================== =========================================
|
||||
``device_realize()`` ``DeviceMethods::realize()``
|
||||
``sysbus_realize()`` ``SysbusDeviceMethods::sysbus_realize()``
|
||||
``pci_realize()`` ``PciDeviceMethods::pci_realize()``
|
||||
========================== =========================================
|
||||
|
||||
Here, the lower classes do not add any functionality, and mostly
|
||||
provide extra compile-time checking; the basic *realize* functionality
|
||||
is the same for all devices. Therefore, ``DeviceMethods`` does not
|
||||
add the prefix.
|
||||
|
||||
Whenever a name is unique in the hierarchy, instead, you should
|
||||
always remove the class name prefix.
|
||||
|
||||
Common pitfalls
|
||||
'''''''''''''''
|
||||
|
||||
Rust has very strict rules with respect to how you get an exclusive (``&mut``)
|
||||
reference; failure to respect those rules is a source of undefined behavior.
|
||||
In particular, even if a value is loaded from a raw mutable pointer (``*mut``),
|
||||
it *cannot* be casted to ``&mut`` unless the value was stored to the ``*mut``
|
||||
from a mutable reference. Furthermore, it is undefined behavior if any
|
||||
shared reference was created between the store to the ``*mut`` and the load::
|
||||
|
||||
let mut p: u32 = 42;
|
||||
let p_mut = &mut p; // 1
|
||||
let p_raw = p_mut as *mut u32; // 2
|
||||
|
||||
// p_raw keeps the mutable reference "alive"
|
||||
|
||||
let p_shared = &p; // 3
|
||||
println!("access from &u32: {}", *p_shared);
|
||||
|
||||
// Bring back the mutable reference, its lifetime overlaps
|
||||
// with that of a shared reference.
|
||||
let p_mut = unsafe { &mut *p_raw }; // 4
|
||||
println!("access from &mut 32: {}", *p_mut);
|
||||
|
||||
println!("access from &u32: {}", *p_shared); // 5
|
||||
|
||||
These rules can be tested with `MIRI`__, for example.
|
||||
|
||||
__ https://github.com/rust-lang/miri
|
||||
|
||||
Almost all Rust code in QEMU will involve QOM objects, and pointers to these
|
||||
objects are *shared*, for example because they are part of the QOM composition
|
||||
tree. This creates exactly the above scenario:
|
||||
|
||||
1. a QOM object is created
|
||||
|
||||
2. a ``*mut`` is created, for example as the opaque value for a ``MemoryRegion``
|
||||
|
||||
3. the QOM object is placed in the composition tree
|
||||
|
||||
4. a memory access dereferences the opaque value to a ``&mut``
|
||||
|
||||
5. but the shared reference is still present in the composition tree
|
||||
|
||||
Because of this, QOM objects should almost always use ``&self`` instead
|
||||
of ``&mut self``; access to internal fields must use *interior mutability*
|
||||
to go from a shared reference to a ``&mut``.
|
||||
|
||||
Whenever C code provides you with an opaque ``void *``, avoid converting it
|
||||
to a Rust mutable reference, and use a shared reference instead. The
|
||||
``bql::cell`` module provides wrappers that can be used to tell the
|
||||
Rust compiler about interior mutability, and optionally to enforce locking
|
||||
rules for the "Big QEMU Lock". In the future, similar cell types might
|
||||
also be provided for ``AioContext``-based locking as well.
|
||||
|
||||
In particular, device code will usually rely on the ``BqlRefCell`` and
|
||||
``BqlCell`` type to ensure that data is accessed correctly under the
|
||||
"Big QEMU Lock". These cell types are also known to the ``vmstate``
|
||||
crate, which is able to "look inside" them when building an in-memory
|
||||
representation of a ``struct``'s layout. Note that the same is not true
|
||||
of a ``RefCell`` or ``Mutex``.
|
||||
|
||||
Bindings code instead will usually use the ``Opaque`` type, which hides
|
||||
the contents of the underlying struct and can be easily converted to
|
||||
a raw pointer, for use in calls to C functions. It can be used for
|
||||
example as follows::
|
||||
|
||||
#[repr(transparent)]
|
||||
#[derive(Debug, common::Wrapper)]
|
||||
pub struct Object(Opaque<bindings::Object>);
|
||||
|
||||
where the special ``derive`` macro provides useful methods such as
|
||||
``from_raw``, ``as_ptr`, ``as_mut_ptr`` and ``raw_get``. The bindings will
|
||||
then manually check for the big QEMU lock with assertions, which allows
|
||||
the wrapper to be declared thread-safe::
|
||||
|
||||
unsafe impl Send for Object {}
|
||||
unsafe impl Sync for Object {}
|
||||
|
||||
Writing bindings to C code
|
||||
''''''''''''''''''''''''''
|
||||
|
||||
Here are some things to keep in mind when working on the QEMU Rust crate.
|
||||
|
||||
**Look at existing code**
|
||||
Very often, similar idioms in C code correspond to similar tricks in
|
||||
Rust bindings. If the C code uses ``offsetof``, look at qdev properties
|
||||
or ``vmstate``. If the C code has a complex const struct, look at
|
||||
``MemoryRegion``. Reuse existing patterns for handling lifetimes;
|
||||
for example use ``&T`` for QOM objects that do not need a reference
|
||||
count (including those that can be embedded in other objects) and
|
||||
``Owned<T>`` for those that need it.
|
||||
|
||||
**Use the type system**
|
||||
Bindings often will need access information that is specific to a type
|
||||
(either a builtin one or a user-defined one) in order to pass it to C
|
||||
functions. Put them in a trait and access it through generic parameters.
|
||||
The ``vmstate`` module has examples of how to retrieve type information
|
||||
for the fields of a Rust ``struct``.
|
||||
|
||||
**Prefer unsafe traits to unsafe functions**
|
||||
Unsafe traits are much easier to prove correct than unsafe functions.
|
||||
They are an excellent place to store metadata that can later be accessed
|
||||
by generic functions. C code usually places metadata in global variables;
|
||||
in Rust, they can be stored in traits and then turned into ``static``
|
||||
variables. Often, unsafe traits can be generated by procedural macros.
|
||||
|
||||
**Document limitations due to old Rust versions**
|
||||
If you need to settle for an inferior solution because of the currently
|
||||
supported set of Rust versions, document it in the source and in this
|
||||
file. This ensures that it can be fixed when the minimum supported
|
||||
version is bumped.
|
||||
|
||||
**Keep locking in mind**.
|
||||
When marking a type ``Sync``, be careful of whether it needs the big
|
||||
QEMU lock. Use ``BqlCell`` and ``BqlRefCell`` for interior data,
|
||||
or assert ``bql_locked()``.
|
||||
|
||||
**Don't be afraid of complexity, but document and isolate it**
|
||||
It's okay to be tricky; device code is written more often than bindings
|
||||
code and it's important that it is idiomatic. However, you should strive
|
||||
to isolate any tricks in a place (for example a ``struct``, a trait
|
||||
or a macro) where it can be documented and tested. If needed, include
|
||||
toy versions of the code in the documentation.
|
||||
|
||||
FFI Binding Generation
|
||||
''''''''''''''''''''''
|
||||
|
||||
QEMU's Rust integration uses multiple ``*-sys`` crates that contain raw FFI
|
||||
bindings to different QEMU subsystems. These crates mirror the dependency
|
||||
structure that meson.build uses for C code, and which is reflected in
|
||||
``static_library()`` declarations. For example:
|
||||
|
||||
* util-sys: Basic utilities (no dependencies)
|
||||
* qom-sys: QEMU Object Model (depends on util-sys)
|
||||
* chardev-sys: Character devices (depends on qom-sys, util-sys)
|
||||
* hwcore-sys: Hardware core (depends on qom-sys, util-sys)
|
||||
* migration-sys: Migration (depends on util-sys)
|
||||
* system-sys: System-level APIs (depends on all others)
|
||||
|
||||
Having multiple crates avoids massive rebuilds of all Rust code when C headers
|
||||
are changed. On the other hand, bindgen is not aware of how headers are split
|
||||
across crates, and therefore it would generate declarations for dependencies
|
||||
again. These duplicate declarations are not only large, they create distinct
|
||||
types and therefore they are incompatible with each other.
|
||||
|
||||
Bindgen Configuration
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Bindgen options such as symbol blocklists or how to configure enums can be
|
||||
defined in each crate's ``Cargo.toml`` via a ``[package.metadata.bindgen]`` section.
|
||||
For example::
|
||||
|
||||
[package.metadata.bindgen]
|
||||
header = "wrapper.h" # Main header file for this crate
|
||||
rustified-enum = ["QEMUClockType"] # Enums to generate as Rust enums
|
||||
bitfield-enum = ["VMStateFlags"] # Enums to treat as bitfields
|
||||
blocklist-function = [ # Functions to exclude
|
||||
"vmstate_register_ram",
|
||||
"vmstate_unregister_ram"
|
||||
]
|
||||
additional-files = [ # Extra files to allowlist
|
||||
"include/system/memory_ldst.*"
|
||||
]
|
||||
|
||||
All bindgen options are supported in the metadata section. The complete list
|
||||
can be found in ``rust/bindings/generate_bindgen_args.py``.
|
||||
|
||||
Dependency Management
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
By examining the dependency chain before bindgen creates the code for
|
||||
the ``*-sys`` crates, the build system ensures that header files included in
|
||||
one crate are blocked from appearing in dependent crates, thus avoiding
|
||||
duplicate definitions. Dependent crates can import the definition via
|
||||
"use" statements.
|
||||
|
||||
This dependency-aware binding generation is handled automatically by
|
||||
``rust/bindings/generate_bindgen_args.py``, which processes the Cargo.toml
|
||||
files in dependency order and generates appropriate ``--allowlist-file`` and
|
||||
``--blocklist-file`` arguments for bindgen.
|
||||
|
||||
Writing procedural macros
|
||||
'''''''''''''''''''''''''
|
||||
|
||||
By conventions, procedural macros are split in two functions, one
|
||||
returning ``Result<proc_macro2::TokenStream, syn::Error>`` with the body of
|
||||
the procedural macro, and the second returning ``proc_macro::TokenStream``
|
||||
which is the actual procedural macro. The former's name is the same as
|
||||
the latter with the ``_or_error`` suffix. The code for the latter is more
|
||||
or less fixed; it follows the following template, which is fixed apart
|
||||
from the type after ``as`` in the invocation of ``parse_macro_input!``::
|
||||
|
||||
#[proc_macro_derive(Object)]
|
||||
pub fn derive_object(input: TokenStream) -> TokenStream {
|
||||
let input = parse_macro_input!(input as DeriveInput);
|
||||
|
||||
derive_object_or_error(input)
|
||||
.unwrap_or_else(syn::Error::into_compile_error)
|
||||
.into()
|
||||
}
|
||||
|
||||
The ``qemu_macros`` crate has utility functions to examine a
|
||||
``DeriveInput`` and perform common checks (e.g. looking for a struct
|
||||
with named fields). These functions return ``Result<..., syn::Error>``
|
||||
and can be used easily in the procedural macro function::
|
||||
|
||||
fn derive_object_or_error(input: DeriveInput) ->
|
||||
Result<proc_macro2::TokenStream, Error>
|
||||
{
|
||||
is_c_repr(&input, "#[derive(Object)]")?;
|
||||
|
||||
let name = &input.ident;
|
||||
let parent = &get_fields(&input, "#[derive(Object)]")?[0].ident;
|
||||
...
|
||||
}
|
||||
|
||||
Use procedural macros with care. They are mostly useful for two purposes:
|
||||
|
||||
* Performing consistency checks; for example ``#[derive(Object)]`` checks
|
||||
that the structure has ``#[repr[C])`` and that the type of the first field
|
||||
is consistent with the ``ObjectType`` declaration.
|
||||
|
||||
* Extracting information from Rust source code into traits, typically based
|
||||
on types and attributes. For example, ``#[derive(TryInto)]`` builds an
|
||||
implementation of ``TryFrom``, and it uses the ``#[repr(...)]`` attribute
|
||||
as the ``TryFrom`` source and error types.
|
||||
|
||||
Procedural macros can be hard to debug and test; if the code generation
|
||||
exceeds a few lines of code, it may be worthwhile to delegate work to
|
||||
"regular" declarative (``macro_rules!``) macros and write unit tests for
|
||||
those instead.
|
||||
|
||||
|
||||
Coding style
|
||||
''''''''''''
|
||||
|
||||
Code should pass clippy and be formatted with rustfmt.
|
||||
|
||||
Right now, only the nightly version of ``rustfmt`` is supported. This
|
||||
might change in the future. While CI checks for correct formatting via
|
||||
``cargo fmt --check``, maintainers can fix this for you when applying patches.
|
||||
|
||||
It is expected that QEMU Rust crates provides full ``rustdoc`` documentation for
|
||||
bindings that are in their final shape or close.
|
||||
|
||||
Adding dependencies
|
||||
-------------------
|
||||
|
||||
Generally, the set of dependent crates is kept small. Think twice before
|
||||
adding a new external crate, especially if it comes with a large set of
|
||||
dependencies itself. Sometimes QEMU only needs a small subset of the
|
||||
functionality; see for example QEMU's ``assertions`` module. Also,
|
||||
choose a version of the crate that works with QEMU's minimum supported
|
||||
Rust version (|msrv|).
|
||||
|
||||
On top of this recommendation, adding external crates to QEMU is a
|
||||
slightly complicated process, mostly due to the need to teach Meson how
|
||||
to download them. While QEMU uses Meson's support for parsing ``Cargo.toml``
|
||||
files, it ships ``.wrap`` files instead of using ``Cargo.lock``; this way,
|
||||
distros can adjust the set of dependencies to the exact versions they use.
|
||||
The versions specified in QEMU's ``Cargo.lock`` must be the same as the
|
||||
one in the wrap file.
|
||||
|
||||
The wrap file must be named ``NAME-SEMVER-rs.wrap``, where ``NAME``
|
||||
is the name of the crate and ``SEMVER`` is the version up to and including the
|
||||
first non-zero number. For example, a crate with version ``0.2.3`` will use
|
||||
``0.2`` for its ``SEMVER``, while a crate with version ``1.0.84`` will use ``1``.
|
||||
|
||||
Usually, Meson is able to figure out how to build the crate, and also handles
|
||||
cross compilation correctly. For crates that have a ``build.rs`` file,
|
||||
equivalent rules must be added to
|
||||
``subprojects/packagefiles/NAME-SEMVER-rs/meson/meson.build``.
|
||||
The file can modify the ``extra_args`` and ``extra_deps`` variables,
|
||||
which contain respectively the compiler arguments and external dependencies
|
||||
for the crate.
|
||||
|
||||
After every change to the ``meson/meson.build`` file you have to update the
|
||||
patched version with ``meson subprojects update --reset ``NAME-SEMVER-rs``.
|
||||
This might be automated in the future.
|
||||
|
||||
Also, after every change to the file it is strongly suggested to do a dummy
|
||||
change to the ``.wrap`` file (for example adding a comment like ``# version 2``),
|
||||
which will help Meson notice that the subproject is out of date.
|
||||
|
||||
As a last step, add the new subproject to ``scripts/archive-source.sh``,
|
||||
``scripts/make-release`` and ``subprojects/.gitignore``.
|
||||
@@ -0,0 +1,170 @@
|
||||
QAPI interface for S390 CPU topology
|
||||
====================================
|
||||
|
||||
The following sections will explain the QAPI interface for S390 CPU topology
|
||||
with the help of exemplary output.
|
||||
For this, let's assume that QEMU has been started with the following
|
||||
command, defining 4 CPUs, where CPU[0] is defined by the -smp argument and will
|
||||
have default values:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
qemu-system-s390x \
|
||||
-enable-kvm \
|
||||
-cpu z14,ctop=on \
|
||||
-smp 1,drawers=3,books=3,sockets=2,cores=2,maxcpus=36 \
|
||||
-device z14-s390x-cpu,core-id=19,entitlement=high \
|
||||
-device z14-s390x-cpu,core-id=11,entitlement=low \
|
||||
-device z14-s390x-cpu,core-id=12,entitlement=high \
|
||||
...
|
||||
|
||||
Additions to query-cpus-fast
|
||||
----------------------------
|
||||
|
||||
The command query-cpus-fast allows querying the topology tree and
|
||||
modifiers for all configured vCPUs.
|
||||
|
||||
.. code-block:: QMP
|
||||
|
||||
{ "execute": "query-cpus-fast" }
|
||||
{
|
||||
"return": [
|
||||
{
|
||||
"dedicated": false,
|
||||
"thread-id": 536993,
|
||||
"props": {
|
||||
"core-id": 0,
|
||||
"socket-id": 0,
|
||||
"drawer-id": 0,
|
||||
"book-id": 0
|
||||
},
|
||||
"cpu-state": "operating",
|
||||
"entitlement": "medium",
|
||||
"qom-path": "/machine/unattached/device[0]",
|
||||
"cpu-index": 0,
|
||||
"target": "s390x"
|
||||
},
|
||||
{
|
||||
"dedicated": false,
|
||||
"thread-id": 537003,
|
||||
"props": {
|
||||
"core-id": 19,
|
||||
"socket-id": 1,
|
||||
"drawer-id": 0,
|
||||
"book-id": 2
|
||||
},
|
||||
"cpu-state": "operating",
|
||||
"entitlement": "high",
|
||||
"qom-path": "/machine/peripheral-anon/device[0]",
|
||||
"cpu-index": 19,
|
||||
"target": "s390x"
|
||||
},
|
||||
{
|
||||
"dedicated": false,
|
||||
"thread-id": 537004,
|
||||
"props": {
|
||||
"core-id": 11,
|
||||
"socket-id": 1,
|
||||
"drawer-id": 0,
|
||||
"book-id": 1
|
||||
},
|
||||
"cpu-state": "operating",
|
||||
"entitlement": "low",
|
||||
"qom-path": "/machine/peripheral-anon/device[1]",
|
||||
"cpu-index": 11,
|
||||
"target": "s390x"
|
||||
},
|
||||
{
|
||||
"dedicated": true,
|
||||
"thread-id": 537005,
|
||||
"props": {
|
||||
"core-id": 12,
|
||||
"socket-id": 0,
|
||||
"drawer-id": 3,
|
||||
"book-id": 2
|
||||
},
|
||||
"cpu-state": "operating",
|
||||
"entitlement": "high",
|
||||
"qom-path": "/machine/peripheral-anon/device[2]",
|
||||
"cpu-index": 12,
|
||||
"target": "s390x"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
QAPI command: set-cpu-topology
|
||||
------------------------------
|
||||
|
||||
The command set-cpu-topology allows modifying the topology tree
|
||||
or the topology modifiers of a vCPU in the configuration.
|
||||
|
||||
.. code-block:: QMP
|
||||
|
||||
{ "execute": "set-cpu-topology",
|
||||
"arguments": {
|
||||
"core-id": 11,
|
||||
"socket-id": 0,
|
||||
"book-id": 0,
|
||||
"drawer-id": 0,
|
||||
"entitlement": "low",
|
||||
"dedicated": false
|
||||
}
|
||||
}
|
||||
{"return": {}}
|
||||
|
||||
The core-id parameter is the only mandatory parameter and every
|
||||
unspecified parameter keeps its previous value.
|
||||
|
||||
QAPI event CPU_POLARIZATION_CHANGE
|
||||
----------------------------------
|
||||
|
||||
When a guest requests a modification of the polarization,
|
||||
QEMU sends a CPU_POLARIZATION_CHANGE event.
|
||||
|
||||
When requesting the change, the guest only specifies horizontal or
|
||||
vertical polarization.
|
||||
It is the job of the entity administrating QEMU to set the dedication and fine
|
||||
grained vertical entitlement in response to this event.
|
||||
|
||||
Note that a vertical polarized dedicated vCPU can only have a high
|
||||
entitlement, giving 6 possibilities for vCPU polarization:
|
||||
|
||||
- Horizontal
|
||||
- Horizontal dedicated
|
||||
- Vertical low
|
||||
- Vertical medium
|
||||
- Vertical high
|
||||
- Vertical high dedicated
|
||||
|
||||
Example of the event received when the guest issues the CPU instruction
|
||||
Perform Topology Function PTF(0) to request an horizontal polarization:
|
||||
|
||||
.. code-block:: QMP
|
||||
|
||||
{
|
||||
"timestamp": {
|
||||
"seconds": 1687870305,
|
||||
"microseconds": 566299
|
||||
},
|
||||
"event": "CPU_POLARIZATION_CHANGE",
|
||||
"data": {
|
||||
"polarization": "horizontal"
|
||||
}
|
||||
}
|
||||
|
||||
QAPI query command: query-s390x-cpu-polarization
|
||||
------------------------------------------------
|
||||
|
||||
The query command query-s390x-cpu-polarization returns the current
|
||||
CPU polarization of the machine.
|
||||
In this case the guest previously issued a PTF(1) to request vertical polarization:
|
||||
|
||||
.. code-block:: QMP
|
||||
|
||||
{ "execute": "query-s390x-cpu-polarization" }
|
||||
{
|
||||
"return": {
|
||||
"polarization": "vertical"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,138 @@
|
||||
Booting from real channel-attached devices on s390x
|
||||
===================================================
|
||||
|
||||
s390 hardware IPL
|
||||
-----------------
|
||||
|
||||
The s390 hardware IPL process consists of the following steps.
|
||||
|
||||
1. A READ IPL ccw is constructed in memory location ``0x0``.
|
||||
This ccw, by definition, reads the IPL1 record which is located on the disk
|
||||
at cylinder 0 track 0 record 1. Note that the chain flag is on in this ccw
|
||||
so when it is complete another ccw will be fetched and executed from memory
|
||||
location ``0x08``.
|
||||
|
||||
2. Execute the Read IPL ccw at ``0x00``, thereby reading IPL1 data into ``0x00``.
|
||||
IPL1 data is 24 bytes in length and consists of the following pieces of
|
||||
information: ``[psw][read ccw][tic ccw]``. When the machine executes the Read
|
||||
IPL ccw it read the 24-bytes of IPL1 to be read into memory starting at
|
||||
location ``0x0``. Then the ccw program at ``0x08`` which consists of a read
|
||||
ccw and a tic ccw is automatically executed because of the chain flag from
|
||||
the original READ IPL ccw. The read ccw will read the IPL2 data into memory
|
||||
and the TIC (Transfer In Channel) will transfer control to the channel
|
||||
program contained in the IPL2 data. The TIC channel command is the
|
||||
equivalent of a branch/jump/goto instruction for channel programs.
|
||||
|
||||
NOTE: The ccws in IPL1 are defined by the architecture to be format 0.
|
||||
|
||||
3. Execute IPL2.
|
||||
The TIC ccw instruction at the end of the IPL1 channel program will begin
|
||||
the execution of the IPL2 channel program. IPL2 is stage-2 of the boot
|
||||
process and will contain a larger channel program than IPL1. The point of
|
||||
IPL2 is to find and load either the operating system or a small program that
|
||||
loads the operating system from disk. At the end of this step all or some of
|
||||
the real operating system is loaded into memory and we are ready to hand
|
||||
control over to the guest operating system. At this point the guest
|
||||
operating system is entirely responsible for loading any more data it might
|
||||
need to function.
|
||||
|
||||
NOTE: The IPL2 channel program might read data into memory
|
||||
location ``0x0`` thereby overwriting the IPL1 psw and channel program. This is ok
|
||||
as long as the data placed in location ``0x0`` contains a psw whose instruction
|
||||
address points to the guest operating system code to execute at the end of
|
||||
the IPL/boot process.
|
||||
|
||||
NOTE: The ccws in IPL2 are defined by the architecture to be format 0.
|
||||
|
||||
4. Start executing the guest operating system.
|
||||
The psw that was loaded into memory location ``0x0`` as part of the ipl process
|
||||
should contain the needed flags for the operating system we have loaded. The
|
||||
psw's instruction address will point to the location in memory where we want
|
||||
to start executing the operating system. This psw is loaded (via LPSW
|
||||
instruction) causing control to be passed to the operating system code.
|
||||
|
||||
In a non-virtualized environment this process, handled entirely by the hardware,
|
||||
is kicked off by the user initiating a "Load" procedure from the hardware
|
||||
management console. This "Load" procedure crafts a special "Read IPL" ccw in
|
||||
memory location 0x0 that reads IPL1. It then executes this ccw thereby kicking
|
||||
off the reading of IPL1 data. Since the channel program from IPL1 will be
|
||||
written immediately after the special "Read IPL" ccw, the IPL1 channel program
|
||||
will be executed immediately (the special read ccw has the chaining bit turned
|
||||
on). The TIC at the end of the IPL1 channel program will cause the IPL2 channel
|
||||
program to be executed automatically. After this sequence completes the "Load"
|
||||
procedure then loads the psw from ``0x0``.
|
||||
|
||||
How this all pertains to QEMU (and the kernel)
|
||||
----------------------------------------------
|
||||
|
||||
In theory we should merely have to do the following to IPL/boot a guest
|
||||
operating system from a DASD device:
|
||||
|
||||
1. Place a "Read IPL" ccw into memory location ``0x0`` with chaining bit on.
|
||||
2. Execute channel program at ``0x0``.
|
||||
3. LPSW ``0x0``.
|
||||
|
||||
However, our emulation of the machine's channel program logic within the kernel
|
||||
is missing one key feature that is required for this process to work:
|
||||
non-prefetch of ccw data.
|
||||
|
||||
When we start a channel program we pass the channel subsystem parameters via an
|
||||
ORB (Operation Request Block). One of those parameters is a prefetch bit. If the
|
||||
bit is on then the vfio-ccw kernel driver is allowed to read the entire channel
|
||||
program from guest memory before it starts executing it. This means that any
|
||||
channel commands that read additional channel commands will not work as expected
|
||||
because the newly read commands will only exist in guest memory and NOT within
|
||||
the kernel's channel subsystem memory. The kernel vfio-ccw driver currently
|
||||
requires this bit to be on for all channel programs. This is a problem because
|
||||
the IPL process consists of transferring control from the "Read IPL" ccw
|
||||
immediately to the IPL1 channel program that was read by "Read IPL".
|
||||
|
||||
Not being able to turn off prefetch will also prevent the TIC at the end of the
|
||||
IPL1 channel program from transferring control to the IPL2 channel program.
|
||||
|
||||
Lastly, in some cases (the zipl bootloader for example) the IPL2 program also
|
||||
transfers control to another channel program segment immediately after reading
|
||||
it from the disk. So we need to be able to handle this case.
|
||||
|
||||
What QEMU does
|
||||
--------------
|
||||
|
||||
Since we are forced to live with prefetch we cannot use the very simple IPL
|
||||
procedure we defined in the preceding section. So we compensate by doing the
|
||||
following.
|
||||
|
||||
1. Place "Read IPL" ccw into memory location ``0x0``, but turn off chaining bit.
|
||||
2. Execute "Read IPL" at ``0x0``.
|
||||
|
||||
So now IPL1's psw is at ``0x0`` and IPL1's channel program is at ``0x08``.
|
||||
|
||||
3. Write a custom channel program that will seek to the IPL2 record and then
|
||||
execute the READ and TIC ccws from IPL1. Normally the seek is not required
|
||||
because after reading the IPL1 record the disk is automatically positioned
|
||||
to read the very next record which will be IPL2. But since we are not reading
|
||||
both IPL1 and IPL2 as part of the same channel program we must manually set
|
||||
the position.
|
||||
|
||||
4. Grab the target address of the TIC instruction from the IPL1 channel program.
|
||||
This address is where the IPL2 channel program starts.
|
||||
|
||||
Now IPL2 is loaded into memory somewhere, and we know the address.
|
||||
|
||||
5. Execute the IPL2 channel program at the address obtained in step #4.
|
||||
|
||||
Because this channel program can be dynamic, we must use a special algorithm
|
||||
that detects a READ immediately followed by a TIC and breaks the ccw chain
|
||||
by turning off the chain bit in the READ ccw. When control is returned from
|
||||
the kernel/hardware to the QEMU bios code we immediately issue another start
|
||||
subchannel to execute the remaining TIC instruction. This causes the entire
|
||||
channel program (starting from the TIC) and all needed data to be refetched
|
||||
thereby stepping around the limitation that would otherwise prevent this
|
||||
channel program from executing properly.
|
||||
|
||||
Now the operating system code is loaded somewhere in guest memory and the psw
|
||||
in memory location ``0x0`` will point to entry code for the guest operating
|
||||
system.
|
||||
|
||||
6. LPSW ``0x0``
|
||||
|
||||
LPSW transfers control to the guest operating system and we're done.
|
||||
@@ -0,0 +1,115 @@
|
||||
=======================
|
||||
Secure Coding Practices
|
||||
=======================
|
||||
This document covers topics that both developers and security researchers must
|
||||
be aware of so that they can develop safe code and audit existing code
|
||||
properly.
|
||||
|
||||
Reporting Security Bugs
|
||||
-----------------------
|
||||
For details on how to report security bugs or ask questions about potential
|
||||
security bugs, see the `Security Process wiki page
|
||||
<https://wiki.qemu.org/SecurityProcess>`_.
|
||||
|
||||
General Secure C Coding Practices
|
||||
---------------------------------
|
||||
Most CVEs (security bugs) reported against QEMU are not specific to
|
||||
virtualization or emulation. They are simply C programming bugs. Therefore
|
||||
it's critical to be aware of common classes of security bugs.
|
||||
|
||||
There is a wide selection of resources available covering secure C coding. For
|
||||
example, the `CERT C Coding Standard
|
||||
<https://wiki.sei.cmu.edu/confluence/display/c/SEI+CERT+C+Coding+Standard>`_
|
||||
covers the most important classes of security bugs.
|
||||
|
||||
Instead of describing them in detail here, only the names of the most important
|
||||
classes of security bugs are mentioned:
|
||||
|
||||
* Buffer overflows
|
||||
* Use-after-free and double-free
|
||||
* Integer overflows
|
||||
* Format string vulnerabilities
|
||||
|
||||
Some of these classes of bugs can be detected by analyzers. Static analysis is
|
||||
performed regularly by Coverity and the most obvious of these bugs are even
|
||||
reported by compilers. Dynamic analysis is possible with valgrind, tsan, and
|
||||
asan.
|
||||
|
||||
Input Validation
|
||||
----------------
|
||||
Inputs from the guest or external sources (e.g. network, files) cannot be
|
||||
trusted and may be invalid. Inputs must be checked before using them in a way
|
||||
that could crash the program, expose host memory to the guest, or otherwise be
|
||||
exploitable by an attacker.
|
||||
|
||||
The most sensitive attack surface is device emulation. All hardware register
|
||||
accesses and data read from guest memory must be validated. A typical example
|
||||
is a device that contains multiple units that are selectable by the guest via
|
||||
an index register::
|
||||
|
||||
typedef struct {
|
||||
ProcessingUnit unit[2];
|
||||
...
|
||||
} MyDeviceState;
|
||||
|
||||
static void mydev_writel(void *opaque, uint32_t addr, uint32_t val)
|
||||
{
|
||||
MyDeviceState *mydev = opaque;
|
||||
ProcessingUnit *unit;
|
||||
|
||||
switch (addr) {
|
||||
case MYDEV_SELECT_UNIT:
|
||||
unit = &mydev->unit[val]; <-- this input wasn't validated!
|
||||
...
|
||||
}
|
||||
}
|
||||
|
||||
If ``val`` is not in range [0, 1] then an out-of-bounds memory access will take
|
||||
place when ``unit`` is dereferenced. The code must check that ``val`` is 0 or
|
||||
1 and handle the case where it is invalid.
|
||||
|
||||
Unexpected Device Accesses
|
||||
--------------------------
|
||||
The guest may access device registers in unusual orders or at unexpected
|
||||
moments. Device emulation code must not assume that the guest follows the
|
||||
typical "theory of operation" presented in driver writer manuals. The guest
|
||||
may make nonsense accesses to device registers such as starting operations
|
||||
before the device has been fully initialized.
|
||||
|
||||
A related issue is that device emulation code must be prepared for unexpected
|
||||
device register accesses while asynchronous operations are in progress. A
|
||||
well-behaved guest might wait for a completion interrupt before accessing
|
||||
certain device registers. Device emulation code must handle the case where the
|
||||
guest overwrites registers or submits further requests before an ongoing
|
||||
request completes. Unexpected accesses must not cause memory corruption or
|
||||
leaks in QEMU.
|
||||
|
||||
Invalid device register accesses can be reported with
|
||||
``qemu_log_mask(LOG_GUEST_ERROR, ...)``. The ``-d guest_errors`` command-line
|
||||
option enables these log messages.
|
||||
|
||||
Live Migration
|
||||
--------------
|
||||
Device state can be saved to disk image files and shared with other users.
|
||||
Live migration code must validate inputs when loading device state so an
|
||||
attacker cannot gain control by crafting invalid device states. Device state
|
||||
is therefore considered untrusted even though it is typically generated by QEMU
|
||||
itself.
|
||||
|
||||
Guest Memory Access Races
|
||||
-------------------------
|
||||
Guests with multiple vCPUs may modify guest RAM while device emulation code is
|
||||
running. Device emulation code must copy in descriptors and other guest RAM
|
||||
structures and only process the local copy. This prevents
|
||||
time-of-check-to-time-of-use (TOCTOU) race conditions that could cause QEMU to
|
||||
crash when a vCPU thread modifies guest RAM while device emulation is
|
||||
processing it.
|
||||
|
||||
Use of null-co block drivers
|
||||
----------------------------
|
||||
|
||||
The ``null-co`` block driver is designed for performance: its read accesses are
|
||||
not initialized by default. In case this driver has to be used for security
|
||||
research, it must be used with the ``read-zeroes=on`` option which fills read
|
||||
buffers with zeroes. Security issues reported with the default
|
||||
(``read-zeroes=off``) will be discarded.
|
||||
@@ -0,0 +1,132 @@
|
||||
================================
|
||||
SSI devices and SPI flash models
|
||||
================================
|
||||
|
||||
QEMU's Synchronous Serial Interface (SSI) bus models the full-duplex transfer
|
||||
of words between a controller and one selected peripheral. Most SPI flash
|
||||
models, including ``m25p80``, are attached to controllers through this bus.
|
||||
|
||||
This page documents the expected boundary between a controller model and a
|
||||
flash model for SPI fast-read dummy cycles. The boundary is important because
|
||||
many real controllers expose dummy-cycle configuration in registers, while the
|
||||
flash model observes only the byte stream delivered through ``ssi_transfer()``.
|
||||
|
||||
SSI transfer granularity
|
||||
------------------------
|
||||
|
||||
``ssi_transfer()`` transfers one SSI word. Flash models that implement common
|
||||
SPI NOR command streams usually consume one 8-bit word at a time:
|
||||
|
||||
* command opcode;
|
||||
* address bytes;
|
||||
* optional mode or continuous-read bytes;
|
||||
* dummy bytes;
|
||||
* data bytes.
|
||||
|
||||
The SSI core does not model individual clock edges or the number of active SPI
|
||||
data lines. If a real transaction has a dummy phase expressed in clock cycles,
|
||||
the device model that generates transfers on the SSI bus must represent that
|
||||
phase as a number of dummy byte transfers.
|
||||
|
||||
Flash model responsibilities
|
||||
----------------------------
|
||||
|
||||
A SPI flash model owns the command semantics for the flash device:
|
||||
|
||||
* which opcodes are recognized;
|
||||
* how many address bytes are required;
|
||||
* whether a command has mode bytes;
|
||||
* how many dummy bytes must be consumed before data can be returned;
|
||||
* manufacturer-specific differences in fast-read command behavior.
|
||||
|
||||
For the ``m25p80`` model, ``needed_bytes`` is a byte count. It must not store
|
||||
raw dummy cycles. When a flash datasheet describes the dummy phase in cycles,
|
||||
the flash model converts the cycles to bytes using the bus width used for the
|
||||
dummy phase::
|
||||
|
||||
dummy_bytes = DIV_ROUND_UP(dummy_cycles * dummy_bus_width, 8)
|
||||
|
||||
For SPI NOR fast-read commands modeled by ``m25p80``, the dummy phase follows
|
||||
the address phase width. For example, output-only dual and quad read commands
|
||||
such as DOR and QOR use one line for command, address, and dummy phases, then
|
||||
use two or four lines only for the data phase. Dual I/O and Quad I/O commands
|
||||
such as DIOR and QIOR use the wider bus for both address and dummy phases.
|
||||
|
||||
If the exact dummy phase cannot be represented as a whole number of SSI byte
|
||||
transfers, the model should round up and log the limitation instead of silently
|
||||
treating cycles as bytes.
|
||||
|
||||
Controller model responsibilities
|
||||
---------------------------------
|
||||
|
||||
A controller model owns the behavior of the controller hardware:
|
||||
|
||||
* how guest-visible registers select command, address width, bus width, and
|
||||
dummy-cycle count;
|
||||
* whether the guest supplies dummy bytes in a transmit FIFO;
|
||||
* whether the controller itself generates the dummy phase for a memory-mapped,
|
||||
direct-read, or other automatic transfer mode;
|
||||
* how chip-select state changes around controller-generated transfers.
|
||||
|
||||
When guest software writes dummy bytes into a transmit FIFO or manual transfer
|
||||
path, the controller should pass those bytes to ``ssi_transfer()`` like any
|
||||
other guest-provided byte. It should not add more dummy transfers on behalf of
|
||||
the flash.
|
||||
|
||||
When hardware registers instruct the controller to generate a dummy phase, the
|
||||
controller must emit dummy byte transfers before data transfers reach the flash
|
||||
model. The controller should convert the configured cycle count using the bus
|
||||
width that the controller uses during the dummy phase. For example:
|
||||
|
||||
* 8 dummy cycles on a single data line become 1 dummy byte;
|
||||
* 8 dummy cycles on two data lines become 2 dummy bytes;
|
||||
* 8 dummy cycles on four data lines become 4 dummy bytes.
|
||||
|
||||
The controller should not duplicate flash-specific opcode tables merely to
|
||||
guess which commands need dummy cycles. In automatic modes the controller
|
||||
already has enough hardware configuration to know whether it must generate a
|
||||
dummy phase. In manual modes the guest-provided byte stream is authoritative.
|
||||
|
||||
Avoiding double counting
|
||||
------------------------
|
||||
|
||||
Exactly one side should generate each dummy byte transfer seen by the flash:
|
||||
|
||||
* If the guest sends dummy bytes through the controller, the controller forwards
|
||||
them and the flash consumes them.
|
||||
* If the guest programs a controller dummy-cycle register, the controller
|
||||
converts those cycles to dummy byte transfers and the flash consumes them.
|
||||
* The flash may know that a command requires dummy bytes, but it does not create
|
||||
transfers on the SSI bus.
|
||||
|
||||
Do not implement controller-side snooping that watches manual-mode opcode
|
||||
streams and injects extra dummy transfers based on flash opcodes. That mixes
|
||||
flash command semantics into the controller and is fragile when flash models
|
||||
gain correct dummy-byte accounting.
|
||||
|
||||
Examples in the tree
|
||||
--------------------
|
||||
|
||||
The following models illustrate the boundary:
|
||||
|
||||
* ``hw/block/m25p80.c`` keeps fast-read dummy requirements as byte counts in
|
||||
``needed_bytes``. Manufacturer-specific helpers convert datasheet dummy
|
||||
cycles to the byte stream expected by the model.
|
||||
* ``hw/ssi/aspeed_smc.c`` generates dummy byte transfers for direct fast-read
|
||||
mode from controller registers, but manual user-mode writes are forwarded as
|
||||
guest-provided bytes.
|
||||
* ``hw/ssi/npcm7xx_fiu.c`` converts the direct-read dummy configuration to the
|
||||
number of dummy byte transfers sent before reading data.
|
||||
|
||||
Review checklist
|
||||
----------------
|
||||
|
||||
When adding or changing a SPI flash controller or flash model, check:
|
||||
|
||||
* Are dummy counts stored in byte units when they drive flash state machines?
|
||||
* If a hardware register stores cycles, is the conversion to bytes based on the
|
||||
bus width of the dummy phase?
|
||||
* Are manual guest-provided dummy bytes forwarded without extra injection?
|
||||
* Are automatic controller-generated dummy phases modeled by the controller?
|
||||
* Is flash-specific opcode knowledge kept in the flash model rather than copied
|
||||
into controller snooping paths?
|
||||
@@ -0,0 +1,73 @@
|
||||
.. _stable-process:
|
||||
|
||||
QEMU and the stable process
|
||||
===========================
|
||||
|
||||
QEMU stable releases
|
||||
--------------------
|
||||
|
||||
QEMU stable releases are based upon the last released QEMU version
|
||||
and marked by an additional version number, e.g. 2.10.1. Occasionally,
|
||||
a four-number version is released, if a single urgent fix needs to go
|
||||
on top.
|
||||
|
||||
Usually, stable releases are only provided for the last major QEMU
|
||||
release. For example, when QEMU 2.11.0 is released, 2.11.x or 2.11.x.y
|
||||
stable releases are produced only until QEMU 2.12.0 is released, at
|
||||
which point the stable process moves to producing 2.12.x/2.12.x.y releases.
|
||||
|
||||
What should go into a stable release?
|
||||
-------------------------------------
|
||||
|
||||
Generally, the following patches are considered stable material:
|
||||
|
||||
* Patches that fix severe issues, like fixes for CVEs
|
||||
|
||||
* Patches that fix regressions
|
||||
|
||||
If you think the patch would be important for users of the current release
|
||||
(or for a distribution picking fixes), it is usually a good candidate
|
||||
for stable.
|
||||
|
||||
|
||||
How to get a patch into QEMU stable
|
||||
-----------------------------------
|
||||
|
||||
There are various ways to get a patch into stable:
|
||||
|
||||
* Preferred: Make sure that the stable maintainers are on copy when you send
|
||||
the patch by adding
|
||||
|
||||
.. code::
|
||||
|
||||
Cc: [email protected]
|
||||
|
||||
to the patch description. By default, this will send a copy of the patch
|
||||
to ``[email protected]`` if you use git send-email, which is where
|
||||
patches that are stable candidates are tracked by the maintainers.
|
||||
|
||||
* You can also reply to a patch and put ``[email protected]`` on copy
|
||||
directly in your mail client if you think a previously submitted patch
|
||||
should be considered for a stable release.
|
||||
|
||||
* If a maintainer judges the patch appropriate for stable later on (or you
|
||||
notify them), they will add the same line to the patch, meaning that
|
||||
the stable maintainers will be on copy on the maintainer's pull request.
|
||||
|
||||
* If you judge an already merged patch suitable for stable, send a mail
|
||||
(preferably as a reply to the most recent patch submission) to
|
||||
``[email protected]`` along with ``[email protected]`` and
|
||||
appropriate other people (like the patch author or the relevant maintainer)
|
||||
on copy.
|
||||
|
||||
Stable release process
|
||||
----------------------
|
||||
|
||||
When the stable maintainers prepare a new stable release, they will prepare
|
||||
a git branch with a release candidate and send the patches out to
|
||||
``[email protected]`` for review. If any of your patches are included,
|
||||
please verify that they look fine, especially if the maintainer had to tweak
|
||||
the patch as part of back-porting things across branches. You may also
|
||||
nominate other patches that you think are suitable for inclusion. After
|
||||
review is complete (may involve more release candidates), a new stable release
|
||||
is made available.
|
||||
@@ -0,0 +1,854 @@
|
||||
.. _coding-style:
|
||||
|
||||
=================
|
||||
QEMU Coding Style
|
||||
=================
|
||||
|
||||
.. contents:: Table of Contents
|
||||
|
||||
Please use the script checkpatch.pl in the scripts directory to check
|
||||
patches before submitting.
|
||||
|
||||
Formatting and style
|
||||
********************
|
||||
|
||||
The repository includes a ``.editorconfig`` file which can help with
|
||||
getting the right settings for your preferred $EDITOR. See
|
||||
`<https://editorconfig.org/>`_ for details.
|
||||
|
||||
Whitespace
|
||||
==========
|
||||
|
||||
Of course, the most important aspect in any coding style is whitespace.
|
||||
Crusty old coders who have trouble spotting the glasses on their noses
|
||||
can tell the difference between a tab and eight spaces from a distance
|
||||
of approximately fifteen parsecs. Many a flamewar has been fought and
|
||||
lost on this issue.
|
||||
|
||||
QEMU indents are four spaces. Tabs are never used, except in Makefiles
|
||||
where they have been irreversibly coded into the syntax.
|
||||
Spaces of course are superior to tabs because:
|
||||
|
||||
* You have just one way to specify whitespace, not two. Ambiguity breeds
|
||||
mistakes.
|
||||
* The confusion surrounding 'use tabs to indent, spaces to justify' is gone.
|
||||
* Tab indents push your code to the right, making your screen seriously
|
||||
unbalanced.
|
||||
* Tabs will be rendered incorrectly on editors who are misconfigured not
|
||||
to use tab stops of eight positions.
|
||||
* Tabs are rendered badly in patches, causing off-by-one errors in almost
|
||||
every line.
|
||||
* It is the QEMU coding style.
|
||||
|
||||
Do not leave whitespace dangling off the ends of lines.
|
||||
|
||||
Multiline Indent
|
||||
----------------
|
||||
|
||||
There are several places where indent is necessary:
|
||||
|
||||
* if/else
|
||||
* while/for
|
||||
* function definition & call
|
||||
|
||||
When breaking up a long line to fit within line width, we need a proper indent
|
||||
for the following lines.
|
||||
|
||||
In case of if/else, while/for, align the secondary lines just after the
|
||||
opening parenthesis of the first.
|
||||
|
||||
For example:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
if (a == 1 &&
|
||||
b == 2) {
|
||||
|
||||
while (a == 1 &&
|
||||
b == 2) {
|
||||
|
||||
In case of function, there are several variants:
|
||||
|
||||
* 4 spaces indent from the beginning
|
||||
* align the secondary lines just after the opening parenthesis of the first
|
||||
|
||||
For example:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
do_something(x, y,
|
||||
z);
|
||||
|
||||
do_something(x, y,
|
||||
z);
|
||||
|
||||
do_something(x, do_another(y,
|
||||
z));
|
||||
|
||||
Line width
|
||||
==========
|
||||
|
||||
Lines should be 80 characters; try not to make them longer.
|
||||
|
||||
Sometimes it is hard to do, especially when dealing with QEMU subsystems
|
||||
that use long function or symbol names. If wrapping the line at 80 columns
|
||||
is obviously less readable and more awkward, prefer not to wrap it; better
|
||||
to have an 85 character line than one which is awkwardly wrapped.
|
||||
|
||||
Even in that case, try not to make lines much longer than 80 characters.
|
||||
(The checkpatch script will warn at 100 characters, but this is intended
|
||||
as a guard against obviously-overlength lines, not a target.)
|
||||
|
||||
Rationale:
|
||||
|
||||
* Some people like to tile their 24" screens with a 6x4 matrix of 80x24
|
||||
xterms and use vi in all of them. The best way to punish them is to
|
||||
let them keep doing it.
|
||||
* Code and especially patches is much more readable if limited to a sane
|
||||
line length. Eighty is traditional.
|
||||
* The four-space indentation makes the most common excuse ("But look
|
||||
at all that white space on the left!") moot.
|
||||
* It is the QEMU coding style.
|
||||
|
||||
Naming
|
||||
======
|
||||
|
||||
Variables are lower_case_with_underscores; easy to type and read. Structured
|
||||
type names are in CamelCase; harder to type but standing out. Enum type
|
||||
names and function type names should also be in CamelCase. Scalar type
|
||||
names are lower_case_with_underscores_ending_with_a_t, like the POSIX
|
||||
uint64_t and family. Note that this last convention contradicts POSIX
|
||||
and is therefore likely to be changed.
|
||||
|
||||
Variable Naming Conventions
|
||||
---------------------------
|
||||
|
||||
A number of short naming conventions exist for variables that use
|
||||
common QEMU types. For example, the architecture independent CPUState
|
||||
is often held as a ``cs`` pointer variable, whereas the concrete
|
||||
CPUArchState is usually held in a pointer called ``env``.
|
||||
|
||||
Likewise, in device emulation code the common DeviceState is usually
|
||||
called ``dev``.
|
||||
|
||||
Function Naming Conventions
|
||||
---------------------------
|
||||
|
||||
Wrapped version of standard library or GLib functions use a ``qemu_``
|
||||
prefix to alert readers that they are seeing a wrapped version, for
|
||||
example ``qemu_strtol`` or ``qemu_mutex_lock``. Other utility functions
|
||||
that are widely called from across the codebase should not have any
|
||||
prefix, for example ``pstrcpy`` or bit manipulation functions such as
|
||||
``find_first_bit``.
|
||||
|
||||
The ``qemu_`` prefix is also used for functions that modify global
|
||||
emulator state, for example ``qemu_add_vm_change_state_handler``.
|
||||
However, if there is an obvious subsystem-specific prefix it should be
|
||||
used instead.
|
||||
|
||||
Public functions from a file or subsystem (declared in headers) tend
|
||||
to have a consistent prefix to show where they came from. For example,
|
||||
``tlb_`` for functions from ``cputlb.c`` or ``cpu_`` for functions
|
||||
from cpus.c.
|
||||
|
||||
If there are two versions of a function to be called with or without a
|
||||
lock held, the function that expects the lock to be already held
|
||||
usually uses the suffix ``_locked``.
|
||||
|
||||
If a function is a shim designed to deal with compatibility
|
||||
workarounds we use the suffix ``_compat``. These are generally not
|
||||
called directly and aliased to the plain function name via the
|
||||
pre-processor. Another common suffix is ``_impl``; it is used for the
|
||||
concrete implementation of a function that will not be called
|
||||
directly, but rather through a macro or an inline function.
|
||||
|
||||
Block structure
|
||||
===============
|
||||
|
||||
Every indented statement is braced; even if the block contains just one
|
||||
statement. The opening brace is on the line that contains the control
|
||||
flow statement that introduces the new block; the closing brace is on the
|
||||
same line as the else keyword, or on a line by itself if there is no else
|
||||
keyword. Example:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
if (a == 5) {
|
||||
printf("a was 5.\n");
|
||||
} else if (a == 6) {
|
||||
printf("a was 6.\n");
|
||||
} else {
|
||||
printf("a was something else entirely.\n");
|
||||
}
|
||||
|
||||
Note that 'else if' is considered a single statement; otherwise a long if/
|
||||
else if/else if/.../else sequence would need an indent for every else
|
||||
statement.
|
||||
|
||||
An exception is the opening brace for a function; for reasons of tradition
|
||||
and clarity it comes on a line by itself:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
void a_function(void)
|
||||
{
|
||||
do_something();
|
||||
}
|
||||
|
||||
Rationale: a consistent (except for functions...) bracing style reduces
|
||||
ambiguity and avoids needless churn when lines are added or removed.
|
||||
Furthermore, it is the QEMU coding style.
|
||||
|
||||
Declarations
|
||||
============
|
||||
|
||||
Mixed declarations (interleaving statements and declarations within
|
||||
blocks) are generally not allowed; declarations should be at the beginning
|
||||
of blocks. To avoid accidental re-use it is permissible to declare
|
||||
loop variables inside for loops:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
for (int i = 0; i < ARRAY_SIZE(thing); i++) {
|
||||
/* do something loopy */
|
||||
}
|
||||
|
||||
Every now and then, an exception is made for declarations inside a
|
||||
#ifdef or #ifndef block: if the code looks nicer, such declarations can
|
||||
be placed at the top of the block even if there are statements above.
|
||||
On the other hand, however, it's often best to move that #ifdef/#ifndef
|
||||
block to a separate function altogether.
|
||||
|
||||
Conditional statements
|
||||
======================
|
||||
|
||||
When comparing a variable for (in)equality with a constant, list the
|
||||
constant on the right, as in:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
if (a == 1) {
|
||||
/* Reads like: "If a equals 1" */
|
||||
do_something();
|
||||
}
|
||||
|
||||
Rationale: Yoda conditions (as in 'if (1 == a)') are awkward to read.
|
||||
Besides, good compilers already warn users when '==' is mis-typed as '=',
|
||||
even when the constant is on the right.
|
||||
|
||||
Comment style
|
||||
=============
|
||||
|
||||
We use traditional C-style /``*`` ``*``/ comments and avoid // comments.
|
||||
|
||||
Rationale: The // form is valid in C99, so this is purely a matter of
|
||||
consistency of style. The checkpatch script will warn you about this.
|
||||
|
||||
Multiline comment blocks should have a row of stars on the left,
|
||||
and the initial /``*`` and terminating ``*``/ both on their own lines:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
/*
|
||||
* like
|
||||
* this
|
||||
*/
|
||||
|
||||
This is the same format required by the Linux kernel coding style.
|
||||
|
||||
(Some of the existing comments in the codebase use the GNU Coding
|
||||
Standards form which does not have stars on the left, or other
|
||||
variations; avoid these when writing new comments, but don't worry
|
||||
about converting to the preferred form unless you're editing that
|
||||
comment anyway.)
|
||||
|
||||
Rationale: Consistency, and ease of visually picking out a multiline
|
||||
comment from the surrounding code.
|
||||
|
||||
Language usage
|
||||
**************
|
||||
|
||||
Preprocessor
|
||||
============
|
||||
|
||||
Variadic macros
|
||||
---------------
|
||||
|
||||
For variadic macros, stick with this C99-like syntax:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
#define DPRINTF(fmt, ...) \
|
||||
do { printf("IRQ: " fmt, ## __VA_ARGS__); } while (0)
|
||||
|
||||
Include directives
|
||||
------------------
|
||||
|
||||
Order include directives as follows:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
#include "qemu/osdep.h" /* Always first... */
|
||||
#include <...> /* then system headers... */
|
||||
#include "..." /* and finally QEMU headers. */
|
||||
|
||||
The "qemu/osdep.h" header contains preprocessor macros that affect the behavior
|
||||
of core system headers like <stdint.h>. It must be the first include so that
|
||||
core system headers included by external libraries get the preprocessor macros
|
||||
that QEMU depends on.
|
||||
|
||||
Do not include "qemu/osdep.h" from header files since the .c file will have
|
||||
already included it.
|
||||
|
||||
Headers should normally include everything they need beyond osdep.h.
|
||||
If exceptions are needed for some reason, they must be documented in
|
||||
the header. If all that's needed from a header is typedefs, consider
|
||||
putting those into qemu/typedefs.h instead of including the header.
|
||||
|
||||
Cyclic inclusion is forbidden.
|
||||
|
||||
Generative Includes
|
||||
-------------------
|
||||
|
||||
QEMU makes fairly extensive use of the macro pre-processor to
|
||||
instantiate multiple similar functions. While such abuse of the macro
|
||||
processor isn't discouraged it can make debugging and code navigation
|
||||
harder. You should consider carefully if the same effect can be
|
||||
achieved by making it easy for the compiler to constant fold or using
|
||||
python scripting to generate grep friendly code.
|
||||
|
||||
If you do use template header files they should be named with the
|
||||
``.c.inc`` or ``.h.inc`` suffix to make it clear they are being
|
||||
included for expansion.
|
||||
|
||||
C types
|
||||
=======
|
||||
|
||||
It should be common sense to use the right type, but we have collected
|
||||
a few useful guidelines here.
|
||||
|
||||
Scalars
|
||||
-------
|
||||
|
||||
If you're using "int" or "long", odds are good that there's a better type.
|
||||
If a variable is counting something, it should be declared with an
|
||||
unsigned type.
|
||||
|
||||
If it's host memory-size related, size_t should be a good choice (use
|
||||
ssize_t only if required). Guest RAM memory offsets must use ram_addr_t,
|
||||
but only for RAM, it may not cover whole guest address space.
|
||||
|
||||
If it's file-size related, use off_t.
|
||||
If it's file-offset related (i.e., signed), use off_t.
|
||||
If it's just counting small numbers use "unsigned int";
|
||||
(on all but oddball embedded systems, you can assume that that
|
||||
type is at least four bytes wide).
|
||||
|
||||
In the event that you require a specific width, use a standard type
|
||||
like int32_t, uint32_t, uint64_t, etc. The specific types are
|
||||
mandatory for VMState fields.
|
||||
|
||||
Don't use Linux kernel internal types like u32, __u32 or __le32.
|
||||
|
||||
Use hwaddr for guest physical addresses except pcibus_t
|
||||
for PCI addresses. In addition, ram_addr_t is a QEMU internal address
|
||||
space that maps guest RAM physical addresses into an intermediate
|
||||
address space that can map to host virtual address spaces. Generally
|
||||
speaking, the size of guest memory can always fit into ram_addr_t but
|
||||
it would not be correct to store an actual guest physical address in a
|
||||
ram_addr_t.
|
||||
|
||||
For CPU virtual addresses there are several possible types.
|
||||
vaddr is the best type to use to hold a CPU virtual address in
|
||||
target-independent code. It is guaranteed to be large enough to hold a
|
||||
virtual address for any target, and it does not change size from target
|
||||
to target. It is always unsigned.
|
||||
target_ulong is a type the size of a virtual address on the CPU; this means
|
||||
it may be 32 or 64 bits depending on which target is being built. It should
|
||||
therefore be used only in target-specific code, and in some
|
||||
performance-critical built-per-target core code such as the TLB code.
|
||||
There is also a signed version, target_long.
|
||||
abi_ulong is for the ``*``-user targets, and represents a type the size of
|
||||
'void ``*``' in that target's ABI. (This may not be the same as the size of a
|
||||
full CPU virtual address in the case of target ABIs which use 32 bit pointers
|
||||
on 64 bit CPUs, like sparc32plus.) Definitions of structures that must match
|
||||
the target's ABI must use this type for anything that on the target is defined
|
||||
to be an 'unsigned long' or a pointer type.
|
||||
There is also a signed version, abi_long.
|
||||
|
||||
Of course, take all of the above with a grain of salt. If you're about
|
||||
to use some system interface that requires a type like size_t, pid_t or
|
||||
off_t, use matching types for any corresponding variables.
|
||||
|
||||
Also, if you try to use e.g., "unsigned int" as a type, and that
|
||||
conflicts with the signedness of a related variable, sometimes
|
||||
it's best just to use the *wrong* type, if "pulling the thread"
|
||||
and fixing all related variables would be too invasive.
|
||||
|
||||
Finally, while using descriptive types is important, be careful not to
|
||||
go overboard. If whatever you're doing causes warnings, or requires
|
||||
casts, then reconsider or ask for help.
|
||||
|
||||
Pointers
|
||||
--------
|
||||
|
||||
Ensure that all of your pointers are "const-correct".
|
||||
Unless a pointer is used to modify the pointed-to storage,
|
||||
give it the "const" attribute. That way, the reader knows
|
||||
up-front that this is a read-only pointer. Perhaps more
|
||||
importantly, if we're diligent about this, when you see a non-const
|
||||
pointer, you're guaranteed that it is used to modify the storage
|
||||
it points to, or it is aliased to another pointer that is.
|
||||
|
||||
Typedefs
|
||||
--------
|
||||
|
||||
Typedefs are used to eliminate the redundant 'struct' keyword, since type
|
||||
names have a different style than other identifiers ("CamelCase" versus
|
||||
"snake_case"). Each named struct type should have a CamelCase name and a
|
||||
corresponding typedef.
|
||||
|
||||
Since certain C compilers choke on duplicated typedefs, you should avoid
|
||||
them and declare a typedef only in one header file. For common types,
|
||||
you can use "include/qemu/typedefs.h" for example. However, as a matter
|
||||
of convenience it is also perfectly fine to use forward struct
|
||||
definitions instead of typedefs in headers and function prototypes; this
|
||||
avoids problems with duplicated typedefs and reduces the need to include
|
||||
headers from other headers.
|
||||
|
||||
Bitfields
|
||||
---------
|
||||
|
||||
C bitfields can be a cause of non-portability issues, especially under windows
|
||||
where `MSVC has a different way to lay them out than GCC
|
||||
<https://gcc.gnu.org/onlinedocs/gcc/x86-Type-Attributes.html>`_, or where
|
||||
endianness matters.
|
||||
|
||||
For this reason, we disallow usage of bitfields in packed structures and in any
|
||||
structures which are supposed to exactly match a specific layout in guest
|
||||
memory. Some existing code may use it, and we carefully ensured the layout was
|
||||
the one expected.
|
||||
|
||||
We also suggest avoiding bitfields even in structures where the exact
|
||||
layout does not matter, unless you can show that they provide a significant
|
||||
usability benefit.
|
||||
|
||||
We encourage the usage of ``include/hw/core/registerfields.h`` as a safe replacement
|
||||
for bitfields.
|
||||
|
||||
Reserved namespaces in C and POSIX
|
||||
----------------------------------
|
||||
|
||||
Underscore capital, double underscore, and underscore 't' suffixes should be
|
||||
avoided.
|
||||
|
||||
Low level memory management
|
||||
===========================
|
||||
|
||||
Use of the ``malloc/free/realloc/calloc/valloc/memalign/posix_memalign``
|
||||
or ``alloca/g_alloca/g_newa/g_newa0`` APIs is not allowed in the QEMU codebase.
|
||||
Instead of these routines, use the GLib memory allocation routines
|
||||
``g_malloc/g_malloc0/g_new/g_new0/g_realloc/g_free``
|
||||
or QEMU's ``qemu_memalign/qemu_blockalign/qemu_vfree`` APIs.
|
||||
|
||||
Please note that ``g_malloc`` will exit on allocation failure, so
|
||||
there is no need to test for failure (as you would have to with
|
||||
``malloc``). Generally using ``g_malloc`` on start-up is fine as the
|
||||
result of a failure to allocate memory is going to be a fatal exit
|
||||
anyway. There may be some start-up cases where failing is unreasonable
|
||||
(for example speculatively loading a large debug symbol table).
|
||||
|
||||
Care should be taken to avoid introducing places where the guest could
|
||||
trigger an exit by causing a large allocation. For small allocations,
|
||||
of the order of 4k, a failure to allocate is likely indicative of an
|
||||
overloaded host and allowing ``g_malloc`` to ``exit`` is a reasonable
|
||||
approach. However for larger allocations where we could realistically
|
||||
fall-back to a smaller one if need be we should use functions like
|
||||
``g_try_new`` and check the result. For example this is valid approach
|
||||
for a time/space trade-off like ``tlb_mmu_resize_locked`` in the
|
||||
SoftMMU TLB code.
|
||||
|
||||
If the lifetime of the allocation is within the function and there are
|
||||
multiple exit paths you can also improve the readability of the code
|
||||
by using ``g_autofree`` and related annotations. See :ref:`autofree-ref`
|
||||
for more details.
|
||||
|
||||
Calling ``g_malloc`` with a zero size is valid and will return NULL.
|
||||
|
||||
Prefer ``g_new(T, n)`` instead of ``g_malloc(sizeof(T) * n)`` for the following
|
||||
reasons:
|
||||
|
||||
* It catches multiplication overflowing size_t;
|
||||
* It returns T ``*`` instead of void ``*``, letting compiler catch more type errors.
|
||||
|
||||
Declarations like
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
T *v = g_malloc(sizeof(*v))
|
||||
|
||||
are acceptable, though.
|
||||
|
||||
Memory allocated by ``qemu_memalign`` or ``qemu_blockalign`` must be freed with
|
||||
``qemu_vfree``, since breaking this will cause problems on Win32.
|
||||
|
||||
String manipulation
|
||||
===================
|
||||
|
||||
Do not use the strncpy function. As mentioned in the man page, it does *not*
|
||||
guarantee a NULL-terminated buffer, which makes it extremely dangerous to use.
|
||||
It also zeros trailing destination bytes out to the specified length. Instead,
|
||||
use this similar function when possible, but note its different signature:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
void pstrcpy(char *dest, int dest_buf_size, const char *src)
|
||||
|
||||
Don't use strcat because it can't check for buffer overflows, but:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
char *pstrcat(char *buf, int buf_size, const char *s)
|
||||
|
||||
The same limitation exists with sprintf and vsprintf, so use snprintf and
|
||||
vsnprintf.
|
||||
|
||||
QEMU provides other useful string functions:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
int strstart(const char *str, const char *val, const char **ptr)
|
||||
int stristart(const char *str, const char *val, const char **ptr)
|
||||
|
||||
There are also replacement character processing macros for isxyz and toxyz,
|
||||
so instead of e.g. isalnum you should use qemu_isalnum.
|
||||
|
||||
Because of the memory management rules, you must use g_strdup/g_strndup
|
||||
instead of plain strdup/strndup.
|
||||
|
||||
Printf-style functions
|
||||
======================
|
||||
|
||||
Whenever you add a new printf-style function, i.e., one with a format
|
||||
string argument and following "..." in its prototype, be sure to use
|
||||
gcc's printf attribute directive in the prototype.
|
||||
|
||||
This makes it so gcc's -Wformat and -Wformat-security options can do
|
||||
their jobs and cross-check format strings with the number and types
|
||||
of arguments.
|
||||
|
||||
C standard, implementation defined and undefined behaviors
|
||||
==========================================================
|
||||
|
||||
C code in QEMU should be written to the C11 language specification. A
|
||||
copy of the final version of the C11 standard formatted as a draft,
|
||||
can be downloaded from:
|
||||
|
||||
`<http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1548.pdf>`_
|
||||
|
||||
The C language specification defines regions of undefined behavior and
|
||||
implementation defined behavior (to give compiler authors enough leeway to
|
||||
produce better code). In general, code in QEMU should follow the language
|
||||
specification and avoid both undefined and implementation defined
|
||||
constructs. ("It works fine on the gcc I tested it with" is not a valid
|
||||
argument...) However there are a few areas where we allow ourselves to
|
||||
assume certain behaviors because in practice all the platforms we care about
|
||||
behave in the same way and writing strictly conformant code would be
|
||||
painful. These are:
|
||||
|
||||
* you may assume that integers are 2s complement representation
|
||||
* you may assume that right shift of a signed integer duplicates
|
||||
the sign bit (ie it is an arithmetic shift, not a logical shift)
|
||||
|
||||
In addition, QEMU assumes that the compiler does not use the latitude
|
||||
given in C99 and C11 to treat aspects of signed '<<' as undefined, as
|
||||
documented in the GNU Compiler Collection manual starting at version 4.0.
|
||||
|
||||
.. _autofree-ref:
|
||||
|
||||
Automatic memory deallocation
|
||||
=============================
|
||||
|
||||
QEMU has a mandatory dependency on either the GCC or the Clang compiler. As
|
||||
such it has the freedom to make use of a C language extension for
|
||||
automatically running a cleanup function when a stack variable goes
|
||||
out of scope. This can be used to simplify function cleanup paths,
|
||||
often allowing many goto jumps to be eliminated, through automatic
|
||||
free'ing of memory.
|
||||
|
||||
The GLib2 library provides a number of functions/macros for enabling
|
||||
automatic cleanup:
|
||||
|
||||
`<https://developer.gnome.org/glib/stable/glib-Miscellaneous-Macros.html>`_
|
||||
|
||||
Most notably:
|
||||
|
||||
* g_autofree - will invoke g_free() on the variable going out of scope
|
||||
|
||||
* g_autoptr - for structs / objects, will invoke the cleanup func created
|
||||
by a previous use of G_DEFINE_AUTOPTR_CLEANUP_FUNC. This is
|
||||
supported for most GLib data types and GObjects
|
||||
|
||||
For example, instead of
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
int somefunc(void)
|
||||
{
|
||||
int ret = -1;
|
||||
char *foo = g_strdup_printf("foo%", "wibble");
|
||||
GList *bar = .....
|
||||
|
||||
if (eek) {
|
||||
goto cleanup;
|
||||
}
|
||||
|
||||
ret = 0;
|
||||
|
||||
cleanup:
|
||||
g_free(foo);
|
||||
g_list_free(bar);
|
||||
return ret;
|
||||
}
|
||||
|
||||
Using g_autofree/g_autoptr enables the code to be written as:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
int somefunc(void)
|
||||
{
|
||||
g_autofree char *foo = g_strdup_printf("foo%", "wibble");
|
||||
g_autoptr (GList) bar = .....
|
||||
|
||||
if (eek) {
|
||||
return -1;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
While this generally results in simpler, less leak-prone code, there
|
||||
are still some caveats to beware of
|
||||
|
||||
* Variables declared with g_auto* MUST always be initialized,
|
||||
otherwise the cleanup function will use uninitialized stack memory
|
||||
|
||||
* If a variable declared with g_auto* holds a value which must
|
||||
live beyond the life of the function, that value must be saved
|
||||
and the original variable NULL'd out. This can be simpler using
|
||||
g_steal_pointer
|
||||
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
char *somefunc(void)
|
||||
{
|
||||
g_autofree char *foo = g_strdup_printf("foo%", "wibble");
|
||||
g_autoptr (GList) bar = .....
|
||||
|
||||
if (eek) {
|
||||
return NULL;
|
||||
}
|
||||
|
||||
return g_steal_pointer(&foo);
|
||||
}
|
||||
|
||||
|
||||
QEMU Specific Idioms
|
||||
********************
|
||||
|
||||
QEMU Object Model Declarations
|
||||
==============================
|
||||
|
||||
The QEMU Object Model (QOM) provides a framework for handling objects
|
||||
in the base C language. The first declaration of a storage or class
|
||||
structure should always be the parent and leave a visual space between
|
||||
that declaration and the new code. It is also useful to separate
|
||||
backing for properties (options driven by the user) and internal state
|
||||
to make navigation easier.
|
||||
|
||||
For a storage structure the first declaration should always be called
|
||||
"parent_obj" and for a class structure the first member should always
|
||||
be called "parent_class" as below:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
struct MyDeviceState {
|
||||
DeviceState parent_obj;
|
||||
|
||||
/* Properties */
|
||||
int prop_a;
|
||||
char *prop_b;
|
||||
/* Other stuff */
|
||||
int internal_state;
|
||||
};
|
||||
|
||||
struct MyDeviceClass {
|
||||
DeviceClass parent_class;
|
||||
|
||||
void (*new_fn1)(void);
|
||||
bool (*new_fn2)(CPUState *);
|
||||
};
|
||||
|
||||
Note that there is no need to provide typedefs for QOM structures
|
||||
since these are generated automatically by the QOM declaration macros.
|
||||
See :ref:`qom` for more details.
|
||||
|
||||
QEMU GUARD macros
|
||||
=================
|
||||
|
||||
QEMU provides a number of ``_GUARD`` macros intended to make the
|
||||
handling of multiple exit paths easier. For example using
|
||||
``QEMU_LOCK_GUARD`` to take a lock will ensure the lock is released on
|
||||
exit from the function.
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
static int my_critical_function(SomeState *s, void *data)
|
||||
{
|
||||
QEMU_LOCK_GUARD(&s->lock);
|
||||
do_thing1(data);
|
||||
if (check_state2(data)) {
|
||||
return -1;
|
||||
}
|
||||
do_thing3(data);
|
||||
return 0;
|
||||
}
|
||||
|
||||
will ensure s->lock is released however the function is exited. The
|
||||
equivalent code without _GUARD macro makes us to carefully put
|
||||
qemu_mutex_unlock() on all exit points:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
static int my_critical_function(SomeState *s, void *data)
|
||||
{
|
||||
qemu_mutex_lock(&s->lock);
|
||||
do_thing1(data);
|
||||
if (check_state2(data)) {
|
||||
qemu_mutex_unlock(&s->lock);
|
||||
return -1;
|
||||
}
|
||||
do_thing3(data);
|
||||
qemu_mutex_unlock(&s->lock);
|
||||
return 0;
|
||||
}
|
||||
|
||||
There are often ``WITH_`` forms of macros which more easily wrap
|
||||
around a block inside a function.
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
WITH_RCU_READ_LOCK_GUARD() {
|
||||
QTAILQ_FOREACH_RCU(kid, &bus->children, sibling) {
|
||||
err = do_the_thing(kid->child);
|
||||
if (err < 0) {
|
||||
return err;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Error handling and reporting
|
||||
============================
|
||||
|
||||
Reporting errors to the human user
|
||||
----------------------------------
|
||||
|
||||
Do not use printf(), fprintf() or monitor_printf(). Instead, use
|
||||
error_report() or error_vreport() from error-report.h. This ensures the
|
||||
error is reported in the right place (current monitor or stderr), and in
|
||||
a uniform format.
|
||||
|
||||
Use error_printf() & friends to print additional information.
|
||||
|
||||
error_report() prints the current location. In certain common cases
|
||||
like command line parsing, the current location is tracked
|
||||
automatically. To manipulate it manually, use the loc_``*``() from
|
||||
error-report.h.
|
||||
|
||||
Propagating errors
|
||||
------------------
|
||||
|
||||
An error can't always be reported to the user right where it's detected,
|
||||
but often needs to be propagated up the call chain to a place that can
|
||||
handle it. This can be done in various ways.
|
||||
|
||||
The most flexible one is Error objects. See error.h for usage
|
||||
information.
|
||||
|
||||
Use the simplest suitable method to communicate success / failure to
|
||||
callers. Stick to common methods: non-negative on success / -1 on
|
||||
error, non-negative / -errno, non-null / null, or Error objects.
|
||||
|
||||
Example: when a function returns a non-null pointer on success, and it
|
||||
can fail only in one way (as far as the caller is concerned), returning
|
||||
null on failure is just fine, and certainly simpler and a lot easier on
|
||||
the eyes than propagating an Error object through an ``Error **`` parameter.
|
||||
|
||||
Example: when a function's callers need to report details on failure
|
||||
only the function really knows, use ``Error **``, and set suitable errors.
|
||||
|
||||
Do not report an error to the user when you're also returning an error
|
||||
for somebody else to handle. Leave the reporting to the place that
|
||||
consumes the error returned.
|
||||
|
||||
Handling errors
|
||||
---------------
|
||||
|
||||
Calling exit() is fine when handling configuration errors during
|
||||
startup. It's problematic during normal operation. In particular,
|
||||
monitor commands should never exit().
|
||||
|
||||
Do not call exit() or abort() to handle an error that can be triggered
|
||||
by the guest (e.g., some unimplemented corner case in guest code
|
||||
translation or device emulation). Guests should not be able to
|
||||
terminate QEMU.
|
||||
|
||||
Note that &error_fatal is just another way to exit(1), and &error_abort
|
||||
is just another way to abort().
|
||||
|
||||
|
||||
trace-events style
|
||||
==================
|
||||
|
||||
0x prefix
|
||||
---------
|
||||
|
||||
In trace-events files, use a '0x' prefix to specify hex numbers, as in:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
some_trace(unsigned x, uint64_t y) "x 0x%x y 0x" PRIx64
|
||||
|
||||
An exception is made for groups of numbers that are hexadecimal by
|
||||
convention and separated by the symbols '.', '/', ':', or ' ' (such as
|
||||
PCI bus id):
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
another_trace(int cssid, int ssid, int dev_num) "bus id: %x.%x.%04x"
|
||||
|
||||
However, you can use '0x' for such groups if you want. Anyway, be sure that
|
||||
it is obvious that numbers are in hex, ex.:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
data_dump(uint8_t c1, uint8_t c2, uint8_t c3) "bytes (in hex): %02x %02x %02x"
|
||||
|
||||
Rationale: hex numbers are hard to read in logs when there is no 0x prefix,
|
||||
especially when (occasionally) the representation doesn't contain any letters
|
||||
and especially in one line with other decimal numbers. Number groups are allowed
|
||||
to not use '0x' because for some things notations like %x.%x.%x are used not
|
||||
only in QEMU. Also dumping raw data bytes with '0x' is less readable.
|
||||
|
||||
'#' printf flag
|
||||
---------------
|
||||
|
||||
Do not use printf flag '#', like '%#x'.
|
||||
|
||||
Rationale: there are two ways to add a '0x' prefix to printed number: '0x%...'
|
||||
and '%#...'. For consistency the only one way should be used. Arguments for
|
||||
'0x%' are:
|
||||
|
||||
* it is more popular
|
||||
* '%#' omits the 0x for the value 0 which makes output inconsistent
|
||||
@@ -0,0 +1,651 @@
|
||||
.. _submitting-a-patch:
|
||||
|
||||
Submitting a Patch
|
||||
==================
|
||||
|
||||
QEMU welcomes contributions to fix bugs, add functionality or improve
|
||||
the documentation. However, we get a lot of patches, and so we have
|
||||
some guidelines about submitting them. If you follow these, you'll
|
||||
help make our task of contribution review easier and your change is
|
||||
likely to be accepted and committed faster.
|
||||
|
||||
This page seems very long, so if you are only trying to post a quick
|
||||
one-shot fix, the bare minimum we ask is that:
|
||||
|
||||
.. list-table:: Minimal Checklist for Patches
|
||||
:widths: 35 65
|
||||
:header-rows: 1
|
||||
|
||||
* - Check
|
||||
- Reason
|
||||
* - Patches contain ``Signed-off-by: Your Name <author@email>``
|
||||
- States you are legally able to contribute the code. See :ref:`patch_emails_must_include_a_signed_off_by_line`
|
||||
* - Sent as patch emails to ``[email protected]``
|
||||
- The project uses an email list based workflow. See :ref:`submitting_your_patches`
|
||||
* - Be prepared to respond to review comments
|
||||
- Code that doesn't pass review will not get merged. See :ref:`participating_in_code_review`
|
||||
|
||||
You do not have to subscribe to post (list policy is to reply-to-all to
|
||||
preserve CCs and keep non-subscribers in the loop on the threads they
|
||||
start), although you may find it easier as a subscriber to pick up good
|
||||
ideas from other posts. If you do subscribe, be prepared for a high
|
||||
volume of email, often over one thousand messages in a week. The list is
|
||||
moderated; first-time posts from an email address (whether or not you
|
||||
subscribed) may be subject to some delay while waiting for a moderator
|
||||
to allow your address.
|
||||
|
||||
The larger your contribution is, or if you plan on becoming a long-term
|
||||
contributor, then the more important the rest of this page becomes.
|
||||
Reading the table of contents below should already give you an idea of
|
||||
the basic requirements. Use the table of contents as a reference, and
|
||||
read the parts that you have doubts about.
|
||||
|
||||
.. contents:: Table of Contents
|
||||
|
||||
.. _writing_your_patches:
|
||||
|
||||
Writing your Patches
|
||||
--------------------
|
||||
|
||||
.. _use_the_qemu_coding_style:
|
||||
|
||||
Use the QEMU coding style
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can run run *scripts/checkpatch.pl <patchfile>* before submitting to
|
||||
check that you are in compliance with our coding standards. Be aware
|
||||
that ``checkpatch.pl`` is not infallible, though, especially where C
|
||||
preprocessor macros are involved; use some common sense too. See also:
|
||||
|
||||
- :ref:`coding-style`
|
||||
- `Automate a checkpatch run on
|
||||
commit <https://blog.vmsplice.net/2011/03/how-to-automatically-run-checkpatchpl.html>`__
|
||||
|
||||
.. _base_patches_against_current_git_master:
|
||||
|
||||
Base patches against current git master
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
There's no point submitting a patch which is based on a released version
|
||||
of QEMU because development will have moved on from then and it probably
|
||||
won't even apply to master. We only apply selected bugfixes to release
|
||||
branches and then only as backports once the code has gone into master.
|
||||
|
||||
It is also okay to base patches on top of other on-going work that is
|
||||
not yet part of the git master branch. To aid continuous integration
|
||||
tools, such as `patchew <http://patchew.org/QEMU/>`__, you should `add a
|
||||
tag <https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg01288.html>`__
|
||||
line ``Based-on: $MESSAGE_ID`` to your cover letter to make the series
|
||||
dependency obvious.
|
||||
|
||||
.. _split_up_long_patches:
|
||||
|
||||
Split up long patches
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Split up longer patches into a patch series of logical code changes.
|
||||
Each change should compile and execute successfully. For instance, don't
|
||||
add a file to the makefile in patch one and then add the file itself in
|
||||
patch two. (This rule is here so that people can later use tools like
|
||||
`git bisect <http://git-scm.com/docs/git-bisect>`__ without hitting
|
||||
points in the commit history where QEMU doesn't work for reasons
|
||||
unrelated to the bug they're chasing.) Put documentation first, not
|
||||
last, so that someone reading the series can do a clean-room evaluation
|
||||
of the documentation, then validate that the code matched the
|
||||
documentation. A commit message that mentions "Also, ..." is often a
|
||||
good candidate for splitting into multiple patches. For more thoughts on
|
||||
properly splitting patches and writing good commit messages, see `this
|
||||
advice from
|
||||
OpenStack <https://wiki.openstack.org/wiki/GitCommitMessages>`__.
|
||||
|
||||
.. _make_code_motion_patches_easy_to_review:
|
||||
|
||||
Make code motion patches easy to review
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
If a series requires large blocks of code motion, there are tricks for
|
||||
making the refactoring easier to review. Split up the series so that
|
||||
semantic changes (or even function renames) are done in a separate patch
|
||||
from the raw code motion. Use a one-time setup of ``git config
|
||||
diff.renames true;`` ``git config diff.algorithm patience`` (refer to
|
||||
`git-config <http://git-scm.com/docs/git-config>`__). The 'diff.renames'
|
||||
property ensures file rename patches will be given in a more compact
|
||||
representation that focuses only on the differences across the file
|
||||
rename, instead of showing the entire old file as a deletion and the new
|
||||
file as an insertion. Meanwhile, the 'diff.algorithm' property ensures
|
||||
that extracting a non-contiguous subset of one file into a new file, but
|
||||
where all extracted parts occur in the same order both before and after
|
||||
the patch, will reduce churn in trying to treat unrelated ``}`` lines in
|
||||
the original file as separating hunks of changes.
|
||||
|
||||
Ideally, a code motion patch can be reviewed by doing::
|
||||
|
||||
git format-patch --stdout -1 > patch;
|
||||
diff -u <(sed -n 's/^-//p' patch) <(sed -n 's/^\+//p' patch)
|
||||
|
||||
to focus on the few changes that weren't wholesale code motion.
|
||||
|
||||
.. _dont_include_irrelevant_changes:
|
||||
|
||||
Don't include irrelevant changes
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
In particular, don't include formatting, coding style or whitespace
|
||||
changes to bits of code that would otherwise not be touched by the
|
||||
patch. (It's OK to fix coding style issues in the immediate area (few
|
||||
lines) of the lines you're changing.) If you think a section of code
|
||||
really does need a reindent or other large-scale style fix, submit this
|
||||
as a separate patch which makes no semantic changes; don't put it in the
|
||||
same patch as your bug fix.
|
||||
|
||||
For smaller patches in less frequently changed areas of QEMU, consider
|
||||
using the :ref:`trivial-patches` process.
|
||||
|
||||
.. _write_a_meaningful_commit_message:
|
||||
|
||||
Write a meaningful commit message
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Commit messages should be meaningful and should stand on their own as a
|
||||
historical record of why the changes you applied were necessary or
|
||||
useful.
|
||||
|
||||
QEMU follows the usual standard for git commit messages: the first line
|
||||
(which becomes the email subject line) is "subsystem: single line
|
||||
summary of change". Whether the "single line summary of change" starts
|
||||
with a capital is a matter of taste, but we prefer that the summary does
|
||||
not end in a dot. Look at ``git shortlog -30`` for an idea of sample
|
||||
subject lines. Then there is a blank line and a more detailed
|
||||
description of the patch, another blank and your Signed-off-by: line.
|
||||
Please do not use lines that are longer than 76 characters in your
|
||||
commit message (so that the text still shows up nicely with "git show"
|
||||
in a 80-columns terminal window).
|
||||
|
||||
The body of the commit message is a good place to document why your
|
||||
change is important. Don't include comments like "This is a suggestion
|
||||
for fixing this bug" (they can go below the ``---`` line in the email so
|
||||
they don't go into the final commit message). Make sure the body of the
|
||||
commit message can be read in isolation even if the reader's mailer
|
||||
displays the subject line some distance apart (that is, a body that
|
||||
starts with "... so that" as a continuation of the subject line is
|
||||
harder to follow).
|
||||
|
||||
If your patch fixes a commit that is already in the repository, please
|
||||
add an additional line with "Fixes: <at-least-12-digits-of-SHA-commit-id>
|
||||
("Fixed commit subject")" below the patch description / before your
|
||||
"Signed-off-by:" line in the commit message.
|
||||
|
||||
If your patch fixes a bug in the gitlab bug tracker, please add a line
|
||||
with "Resolves: <URL-of-the-bug>" to the commit message, too. Gitlab can
|
||||
close bugs automatically once commits with the "Resolves:" keyword get
|
||||
merged into the master branch of the project. And if your patch addresses
|
||||
a bug in another public bug tracker, you can also use a line with
|
||||
"Buglink: <URL-of-the-bug>" for reference here, too.
|
||||
|
||||
Example::
|
||||
|
||||
Fixes: 14055ce53c2d ("s390x/tcg: avoid overflows in time2tod/tod2time")
|
||||
Resolves: https://gitlab.com/qemu-project/qemu/-/issues/42
|
||||
Buglink: https://bugs.launchpad.net/qemu/+bug/1804323``
|
||||
|
||||
Some other tags that are used in commit messages include "Message-Id:"
|
||||
"Tested-by:", "Acked-by:", "Reported-by:", "Suggested-by:". See ``git
|
||||
log`` for these keywords for example usage.
|
||||
|
||||
.. _test_your_patches:
|
||||
|
||||
Test your patches
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
Although QEMU uses various :ref:`ci` services that attempt to test
|
||||
patches submitted to the list, it still saves everyone time if you
|
||||
have already tested that your patch compiles and works. Because QEMU
|
||||
is such a large project the default configuration won't create a
|
||||
testing pipeline on GitLab when a branch is pushed. See the :ref:`CI
|
||||
variable documentation<ci_var>` for details on how to control the
|
||||
running of tests; but it is still wise to also check that your patches
|
||||
work with a full build before submitting a series, especially if your
|
||||
changes might have an unintended effect on other areas of the code you
|
||||
don't normally experiment with. See :ref:`testing` for more details on
|
||||
what tests are available.
|
||||
|
||||
Also, it is a wise idea to include a testsuite addition as part of
|
||||
your patches - either to ensure that future changes won't regress your
|
||||
new feature, or to add a test which exposes the bug that the rest of
|
||||
your series fixes. Keeping separate commits for the test and the fix
|
||||
allows reviewers to rebase the test to occur first to prove it catches
|
||||
the problem, then again to place it last in the series so that
|
||||
bisection doesn't land on a known-broken state.
|
||||
|
||||
.. _submitting_your_patches:
|
||||
|
||||
Submitting your Patches
|
||||
-----------------------
|
||||
|
||||
The QEMU project uses a public email based workflow for reviewing and
|
||||
merging patches. As a result all contributions to QEMU must be **sent
|
||||
as patches** to the qemu-devel `mailing list
|
||||
<https://wiki.qemu.org/Contribute/MailingLists>`__. Patch
|
||||
contributions should not be posted on the bug tracker, posted on
|
||||
forums, or externally hosted and linked to. (We have other mailing
|
||||
lists too, but all patches must go to qemu-devel, possibly with a Cc:
|
||||
to another list.) ``git send-email`` (`step-by-step setup guide
|
||||
<https://git-send-email.io/>`__ and `hints and tips
|
||||
<https://elixir.bootlin.com/linux/latest/source/Documentation/process/email-clients.rst>`__)
|
||||
works best for delivering the patch without mangling it, but
|
||||
attachments can be used as a last resort on a first-time submission.
|
||||
|
||||
.. _use_b4:
|
||||
|
||||
Use B4
|
||||
~~~~~~
|
||||
|
||||
The `b4`_ tool, used for Linux kernel development, can also be used for QEMU
|
||||
development. It is packaged in most distros and PyPi. The QEMU source tree
|
||||
includes a ``b4`` project configuration file at the root: ``.b4-config``.
|
||||
|
||||
Example workflow to prepare a patch series:
|
||||
|
||||
1. Start with a clean checkout of the ``master`` branch.
|
||||
2. Create a new series with a topical branch name using ``b4 prep -n descriptive-name``.
|
||||
``b4`` will create a ``b4/descriptive-name`` branch and switch to it.
|
||||
3. Commit your changes, following this page's guidelines about proper commit messages etc.
|
||||
4. Write a descriptive cover letter with ``b4 prep --edit-cover``.
|
||||
5. Add maintainer and reviewer CCs with ``b4 prep --auto-to-cc``. You can make
|
||||
changes to Cc: and To: recipients by editing the cover letter.
|
||||
6. Run patch checks with ``b4 prep --check``.
|
||||
7. Optionally review the patches with ``b4 send --dry-run`` which will print the
|
||||
raw patches in standard output.
|
||||
|
||||
To send the patches, you can:
|
||||
|
||||
- Setup ``git-send-email`` and use ``b4 send``, or
|
||||
- Export the patches to files using ``b4 send -o OUTPUT_DIR`` and send them manually.
|
||||
|
||||
For more details, consult the `b4 documentation`_.
|
||||
|
||||
.. _b4 documentation: https://b4.docs.kernel.org/
|
||||
.. _b4: https://github.com/mricon/b4/
|
||||
|
||||
.. _use_git_publish:
|
||||
|
||||
Use git-publish
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
If you already configured git send-email, you can simply use `git-publish
|
||||
<https://github.com/stefanha/git-publish>`__ to send series.
|
||||
|
||||
::
|
||||
|
||||
$ git checkout master -b my-feature
|
||||
$ # work on new commits, add your 'Signed-off-by' lines to each
|
||||
$ git publish
|
||||
$ ... more work, rebase on master, ...
|
||||
$ git publish # will send a v2
|
||||
|
||||
Each time you post a series, git-publish will create a local tag with the format
|
||||
``<branchname>-v<version>`` to record the patch series.
|
||||
|
||||
When sending patch emails, 'git publish' will consult the output of
|
||||
'scripts/get_maintainers.pl' and automatically CC anyone listed as maintainers
|
||||
of the affected code. Generally you should accept the suggested CC list, but
|
||||
there may sometimes be scenarios where it is appropriate to cut it down (eg on
|
||||
certain large tree-wide cleanups), or augment it with other interested people.
|
||||
|
||||
.. _if_you_cannot_send_patch_emails:
|
||||
|
||||
If you cannot send patch emails
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
In rare cases it may not be possible to send properly formatted patch
|
||||
emails. You can use `sourcehut <https://sourcehut.org/>`__ to send your
|
||||
patches to the QEMU mailing list by following these steps:
|
||||
|
||||
#. Register or sign in to your account
|
||||
#. Add your SSH public key in `meta \|
|
||||
keys <https://meta.sr.ht/keys>`__.
|
||||
#. Publish your git branch using **git push [email protected]:~USERNAME/qemu
|
||||
HEAD**
|
||||
#. Send your patches to the QEMU mailing list using the web-based
|
||||
``git-send-email`` UI at https://git.sr.ht/~USERNAME/qemu/send-email
|
||||
|
||||
Documentation for sourcehut is available `here
|
||||
<https://man.sr.ht/git.sr.ht/#sending-patches-upstream>`__.
|
||||
|
||||
.. _cc_the_relevant_maintainer:
|
||||
|
||||
CC the relevant maintainer
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Send patches both to the mailing list and CC the maintainer(s) of the
|
||||
files you are modifying. look in the MAINTAINERS file to find out who
|
||||
that is. Also try using scripts/get_maintainer.pl from the repository
|
||||
for learning the most common committers for the files you touched.
|
||||
|
||||
Example::
|
||||
|
||||
~/src/qemu/scripts/get_maintainer.pl -f hw/ide/core.c
|
||||
|
||||
In fact, you can automate this, via a one-time setup of ``git config
|
||||
sendemail.cccmd 'scripts/get_maintainer.pl --nogit-fallback'`` (Refer to
|
||||
`git-config <http://git-scm.com/docs/git-config>`__.)
|
||||
|
||||
.. _do_not_send_as_an_attachment:
|
||||
|
||||
Do not send as an attachment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Send patches inline so they are easy to reply to with review comments.
|
||||
Do not put patches in attachments.
|
||||
|
||||
.. _use_git_format_patch:
|
||||
|
||||
Use ``git format-patch``
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the right diff format.
|
||||
`git format-patch <http://git-scm.com/docs/git-format-patch>`__ will
|
||||
produce patch emails in the right format (check the documentation to
|
||||
find out how to drive it). You can then edit the cover letter before
|
||||
using ``git send-email`` to mail the files to the mailing list. (We
|
||||
recommend `git send-email <http://git-scm.com/docs/git-send-email>`__
|
||||
because mail clients often mangle patches by wrapping long lines or
|
||||
messing up whitespace. Some distributions do not include send-email in a
|
||||
default install of git; you may need to download additional packages,
|
||||
such as 'git-email' on Fedora-based systems.) Patch series need a cover
|
||||
letter, with shallow threading (all patches in the series are
|
||||
in-reply-to the cover letter, but not to each other); single unrelated
|
||||
patches do not need a cover letter (but if you do send a cover letter,
|
||||
use ``--numbered`` so the cover and the patch have distinct subject lines).
|
||||
Patches are easier to find if they start a new top-level thread, rather
|
||||
than being buried in-reply-to another existing thread.
|
||||
|
||||
.. _avoid_posting_large_binary_blob:
|
||||
|
||||
Avoid posting large binary blob
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
If you added binaries to the repository, consider producing the patch
|
||||
emails using ``git format-patch --no-binary`` and include a link to a
|
||||
git repository to fetch the original commit.
|
||||
|
||||
.. _patch_emails_must_include_a_signed_off_by_line:
|
||||
|
||||
Patch emails must include a ``Signed-off-by:`` line
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Your patches **must** include a Signed-off-by: line. This is a hard
|
||||
requirement because it's how you say "I'm legally okay to contribute
|
||||
this and happy for it to go into QEMU". For full guidance, read the
|
||||
:ref:`code-provenance` documentation.
|
||||
|
||||
|
||||
.. _include_a_meaningful_cover_letter:
|
||||
|
||||
Include a meaningful cover letter
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
This is a requirement for any series with multiple patches (as it aids
|
||||
continuous integration), but optional for an isolated patch. The cover
|
||||
letter explains the overall goal of such a series, and also provides a
|
||||
convenient 0/N email for others to reply to the series as a whole. A
|
||||
one-time setup of ``git config format.coverletter auto`` (refer to
|
||||
`git-config <http://git-scm.com/docs/git-config>`__) will generate the
|
||||
cover letter as needed.
|
||||
|
||||
When reviewers don't know your goal at the start of their review, they
|
||||
may object to early changes that don't make sense until the end of the
|
||||
series, because they do not have enough context yet at that point of
|
||||
their review. A series where the goal is unclear also risks a higher
|
||||
number of review-fix cycles because the reviewers haven't bought into
|
||||
the idea yet. If the cover letter can explain these points to the
|
||||
reviewer, the process will be smoother patches will get merged faster.
|
||||
Make sure your cover letter includes a diffstat of changes made over the
|
||||
entire series; potential reviewers know what files they are interested
|
||||
in, and they need an easy way determine if your series touches them.
|
||||
|
||||
.. _use_the_rfc_tag_if_needed:
|
||||
|
||||
Use the RFC tag if needed
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
For example, "[PATCH RFC v2]". ``git format-patch --subject-prefix=RFC``
|
||||
can help.
|
||||
|
||||
"RFC" means "Request For Comments" and is a statement that you don't
|
||||
intend for your patchset to be applied to master, but would like some
|
||||
review on it anyway. Reasons for doing this include:
|
||||
|
||||
- the patch depends on some pending kernel changes which haven't yet
|
||||
been accepted, so the QEMU patch series is blocked until that
|
||||
dependency has been dealt with, but is worth reviewing anyway
|
||||
- the patch set is not finished yet (perhaps it doesn't cover all use
|
||||
cases or work with all targets) but you want early review of a major
|
||||
API change or design structure before continuing
|
||||
|
||||
In general, since it's asking other people to do review work on a
|
||||
patchset that the submitter themselves is saying shouldn't be applied,
|
||||
it's best to:
|
||||
|
||||
- use it sparingly
|
||||
- in the cover letter, be clear about why a patch is an RFC, what areas
|
||||
of the patchset you're looking for review on, and why reviewers
|
||||
should care
|
||||
|
||||
.. _consider_whether_your_patch_is_applicable_for_stable:
|
||||
|
||||
Consider whether your patch is applicable for stable
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
If your patch fixes a severe issue or a regression, it may be applicable
|
||||
for stable. In that case, consider adding ``Cc: [email protected]``
|
||||
to your patch to notify the stable maintainers.
|
||||
|
||||
For more details on how QEMU's stable process works, refer to the
|
||||
:ref:`stable-process` page.
|
||||
|
||||
.. _participating_in_code_review:
|
||||
|
||||
Retrieve an existing series
|
||||
---------------------------
|
||||
|
||||
If you want to apply an existing series on top of your tree, you can simply use
|
||||
`b4`_.
|
||||
|
||||
::
|
||||
|
||||
b4 shazam $msg-id
|
||||
|
||||
The message id is related to the patch series that has been sent to the mailing
|
||||
list. You need to retrieve the "Message-Id:" header from one of the patches. Any
|
||||
of them can be used and b4 will apply the whole series.
|
||||
|
||||
Participating in Code Review
|
||||
----------------------------
|
||||
|
||||
All patches submitted to the QEMU project go through a code review
|
||||
process before they are accepted. This will often mean a series will
|
||||
go through a number of iterations before being picked up by
|
||||
:ref:`maintainers<maintainers>`. You therefore should be prepared to
|
||||
read replies to your messages and be willing to act on them.
|
||||
|
||||
Maintainers are often willing to manually fix up first-time
|
||||
contributions, since there is a learning curve involved in making an
|
||||
ideal patch submission. However for the best results you should
|
||||
proactively respond to suggestions with changes or justifications for
|
||||
your current approach.
|
||||
|
||||
Some areas of code that are well maintained may review patches
|
||||
quickly, lesser-loved areas of code may have a longer delay.
|
||||
|
||||
.. _stay_around_to_fix_problems_raised_in_code_review:
|
||||
|
||||
Stay around to fix problems raised in code review
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Not many patches get into QEMU straight away -- it is quite common that
|
||||
developers will identify bugs, or suggest a cleaner approach, or even
|
||||
just point out code style issues or commit message typos. You'll need to
|
||||
respond to these, and then send a second version of your patches with
|
||||
the issues fixed. This takes a little time and effort on your part, but
|
||||
if you don't do it then your changes will never get into QEMU.
|
||||
|
||||
Remember that a maintainer is under no obligation to take your
|
||||
patches. If someone has spent the time reviewing your code and
|
||||
suggesting improvements and you simply re-post without either
|
||||
addressing the comment directly or providing additional justification
|
||||
for the change then it becomes wasted effort. You cannot demand others
|
||||
merge and then fix up your code after the fact.
|
||||
|
||||
When replying to comments on your patches **reply to all and not just
|
||||
the sender** -- keeping discussion on the mailing list means everybody
|
||||
can follow it. Remember the spirit of the :ref:`code_of_conduct` and
|
||||
keep discussions respectful and collaborative and avoid making
|
||||
personal comments.
|
||||
|
||||
.. _pay_attention_to_review_comments:
|
||||
|
||||
Pay attention to review comments
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Someone took their time to review your work, and it pays to respect that
|
||||
effort; repeatedly submitting a series without addressing all comments
|
||||
from the previous round tends to alienate reviewers and stall your
|
||||
patch. Reviewers aren't always perfect, so it is okay if you want to
|
||||
argue that your code was correct in the first place instead of blindly
|
||||
doing everything the reviewer asked. On the other hand, if someone
|
||||
pointed out a potential issue during review, then even if your code
|
||||
turns out to be correct, it's probably a sign that you should improve
|
||||
your commit message and/or comments in the code explaining why the code
|
||||
is correct.
|
||||
|
||||
If you fix issues that are raised during review **resend the entire
|
||||
patch series** not just the one patch that was changed. This allows
|
||||
maintainers to easily apply the fixed series without having to manually
|
||||
identify which patches are relevant. Send the new version as a complete
|
||||
fresh email or series of emails -- don't try to make it a followup to
|
||||
version 1. (This helps automatic patch email handling tools distinguish
|
||||
between v1 and v2 emails.)
|
||||
|
||||
.. _when_resending_patches_add_a_version_tag:
|
||||
|
||||
When resending patches add a version tag
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
All patches beyond the first version should include a version tag -- for
|
||||
example, "[PATCH v2]". This means people can easily identify whether
|
||||
they're looking at the most recent version. (The first version of a
|
||||
patch need not say "v1", just [PATCH] is sufficient.) For patch series,
|
||||
the version applies to the whole series -- even if you only change one
|
||||
patch, you resend the entire series and mark it as "v2". Don't try to
|
||||
track versions of different patches in the series separately. `git
|
||||
format-patch <http://git-scm.com/docs/git-format-patch>`__ and `git
|
||||
send-email <http://git-scm.com/docs/git-send-email>`__ both understand
|
||||
the ``-v2`` option to make this easier. Send each new revision as a new
|
||||
top-level thread, rather than burying it in-reply-to an earlier
|
||||
revision, as many reviewers are not looking inside deep threads for new
|
||||
patches.
|
||||
|
||||
.. _include_version_history_in_patchset_revisions:
|
||||
|
||||
Include version history in patchset revisions
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
For later versions of patches, include a summary of changes from
|
||||
previous versions, but not in the commit message itself. In an email
|
||||
formatted as a git patch, the commit message is the part above the ``---``
|
||||
line, and this will go into the git changelog when the patch is
|
||||
committed. This part should be a self-contained description of what this
|
||||
version of the patch does, written to make sense to anybody who comes
|
||||
back to look at this commit in git in six months' time. The part below
|
||||
the ``---`` line and above the patch proper (git format-patch puts the
|
||||
diffstat here) is a good place to put remarks for people reading the
|
||||
patch email, and this is where the "changes since previous version"
|
||||
summary belongs. The `git-publish
|
||||
<https://github.com/stefanha/git-publish>`__ script can help with
|
||||
tracking a good summary across versions. Also, the `git-backport-diff
|
||||
<https://github.com/codyprime/git-scripts>`__ script can help focus
|
||||
reviewers on what changed between revisions. The ``b4`` tool automatically
|
||||
generates a version history section in the cover letter, including links to the
|
||||
previous versions on `Lore`_.
|
||||
|
||||
.. _Lore: https://lore.kernel.org/
|
||||
|
||||
.. _tips_and_tricks:
|
||||
|
||||
Tips and Tricks
|
||||
---------------
|
||||
|
||||
.. _proper_use_of_reviewed_by_tags_can_aid_review:
|
||||
|
||||
Proper use of Reviewed-by: tags can aid review
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
When reviewing a large series, a reviewer can reply to some of the
|
||||
patches with a Reviewed-by tag, stating that they are happy with that
|
||||
patch in isolation (sometimes conditional on minor cleanup, like fixing
|
||||
whitespace, that doesn't affect code content). You should then update
|
||||
those commit messages by hand to include the Reviewed-by tag, so that in
|
||||
the next revision, reviewers can spot which patches were already clean
|
||||
from the previous round. Conversely, if you significantly modify a patch
|
||||
that was previously reviewed, remove the reviewed-by tag out of the
|
||||
commit message, as well as listing the changes from the previous
|
||||
version, to make it easier to focus a reviewer's attention to your
|
||||
changes.
|
||||
|
||||
.. _if_your_patch_seems_to_have_been_ignored:
|
||||
|
||||
If your patch seems to have been ignored
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
If your patchset has received no replies you should "ping" it after a
|
||||
week or two, by sending an email as a reply-to-all to the patch mail,
|
||||
including the word "ping" and ideally also a link to the page for the
|
||||
patch on `patchew <https://patchew.org/QEMU/>`__ or
|
||||
`lore.kernel.org <https://lore.kernel.org/qemu-devel/>`__. It's worth
|
||||
double-checking for reasons why your patch might have been ignored
|
||||
(forgot to CC the maintainer? annoyed people by failing to respond to
|
||||
review comments on an earlier version?), but often for less-maintained
|
||||
areas of QEMU patches do just slip through the cracks. If your ping is
|
||||
also ignored, ping again after another week or so. As the submitter, you
|
||||
are the person with the most motivation to get your patch applied, so
|
||||
you have to be persistent.
|
||||
|
||||
.. _is_my_patch_in:
|
||||
|
||||
Is my patch in?
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
QEMU has some Continuous Integration machines that try to catch patch
|
||||
submission problems as soon as possible. `patchew
|
||||
<http://patchew.org/QEMU/>`__ includes a web interface for tracking the
|
||||
status of various threads that have been posted to the list, and may
|
||||
send you an automated mail if it detected a problem with your patch.
|
||||
|
||||
Once your patch has had enough review on list, the maintainer for that
|
||||
area of code will send notification to the list that they are including
|
||||
your patch in a particular staging branch. Periodically, the maintainer
|
||||
then takes care of :ref:`submitting-a-pull-request`
|
||||
for aggregating topic branches into mainline QEMU. Generally, you do not
|
||||
need to send a pull request unless you have contributed enough patches
|
||||
to become a maintainer over a particular section of code. Maintainers
|
||||
may further modify your commit, by resolving simple merge conflicts or
|
||||
fixing minor typos pointed out during review, but will always add a
|
||||
Signed-off-by line in addition to yours, indicating that it went through
|
||||
their tree. Occasionally, the maintainer's pull request may hit more
|
||||
difficult merge conflicts, where you may be requested to help rebase and
|
||||
resolve the problems. It may take a couple of weeks between when your
|
||||
patch first had a positive review to when it finally lands in qemu.git;
|
||||
release cycle freezes may extend that time even longer.
|
||||
|
||||
.. _return_the_favor:
|
||||
|
||||
Return the favor
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
Peer review only works if everyone chips in a bit of review time. If
|
||||
everyone submitted more patches than they reviewed, we would have a
|
||||
patch backlog. A good goal is to try to review at least as many patches
|
||||
from others as what you submit. Don't worry if you don't know the code
|
||||
base as well as a maintainer; it's perfectly fine to admit when your
|
||||
review is weak because you are unfamiliar with the code.
|
||||
@@ -0,0 +1,73 @@
|
||||
.. _submitting-a-pull-request:
|
||||
|
||||
Submitting a Pull Request
|
||||
=========================
|
||||
|
||||
QEMU welcomes contributions of code, but we generally expect these to be
|
||||
sent as simple patch emails to the mailing list (see our page on
|
||||
:ref:`submitting-a-patch`
|
||||
for more details). Generally only existing submaintainers of a tree
|
||||
will need to submit pull requests, although occasionally for a large
|
||||
patch series we might ask a submitter to send a pull request. This page
|
||||
documents our recommendations on pull requests for those people.
|
||||
|
||||
A good rule of thumb is not to send a pull request unless somebody asks
|
||||
you to.
|
||||
|
||||
**Resend the patches with the pull request** as emails which are
|
||||
threaded as follow-ups to the pull request itself. The simplest way to
|
||||
do this is to use ``git format-patch --cover-letter`` to create the
|
||||
emails, and then edit the cover letter to include the pull request
|
||||
details that ``git request-pull`` outputs.
|
||||
|
||||
**Use PULL as the subject line tag** in both the cover letter and the
|
||||
retransmitted patch mails (for example, by using
|
||||
``--subject-prefix=PULL`` in your ``git format-patch`` command). This
|
||||
helps people to filter in or out the resulting emails (especially useful
|
||||
if they are only CC'd on one email out of the set).
|
||||
|
||||
**Each patch must have your own Signed-off-by: line** as well as that of
|
||||
the original author if the patch was not written by you. This is because
|
||||
with a pull request you're now indicating that the patch has passed via
|
||||
you rather than directly from the original author.
|
||||
|
||||
**Don't forget to add Reviewed-by: and Acked-by: lines**. When other
|
||||
people have reviewed the patches you're putting in the pull request,
|
||||
make sure you've copied their signoffs across. (If you use the `patches
|
||||
tool <https://github.com/stefanha/patches>`__ to add patches from email
|
||||
directly to your git repo it will include the tags automatically; if
|
||||
you're updating patches manually or in some other way you'll need to
|
||||
edit the commit messages by hand.)
|
||||
|
||||
**Don't send pull requests for code that hasn't passed review**. A pull
|
||||
request says these patches are ready to go into QEMU now, so they must
|
||||
have passed the standard code review processes. In particular if you've
|
||||
corrected issues in one round of code review, you need to send your
|
||||
fixed patch series as normal to the list; you can't put it in a pull
|
||||
request until it's gone through. (Extremely trivial fixes may be OK to
|
||||
just fix in passing, but if in doubt err on the side of not.)
|
||||
|
||||
**Test before sending**. This is an obvious thing to say, but make sure
|
||||
everything builds (including that it compiles at each step of the patch
|
||||
series) and that "make check" passes before sending out the pull
|
||||
request. As a submaintainer you're one of QEMU's lines of defense
|
||||
against bad code, so double check the details.
|
||||
|
||||
**All pull requests must be signed**. By "signed" here we mean that
|
||||
the pullreq email should quote a tag which is a GPG-signed tag (as
|
||||
created with 'gpg tag -s ...'). See :ref:`maintainer_keys` for
|
||||
details.
|
||||
|
||||
**Pull requests not for master should say "not for master" and have
|
||||
"PULL SUBSYSTEM whatever" in the subject tag**. If your pull request is
|
||||
targeting a stable branch or some submaintainer tree, please include the
|
||||
string "not for master" in the cover letter email, and make sure the
|
||||
subject tag is "PULL SUBSYSTEM s390/block/whatever" rather than just
|
||||
"PULL". This allows it to be automatically filtered out of the set of
|
||||
pull requests that should be applied to master.
|
||||
|
||||
You might be interested in the `make-pullreq
|
||||
<https://gitlab.com/pm215/misc-scripts/-/blob/master/make-pullreq>`__
|
||||
script which automates some of this process for you and includes a few
|
||||
sanity checks. Note that you must edit it to configure it suitably for
|
||||
your local situation!
|
||||
@@ -0,0 +1,88 @@
|
||||
..
|
||||
Copyright (c) 2020, Linaro Limited
|
||||
Written by Alex Bennée
|
||||
|
||||
|
||||
========================
|
||||
TCG Instruction Counting
|
||||
========================
|
||||
|
||||
TCG has long supported a feature known as icount which allows for
|
||||
instruction counting during execution. This should not be confused
|
||||
with cycle accurate emulation - QEMU does not attempt to emulate how
|
||||
long an instruction would take on real hardware. That is a job for
|
||||
other more detailed (and slower) tools that simulate the rest of a
|
||||
micro-architecture.
|
||||
|
||||
This feature is only available for system emulation and is
|
||||
incompatible with multi-threaded TCG. It can be used to better align
|
||||
execution time with wall-clock time so a "slow" device doesn't run too
|
||||
fast on modern hardware. It can also provides for a degree of
|
||||
deterministic execution and is an essential part of the record/replay
|
||||
support in QEMU.
|
||||
|
||||
Core Concepts
|
||||
=============
|
||||
|
||||
At its heart icount is simply a count of executed instructions which
|
||||
is stored in the TimersState of QEMU's timer sub-system. The number of
|
||||
executed instructions can then be used to calculate QEMU_CLOCK_VIRTUAL
|
||||
which represents the amount of elapsed time in the system since
|
||||
execution started. Depending on the icount mode this may either be a
|
||||
fixed number of ns per instruction or adjusted as execution continues
|
||||
to keep wall clock time and virtual time in sync.
|
||||
|
||||
To be able to calculate the number of executed instructions the
|
||||
translator starts by allocating a budget of instructions to be
|
||||
executed. The budget of instructions is limited by how long it will be
|
||||
until the next timer will expire. We store this budget as part of a
|
||||
vCPU icount_decr field which shared with the machinery for handling
|
||||
qemu_cpu_kick(). The whole field is checked at the start of every
|
||||
translated block and will cause a return to the outer loop to deal
|
||||
with whatever caused the exit.
|
||||
|
||||
In the case of icount, before the flag is checked we subtract the
|
||||
number of instructions the translation block would execute. If this
|
||||
would cause the instruction budget to go negative we exit the main
|
||||
loop and regenerate a new translation block with exactly the right
|
||||
number of instructions to take the budget to 0 meaning whatever timer
|
||||
was due to expire will expire exactly when we exit the main run loop.
|
||||
|
||||
Dealing with MMIO
|
||||
-----------------
|
||||
|
||||
While we can adjust the instruction budget for known events like timer
|
||||
expiry we cannot do the same for MMIO. Every load/store we execute
|
||||
might potentially trigger an I/O event, at which point we will need an
|
||||
up to date and accurate reading of the icount number.
|
||||
|
||||
To deal with this case, when an I/O access is made we:
|
||||
|
||||
- restore un-executed instructions to the icount budget
|
||||
- re-compile a single [1]_ instruction block for the current PC
|
||||
- exit the cpu loop and execute the re-compiled block
|
||||
|
||||
.. [1] sometimes two instructions if dealing with delay slots
|
||||
|
||||
Other I/O operations
|
||||
--------------------
|
||||
|
||||
MMIO isn't the only type of operation for which we might need a
|
||||
correct and accurate clock. IO port instructions and accesses to
|
||||
system registers are the common examples here. These instructions have
|
||||
to be handled by the individual translators which have the knowledge
|
||||
of which operations are I/O operations.
|
||||
|
||||
When the translator is handling an instruction of this kind:
|
||||
|
||||
* it must call gen_io_start() if icount is enabled, at some
|
||||
point before the generation of the code which actually does
|
||||
the I/O, using a code fragment similar to:
|
||||
|
||||
.. code:: c
|
||||
|
||||
if (tb_cflags(s->base.tb) & CF_USE_ICOUNT) {
|
||||
gen_io_start();
|
||||
}
|
||||
|
||||
* it must end the TB immediately after this instruction
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,172 @@
|
||||
..
|
||||
Copyright (C) 2017, Emilio G. Cota <[email protected]>
|
||||
Copyright (c) 2019, Linaro Limited
|
||||
Written by Emilio Cota and Alex Bennée
|
||||
|
||||
.. _TCG Plugins:
|
||||
|
||||
QEMU TCG Plugins
|
||||
================
|
||||
|
||||
|
||||
Writing plugins
|
||||
---------------
|
||||
|
||||
API versioning
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
This is a new feature for QEMU and it does allow people to develop
|
||||
out-of-tree plugins that can be dynamically linked into a running QEMU
|
||||
process. However the project reserves the right to change or break the
|
||||
API should it need to do so. The best way to avoid this is to submit
|
||||
your plugin upstream so they can be updated if/when the API changes.
|
||||
|
||||
All plugins need to declare a symbol which exports the plugin API
|
||||
version they were built against. This can be done simply by::
|
||||
|
||||
QEMU_PLUGIN_EXPORT int qemu_plugin_version = QEMU_PLUGIN_VERSION;
|
||||
|
||||
The core code will refuse to load a plugin that doesn't export a
|
||||
``qemu_plugin_version`` symbol or if plugin version is outside of QEMU's
|
||||
supported range of API versions.
|
||||
|
||||
Additionally the ``qemu_info_t`` structure which is passed to the
|
||||
``qemu_plugin_install`` method of a plugin will detail the minimum and
|
||||
current API versions supported by QEMU. The API version will be
|
||||
incremented if new APIs are added. The minimum API version will be
|
||||
incremented if existing APIs are changed or removed.
|
||||
|
||||
Lifetime of the query handle
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Each callback provides an opaque anonymous information handle which
|
||||
can usually be further queried to find out information about a
|
||||
translation, instruction or operation. The handles themselves are only
|
||||
valid during the lifetime of the callback so it is important that any
|
||||
information that is needed is extracted during the callback and saved
|
||||
by the plugin.
|
||||
|
||||
Plugin life cycle
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
First the plugin is loaded and the public qemu_plugin_install function
|
||||
is called. The plugin will then register callbacks for various plugin
|
||||
events. Generally plugins will register a handler for the *atexit*
|
||||
if they want to dump a summary of collected information once the
|
||||
program/system has finished running.
|
||||
|
||||
When a registered event occurs the plugin callback is invoked. The
|
||||
callbacks may provide additional information. In the case of a
|
||||
translation event the plugin has an option to enumerate the
|
||||
instructions in a block of instructions and optionally register
|
||||
callbacks to some or all instructions when they are executed.
|
||||
|
||||
There is also a facility to add inline instructions doing various operations,
|
||||
like adding or storing an immediate value. It is also possible to execute a
|
||||
callback conditionally, with condition being evaluated inline. All those inline
|
||||
operations are associated to a ``scoreboard``, which is a thread-local storage
|
||||
automatically expanded when new cores/threads are created and that can be
|
||||
accessed/modified in a thread-safe way without any lock needed. Combining inline
|
||||
operations and conditional callbacks offer a more efficient way to instrument
|
||||
binaries, compared to classic callbacks.
|
||||
|
||||
Finally when QEMU exits all the registered *atexit* callbacks are
|
||||
invoked.
|
||||
|
||||
Exposure of QEMU internals
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The plugin architecture actively avoids leaking implementation details
|
||||
about how QEMU's translation works to the plugins. While there are
|
||||
conceptions such as translation time and translation blocks the
|
||||
details are opaque to plugins. The plugin is able to query select
|
||||
details of instructions and system configuration only through the
|
||||
exported *qemu_plugin* functions.
|
||||
|
||||
However the following assumptions can be made:
|
||||
|
||||
Translation Blocks
|
||||
++++++++++++++++++
|
||||
|
||||
All code will go through a translation phase although not all
|
||||
translations will be necessarily be executed. You need to instrument
|
||||
actual executions to track what is happening.
|
||||
|
||||
It is quite normal to see the same address translated multiple times.
|
||||
If you want to track the code in system emulation you should examine
|
||||
the underlying physical address (``qemu_plugin_insn_haddr``) to take
|
||||
into account the effects of virtual memory although if the system does
|
||||
paging this will change too.
|
||||
|
||||
Not all instructions in a block will always execute so if its
|
||||
important to track individual instruction execution you need to
|
||||
instrument them directly. However asynchronous interrupts will not
|
||||
change control flow mid-block.
|
||||
|
||||
Instructions
|
||||
++++++++++++
|
||||
|
||||
Instruction instrumentation runs before the instruction executes. You
|
||||
can be can be sure the instruction will be dispatched, but you can't
|
||||
be sure it will complete. Generally this will be because of a
|
||||
synchronous exception (e.g. SIGILL) triggered by the instruction
|
||||
attempting to execute. If you want to be sure you will need to
|
||||
instrument the next instruction as well. See the ``execlog.c`` plugin
|
||||
for examples of how to track this and finalise details after execution.
|
||||
|
||||
Memory Accesses
|
||||
+++++++++++++++
|
||||
|
||||
Memory callbacks are called after a successful load or store.
|
||||
Unsuccessful operations (i.e. faults) will not be visible to memory
|
||||
instrumentation although the execution side effects can be observed
|
||||
(e.g. entering a exception handler).
|
||||
|
||||
System Idle and Resume States
|
||||
+++++++++++++++++++++++++++++
|
||||
|
||||
The ``qemu_plugin_register_vcpu_idle_cb`` and
|
||||
``qemu_plugin_register_vcpu_resume_cb`` functions can be used to track
|
||||
when CPUs go into and return from sleep states when waiting for
|
||||
external I/O. Be aware though that these may occur less frequently
|
||||
than in real HW due to the inefficiencies of emulation giving less
|
||||
chance for the CPU to idle.
|
||||
|
||||
Internals
|
||||
---------
|
||||
|
||||
Locking
|
||||
~~~~~~~
|
||||
|
||||
We have to ensure we cannot deadlock, particularly under MTTCG. For
|
||||
this we acquire a lock when called from plugin code. We also keep the
|
||||
list of callbacks under RCU so that we do not have to hold the lock
|
||||
when calling the callbacks. This is also for performance, since some
|
||||
callbacks (e.g. memory access callbacks) might be called very
|
||||
frequently.
|
||||
|
||||
* A consequence of this is that we keep our own list of CPUs, so that
|
||||
we do not have to worry about locking order wrt cpu_list_lock.
|
||||
* Use a recursive lock, since we can get registration calls from
|
||||
callbacks.
|
||||
|
||||
As a result registering/unregistering callbacks is "slow", since it
|
||||
takes a lock. But this is very infrequent; we want performance when
|
||||
calling (or not calling) callbacks, not when registering them. Using
|
||||
RCU is great for this.
|
||||
|
||||
We support the uninstallation of a plugin at any time (e.g. from
|
||||
plugin callbacks). This allows plugins to remove themselves if they no
|
||||
longer want to instrument the code. This operation is asynchronous
|
||||
which means callbacks may still occur after the uninstall operation is
|
||||
requested. The plugin isn't completely uninstalled until the safe work
|
||||
has executed while all vCPUs are quiescent.
|
||||
|
||||
Plugin API
|
||||
==========
|
||||
|
||||
The following API is generated from the inline documentation in
|
||||
``include/plugins/qemu-plugin.h``. Please ensure any updates to the API
|
||||
include the full kernel-doc annotations.
|
||||
|
||||
.. kernel-doc:: include/plugins/qemu-plugin.h
|
||||
@@ -0,0 +1,215 @@
|
||||
.. _tcg_internals:
|
||||
|
||||
====================
|
||||
Translator Internals
|
||||
====================
|
||||
|
||||
QEMU is a dynamic translator. When it first encounters a piece of code,
|
||||
it converts it to the host instruction set. Usually dynamic translators
|
||||
are very complicated and highly CPU dependent. QEMU uses some tricks
|
||||
which make it relatively easily portable and simple while achieving good
|
||||
performances.
|
||||
|
||||
QEMU's dynamic translation backend is called TCG, for "Tiny Code
|
||||
Generator". For more information, please take a look at :ref:`tcg-ops-ref`.
|
||||
|
||||
The following sections outline some notable features and implementation
|
||||
details of QEMU's dynamic translator.
|
||||
|
||||
CPU state optimisations
|
||||
-----------------------
|
||||
|
||||
The target CPUs have many internal states which change the way they
|
||||
evaluate instructions. In order to achieve a good speed, the
|
||||
translation phase considers that some state information of the virtual
|
||||
CPU cannot change in it. The state is recorded in the Translation
|
||||
Block (TB). If the state changes (e.g. privilege level), a new TB will
|
||||
be generated and the previous TB won't be used anymore until the state
|
||||
matches the state recorded in the previous TB. The same idea can be applied
|
||||
to other aspects of the CPU state. For example, on x86, if the SS,
|
||||
DS and ES segments have a zero base, then the translator does not even
|
||||
generate an addition for the segment base.
|
||||
|
||||
Direct block chaining
|
||||
---------------------
|
||||
|
||||
After each translated basic block is executed, QEMU uses the simulated
|
||||
Program Counter (PC) and other CPU state information (such as the CS
|
||||
segment base value) to find the next basic block.
|
||||
|
||||
In its simplest, less optimized form, this is done by exiting from the
|
||||
current TB, going through the TB epilogue, and then back to the
|
||||
main loop. That’s where QEMU looks for the next TB to execute,
|
||||
translating it from the guest architecture if it isn’t already available
|
||||
in memory. Then QEMU proceeds to execute this next TB, starting at the
|
||||
prologue and then moving on to the translated instructions.
|
||||
|
||||
Exiting from the TB this way will cause the ``cpu_exec_interrupt()``
|
||||
callback to be re-evaluated before executing additional instructions.
|
||||
It is mandatory to exit this way after any CPU state changes that may
|
||||
unmask interrupts.
|
||||
|
||||
In order to accelerate the cases where the TB for the new
|
||||
simulated PC is already available, QEMU has mechanisms that allow
|
||||
multiple TBs to be chained directly, without having to go back to the
|
||||
main loop as described above. These mechanisms are:
|
||||
|
||||
``lookup_and_goto_ptr``
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Calling ``tcg_gen_lookup_and_goto_ptr()`` will emit a call to
|
||||
``helper_lookup_tb_ptr``. This helper will look for an existing TB that
|
||||
matches the current CPU state. If the destination TB is available its
|
||||
code address is returned, otherwise the address of the JIT epilogue is
|
||||
returned. The call to the helper is always followed by the tcg ``goto_ptr``
|
||||
opcode, which branches to the returned address. In this way, we either
|
||||
branch to the next TB or return to the main loop.
|
||||
|
||||
``goto_tb + exit_tb``
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The translation code usually implements branching by performing the
|
||||
following steps:
|
||||
|
||||
1. Call ``tcg_gen_goto_tb()`` passing a jump slot index (either 0 or 1)
|
||||
as a parameter.
|
||||
|
||||
2. Emit TCG instructions to update the CPU state with any information
|
||||
that has been assumed constant and is required by the main loop to
|
||||
correctly locate and execute the next TB. For most guests, this is
|
||||
just the PC of the branch destination, but others may store additional
|
||||
data. The information updated in this step must be inferable from both
|
||||
``cpu_get_tb_cpu_state()`` and ``cpu_restore_state()``.
|
||||
|
||||
3. Call ``tcg_gen_exit_tb()`` passing the address of the current TB and
|
||||
the jump slot index again.
|
||||
|
||||
Step 1, ``tcg_gen_goto_tb()``, will emit a ``goto_tb`` TCG
|
||||
instruction that later on gets translated to a jump to an address
|
||||
associated with the specified jump slot. Initially, this is the address
|
||||
of step 2's instructions, which update the CPU state information. Step 3,
|
||||
``tcg_gen_exit_tb()``, exits from the current TB returning a tagged
|
||||
pointer composed of the last executed TB’s address and the jump slot
|
||||
index.
|
||||
|
||||
The first time this whole sequence is executed, step 1 simply jumps
|
||||
to step 2. Then the CPU state information gets updated and we exit from
|
||||
the current TB. As a result, the behavior is very similar to the less
|
||||
optimized form described earlier in this section.
|
||||
|
||||
Next, the main loop looks for the next TB to execute using the
|
||||
current CPU state information (creating the TB if it wasn’t already
|
||||
available) and, before starting to execute the new TB’s instructions,
|
||||
patches the previously executed TB by associating one of its jump
|
||||
slots (the one specified in the call to ``tcg_gen_exit_tb()``) with the
|
||||
address of the new TB.
|
||||
|
||||
The next time this previous TB is executed and we get to that same
|
||||
``goto_tb`` step, it will already be patched (assuming the destination TB
|
||||
is still in memory) and will jump directly to the first instruction of
|
||||
the destination TB, without going back to the main loop.
|
||||
|
||||
For the ``goto_tb + exit_tb`` mechanism to be used, the following
|
||||
conditions need to be satisfied:
|
||||
|
||||
* The change in CPU state must be constant, e.g., a direct branch and
|
||||
not an indirect branch.
|
||||
|
||||
* The direct branch cannot cross a page boundary. Memory mappings
|
||||
may change, causing the code at the destination address to change.
|
||||
|
||||
Note that, on step 3 (``tcg_gen_exit_tb()``), in addition to the
|
||||
jump slot index, the address of the TB just executed is also returned.
|
||||
This address corresponds to the TB that will be patched; it may be
|
||||
different than the one that was directly executed from the main loop
|
||||
if the latter had already been chained to other TBs.
|
||||
|
||||
Self-modifying code and translated code invalidation
|
||||
----------------------------------------------------
|
||||
|
||||
Self-modifying code is a special challenge in x86 emulation because no
|
||||
instruction cache invalidation is signaled by the application when code
|
||||
is modified.
|
||||
|
||||
User-mode emulation marks a host page as write-protected (if it is
|
||||
not already read-only) every time translated code is generated for a
|
||||
basic block. Then, if a write access is done to the page, Linux raises
|
||||
a SEGV signal. QEMU then invalidates all the translated code in the page
|
||||
and enables write accesses to the page. For system emulation, write
|
||||
protection is achieved through the software MMU.
|
||||
|
||||
Correct translated code invalidation is done efficiently by maintaining
|
||||
a linked list of every translated block contained in a given page. Other
|
||||
linked lists are also maintained to undo direct block chaining.
|
||||
|
||||
On RISC targets, correctly written software uses memory barriers and
|
||||
cache flushes, so some of the protection above would not be
|
||||
necessary. However, QEMU still requires that the generated code always
|
||||
matches the target instructions in memory in order to handle
|
||||
exceptions correctly.
|
||||
|
||||
Exception support
|
||||
-----------------
|
||||
|
||||
longjmp() is used when an exception such as division by zero is
|
||||
encountered.
|
||||
|
||||
The host SIGSEGV and SIGBUS signal handlers are used to get invalid
|
||||
memory accesses. QEMU keeps a map from host program counter to
|
||||
target program counter, and looks up where the exception happened
|
||||
based on the host program counter at the exception point.
|
||||
|
||||
On some targets, some bits of the virtual CPU's state are not flushed to the
|
||||
memory until the end of the translation block. This is done for internal
|
||||
emulation state that is rarely accessed directly by the program and/or changes
|
||||
very often throughout the execution of a translation block---this includes
|
||||
condition codes on x86, delay slots on SPARC, conditional execution on
|
||||
Arm, and so on. This state is stored for each target instruction, and
|
||||
looked up on exceptions.
|
||||
|
||||
MMU emulation
|
||||
-------------
|
||||
|
||||
For system emulation QEMU uses a software MMU. In that mode, the MMU
|
||||
virtual to physical address translation is done at every memory
|
||||
access.
|
||||
|
||||
QEMU uses an address translation cache (TLB) to speed up the translation.
|
||||
In order to avoid flushing the translated code each time the MMU
|
||||
mappings change, all caches in QEMU are physically indexed. This
|
||||
means that each basic block is indexed with its physical address.
|
||||
|
||||
In order to avoid invalidating the basic block chain when MMU mappings
|
||||
change, chaining is only performed when the destination of the jump
|
||||
shares a page with the basic block that is performing the jump.
|
||||
|
||||
The MMU can also distinguish RAM and ROM memory areas from MMIO memory
|
||||
areas. Access is faster for RAM and ROM because the translation cache also
|
||||
hosts the offset between guest address and host memory. Accessing MMIO
|
||||
memory areas instead calls out to C code for device emulation.
|
||||
Finally, the MMU helps tracking dirty pages and pages pointed to by
|
||||
translation blocks.
|
||||
|
||||
Profiling JITted code
|
||||
---------------------
|
||||
|
||||
The Linux ``perf`` tool will treat all JITted code as a single block as
|
||||
unlike the main code it can't use debug information to link individual
|
||||
program counter samples with larger functions. To overcome this
|
||||
limitation you can use the ``-perfmap`` or the ``-jitdump`` option to generate
|
||||
map files. ``-perfmap`` is lightweight and produces only guest-host mappings.
|
||||
``-jitdump`` additionally saves JITed code and guest debug information (if
|
||||
available); its output needs to be integrated with the ``perf.data`` file
|
||||
before the final report can be viewed.
|
||||
|
||||
.. code::
|
||||
|
||||
perf record $QEMU -perfmap $REMAINING_ARGS
|
||||
perf report
|
||||
|
||||
perf record -k 1 $QEMU -jitdump $REMAINING_ARGS
|
||||
DEBUGINFOD_URLS= perf inject -j -i perf.data -o perf.data.jitted
|
||||
perf report -i perf.data.jitted
|
||||
|
||||
Note that qemu-system generates mappings only for ``-kernel`` files in ELF
|
||||
format.
|
||||
@@ -0,0 +1,155 @@
|
||||
==================================
|
||||
ACPI/SMBIOS testing using biosbits
|
||||
==================================
|
||||
************
|
||||
Introduction
|
||||
************
|
||||
Biosbits is a software written by Josh Triplett that can be downloaded
|
||||
from https://biosbits.org/. The github codebase can be found
|
||||
`here <https://github.com/biosbits/bits/tree/master>`__. It is a software that
|
||||
executes the bios components such as acpi and smbios tables directly through
|
||||
acpica bios interpreter (a freely available C based library written by Intel,
|
||||
downloadable from https://acpica.org/ and is included with biosbits) without an
|
||||
operating system getting involved in between. Bios-bits has python integration
|
||||
with grub so actual routines that executes bios components can be written in
|
||||
python instead of bash-ish (grub's native scripting language).
|
||||
There are several advantages to directly testing the bios in a real physical
|
||||
machine or in a VM as opposed to indirectly discovering bios issues through the
|
||||
operating system (the OS). Operating systems tend to bypass bios problems and
|
||||
hide them from the end user. We have more control of what we wanted to test and
|
||||
how by being as close to the bios on a running system as possible without a
|
||||
complicated software component such as an operating system coming in between.
|
||||
Another issue is that we cannot exercise bios components such as ACPI and
|
||||
SMBIOS without being in the highest hardware privilege level, ring 0 for
|
||||
example in case of x86. Since the OS executes from ring 0 whereas normal user
|
||||
land software resides in unprivileged ring 3, operating system must be modified
|
||||
in order to write our test routines that exercise and test the bios. This is
|
||||
not possible in all cases. Lastly, test frameworks and routines are preferably
|
||||
written using a high level scripting language such as python. OSes and
|
||||
OS modules are generally written using low level languages such as C and
|
||||
low level assembly machine language. Writing test routines in a low level
|
||||
language makes things more cumbersome. These and other reasons makes using
|
||||
bios-bits very attractive for testing bioses. More details on the inspiration
|
||||
for developing biosbits and its real life uses were presented `at Plumbers
|
||||
in 2011 <Plumbers_>`__ and `at Linux.conf.au in 2012 <Linux.conf.au_>`__.
|
||||
|
||||
For QEMU, we maintain a fork of bios bits in `gitlab`_, along with all
|
||||
the dependent submodules. This fork contains numerous fixes, a newer
|
||||
acpica and changes specific to running these functional QEMU tests using
|
||||
bits. The author of this document is the current maintainer of the QEMU
|
||||
fork of bios bits repository. For more information, please see `the
|
||||
author's FOSDEM presentation <FOSDEM_>`__ on this bios-bits based test framework.
|
||||
|
||||
.. _Plumbers: https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
|
||||
.. _Linux.conf.au: https://www.youtube.com/watch?v=36QIepyUuhg
|
||||
.. _gitlab: https://gitlab.com/qemu-project/biosbits-bits
|
||||
.. _FOSDEM: https://fosdem.org/2024/schedule/event/fosdem-2024-2262-exercising-qemu-generated-acpi-smbios-tables-using-biosbits-from-within-a-guest-vm-/
|
||||
|
||||
*********************************
|
||||
Description of the test framework
|
||||
*********************************
|
||||
|
||||
Under the directory ``tests/functional/``, ``test_acpi_bits.py`` is a QEMU
|
||||
functional test that drives all this.
|
||||
|
||||
A brief description of the various test files follows.
|
||||
|
||||
Under ``tests/functional/`` as the root we have:
|
||||
|
||||
::
|
||||
|
||||
├── acpi-bits
|
||||
│ ├── bits-config
|
||||
│ │ └── bits-cfg.txt
|
||||
│ ├── bits-tests
|
||||
│ ├── smbios.py2
|
||||
│ ├── testacpi.py2
|
||||
│ └── testcpuid.py2
|
||||
├── test_acpi_bits.py
|
||||
|
||||
* ``tests/functional``:
|
||||
|
||||
``test_acpi_bits.py``:
|
||||
This is the main python functional test script that generates a
|
||||
biosbits iso. It then spawns a QEMU VM with it, collects the log and reports
|
||||
test failures. This is the script one would be interested in if they wanted
|
||||
to add or change some component of the log parsing, add a new command line
|
||||
to alter how QEMU is spawned etc. Test writers typically would not need to
|
||||
modify this script unless they wanted to enhance or change the log parsing
|
||||
for their tests. In order to enable debugging, you can set **V=1**
|
||||
environment variable. This enables verbose mode for the test and also dumps
|
||||
the entire log from bios bits and more information in case failure happens.
|
||||
You can also set **BITS_DEBUG=1** to turn on debug mode. It will enable
|
||||
verbose logs and also retain the temporary work directory the test used for
|
||||
you to inspect and run the specific commands manually.
|
||||
|
||||
In order to run this test, please perform the following steps from the QEMU
|
||||
build directory (assuming that the sources are in ".."):
|
||||
::
|
||||
|
||||
$ export PYTHONPATH=../python:../tests/functional
|
||||
$ export QEMU_TEST_QEMU_BINARY=$PWD/qemu-system-x86_64
|
||||
$ python3 ../tests/functional/test_acpi_bits.py
|
||||
|
||||
The above will run all acpi-bits functional tests (producing output in
|
||||
tap format).
|
||||
|
||||
You can inspect the log files in tests/functional/x86_64/test_acpi_bits.*/
|
||||
for more information about the run or in order to diagnoze issues.
|
||||
If you pass V=1 in the environment, more diagnostic logs will be put into
|
||||
the test log.
|
||||
|
||||
* ``tests/functional/acpi-bits/bits-config``:
|
||||
|
||||
This location contains biosbits configuration files that determine how the
|
||||
software runs the tests.
|
||||
|
||||
``bits-config.txt``:
|
||||
This is the biosbits config file that determines what tests
|
||||
or actions are performed by bits. The description of the config options are
|
||||
provided in the file itself.
|
||||
|
||||
* ``tests/functional/acpi-bits/bits-tests``:
|
||||
|
||||
This directory contains biosbits python based tests that are run from within
|
||||
the biosbits environment in the spawned VM. New additions of test cases can
|
||||
be made in the appropriate test file. For example, new acpi tests can go
|
||||
into testacpi.py2 and one would call testsuite.add_test() to register the new
|
||||
test so that it gets executed as a part of the ACPI tests.
|
||||
It might be occasionally necessary to disable some subtests or add a new
|
||||
test that belongs to a test suite not already present in this directory. To
|
||||
do this, please clone the bits source from
|
||||
https://gitlab.com/qemu-project/biosbits-bits/-/tree/qemu-bits.
|
||||
Note that this is the "qemu-bits" branch and not the "bits" branch of the
|
||||
repository. "qemu-bits" is the branch where we have made all the QEMU
|
||||
specific enhancements and we must use the source from this branch only.
|
||||
Copy the test suite/script that needs modification (addition of new tests
|
||||
or disabling them) from python directory into this directory. For
|
||||
example, in order to change cpuid related tests, copy the following
|
||||
file into this directory and rename it with .py2 extension:
|
||||
https://gitlab.com/qemu-project/biosbits-bits/-/blob/qemu-bits/python/testcpuid.py
|
||||
Then make your additions and changes here. Therefore, the steps are:
|
||||
|
||||
(a) Copy unmodified test script to this directory from bits source.
|
||||
(b) Add a SPDX license header.
|
||||
(c) Perform modifications to the test.
|
||||
|
||||
Commits (a), (b) and (c) preferably should go under separate commits so that
|
||||
the original test script and the changes we have made are separated and
|
||||
clear. (a) and (b) can sometimes be combined into a single step.
|
||||
|
||||
The test framework will then use your modified test script to run the test.
|
||||
No further changes would be needed. Please check the logs to make sure that
|
||||
appropriate changes have taken effect.
|
||||
|
||||
The tests have an extension .py2 in order to indicate that:
|
||||
|
||||
(a) They are python2.7 based scripts and not python 3 scripts.
|
||||
(b) They are run from within the bios bits VM and is not subjected to QEMU
|
||||
build/test python script maintenance and dependency resolutions.
|
||||
(c) They need not be loaded by the test framework by accident when running
|
||||
tests.
|
||||
|
||||
|
||||
Author: Ani Sinha <[email protected]>
|
||||
|
||||
@@ -0,0 +1,177 @@
|
||||
Block I/O error injection using ``blkdebug``
|
||||
============================================
|
||||
|
||||
..
|
||||
Copyright (C) 2014-2015 Red Hat Inc
|
||||
|
||||
This work is licensed under the terms of the GNU GPL, version 2 or later. See
|
||||
the COPYING file in the top-level directory.
|
||||
|
||||
The ``blkdebug`` block driver is a rule-based error injection engine. It can be
|
||||
used to exercise error code paths in block drivers including ``ENOSPC`` (out of
|
||||
space) and ``EIO``.
|
||||
|
||||
This document gives an overview of the features available in ``blkdebug``.
|
||||
|
||||
Background
|
||||
----------
|
||||
Block drivers have many error code paths that handle I/O errors. Image formats
|
||||
are especially complex since metadata I/O errors during cluster allocation or
|
||||
while updating tables happen halfway through request processing and require
|
||||
discipline to keep image files consistent.
|
||||
|
||||
Error injection allows test cases to trigger I/O errors at specific points.
|
||||
This way, all error paths can be tested to make sure they are correct.
|
||||
|
||||
Rules
|
||||
-----
|
||||
The ``blkdebug`` block driver takes a list of "rules" that tell the error injection
|
||||
engine when to fail an I/O request.
|
||||
|
||||
Each I/O request is evaluated against the rules. If a rule matches the request
|
||||
then its "action" is executed.
|
||||
|
||||
Rules can be placed in a configuration file; the configuration file
|
||||
follows the same .ini-like format used by QEMU's ``-readconfig`` option, and
|
||||
each section of the file represents a rule.
|
||||
|
||||
The following configuration file defines a single rule::
|
||||
|
||||
$ cat blkdebug.conf
|
||||
[inject-error]
|
||||
event = "read_aio"
|
||||
errno = "28"
|
||||
|
||||
This rule fails all aio read requests with ``ENOSPC`` (28). Note that the errno
|
||||
value depends on the host. On Linux, see
|
||||
``/usr/include/asm-generic/errno-base.h`` for errno values.
|
||||
|
||||
Invoke QEMU as follows::
|
||||
|
||||
$ qemu-system-x86_64
|
||||
-drive if=none,cache=none,file=blkdebug:blkdebug.conf:test.img,id=drive0 \
|
||||
-device virtio-blk-pci,drive=drive0,id=virtio-blk-pci0
|
||||
|
||||
Rules support the following attributes:
|
||||
|
||||
``event``
|
||||
which type of operation to match (e.g. ``read_aio``, ``write_aio``,
|
||||
``flush_to_os``, ``flush_to_disk``). See `Events`_ for
|
||||
information on events.
|
||||
|
||||
``state``
|
||||
(optional) the engine must be in this state number in order for this
|
||||
rule to match. See `State transitions`_ for information
|
||||
on states.
|
||||
|
||||
``errno``
|
||||
the numeric errno value to return when a request matches this rule.
|
||||
The errno values depend on the host since the numeric values are not
|
||||
standardized in the POSIX specification.
|
||||
|
||||
``sector``
|
||||
(optional) a sector number that the request must overlap in order to
|
||||
match this rule
|
||||
|
||||
``once``
|
||||
(optional, default ``off``) only execute this action on the first
|
||||
matching request
|
||||
|
||||
``immediately``
|
||||
(optional, default ``off``) return a NULL ``BlockAIOCB``
|
||||
pointer and fail without an errno instead. This
|
||||
exercises the code path where ``BlockAIOCB`` fails and the
|
||||
caller's ``BlockCompletionFunc`` is not invoked.
|
||||
|
||||
Events
|
||||
------
|
||||
Block drivers provide information about the type of I/O request they are about
|
||||
to make so rules can match specific types of requests. For example, the ``qcow2``
|
||||
block driver tells ``blkdebug`` when it accesses the L1 table so rules can match
|
||||
only L1 table accesses and not other metadata or guest data requests.
|
||||
|
||||
The core events are:
|
||||
|
||||
``read_aio``
|
||||
guest data read
|
||||
|
||||
``write_aio``
|
||||
guest data write
|
||||
|
||||
``flush_to_os``
|
||||
write out unwritten block driver state (e.g. cached metadata)
|
||||
|
||||
``flush_to_disk``
|
||||
flush the host block device's disk cache
|
||||
|
||||
See ``qapi/block-core.json:BlkdebugEvent`` for the full list of events.
|
||||
You may need to grep block driver source code to understand the
|
||||
meaning of specific events.
|
||||
|
||||
State transitions
|
||||
-----------------
|
||||
There are cases where more power is needed to match a particular I/O request in
|
||||
a longer sequence of requests. For example::
|
||||
|
||||
write_aio
|
||||
flush_to_disk
|
||||
write_aio
|
||||
|
||||
How do we match the 2nd ``write_aio`` but not the first? This is where state
|
||||
transitions come in.
|
||||
|
||||
The error injection engine has an integer called the "state" that always starts
|
||||
initialized to 1. The state integer is internal to ``blkdebug`` and cannot be
|
||||
observed from outside but rules can interact with it for powerful matching
|
||||
behavior.
|
||||
|
||||
Rules can be conditional on the current state and they can transition to a new
|
||||
state.
|
||||
|
||||
When a rule's "state" attribute is non-zero then the current state must equal
|
||||
the attribute in order for the rule to match.
|
||||
|
||||
For example, to match the 2nd write_aio::
|
||||
|
||||
[set-state]
|
||||
event = "write_aio"
|
||||
state = "1"
|
||||
new_state = "2"
|
||||
|
||||
[inject-error]
|
||||
event = "write_aio"
|
||||
state = "2"
|
||||
errno = "5"
|
||||
|
||||
The first ``write_aio`` request matches the ``set-state`` rule and transitions from
|
||||
state 1 to state 2. Once state 2 has been entered, the ``set-state`` rule no
|
||||
longer matches since it requires state 1. But the ``inject-error`` rule now
|
||||
matches the next ``write_aio`` request and injects ``EIO`` (5).
|
||||
|
||||
State transition rules support the following attributes:
|
||||
|
||||
``event``
|
||||
which type of operation to match (e.g. ``read_aio``, ``write_aio``,
|
||||
``flush_to_os`, ``flush_to_disk``). See `Events`_ for
|
||||
information on events.
|
||||
|
||||
``state``
|
||||
(optional) the engine must be in this state number in order for this
|
||||
rule to match
|
||||
|
||||
``new_state``
|
||||
transition to this state number
|
||||
|
||||
Suspend and resume
|
||||
------------------
|
||||
Exercising code paths in block drivers may require specific ordering amongst
|
||||
concurrent requests. The "breakpoint" feature allows requests to be halted on
|
||||
a ``blkdebug`` event and resumed later. This makes it possible to achieve
|
||||
deterministic ordering when multiple requests are in flight.
|
||||
|
||||
Breakpoints on ``blkdebug`` events are associated with a user-defined ``tag`` string.
|
||||
This tag serves as an identifier by which the request can be resumed at a later
|
||||
point.
|
||||
|
||||
See the ``qemu-io(1)`` ``break``, ``resume``, ``remove_break``, and ``wait_break``
|
||||
commands for details.
|
||||
@@ -0,0 +1,73 @@
|
||||
Block driver correctness testing with ``blkverify``
|
||||
===================================================
|
||||
|
||||
Introduction
|
||||
------------
|
||||
|
||||
This document describes how to use the ``blkverify`` protocol to test that a block
|
||||
driver is operating correctly.
|
||||
|
||||
It is difficult to test and debug block drivers against real guests. Often
|
||||
processes inside the guest will crash because corrupt sectors were read as part
|
||||
of the executable. Other times obscure errors are raised by a program inside
|
||||
the guest. These issues are extremely hard to trace back to bugs in the block
|
||||
driver.
|
||||
|
||||
``blkverify`` solves this problem by catching data corruption inside QEMU the first
|
||||
time bad data is read and reporting the disk sector that is corrupted.
|
||||
|
||||
How it works
|
||||
------------
|
||||
|
||||
The ``blkverify`` protocol has two child block devices, the "test" device and the
|
||||
"raw" device. Read/write operations are mirrored to both devices so their
|
||||
state should always be in sync.
|
||||
|
||||
The "raw" device is a raw image, a flat file, that has identical starting
|
||||
contents to the "test" image. The idea is that the "raw" device will handle
|
||||
read/write operations correctly and not corrupt data. It can be used as a
|
||||
reference for comparison against the "test" device.
|
||||
|
||||
After a mirrored read operation completes, ``blkverify`` will compare the data and
|
||||
raise an error if it is not identical. This makes it possible to catch the
|
||||
first instance where corrupt data is read.
|
||||
|
||||
Example
|
||||
-------
|
||||
|
||||
Imagine raw.img has 0xcd repeated throughout its first sector::
|
||||
|
||||
$ ./qemu-io -c 'read -v 0 512' raw.img
|
||||
00000000: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................
|
||||
00000010: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................
|
||||
[...]
|
||||
000001e0: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................
|
||||
000001f0: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................
|
||||
read 512/512 bytes at offset 0
|
||||
512.000000 bytes, 1 ops; 0.0000 sec (97.656 MiB/sec and 200000.0000 ops/sec)
|
||||
|
||||
And test.img is corrupt, its first sector is zeroed when it shouldn't be::
|
||||
|
||||
$ ./qemu-io -c 'read -v 0 512' test.img
|
||||
00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
|
||||
00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
|
||||
[...]
|
||||
000001e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
|
||||
000001f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
|
||||
read 512/512 bytes at offset 0
|
||||
512.000000 bytes, 1 ops; 0.0000 sec (81.380 MiB/sec and 166666.6667 ops/sec)
|
||||
|
||||
This error is caught by ``blkverify``::
|
||||
|
||||
$ ./qemu-io -c 'read 0 512' blkverify:a.img:b.img
|
||||
blkverify: read sector_num=0 nb_sectors=4 contents mismatch in sector 0
|
||||
|
||||
A more realistic scenario is verifying the installation of a guest OS::
|
||||
|
||||
$ ./qemu-img create raw.img 16G
|
||||
$ ./qemu-img create -f qcow2 test.qcow2 16G
|
||||
$ ./qemu-system-x86_64 -cdrom debian.iso \
|
||||
-drive file=blkverify:raw.img:test.qcow2
|
||||
|
||||
If the installation is aborted when ``blkverify`` detects corruption, use ``qemu-io``
|
||||
to explore the contents of the disk image at the sector in question.
|
||||
@@ -0,0 +1,189 @@
|
||||
.. _ci_var:
|
||||
|
||||
Custom CI/CD variables
|
||||
======================
|
||||
|
||||
QEMU CI pipelines can be tuned by setting some CI environment variables.
|
||||
|
||||
Set variable globally in the user's CI namespace
|
||||
------------------------------------------------
|
||||
|
||||
Variables can be set globally in the user's CI namespace setting.
|
||||
|
||||
For further information about how to set these variables, please refer to::
|
||||
|
||||
https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project
|
||||
|
||||
Set variable manually when pushing a branch or tag to the user's repository
|
||||
---------------------------------------------------------------------------
|
||||
|
||||
Variables can be set manually when pushing a branch or tag, using
|
||||
git-push command line arguments.
|
||||
|
||||
Example setting the QEMU_CI_EXAMPLE_VAR variable:
|
||||
|
||||
.. code::
|
||||
|
||||
git push -o ci.variable="QEMU_CI_EXAMPLE_VAR=value" myrepo mybranch
|
||||
|
||||
For further information about how to set these variables, please refer to::
|
||||
|
||||
https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
|
||||
|
||||
Setting aliases in your git config
|
||||
----------------------------------
|
||||
|
||||
You can use aliases to make it easier to push branches with different
|
||||
CI configurations. For example define an alias for triggering CI:
|
||||
|
||||
.. code::
|
||||
|
||||
git config --local alias.push-ci "push -o ci.variable=QEMU_CI=1"
|
||||
git config --local alias.push-ci-now "push -o ci.variable=QEMU_CI=2"
|
||||
|
||||
Which lets you run:
|
||||
|
||||
.. code::
|
||||
|
||||
git push-ci
|
||||
|
||||
to create the pipeline, or:
|
||||
|
||||
.. code::
|
||||
|
||||
git push-ci-now
|
||||
|
||||
to create and run the pipeline
|
||||
|
||||
|
||||
Variable naming and grouping
|
||||
----------------------------
|
||||
|
||||
The variables used by QEMU's CI configuration are grouped together
|
||||
in a handful of namespaces
|
||||
|
||||
* QEMU_JOB_nnnn - variables to be defined in individual jobs
|
||||
or templates, to influence the shared rules defined in the
|
||||
.base_job_template.
|
||||
|
||||
* QEMU_CI_nnn - variables to be set by contributors in their
|
||||
repository CI settings, or as git push variables, to influence
|
||||
which jobs get run in a pipeline
|
||||
|
||||
* QEMU_CI_CONTAINER_TAG - the tag used to publish containers
|
||||
in stage 1, for use by build jobs in stage 2. Defaults to
|
||||
'latest', but if running pipelines for different branches
|
||||
concurrently, it should be overridden per pipeline.
|
||||
|
||||
* QEMU_CI_UPSTREAM - gitlab namespace that is considered to be
|
||||
the 'upstream'. This defaults to 'qemu-project'. Contributors
|
||||
may choose to override this if they are modifying rules in
|
||||
base.yml and need to validate how they will operate when in
|
||||
an upstream context, as opposed to their fork context.
|
||||
|
||||
* nnn - other misc variables not falling into the above
|
||||
categories, or using different names for historical reasons
|
||||
and not yet converted.
|
||||
|
||||
Maintainer controlled job variables
|
||||
-----------------------------------
|
||||
|
||||
The following variables may be set when defining a job in the
|
||||
CI configuration file.
|
||||
|
||||
QEMU_JOB_OPTIONAL
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
The job is expected to be successful in general, but is not run
|
||||
by default due to need to conserve limited CI resources. It is
|
||||
available to be started manually by the contributor in the CI
|
||||
pipelines UI.
|
||||
|
||||
QEMU_JOB_ONLY_FORKS
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The job results are only of interest to contributors prior to
|
||||
submitting code. They are not required as part of the gating
|
||||
CI pipeline.
|
||||
|
||||
QEMU_JOB_SKIPPED
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
The job is not reliably successful in general, so is not
|
||||
currently suitable to be run by default. Ideally this should
|
||||
be a temporary marker until the problems can be addressed, or
|
||||
the job permanently removed.
|
||||
|
||||
QEMU_JOB_PUBLISH
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
The job is for publishing content after a branch has been
|
||||
merged into the upstream default branch.
|
||||
|
||||
QEMU_JOB_FUNCTIONAL
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The job runs the functional test suite
|
||||
|
||||
QEMU_JOB_MACOS
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
The job requires access to the macOS shared runners, which are
|
||||
only available to Premium/Ultimate tiers. This includes OSS project
|
||||
members, but not forks.
|
||||
|
||||
Contributor controlled runtime variables
|
||||
----------------------------------------
|
||||
|
||||
The following variables may be set by contributors to control
|
||||
job execution
|
||||
|
||||
QEMU_CI
|
||||
~~~~~~~
|
||||
|
||||
By default, no pipelines will be created on contributor forks
|
||||
in order to preserve CI credits
|
||||
|
||||
Set this variable to 1 to create the pipelines, but leave all
|
||||
the jobs to be manually started from the UI
|
||||
|
||||
Set this variable to 2 to create the pipelines and run all
|
||||
the jobs immediately, as was the historical behaviour
|
||||
|
||||
QEMU_CI_FUNCTIONAL
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
By default, tests using the functional framework are not run automatically
|
||||
in the pipelines (because multiple artifacts have to be downloaded, which
|
||||
might cause a lot of network traffic). Set this variable to have the tests
|
||||
using the functional framework run automatically.
|
||||
|
||||
Other misc variables
|
||||
--------------------
|
||||
|
||||
These variables are primarily to control execution of jobs on
|
||||
private runners
|
||||
|
||||
AARCH64_RUNNER_AVAILABLE
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
If you've got access to an aarch64 host that can be used as a gitlab-CI
|
||||
runner, you can set this variable to enable the tests that require this
|
||||
kind of host. The runner should be tagged with "aarch64".
|
||||
|
||||
S390X_RUNNER_AVAILABLE
|
||||
~~~~~~~~~~~~~~~~~~~~~~
|
||||
If you've got access to an IBM Z host that can be used as a gitlab-CI
|
||||
runner, you can set this variable to enable the tests that require this
|
||||
kind of host. The runner should be tagged with "s390x".
|
||||
|
||||
PPC64LE_RUNNER_AVAILABLE
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
If you've got access to an PPC64LE host that can be used as a gitlab-CI
|
||||
runner, you can set this variable to enable the tests that require this
|
||||
kind of host. The runner should be tagged with "ppc64le".
|
||||
|
||||
CCACHE_DISABLE
|
||||
~~~~~~~~~~~~~~
|
||||
The jobs are configured to use "ccache" by default since this typically
|
||||
reduces compilation time, at the cost of increased storage. If the
|
||||
use of "ccache" is suspected to be hurting the overall job execution
|
||||
time, setting the "CCACHE_DISABLE=1" env variable to disable it.
|
||||
@@ -0,0 +1,116 @@
|
||||
Jobs on Custom Runners
|
||||
======================
|
||||
|
||||
Besides the jobs run under the various CI systems listed before, there
|
||||
are a number additional jobs that will run before an actual merge.
|
||||
These use the same GitLab CI's service/framework already used for all
|
||||
other GitLab based CI jobs, but rely on additional systems, not the
|
||||
ones provided by GitLab as "shared runners".
|
||||
|
||||
The architecture of GitLab's CI service allows different machines to
|
||||
be set up with GitLab's "agent", called gitlab-runner, which will take
|
||||
care of running jobs created by events such as a push to a branch.
|
||||
Here, the combination of a machine, properly configured with GitLab's
|
||||
gitlab-runner, is called a "custom runner".
|
||||
|
||||
The GitLab CI jobs definition for the custom runners are located under::
|
||||
|
||||
.gitlab-ci.d/custom-runners.yml
|
||||
|
||||
Custom runners entail custom machines. To see a list of the machines
|
||||
currently deployed in the QEMU GitLab CI and their maintainers, please
|
||||
refer to the QEMU `wiki <https://wiki.qemu.org/AdminContacts>`__.
|
||||
|
||||
Machine Setup Howto
|
||||
-------------------
|
||||
|
||||
For all Linux based systems, the setup can be mostly automated by the
|
||||
execution of two Ansible playbooks. Create an ``inventory`` file
|
||||
under ``scripts/ci/setup``, such as this::
|
||||
|
||||
fully.qualified.domain
|
||||
other.machine.hostname
|
||||
|
||||
You may need to set some variables in the inventory file itself. One
|
||||
very common need is to tell Ansible to use a Python 3 interpreter on
|
||||
those hosts. This would look like::
|
||||
|
||||
fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
|
||||
other.machine.hostname ansible_python_interpreter=/usr/bin/python3
|
||||
|
||||
Build environment
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
The ``scripts/ci/setup/$DISTRO/build-environment.yml`` Ansible
|
||||
playbook will set up machines with the environment needed to perform
|
||||
builds and run QEMU tests. This playbook consists on the installation
|
||||
of various required packages (and a general package update while at
|
||||
it).
|
||||
|
||||
The minimum required version of Ansible successfully tested in this
|
||||
playbook is 2.8.0 (a version check is embedded within the playbook
|
||||
itself). To run the playbook, execute::
|
||||
|
||||
cd scripts/ci/setup
|
||||
ansible-playbook -i inventory $DISTRO/build-environment.yml
|
||||
|
||||
Please note that most of the tasks in the playbook require superuser
|
||||
privileges, such as those from the ``root`` account or those obtained
|
||||
by ``sudo``. If necessary, please refer to ``ansible-playbook``
|
||||
options such as ``--become``, ``--become-method``, ``--become-user``
|
||||
and ``--ask-become-pass``.
|
||||
|
||||
gitlab-runner setup and registration
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The gitlab-runner agent needs to be installed on each machine that
|
||||
will run jobs. The association between a machine and a GitLab project
|
||||
happens with a registration token. To find the registration token for
|
||||
your repository/project, navigate on GitLab's web UI to:
|
||||
|
||||
* Settings (the gears-like icon at the bottom of the left hand side
|
||||
vertical toolbar), then
|
||||
* CI/CD, then
|
||||
* Runners, and click on the "Expand" button, then
|
||||
* Under "Set up a specific Runner manually", look for the value under
|
||||
"And this registration token:"
|
||||
|
||||
Copy the ``scripts/ci/setup/vars.yml.template`` file to
|
||||
``scripts/ci/setup/vars.yml``. Then, set the
|
||||
``gitlab_runner_registration_token`` variable to the value obtained
|
||||
earlier.
|
||||
|
||||
To run the playbook, execute::
|
||||
|
||||
cd scripts/ci/setup
|
||||
ansible-playbook -i inventory gitlab-runner.yml
|
||||
|
||||
Following the registration, it's necessary to configure the runner tags,
|
||||
and optionally other configurations on the GitLab UI. Navigate to:
|
||||
|
||||
* Settings (the gears like icon), then
|
||||
* CI/CD, then
|
||||
* Runners, and click on the "Expand" button, then
|
||||
* "Runners activated for this project", then
|
||||
* Click on the "Edit" icon (next to the "Lock" Icon)
|
||||
|
||||
Tags are very important as they are used to route specific jobs to
|
||||
specific types of runners, so it's a good idea to double check that
|
||||
the automatically created tags are consistent with the OS and
|
||||
architecture. For instance, an Ubuntu 20.04 aarch64 system should
|
||||
have tags set as::
|
||||
|
||||
ubuntu_20.04,aarch64
|
||||
|
||||
Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
|
||||
would contain::
|
||||
|
||||
ubuntu-20.04-aarch64-all:
|
||||
tags:
|
||||
- ubuntu_20.04
|
||||
- aarch64
|
||||
|
||||
It's also recommended to:
|
||||
|
||||
* increase the "Maximum job timeout" to something like ``2h``
|
||||
* give it a better Description
|
||||
@@ -0,0 +1,34 @@
|
||||
.. _ci:
|
||||
|
||||
Continuous Integration (CI)
|
||||
===========================
|
||||
|
||||
Continuous integration (CI) requires the builds of the entire application and
|
||||
the execution of a comprehensive set of automated tests every time there is a
|
||||
need to commit any set of changes [1]_. The automated tests are composed
|
||||
of unit, functional and other tests.
|
||||
|
||||
Most of QEMU's CI is run on GitLab's infrastructure although a number
|
||||
of other CI services are used for specialised purposes. The most up to
|
||||
date information about them and their status can be found on the
|
||||
`project wiki testing page <https://wiki.qemu.org/Testing/CI>`_.
|
||||
|
||||
These tests are also used as gating tests before merging pull requests.
|
||||
A gating test restricts the move of code from one stage to another on a
|
||||
test/deployment pipeline. The step move is granted with approval. The approval
|
||||
can be a manual intervention or a set of tests succeeding [2]_.
|
||||
|
||||
On QEMU, the gating process happens during the pull request. The approval is
|
||||
done by the project leader running its own set of tests. The pull request gets
|
||||
merged when the tests succeed.
|
||||
|
||||
.. include:: ci-jobs.rst.inc
|
||||
.. include:: ci-runners.rst.inc
|
||||
|
||||
References
|
||||
----------
|
||||
|
||||
.. [1] Humble, Jez & Farley, David (2010). Continuous Delivery:
|
||||
Reliable Software Releases Through Build, Test, and Deployment, p. 55.
|
||||
.. [2] Humble, Jez & Farley, David (2010). Continuous Delivery:
|
||||
Reliable Software Releases Through Build, Test, and Deployment, p. 122.
|
||||
@@ -0,0 +1,397 @@
|
||||
.. _checkfunctional-ref:
|
||||
|
||||
Functional testing with Python
|
||||
==============================
|
||||
|
||||
The ``tests/functional`` directory hosts functional tests written in
|
||||
Python. They are usually higher level tests, and may interact with
|
||||
external resources and with various guest operating systems.
|
||||
|
||||
The tests should be written in the style of the Python `unittest`_ framework,
|
||||
using stdio for the TAP protocol. The folder ``tests/functional/qemu_test``
|
||||
provides classes (e.g. the ``QemuBaseTest``, ``QemuUserTest`` and the
|
||||
``QemuSystemTest`` classes) and utility functions that help to get your test
|
||||
into the right shape, e.g. by replacing the 'stdout' python object to redirect
|
||||
the normal output of your test to stderr instead.
|
||||
|
||||
Note that if you don't use one of the QemuBaseTest based classes for your
|
||||
test, or if you spawn subprocesses from your test, you have to make sure
|
||||
that there is no TAP-incompatible output written to stdio, e.g. either by
|
||||
prefixing every line with a "# " to mark the output as a TAP comment, or
|
||||
e.g. by capturing the stdout output of subprocesses (redirecting it to
|
||||
stderr is OK).
|
||||
|
||||
Tests based on ``qemu_test.QemuSystemTest`` can easily:
|
||||
|
||||
* Customize the command line arguments given to the convenience
|
||||
``self.vm`` attribute (a QEMUMachine instance)
|
||||
|
||||
* Interact with the QEMU monitor, send QMP commands and check
|
||||
their results
|
||||
|
||||
* Interact with the guest OS, using the convenience console device
|
||||
(which may be useful to assert the effectiveness and correctness of
|
||||
command line arguments or QMP commands)
|
||||
|
||||
* Download (and cache) remote data files, such as firmware and kernel
|
||||
images
|
||||
|
||||
Running tests
|
||||
-------------
|
||||
|
||||
You can run the functional tests simply by executing:
|
||||
|
||||
.. code::
|
||||
|
||||
make check-functional
|
||||
|
||||
It is also possible to run tests for a certain target only, for example
|
||||
the following line will only run the tests for the x86_64 target:
|
||||
|
||||
.. code::
|
||||
|
||||
make check-functional-x86_64
|
||||
|
||||
To run a single test file without the meson test runner, you can also
|
||||
execute the file directly by specifying the name of the emulator target
|
||||
binary as an env variable.
|
||||
|
||||
Assuming the current working directory is the top level source checkout
|
||||
and the build directory is './build'::
|
||||
|
||||
$ export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
|
||||
|
||||
Run all tests from a test file::
|
||||
|
||||
$ ./build/run tests/functional/x86_64/test_virtio_version.py
|
||||
|
||||
Run all tests from a test class::
|
||||
|
||||
$ ./build/run tests/functional/x86_64/test_virtio_version.py VirtioVersionCheck
|
||||
|
||||
Or a single test::
|
||||
|
||||
$ ./build/run tests/functional/x86_64/test_virtio_version.py VirtioVersionCheck.test_modern_only_devs
|
||||
|
||||
Filtering test names also works::
|
||||
|
||||
$ ./build/run tests/functional/x86_64/test_virtio_version.py -k modern
|
||||
|
||||
The test framework will automatically purge any scratch files created during
|
||||
the tests. If needing to debug a failed test, it is possible to keep these
|
||||
files around on disk by setting ``QEMU_TEST_KEEP_SCRATCH=1`` as an env
|
||||
variable. Any preserved files will be deleted the next time the test is run
|
||||
without this variable set.
|
||||
|
||||
Logging
|
||||
-------
|
||||
|
||||
The framework collects log files for each test in the build directory
|
||||
in the following subfolder::
|
||||
|
||||
<builddir>/tests/functional/<arch>/<fileid>.<classid>.<testname>/
|
||||
|
||||
There are usually three log files:
|
||||
|
||||
* ``base.log`` contains the generic logging information that is written
|
||||
by the calls to the logging functions in the test code (e.g. by calling
|
||||
the ``self.log.info()`` or ``self.log.debug()`` functions).
|
||||
* ``console.log`` contains the output of the serial console of the guest.
|
||||
* ``default.log`` contains the output of QEMU. This file could be named
|
||||
differently if the test chooses to use a different identifier for
|
||||
the guest VM (e.g. when the test spins up multiple VMs).
|
||||
|
||||
Introduction to writing tests
|
||||
-----------------------------
|
||||
|
||||
The ``tests/functional/qemu_test`` directory provides the ``qemu_test``
|
||||
Python module, containing the ``qemu_test.QemuSystemTest`` class.
|
||||
Here is a simple usage example:
|
||||
|
||||
.. code::
|
||||
|
||||
#!/usr/bin/env python3
|
||||
|
||||
from qemu_test import QemuSystemTest
|
||||
|
||||
class Version(QemuSystemTest):
|
||||
|
||||
def test_qmp_human_info_version(self):
|
||||
self.vm.launch()
|
||||
res = self.vm.cmd('human-monitor-command',
|
||||
command_line='info version')
|
||||
self.assertRegex(res, r'^(\d+\.\d+\.\d)')
|
||||
|
||||
if __name__ == '__main__':
|
||||
QemuSystemTest.main()
|
||||
|
||||
By providing the "hash bang" line at the beginning of the script, marking
|
||||
the file as executable and by calling into QemuSystemTest.main(), the test
|
||||
can also be run stand-alone, without a test runner. OTOH when run via a test
|
||||
runner, the QemuSystemTest.main() function takes care of running the test
|
||||
functions in the right fassion (e.g. with TAP output that is required by the
|
||||
meson test runner).
|
||||
|
||||
The ``qemu_test.QemuSystemTest`` base test class
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The ``qemu_test.QemuSystemTest`` class has a number of characteristics
|
||||
that are worth being mentioned.
|
||||
|
||||
First of all, it attempts to give each test a ready to use QEMUMachine
|
||||
instance, available at ``self.vm``. Because many tests will tweak the
|
||||
QEMU command line, launching the QEMUMachine (by using ``self.vm.launch()``)
|
||||
is left to the test writer.
|
||||
|
||||
The base test class has also support for tests with more than one
|
||||
QEMUMachine. The way to get machines is through the ``self.get_vm()``
|
||||
method which will return a QEMUMachine instance. The ``self.get_vm()``
|
||||
method accepts arguments that will be passed to the QEMUMachine creation
|
||||
and also an optional ``name`` attribute so you can identify a specific
|
||||
machine and get it more than once through the tests methods. A simple
|
||||
and hypothetical example follows:
|
||||
|
||||
.. code::
|
||||
|
||||
from qemu_test import QemuSystemTest
|
||||
|
||||
class MultipleMachines(QemuSystemTest):
|
||||
def test_multiple_machines(self):
|
||||
first_machine = self.get_vm()
|
||||
second_machine = self.get_vm()
|
||||
self.get_vm(name='third_machine').launch()
|
||||
|
||||
first_machine.launch()
|
||||
second_machine.launch()
|
||||
|
||||
first_res = first_machine.cmd(
|
||||
'human-monitor-command',
|
||||
command_line='info version')
|
||||
|
||||
second_res = second_machine.cmd(
|
||||
'human-monitor-command',
|
||||
command_line='info version')
|
||||
|
||||
third_res = self.get_vm(name='third_machine').cmd(
|
||||
'human-monitor-command',
|
||||
command_line='info version')
|
||||
|
||||
self.assertEqual(first_res, second_res, third_res)
|
||||
|
||||
At test "tear down", ``qemu_test.QemuSystemTest`` handles all the QEMUMachines
|
||||
shutdown.
|
||||
|
||||
QEMUMachine
|
||||
-----------
|
||||
|
||||
The QEMUMachine API is already widely used in the Python iotests,
|
||||
device-crash-test and other Python scripts. It's a wrapper around the
|
||||
execution of a QEMU binary, giving its users:
|
||||
|
||||
* the ability to set command line arguments to be given to the QEMU
|
||||
binary
|
||||
|
||||
* a ready to use QMP connection and interface, which can be used to
|
||||
send commands and inspect its results, as well as asynchronous
|
||||
events
|
||||
|
||||
* convenience methods to set commonly used command line arguments in
|
||||
a more succinct and intuitive way
|
||||
|
||||
QEMU binary selection
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The QEMU binary used for the ``self.vm`` QEMUMachine instance will
|
||||
primarily depend on the value of the ``qemu_bin`` instance attribute.
|
||||
If it is not explicitly set by the test code, its default value will
|
||||
be the result the QEMU_TEST_QEMU_BINARY environment variable.
|
||||
|
||||
Debugging hung QEMU
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
When test cases go wrong it may be helpful to debug a stalled QEMU
|
||||
process. While the QEMUMachine class owns the primary QMP monitor
|
||||
socket, it is possible to request a second QMP monitor be created
|
||||
by setting the ``QEMU_TEST_QMP_BACKDOOR`` env variable to refer
|
||||
to a UNIX socket name. The ``qmp-shell`` command can then be
|
||||
attached to the stalled QEMU to examine its live state.
|
||||
|
||||
Attribute reference
|
||||
-------------------
|
||||
|
||||
QemuBaseTest
|
||||
^^^^^^^^^^^^
|
||||
|
||||
The following attributes are available on any ``qemu_test.QemuBaseTest``
|
||||
instance.
|
||||
|
||||
arch
|
||||
""""
|
||||
|
||||
The target architecture of the QEMU binary.
|
||||
|
||||
Tests are also free to use this attribute value, for their own needs.
|
||||
A test may, for instance, use this value when selecting the architecture
|
||||
of a kernel or disk image to boot a VM with.
|
||||
|
||||
qemu_bin
|
||||
""""""""
|
||||
|
||||
The preserved value of the ``QEMU_TEST_QEMU_BINARY`` environment
|
||||
variable.
|
||||
|
||||
QemuUserTest
|
||||
^^^^^^^^^^^^
|
||||
|
||||
The QemuUserTest class can be used for running an executable via the
|
||||
usermode emulation binaries.
|
||||
|
||||
QemuSystemTest
|
||||
^^^^^^^^^^^^^^
|
||||
|
||||
The QemuSystemTest class can be used for running tests via one of the
|
||||
qemu-system-* binaries.
|
||||
|
||||
vm
|
||||
""
|
||||
|
||||
A QEMUMachine instance, initially configured according to the given
|
||||
``qemu_bin`` parameter.
|
||||
|
||||
cpu
|
||||
"""
|
||||
|
||||
The cpu model that will be set to all QEMUMachine instances created
|
||||
by the test.
|
||||
|
||||
machine
|
||||
"""""""
|
||||
|
||||
The machine type that will be set to all QEMUMachine instances created
|
||||
by the test. By using the set_machine() function of the QemuSystemTest
|
||||
class to set this attribute, you can automatically check whether the
|
||||
machine is available to skip the test in case it is not built into the
|
||||
QEMU binary.
|
||||
|
||||
Asset handling
|
||||
--------------
|
||||
|
||||
Many functional tests download assets (e.g. Linux kernels, initrds,
|
||||
firmware images, etc.) from the internet to be able to run tests with
|
||||
them. This imposes additional challenges to the test framework.
|
||||
|
||||
First there is the problem that some people might not have an
|
||||
unconstrained internet connection, so such tests should not be run by
|
||||
default when running ``make check``. To accomplish this situation,
|
||||
the tests that download files should only be added to the "thorough"
|
||||
speed mode in the meson.build file, while the "quick" speed mode is
|
||||
fine for functional tests that can be run without downloading files.
|
||||
``make check`` then only runs the quick functional tests along with
|
||||
the other quick tests from the other test suites. If you choose to
|
||||
run only ``make check-functional``, the "thorough" tests will be
|
||||
executed, too. And to run all functional tests along with the others,
|
||||
you can use something like::
|
||||
|
||||
make -j$(nproc) check SPEED=thorough
|
||||
|
||||
The second problem with downloading files from the internet are time
|
||||
constraints. The time for downloading files should not be taken into
|
||||
account when the test is running and the timeout of the test is ticking
|
||||
(since downloading can be very slow, depending on the network bandwidth).
|
||||
This problem is solved by downloading the assets ahead of time, before
|
||||
the tests are run. This pre-caching is done with the qemu_test.Asset
|
||||
class. To use it in your test, declare an asset in your test class with
|
||||
its URL and SHA256 checksum like this::
|
||||
|
||||
from qemu_test import Asset
|
||||
|
||||
ASSET_somename = Asset(
|
||||
('https://www.qemu.org/assets/images/qemu_head_200.png'),
|
||||
'34b74cad46ea28a2966c1d04e102510daf1fd73e6582b6b74523940d5da029dd')
|
||||
|
||||
In your test function, you can then get the file name of the cached
|
||||
asset like this::
|
||||
|
||||
def test_function(self):
|
||||
file_path = self.ASSET_somename.fetch()
|
||||
|
||||
The pre-caching will be done automatically when running
|
||||
``make check-functional`` (but not when running e.g.
|
||||
``make check-functional-<target>``). In case you just want to download
|
||||
the assets without running the tests, you can do so by running::
|
||||
|
||||
make precache-functional
|
||||
|
||||
The cache is populated in the ``~/.cache/qemu/download`` directory by
|
||||
default, but the location can be changed by setting the
|
||||
``QEMU_TEST_CACHE_DIR`` environment variable.
|
||||
|
||||
To force the test suite to re-download the cache, even if still valid,
|
||||
set the ``QEMU_TEST_REFRESH_CACHE`` environment variable.
|
||||
|
||||
Skipping tests
|
||||
--------------
|
||||
|
||||
Since the test framework is based on the common Python unittest framework,
|
||||
you can use the usual Python decorators which allow for easily skipping
|
||||
tests running under certain conditions, for example, on the lack of a binary
|
||||
on the test system or when the running environment is a CI system. For further
|
||||
information about those decorators, please refer to:
|
||||
|
||||
https://docs.python.org/3/library/unittest.html#skipping-tests-and-expected-failures
|
||||
|
||||
While the conditions for skipping tests are often specifics of each one, there
|
||||
are recurring scenarios identified by the QEMU developers and the use of
|
||||
environment variables became a kind of standard way to enable/disable tests.
|
||||
|
||||
Here is a list of the most used variables:
|
||||
|
||||
QEMU_TEST_ALLOW_LARGE_STORAGE
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Tests which are going to fetch or produce assets considered *large* are not
|
||||
going to run unless that ``QEMU_TEST_ALLOW_LARGE_STORAGE=1`` is exported on
|
||||
the environment.
|
||||
|
||||
The definition of *large* is a bit arbitrary here, but it usually means an
|
||||
asset which occupies at least 1GB of size on disk when uncompressed.
|
||||
|
||||
QEMU_TEST_ALLOW_UNTRUSTED_CODE
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
There are tests which will boot a kernel image or firmware that can be
|
||||
considered not safe to run on the developer's workstation, thus they are
|
||||
skipped by default. The definition of *not safe* is also arbitrary but
|
||||
usually it means a blob which either its source or build process aren't
|
||||
public available.
|
||||
|
||||
You should export ``QEMU_TEST_ALLOW_UNTRUSTED_CODE=1`` on the environment in
|
||||
order to allow tests which make use of those kind of assets.
|
||||
|
||||
QEMU_TEST_FLAKY_TESTS
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
Some tests are not working reliably and thus are disabled by default.
|
||||
This includes tests that don't run reliably on GitLab's CI which
|
||||
usually expose real issues that are rarely seen on developer machines
|
||||
due to the constraints of the CI environment. If you encounter a
|
||||
similar situation then raise a bug and then mark the test as shown on
|
||||
the code snippet below:
|
||||
|
||||
.. code::
|
||||
|
||||
# See https://gitlab.com/qemu-project/qemu/-/issues/nnnn
|
||||
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
|
||||
def test(self):
|
||||
do_something()
|
||||
|
||||
Tests should not live in this state forever and should either be fixed
|
||||
or eventually removed.
|
||||
|
||||
QEMU_TEST_ALLOW_SLOW
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
Tests that have a very long runtime and might run into timeout issues
|
||||
e.g. if the QEMU binary has been compiled with debugging options enabled.
|
||||
To avoid these timeout issues by default and to save some precious CPU
|
||||
cycles during normal testing, such tests are disabled by default unless
|
||||
the QEMU_TEST_ALLOW_SLOW environment variable has been set.
|
||||
|
||||
|
||||
.. _unittest: https://docs.python.org/3/library/unittest.html
|
||||
@@ -0,0 +1,314 @@
|
||||
========
|
||||
Fuzzing
|
||||
========
|
||||
|
||||
This document describes the virtual-device fuzzing infrastructure in QEMU and
|
||||
how to use it to implement additional fuzzers.
|
||||
|
||||
Basics
|
||||
------
|
||||
|
||||
Fuzzing operates by passing inputs to an entry point/target function. The
|
||||
fuzzer tracks the code coverage triggered by the input. Based on these
|
||||
findings, the fuzzer mutates the input and repeats the fuzzing.
|
||||
|
||||
To fuzz QEMU, we rely on libfuzzer. Unlike other fuzzers such as AFL, libfuzzer
|
||||
is an *in-process* fuzzer. For the developer, this means that it is their
|
||||
responsibility to ensure that state is reset between fuzzing-runs.
|
||||
|
||||
Building the fuzzers
|
||||
--------------------
|
||||
|
||||
To build the fuzzers, install a recent version of clang:
|
||||
Configure with (substitute the clang binaries with the version you installed).
|
||||
Here, enable-asan and enable-ubsan are optional but they allow us to reliably
|
||||
detect bugs such as out-of-bounds accesses, uses-after-free, double-frees
|
||||
etc.::
|
||||
|
||||
CC=clang-8 CXX=clang++-8 /path/to/configure \
|
||||
--enable-fuzzing --enable-asan --enable-ubsan
|
||||
|
||||
Fuzz targets are built similarly to system targets::
|
||||
|
||||
make qemu-fuzz-i386
|
||||
|
||||
This builds ``./qemu-fuzz-i386``
|
||||
|
||||
The first option to this command is: ``--fuzz-target=FUZZ_NAME``
|
||||
To list all of the available fuzzers run ``qemu-fuzz-i386`` with no arguments.
|
||||
|
||||
For example::
|
||||
|
||||
./qemu-fuzz-i386 --fuzz-target=virtio-scsi-fuzz
|
||||
|
||||
Internally, libfuzzer parses all arguments that do not begin with ``"--"``.
|
||||
Information about these is available by passing ``-help=1``
|
||||
|
||||
Now the only thing left to do is wait for the fuzzer to trigger potential
|
||||
crashes.
|
||||
|
||||
Useful libFuzzer flags
|
||||
----------------------
|
||||
|
||||
As mentioned above, libFuzzer accepts some arguments. Passing ``-help=1`` will
|
||||
list the available arguments. In particular, these arguments might be helpful:
|
||||
|
||||
* ``CORPUS_DIR/`` : Specify a directory as the last argument to libFuzzer.
|
||||
libFuzzer stores each "interesting" input in this corpus directory. The next
|
||||
time you run libFuzzer, it will read all of the inputs from the corpus, and
|
||||
continue fuzzing from there. You can also specify multiple directories.
|
||||
libFuzzer loads existing inputs from all specified directories, but will only
|
||||
write new ones to the first one specified.
|
||||
|
||||
* ``-max_len=4096`` : specify the maximum byte-length of the inputs libFuzzer
|
||||
will generate.
|
||||
|
||||
* ``-close_fd_mask={1,2,3}`` : close, stderr, or both. Useful for targets that
|
||||
trigger many debug/error messages, or create output on the serial console.
|
||||
|
||||
* ``-jobs=4 -workers=4`` : These arguments configure libFuzzer to run 4 fuzzers in
|
||||
parallel (4 fuzzing jobs in 4 worker processes). Alternatively, with only
|
||||
``-jobs=N``, libFuzzer automatically spawns a number of workers less than or equal
|
||||
to half the available CPU cores. Replace 4 with a number appropriate for your
|
||||
machine. Make sure to specify a ``CORPUS_DIR``, which will allow the parallel
|
||||
fuzzers to share information about the interesting inputs they find.
|
||||
|
||||
* ``-use_value_profile=1`` : For each comparison operation, libFuzzer computes
|
||||
``(caller_pc&4095) | (popcnt(Arg1 ^ Arg2) << 12)`` and places this in the
|
||||
coverage table. Useful for targets with "magic" constants. If Arg1 came from
|
||||
the fuzzer's input and Arg2 is a magic constant, then each time the Hamming
|
||||
distance between Arg1 and Arg2 decreases, libFuzzer adds the input to the
|
||||
corpus.
|
||||
|
||||
* ``-shrink=1`` : Tries to make elements of the corpus "smaller". Might lead to
|
||||
better coverage performance, depending on the target.
|
||||
|
||||
Note that libFuzzer's exact behavior will depend on the version of
|
||||
clang and libFuzzer used to build the device fuzzers.
|
||||
|
||||
Generating Coverage Reports
|
||||
---------------------------
|
||||
|
||||
Code coverage is a crucial metric for evaluating a fuzzer's performance.
|
||||
libFuzzer's output provides a "cov: " column that provides a total number of
|
||||
unique blocks/edges covered. To examine coverage on a line-by-line basis we
|
||||
can use Clang coverage:
|
||||
|
||||
1. Configure libFuzzer to store a corpus of all interesting inputs (see
|
||||
CORPUS_DIR above)
|
||||
2. ``./configure`` the QEMU build with ::
|
||||
|
||||
--enable-fuzzing \
|
||||
--extra-cflags="-fprofile-instr-generate -fcoverage-mapping"
|
||||
|
||||
3. Re-run the fuzzer. Specify $CORPUS_DIR/* as an argument, telling libfuzzer
|
||||
to execute all of the inputs in $CORPUS_DIR and exit. Once the process
|
||||
exits, you should find a file, "default.profraw" in the working directory.
|
||||
4. Execute these commands to generate a detailed HTML coverage-report::
|
||||
|
||||
llvm-profdata merge -output=default.profdata default.profraw
|
||||
llvm-cov show ./path/to/qemu-fuzz-i386 -instr-profile=default.profdata \
|
||||
--format html -output-dir=/path/to/output/report
|
||||
|
||||
Adding a new fuzzer
|
||||
-------------------
|
||||
|
||||
Coverage over virtual devices can be improved by adding additional fuzzers.
|
||||
Fuzzers are kept in ``tests/qtest/fuzz/`` and should be added to
|
||||
``tests/qtest/fuzz/meson.build``
|
||||
|
||||
Fuzzers can rely on both qtest and libqos to communicate with virtual devices.
|
||||
|
||||
1. Create a new source file. For example ``tests/qtest/fuzz/foo-device-fuzz.c``.
|
||||
|
||||
2. Write the fuzzing code using the libqtest/libqos API. See existing fuzzers
|
||||
for reference.
|
||||
|
||||
3. Add the fuzzer to ``tests/qtest/fuzz/meson.build``.
|
||||
|
||||
Fuzzers can be more-or-less thought of as special qtest programs which can
|
||||
modify the qtest commands and/or qtest command arguments based on inputs
|
||||
provided by libfuzzer. Libfuzzer passes a byte array and length. Commonly the
|
||||
fuzzer loops over the byte-array interpreting it as a list of qtest commands,
|
||||
addresses, or values.
|
||||
|
||||
The Generic Fuzzer
|
||||
------------------
|
||||
|
||||
Writing a fuzz target can be a lot of effort (especially if a device driver has
|
||||
not be built-out within libqos). Many devices can be fuzzed to some degree,
|
||||
without any device-specific code, using the generic-fuzz target.
|
||||
|
||||
The generic-fuzz target is capable of fuzzing devices over their PIO, MMIO,
|
||||
and DMA input-spaces. To apply the generic-fuzz to a device, we need to define
|
||||
two env-variables, at minimum:
|
||||
|
||||
* ``QEMU_FUZZ_ARGS=`` is the set of QEMU arguments used to configure a machine, with
|
||||
the device attached. For example, if we want to fuzz the virtio-net device
|
||||
attached to a pc-i440fx machine, we can specify::
|
||||
|
||||
QEMU_FUZZ_ARGS="-M pc -nodefaults -netdev user,id=user0 \
|
||||
-device virtio-net,netdev=user0"
|
||||
|
||||
* ``QEMU_FUZZ_OBJECTS=`` is a set of space-delimited strings used to identify
|
||||
the MemoryRegions that will be fuzzed. These strings are compared against
|
||||
MemoryRegion names and MemoryRegion owner names, to decide whether each
|
||||
MemoryRegion should be fuzzed. These strings support globbing. For the
|
||||
virtio-net example, we could use one of ::
|
||||
|
||||
QEMU_FUZZ_OBJECTS='virtio-net'
|
||||
QEMU_FUZZ_OBJECTS='virtio*'
|
||||
QEMU_FUZZ_OBJECTS='virtio* pcspk' # Fuzz the virtio devices and the speaker
|
||||
QEMU_FUZZ_OBJECTS='*' # Fuzz the whole machine``
|
||||
|
||||
The ``"info mtree"`` and ``"info qom-tree"`` monitor commands can be especially
|
||||
useful for identifying the ``MemoryRegion`` and ``Object`` names used for
|
||||
matching.
|
||||
|
||||
As a generic rule-of-thumb, the more ``MemoryRegions``/Devices we match, the
|
||||
greater the input-space, and the smaller the probability of finding crashing
|
||||
inputs for individual devices. As such, it is usually a good idea to limit the
|
||||
fuzzer to only a few ``MemoryRegions``.
|
||||
|
||||
To ensure that these env variables have been configured correctly, we can use::
|
||||
|
||||
./qemu-fuzz-i386 --fuzz-target=generic-fuzz -runs=0
|
||||
|
||||
The output should contain a complete list of matched MemoryRegions.
|
||||
|
||||
OSS-Fuzz
|
||||
--------
|
||||
QEMU is continuously fuzzed on `OSS-Fuzz
|
||||
<https://github.com/google/oss-fuzz>`_. By default, the OSS-Fuzz build
|
||||
will try to fuzz every fuzz-target. Since the generic-fuzz target
|
||||
requires additional information provided in environment variables, we
|
||||
pre-define some generic-fuzz configs in
|
||||
``tests/qtest/fuzz/generic_fuzz_configs.h``. Each config must specify:
|
||||
|
||||
- ``.name``: To identify the fuzzer config
|
||||
|
||||
- ``.args`` OR ``.argfunc``: A string or pointer to a function returning a
|
||||
string. These strings are used to specify the ``QEMU_FUZZ_ARGS``
|
||||
environment variable. ``argfunc`` is useful when the config relies on e.g.
|
||||
a dynamically created temp directory, or a free tcp/udp port.
|
||||
|
||||
- ``.objects``: A string that specifies the ``QEMU_FUZZ_OBJECTS`` environment
|
||||
variable.
|
||||
|
||||
To fuzz additional devices/device configuration on OSS-Fuzz, send patches for
|
||||
either a new device-specific fuzzer or a new generic-fuzz config.
|
||||
|
||||
Build details:
|
||||
|
||||
- The Dockerfile that sets up the environment for building QEMU's
|
||||
fuzzers on OSS-Fuzz can be fund in the OSS-Fuzz repository
|
||||
__(https://github.com/google/oss-fuzz/blob/master/projects/qemu/Dockerfile)
|
||||
|
||||
- The script responsible for building the fuzzers can be found in the
|
||||
QEMU source tree at ``scripts/oss-fuzz/build.sh``
|
||||
|
||||
Building Crash Reproducers
|
||||
-----------------------------------------
|
||||
When we find a crash, we should try to create an independent reproducer, that
|
||||
can be used on a non-fuzzer build of QEMU. This filters out any potential
|
||||
false-positives, and improves the debugging experience for developers.
|
||||
Here are the steps for building a reproducer for a crash found by the
|
||||
generic-fuzz target.
|
||||
|
||||
- Ensure the crash reproduces::
|
||||
|
||||
qemu-fuzz-i386 --fuzz-target... ./crash-...
|
||||
|
||||
- Gather the QTest output for the crash::
|
||||
|
||||
QEMU_FUZZ_TIMEOUT=0 QTEST_LOG=1 FUZZ_SERIALIZE_QTEST=1 \
|
||||
qemu-fuzz-i386 --fuzz-target... ./crash-... &> /tmp/trace
|
||||
|
||||
- Reorder and clean-up the resulting trace::
|
||||
|
||||
scripts/oss-fuzz/reorder_fuzzer_qtest_trace.py /tmp/trace > /tmp/reproducer
|
||||
|
||||
- Get the arguments needed to start qemu, and provide a path to qemu::
|
||||
|
||||
less /tmp/trace # The args should be logged at the top of this file
|
||||
export QEMU_ARGS="-machine ..."
|
||||
export QEMU_PATH="path/to/qemu-system"
|
||||
|
||||
- Ensure the crash reproduces in qemu-system::
|
||||
|
||||
$QEMU_PATH $QEMU_ARGS -qtest stdio < /tmp/reproducer
|
||||
|
||||
- From the crash output, obtain some string that identifies the crash. This
|
||||
can be a line in the stack-trace, for example::
|
||||
|
||||
export CRASH_TOKEN="hw/usb/hcd-xhci.c:1865"
|
||||
|
||||
- Minimize the reproducer::
|
||||
|
||||
scripts/oss-fuzz/minimize_qtest_trace.py -M1 -M2 \
|
||||
/tmp/reproducer /tmp/reproducer-minimized
|
||||
|
||||
- Confirm that the minimized reproducer still crashes::
|
||||
|
||||
$QEMU_PATH $QEMU_ARGS -qtest stdio < /tmp/reproducer-minimized
|
||||
|
||||
- Create a one-liner reproducer that can be sent over email::
|
||||
|
||||
./scripts/oss-fuzz/output_reproducer.py -bash /tmp/reproducer-minimized
|
||||
|
||||
- Output the C source code for a test case that will reproduce the bug::
|
||||
|
||||
./scripts/oss-fuzz/output_reproducer.py -owner "John Smith <[email protected]>"\
|
||||
-name "test_function_name" /tmp/reproducer-minimized
|
||||
|
||||
- Report the bug and send a patch with the C reproducer upstream
|
||||
|
||||
QEMU can also read the reproducer directly from a file rather than
|
||||
from standard input::
|
||||
|
||||
$QEMU_PATH $QEMU_ARGS -qtest chardev:repro \
|
||||
-chardev file,id=repro,path=/dev/null,input-path=/tmp/reproducer
|
||||
|
||||
This is useful if you want to run QEMU under a debugger to investigate
|
||||
the failure.
|
||||
|
||||
Implementation Details / Fuzzer Lifecycle
|
||||
-----------------------------------------
|
||||
|
||||
The fuzzer has two entrypoints that libfuzzer calls. libfuzzer provides it's
|
||||
own ``main()``, which performs some setup, and calls the entrypoints:
|
||||
|
||||
``LLVMFuzzerInitialize``: called prior to fuzzing. Used to initialize all of the
|
||||
necessary state
|
||||
|
||||
``LLVMFuzzerTestOneInput``: called for each fuzzing run. Processes the input and
|
||||
resets the state at the end of each run.
|
||||
|
||||
In more detail:
|
||||
|
||||
``LLVMFuzzerInitialize`` parses the arguments to the fuzzer (must start with two
|
||||
dashes, so they are ignored by libfuzzer ``main()``). Currently, the arguments
|
||||
select the fuzz target. Then, the qtest client is initialized. If the target
|
||||
requires qos, qgraph is set up and the QOM/LIBQOS modules are initialized.
|
||||
Then the QGraph is walked and the QEMU cmd_line is determined and saved.
|
||||
|
||||
After this, the ``vl.c:main`` is called to set up the guest. There are
|
||||
target-specific hooks that can be called before and after main, for
|
||||
additional setup(e.g. PCI setup, or VM snapshotting).
|
||||
|
||||
``LLVMFuzzerTestOneInput``: Uses qtest/qos functions to act based on the fuzz
|
||||
input. It is also responsible for manually calling ``main_loop_wait`` to ensure
|
||||
that bottom halves are executed and any cleanup required before the next input.
|
||||
|
||||
Since the same process is reused for many fuzzing runs, QEMU state needs to
|
||||
be reset at the end of each run. For example, this can be done by rebooting the
|
||||
VM, after each run.
|
||||
|
||||
- *Pros*: Straightforward and fast for simple fuzz targets.
|
||||
|
||||
- *Cons*: Depending on the device, does not reset all device state. If the
|
||||
device requires some initialization prior to being ready for fuzzing (common
|
||||
for QOS-based targets), this initialization needs to be done after each
|
||||
reboot.
|
||||
|
||||
- *Example target*: ``i440fx-qtest-reboot-fuzz``
|
||||
@@ -0,0 +1,17 @@
|
||||
Testing QEMU
|
||||
------------
|
||||
|
||||
Details about how to test QEMU and how it is integrated into our CI
|
||||
testing infrastructure.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 3
|
||||
|
||||
main
|
||||
qtest
|
||||
functional
|
||||
acpi-bits
|
||||
ci
|
||||
fuzzing
|
||||
blkdebug
|
||||
blkverify
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,628 @@
|
||||
.. _qgraph:
|
||||
|
||||
Qtest Driver Framework
|
||||
======================
|
||||
|
||||
In order to test a specific driver, plain libqos tests need to
|
||||
take care of booting QEMU with the right machine and devices.
|
||||
This makes each test "hardcoded" for a specific configuration, reducing
|
||||
the possible coverage that it can reach.
|
||||
|
||||
For example, the sdhci device is supported on both x86_64 and Arm boards,
|
||||
therefore a generic sdhci test should test all machines and drivers that
|
||||
support that device.
|
||||
Using only libqos APIs, the test has to manually take care of
|
||||
covering all the setups, and build the correct command line.
|
||||
|
||||
This also introduces backward compatibility issues: if a device/driver command
|
||||
line name is changed, all tests that use that will not work
|
||||
properly anymore and need to be adjusted.
|
||||
|
||||
The aim of qgraph is to create a graph of drivers, machines and tests such that
|
||||
a test aimed to a certain driver does not have to care of
|
||||
booting the right QEMU machine, pick the right device, build the command line
|
||||
and so on. Instead, it only defines what type of device it is testing
|
||||
(interface in qgraph terms) and the framework takes care of
|
||||
covering all supported types of devices and machine architectures.
|
||||
|
||||
Following the above example, an interface would be ``sdhci``,
|
||||
so the sdhci-test should only care of linking its qgraph node with
|
||||
that interface. In this way, if the command line of a sdhci driver
|
||||
is changed, only the respective qgraph driver node has to be adjusted.
|
||||
|
||||
QGraph concepts
|
||||
---------------
|
||||
|
||||
The graph is composed by nodes that represent machines, drivers, tests
|
||||
and edges that define the relationships between them (``CONSUMES``, ``PRODUCES``, and
|
||||
``CONTAINS``).
|
||||
|
||||
Nodes
|
||||
~~~~~
|
||||
|
||||
A node can be of four types:
|
||||
|
||||
- **QNODE_MACHINE**: for example ``arm/raspi2b``
|
||||
- **QNODE_DRIVER**: for example ``generic-sdhci``
|
||||
- **QNODE_INTERFACE**: for example ``sdhci`` (interface for all ``-sdhci``
|
||||
drivers).
|
||||
An interface is not explicitly created, it will be automatically
|
||||
instantiated when a node consumes or produces it.
|
||||
An interface is simply a struct that abstracts the various drivers
|
||||
for the same type of device, and offers an API to the nodes that
|
||||
use it ("consume" relation in qgraph terms) that is implemented/backed up by the drivers that implement it ("produce" relation in qgraph terms).
|
||||
- **QNODE_TEST**: for example ``sdhci-test``. A test consumes an interface
|
||||
and tests the functions provided by it.
|
||||
|
||||
Notes for the nodes:
|
||||
|
||||
- QNODE_MACHINE: each machine struct must have a ``QGuestAllocator`` and
|
||||
implement ``get_driver()`` to return the allocator mapped to the interface
|
||||
"memory". The function can also return ``NULL`` if the allocator
|
||||
is not set.
|
||||
- QNODE_DRIVER: driver names must be unique, and machines and nodes
|
||||
planned to be "consumed" by other nodes must match QEMU
|
||||
drivers name, otherwise they won't be discovered
|
||||
|
||||
Edges
|
||||
~~~~~
|
||||
|
||||
An edge relation between two nodes (drivers or machines) ``X`` and ``Y`` can be:
|
||||
|
||||
- ``X CONSUMES Y``: ``Y`` can be plugged into ``X``
|
||||
- ``X PRODUCES Y``: ``X`` provides the interface ``Y``
|
||||
- ``X CONTAINS Y``: ``Y`` is part of ``X`` component
|
||||
|
||||
Execution steps
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
The basic framework steps are the following:
|
||||
|
||||
- All nodes and edges are created in their respective
|
||||
machine/driver/test files
|
||||
- The framework starts QEMU and asks for a list of available devices
|
||||
and machines (note that only machines and "consumed" nodes are mapped
|
||||
1:1 with QEMU devices)
|
||||
- The framework walks the graph starting from the available machines and
|
||||
performs a Depth First Search for tests
|
||||
- Once a test is found, the path is walked again and all drivers are
|
||||
allocated accordingly and the final interface is passed to the test
|
||||
- The test is executed
|
||||
- Unused objects are cleaned and the path discovery is continued
|
||||
|
||||
Depending on the QEMU binary used, only some drivers/machines will be
|
||||
available and only test that are reached by them will be executed.
|
||||
|
||||
Command line
|
||||
~~~~~~~~~~~~
|
||||
|
||||
Command line is built by using node names and optional arguments
|
||||
passed by the user when building the edges.
|
||||
|
||||
There are three types of command line arguments:
|
||||
|
||||
- ``in node`` : created from the node name. For example, machines will
|
||||
have ``-M <machine>`` to its command line, while devices
|
||||
``-device <device>``. It is automatically done by the framework.
|
||||
- ``after node`` : added as additional argument to the node name.
|
||||
This argument is added optionally when creating edges,
|
||||
by setting the parameter ``after_cmd_line`` and
|
||||
``extra_edge_opts`` in ``QOSGraphEdgeOptions``.
|
||||
The framework automatically adds
|
||||
a comma before ``extra_edge_opts``,
|
||||
because it is going to add attributes
|
||||
after the destination node pointed by
|
||||
the edge containing these options, and automatically
|
||||
adds a space before ``after_cmd_line``, because it
|
||||
adds an additional device, not an attribute.
|
||||
- ``before node`` : added as additional argument to the node name.
|
||||
This argument is added optionally when creating edges,
|
||||
by setting the parameter ``before_cmd_line`` in
|
||||
``QOSGraphEdgeOptions``. This attribute
|
||||
is going to add attributes before the destination node
|
||||
pointed by the edge containing these options. It is
|
||||
helpful to commands that are not node-representable,
|
||||
such as ``-fdsev`` or ``-netdev``.
|
||||
|
||||
While adding command line in edges is always used, not all nodes names are
|
||||
used in every path walk: this is because the contained or produced ones
|
||||
are already added by QEMU, so only nodes that "consumes" will be used to
|
||||
build the command line. Also, nodes that will have ``{ "abstract" : true }``
|
||||
as QMP attribute will loose their command line, since they are not proper
|
||||
devices to be added in QEMU.
|
||||
|
||||
Example::
|
||||
|
||||
QOSGraphEdgeOptions opts = {
|
||||
.before_cmd_line = "-drive id=drv0,if=none,file=null-co://,"
|
||||
"file.read-zeroes=on,format=raw",
|
||||
.after_cmd_line = "-device scsi-hd,bus=vs0.0,drive=drv0",
|
||||
|
||||
opts.extra_device_opts = "id=vs0";
|
||||
};
|
||||
|
||||
qos_node_create_driver("virtio-scsi-device",
|
||||
virtio_scsi_device_create);
|
||||
qos_node_consumes("virtio-scsi-device", "virtio-bus", &opts);
|
||||
|
||||
Will produce the following command line:
|
||||
``-drive id=drv0,if=none,file=null-co://, -device virtio-scsi-device,id=vs0 -device scsi-hd,bus=vs0.0,drive=drv0``
|
||||
|
||||
Troubleshooting unavailable tests
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
If there is no path from an available machine to a test then that test will be
|
||||
unavailable and won't execute. This can happen if a test or driver did not set
|
||||
up its qgraph node correctly. It can also happen if the necessary machine type
|
||||
or device is missing from the QEMU binary because it was compiled out or
|
||||
otherwise.
|
||||
|
||||
It is possible to troubleshoot unavailable tests by running::
|
||||
|
||||
$ QTEST_QEMU_BINARY=build/qemu-system-x86_64 build/tests/qtest/qos-test --verbose
|
||||
# ALL QGRAPH EDGES: {
|
||||
# src='virtio-net'
|
||||
# |-> dest='virtio-net-tests/vhost-user/multiqueue' type=2 (node=0x559142109e30)
|
||||
# |-> dest='virtio-net-tests/vhost-user/migrate' type=2 (node=0x559142109d00)
|
||||
# src='virtio-net-pci'
|
||||
# |-> dest='virtio-net' type=1 (node=0x55914210d740)
|
||||
# src='pci-bus'
|
||||
# |-> dest='virtio-net-pci' type=2 (node=0x55914210d880)
|
||||
# src='pci-bus-pc'
|
||||
# |-> dest='pci-bus' type=1 (node=0x559142103f40)
|
||||
# src='i440FX-pcihost'
|
||||
# |-> dest='pci-bus-pc' type=0 (node=0x55914210ac70)
|
||||
# src='x86_64/pc'
|
||||
# |-> dest='i440FX-pcihost' type=0 (node=0x5591421117f0)
|
||||
# src=''
|
||||
# |-> dest='x86_64/pc' type=0 (node=0x559142111600)
|
||||
# |-> dest='arm/raspi2b' type=0 (node=0x559142110740)
|
||||
...
|
||||
# }
|
||||
# ALL QGRAPH NODES: {
|
||||
# name='virtio-net-tests/announce-self' type=3 cmd_line='(null)' [available]
|
||||
# name='arm/raspi2b' type=0 cmd_line='-M raspi2b ' [UNAVAILABLE]
|
||||
...
|
||||
# }
|
||||
|
||||
The ``virtio-net-tests/announce-self`` test is listed as "available" in the
|
||||
"ALL QGRAPH NODES" output. This means the test will execute. We can follow the
|
||||
qgraph path in the "ALL QGRAPH EDGES" output as follows: '' -> 'x86_64/pc' ->
|
||||
'i440FX-pcihost' -> 'pci-bus-pc' -> 'pci-bus' -> 'virtio-net-pci' ->
|
||||
'virtio-net'. The root of the qgraph is '' and the depth first search begins
|
||||
there.
|
||||
|
||||
The ``arm/raspi2b`` machine node is listed as "UNAVAILABLE". Although it is
|
||||
reachable from the root via '' -> 'arm/raspi2b' the node is unavailable because
|
||||
the QEMU binary did not list it when queried by the framework. This is expected
|
||||
because we used the ``qemu-system-x86_64`` binary which does not support Arm
|
||||
machine types.
|
||||
|
||||
If a test is unexpectedly listed as "UNAVAILABLE", first check that the "ALL
|
||||
QGRAPH EDGES" output reports edge connectivity from the root ('') to the test.
|
||||
If there is no connectivity then the qgraph nodes were not set up correctly and
|
||||
the driver or test code is incorrect. If there is connectivity, check the
|
||||
availability of each node in the path in the "ALL QGRAPH NODES" output. The
|
||||
first unavailable node in the path is the reason why the test is unavailable.
|
||||
Typically this is because the QEMU binary lacks support for the necessary
|
||||
machine type or device.
|
||||
|
||||
Creating a new driver and its interface
|
||||
---------------------------------------
|
||||
|
||||
Here we continue the ``sdhci`` use case, with the following scenario:
|
||||
|
||||
- ``sdhci-test`` aims to test the ``read[q,w], writeq`` functions
|
||||
offered by the ``sdhci`` drivers.
|
||||
- The current ``sdhci`` device is supported by both ``x86_64/pc`` and Arm
|
||||
(in this example we focus on the ``arm-raspi2b``) machines.
|
||||
- QEMU offers 2 types of drivers: ``QSDHCI_MemoryMapped`` for Arm and
|
||||
``QSDHCI_PCI`` for ``x86_64/pc``. Both implement the
|
||||
``read[q,w], writeq`` functions.
|
||||
|
||||
In order to implement such scenario in qgraph, the test developer needs to:
|
||||
|
||||
- Create the ``x86_64/pc`` machine node. This machine uses the
|
||||
``pci-bus`` architecture so it ``contains`` a PCI driver,
|
||||
``pci-bus-pc``. The actual path is
|
||||
|
||||
``x86_64/pc --contains--> 1440FX-pcihost --contains-->
|
||||
pci-bus-pc --produces--> pci-bus``.
|
||||
|
||||
For the sake of this example,
|
||||
we do not focus on the PCI interface implementation.
|
||||
- Create the ``sdhci-pci`` driver node, representing ``QSDHCI_PCI``.
|
||||
The driver uses the PCI bus (and its API),
|
||||
so it must ``consume`` the ``pci-bus`` generic interface (which abstracts
|
||||
all the pci drivers available)
|
||||
|
||||
``sdhci-pci --consumes--> pci-bus``
|
||||
- Create an ``arm/raspi2b`` machine node. This machine ``contains``
|
||||
a ``generic-sdhci`` memory mapped ``sdhci`` driver node, representing
|
||||
``QSDHCI_MemoryMapped``.
|
||||
|
||||
``arm/raspi2b --contains--> generic-sdhci``
|
||||
- Create the ``sdhci`` interface node. This interface offers the
|
||||
functions that are shared by all ``sdhci`` devices.
|
||||
The interface is produced by ``sdhci-pci`` and ``generic-sdhci``,
|
||||
the available architecture-specific drivers.
|
||||
|
||||
``sdhci-pci --produces--> sdhci``
|
||||
|
||||
``generic-sdhci --produces--> sdhci``
|
||||
- Create the ``sdhci-test`` test node. The test ``consumes`` the
|
||||
``sdhci`` interface, using its API. It doesn't need to look at
|
||||
the supported machines or drivers.
|
||||
|
||||
``sdhci-test --consumes--> sdhci``
|
||||
|
||||
``arm-raspi2b`` machine, simplified from
|
||||
``tests/qtest/libqos/arm-raspi2-machine.c``::
|
||||
|
||||
#include "qgraph.h"
|
||||
|
||||
struct QRaspi2Machine {
|
||||
QOSGraphObject obj;
|
||||
QGuestAllocator alloc;
|
||||
QSDHCI_MemoryMapped sdhci;
|
||||
};
|
||||
|
||||
static void *raspi2_get_driver(void *object, const char *interface)
|
||||
{
|
||||
QRaspi2Machine *machine = object;
|
||||
if (!g_strcmp0(interface, "memory")) {
|
||||
return &machine->alloc;
|
||||
}
|
||||
|
||||
fprintf(stderr, "%s not present in arm/raspi2b\n", interface);
|
||||
g_assert_not_reached();
|
||||
}
|
||||
|
||||
static QOSGraphObject *raspi2_get_device(void *obj,
|
||||
const char *device)
|
||||
{
|
||||
QRaspi2Machine *machine = obj;
|
||||
if (!g_strcmp0(device, "generic-sdhci")) {
|
||||
return &machine->sdhci.obj;
|
||||
}
|
||||
|
||||
fprintf(stderr, "%s not present in arm/raspi2b\n", device);
|
||||
g_assert_not_reached();
|
||||
}
|
||||
|
||||
static void *qos_create_machine_arm_raspi2(QTestState *qts)
|
||||
{
|
||||
QRaspi2Machine *machine = g_new0(QRaspi2Machine, 1);
|
||||
|
||||
alloc_init(&machine->alloc, ...);
|
||||
|
||||
/* Get node(s) contained inside (CONTAINS) */
|
||||
machine->obj.get_device = raspi2_get_device;
|
||||
|
||||
/* Get node(s) produced (PRODUCES) */
|
||||
machine->obj.get_driver = raspi2_get_driver;
|
||||
|
||||
/* free the object */
|
||||
machine->obj.destructor = raspi2_destructor;
|
||||
qos_init_sdhci_mm(&machine->sdhci, ...);
|
||||
return &machine->obj;
|
||||
}
|
||||
|
||||
static void raspi2_register_nodes(void)
|
||||
{
|
||||
/* arm/raspi2b --contains--> generic-sdhci */
|
||||
qos_node_create_machine("arm/raspi2b",
|
||||
qos_create_machine_arm_raspi2);
|
||||
qos_node_contains("arm/raspi2b", "generic-sdhci", NULL);
|
||||
}
|
||||
|
||||
libqos_init(raspi2_register_nodes);
|
||||
|
||||
``x86_64/pc`` machine, simplified from
|
||||
``tests/qtest/libqos/x86_64_pc-machine.c``::
|
||||
|
||||
#include "qgraph.h"
|
||||
|
||||
struct i440FX_pcihost {
|
||||
QOSGraphObject obj;
|
||||
QPCIBusPC pci;
|
||||
};
|
||||
|
||||
struct QX86PCMachine {
|
||||
QOSGraphObject obj;
|
||||
QGuestAllocator alloc;
|
||||
i440FX_pcihost bridge;
|
||||
};
|
||||
|
||||
/* i440FX_pcihost */
|
||||
|
||||
static QOSGraphObject *i440FX_host_get_device(void *obj,
|
||||
const char *device)
|
||||
{
|
||||
i440FX_pcihost *host = obj;
|
||||
if (!g_strcmp0(device, "pci-bus-pc")) {
|
||||
return &host->pci.obj;
|
||||
}
|
||||
fprintf(stderr, "%s not present in i440FX-pcihost\n", device);
|
||||
g_assert_not_reached();
|
||||
}
|
||||
|
||||
/* x86_64/pc machine */
|
||||
|
||||
static void *pc_get_driver(void *object, const char *interface)
|
||||
{
|
||||
QX86PCMachine *machine = object;
|
||||
if (!g_strcmp0(interface, "memory")) {
|
||||
return &machine->alloc;
|
||||
}
|
||||
|
||||
fprintf(stderr, "%s not present in x86_64/pc\n", interface);
|
||||
g_assert_not_reached();
|
||||
}
|
||||
|
||||
static QOSGraphObject *pc_get_device(void *obj, const char *device)
|
||||
{
|
||||
QX86PCMachine *machine = obj;
|
||||
if (!g_strcmp0(device, "i440FX-pcihost")) {
|
||||
return &machine->bridge.obj;
|
||||
}
|
||||
|
||||
fprintf(stderr, "%s not present in x86_64/pc\n", device);
|
||||
g_assert_not_reached();
|
||||
}
|
||||
|
||||
static void *qos_create_machine_pc(QTestState *qts)
|
||||
{
|
||||
QX86PCMachine *machine = g_new0(QX86PCMachine, 1);
|
||||
|
||||
/* Get node(s) contained inside (CONTAINS) */
|
||||
machine->obj.get_device = pc_get_device;
|
||||
|
||||
/* Get node(s) produced (PRODUCES) */
|
||||
machine->obj.get_driver = pc_get_driver;
|
||||
|
||||
/* free the object */
|
||||
machine->obj.destructor = pc_destructor;
|
||||
pc_alloc_init(&machine->alloc, qts, ALLOC_NO_FLAGS);
|
||||
|
||||
/* Get node(s) contained inside (CONTAINS) */
|
||||
machine->bridge.obj.get_device = i440FX_host_get_device;
|
||||
|
||||
return &machine->obj;
|
||||
}
|
||||
|
||||
static void pc_machine_register_nodes(void)
|
||||
{
|
||||
/* x86_64/pc --contains--> 1440FX-pcihost --contains-->
|
||||
* pci-bus-pc [--produces--> pci-bus (in pci.h)] */
|
||||
qos_node_create_machine("x86_64/pc", qos_create_machine_pc);
|
||||
qos_node_contains("x86_64/pc", "i440FX-pcihost", NULL);
|
||||
|
||||
/* contained drivers don't need a constructor,
|
||||
* they will be init by the parent */
|
||||
qos_node_create_driver("i440FX-pcihost", NULL);
|
||||
qos_node_contains("i440FX-pcihost", "pci-bus-pc", NULL);
|
||||
}
|
||||
|
||||
libqos_init(pc_machine_register_nodes);
|
||||
|
||||
``sdhci`` taken from ``tests/qtest/libqos/sdhci.c``::
|
||||
|
||||
/* Interface node, offers the sdhci API */
|
||||
struct QSDHCI {
|
||||
uint16_t (*readw)(QSDHCI *s, uint32_t reg);
|
||||
uint64_t (*readq)(QSDHCI *s, uint32_t reg);
|
||||
void (*writeq)(QSDHCI *s, uint32_t reg, uint64_t val);
|
||||
/* other fields */
|
||||
};
|
||||
|
||||
/* Memory Mapped implementation of QSDHCI */
|
||||
struct QSDHCI_MemoryMapped {
|
||||
QOSGraphObject obj;
|
||||
QSDHCI sdhci;
|
||||
/* other driver-specific fields */
|
||||
};
|
||||
|
||||
/* PCI implementation of QSDHCI */
|
||||
struct QSDHCI_PCI {
|
||||
QOSGraphObject obj;
|
||||
QSDHCI sdhci;
|
||||
/* other driver-specific fields */
|
||||
};
|
||||
|
||||
/* Memory mapped implementation of QSDHCI */
|
||||
|
||||
static void *sdhci_mm_get_driver(void *obj, const char *interface)
|
||||
{
|
||||
QSDHCI_MemoryMapped *smm = obj;
|
||||
if (!g_strcmp0(interface, "sdhci")) {
|
||||
return &smm->sdhci;
|
||||
}
|
||||
fprintf(stderr, "%s not present in generic-sdhci\n", interface);
|
||||
g_assert_not_reached();
|
||||
}
|
||||
|
||||
void qos_init_sdhci_mm(QSDHCI_MemoryMapped *sdhci, QTestState *qts,
|
||||
uint32_t addr, QSDHCIProperties *common)
|
||||
{
|
||||
/* Get node contained inside (CONTAINS) */
|
||||
sdhci->obj.get_driver = sdhci_mm_get_driver;
|
||||
|
||||
/* SDHCI interface API */
|
||||
sdhci->sdhci.readw = sdhci_mm_readw;
|
||||
sdhci->sdhci.readq = sdhci_mm_readq;
|
||||
sdhci->sdhci.writeq = sdhci_mm_writeq;
|
||||
sdhci->qts = qts;
|
||||
}
|
||||
|
||||
/* PCI implementation of QSDHCI */
|
||||
|
||||
static void *sdhci_pci_get_driver(void *object,
|
||||
const char *interface)
|
||||
{
|
||||
QSDHCI_PCI *spci = object;
|
||||
if (!g_strcmp0(interface, "sdhci")) {
|
||||
return &spci->sdhci;
|
||||
}
|
||||
|
||||
fprintf(stderr, "%s not present in sdhci-pci\n", interface);
|
||||
g_assert_not_reached();
|
||||
}
|
||||
|
||||
static void *sdhci_pci_create(void *pci_bus,
|
||||
QGuestAllocator *alloc,
|
||||
void *addr)
|
||||
{
|
||||
QSDHCI_PCI *spci = g_new0(QSDHCI_PCI, 1);
|
||||
QPCIBus *bus = pci_bus;
|
||||
uint64_t barsize;
|
||||
|
||||
qpci_device_init(&spci->dev, bus, addr);
|
||||
|
||||
/* SDHCI interface API */
|
||||
spci->sdhci.readw = sdhci_pci_readw;
|
||||
spci->sdhci.readq = sdhci_pci_readq;
|
||||
spci->sdhci.writeq = sdhci_pci_writeq;
|
||||
|
||||
/* Get node(s) produced (PRODUCES) */
|
||||
spci->obj.get_driver = sdhci_pci_get_driver;
|
||||
|
||||
spci->obj.start_hw = sdhci_pci_start_hw;
|
||||
spci->obj.destructor = sdhci_destructor;
|
||||
return &spci->obj;
|
||||
}
|
||||
|
||||
static void qsdhci_register_nodes(void)
|
||||
{
|
||||
QOSGraphEdgeOptions opts = {
|
||||
.extra_device_opts = "addr=04.0",
|
||||
};
|
||||
|
||||
/* generic-sdhci */
|
||||
/* generic-sdhci --produces--> sdhci */
|
||||
qos_node_create_driver("generic-sdhci", NULL);
|
||||
qos_node_produces("generic-sdhci", "sdhci");
|
||||
|
||||
/* sdhci-pci */
|
||||
/* sdhci-pci --produces--> sdhci
|
||||
* sdhci-pci --consumes--> pci-bus */
|
||||
qos_node_create_driver("sdhci-pci", sdhci_pci_create);
|
||||
qos_node_produces("sdhci-pci", "sdhci");
|
||||
qos_node_consumes("sdhci-pci", "pci-bus", &opts);
|
||||
}
|
||||
|
||||
libqos_init(qsdhci_register_nodes);
|
||||
|
||||
In the above example, all possible types of relations are created::
|
||||
|
||||
x86_64/pc --contains--> 1440FX-pcihost --contains--> pci-bus-pc
|
||||
|
|
||||
sdhci-pci --consumes--> pci-bus <--produces--+
|
||||
|
|
||||
+--produces--+
|
||||
|
|
||||
v
|
||||
sdhci
|
||||
^
|
||||
|
|
||||
+--produces-- +
|
||||
|
|
||||
arm/raspi2b --contains--> generic-sdhci
|
||||
|
||||
or inverting the consumes edge in consumed_by::
|
||||
|
||||
x86_64/pc --contains--> 1440FX-pcihost --contains--> pci-bus-pc
|
||||
|
|
||||
sdhci-pci <--consumed by-- pci-bus <--produces--+
|
||||
|
|
||||
+--produces--+
|
||||
|
|
||||
v
|
||||
sdhci
|
||||
^
|
||||
|
|
||||
+--produces-- +
|
||||
|
|
||||
arm/raspi2b --contains--> generic-sdhci
|
||||
|
||||
Adding a new test
|
||||
-----------------
|
||||
|
||||
Given the above setup, adding a new test is very simple.
|
||||
``sdhci-test``, taken from ``tests/qtest/sdhci-test.c``::
|
||||
|
||||
static void check_capab_sdma(QSDHCI *s, bool supported)
|
||||
{
|
||||
uint64_t capab, capab_sdma;
|
||||
|
||||
capab = s->readq(s, SDHC_CAPAB);
|
||||
capab_sdma = FIELD_EX64(capab, SDHC_CAPAB, SDMA);
|
||||
g_assert_cmpuint(capab_sdma, ==, supported);
|
||||
}
|
||||
|
||||
static void test_registers(void *obj, void *data,
|
||||
QGuestAllocator *alloc)
|
||||
{
|
||||
QSDHCI *s = obj;
|
||||
|
||||
/* example test */
|
||||
check_capab_sdma(s, s->props.capab.sdma);
|
||||
}
|
||||
|
||||
static void register_sdhci_test(void)
|
||||
{
|
||||
/* sdhci-test --consumes--> sdhci */
|
||||
qos_add_test("registers", "sdhci", test_registers, NULL);
|
||||
}
|
||||
|
||||
libqos_init(register_sdhci_test);
|
||||
|
||||
Here a new test is created, consuming ``sdhci`` interface node
|
||||
and creating a valid path from both machines to a test.
|
||||
Final graph will be like this::
|
||||
|
||||
x86_64/pc --contains--> 1440FX-pcihost --contains--> pci-bus-pc
|
||||
|
|
||||
sdhci-pci --consumes--> pci-bus <--produces--+
|
||||
|
|
||||
+--produces--+
|
||||
|
|
||||
v
|
||||
sdhci <--consumes-- sdhci-test
|
||||
^
|
||||
|
|
||||
+--produces-- +
|
||||
|
|
||||
arm/raspi2b --contains--> generic-sdhci
|
||||
|
||||
or inverting the consumes edge in consumed_by::
|
||||
|
||||
x86_64/pc --contains--> 1440FX-pcihost --contains--> pci-bus-pc
|
||||
|
|
||||
sdhci-pci <--consumed by-- pci-bus <--produces--+
|
||||
|
|
||||
+--produces--+
|
||||
|
|
||||
v
|
||||
sdhci --consumed by--> sdhci-test
|
||||
^
|
||||
|
|
||||
+--produces-- +
|
||||
|
|
||||
arm/raspi2b --contains--> generic-sdhci
|
||||
|
||||
Assuming there the binary is
|
||||
``QTEST_QEMU_BINARY=./qemu-system-x86_64``
|
||||
a valid test path will be:
|
||||
``/x86_64/pc/1440FX-pcihost/pci-bus-pc/pci-bus/sdhci-pc/sdhci/sdhci-test``
|
||||
|
||||
and for the binary ``QTEST_QEMU_BINARY=./qemu-system-arm``:
|
||||
|
||||
``/arm/raspi2b/generic-sdhci/sdhci/sdhci-test``
|
||||
|
||||
Additional examples are also in ``test-qgraph.c``
|
||||
|
||||
Qgraph API reference
|
||||
--------------------
|
||||
|
||||
.. kernel-doc:: tests/qtest/libqos/qgraph.h
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user