ShuriZma
/
xemu
mirror of https://github.com/xemu-project/xemu.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

docs/devel: Align VFIO migration docs to v2 protocol 

Now that VFIO migration protocol v2 has been implemented and v1 protocol
has been removed, update the documentation according to v2 protocol.

Signed-off-by: Avihai Horon <avihaih@nvidia.com>
Reviewed-by: Cédric Le Goater <clg@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Link: https://lore.kernel.org/r/20230216143630.25610-12-avihaih@nvidia.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
This commit is contained in:
Avihai Horon 2023-02-16 16:36:30 +02:00 committed by Alex Williamson
parent 48e4d8289f
commit 2b0ab9e9d6
1 changed files with 34 additions and 38 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

72
docs/devel/vfio-migration.rst
View File

@ -7,46 +7,43 @@ the guest is running on source host and restoring this saved state on the
destination host. This document details how saving and restoring of VFIO destination host. This document details how saving and restoring of VFIO
devices is done in QEMU. devices is done in QEMU.
Migration of VFIO devices consists of two phases: the optional pre-copy phase, Migration of VFIO devices currently consists of a single stop-and-copy phase.
and the stop-and-copy phase. The pre-copy phase is iterative and allows to During the stop-and-copy phase the guest is stopped and the entire VFIO device
accommodate VFIO devices that have a large amount of data that needs to be data is transferred to the destination.
transferred. The iterative pre-copy phase of migration allows for the guest to
continue whilst the VFIO device state is transferred to the destination, this The pre-copy phase of migration is currently not supported for VFIO devices.
helps to reduce the total downtime of the VM. VFIO devices can choose to skip Support for VFIO pre-copy will be added later on.
the pre-copy phase of migration by returning pending_bytes as zero during the
pre-copy phase. Note that currently VFIO migration is supported only for a single device. This
is due to VFIO migration's lack of P2P support. However, P2P support is planned
to be added later on.
A detailed description of the UAPI for VFIO device migration can be found in A detailed description of the UAPI for VFIO device migration can be found in
the comment for the ``vfio_device_migration_info`` structure in the header the comment for the ``vfio_device_mig_state`` structure in the header file
file linux-headers/linux/vfio.h. linux-headers/linux/vfio.h.
VFIO implements the device hooks for the iterative approach as follows: VFIO implements the device hooks for the iterative approach as follows:
* A ``save_setup`` function that sets up the migration region and sets _SAVING * A ``save_setup`` function that sets up migration on the source.
flag in the VFIO device state.
* A ``load_setup`` function that sets up the migration region on the * A ``load_setup`` function that sets the VFIO device on the destination in
destination and sets _RESUMING flag in the VFIO device state. _RESUMING state.
* A ``state_pending_exact`` function that reads pending_bytes from the vendor * A ``state_pending_exact`` function that reads pending_bytes from the vendor
driver, which indicates the amount of data that the vendor driver has yet to driver, which indicates the amount of data that the vendor driver has yet to
save for the VFIO device. save for the VFIO device.
* A ``save_live_iterate`` function that reads the VFIO device's data from the
vendor driver through the migration region during iterative phase.
* A ``save_state`` function to save the device config space if it is present. * A ``save_state`` function to save the device config space if it is present.
* A ``save_live_complete_precopy`` function that resets _RUNNING flag from the * A ``save_live_complete_precopy`` function that sets the VFIO device in
VFIO device state and iteratively copies the remaining data for the VFIO _STOP_COPY state and iteratively copies the data for the VFIO device until
device until the vendor driver indicates that no data remains (pending bytes the vendor driver indicates that no data remains.
is zero).
* A ``load_state`` function that loads the config section and the data * A ``load_state`` function that loads the config section and the data
sections that are generated by the save functions above sections that are generated by the save functions above.
* ``cleanup`` functions for both save and load that perform any migration * ``cleanup`` functions for both save and load that perform any migration
related cleanup, including unmapping the migration region related cleanup.
The VFIO migration code uses a VM state change handler to change the VFIO The VFIO migration code uses a VM state change handler to change the VFIO
@ -71,13 +68,13 @@ tracking can identify dirtied pages, but any page pinned by the vendor driver
can also be written by the device. There is currently no device or IOMMU can also be written by the device. There is currently no device or IOMMU
support for dirty page tracking in hardware. support for dirty page tracking in hardware.
By default, dirty pages are tracked when the device is in pre-copy as well as By default, dirty pages are tracked during pre-copy as well as stop-and-copy
stop-and-copy phase. So, a page pinned by the vendor driver will be copied to phase. So, a page pinned by the vendor driver will be copied to the destination
the destination in both phases. Copying dirty pages in pre-copy phase helps in both phases. Copying dirty pages in pre-copy phase helps QEMU to predict if
QEMU to predict if it can achieve its downtime tolerances. If QEMU during it can achieve its downtime tolerances. If QEMU during pre-copy phase keeps
pre-copy phase keeps finding dirty pages continuously, then it understands finding dirty pages continuously, then it understands that even in stop-and-copy
that even in stop-and-copy phase, it is likely to find dirty pages and can phase, it is likely to find dirty pages and can predict the downtime
predict the downtime accordingly. accordingly.
QEMU also provides a per device opt-out option ``pre-copy-dirty-page-tracking`` QEMU also provides a per device opt-out option ``pre-copy-dirty-page-tracking``
which disables querying the dirty bitmap during pre-copy phase. If it is set to which disables querying the dirty bitmap during pre-copy phase. If it is set to
@ -111,23 +108,22 @@ Live migration save path
| |
migrate_init spawns migration_thread migrate_init spawns migration_thread
Migration thread then calls each device's .save_setup() Migration thread then calls each device's .save_setup()
(RUNNING, _SETUP, _RUNNING|_SAVING) (RUNNING, _SETUP, _RUNNING)
| |
(RUNNING, _ACTIVE, _RUNNING|_SAVING) (RUNNING, _ACTIVE, _RUNNING)
If device is active, get pending_bytes by .state_pending_exact() If device is active, get pending_bytes by .state_pending_exact()
If total pending_bytes >= threshold_size, call .save_live_iterate() If total pending_bytes >= threshold_size, call .save_live_iterate()
Data of VFIO device for pre-copy phase is copied
Iterate till total pending bytes converge and are less than threshold Iterate till total pending bytes converge and are less than threshold
| |
On migration completion, vCPU stops and calls .save_live_complete_precopy for On migration completion, vCPU stops and calls .save_live_complete_precopy for
each active device. The VFIO device is then transitioned into _SAVING state each active device. The VFIO device is then transitioned into _STOP_COPY state
(FINISH_MIGRATE, _DEVICE, _SAVING) (FINISH_MIGRATE, _DEVICE, _STOP_COPY)
| |
For the VFIO device, iterate in .save_live_complete_precopy until For the VFIO device, iterate in .save_live_complete_precopy until
pending data is 0 pending data is 0
(FINISH_MIGRATE, _DEVICE, _STOPPED) (FINISH_MIGRATE, _DEVICE, _STOP)
| |
(FINISH_MIGRATE, _COMPLETED, _STOPPED) (FINISH_MIGRATE, _COMPLETED, _STOP)
Migraton thread schedules cleanup bottom half and exits Migraton thread schedules cleanup bottom half and exits
Live migration resume path Live migration resume path
@ -136,7 +132,7 @@ Live migration resume path
:: ::
Incoming migration calls .load_setup for each device Incoming migration calls .load_setup for each device
(RESTORE_VM, _ACTIVE, _STOPPED) (RESTORE_VM, _ACTIVE, _STOP)
| |
For each device, .load_state is called for that device section data For each device, .load_state is called for that device section data
(RESTORE_VM, _ACTIVE, _RESUMING) (RESTORE_VM, _ACTIVE, _RESUMING)