From 3ae3fcfae533086d3e210589c10f317717c9d789 Mon Sep 17 00:00:00 2001 From: Vladimir Sementsov-Ogievskiy Date: Fri, 31 Jan 2020 17:22:18 +0300 Subject: [PATCH 01/18] docs: improve qcow2 spec about extending image header Make it more obvious how to add new fields to the version 3 header and how to interpret them. The specification is adjusted so that for new defined optional fields: 1. Software may support some of these optional fields and ignore the others, which means that features may be backported to downstream Qemu independently. 2. If we want to add incompatible field (or a field, for which some of its values would be incompatible), it must be accompanied by incompatible feature bit. Also the concept of "default is zero" is clarified, as it's strange to say that the value of the field is assumed to be zero for the software version which don't know about the field at all and don't know how to treat it be it zero or not. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake Message-Id: <20200131142219.3264-2-vsementsov@virtuozzo.com> Reviewed-by: Alberto Garcia [mreitz: s/some its/some of its/] Signed-off-by: Max Reitz --- docs/interop/qcow2.txt | 45 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 42 insertions(+), 3 deletions(-) diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt index af5711e533..823cc266e0 100644 --- a/docs/interop/qcow2.txt +++ b/docs/interop/qcow2.txt @@ -79,9 +79,9 @@ The first cluster of a qcow2 image contains the file header: Offset into the image file at which the snapshot table starts. Must be aligned to a cluster boundary. -If the version is 3 or higher, the header has the following additional fields. -For version 2, the values are assumed to be zero, unless specified otherwise -in the description of a field. +For version 2, the header is exactly 72 bytes in length, and finishes here. +For version 3 or higher, the header length is at least 104 bytes, including +the next fields through header_length. 72 - 79: incompatible_features Bitmask of incompatible features. An implementation must @@ -164,6 +164,45 @@ in the description of a field. 100 - 103: header_length Length of the header structure in bytes. For version 2 images, the length is always assumed to be 72 bytes. + For version 3 it's at least 104 bytes and must be a multiple + of 8. + + +=== Additional fields (version 3 and higher) === + +In general, these fields are optional and may be safely ignored by the software, +as well as filled by zeros (which is equal to field absence), if software needs +to set field B, but does not care about field A which precedes B. More +formally, additional fields have the following compatibility rules: + +1. If the value of the additional field must not be ignored for correct +handling of the file, it will be accompanied by a corresponding incompatible +feature bit. + +2. If there are no unrecognized incompatible feature bits set, an unknown +additional field may be safely ignored other than preserving its value when +rewriting the image header. + +3. An explicit value of 0 will have the same behavior as when the field is not +present*, if not altered by a specific incompatible bit. + +*. A field is considered not present when header_length is less than or equal +to the field's offset. Also, all additional fields are not present for +version 2. + + < ... No additional fields in the header currently ... > + + +=== Header padding === + +@header_length must be a multiple of 8, which means that if the end of the last +additional field is not aligned, some padding is needed. This padding must be +zeroed, so that if some existing (or future) additional field will fall into +the padding, it will be interpreted accordingly to point [3.] of the previous +paragraph, i.e. in the same manner as when this field is not present. + + +=== Header extensions === Directly after the image header, optional sections called header extensions can be stored. Each extension has a structure like the following: From 66fcbca5a7202d2ff2973b61a30f915204917eb7 Mon Sep 17 00:00:00 2001 From: Vladimir Sementsov-Ogievskiy Date: Fri, 31 Jan 2020 17:22:19 +0300 Subject: [PATCH 02/18] docs: qcow2: introduce compression type feature The patch adds a new additional field to the qcow2 header: compression_type, which specifies compression type. If field is absent or zero, default compression type is set: ZLIB, which corresponds to current behavior. New compression type (ZSTD) is to be added in further commit. Suggested-by: Denis Plotnikov Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20200131142219.3264-3-vsementsov@virtuozzo.com> [mreitz: s/Bits 3-63: Reserved/Bits 4-63: Reserved/] Signed-off-by: Max Reitz --- docs/interop/qcow2.txt | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt index 823cc266e0..5597e24474 100644 --- a/docs/interop/qcow2.txt +++ b/docs/interop/qcow2.txt @@ -109,7 +109,12 @@ the next fields through header_length. An External Data File Name header extension may be present if this bit is set. - Bits 3-63: Reserved (set to 0) + Bit 3: Compression type bit. If this bit is set, + a non-default compression is used for compressed + clusters. The compression_type field must be + present and not zero. + + Bits 4-63: Reserved (set to 0) 80 - 87: compatible_features Bitmask of compatible features. An implementation can @@ -190,7 +195,19 @@ present*, if not altered by a specific incompatible bit. to the field's offset. Also, all additional fields are not present for version 2. - < ... No additional fields in the header currently ... > + 104: compression_type + + Defines the compression method used for compressed clusters. + All compressed clusters in an image use the same compression + type. + + If the incompatible bit "Compression type" is set: the field + must be present and non-zero (which means non-zlib + compression type). Otherwise, this field must not be present + or must be zero (which means zlib). + + Available compression type values: + 0: zlib === Header padding === From 6756696e38117f9e7c769dd33b339f5ab71f7d47 Mon Sep 17 00:00:00 2001 From: Thomas Huth Date: Wed, 29 Jan 2020 15:17:51 +0100 Subject: [PATCH 03/18] iotests: Remove the superfluous 2nd check for the availability of quorum Commit d9df28e7b07 ("iotests: check whitelisted formats") added the modern @iotests.skip_if_unsupported() to the functions in this test, so we don't need the old explicit test here anymore. Signed-off-by: Thomas Huth Message-Id: <20200129141751.32652-1-thuth@redhat.com> Reviewed-by: Alberto Garcia Reviewed-by: Andrey Shinkevich Tested-by: Andrey Shinkevich Signed-off-by: Max Reitz --- tests/qemu-iotests/139 | 3 --- 1 file changed, 3 deletions(-) diff --git a/tests/qemu-iotests/139 b/tests/qemu-iotests/139 index 6b1a444364..7120d3142b 100755 --- a/tests/qemu-iotests/139 +++ b/tests/qemu-iotests/139 @@ -344,9 +344,6 @@ class TestBlockdevDel(iotests.QMPTestCase): @iotests.skip_if_unsupported(['quorum']) def testQuorum(self): - if not iotests.supports_quorum(): - return - self.addQuorum('quorum0', 'node0', 'node1') # We cannot remove the children of a Quorum device self.delBlockDriverState('node0', expect_error = True) From b0c4cf21b0f26533e4aa6ca1930b2c9f52371bfa Mon Sep 17 00:00:00 2001 From: Max Reitz Date: Thu, 6 Feb 2020 14:08:12 +0100 Subject: [PATCH 04/18] iotests/147: Fix drive parameters 8dff69b94 added an aio parameter to the drive parameter but forgot to add a comma before, thus breaking the test. Fix it again. Fixes: 8dff69b9415b4287e900358744b732195e1ab2e2 Signed-off-by: Max Reitz Message-Id: <20200206130812.612960-1-mreitz@redhat.com> Reviewed-by: Eric Blake Tested-by: Eric Blake Signed-off-by: Max Reitz --- tests/qemu-iotests/147 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/qemu-iotests/147 b/tests/qemu-iotests/147 index f4b0a11dba..d7a9f31089 100755 --- a/tests/qemu-iotests/147 +++ b/tests/qemu-iotests/147 @@ -134,7 +134,7 @@ class BuiltinNBD(NBDBlockdevAddBase): self.server.add_drive_raw('if=none,id=nbd-export,' + 'file=%s,' % test_img + 'format=%s,' % imgfmt + - 'cache=%s' % cachemode + + 'cache=%s,' % cachemode + 'aio=%s' % aiomode) self.server.launch() From facda5443f5a8676fb635b82ac1046ac6b6a67ce Mon Sep 17 00:00:00 2001 From: Peter Krempa Date: Mon, 20 Jan 2020 09:50:49 +0100 Subject: [PATCH 05/18] qapi: Allow getting flat output from 'query-named-block-nodes' When a management application manages node names there's no reason to recurse into backing images in the output of query-named-block-nodes. Add a parameter to the command which will return just the top level structs. Signed-off-by: Peter Krempa Message-Id: <4470f8c779abc404dcf65e375db195cd91a80651.1579509782.git.pkrempa@redhat.com> Reviewed-by: Eric Blake [mreitz: Fixed coding style] Signed-off-by: Max Reitz --- block.c | 5 +++-- block/qapi.c | 11 +++++++++-- blockdev.c | 8 ++++++-- include/block/block.h | 2 +- include/block/qapi.h | 4 +++- monitor/hmp-cmds.c | 2 +- qapi/block-core.json | 7 ++++++- 7 files changed, 29 insertions(+), 10 deletions(-) diff --git a/block.c b/block.c index 946e3c896e..08372ced26 100644 --- a/block.c +++ b/block.c @@ -4807,14 +4807,15 @@ BlockDriverState *bdrv_find_node(const char *node_name) } /* Put this QMP function here so it can access the static graph_bdrv_states. */ -BlockDeviceInfoList *bdrv_named_nodes_list(Error **errp) +BlockDeviceInfoList *bdrv_named_nodes_list(bool flat, + Error **errp) { BlockDeviceInfoList *list, *entry; BlockDriverState *bs; list = NULL; QTAILQ_FOREACH(bs, &graph_bdrv_states, node_list) { - BlockDeviceInfo *info = bdrv_block_device_info(NULL, bs, errp); + BlockDeviceInfo *info = bdrv_block_device_info(NULL, bs, flat, errp); if (!info) { qapi_free_BlockDeviceInfoList(list); return NULL; diff --git a/block/qapi.c b/block/qapi.c index 9a5d0c9b27..3f09477cc5 100644 --- a/block/qapi.c +++ b/block/qapi.c @@ -42,7 +42,9 @@ #include "qemu/cutils.h" BlockDeviceInfo *bdrv_block_device_info(BlockBackend *blk, - BlockDriverState *bs, Error **errp) + BlockDriverState *bs, + bool flat, + Error **errp) { ImageInfo **p_image_info; BlockDriverState *bs0; @@ -156,6 +158,11 @@ BlockDeviceInfo *bdrv_block_device_info(BlockBackend *blk, return NULL; } + /* stop gathering data for flat output */ + if (flat) { + break; + } + if (bs0->drv && bs0->backing) { info->backing_file_depth++; bs0 = bs0->backing->bs; @@ -389,7 +396,7 @@ static void bdrv_query_info(BlockBackend *blk, BlockInfo **p_info, if (bs && bs->drv) { info->has_inserted = true; - info->inserted = bdrv_block_device_info(blk, bs, errp); + info->inserted = bdrv_block_device_info(blk, bs, false, errp); if (info->inserted == NULL) { goto err; } diff --git a/blockdev.c b/blockdev.c index 45de0ba37e..011dcfec27 100644 --- a/blockdev.c +++ b/blockdev.c @@ -3734,9 +3734,13 @@ void qmp_drive_backup(DriveBackup *backup, Error **errp) blockdev_do_action(&action, errp); } -BlockDeviceInfoList *qmp_query_named_block_nodes(Error **errp) +BlockDeviceInfoList *qmp_query_named_block_nodes(bool has_flat, + bool flat, + Error **errp) { - return bdrv_named_nodes_list(errp); + bool return_flat = has_flat && flat; + + return bdrv_named_nodes_list(return_flat, errp); } XDbgBlockGraph *qmp_x_debug_query_block_graph(Error **errp) diff --git a/include/block/block.h b/include/block/block.h index 314ce63ed9..cd6b5b95aa 100644 --- a/include/block/block.h +++ b/include/block/block.h @@ -454,7 +454,7 @@ void bdrv_lock_medium(BlockDriverState *bs, bool locked); void bdrv_eject(BlockDriverState *bs, bool eject_flag); const char *bdrv_get_format_name(BlockDriverState *bs); BlockDriverState *bdrv_find_node(const char *node_name); -BlockDeviceInfoList *bdrv_named_nodes_list(Error **errp); +BlockDeviceInfoList *bdrv_named_nodes_list(bool flat, Error **errp); XDbgBlockGraph *bdrv_get_xdbg_block_graph(Error **errp); BlockDriverState *bdrv_lookup_bs(const char *device, const char *node_name, diff --git a/include/block/qapi.h b/include/block/qapi.h index cd9410dee3..22c7807c89 100644 --- a/include/block/qapi.h +++ b/include/block/qapi.h @@ -29,7 +29,9 @@ #include "block/snapshot.h" BlockDeviceInfo *bdrv_block_device_info(BlockBackend *blk, - BlockDriverState *bs, Error **errp); + BlockDriverState *bs, + bool flat, + Error **errp); int bdrv_query_snapshot_info_list(BlockDriverState *bs, SnapshotInfoList **p_list, Error **errp); diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c index b237613e0d..53bc3f76c4 100644 --- a/monitor/hmp-cmds.c +++ b/monitor/hmp-cmds.c @@ -620,7 +620,7 @@ void hmp_info_block(Monitor *mon, const QDict *qdict) } /* Print node information */ - blockdev_list = qmp_query_named_block_nodes(NULL); + blockdev_list = qmp_query_named_block_nodes(false, false, NULL); for (blockdev = blockdev_list; blockdev; blockdev = blockdev->next) { assert(blockdev->value->has_node_name); if (device && strcmp(device, blockdev->value->node_name)) { diff --git a/qapi/block-core.json b/qapi/block-core.json index 37d7ea7295..85e27bb61f 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -1758,6 +1758,9 @@ # # Get the named block driver list # +# @flat: Omit the nested data about backing image ("backing-image" key) if true. +# Default is false (Since 5.0) +# # Returns: the list of BlockDeviceInfo # # Since: 2.0 @@ -1811,7 +1814,9 @@ # } } ] } # ## -{ 'command': 'query-named-block-nodes', 'returns': [ 'BlockDeviceInfo' ] } +{ 'command': 'query-named-block-nodes', + 'returns': [ 'BlockDeviceInfo' ], + 'data': { '*flat': 'bool' } } ## # @XDbgBlockGraphNodeType: From 168468fe19c872d31d17581e6a6cf7b69bd105b1 Mon Sep 17 00:00:00 2001 From: David Edmondson Date: Wed, 5 Feb 2020 11:02:48 +0000 Subject: [PATCH 06/18] qemu-img: Add --target-is-zero to convert In many cases the target of a convert operation is a newly provisioned target that the user knows is blank (reads as zero). In this situation there is no requirement for qemu-img to wastefully zero out the entire device. Add a new option, --target-is-zero, allowing the user to indicate that an existing target device will return zeros for all reads. Signed-off-by: David Edmondson Message-Id: <20200205110248.2009589-2-david.edmondson@oracle.com> Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake Signed-off-by: Max Reitz --- docs/interop/qemu-img.rst | 9 ++++++++- qemu-img-cmds.hx | 4 ++-- qemu-img.c | 26 +++++++++++++++++++++++--- 3 files changed, 33 insertions(+), 6 deletions(-) diff --git a/docs/interop/qemu-img.rst b/docs/interop/qemu-img.rst index 42e4451db4..5f40137c10 100644 --- a/docs/interop/qemu-img.rst +++ b/docs/interop/qemu-img.rst @@ -214,6 +214,13 @@ Parameters to convert subcommand: will still be printed. Areas that cannot be read from the source will be treated as containing only zeroes. +.. option:: --target-is-zero + + Assume that reading the destination image will always return + zeros. This parameter is mutually exclusive with a destination image + that has a backing file. It is required to also use the ``-n`` + parameter to skip image creation. + Parameters to dd subcommand: .. program:: qemu-img-dd @@ -366,7 +373,7 @@ Command description: 4 Error on reading data -.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME +.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME Convert the disk image *FILENAME* or a snapshot *SNAPSHOT_PARAM* to disk image *OUTPUT_FILENAME* using format *OUTPUT_FMT*. It can diff --git a/qemu-img-cmds.hx b/qemu-img-cmds.hx index d7fbc6b1f4..c9c54de1df 100644 --- a/qemu-img-cmds.hx +++ b/qemu-img-cmds.hx @@ -39,9 +39,9 @@ SRST ERST DEF("convert", img_convert, - "convert [--object objectdef] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename") + "convert [--object objectdef] [--image-opts] [--target-image-opts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename") SRST -.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] [--salvage] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME +.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] [--salvage] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME ERST DEF("create", img_create, diff --git a/qemu-img.c b/qemu-img.c index 2b4562b9d9..0faf2cd2f5 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -70,6 +70,7 @@ enum { OPTION_PREALLOCATION = 265, OPTION_SHRINK = 266, OPTION_SALVAGE = 267, + OPTION_TARGET_IS_ZERO = 268, }; typedef enum OutputFormat { @@ -1984,10 +1985,9 @@ static int convert_do_copy(ImgConvertState *s) int64_t sector_num = 0; /* Check whether we have zero initialisation or can get it efficiently */ - if (s->target_is_new && s->min_sparse && !s->target_has_backing) { + if (!s->has_zero_init && s->target_is_new && s->min_sparse && + !s->target_has_backing) { s->has_zero_init = bdrv_has_zero_init(blk_bs(s->target)); - } else { - s->has_zero_init = false; } if (!s->has_zero_init && !s->target_has_backing && @@ -2086,6 +2086,7 @@ static int img_convert(int argc, char **argv) {"force-share", no_argument, 0, 'U'}, {"target-image-opts", no_argument, 0, OPTION_TARGET_IMAGE_OPTS}, {"salvage", no_argument, 0, OPTION_SALVAGE}, + {"target-is-zero", no_argument, 0, OPTION_TARGET_IS_ZERO}, {0, 0, 0, 0} }; c = getopt_long(argc, argv, ":hf:O:B:Cco:l:S:pt:T:qnm:WU", @@ -2209,6 +2210,14 @@ static int img_convert(int argc, char **argv) case OPTION_TARGET_IMAGE_OPTS: tgt_image_opts = true; break; + case OPTION_TARGET_IS_ZERO: + /* + * The user asserting that the target is blank has the + * same effect as the target driver supporting zero + * initialisation. + */ + s.has_zero_init = true; + break; } } @@ -2247,6 +2256,11 @@ static int img_convert(int argc, char **argv) warn_report("This will become an error in future QEMU versions."); } + if (s.has_zero_init && !skip_create) { + error_report("--target-is-zero requires use of -n flag"); + goto fail_getopt; + } + s.src_num = argc - optind - 1; out_filename = s.src_num >= 1 ? argv[argc - 1] : NULL; @@ -2380,6 +2394,12 @@ static int img_convert(int argc, char **argv) } s.target_has_backing = (bool) out_baseimg; + if (s.has_zero_init && s.target_has_backing) { + error_report("Cannot use --target-is-zero when the destination " + "image has a backing file"); + goto out; + } + if (s.src_num > 1 && out_baseimg) { error_report("Having a backing file for the target makes no sense when " "concatenating multiple input images"); From 087ab8e775f48766068e65de1bc99d03b40d1670 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 7 Feb 2020 13:55:20 +0000 Subject: [PATCH 07/18] block: always fill entire LUKS header space with zeros MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When initializing the LUKS header the size with default encryption parameters will currently be 2068480 bytes. This is rounded up to a multiple of the cluster size, 2081792, with 64k sectors. If the end of the header is not the same as the end of the cluster we fill the extra space with zeros. This was forgetting that not even the space allocated for the header will be fully initialized, as we only write key material for the first key slot. The space left for the other 7 slots is never written to. An optimization to the ref count checking code: commit a5fff8d4b4d928311a5005efa12d0991fe3b66f9 (refs/bisect/bad) Author: Vladimir Sementsov-Ogievskiy Date: Wed Feb 27 16:14:30 2019 +0300 qcow2-refcount: avoid eating RAM made the assumption that every cluster which was allocated would have at least some data written to it. This was violated by way the LUKS header is only partially written, with much space simply reserved for future use. Depending on the cluster size this problem was masked by the logic which wrote zeros between the end of the LUKS header and the end of the cluster. $ qemu-img create --object secret,id=cluster_encrypt0,data=123456 \ -f qcow2 -o cluster_size=2k,encrypt.iter-time=1,\ encrypt.format=luks,encrypt.key-secret=cluster_encrypt0 \ cluster_size_check.qcow2 100M Formatting 'cluster_size_check.qcow2', fmt=qcow2 size=104857600 encrypt.format=luks encrypt.key-secret=cluster_encrypt0 encrypt.iter-time=1 cluster_size=2048 lazy_refcounts=off refcount_bits=16 $ qemu-img check --object secret,id=cluster_encrypt0,data=redhat \ 'json:{"driver": "qcow2", "encrypt.format": "luks", \ "encrypt.key-secret": "cluster_encrypt0", \ "file.driver": "file", "file.filename": "cluster_size_check.qcow2"}' ERROR: counting reference for region exceeding the end of the file by one cluster or more: offset 0x2000 size 0x1f9000 Leaked cluster 4 refcount=1 reference=0 ...snip... Leaked cluster 130 refcount=1 reference=0 1 errors were found on the image. Data may be corrupted, or further writes to the image may corrupt it. 127 leaked clusters were found on the image. This means waste of disk space, but no harm to data. Image end offset: 268288 The problem only exists when the disk image is entirely empty. Writing data to the disk image payload will solve the problem by causing the end of the file to be extended further. The change fixes it by ensuring that the entire allocated LUKS header region is fully initialized with zeros. The qemu-img check will still fail for any pre-existing disk images created prior to this change, unless at least 1 byte of the payload is written to. Fully writing zeros to the entire LUKS header is a good idea regardless as it ensures that space has been allocated on the host filesystem (or whatever block storage backend is used). Signed-off-by: Daniel P. Berrangé Message-Id: <20200207135520.2669430-1-berrange@redhat.com> Reviewed-by: Eric Blake Signed-off-by: Max Reitz --- block/qcow2.c | 11 +++-- tests/qemu-iotests/284 | 97 ++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/284.out | 62 ++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 4 files changed, 167 insertions(+), 4 deletions(-) create mode 100755 tests/qemu-iotests/284 create mode 100644 tests/qemu-iotests/284.out diff --git a/block/qcow2.c b/block/qcow2.c index 8dcee5efec..3c754f616b 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -135,13 +135,16 @@ static ssize_t qcow2_crypto_hdr_init_func(QCryptoBlock *block, size_t headerlen, s->crypto_header.length = headerlen; s->crypto_header.offset = ret; - /* Zero fill remaining space in cluster so it has predictable - * content in case of future spec changes */ + /* + * Zero fill all space in cluster so it has predictable + * content, as we may not initialize some regions of the + * header (eg only 1 out of 8 key slots will be initialized) + */ clusterlen = size_to_clusters(s, headerlen) * s->cluster_size; assert(qcow2_pre_write_overlap_check(bs, 0, ret, clusterlen, false) == 0); ret = bdrv_pwrite_zeroes(bs->file, - ret + headerlen, - clusterlen - headerlen, 0); + ret, + clusterlen, 0); if (ret < 0) { error_setg_errno(errp, -ret, "Could not zero fill encryption header"); return -1; diff --git a/tests/qemu-iotests/284 b/tests/qemu-iotests/284 new file mode 100755 index 0000000000..071e89b33e --- /dev/null +++ b/tests/qemu-iotests/284 @@ -0,0 +1,97 @@ +#!/usr/bin/env bash +# +# Test ref count checks on encrypted images +# +# Copyright (C) 2019 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +# creator +owner=berrange@redhat.com + +seq=`basename $0` +echo "QA output created by $seq" + +status=1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter + +_supported_fmt qcow2 +_supported_proto generic +_supported_os Linux + + +size=1M + +SECRET="secret,id=sec0,data=astrochicken" + +IMGSPEC="driver=$IMGFMT,file.filename=$TEST_IMG,encrypt.key-secret=sec0" +QEMU_IO_OPTIONS=$QEMU_IO_OPTIONS_NO_FMT + +_run_test() +{ + IMGOPTSSYNTAX=true + OLD_TEST_IMG="$TEST_IMG" + TEST_IMG="driver=$IMGFMT,file.filename=$TEST_IMG,encrypt.key-secret=sec0" + QEMU_IMG_EXTRA_ARGS="--image-opts --object $SECRET" + + echo + echo "== cluster size $csize" + echo "== checking image refcounts ==" + _check_test_img + + echo + echo "== writing some data ==" + $QEMU_IO -c "write -P 0x9 0 1" $QEMU_IMG_EXTRA_ARGS $TEST_IMG | _filter_qemu_io | _filter_testdir + echo + echo "== rechecking image refcounts ==" + _check_test_img + + echo + echo "== writing some more data ==" + $QEMU_IO -c "write -P 0x9 $csize 1" $QEMU_IMG_EXTRA_ARGS $TEST_IMG | _filter_qemu_io | _filter_testdir + echo + echo "== rechecking image refcounts ==" + _check_test_img + + TEST_IMG="$OLD_TEST_IMG" + QEMU_IMG_EXTRA_ARGS= + IMGOPTSSYNTAX= +} + + +echo +echo "testing LUKS qcow2 encryption" +echo + +for csize in 512 2048 32768 +do + _make_test_img --object $SECRET -o "encrypt.format=luks,encrypt.key-secret=sec0,encrypt.iter-time=10,cluster_size=$csize" $size + _run_test + _cleanup_test_img +done + +# success, all done +echo "*** done" +rm -f $seq.full +status=0 diff --git a/tests/qemu-iotests/284.out b/tests/qemu-iotests/284.out new file mode 100644 index 0000000000..48216f5742 --- /dev/null +++ b/tests/qemu-iotests/284.out @@ -0,0 +1,62 @@ +QA output created by 284 + +testing LUKS qcow2 encryption + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 encrypt.format=luks encrypt.key-secret=sec0 encrypt.iter-time=10 + +== cluster size 512 +== checking image refcounts == +No errors were found on the image. + +== writing some data == +wrote 1/1 bytes at offset 0 +1 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +== rechecking image refcounts == +No errors were found on the image. + +== writing some more data == +wrote 1/1 bytes at offset 512 +1 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +== rechecking image refcounts == +No errors were found on the image. +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 encrypt.format=luks encrypt.key-secret=sec0 encrypt.iter-time=10 + +== cluster size 2048 +== checking image refcounts == +No errors were found on the image. + +== writing some data == +wrote 1/1 bytes at offset 0 +1 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +== rechecking image refcounts == +No errors were found on the image. + +== writing some more data == +wrote 1/1 bytes at offset 2048 +1 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +== rechecking image refcounts == +No errors were found on the image. +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 encrypt.format=luks encrypt.key-secret=sec0 encrypt.iter-time=10 + +== cluster size 32768 +== checking image refcounts == +No errors were found on the image. + +== writing some data == +wrote 1/1 bytes at offset 0 +1 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +== rechecking image refcounts == +No errors were found on the image. + +== writing some more data == +wrote 1/1 bytes at offset 32768 +1 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +== rechecking image refcounts == +No errors were found on the image. +*** done diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group index 1904223020..818380a8f0 100644 --- a/tests/qemu-iotests/group +++ b/tests/qemu-iotests/group @@ -290,3 +290,4 @@ 280 rw migration quick 281 rw quick 283 auto quick +284 rw From 4bc267a7c7cf204b05d3d1f01625691540a7b965 Mon Sep 17 00:00:00 2001 From: Vladimir Sementsov-Ogievskiy Date: Fri, 7 Feb 2020 19:12:31 +0300 Subject: [PATCH 08/18] block/backup-top: fix flags handling backup-top "supports" write-unchanged, by skipping CBW operation in backup_top_co_pwritev. But it forgets to do the same in backup_top_co_pwrite_zeroes, as well as declare support for BDRV_REQ_WRITE_UNCHANGED. Fix this, and, while being here, declare also support for flags supported by source child. Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20200207161231.32707-1-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz --- block/backup-top.c | 31 ++++++++++++++++++++----------- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/block/backup-top.c b/block/backup-top.c index fa78f3256d..1bfb360bd3 100644 --- a/block/backup-top.c +++ b/block/backup-top.c @@ -48,11 +48,17 @@ static coroutine_fn int backup_top_co_preadv( } static coroutine_fn int backup_top_cbw(BlockDriverState *bs, uint64_t offset, - uint64_t bytes) + uint64_t bytes, BdrvRequestFlags flags) { BDRVBackupTopState *s = bs->opaque; - uint64_t end = QEMU_ALIGN_UP(offset + bytes, s->bcs->cluster_size); - uint64_t off = QEMU_ALIGN_DOWN(offset, s->bcs->cluster_size); + uint64_t off, end; + + if (flags & BDRV_REQ_WRITE_UNCHANGED) { + return 0; + } + + off = QEMU_ALIGN_DOWN(offset, s->bcs->cluster_size); + end = QEMU_ALIGN_UP(offset + bytes, s->bcs->cluster_size); return block_copy(s->bcs, off, end - off, NULL); } @@ -60,7 +66,7 @@ static coroutine_fn int backup_top_cbw(BlockDriverState *bs, uint64_t offset, static int coroutine_fn backup_top_co_pdiscard(BlockDriverState *bs, int64_t offset, int bytes) { - int ret = backup_top_cbw(bs, offset, bytes); + int ret = backup_top_cbw(bs, offset, bytes, 0); if (ret < 0) { return ret; } @@ -71,7 +77,7 @@ static int coroutine_fn backup_top_co_pdiscard(BlockDriverState *bs, static int coroutine_fn backup_top_co_pwrite_zeroes(BlockDriverState *bs, int64_t offset, int bytes, BdrvRequestFlags flags) { - int ret = backup_top_cbw(bs, offset, bytes); + int ret = backup_top_cbw(bs, offset, bytes, flags); if (ret < 0) { return ret; } @@ -84,11 +90,9 @@ static coroutine_fn int backup_top_co_pwritev(BlockDriverState *bs, uint64_t bytes, QEMUIOVector *qiov, int flags) { - if (!(flags & BDRV_REQ_WRITE_UNCHANGED)) { - int ret = backup_top_cbw(bs, offset, bytes); - if (ret < 0) { - return ret; - } + int ret = backup_top_cbw(bs, offset, bytes, flags); + if (ret < 0) { + return ret; } return bdrv_co_pwritev(bs->backing, offset, bytes, qiov, flags); @@ -196,8 +200,13 @@ BlockDriverState *bdrv_backup_top_append(BlockDriverState *source, return NULL; } - top->total_sectors = source->total_sectors; state = top->opaque; + top->total_sectors = source->total_sectors; + top->supported_write_flags = BDRV_REQ_WRITE_UNCHANGED | + (BDRV_REQ_FUA & source->supported_write_flags); + top->supported_zero_flags = BDRV_REQ_WRITE_UNCHANGED | + ((BDRV_REQ_FUA | BDRV_REQ_MAY_UNMAP | BDRV_REQ_NO_FALLBACK) & + source->supported_zero_flags); bdrv_ref(target); state->target = bdrv_attach_child(top, target, "target", &child_file, errp); From b7e9eae98c160f3e7eb9ffbedba8821e358d963c Mon Sep 17 00:00:00 2001 From: Max Reitz Date: Thu, 19 Dec 2019 15:42:43 +0100 Subject: [PATCH 09/18] iotests/279: Fix for non-qcow2 formats First, driver=qcow2 will not work so well with non-qcow2 formats (and this test claims to support qcow, qed, and vmdk). Second, vmdk will always report the backing file format to be vmdk. Filter that out so the output looks like for all other formats. Third, the flat vmdk subformats do not support backing files, so they will not work with this test. Signed-off-by: Max Reitz Message-Id: <20191219144243.1763246-1-mreitz@redhat.com> Tested-by: Thomas Huth Signed-off-by: Max Reitz --- tests/qemu-iotests/279 | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/tests/qemu-iotests/279 b/tests/qemu-iotests/279 index 6682376808..30d29b1cb2 100755 --- a/tests/qemu-iotests/279 +++ b/tests/qemu-iotests/279 @@ -38,6 +38,8 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow qcow2 vmdk qed _supported_proto file _supported_os Linux +_unsupported_imgopts "subformat=monolithicFlat" \ + "subformat=twoGbMaxExtentFlat" \ TEST_IMG="$TEST_IMG.base" _make_test_img 64M TEST_IMG="$TEST_IMG.mid" _make_test_img -b "$TEST_IMG.base" @@ -45,11 +47,12 @@ _make_test_img -b "$TEST_IMG.mid" echo echo '== qemu-img info --backing-chain ==' -_img_info --backing-chain | _filter_img_info +_img_info --backing-chain | _filter_img_info | grep -v 'backing file format' echo echo '== qemu-img info --backing-chain --image-opts ==' -TEST_IMG="driver=qcow2,file.driver=file,file.filename=$TEST_IMG" _img_info --backing-chain --image-opts | _filter_img_info +TEST_IMG="driver=$IMGFMT,file.driver=file,file.filename=$TEST_IMG" _img_info --backing-chain --image-opts \ + | _filter_img_info | grep -v 'backing file format' # success, all done echo "*** done" From 78c81a3f108870d325b0a39d88711366afe6f703 Mon Sep 17 00:00:00 2001 From: Max Reitz Date: Wed, 22 Jan 2020 17:45:28 +0100 Subject: [PATCH 10/18] block/nbd: Fix hang in .bdrv_close() When nbd_close() is called from a coroutine, the connection_co never gets to run, and thus nbd_teardown_connection() hangs. This is because aio_co_enter() only puts the connection_co into the main coroutine's wake-up queue, so this main coroutine needs to yield and wait for connection_co to terminate. Suggested-by: Kevin Wolf Signed-off-by: Max Reitz Message-Id: <20200122164532.178040-2-mreitz@redhat.com> Reviewed-by: Eric Blake Reviewed-by: Maxim Levitsky Signed-off-by: Max Reitz --- block/nbd.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/block/nbd.c b/block/nbd.c index d085554f21..6d3b22f844 100644 --- a/block/nbd.c +++ b/block/nbd.c @@ -70,6 +70,7 @@ typedef struct BDRVNBDState { CoMutex send_mutex; CoQueue free_sema; Coroutine *connection_co; + Coroutine *teardown_co; QemuCoSleepState *connection_co_sleep_ns_state; bool drained; bool wait_drained_end; @@ -203,7 +204,15 @@ static void nbd_teardown_connection(BlockDriverState *bs) qemu_co_sleep_wake(s->connection_co_sleep_ns_state); } } - BDRV_POLL_WHILE(bs, s->connection_co); + if (qemu_in_coroutine()) { + s->teardown_co = qemu_coroutine_self(); + /* connection_co resumes us when it terminates */ + qemu_coroutine_yield(); + s->teardown_co = NULL; + } else { + BDRV_POLL_WHILE(bs, s->connection_co); + } + assert(!s->connection_co); } static bool nbd_client_connecting(BDRVNBDState *s) @@ -395,6 +404,9 @@ static coroutine_fn void nbd_connection_entry(void *opaque) s->ioc = NULL; } + if (s->teardown_co) { + aio_co_wake(s->teardown_co); + } aio_wait_kick(); } From fd17146cd93d1704cd96d7c2757b325fc7aac6fd Mon Sep 17 00:00:00 2001 From: Max Reitz Date: Wed, 22 Jan 2020 17:45:29 +0100 Subject: [PATCH 11/18] block: Generic file creation fallback If a protocol driver does not support image creation, we can see whether maybe the file exists already. If so, just truncating it will be sufficient. Signed-off-by: Max Reitz Message-Id: <20200122164532.178040-3-mreitz@redhat.com> Signed-off-by: Max Reitz --- block.c | 159 +++++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 147 insertions(+), 12 deletions(-) diff --git a/block.c b/block.c index 08372ced26..308a91c96b 100644 --- a/block.c +++ b/block.c @@ -532,20 +532,139 @@ out: return ret; } +/** + * Helper function for bdrv_create_file_fallback(): Resize @blk to at + * least the given @minimum_size. + * + * On success, return @blk's actual length. + * Otherwise, return -errno. + */ +static int64_t create_file_fallback_truncate(BlockBackend *blk, + int64_t minimum_size, Error **errp) +{ + Error *local_err = NULL; + int64_t size; + int ret; + + ret = blk_truncate(blk, minimum_size, false, PREALLOC_MODE_OFF, &local_err); + if (ret < 0 && ret != -ENOTSUP) { + error_propagate(errp, local_err); + return ret; + } + + size = blk_getlength(blk); + if (size < 0) { + error_free(local_err); + error_setg_errno(errp, -size, + "Failed to inquire the new image file's length"); + return size; + } + + if (size < minimum_size) { + /* Need to grow the image, but we failed to do that */ + error_propagate(errp, local_err); + return -ENOTSUP; + } + + error_free(local_err); + local_err = NULL; + + return size; +} + +/** + * Helper function for bdrv_create_file_fallback(): Zero the first + * sector to remove any potentially pre-existing image header. + */ +static int create_file_fallback_zero_first_sector(BlockBackend *blk, + int64_t current_size, + Error **errp) +{ + int64_t bytes_to_clear; + int ret; + + bytes_to_clear = MIN(current_size, BDRV_SECTOR_SIZE); + if (bytes_to_clear) { + ret = blk_pwrite_zeroes(blk, 0, bytes_to_clear, BDRV_REQ_MAY_UNMAP); + if (ret < 0) { + error_setg_errno(errp, -ret, + "Failed to clear the new image's first sector"); + return ret; + } + } + + return 0; +} + +static int bdrv_create_file_fallback(const char *filename, BlockDriver *drv, + QemuOpts *opts, Error **errp) +{ + BlockBackend *blk; + QDict *options = qdict_new(); + int64_t size = 0; + char *buf = NULL; + PreallocMode prealloc; + Error *local_err = NULL; + int ret; + + size = qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0); + buf = qemu_opt_get_del(opts, BLOCK_OPT_PREALLOC); + prealloc = qapi_enum_parse(&PreallocMode_lookup, buf, + PREALLOC_MODE_OFF, &local_err); + g_free(buf); + if (local_err) { + error_propagate(errp, local_err); + return -EINVAL; + } + + if (prealloc != PREALLOC_MODE_OFF) { + error_setg(errp, "Unsupported preallocation mode '%s'", + PreallocMode_str(prealloc)); + return -ENOTSUP; + } + + qdict_put_str(options, "driver", drv->format_name); + + blk = blk_new_open(filename, NULL, options, + BDRV_O_RDWR | BDRV_O_RESIZE, errp); + if (!blk) { + error_prepend(errp, "Protocol driver '%s' does not support image " + "creation, and opening the image failed: ", + drv->format_name); + return -EINVAL; + } + + size = create_file_fallback_truncate(blk, size, errp); + if (size < 0) { + ret = size; + goto out; + } + + ret = create_file_fallback_zero_first_sector(blk, size, errp); + if (ret < 0) { + goto out; + } + + ret = 0; +out: + blk_unref(blk); + return ret; +} + int bdrv_create_file(const char *filename, QemuOpts *opts, Error **errp) { BlockDriver *drv; - Error *local_err = NULL; - int ret; drv = bdrv_find_protocol(filename, true, errp); if (drv == NULL) { return -ENOENT; } - ret = bdrv_create(drv, filename, opts, &local_err); - error_propagate(errp, local_err); - return ret; + if (drv->bdrv_co_create_opts) { + return bdrv_create(drv, filename, opts, errp); + } else { + return bdrv_create_file_fallback(filename, drv, opts, errp); + } } /** @@ -1444,6 +1563,24 @@ QemuOptsList bdrv_runtime_opts = { }, }; +static QemuOptsList fallback_create_opts = { + .name = "fallback-create-opts", + .head = QTAILQ_HEAD_INITIALIZER(fallback_create_opts.head), + .desc = { + { + .name = BLOCK_OPT_SIZE, + .type = QEMU_OPT_SIZE, + .help = "Virtual disk size" + }, + { + .name = BLOCK_OPT_PREALLOC, + .type = QEMU_OPT_STRING, + .help = "Preallocation mode (allowed values: off)" + }, + { /* end of list */ } + } +}; + /* * Common part for opening disk images and files * @@ -5772,15 +5909,13 @@ void bdrv_img_create(const char *filename, const char *fmt, return; } - if (!proto_drv->create_opts) { - error_setg(errp, "Protocol driver '%s' does not support image creation", - proto_drv->format_name); - return; - } - /* Create parameter list */ create_opts = qemu_opts_append(create_opts, drv->create_opts); - create_opts = qemu_opts_append(create_opts, proto_drv->create_opts); + if (proto_drv->create_opts) { + create_opts = qemu_opts_append(create_opts, proto_drv->create_opts); + } else { + create_opts = qemu_opts_append(create_opts, &fallback_create_opts); + } opts = qemu_opts_create(create_opts, NULL, 0, &error_abort); From 87ca3b8fa615b278b33cabf9ed22b3f44b5214ba Mon Sep 17 00:00:00 2001 From: Max Reitz Date: Wed, 22 Jan 2020 17:45:30 +0100 Subject: [PATCH 12/18] file-posix: Drop hdev_co_create_opts() The generic fallback implementation effectively does the same. Reviewed-by: Maxim Levitsky Signed-off-by: Max Reitz Message-Id: <20200122164532.178040-4-mreitz@redhat.com> Signed-off-by: Max Reitz --- block/file-posix.c | 67 ---------------------------------------------- 1 file changed, 67 deletions(-) diff --git a/block/file-posix.c b/block/file-posix.c index ab82ee1a67..6345477112 100644 --- a/block/file-posix.c +++ b/block/file-posix.c @@ -3477,67 +3477,6 @@ static coroutine_fn int hdev_co_pwrite_zeroes(BlockDriverState *bs, return raw_do_pwrite_zeroes(bs, offset, bytes, flags, true); } -static int coroutine_fn hdev_co_create_opts(const char *filename, QemuOpts *opts, - Error **errp) -{ - int fd; - int ret = 0; - struct stat stat_buf; - int64_t total_size = 0; - bool has_prefix; - - /* This function is used by both protocol block drivers and therefore either - * of these prefixes may be given. - * The return value has to be stored somewhere, otherwise this is an error - * due to -Werror=unused-value. */ - has_prefix = - strstart(filename, "host_device:", &filename) || - strstart(filename, "host_cdrom:" , &filename); - - (void)has_prefix; - - ret = raw_normalize_devicepath(&filename, errp); - if (ret < 0) { - return ret; - } - - /* Read out options */ - total_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0), - BDRV_SECTOR_SIZE); - - fd = qemu_open(filename, O_WRONLY | O_BINARY); - if (fd < 0) { - ret = -errno; - error_setg_errno(errp, -ret, "Could not open device"); - return ret; - } - - if (fstat(fd, &stat_buf) < 0) { - ret = -errno; - error_setg_errno(errp, -ret, "Could not stat device"); - } else if (!S_ISBLK(stat_buf.st_mode) && !S_ISCHR(stat_buf.st_mode)) { - error_setg(errp, - "The given file is neither a block nor a character device"); - ret = -ENODEV; - } else if (lseek(fd, 0, SEEK_END) < total_size) { - error_setg(errp, "Device is too small"); - ret = -ENOSPC; - } - - if (!ret && total_size) { - uint8_t buf[BDRV_SECTOR_SIZE] = { 0 }; - int64_t zero_size = MIN(BDRV_SECTOR_SIZE, total_size); - if (lseek(fd, 0, SEEK_SET) == -1) { - ret = -errno; - } else { - ret = qemu_write_full(fd, buf, zero_size); - ret = ret == zero_size ? 0 : -errno; - } - } - qemu_close(fd); - return ret; -} - static BlockDriver bdrv_host_device = { .format_name = "host_device", .protocol_name = "host_device", @@ -3550,8 +3489,6 @@ static BlockDriver bdrv_host_device = { .bdrv_reopen_prepare = raw_reopen_prepare, .bdrv_reopen_commit = raw_reopen_commit, .bdrv_reopen_abort = raw_reopen_abort, - .bdrv_co_create_opts = hdev_co_create_opts, - .create_opts = &raw_create_opts, .mutable_opts = mutable_opts, .bdrv_co_invalidate_cache = raw_co_invalidate_cache, .bdrv_co_pwrite_zeroes = hdev_co_pwrite_zeroes, @@ -3678,8 +3615,6 @@ static BlockDriver bdrv_host_cdrom = { .bdrv_reopen_prepare = raw_reopen_prepare, .bdrv_reopen_commit = raw_reopen_commit, .bdrv_reopen_abort = raw_reopen_abort, - .bdrv_co_create_opts = hdev_co_create_opts, - .create_opts = &raw_create_opts, .mutable_opts = mutable_opts, .bdrv_co_invalidate_cache = raw_co_invalidate_cache, @@ -3812,8 +3747,6 @@ static BlockDriver bdrv_host_cdrom = { .bdrv_reopen_prepare = raw_reopen_prepare, .bdrv_reopen_commit = raw_reopen_commit, .bdrv_reopen_abort = raw_reopen_abort, - .bdrv_co_create_opts = hdev_co_create_opts, - .create_opts = &raw_create_opts, .mutable_opts = mutable_opts, .bdrv_co_preadv = raw_co_preadv, From 80f0900905b555f00d644894c786b6d66ac2e00e Mon Sep 17 00:00:00 2001 From: Max Reitz Date: Wed, 22 Jan 2020 17:45:31 +0100 Subject: [PATCH 13/18] iscsi: Drop iscsi_co_create_opts() The generic fallback implementation effectively does the same. Reviewed-by: Maxim Levitsky Signed-off-by: Max Reitz Message-Id: <20200122164532.178040-5-mreitz@redhat.com> Signed-off-by: Max Reitz --- block/iscsi.c | 56 --------------------------------------------------- 1 file changed, 56 deletions(-) diff --git a/block/iscsi.c b/block/iscsi.c index c8feaa2f0e..682abd8e09 100644 --- a/block/iscsi.c +++ b/block/iscsi.c @@ -2164,58 +2164,6 @@ static int coroutine_fn iscsi_co_truncate(BlockDriverState *bs, int64_t offset, return 0; } -static int coroutine_fn iscsi_co_create_opts(const char *filename, QemuOpts *opts, - Error **errp) -{ - int ret = 0; - int64_t total_size = 0; - BlockDriverState *bs; - IscsiLun *iscsilun = NULL; - QDict *bs_options; - Error *local_err = NULL; - - bs = bdrv_new(); - - /* Read out options */ - total_size = DIV_ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0), - BDRV_SECTOR_SIZE); - bs->opaque = g_new0(struct IscsiLun, 1); - iscsilun = bs->opaque; - - bs_options = qdict_new(); - iscsi_parse_filename(filename, bs_options, &local_err); - if (local_err) { - error_propagate(errp, local_err); - ret = -EINVAL; - } else { - ret = iscsi_open(bs, bs_options, 0, NULL); - } - qobject_unref(bs_options); - - if (ret != 0) { - goto out; - } - iscsi_detach_aio_context(bs); - if (iscsilun->type != TYPE_DISK) { - ret = -ENODEV; - goto out; - } - if (bs->total_sectors < total_size) { - ret = -ENOSPC; - goto out; - } - - ret = 0; -out: - if (iscsilun->iscsi != NULL) { - iscsi_destroy_context(iscsilun->iscsi); - } - g_free(bs->opaque); - bs->opaque = NULL; - bdrv_unref(bs); - return ret; -} - static int iscsi_get_info(BlockDriverState *bs, BlockDriverInfo *bdi) { IscsiLun *iscsilun = bs->opaque; @@ -2486,8 +2434,6 @@ static BlockDriver bdrv_iscsi = { .bdrv_parse_filename = iscsi_parse_filename, .bdrv_file_open = iscsi_open, .bdrv_close = iscsi_close, - .bdrv_co_create_opts = iscsi_co_create_opts, - .create_opts = &iscsi_create_opts, .bdrv_reopen_prepare = iscsi_reopen_prepare, .bdrv_reopen_commit = iscsi_reopen_commit, .bdrv_co_invalidate_cache = iscsi_co_invalidate_cache, @@ -2525,8 +2471,6 @@ static BlockDriver bdrv_iser = { .bdrv_parse_filename = iscsi_parse_filename, .bdrv_file_open = iscsi_open, .bdrv_close = iscsi_close, - .bdrv_co_create_opts = iscsi_co_create_opts, - .create_opts = &iscsi_create_opts, .bdrv_reopen_prepare = iscsi_reopen_prepare, .bdrv_reopen_commit = iscsi_reopen_commit, .bdrv_co_invalidate_cache = iscsi_co_invalidate_cache, From 4dddeac115c5a2c5f74731fda0afd031a0b45490 Mon Sep 17 00:00:00 2001 From: Max Reitz Date: Wed, 22 Jan 2020 17:45:32 +0100 Subject: [PATCH 14/18] iotests: Add test for image creation fallback Signed-off-by: Max Reitz Message-Id: <20200122164532.178040-6-mreitz@redhat.com> Reviewed-by: Eric Blake Reviewed-by: Maxim Levitsky [mreitz: Added a note that NBD does not support resizing, which is why the second case is expected to fail] Signed-off-by: Max Reitz --- tests/qemu-iotests/259 | 62 ++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/259.out | 14 +++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 77 insertions(+) create mode 100755 tests/qemu-iotests/259 create mode 100644 tests/qemu-iotests/259.out diff --git a/tests/qemu-iotests/259 b/tests/qemu-iotests/259 new file mode 100755 index 0000000000..62e29af05f --- /dev/null +++ b/tests/qemu-iotests/259 @@ -0,0 +1,62 @@ +#!/usr/bin/env bash +# +# Test generic image creation fallback (by using NBD) +# +# Copyright (C) 2019 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +# creator +owner=mreitz@redhat.com + +seq=$(basename $0) +echo "QA output created by $seq" + +status=1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter + +_supported_fmt raw +_supported_proto nbd +_supported_os Linux + + +_make_test_img 64M + +echo +echo '--- Testing creation ---' + +$QEMU_IMG create -f qcow2 "$TEST_IMG" 64M | _filter_img_create +$QEMU_IMG info "$TEST_IMG" | _filter_img_info + +echo +echo '--- Testing creation for which the node would need to grow ---' + +# NBD does not support resizing, so this will fail +$QEMU_IMG create -f qcow2 -o preallocation=metadata "$TEST_IMG" 64M 2>&1 \ + | _filter_img_create + +# success, all done +echo "*** done" +rm -f $seq.full +status=0 diff --git a/tests/qemu-iotests/259.out b/tests/qemu-iotests/259.out new file mode 100644 index 0000000000..ffed19c2a0 --- /dev/null +++ b/tests/qemu-iotests/259.out @@ -0,0 +1,14 @@ +QA output created by 259 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 + +--- Testing creation --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=qcow2 size=67108864 +image: TEST_DIR/t.IMGFMT +file format: qcow2 +virtual size: 64 MiB (67108864 bytes) +disk size: unavailable + +--- Testing creation for which the node would need to grow --- +qemu-img: TEST_DIR/t.IMGFMT: Could not resize image: Image format driver does not support resize +Formatting 'TEST_DIR/t.IMGFMT', fmt=qcow2 size=67108864 preallocation=metadata +*** done diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group index 818380a8f0..b17711d17d 100644 --- a/tests/qemu-iotests/group +++ b/tests/qemu-iotests/group @@ -273,6 +273,7 @@ 256 rw auto quick 257 rw 258 rw quick +259 rw auto quick 260 rw quick 261 rw 262 rw quick migration From c69291e712ae4ef95f628424db6586473da61d43 Mon Sep 17 00:00:00 2001 From: Max Reitz Date: Tue, 21 Jan 2020 16:59:14 +0100 Subject: [PATCH 15/18] qemu-img: Fix convert -n -B for backing-less targets s.target_has_backing does not reflect whether the target BDS has a backing file; it only tells whether we should use a backing file during conversion (specified by -B). As such, if you use convert -n, the target does not necessarily actually have a backing file, and then dereferencing out_bs->backing fails here. When converting to an existing file, we should set target_backing_sectors to a negative value, because first, as the comment explains, this value is only used for optimization, so it is always fine to do that. Second, we use this value to determine where the target must be initialized to zeroes (overlays are initialized to zero after the end of their backing file). When converting to an existing file, we cannot assume that to be true. Cc: qemu-stable@nongnu.org Fixes: 351c8efff9ad809c822d55620df54d575d536f68 ("qemu-img: Special post-backing convert handling") Signed-off-by: Max Reitz Message-Id: <20200121155915.98232-2-mreitz@redhat.com> Reviewed-by: John Snow Signed-off-by: Max Reitz --- qemu-img.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/qemu-img.c b/qemu-img.c index 0faf2cd2f5..804630a368 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -2523,7 +2523,7 @@ static int img_convert(int argc, char **argv) } } - if (s.target_has_backing) { + if (s.target_has_backing && s.target_is_new) { /* Errors are treated as "backing length unknown" (which means * s.target_backing_sectors has to be negative, which it will * be automatically). The backing file length is used only From f535cc90b50b1be2e6c204b844eab991c872b78b Mon Sep 17 00:00:00 2001 From: Max Reitz Date: Tue, 21 Jan 2020 16:59:15 +0100 Subject: [PATCH 16/18] iotests: Test convert -n -B to backing-less target This must not crash. Signed-off-by: Max Reitz Message-Id: <20200121155915.98232-3-mreitz@redhat.com> Reviewed-by: John Snow Signed-off-by: Max Reitz --- tests/qemu-iotests/122 | 14 ++++++++++++++ tests/qemu-iotests/122.out | 5 +++++ 2 files changed, 19 insertions(+) diff --git a/tests/qemu-iotests/122 b/tests/qemu-iotests/122 index dfa350936f..f7a3ae684a 100755 --- a/tests/qemu-iotests/122 +++ b/tests/qemu-iotests/122 @@ -276,6 +276,20 @@ $QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig $QEMU_IMG compare "$TEST_IMG" "$TEST_IMG".orig +echo +echo '=== -n -B to an image without a backing file ===' +echo + +# Base for the output +TEST_IMG="$TEST_IMG".base _make_test_img 64M + +# Output that does have $TEST_IMG.base set as its (implicit) backing file +TEST_IMG="$TEST_IMG".orig _make_test_img 64M + +# Convert with -n, which should not confuse -B with "target BDS has a +# backing file" +$QEMU_IMG convert -O $IMGFMT -B "$TEST_IMG".base -n "$TEST_IMG" "$TEST_IMG".orig + # success, all done echo '*** done' rm -f $seq.full diff --git a/tests/qemu-iotests/122.out b/tests/qemu-iotests/122.out index 849b6cc2ef..1a35951a80 100644 --- a/tests/qemu-iotests/122.out +++ b/tests/qemu-iotests/122.out @@ -228,4 +228,9 @@ Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864 wrote 65536/65536 bytes at offset 0 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) Images are identical. + +=== -n -B to an image without a backing file === + +Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=67108864 +Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864 *** done From 804359b8b90f76d9d8fbe8d85a6544b68f107f10 Mon Sep 17 00:00:00 2001 From: Max Reitz Date: Fri, 17 Jan 2020 11:58:58 +0100 Subject: [PATCH 17/18] block: Fix VM size field width in snapshot dump When printing the snapshot list (e.g. with qemu-img snapshot -l), the VM size field is only seven characters wide. As of de38b5005e9, this is not necessarily sufficient: We generally print three digits, and this may require a decimal point. Also, the unit field grew from something as plain as "M" to " MiB". This means that number and unit may take up eight characters in total; but we also want spaces in front. Considering previously the maximum width was four characters and the field width was chosen to be three characters wider, let us adjust the field width to be eleven now. Fixes: de38b5005e946aa3714963ea4c501e279e7d3666 Buglink: https://bugs.launchpad.net/qemu/+bug/1859989 Signed-off-by: Max Reitz Message-Id: <20200117105859.241818-2-mreitz@redhat.com> Reviewed-by: Eric Blake Signed-off-by: Max Reitz --- block/qapi.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/block/qapi.c b/block/qapi.c index 3f09477cc5..afd9f3b4a7 100644 --- a/block/qapi.c +++ b/block/qapi.c @@ -664,7 +664,7 @@ void bdrv_snapshot_dump(QEMUSnapshotInfo *sn) char *sizing = NULL; if (!sn) { - qemu_printf("%-10s%-20s%7s%20s%15s", + qemu_printf("%-10s%-20s%11s%20s%15s", "ID", "TAG", "VM SIZE", "DATE", "VM CLOCK"); } else { ti = sn->date_sec; @@ -679,7 +679,7 @@ void bdrv_snapshot_dump(QEMUSnapshotInfo *sn) (int)(secs % 60), (int)((sn->vm_clock_nsec / 1000000) % 1000)); sizing = size_to_str(sn->vm_state_size); - qemu_printf("%-10s%-20s%7s%20s%15s", + qemu_printf("%-10s%-20s%11s%20s%15s", sn->id_str, sn->name, sizing, date_buf, From dff8d44c96f128480430b0c59ed8760917dbd427 Mon Sep 17 00:00:00 2001 From: Max Reitz Date: Fri, 17 Jan 2020 11:58:59 +0100 Subject: [PATCH 18/18] iotests: Test snapshot -l field separation Add a test that all fields in "qemu-img snapshot -l"s output are separated by spaces. Signed-off-by: Max Reitz Message-Id: <20200117105859.241818-3-mreitz@redhat.com> Reviewed-by: Eric Blake [mreitz: Renamed test from 284 to 286] Signed-off-by: Max Reitz --- tests/qemu-iotests/286 | 76 ++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/286.out | 8 ++++ tests/qemu-iotests/group | 1 + 3 files changed, 85 insertions(+) create mode 100755 tests/qemu-iotests/286 create mode 100644 tests/qemu-iotests/286.out diff --git a/tests/qemu-iotests/286 b/tests/qemu-iotests/286 new file mode 100755 index 0000000000..f14445ba4a --- /dev/null +++ b/tests/qemu-iotests/286 @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# +# Test qemu-img snapshot -l +# +# Copyright (C) 2019 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +seq=$(basename "$0") +echo "QA output created by $seq" + +status=1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter +. ./common.qemu + +_supported_fmt qcow2 +_supported_proto file +# Internal snapshots are (currently) impossible with refcount_bits=1, +# and generally impossible with external data files +_unsupported_imgopts 'refcount_bits=1[^0-9]' data_file + +_make_test_img 64M + +# Should be so long as to take up the whole field width +sn_name=abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz + +# More memory will give us a larger VM state, i.e. one above 1 MB. +# This way, we get a number with a decimal point. +qemu_comm_method=monitor _launch_qemu -m 512 "$TEST_IMG" + +_send_qemu_cmd $QEMU_HANDLE "savevm $sn_name" '(qemu)' +_send_qemu_cmd $QEMU_HANDLE 'quit' '(qemu)' +wait=yes _cleanup_qemu + +# Check that all fields are separated by spaces. +# We first collapse all space sequences into one space each; +# then we turn every space-separated field into a '.'; +# and finally, we name the '.'s so the output is not just a confusing +# sequence of dots. + +echo 'Output structure:' +$QEMU_IMG snapshot -l "$TEST_IMG" | tail -n 1 | tr -s ' ' \ + | sed -e 's/\S\+/./g' \ + | sed -e 's/\./(snapshot ID)/' \ + -e 's/\./(snapshot name)/' \ + -e 's/\./(VM state size value)/' \ + -e 's/\./(VM state size unit)/' \ + -e 's/\./(snapshot date)/' \ + -e 's/\./(snapshot time)/' \ + -e 's/\./(VM clock)/' + +# success, all done +echo "*** done" +rm -f $seq.full +status=0 diff --git a/tests/qemu-iotests/286.out b/tests/qemu-iotests/286.out new file mode 100644 index 0000000000..39ff07e12c --- /dev/null +++ b/tests/qemu-iotests/286.out @@ -0,0 +1,8 @@ +QA output created by 286 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 +QEMU X.Y.Z monitor - type 'help' for more information +(qemu) savevm abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz +(qemu) quit +Output structure: +(snapshot ID) (snapshot name) (VM state size value) (VM state size unit) (snapshot date) (snapshot time) (VM clock) +*** done diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group index b17711d17d..0317667695 100644 --- a/tests/qemu-iotests/group +++ b/tests/qemu-iotests/group @@ -292,3 +292,4 @@ 281 rw quick 283 auto quick 284 rw +286 rw quick