Flaky avocado tests, gdbstub and gitlab tweaks

- gdbstub, properly halt when QEMU is having IO issues
   - convert skipIf(GITLAB_CI) to skipUnless(QEMU_TEST_FLAKY_TESTS)
   - tag sbsa-ref tests as TCG only
   - build the correct microblaze for avocado-system-ubuntu
   - add optional flaky tests job to CI
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCgAdFiEEZoWumedRZ7yvyN81+9DbCVqeKkQFAmVqHFgACgkQ+9DbCVqe
 KkQHLwgAjP2iL5LSa3FaMUoESJQqRB0rpoJ80gtEtmvmgRF0fHsRfHtDdMN9h2Ed
 YilCDhMKLyr2ZoK4atyuc5SR6vCXI5RAvfTddex0xSxlvBX5Z5+1FMC6yA8SDJM7
 ezEXACEKHiGv+l8gvOZOf9ZYEgh8DMJYFMbrtxuxKWw/kAjZ3R3X/ChCL94ZCPRe
 486wqPIQfp5EPs2ddsW4DYFTjLpK5ImX+u/5kdaEGXwcg8UoLmQ9BVIrN/hYJ6u5
 t/mAp1qVIQwSOSUBnerQ4ZkVQfCgLtEtiDtt8EZjUbQD3DcLjfHFjTwVlpqcC1zs
 wHXYpLbD5jkthqav5E0DObCF9gIZdA==
 =qtvU
 -----END PGP SIGNATURE-----

Merge tag 'pull-more-8.2-fixes-011223-2' of https://gitlab.com/stsquad/qemu into staging

Flaky avocado tests, gdbstub and gitlab tweaks

  - gdbstub, properly halt when QEMU is having IO issues
  - convert skipIf(GITLAB_CI) to skipUnless(QEMU_TEST_FLAKY_TESTS)
  - tag sbsa-ref tests as TCG only
  - build the correct microblaze for avocado-system-ubuntu
  - add optional flaky tests job to CI

# -----BEGIN PGP SIGNATURE-----
#
# iQEzBAABCgAdFiEEZoWumedRZ7yvyN81+9DbCVqeKkQFAmVqHFgACgkQ+9DbCVqe
# KkQHLwgAjP2iL5LSa3FaMUoESJQqRB0rpoJ80gtEtmvmgRF0fHsRfHtDdMN9h2Ed
# YilCDhMKLyr2ZoK4atyuc5SR6vCXI5RAvfTddex0xSxlvBX5Z5+1FMC6yA8SDJM7
# ezEXACEKHiGv+l8gvOZOf9ZYEgh8DMJYFMbrtxuxKWw/kAjZ3R3X/ChCL94ZCPRe
# 486wqPIQfp5EPs2ddsW4DYFTjLpK5ImX+u/5kdaEGXwcg8UoLmQ9BVIrN/hYJ6u5
# t/mAp1qVIQwSOSUBnerQ4ZkVQfCgLtEtiDtt8EZjUbQD3DcLjfHFjTwVlpqcC1zs
# wHXYpLbD5jkthqav5E0DObCF9gIZdA==
# =qtvU
# -----END PGP SIGNATURE-----
# gpg: Signature made Fri 01 Dec 2023 12:48:08 EST
# gpg:                using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>" [unknown]
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg:          There is no indication that the signature belongs to the owner.
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8  DF35 FBD0 DB09 5A9E 2A44

* tag 'pull-more-8.2-fixes-011223-2' of https://gitlab.com/stsquad/qemu:
  gitlab: add optional job to run flaky avocado tests
  gitlab: build the correct microblaze target
  tests/avocado: tag sbsa tests as tcg only
  docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS
  gdbstub: use a better signal when we halt for IO reasons

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
This commit is contained in:
Stefan Hajnoczi 2023-12-04 08:03:18 -05:00
commit 17dacf7ac9
17 changed files with 128 additions and 46 deletions

View File

@ -41,7 +41,7 @@ build-system-ubuntu:
variables:
IMAGE: ubuntu2204
CONFIGURE_ARGS: --enable-docs
TARGETS: alpha-softmmu microblazeel-softmmu mips64el-softmmu
TARGETS: alpha-softmmu microblaze-softmmu mips64el-softmmu
MAKE_CHECK_ARGS: check-build
check-system-ubuntu:
@ -217,6 +217,36 @@ avocado-system-opensuse:
MAKE_CHECK_ARGS: check-avocado
AVOCADO_TAGS: arch:s390x arch:x86_64 arch:aarch64
#
# Flaky tests. We don't run these by default and they are allow fail
# but often the CI system is the only way to trigger the failures.
#
build-system-flaky:
extends:
- .native_build_job_template
- .native_build_artifact_template
needs:
job: amd64-debian-container
variables:
IMAGE: debian
QEMU_JOB_OPTIONAL: 1
TARGETS: aarch64-softmmu arm-softmmu mips64el-softmmu
ppc64-softmmu rx-softmmu s390x-softmmu sh4-softmmu x86_64-softmmu
MAKE_CHECK_ARGS: check-build
avocado-system-flaky:
extends: .avocado_test_job_template
needs:
- job: build-system-flaky
artifacts: true
allow_failure: true
variables:
IMAGE: debian
MAKE_CHECK_ARGS: check-avocado
QEMU_JOB_OPTIONAL: 1
QEMU_TEST_FLAKY_TESTS: 1
AVOCADO_TAGS: flaky
# This jobs explicitly disable TCG (--disable-tcg), KVM is detected by
# the configure script. The container doesn't contain Xen headers so

View File

@ -1371,23 +1371,33 @@ conditions. For example, tests that take longer to execute when QEMU is
compiled with debug flags. Therefore, the ``AVOCADO_TIMEOUT_EXPECTED`` variable
has been used to determine whether those tests should run or not.
GITLAB_CI
^^^^^^^^^
A number of tests are flagged to not run on the GitLab CI. Usually because
they proved to the flaky or there are constraints on the CI environment which
would make them fail. If you encounter a similar situation then use that
variable as shown on the code snippet below to skip the test:
.. code::
@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
def test(self):
do_something()
QEMU_TEST_FLAKY_TESTS
^^^^^^^^^^^^^^^^^^^^^
Some tests are not working reliably and thus are disabled by default.
Set this environment variable to enable them.
This includes tests that don't run reliably on GitLab's CI which
usually expose real issues that are rarely seen on developer machines
due to the constraints of the CI environment. If you encounter a
similar situation then raise a bug and then mark the test as shown on
the code snippet below:
.. code::
# See https://gitlab.com/qemu-project/qemu/-/issues/nnnn
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
def test(self):
do_something()
You can also add ``:avocado: tags=flaky`` to the test meta-data so
only the flaky tests can be run as a group:
.. code::
env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/avocado \
run tests/avocado -filter-by-tags=flaky
Tests should not live in this state forever and should either be fixed
or eventually removed.
Uninstalling Avocado
~~~~~~~~~~~~~~~~~~~~

View File

@ -24,6 +24,7 @@ enum {
GDB_SIGNAL_TRAP = 5,
GDB_SIGNAL_ABRT = 6,
GDB_SIGNAL_ALRM = 14,
GDB_SIGNAL_STOP = 17,
GDB_SIGNAL_IO = 23,
GDB_SIGNAL_XCPU = 24,
GDB_SIGNAL_UNKNOWN = 143

View File

@ -183,7 +183,7 @@ static void gdb_vm_state_change(void *opaque, bool running, RunState state)
break;
case RUN_STATE_IO_ERROR:
trace_gdbstub_hit_io_error();
ret = GDB_SIGNAL_IO;
ret = GDB_SIGNAL_STOP;
break;
case RUN_STATE_WATCHDOG:
trace_gdbstub_hit_watchdog();

View File

@ -12,7 +12,7 @@ import os
from avocado_qemu import LinuxTest, BUILD_DIR
from avocado import skipIf
from avocado import skipUnless
class BootLinuxX8664(LinuxTest):
@ -93,11 +93,13 @@ class BootLinuxPPC64(LinuxTest):
timeout = 360
@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
def test_pseries_tcg(self):
"""
:avocado: tags=machine:pseries
:avocado: tags=accel:tcg
:avocado: tags=flaky
"""
self.require_accelerator("tcg")
self.vm.add_args("-accel", "tcg")
@ -111,11 +113,13 @@ class BootLinuxS390X(LinuxTest):
timeout = 240
@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
def test_s390_ccw_virtio_tcg(self):
"""
:avocado: tags=machine:s390-ccw-virtio
:avocado: tags=accel:tcg
:avocado: tags=flaky
"""
self.require_accelerator("tcg")
self.vm.add_args("-accel", "tcg")

View File

@ -15,7 +15,7 @@ import shutil
from avocado import skip
from avocado import skipUnless
from avocado import skipIf
from avocado import skipUnless
from avocado_qemu import QemuSystemTest
from avocado_qemu import exec_command
from avocado_qemu import exec_command_and_wait_for_pattern
@ -1419,11 +1419,13 @@ class BootLinuxConsole(LinuxKernelTest):
# This test has a 6-10% failure rate on various hosts that look
# like issues with a buggy kernel. As a result we don't want it
# gating releases on Gitlab.
@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
def test_sh4_r2d(self):
"""
:avocado: tags=arch:sh4
:avocado: tags=machine:r2d
:avocado: tags=flaky
"""
tar_hash = 'fe06a4fd8ccbf2e27928d64472939d47829d4c7e'
self.vm.add_args('-append', 'console=ttySC1')

View File

@ -9,10 +9,11 @@
# later. See the COPYING file in the top-level directory.
import os
from avocado import skipIf
from avocado import skipUnless
from avocado_qemu import LinuxTest
@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
class IntelIOMMU(LinuxTest):
"""
:avocado: tags=arch:x86_64
@ -21,6 +22,7 @@ class IntelIOMMU(LinuxTest):
:avocado: tags=machine:q35
:avocado: tags=accel:kvm
:avocado: tags=intel_iommu
:avocado: tags=flaky
"""
IOMMU_ADDON = ',iommu_platform=on,disable-modern=off,disable-legacy=on'

View File

@ -13,7 +13,7 @@ import logging
import tempfile
from avocado_qemu import QemuSystemTest
from avocado import skipIf
from avocado import skipUnless
class LinuxInitrd(QemuSystemTest):
@ -53,9 +53,12 @@ class LinuxInitrd(QemuSystemTest):
max_size + 1)
self.assertRegex(self.vm.get_log(), expected_msg)
@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
def test_with_2gib_file_should_work_with_linux_v4_16(self):
"""
:avocado: tags=flaky
QEMU has supported up to 4 GiB initrd for recent kernel
Expect guest can reach 'Unpacking initramfs...'
"""

View File

@ -20,6 +20,10 @@ class Aarch64SbsarefMachine(QemuSystemTest):
"""
:avocado: tags=arch:aarch64
:avocado: tags=machine:sbsa-ref
:avocado: tags=accel:tcg
As firmware runs at a higher privilege level than the hypervisor we
can only run these tests under TCG emulation.
"""
timeout = 180

View File

@ -18,7 +18,7 @@ from avocado_qemu import exec_command_and_wait_for_pattern
from avocado_qemu import interrupt_interactive_console_until_pattern
from avocado_qemu import has_cmd
from avocado.utils import archive
from avocado import skipIf
from avocado import skipUnless
from avocado import skipUnless
@ -311,11 +311,13 @@ class AST2x00MachineSDK(QemuSystemTest, LinuxSSHMixIn):
self, 'boot', '## Loading kernel from FIT Image')
self.wait_for_console_pattern('Starting kernel ...')
@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
def test_arm_ast2500_evb_sdk(self):
"""
:avocado: tags=arch:arm
:avocado: tags=machine:ast2500-evb
:avocado: tags=flaky
"""
image_url = ('https://github.com/AspeedTech-BMC/openbmc/releases/'
@ -329,11 +331,13 @@ class AST2x00MachineSDK(QemuSystemTest, LinuxSSHMixIn):
self.workdir + '/ast2500-default/image-bmc')
self.wait_for_console_pattern('nodistro.0 ast2500-default ttyS4')
@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
def test_arm_ast2600_evb_sdk(self):
"""
:avocado: tags=arch:arm
:avocado: tags=machine:ast2600-evb
:avocado: tags=flaky
"""
image_url = ('https://github.com/AspeedTech-BMC/openbmc/releases/'

View File

@ -11,7 +11,7 @@ import os
import gzip
import logging
from avocado import skipIf
from avocado import skipUnless
from avocado import skipUnless
from avocado.utils import archive
from avocado_qemu import QemuSystemTest
@ -101,23 +101,27 @@ class MaltaMachineFramebuffer(QemuSystemTest):
"""
self.do_test_i6400_framebuffer_logo(1)
@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
def test_mips_malta_i6400_framebuffer_logo_7cores(self):
"""
:avocado: tags=arch:mips64el
:avocado: tags=machine:malta
:avocado: tags=cpu:I6400
:avocado: tags=mips:smp
:avocado: tags=flaky
"""
self.do_test_i6400_framebuffer_logo(7)
@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
def test_mips_malta_i6400_framebuffer_logo_8cores(self):
"""
:avocado: tags=arch:mips64el
:avocado: tags=machine:malta
:avocado: tags=cpu:I6400
:avocado: tags=mips:smp
:avocado: tags=flaky
"""
self.do_test_i6400_framebuffer_logo(8)

View File

@ -10,7 +10,7 @@
import os
from avocado import skipIf
from avocado import skipUnless
from avocado_qemu import QemuSystemTest
from avocado_qemu import exec_command_and_wait_for_pattern
from avocado_qemu import wait_for_console_pattern
@ -22,7 +22,8 @@ class RxGdbSimMachine(QemuSystemTest):
timeout = 30
KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
def test_uboot(self):
"""
U-Boot and checks that the console is operational.
@ -30,6 +31,7 @@ class RxGdbSimMachine(QemuSystemTest):
:avocado: tags=arch:rx
:avocado: tags=machine:gdbsim-r5f562n8
:avocado: tags=endian:little
:avocado: tags=flaky
"""
uboot_url = ('https://acc.dl.osdn.jp/users/23/23888/u-boot.bin.gz')
uboot_hash = '9b78dbd43b40b2526848c0b1ce9de02c24f4dcdb'
@ -46,7 +48,8 @@ class RxGdbSimMachine(QemuSystemTest):
# FIXME limit baudrate on chardev, else we type too fast
#exec_command_and_wait_for_pattern(self, 'version', gcc_version)
@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
def test_linux_sash(self):
"""
Boots a Linux kernel and checks that the console is operational.
@ -54,6 +57,7 @@ class RxGdbSimMachine(QemuSystemTest):
:avocado: tags=arch:rx
:avocado: tags=machine:gdbsim-r5f562n7
:avocado: tags=endian:little
:avocado: tags=flaky
"""
dtb_url = ('https://acc.dl.osdn.jp/users/23/23887/rx-virt.dtb')
dtb_hash = '7b4e4e2c71905da44e86ce47adee2210b026ac18'

View File

@ -12,7 +12,7 @@
import os
import tempfile
from avocado import skipIf
from avocado import skipUnless
from avocado_qemu import QemuSystemTest
from avocado_qemu import exec_command_and_wait_for_pattern
from avocado_qemu import wait_for_console_pattern
@ -167,6 +167,7 @@ class S390CCWVirtioMachine(QemuSystemTest):
:avocado: tags=device:virtio-gpu
:avocado: tags=device:virtio-crypto
:avocado: tags=device:virtio-net
:avocado: tags=flaky
"""
kernel_url = ('https://archives.fedoraproject.org/pub/archive'

View File

@ -15,7 +15,7 @@ import logging
import time
from avocado import skip
from avocado import skipIf
from avocado import skipUnless
from avocado import skipUnless
from avocado_qemu import wait_for_console_pattern
from avocado.utils import archive
@ -82,11 +82,13 @@ class ReplayKernelBase(LinuxKernelTest):
class ReplayKernelNormal(ReplayKernelBase):
# See https://gitlab.com/qemu-project/qemu/-/issues/2010
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test sometimes gets stuck')
def test_x86_64_pc(self):
"""
:avocado: tags=arch:x86_64
:avocado: tags=machine:pc
:avocado: tags=flaky
"""
kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
'/linux/releases/29/Everything/x86_64/os/images/pxeboot'
@ -179,11 +181,13 @@ class ReplayKernelNormal(ReplayKernelBase):
self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=1)
@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
def test_arm_cubieboard_initrd(self):
"""
:avocado: tags=arch:arm
:avocado: tags=machine:cubieboard
:avocado: tags=flaky
"""
deb_url = ('https://apt.armbian.com/pool/main/l/'
'linux-5.10.16-sunxi/linux-image-current-sunxi_21.02.2_armhf.deb')

View File

@ -10,7 +10,7 @@
import os
import logging
from avocado import skipIf
from avocado import skipUnless
from avocado_qemu import BUILD_DIR
from avocado.utils import datadrainer
from avocado.utils import gdb
@ -206,7 +206,8 @@ class ReverseDebugging_X86_64(ReverseDebugging):
+ self.get_reg_le(g, self.REG_CS) * 0x10
# unidentified gitlab timeout problem
@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
def test_x86_64_pc(self):
"""
:avocado: tags=arch:x86_64
@ -223,7 +224,8 @@ class ReverseDebugging_AArch64(ReverseDebugging):
REG_PC = 32
# unidentified gitlab timeout problem
@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
def test_aarch64_virt(self):
"""
:avocado: tags=arch:aarch64
@ -247,11 +249,13 @@ class ReverseDebugging_ppc64(ReverseDebugging):
REG_PC = 0x40
# unidentified gitlab timeout problem
@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
def test_ppc64_pseries(self):
"""
:avocado: tags=arch:ppc64
:avocado: tags=machine:pseries
:avocado: tags=flaky
"""
# SLOF branches back to its entry point, which causes this test
# to take the 'hit a breakpoint again' path. That's not a problem,
@ -260,11 +264,13 @@ class ReverseDebugging_ppc64(ReverseDebugging):
self.reverse_debugging()
# See https://gitlab.com/qemu-project/qemu/-/issues/1992
@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
def test_ppc64_powernv(self):
"""
:avocado: tags=arch:ppc64
:avocado: tags=machine:powernv
:avocado: tags=flaky
"""
self.endian_is_le = False
self.reverse_debugging()

View File

@ -9,10 +9,11 @@
# later. See the COPYING file in the top-level directory.
import os
from avocado import skipIf
from avocado import skipUnless
from avocado_qemu import LinuxTest, BUILD_DIR
@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
class SMMU(LinuxTest):
"""
:avocado: tags=accel:kvm
@ -21,6 +22,7 @@ class SMMU(LinuxTest):
:avocado: tags=machine:virt
:avocado: tags=distro:fedora
:avocado: tags=smmu
:avocado: tags=flaky
"""
IOMMU_ADDON = ',iommu_platform=on,disable-modern=off,disable-legacy=on'

View File

@ -13,7 +13,7 @@ import os
import time
import tempfile
from avocado import skip, skipIf
from avocado import skip, skipUnless
from avocado_qemu import QemuSystemTest
from avocado_qemu import exec_command, exec_command_and_wait_for_pattern
from avocado_qemu import wait_for_console_pattern
@ -551,7 +551,7 @@ class TuxRunBaselineTest(QemuSystemTest):
haltmsg="Requesting system halt")
# Note: some segfaults caused by unaligned userspace access
@skipIf(os.getenv('GITLAB_CI'), 'Skipping unstable test on GitLab')
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
def test_sh4(self):
"""
:avocado: tags=arch:sh4
@ -561,6 +561,7 @@ class TuxRunBaselineTest(QemuSystemTest):
:avocado: tags=image:zImage
:avocado: tags=root:sda
:avocado: tags=console:ttySC1
:avocado: tags=flaky
"""
sums = { "rootfs.ext4.zst" :
"3592a7a3d5a641e8b9821449e77bc43c9904a56c30d45da0694349cfd86743fd",