qapi: Speed up frontend tests

"make check-qapi-schema" takes around 10s user + system time for me.
With -j, it takes a bit over 3s real time.  We have worse tests.  It's
still annoying when you work on the QAPI generator.

Some 1.4s user + system time is consumed by make figuring out what to
do, measured by making a target that does nothing.  There's nothing I
can do about that right now.  But let's see what we can do about the
other 8s.

Almost 7s are spent running test-qapi.py for every test case, the rest
normalizing and diffing test-qapi.py output.  We have 190 test cases.

If I downgrade to python2, it's 4.5s, but python2 is a goner.

Hacking up test-qapi.py to exit(0) without doing anything makes it
only marginally faster.  The problem is Python startup overhead.

Our configure puts -B into $(PYTHON).  Running without -B is faster:
4.4s.

We could improve the Makefile to run test cases only when the test
case or the generator changed.  But I'm after improvement in the case
where the generator changed.

test-qapi.py is designed to be the simplest possible building block
for a shell script to do the complete job (it's actually a Makefile,
not a shell script; no real difference).  Python is just not meant for
that.  It's for bigger blocks.

Move the post-processing and diffing into test-qapi.py, and make it
capable of testing multiple schema files.  Set executable bits while
there.

Running it once per test case now takes slightly longer than 8s.  But
running it once for all of them takes under 0.2s.

Messing with the Makefile to run it only on the tests that need
retesting is clearly not worth the bother.

Expected error output changes because the new normalization strips off
$(SRCDIR)/tests/qapi-schema/ instead of just $(SRCDIR)/.

The .exit files go away, because there is no exit status to test
anymore.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20191018074345.24034-5-armbru@redhat.com>
This commit is contained in:
Markus Armbruster 2019-10-18 09:43:42 +02:00
parent 0002b557b5
commit f01338cce6
364 changed files with 413 additions and 515 deletions

View File

@ -1102,17 +1102,11 @@ check-tests/check-block.sh: tests/check-block.sh qemu-img$(EXESUF) \
$(patsubst %,%/all,$(filter %-softmmu,$(TARGET_DIRS))) $(patsubst %,%/all,$(filter %-softmmu,$(TARGET_DIRS)))
@$< @$<
.PHONY: $(patsubst %, check-%, $(check-qapi-schema-y)) .PHONY: check-tests/qapi-schema/frontend
$(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/%.json check-tests/qapi-schema/frontend: $(addprefix $(SRC_PATH)/, $(check-qapi-schema-y))
$(call quiet-command, PYTHONPATH=$(SRC_PATH)/scripts \ $(call quiet-command, PYTHONPATH=$(SRC_PATH)/scripts \
PYTHONIOENCODING=utf-8 $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \ PYTHONIOENCODING=utf-8 $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py $^, \
$^ >$*.test.out 2>$*.test.err; \ TEST, check-qapi-schema)
echo $$? >$*.test.exit, \
"TEST","$*.out")
@# Sanitize error messages (make them independent of build directory)
@perl -p -e 's|\Q$(SRC_PATH)\E/||g' $*.test.err | diff -u $(SRC_PATH)/$*.err -
@diff -u $(SRC_PATH)/$*.out $*.test.out
@diff -u $(SRC_PATH)/$*.exit $*.test.exit
.PHONY: check-tests/qapi-schema/doc-good.texi .PHONY: check-tests/qapi-schema/doc-good.texi
check-tests/qapi-schema/doc-good.texi: tests/qapi-schema/doc-good.test.texi check-tests/qapi-schema/doc-good.texi: tests/qapi-schema/doc-good.test.texi
@ -1170,7 +1164,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR)
# Consolidated targets # Consolidated targets
.PHONY: check-block check-qapi-schema check-qtest check-unit check check-clean .PHONY: check-block check-qapi-schema check-qtest check-unit check check-clean
check-qapi-schema: $(patsubst %,check-%, $(check-qapi-schema-y)) check-tests/qapi-schema/doc-good.texi check-qapi-schema: check-tests/qapi-schema/frontend check-tests/qapi-schema/doc-good.texi
check-qtest: $(patsubst %,check-qtest-%, $(QTEST_TARGETS)) check-qtest: $(patsubst %,check-qtest-%, $(QTEST_TARGETS))
check-block: $(patsubst %,check-%, $(check-block-y)) check-block: $(patsubst %,check-%, $(check-block-y))
check: check-block check-qapi-schema check-unit check-softfloat check-qtest check-decodetree check: check-block check-qapi-schema check-unit check-softfloat check-qtest check-decodetree

View File

@ -1,2 +1,2 @@
tests/qapi-schema/allow-preconfig-test.json: In command 'allow-preconfig-test': allow-preconfig-test.json: In command 'allow-preconfig-test':
tests/qapi-schema/allow-preconfig-test.json:2: flag 'allow-preconfig' may only use true value allow-preconfig-test.json:2: flag 'allow-preconfig' may only use true value

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/alternate-any.json: In alternate 'Alt': alternate-any.json: In alternate 'Alt':
tests/qapi-schema/alternate-any.json:2: branch 'one' cannot use built-in type 'any' alternate-any.json:2: branch 'one' cannot use built-in type 'any'

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/alternate-array.json: In alternate 'Alt': alternate-array.json: In alternate 'Alt':
tests/qapi-schema/alternate-array.json:5: 'data' member 'two' cannot be an array alternate-array.json:5: 'data' member 'two' cannot be an array

View File

@ -1 +0,0 @@
1

View File

@ -1,3 +1,3 @@
tests/qapi-schema/alternate-base.json: In alternate 'Alt': alternate-base.json: In alternate 'Alt':
tests/qapi-schema/alternate-base.json:4: alternate has unknown key 'base' alternate-base.json:4: alternate has unknown key 'base'
Valid keys are 'alternate', 'data', 'if'. Valid keys are 'alternate', 'data', 'if'.

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/alternate-branch-if-invalid.json: In alternate 'Alt': alternate-branch-if-invalid.json: In alternate 'Alt':
tests/qapi-schema/alternate-branch-if-invalid.json:2: 'if' condition ' ' of 'data' member 'branch' makes no sense alternate-branch-if-invalid.json:2: 'if' condition ' ' of 'data' member 'branch' makes no sense

View File

@ -1,2 +1,2 @@
tests/qapi-schema/alternate-clash.json: In alternate 'Alt1': alternate-clash.json: In alternate 'Alt1':
tests/qapi-schema/alternate-clash.json:7: branch 'a_b' collides with branch 'a-b' alternate-clash.json:7: branch 'a_b' collides with branch 'a-b'

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/alternate-conflict-bool-string.json: In alternate 'Alt': alternate-conflict-bool-string.json: In alternate 'Alt':
tests/qapi-schema/alternate-conflict-bool-string.json:2: branch 'two' can't be distinguished from 'one' alternate-conflict-bool-string.json:2: branch 'two' can't be distinguished from 'one'

View File

@ -1,2 +1,2 @@
tests/qapi-schema/alternate-conflict-dict.json: In alternate 'Alt': alternate-conflict-dict.json: In alternate 'Alt':
tests/qapi-schema/alternate-conflict-dict.json:6: branch 'two' can't be distinguished from 'one' alternate-conflict-dict.json:6: branch 'two' can't be distinguished from 'one'

View File

@ -1,2 +1,2 @@
tests/qapi-schema/alternate-conflict-enum-bool.json: In alternate 'Alt': alternate-conflict-enum-bool.json: In alternate 'Alt':
tests/qapi-schema/alternate-conflict-enum-bool.json:4: branch 'two' can't be distinguished from 'one' alternate-conflict-enum-bool.json:4: branch 'two' can't be distinguished from 'one'

View File

@ -1,2 +1,2 @@
tests/qapi-schema/alternate-conflict-enum-int.json: In alternate 'Alt': alternate-conflict-enum-int.json: In alternate 'Alt':
tests/qapi-schema/alternate-conflict-enum-int.json:4: branch 'two' can't be distinguished from 'one' alternate-conflict-enum-int.json:4: branch 'two' can't be distinguished from 'one'

View File

@ -1,2 +1,2 @@
tests/qapi-schema/alternate-conflict-num-string.json: In alternate 'Alt': alternate-conflict-num-string.json: In alternate 'Alt':
tests/qapi-schema/alternate-conflict-num-string.json:2: branch 'two' can't be distinguished from 'one' alternate-conflict-num-string.json:2: branch 'two' can't be distinguished from 'one'

View File

@ -1,2 +1,2 @@
tests/qapi-schema/alternate-conflict-string.json: In alternate 'Alt': alternate-conflict-string.json: In alternate 'Alt':
tests/qapi-schema/alternate-conflict-string.json:2: branch 'two' can't be distinguished from 'one' alternate-conflict-string.json:2: branch 'two' can't be distinguished from 'one'

View File

@ -1,2 +1,2 @@
tests/qapi-schema/alternate-empty.json: In alternate 'Alt': alternate-empty.json: In alternate 'Alt':
tests/qapi-schema/alternate-empty.json:2: 'data' must not be empty alternate-empty.json:2: 'data' must not be empty

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/alternate-invalid-dict.json: In alternate 'Alt': alternate-invalid-dict.json: In alternate 'Alt':
tests/qapi-schema/alternate-invalid-dict.json:2: 'data' member 'two' misses key 'type' alternate-invalid-dict.json:2: 'data' member 'two' misses key 'type'

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/alternate-nested.json: In alternate 'Alt2': alternate-nested.json: In alternate 'Alt2':
tests/qapi-schema/alternate-nested.json:4: branch 'nested' cannot use alternate type 'Alt1' alternate-nested.json:4: branch 'nested' cannot use alternate type 'Alt1'

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/alternate-unknown.json: In alternate 'Alt': alternate-unknown.json: In alternate 'Alt':
tests/qapi-schema/alternate-unknown.json:2: branch 'unknown' uses unknown type 'MissingType' alternate-unknown.json:2: branch 'unknown' uses unknown type 'MissingType'

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/args-alternate.json: In command 'oops': args-alternate.json: In command 'oops':
tests/qapi-schema/args-alternate.json:3: command's 'data' cannot take alternate type 'Alt' args-alternate.json:3: command's 'data' cannot take alternate type 'Alt'

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/args-any.json: In command 'oops': args-any.json: In command 'oops':
tests/qapi-schema/args-any.json:2: command's 'data' cannot take built-in type 'any' args-any.json:2: command's 'data' cannot take built-in type 'any'

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/args-array-empty.json: In command 'oops': args-array-empty.json: In command 'oops':
tests/qapi-schema/args-array-empty.json:2: 'data' member 'empty': array type must contain single type name args-array-empty.json:2: 'data' member 'empty': array type must contain single type name

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/args-array-unknown.json: In command 'oops': args-array-unknown.json: In command 'oops':
tests/qapi-schema/args-array-unknown.json:2: command uses unknown type 'NoSuchType' args-array-unknown.json:2: command uses unknown type 'NoSuchType'

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/args-bad-boxed.json: In command 'foo': args-bad-boxed.json: In command 'foo':
tests/qapi-schema/args-bad-boxed.json:2: flag 'boxed' may only use true value args-bad-boxed.json:2: flag 'boxed' may only use true value

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/args-boxed-anon.json: In command 'foo': args-boxed-anon.json: In command 'foo':
tests/qapi-schema/args-boxed-anon.json:2: 'data' should be a type name args-boxed-anon.json:2: 'data' should be a type name

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/args-boxed-string.json: In command 'foo': args-boxed-string.json: In command 'foo':
tests/qapi-schema/args-boxed-string.json:2: command's 'data' cannot take built-in type 'str' args-boxed-string.json:2: command's 'data' cannot take built-in type 'str'

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/args-int.json: In command 'oops': args-int.json: In command 'oops':
tests/qapi-schema/args-int.json:2: command's 'data' cannot take built-in type 'int' args-int.json:2: command's 'data' cannot take built-in type 'int'

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/args-invalid.json: In command 'foo': args-invalid.json: In command 'foo':
tests/qapi-schema/args-invalid.json:1: 'data' should be an object or type name args-invalid.json:1: 'data' should be an object or type name

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/args-member-array-bad.json: In command 'oops': args-member-array-bad.json: In command 'oops':
tests/qapi-schema/args-member-array-bad.json:2: 'data' member 'member': array type must contain single type name args-member-array-bad.json:2: 'data' member 'member': array type must contain single type name

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/args-member-case.json: In command 'no-way-this-will-get-whitelisted': args-member-case.json: In command 'no-way-this-will-get-whitelisted':
tests/qapi-schema/args-member-case.json:2: 'data' member 'Arg' uses uppercase in name args-member-case.json:2: 'data' member 'Arg' uses uppercase in name

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/args-member-unknown.json: In command 'oops': args-member-unknown.json: In command 'oops':
tests/qapi-schema/args-member-unknown.json:2: parameter 'member' uses unknown type 'NoSuchType' args-member-unknown.json:2: parameter 'member' uses unknown type 'NoSuchType'

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/args-name-clash.json: In command 'oops': args-name-clash.json: In command 'oops':
tests/qapi-schema/args-name-clash.json:4: parameter 'a_b' collides with parameter 'a-b' args-name-clash.json:4: parameter 'a_b' collides with parameter 'a-b'

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/args-union.json: In command 'oops': args-union.json: In command 'oops':
tests/qapi-schema/args-union.json:3: command's 'data' can take union type 'Uni' only with 'boxed': true args-union.json:3: command's 'data' can take union type 'Uni' only with 'boxed': true

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/args-unknown.json: In command 'oops': args-unknown.json: In command 'oops':
tests/qapi-schema/args-unknown.json:2: command's 'data' uses unknown type 'NoSuchType' args-unknown.json:2: command's 'data' uses unknown type 'NoSuchType'

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/bad-base.json: In struct 'MyType': bad-base.json: In struct 'MyType':
tests/qapi-schema/bad-base.json:3: 'base' requires a struct type, union type 'Union' isn't bad-base.json:3: 'base' requires a struct type, union type 'Union' isn't

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/bad-data.json: In command 'oops': bad-data.json: In command 'oops':
tests/qapi-schema/bad-data.json:2: 'data' cannot be an array bad-data.json:2: 'data' cannot be an array

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/bad-ident.json: In struct '*oops': bad-ident.json: In struct '*oops':
tests/qapi-schema/bad-ident.json:2: struct has an invalid name bad-ident.json:2: struct has an invalid name

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/bad-if-empty-list.json: In struct 'TestIfStruct': bad-if-empty-list.json: In struct 'TestIfStruct':
tests/qapi-schema/bad-if-empty-list.json:2: 'if' condition [] of struct is useless bad-if-empty-list.json:2: 'if' condition [] of struct is useless

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/bad-if-empty.json: In struct 'TestIfStruct': bad-if-empty.json: In struct 'TestIfStruct':
tests/qapi-schema/bad-if-empty.json:2: 'if' condition '' of struct makes no sense bad-if-empty.json:2: 'if' condition '' of struct makes no sense

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/bad-if-list.json: In struct 'TestIfStruct': bad-if-list.json: In struct 'TestIfStruct':
tests/qapi-schema/bad-if-list.json:2: 'if' condition ' ' of struct makes no sense bad-if-list.json:2: 'if' condition ' ' of struct makes no sense

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/bad-if.json: In struct 'TestIfStruct': bad-if.json: In struct 'TestIfStruct':
tests/qapi-schema/bad-if.json:2: 'if' condition of struct must be a string or a list of strings bad-if.json:2: 'if' condition of struct must be a string or a list of strings

View File

@ -1 +0,0 @@
1

View File

@ -1 +1 @@
tests/qapi-schema/bad-type-bool.json:2: 'struct' requires a string name bad-type-bool.json:2: 'struct' requires a string name

View File

@ -1 +0,0 @@
1

View File

@ -1 +1 @@
tests/qapi-schema/bad-type-dict.json:2: 'command' requires a string name bad-type-dict.json:2: 'command' requires a string name

View File

@ -1 +0,0 @@
1

View File

@ -1 +1 @@
tests/qapi-schema/bad-type-int.json:3:13: stray '123' bad-type-int.json:3:13: stray '123'

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/base-cycle-direct.json: In struct 'Loopy': base-cycle-direct.json: In struct 'Loopy':
tests/qapi-schema/base-cycle-direct.json:2: object Loopy contains itself base-cycle-direct.json:2: object Loopy contains itself

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/base-cycle-indirect.json: In struct 'Base1': base-cycle-indirect.json: In struct 'Base1':
tests/qapi-schema/base-cycle-indirect.json:2: object Base1 contains itself base-cycle-indirect.json:2: object Base1 contains itself

View File

@ -1 +0,0 @@
1

View File

@ -1,2 +1,2 @@
tests/qapi-schema/command-int.json: In command 'int': command-int.json: In command 'int':
tests/qapi-schema/command-int.json:2: built-in type 'int' is already defined command-int.json:2: built-in type 'int' is already defined

View File

@ -1 +0,0 @@
1

View File

@ -1 +0,0 @@
0

View File

@ -1 +1 @@
tests/qapi-schema/doc-bad-alternate-member.json:3: the following documented members are not in the declaration: aa, bb doc-bad-alternate-member.json:3: the following documented members are not in the declaration: aa, bb

View File

@ -1 +1 @@
tests/qapi-schema/doc-bad-command-arg.json:3: the following documented members are not in the declaration: b doc-bad-command-arg.json:3: the following documented members are not in the declaration: b

View File

@ -1 +0,0 @@
1

View File

@ -1 +0,0 @@
0

View File

@ -1,2 +1,2 @@
tests/qapi-schema/doc-bad-symbol.json: In command 'foo': doc-bad-symbol.json: In command 'foo':
tests/qapi-schema/doc-bad-symbol.json:6: documentation comment is for 'food' doc-bad-symbol.json:6: documentation comment is for 'food'

View File

@ -1 +0,0 @@
1

View File

@ -1 +1 @@
tests/qapi-schema/doc-bad-union-member.json:3: the following documented members are not in the declaration: a, b doc-bad-union-member.json:3: the following documented members are not in the declaration: a, b

View File

@ -1 +0,0 @@
1

View File

@ -1 +1 @@
tests/qapi-schema/doc-before-include.json:3: documentation for 'foo' is not followed by the definition doc-before-include.json:3: documentation for 'foo' is not followed by the definition

Some files were not shown because too many files have changed in this diff Show More