Add Catch2 unit testing framework + some tests.

Add the Catch2 headers to third_party/include/catch2.

Add Catch.cmake and CatchAddTests.cmake to cmake/.

Add unit tests src/wx/tests/strutils.cpp using Catch2 to test
src/wx/strutils.cpp.

Make some code changes to strutils.cpp to make the tests pass.

See src/wx/tests/CMakeLists.txt for how to set up unit test files; they
plug into the normal CTest mechanism in cmake.

The test binaries are written to the tests/ subdirectory of the build
directory.

Building the tests and enabling the CTest support can be turned off by
passing -DBUILD_TESTING=OFF to cmake, the default is ON, this is the
standard mechanism.

Start running ctest on travis and appveyor.

Move stb to third_party/include where Catch2 is now as well.

Signed-off-by: Rafael Kitover <rkitover@gmail.com>
This commit is contained in:
Rafael Kitover 2020-07-10 13:40:05 +00:00
parent 3ba0529585
commit 81c88227fa
19 changed files with 18891 additions and 33 deletions

View File

@ -39,14 +39,9 @@ build_script:
- ninja
# only debug builds are console mode apps, in them test --help
for:
-
matrix:
except:
- configuration: Release
test_script:
- .\visualboyadvance-m.exe --help
test_script:
- if not %CONFIGURATION%==Release .\visualboyadvance-m.exe --help
- ctest -V
cache:
- c:\vcpkg\installed

View File

@ -85,6 +85,7 @@ script:
elif [ -n "$XVFB_RUN" ]; then
xvfb-run -a ./$PRGNAM --help
fi
ctest -V
env:
global:

View File

@ -13,6 +13,10 @@ if(COMMAND cmake_policy)
if(POLICY CMP0043)
cmake_policy(SET CMP0043 NEW) # for wxWidgets, use generator expressions
endif()
if(POLICY CMP0048)
cmake_policy(SET CMP0048 NEW) # set CMAKE_PROJECT_VERSION*
endif()
endif()
set(CMAKE_MODULE_PATH ${CMAKE_SOURCE_DIR}/cmake)
@ -54,6 +58,12 @@ if(CMAKE_BUILD_TYPE STREQUAL "")
set(CMAKE_BUILD_TYPE "Release" CACHE STRING "Build Type" FORCE)
endif()
include(CTest)
if(BUILD_TESTING)
enable_testing()
endif()
set(ALL_TARGETS fex visualboyadvance-m vbamcore vbam)
add_custom_target(generate)
@ -1012,14 +1022,15 @@ endif()
set(
HDR_STB_IMAGE
headers/stb/stb_image.h
headers/stb/stb_image_write.h
third_party/include/stb/stb_image.h
third_party/include/stb/stb_image_write.h
)
include_directories(
${ZLIB_INCLUDE_DIR}
fex
${SDL2_INCLUDE_DIR}
third_party/include/stb
)
if(ENABLE_FFMPEG)
@ -1130,3 +1141,5 @@ set(CPACK_PACKAGE_VERSION_MINOR "0")
set(CPACK_PACKAGE_VERSION_PATCH "0-Git-" ${COMMITHASH})
list(APPEND CPACK_SOURCE_IGNORE_FILES "${CMAKE_CURRENT_SOURCE_DIR}/dependencies")
include(CPack)
# vim:sw=4 et tw=0:

View File

@ -192,6 +192,7 @@ Here is the complete list:
| ENABLE_SSP | Enable gcc stack protector support (gcc only) | OFF |
| ENABLE_ASAN | Enable libasan sanitizers (by default address, only in debug mode) | OFF |
| UPSTREAM_RELEASE | Do some release tasks, like codesigning, making zip and gpg sigs. | OFF |
| BUILD_TESTING | Build the tests and enable ctest support. | ON |
| VBAM_STATIC | Try link all libs statically (the following are set to ON if ON) | OFF |
| SDL2_STATIC | Try to link static SDL2 libraries | OFF |
| SFML_STATIC_LIBRARIES | Try to link static SFML libraries | OFF |

175
cmake/Catch.cmake Normal file
View File

@ -0,0 +1,175 @@
# Distributed under the OSI-approved BSD 3-Clause License. See accompanying
# file Copyright.txt or https://cmake.org/licensing for details.
#[=======================================================================[.rst:
Catch
-----
This module defines a function to help use the Catch test framework.
The :command:`catch_discover_tests` discovers tests by asking the compiled test
executable to enumerate its tests. This does not require CMake to be re-run
when tests change. However, it may not work in a cross-compiling environment,
and setting test properties is less convenient.
This command is intended to replace use of :command:`add_test` to register
tests, and will create a separate CTest test for each Catch test case. Note
that this is in some cases less efficient, as common set-up and tear-down logic
cannot be shared by multiple test cases executing in the same instance.
However, it provides more fine-grained pass/fail information to CTest, which is
usually considered as more beneficial. By default, the CTest test name is the
same as the Catch name; see also ``TEST_PREFIX`` and ``TEST_SUFFIX``.
.. command:: catch_discover_tests
Automatically add tests with CTest by querying the compiled test executable
for available tests::
catch_discover_tests(target
[TEST_SPEC arg1...]
[EXTRA_ARGS arg1...]
[WORKING_DIRECTORY dir]
[TEST_PREFIX prefix]
[TEST_SUFFIX suffix]
[PROPERTIES name1 value1...]
[TEST_LIST var]
)
``catch_discover_tests`` sets up a post-build command on the test executable
that generates the list of tests by parsing the output from running the test
with the ``--list-test-names-only`` argument. This ensures that the full
list of tests is obtained. Since test discovery occurs at build time, it is
not necessary to re-run CMake when the list of tests changes.
However, it requires that :prop_tgt:`CROSSCOMPILING_EMULATOR` is properly set
in order to function in a cross-compiling environment.
Additionally, setting properties on tests is somewhat less convenient, since
the tests are not available at CMake time. Additional test properties may be
assigned to the set of tests as a whole using the ``PROPERTIES`` option. If
more fine-grained test control is needed, custom content may be provided
through an external CTest script using the :prop_dir:`TEST_INCLUDE_FILES`
directory property. The set of discovered tests is made accessible to such a
script via the ``<target>_TESTS`` variable.
The options are:
``target``
Specifies the Catch executable, which must be a known CMake executable
target. CMake will substitute the location of the built executable when
running the test.
``TEST_SPEC arg1...``
Specifies test cases, wildcarded test cases, tags and tag expressions to
pass to the Catch executable with the ``--list-test-names-only`` argument.
``EXTRA_ARGS arg1...``
Any extra arguments to pass on the command line to each test case.
``WORKING_DIRECTORY dir``
Specifies the directory in which to run the discovered test cases. If this
option is not provided, the current binary directory is used.
``TEST_PREFIX prefix``
Specifies a ``prefix`` to be prepended to the name of each discovered test
case. This can be useful when the same test executable is being used in
multiple calls to ``catch_discover_tests()`` but with different
``TEST_SPEC`` or ``EXTRA_ARGS``.
``TEST_SUFFIX suffix``
Similar to ``TEST_PREFIX`` except the ``suffix`` is appended to the name of
every discovered test case. Both ``TEST_PREFIX`` and ``TEST_SUFFIX`` may
be specified.
``PROPERTIES name1 value1...``
Specifies additional properties to be set on all tests discovered by this
invocation of ``catch_discover_tests``.
``TEST_LIST var``
Make the list of tests available in the variable ``var``, rather than the
default ``<target>_TESTS``. This can be useful when the same test
executable is being used in multiple calls to ``catch_discover_tests()``.
Note that this variable is only available in CTest.
#]=======================================================================]
#------------------------------------------------------------------------------
function(catch_discover_tests TARGET)
cmake_parse_arguments(
""
""
"TEST_PREFIX;TEST_SUFFIX;WORKING_DIRECTORY;TEST_LIST"
"TEST_SPEC;EXTRA_ARGS;PROPERTIES"
${ARGN}
)
if(NOT _WORKING_DIRECTORY)
set(_WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}")
endif()
if(NOT _TEST_LIST)
set(_TEST_LIST ${TARGET}_TESTS)
endif()
## Generate a unique name based on the extra arguments
string(SHA1 args_hash "${_TEST_SPEC} ${_EXTRA_ARGS}")
string(SUBSTRING ${args_hash} 0 7 args_hash)
# Define rule to generate test list for aforementioned test executable
set(ctest_include_file "${CMAKE_CURRENT_BINARY_DIR}/${TARGET}_include-${args_hash}.cmake")
set(ctest_tests_file "${CMAKE_CURRENT_BINARY_DIR}/${TARGET}_tests-${args_hash}.cmake")
get_property(crosscompiling_emulator
TARGET ${TARGET}
PROPERTY CROSSCOMPILING_EMULATOR
)
add_custom_command(
TARGET ${TARGET} POST_BUILD
BYPRODUCTS "${ctest_tests_file}"
COMMAND "${CMAKE_COMMAND}"
-D "TEST_TARGET=${TARGET}"
-D "TEST_EXECUTABLE=$<TARGET_FILE:${TARGET}>"
-D "TEST_EXECUTOR=${crosscompiling_emulator}"
-D "TEST_WORKING_DIR=${_WORKING_DIRECTORY}"
-D "TEST_SPEC=${_TEST_SPEC}"
-D "TEST_EXTRA_ARGS=${_EXTRA_ARGS}"
-D "TEST_PROPERTIES=${_PROPERTIES}"
-D "TEST_PREFIX=${_TEST_PREFIX}"
-D "TEST_SUFFIX=${_TEST_SUFFIX}"
-D "TEST_LIST=${_TEST_LIST}"
-D "CTEST_FILE=${ctest_tests_file}"
-P "${_CATCH_DISCOVER_TESTS_SCRIPT}"
VERBATIM
)
file(WRITE "${ctest_include_file}"
"if(EXISTS \"${ctest_tests_file}\")\n"
" include(\"${ctest_tests_file}\")\n"
"else()\n"
" add_test(${TARGET}_NOT_BUILT-${args_hash} ${TARGET}_NOT_BUILT-${args_hash})\n"
"endif()\n"
)
if(NOT ${CMAKE_VERSION} VERSION_LESS "3.10.0")
# Add discovered tests to directory TEST_INCLUDE_FILES
set_property(DIRECTORY
APPEND PROPERTY TEST_INCLUDE_FILES "${ctest_include_file}"
)
else()
# Add discovered tests as directory TEST_INCLUDE_FILE if possible
get_property(test_include_file_set DIRECTORY PROPERTY TEST_INCLUDE_FILE SET)
if (NOT ${test_include_file_set})
set_property(DIRECTORY
PROPERTY TEST_INCLUDE_FILE "${ctest_include_file}"
)
else()
message(FATAL_ERROR
"Cannot set more than one TEST_INCLUDE_FILE"
)
endif()
endif()
endfunction()
###############################################################################
set(_CATCH_DISCOVER_TESTS_SCRIPT
${CMAKE_CURRENT_LIST_DIR}/CatchAddTests.cmake
)

81
cmake/CatchAddTests.cmake Normal file
View File

@ -0,0 +1,81 @@
# Distributed under the OSI-approved BSD 3-Clause License. See accompanying
# file Copyright.txt or https://cmake.org/licensing for details.
set(prefix "${TEST_PREFIX}")
set(suffix "${TEST_SUFFIX}")
set(spec ${TEST_SPEC})
set(extra_args ${TEST_EXTRA_ARGS})
set(properties ${TEST_PROPERTIES})
set(script)
set(suite)
set(tests)
function(add_command NAME)
set(_args "")
foreach(_arg ${ARGN})
if(_arg MATCHES "[^-./:a-zA-Z0-9_]")
set(_args "${_args} [==[${_arg}]==]") # form a bracket_argument
else()
set(_args "${_args} ${_arg}")
endif()
endforeach()
set(script "${script}${NAME}(${_args})\n" PARENT_SCOPE)
endfunction()
# Run test executable to get list of available tests
if(NOT EXISTS "${TEST_EXECUTABLE}")
message(FATAL_ERROR
"Specified test executable '${TEST_EXECUTABLE}' does not exist"
)
endif()
execute_process(
COMMAND ${TEST_EXECUTOR} "${TEST_EXECUTABLE}" ${spec} --list-test-names-only
OUTPUT_VARIABLE output
RESULT_VARIABLE result
)
# Catch --list-test-names-only reports the number of tests, so 0 is... surprising
if(${result} EQUAL 0)
message(WARNING
"Test executable '${TEST_EXECUTABLE}' contains no tests!\n"
)
elseif(${result} LESS 0)
message(FATAL_ERROR
"Error running test executable '${TEST_EXECUTABLE}':\n"
" Result: ${result}\n"
" Output: ${output}\n"
)
endif()
string(REPLACE "\n" ";" output "${output}")
# Parse output
foreach(line ${output})
set(test ${line})
# Escape characters in test case names that would be parsed by Catch2
set(test_name ${test})
foreach(char , [ ])
string(REPLACE ${char} "\\${char}" test_name ${test_name})
endforeach(char)
# ...and add to script
add_command(add_test
"${prefix}${test}${suffix}"
${TEST_EXECUTOR}
"${TEST_EXECUTABLE}"
"${test_name}"
${extra_args}
)
add_command(set_tests_properties
"${prefix}${test}${suffix}"
PROPERTIES
WORKING_DIRECTORY "${TEST_WORKING_DIR}"
${properties}
)
list(APPEND tests "${prefix}${test}${suffix}")
endforeach()
# Create a list of all discovered tests, which users may use to e.g. set
# properties on the tests
add_command(set ${TEST_LIST} ${tests})
# Write CTest script
file(WRITE "${CTEST_FILE}" "${script}")

View File

@ -16,13 +16,13 @@
#define STB_IMAGE_IMPLEMENTATION
extern "C" {
#include "../headers/stb/stb_image.h"
#include "stb_image.h"
}
#define STBI_MSC_SECURE_CRT
#define STB_IMAGE_WRITE_IMPLEMENTATION
extern "C" {
#include "../headers/stb/stb_image_write.h"
#include "stb_image_write.h"
}
#include "NLS.h"

View File

@ -1288,4 +1288,8 @@ install(
BUNDLE DESTINATION ${CMAKE_INSTALL_BINDIR}
)
# vim:sw=4 tw=0
if(BUILD_TESTING AND (NOT CMAKE_CROSSCOMPILING))
add_subdirectory(tests)
endif()
# vim:sw=4 et tw=0:

View File

@ -3,8 +3,8 @@
// From: https://stackoverflow.com/a/7408245/262458
//
// modified to ignore empty tokens
wxArrayString str_split(const wxString& text, const wxString& sep) {
// Modified to ignore empty tokens or return sep for them.
wxArrayString str_split(const wxString& text, const wxString& sep, bool empty_token_is_sep) {
wxArrayString tokens;
size_t start = 0, end = 0;
@ -13,32 +13,25 @@ wxArrayString str_split(const wxString& text, const wxString& sep) {
if (token.length())
tokens.Add(token);
else if (empty_token_is_sep)
tokens.Add(sep);
start = end + 1;
start = end + sep.length();
}
tokens.Add(text.substr(start));
// Last token.
wxString token = text.substr(start);
if (token.length())
tokens.Add(token);
else if (empty_token_is_sep)
tokens.Add(sep);
return tokens;
}
wxArrayString str_split_with_sep(const wxString& text, const wxString& sep)
{
wxArrayString tokens;
bool sepIsTokenToo = false;
wxStringTokenizer tokenizer(text, sep, wxTOKEN_RET_EMPTY_ALL);
while (tokenizer.HasMoreTokens()) {
wxString token = tokenizer.GetNextToken();
if (token.IsEmpty()) {
if (!sepIsTokenToo) {
sepIsTokenToo = true;
tokens.Add(sep);
}
continue;
}
tokens.Add(token);
}
return tokens;
return str_split(text, sep, true);
}
size_t vec_find(wxArrayString& opts, const wxString& val) {

View File

@ -1,11 +1,14 @@
#ifndef STRUTILS_H
#define STRUTILS_H
#define __STDC_LIMIT_MACROS
#define __STDC_CONSTANT_MACROS
#include <wx/string.h>
#include <wx/arrstr.h>
// From: https://stackoverflow.com/a/7408245/262458
wxArrayString str_split(const wxString& text, const wxString& sep);
wxArrayString str_split(const wxString& text, const wxString& sep, bool empty_token_is_sep=false);
// Same as above, but it includes the sep dir.
// If "A,,,B" is the text and "," is sep, then

View File

@ -0,0 +1,22 @@
include(Catch)
include_directories("${CMAKE_SOURCE_DIR}/third_party/include/catch2")
function(add_catch2_test test_src)
string(REGEX REPLACE ".cpp$" "" test_name "${test_src}")
add_executable("${test_name}" "${ARGV}")
target_link_libraries("${test_name}" ${wxWidgets_LIBRARIES})
target_include_directories("${test_name}" PRIVATE ${wxWidgets_INCLUDE_DIRS})
set_target_properties("${test_name}"
PROPERTIES
RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/tests"
)
catch_discover_tests("${test_name}")
endfunction()
add_catch2_test(strutils.cpp ../strutils.h ../strutils.cpp)

97
src/wx/tests/strutils.cpp Normal file
View File

@ -0,0 +1,97 @@
#define CATCH_CONFIG_MAIN
#include "strutils.h"
#include "catch.hpp"
TEST_CASE("str_split() basic test", "[str_split]") {
wxString foo = "foo|bar|baz";
auto vec = str_split(foo, '|');
REQUIRE(vec.size() == 3);
REQUIRE(vec[0] == "foo");
REQUIRE(vec[1] == "bar");
REQUIRE(vec[2] == "baz");
}
TEST_CASE("str_split() multi-char separator", "[str_split]") {
wxString foo = "foo|-|bar|-|baz";
auto vec = str_split(foo, "|-|");
REQUIRE(vec.size() == 3);
REQUIRE(vec[0] == "foo");
REQUIRE(vec[1] == "bar");
REQUIRE(vec[2] == "baz");
}
TEST_CASE("str_split() skips empty tokens", "[str_split]") {
wxString foo = "|-|foo|-||-|bar|-|baz|-|";
auto vec = str_split(foo, "|-|");
REQUIRE(vec.size() == 3);
REQUIRE(vec[0] == "foo");
REQUIRE(vec[1] == "bar");
REQUIRE(vec[2] == "baz");
}
TEST_CASE("str_split() empty input", "[str_split]") {
wxString foo;
auto vec = str_split(foo, "|-|");
REQUIRE(vec.size() == 0);
}
TEST_CASE("str_split() no tokens, just separators", "[str_split]") {
wxString foo = "|-||-||-||-||-|";
auto vec = str_split(foo, "|-|");
REQUIRE(vec.size() == 0);
}
TEST_CASE("str_split_with_sep() basic test", "[str_split_with_sep]") {
wxString foo = "foo|bar|baz|";
auto vec = str_split_with_sep(foo, '|');
REQUIRE(vec.size() == 4);
REQUIRE(vec[0] == "foo");
REQUIRE(vec[1] == "bar");
REQUIRE(vec[2] == "baz");
REQUIRE(vec[3] == "|");
}
TEST_CASE("str_split_with_sep() multi-char sep", "[str_split_with_sep]") {
wxString foo = "foo|-|bar|-|baz|-|";
auto vec = str_split_with_sep(foo, "|-|");
REQUIRE(vec.size() == 4);
REQUIRE(vec[0] == "foo");
REQUIRE(vec[1] == "bar");
REQUIRE(vec[2] == "baz");
REQUIRE(vec[3] == "|-|");
}
TEST_CASE("str_split_with_sep() multiple sep tokens", "[str_split_with_sep]") {
wxString foo = "|-|foo|-||-|bar|-|baz|-|";
auto vec = str_split_with_sep(foo, "|-|");
REQUIRE(vec.size() == 6);
REQUIRE(vec[0] == "|-|");
REQUIRE(vec[1] == "foo");
REQUIRE(vec[2] == "|-|");
REQUIRE(vec[3] == "bar");
REQUIRE(vec[4] == "baz");
REQUIRE(vec[5] == "|-|");
}

17758
third_party/include/catch2/catch.hpp vendored Normal file

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,62 @@
/*
* Created by Justin R. Wilson on 2/19/2017.
* Copyright 2017 Justin R. Wilson. All rights reserved.
*
* Distributed under the Boost Software License, Version 1.0. (See accompanying
* file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
*/
#ifndef TWOBLUECUBES_CATCH_REPORTER_AUTOMAKE_HPP_INCLUDED
#define TWOBLUECUBES_CATCH_REPORTER_AUTOMAKE_HPP_INCLUDED
// Don't #include any Catch headers here - we can assume they are already
// included before this header.
// This is not good practice in general but is necessary in this case so this
// file can be distributed as a single header that works with the main
// Catch single header.
namespace Catch {
struct AutomakeReporter : StreamingReporterBase<AutomakeReporter> {
AutomakeReporter( ReporterConfig const& _config )
: StreamingReporterBase( _config )
{}
~AutomakeReporter() override;
static std::string getDescription() {
return "Reports test results in the format of Automake .trs files";
}
void assertionStarting( AssertionInfo const& ) override {}
bool assertionEnded( AssertionStats const& /*_assertionStats*/ ) override { return true; }
void testCaseEnded( TestCaseStats const& _testCaseStats ) override {
// Possible values to emit are PASS, XFAIL, SKIP, FAIL, XPASS and ERROR.
stream << ":test-result: ";
if (_testCaseStats.totals.assertions.allPassed()) {
stream << "PASS";
} else if (_testCaseStats.totals.assertions.allOk()) {
stream << "XFAIL";
} else {
stream << "FAIL";
}
stream << ' ' << _testCaseStats.testInfo.name << '\n';
StreamingReporterBase::testCaseEnded( _testCaseStats );
}
void skipTest( TestCaseInfo const& testInfo ) override {
stream << ":test-result: SKIP " << testInfo.name << '\n';
}
};
#ifdef CATCH_IMPL
AutomakeReporter::~AutomakeReporter() {}
#endif
CATCH_REGISTER_REPORTER( "automake", AutomakeReporter)
} // end namespace Catch
#endif // TWOBLUECUBES_CATCH_REPORTER_AUTOMAKE_HPP_INCLUDED

View File

@ -0,0 +1,181 @@
/*
* Created by Daniel Garcia on 2018-12-04.
* Copyright Social Point SL. All rights reserved.
*
* Distributed under the Boost Software License, Version 1.0. (See accompanying
* file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
*/
#ifndef CATCH_REPORTER_SONARQUBE_HPP_INCLUDED
#define CATCH_REPORTER_SONARQUBE_HPP_INCLUDED
// Don't #include any Catch headers here - we can assume they are already
// included before this header.
// This is not good practice in general but is necessary in this case so this
// file can be distributed as a single header that works with the main
// Catch single header.
#include <map>
namespace Catch {
struct SonarQubeReporter : CumulativeReporterBase<SonarQubeReporter> {
SonarQubeReporter(ReporterConfig const& config)
: CumulativeReporterBase(config)
, xml(config.stream()) {
m_reporterPrefs.shouldRedirectStdOut = true;
m_reporterPrefs.shouldReportAllAssertions = true;
}
~SonarQubeReporter() override;
static std::string getDescription() {
return "Reports test results in the Generic Test Data SonarQube XML format";
}
static std::set<Verbosity> getSupportedVerbosities() {
return { Verbosity::Normal };
}
void noMatchingTestCases(std::string const& /*spec*/) override {}
void testRunStarting(TestRunInfo const& testRunInfo) override {
CumulativeReporterBase::testRunStarting(testRunInfo);
xml.startElement("testExecutions");
xml.writeAttribute("version", "1");
}
void testGroupEnded(TestGroupStats const& testGroupStats) override {
CumulativeReporterBase::testGroupEnded(testGroupStats);
writeGroup(*m_testGroups.back());
}
void testRunEndedCumulative() override {
xml.endElement();
}
void writeGroup(TestGroupNode const& groupNode) {
std::map<std::string, TestGroupNode::ChildNodes> testsPerFile;
for(auto const& child : groupNode.children)
testsPerFile[child->value.testInfo.lineInfo.file].push_back(child);
for(auto const& kv : testsPerFile)
writeTestFile(kv.first.c_str(), kv.second);
}
void writeTestFile(const char* filename, TestGroupNode::ChildNodes const& testCaseNodes) {
XmlWriter::ScopedElement e = xml.scopedElement("file");
xml.writeAttribute("path", filename);
for(auto const& child : testCaseNodes)
writeTestCase(*child);
}
void writeTestCase(TestCaseNode const& testCaseNode) {
// All test cases have exactly one section - which represents the
// test case itself. That section may have 0-n nested sections
assert(testCaseNode.children.size() == 1);
SectionNode const& rootSection = *testCaseNode.children.front();
writeSection("", rootSection, testCaseNode.value.testInfo.okToFail());
}
void writeSection(std::string const& rootName, SectionNode const& sectionNode, bool okToFail) {
std::string name = trim(sectionNode.stats.sectionInfo.name);
if(!rootName.empty())
name = rootName + '/' + name;
if(!sectionNode.assertions.empty() || !sectionNode.stdOut.empty() || !sectionNode.stdErr.empty()) {
XmlWriter::ScopedElement e = xml.scopedElement("testCase");
xml.writeAttribute("name", name);
xml.writeAttribute("duration", static_cast<long>(sectionNode.stats.durationInSeconds * 1000));
writeAssertions(sectionNode, okToFail);
}
for(auto const& childNode : sectionNode.childSections)
writeSection(name, *childNode, okToFail);
}
void writeAssertions(SectionNode const& sectionNode, bool okToFail) {
for(auto const& assertion : sectionNode.assertions)
writeAssertion( assertion, okToFail);
}
void writeAssertion(AssertionStats const& stats, bool okToFail) {
AssertionResult const& result = stats.assertionResult;
if(!result.isOk()) {
std::string elementName;
if(okToFail) {
elementName = "skipped";
}
else {
switch(result.getResultType()) {
case ResultWas::ThrewException:
case ResultWas::FatalErrorCondition:
elementName = "error";
break;
case ResultWas::ExplicitFailure:
elementName = "failure";
break;
case ResultWas::ExpressionFailed:
elementName = "failure";
break;
case ResultWas::DidntThrowException:
elementName = "failure";
break;
// We should never see these here:
case ResultWas::Info:
case ResultWas::Warning:
case ResultWas::Ok:
case ResultWas::Unknown:
case ResultWas::FailureBit:
case ResultWas::Exception:
elementName = "internalError";
break;
}
}
XmlWriter::ScopedElement e = xml.scopedElement(elementName);
ReusableStringStream messageRss;
messageRss << result.getTestMacroName() << "(" << result.getExpression() << ")";
xml.writeAttribute("message", messageRss.str());
ReusableStringStream textRss;
if (stats.totals.assertions.total() > 0) {
textRss << "FAILED:\n";
if (result.hasExpression()) {
textRss << "\t" << result.getExpressionInMacro() << "\n";
}
if (result.hasExpandedExpression()) {
textRss << "with expansion:\n\t" << result.getExpandedExpression() << "\n";
}
}
if(!result.getMessage().empty())
textRss << result.getMessage() << "\n";
for(auto const& msg : stats.infoMessages)
if(msg.type == ResultWas::Info)
textRss << msg.message << "\n";
textRss << "at " << result.getSourceInfo();
xml.writeText(textRss.str(), XmlFormatting::Newline);
}
}
private:
XmlWriter xml;
};
#ifdef CATCH_IMPL
SonarQubeReporter::~SonarQubeReporter() {}
#endif
CATCH_REGISTER_REPORTER( "sonarqube", SonarQubeReporter )
} // end namespace Catch
#endif // CATCH_REPORTER_SONARQUBE_HPP_INCLUDED

View File

@ -0,0 +1,253 @@
/*
* Created by Colton Wolkins on 2015-08-15.
* Copyright 2015 Martin Moene. All rights reserved.
*
* Distributed under the Boost Software License, Version 1.0. (See accompanying
* file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
*/
#ifndef TWOBLUECUBES_CATCH_REPORTER_TAP_HPP_INCLUDED
#define TWOBLUECUBES_CATCH_REPORTER_TAP_HPP_INCLUDED
// Don't #include any Catch headers here - we can assume they are already
// included before this header.
// This is not good practice in general but is necessary in this case so this
// file can be distributed as a single header that works with the main
// Catch single header.
#include <algorithm>
namespace Catch {
struct TAPReporter : StreamingReporterBase<TAPReporter> {
using StreamingReporterBase::StreamingReporterBase;
~TAPReporter() override;
static std::string getDescription() {
return "Reports test results in TAP format, suitable for test harnesses";
}
ReporterPreferences getPreferences() const override {
return m_reporterPrefs;
}
void noMatchingTestCases( std::string const& spec ) override {
stream << "# No test cases matched '" << spec << "'" << std::endl;
}
void assertionStarting( AssertionInfo const& ) override {}
bool assertionEnded( AssertionStats const& _assertionStats ) override {
++counter;
stream << "# " << currentTestCaseInfo->name << std::endl;
AssertionPrinter printer( stream, _assertionStats, counter );
printer.print();
stream << std::endl;
return true;
}
void testRunEnded( TestRunStats const& _testRunStats ) override {
printTotals( _testRunStats.totals );
stream << "\n" << std::endl;
StreamingReporterBase::testRunEnded( _testRunStats );
}
private:
std::size_t counter = 0;
class AssertionPrinter {
public:
AssertionPrinter& operator= ( AssertionPrinter const& ) = delete;
AssertionPrinter( AssertionPrinter const& ) = delete;
AssertionPrinter( std::ostream& _stream, AssertionStats const& _stats, std::size_t _counter )
: stream( _stream )
, result( _stats.assertionResult )
, messages( _stats.infoMessages )
, itMessage( _stats.infoMessages.begin() )
, printInfoMessages( true )
, counter(_counter)
{}
void print() {
itMessage = messages.begin();
switch( result.getResultType() ) {
case ResultWas::Ok:
printResultType( passedString() );
printOriginalExpression();
printReconstructedExpression();
if ( ! result.hasExpression() )
printRemainingMessages( Colour::None );
else
printRemainingMessages();
break;
case ResultWas::ExpressionFailed:
if (result.isOk()) {
printResultType(passedString());
} else {
printResultType(failedString());
}
printOriginalExpression();
printReconstructedExpression();
if (result.isOk()) {
printIssue(" # TODO");
}
printRemainingMessages();
break;
case ResultWas::ThrewException:
printResultType( failedString() );
printIssue( "unexpected exception with message:" );
printMessage();
printExpressionWas();
printRemainingMessages();
break;
case ResultWas::FatalErrorCondition:
printResultType( failedString() );
printIssue( "fatal error condition with message:" );
printMessage();
printExpressionWas();
printRemainingMessages();
break;
case ResultWas::DidntThrowException:
printResultType( failedString() );
printIssue( "expected exception, got none" );
printExpressionWas();
printRemainingMessages();
break;
case ResultWas::Info:
printResultType( "info" );
printMessage();
printRemainingMessages();
break;
case ResultWas::Warning:
printResultType( "warning" );
printMessage();
printRemainingMessages();
break;
case ResultWas::ExplicitFailure:
printResultType( failedString() );
printIssue( "explicitly" );
printRemainingMessages( Colour::None );
break;
// These cases are here to prevent compiler warnings
case ResultWas::Unknown:
case ResultWas::FailureBit:
case ResultWas::Exception:
printResultType( "** internal error **" );
break;
}
}
private:
static Colour::Code dimColour() { return Colour::FileName; }
static const char* failedString() { return "not ok"; }
static const char* passedString() { return "ok"; }
void printSourceInfo() const {
Colour colourGuard( dimColour() );
stream << result.getSourceInfo() << ":";
}
void printResultType( std::string const& passOrFail ) const {
if( !passOrFail.empty() ) {
stream << passOrFail << ' ' << counter << " -";
}
}
void printIssue( std::string const& issue ) const {
stream << " " << issue;
}
void printExpressionWas() {
if( result.hasExpression() ) {
stream << ";";
{
Colour colour( dimColour() );
stream << " expression was:";
}
printOriginalExpression();
}
}
void printOriginalExpression() const {
if( result.hasExpression() ) {
stream << " " << result.getExpression();
}
}
void printReconstructedExpression() const {
if( result.hasExpandedExpression() ) {
{
Colour colour( dimColour() );
stream << " for: ";
}
std::string expr = result.getExpandedExpression();
std::replace( expr.begin(), expr.end(), '\n', ' ');
stream << expr;
}
}
void printMessage() {
if ( itMessage != messages.end() ) {
stream << " '" << itMessage->message << "'";
++itMessage;
}
}
void printRemainingMessages( Colour::Code colour = dimColour() ) {
if (itMessage == messages.end()) {
return;
}
// using messages.end() directly (or auto) yields compilation error:
std::vector<MessageInfo>::const_iterator itEnd = messages.end();
const std::size_t N = static_cast<std::size_t>( std::distance( itMessage, itEnd ) );
{
Colour colourGuard( colour );
stream << " with " << pluralise( N, "message" ) << ":";
}
for(; itMessage != itEnd; ) {
// If this assertion is a warning ignore any INFO messages
if( printInfoMessages || itMessage->type != ResultWas::Info ) {
stream << " '" << itMessage->message << "'";
if ( ++itMessage != itEnd ) {
Colour colourGuard( dimColour() );
stream << " and";
}
}
}
}
private:
std::ostream& stream;
AssertionResult const& result;
std::vector<MessageInfo> messages;
std::vector<MessageInfo>::const_iterator itMessage;
bool printInfoMessages;
std::size_t counter;
};
void printTotals( const Totals& totals ) const {
if( totals.testCases.total() == 0 ) {
stream << "1..0 # Skipped: No tests ran.";
} else {
stream << "1.." << counter;
}
}
};
#ifdef CATCH_IMPL
TAPReporter::~TAPReporter() {}
#endif
CATCH_REGISTER_REPORTER( "tap", TAPReporter )
} // end namespace Catch
#endif // TWOBLUECUBES_CATCH_REPORTER_TAP_HPP_INCLUDED

View File

@ -0,0 +1,219 @@
/*
* Created by Phil Nash on 19th December 2014
* Copyright 2014 Two Blue Cubes Ltd. All rights reserved.
*
* Distributed under the Boost Software License, Version 1.0. (See accompanying
* file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
*/
#ifndef TWOBLUECUBES_CATCH_REPORTER_TEAMCITY_HPP_INCLUDED
#define TWOBLUECUBES_CATCH_REPORTER_TEAMCITY_HPP_INCLUDED
// Don't #include any Catch headers here - we can assume they are already
// included before this header.
// This is not good practice in general but is necessary in this case so this
// file can be distributed as a single header that works with the main
// Catch single header.
#include <cstring>
#ifdef __clang__
# pragma clang diagnostic push
# pragma clang diagnostic ignored "-Wpadded"
#endif
namespace Catch {
struct TeamCityReporter : StreamingReporterBase<TeamCityReporter> {
TeamCityReporter( ReporterConfig const& _config )
: StreamingReporterBase( _config )
{
m_reporterPrefs.shouldRedirectStdOut = true;
}
static std::string escape( std::string const& str ) {
std::string escaped = str;
replaceInPlace( escaped, "|", "||" );
replaceInPlace( escaped, "'", "|'" );
replaceInPlace( escaped, "\n", "|n" );
replaceInPlace( escaped, "\r", "|r" );
replaceInPlace( escaped, "[", "|[" );
replaceInPlace( escaped, "]", "|]" );
return escaped;
}
~TeamCityReporter() override;
static std::string getDescription() {
return "Reports test results as TeamCity service messages";
}
void skipTest( TestCaseInfo const& /* testInfo */ ) override {
}
void noMatchingTestCases( std::string const& /* spec */ ) override {}
void testGroupStarting( GroupInfo const& groupInfo ) override {
StreamingReporterBase::testGroupStarting( groupInfo );
stream << "##teamcity[testSuiteStarted name='"
<< escape( groupInfo.name ) << "']\n";
}
void testGroupEnded( TestGroupStats const& testGroupStats ) override {
StreamingReporterBase::testGroupEnded( testGroupStats );
stream << "##teamcity[testSuiteFinished name='"
<< escape( testGroupStats.groupInfo.name ) << "']\n";
}
void assertionStarting( AssertionInfo const& ) override {}
bool assertionEnded( AssertionStats const& assertionStats ) override {
AssertionResult const& result = assertionStats.assertionResult;
if( !result.isOk() ) {
ReusableStringStream msg;
if( !m_headerPrintedForThisSection )
printSectionHeader( msg.get() );
m_headerPrintedForThisSection = true;
msg << result.getSourceInfo() << "\n";
switch( result.getResultType() ) {
case ResultWas::ExpressionFailed:
msg << "expression failed";
break;
case ResultWas::ThrewException:
msg << "unexpected exception";
break;
case ResultWas::FatalErrorCondition:
msg << "fatal error condition";
break;
case ResultWas::DidntThrowException:
msg << "no exception was thrown where one was expected";
break;
case ResultWas::ExplicitFailure:
msg << "explicit failure";
break;
// We shouldn't get here because of the isOk() test
case ResultWas::Ok:
case ResultWas::Info:
case ResultWas::Warning:
CATCH_ERROR( "Internal error in TeamCity reporter" );
// These cases are here to prevent compiler warnings
case ResultWas::Unknown:
case ResultWas::FailureBit:
case ResultWas::Exception:
CATCH_ERROR( "Not implemented" );
}
if( assertionStats.infoMessages.size() == 1 )
msg << " with message:";
if( assertionStats.infoMessages.size() > 1 )
msg << " with messages:";
for( auto const& messageInfo : assertionStats.infoMessages )
msg << "\n \"" << messageInfo.message << "\"";
if( result.hasExpression() ) {
msg <<
"\n " << result.getExpressionInMacro() << "\n"
"with expansion:\n" <<
" " << result.getExpandedExpression() << "\n";
}
if( currentTestCaseInfo->okToFail() ) {
msg << "- failure ignore as test marked as 'ok to fail'\n";
stream << "##teamcity[testIgnored"
<< " name='" << escape( currentTestCaseInfo->name )<< "'"
<< " message='" << escape( msg.str() ) << "'"
<< "]\n";
}
else {
stream << "##teamcity[testFailed"
<< " name='" << escape( currentTestCaseInfo->name )<< "'"
<< " message='" << escape( msg.str() ) << "'"
<< "]\n";
}
}
stream.flush();
return true;
}
void sectionStarting( SectionInfo const& sectionInfo ) override {
m_headerPrintedForThisSection = false;
StreamingReporterBase::sectionStarting( sectionInfo );
}
void testCaseStarting( TestCaseInfo const& testInfo ) override {
m_testTimer.start();
StreamingReporterBase::testCaseStarting( testInfo );
stream << "##teamcity[testStarted name='"
<< escape( testInfo.name ) << "']\n";
stream.flush();
}
void testCaseEnded( TestCaseStats const& testCaseStats ) override {
StreamingReporterBase::testCaseEnded( testCaseStats );
if( !testCaseStats.stdOut.empty() )
stream << "##teamcity[testStdOut name='"
<< escape( testCaseStats.testInfo.name )
<< "' out='" << escape( testCaseStats.stdOut ) << "']\n";
if( !testCaseStats.stdErr.empty() )
stream << "##teamcity[testStdErr name='"
<< escape( testCaseStats.testInfo.name )
<< "' out='" << escape( testCaseStats.stdErr ) << "']\n";
stream << "##teamcity[testFinished name='"
<< escape( testCaseStats.testInfo.name ) << "' duration='"
<< m_testTimer.getElapsedMilliseconds() << "']\n";
stream.flush();
}
private:
void printSectionHeader( std::ostream& os ) {
assert( !m_sectionStack.empty() );
if( m_sectionStack.size() > 1 ) {
os << getLineOfChars<'-'>() << "\n";
std::vector<SectionInfo>::const_iterator
it = m_sectionStack.begin()+1, // Skip first section (test case)
itEnd = m_sectionStack.end();
for( ; it != itEnd; ++it )
printHeaderString( os, it->name );
os << getLineOfChars<'-'>() << "\n";
}
SourceLineInfo lineInfo = m_sectionStack.front().lineInfo;
os << lineInfo << "\n";
os << getLineOfChars<'.'>() << "\n\n";
}
// if string has a : in first line will set indent to follow it on
// subsequent lines
static void printHeaderString( std::ostream& os, std::string const& _string, std::size_t indent = 0 ) {
std::size_t i = _string.find( ": " );
if( i != std::string::npos )
i+=2;
else
i = 0;
os << Column( _string )
.indent( indent+i)
.initialIndent( indent ) << "\n";
}
private:
bool m_headerPrintedForThisSection = false;
Timer m_testTimer;
};
#ifdef CATCH_IMPL
TeamCityReporter::~TeamCityReporter() {}
#endif
CATCH_REGISTER_REPORTER( "teamcity", TeamCityReporter )
} // end namespace Catch
#ifdef __clang__
# pragma clang diagnostic pop
#endif
#endif // TWOBLUECUBES_CATCH_REPORTER_TEAMCITY_HPP_INCLUDED