C&C++ Unit Test Frameworks to Speed-up Prototyping

on

Introduction

Projects Survival
Unit testing is  ubiquitous in the commercial software development, often being accompanied with the respective development methods including Test-driven development (TDD) and Extreme programming (XP). However, majority of projects and produced source code belong to the research and prototyping activities, where unit tests are pretty rare written because the code base is not stable and short development time has higher priority than the code stability and maintainability.
Design by contract (DbC, Contract Programming) is typically preferable and sufficient to ensure correctness of the code not wasting time for the boilerplate code required by the Unit Test Frameworks. Design by Contract was introduced in the Eiffel language and naively supported by various modern languages including Clojure, Perl, Vala, D, Ada, Racket (PLT Scheme), etc. Contract Programming can be effortlessly applied also in C++11 via STL type traits, constexpr-ns, asserts and preprocessor directives (macroses).

Design by Contract
Unit testing naturally complements Design by Contract. DbC ensures that all input-output values satisfy formal criteria. Unit testing is not able to cover all possible use-cases of the input-output data for all functions/classes/modules but it allows to verify execution logic of the specific functions or interfaces without building the whole project. On the spot build of only specific files saves huge amount of time for the C++ projects (especially, heavily employing templates or having lots of dependencies) validation and debugging, which is the topic of this post.

Introduction to the Design by Contract:

KISS Unit Testing to Speed-up Prototyping

Unit Test Framework Workflow
Unit testing provides fast and DRY (Do not Repeat Yourself) validation of the execution logic of non-trivial project components, which often may save huge amount of time otherwise wasted for the debugging. For example, I'm developing a clustering library and extending it with the new feature for fast matching of large containers of objects. This feature includes low-level arithmetic with Big Integral numbers and complex bit-wise operations. The clustering library itself is stable, but the new feature may contain some bugs in the execution logic. To validate the execution logic I need either a) to rebuild the whole library (takes time), execute it on a dataset with ground-truth (takes more time), compare the results and then (!) waste time for the debugging to identify the cause of the invalid results b) or integrate several unit tests directly after the functions having a tricky logic and immediately validate that functions. Unit tests does not guarantee the correct execution of the whole system but significantly reduce the amount of issues saving lots of time.
What about the boilerplate code of the testing framework, which makes the main overhead of the unit testing development process? - In fact, the boilerplate overhead does not always necessary exist. We are coming to the selection criteria of the suitable "Keep It Simple" unit test framework for the prototyping:
  • Non-intrusive. Deployment of the test framework should [almost] not affect the already existing build environment. A default entry point[s] should be provided for the tests execution out of the box to not flood the original source code.
  • Minimal dependencies. A single header inclusion should be sufficient to write the tests and the linking dependency only with the single library is desirable.
  • Concise tests. for the frequently changing code base, it is often convenient to write tests in the original source code files to verify only the required functions and to explicitly see what and where is tested. There should not be any obligation to declare redundant test cases/suites/etc. (excessive verbosity).
  • Comprehensive toolbox. The framework should supports parameterized tests (ideally by both type an value) and theories (to test a specific behavior against permutations of a set of user-defined parameters), fixtures (context of the test execution), customized validations, tests hierarchy management means, etc.
  • Flexible reporting. The tests reports should be customizable and support run-time filtering of the executing tests.
  • Cross-platform and open source. The tests should be executable on all the platforms where the original project is compilable. Open source frameworks allows to clarify anything about the functionality, customize and extend it beyond capabilities of the original APIs.

Outstanding Unit Test Frameworks for C/C++

The most popular open source and comprehensive unit test frameworks for C++ are probably Google Test, Boost.Test and the legacy CppUnit. Each of them provides comprehensive functionality suitable for large commercial projects but they are not always so convenient for the prototyping and research projects being relatively heavy and verbose. Most of the modern unit test frameworks implement the xUnit framework structure.

CppUnit (C++) is one of the first comprehensive and popular unit test frameworks for C++ but it is far from being concise or non-intrusive. This framework requires lots of boilerplate code to both write test cases and deploy them, which does not fit our requirements.
A "brief" tutorial: Crash Course in using CppUnit.

Google Test (C/C++) is probably the most comprehensive existing unit test framework for C/C++, which covers all possible needs of the production code but makes a steeper learning curve and more verbose syntax comparing to more specific frameworks. This framework requires a boilerplate code that explicitly invokes tests execution, which is not the most convenient option for our needs.

Boost.Test (C++)

Boost.Test allows to organize test cases into test suites and modules, declare dependencies between the tests to discard their execution, group test cases and control their execution by semantic labels. Decorators are one of the specific capabilities, which provide a mechanism for updating various attributes of the tests to tune their execution. The framework provides test case template facility to test a template base component with different template parameters besides the ordinary value parameters and many other features. Example of the simple test case:
#include <my_class.hpp>
#define BOOST_TEST_MODULE MyTest
#include <boost/test/unit_test.hpp>

// ...

BOOST_AUTO_TEST_CASE( my_test )
{
  my_class test_object( "qwerty" );
  BOOST_TEST( test_object.is_valid() );
}

A brief tutorial: A testing framework, what for?

Criterion (C/C++)

Criterion is a full-fledged unit test framework for C/C++, which like Boost.Test and Mettle does not require any boilerplate code and the same time provides comprehensive testing capabilities. Specific and extremely useful feature of this framework is theories, which automate testing for sets of parameters providing lots of capabilities including invariants. Theories are much more than just parameterized tests available in other frameworks. AExample of the simple test case:


#include <string.h>
#include <criterion/criterion.h>

Test(sample, test) {
    cr_expect(strlen("Test") == 4, "Expected \"Test\" to have a length of 4.");
    cr_expect(strlen("Hello") == 4, "This will always fail, why did I add this?");
    cr_assert(strlen("") == 0);
}
A brief tutorial: Getting started with Criterion.

Mettle (C++14)

Mettle requires C++14 compiler, which limits the applicability of the framework but on the other hand provides the most concise and powerful syntax! I'm not aware about any other framework, which allows to write parameterized unit tests literally within 3 lines of code (see the Example below). Unlike other test frameworks, Mettle provides opportunity to write test cases using pure template based code instead of macroses, though the build-in macroses are also available. Test cases can be parameterized by both value and type, fixtures factories are available. Test cases have tags and attributes for the filtering on execution. Mettle evaluates expectations (analog of assertions) using composition of matchers (specific function objects), which is a more powerful, concise and expressive solution than the legacy assertion-based approach. Basically. a combination of the filter() matcher with others provides capabilities of the Criterion theories!

An extended example:
#include <mettle.hpp>
using namespace mettle;

suite<> basic("a basic suite", [](auto &_) {
  _.test("a test", []() {  // Simple test case "a test"
    expect(true, equal_to(true));
  });

  for(int i = 0; i < 4; i++) {  // Parameterized test case
    _.test("test number " + std::to_string(i), [i]() {
      expect(i % 2, less(2));
    });
  }

  subsuite<>(_, "a subsuite", [](auto &_) {
    _.test("a sub-test", []() {
      expect(true, equal_to(true));
    });
  });

});

A brief tutorial: Mettle Tutorial.

Conclusion

Each of the three listed unit test frameworks have own advantages being suitable for the prototyping and research projects with the unstable code base due to a) absence of the boilerplate code and b) possibility to write tests in the testing source code. Boots.Test provides the most comprehensive and structured documentation together with comprehensive out of the box features like various logging formats (but lacks the theories feature of the Criterion framework). On the other hand, parameterized tests and various tuning is more verbose and not so straight forward in Boots.Test comparing to Criterion and Mettle frameworks. Criterion is my personal choice for C projects, legacy C++ projects or when the explicit theories feature is essential (math libs, etc.). For the projects supporting C++14, I found Mettle to be the most convenient, concise and expressive unit test framework! Mettle test diver (executor) provides less options than Criterion or Boost.Test and the framework documentation is also less structured, but the expressiveness and power of the framework outweigh that insignificant shortcoming, which might be addressed in near future.
So, just give a try to Mettle or Criterion unit test frameworks in your C/C++ projects and look how much time it saves!

PS You might be also interested in my previous post, Overview of the Efficient Programming Languages.

Comments

Post a Comment