iBoot has a system for building and running unit tests. Currently the system only allows for tests to be run on an x86 host using the native libc, but it's intended that the system will allow tests to run on x86 with iBoot's libc, and built for arm and run on targets (either hosted or freestanding). Overview ======== To add tests to a module, do the following: 1. Create a test driver .c file (details below). 2. Create a test makefile in the module's directory, normally called tests.mk. If the module requires multiple test binaries (usually because of different compilation options), add a makefile per binary, usually called tests-.mk. Details on makefiles are below 3. Add the makefile to the TEST_MAKEFILES list in makefiles/tests.mk Writing tests ============= The only requirement for test binaries is that their main function returns 0 on success and nonzero on failure. By default, tests should only print to stdio on failure. iBoot includes a unit testing framework, but its use is optional is a different testing strategy is needed (or for legacy tests that don't use the framework). Most unit tests should use the testing framework, however. Using the testing framework =========================== To use the testing framework, add tests/unittest-main.o to the test's object file list (normally TEST_SUPPORT_OBJS, not TEST_OBJS) and include in your main test file. unittest-main includes a main() function to run test cases, so only the test cases themselves need to be defined when using the testing framework. The unit testing frameworks works with test suites and test cases. A test suite includes an optional setup function, a list of test cases, and an optional cleanup function. The setup and cleanup function are respectively run before and after each test case. Each test cases is simply a function that takes a uintptr_t parameter. Use struct test_suite to define a test suite, and then use the TEST_SUITE macro to make the test suite visble to the main() function. For example: struct test_suite my_test_suite = { .name = "mymodule", .setup_function = my_setup_function // optional .cleanup_function = my_cleanup_function // optional .test_cases = { { "test1", run_test1, 0 }, { "test2,0", run_test2, 0 }, // passes 0 in parameter { "test2,100", run_test2, 100 }, // passes 100 in parameter TEST_CASE_LAST } }; Test cases are defined by their name, their function, and their argument (a uintptr_t). Test case functions take a single uintptr_t parameter and return void. A test succeeds if no assertions fail, no unexpected panics occur, expected panics (if any) do occur, and TEST_FAIL is not called. Tests are immediately aborted on assertion failures, panics (expected or unexpected), and calls to TEST_FAIL. Assertions can be embedded inside the code under test using the typical iBoot ASSERT and RELEASE_ASSERT macros. These assertions end up in shipping code, of course. For assertions in the test code, the unittest framework provides a number of convenience macros, including: For integeger types: TEST_ASSERT_{EQ,NEQ,GT,LT,GTE,LTE} For C strings: TEST_ASSERT_STR_EQ, TEST_ASSERT_STR_NEQ For binary data: TEST_ASSERT_MEM_EQ, TEST_ASSERT_MEM_NEQ For pointers: TEST_ASSERT_PTR_EQ, TEST_ASSERT_PTR_NEQ, TEST_ASSERT_NULL, TEST_ASSERT_NOT_NULL These macros (with the exception of the MEM macros) support printing the expression and value for the left hand and right hand side of the comparison on failure. A general-purpose TEST_FAIL macro is also provided for things not covered by the assertion macros above. Writing test makefiles ====================== Test makefiles define the following variables: TEST_NAME The name of the test binary, must be unique TEST_OBJS The object files being tested TEST_SUPPORT_OBJS (optional): Object files used to run the tests. These are listed separately so that their source files are not included in test coverage reporting TEST_CFLAGS (optional): CFLAGS to be used while building the test binary TEST_LDFLAGS (optional): LDFLAGS to be used when building the test binary Test coverage ============= The unit tests are automatically built with gcov support. This can be suppressed by passing COVERAGE=NO on the make commandline. make COVERAGE=NO tests The best way to parse the coverage info generated by the tests is with the gcovr tool (find it online until it gets included in iBoot source). Run the tool as follows: mkdir -p build/coverage && ./gcovr -b -r . --html -o build/coverage/index.html --html-details This will only give coverage statistics for the source files as built for unit testing, which may be different from how they're built for real targets. To see how well the tests are covering real world scenarios, it's possible to generate coverage info for real targets. This will just emit info on which source lines emitted executable code; no profiling will actually be performed on the targets. To generate this info, add COVERAGE=YES to the make commandline: make COVERAGE=YES build