Introduction

BullseyeCoverage is a code coverage analyzer for the C and C++ languages. With BullseyeCoverage, you can:

identify code which was not fully exercised during testing
obtain summary measurements of testing completeness to help you decide when you have tested enough

The steps involved in using BullseyeCoverage are:

Enable coverage build and build your program
Run your program one or more times
Generate and view coverage reports
Repeat from step 2 until you are satisfied with testing

BullseyeCoverage cannot measure coverage of assembly code.

How it Works

When coverage build is enabled, BullseyeCoverage intercepts compiler invocations to perform instrumentation. Information about the code is written to the coverage file, named test.cov by default. The instrumented code is compiled using your compiler. At run-time, the coverage file is updated with coverage measurements. The COVFILE environment variable names the coverage file. You can view the coverage file using the Coverage Browser or the command line report programs.

Measurement Technique

Instrumentation Model

BullseyeCoverage measures condition/decision coverage and function coverage. The following specific events are monitored.

BullseyeCoverage indicates whether the true and false outcomes of each decision occurred. A decision is the controlling expression in the if, if-else, while, do-while, and for statements. A decision is an expression that uses the logical operators && or || outside such a controlling expression. The first operand of the ?: operator is a decision if the ?: occurs outside a controlling expression. In the example code below, the decisions are marked by yellow boxes.
```
if (v1 == v2 && (v3 || v4))

v5 = v6 ? v7 : v8;

if (v6 ? v7 : v8)

v9 = (v10 || v11) ? v12 : v13;

if ((v10 || v11) ? v12 : v13)

v14 = v15 || v16;
```
BullseyeCoverage indicates whether the true and false outcomes of each condition occurred. A condition is an operand of the logical operators && and || that does not contain logical operators. The ?: operator is also a logical operator. The first operand of the ?: operator is a condition if it does not contain logical operators and the ?: occurs in a decision. In the example code below, the conditions are marked by blue boxes.
```
if (v1 == v2 && (v3 || v4))

v5 = v6 ? v7 : v8;

if (v6 ? v7 : v8)

v9 = (v10 || v11) ? v12 : v13;

if ((v10 || v11) ? v12 : v13)

v14 = v15 || v16;
```
BullseyeCoverage indicates whether each case and default label was selected. When the default label is omitted, the branch to the end of the switch statement is indicated. For example, BullseyeCoverage indicates whether control flowed to each of the pink boxed areas below.
```
switch (a) {
case 1:
case 2:
    statements
    break;
}	// Implicit default
```
BullseyeCoverage indicates whether each try-block finished without exceptions and whether each exception handler was entered. Coverage of Microsoft structured exception handling (__try, __except) is indicated similarly. For example, BullseyeCoverage indicates whether control flowed to the pink boxed area below.
```
try {
    statements
}
catch (exception-declaration) {
    statements
}
catch (...) {
    statements
}
```
BullseyeCoverage indicates whether each function was entered.
```
void function()
{
    statements
}
```
BullseyeCoverage indicates whether each lambda-expression was entered.
```
[] { statements }
```
BullseyeCoverage indicates whether each range-based for statement body was entered.
```
for (for-range-declaration: expression) statement
```

Other Details

By default, BullseyeCoverage does not measure coverage of code produced by macro expansion.

BullseyeCoverage does not measure coverage of variable initializers at file scope.

BullseyeCoverage does not measure coverage of static variable initializers in C. These initializers are evaluated at compile-time.

BullseyeCoverage does not measure coverage of functions declared consteval or constexpr. These functions might be evaluated at compile-time.

BullseyeCoverage measures coverage in template functions the same way as non-template functions. Coverage is measured by instrumenting the source code. Coverage in all template instantiations is combined and reported together.

Best Practice Recommendations

These are general guidelines for usage of C++ features and tools that can have an impact on coverage measurement.

Update to the latest version of your compiler and the latest version of BullseyeCoverage frequently. Older versions of any software have more bugs than newer versions. Updating frequently allows you to make any necessary adjustments incrementally, minimizing the likelihood of a productivity impact. When updating your tools, update all your projects and users at the same time to avoid version compatibility issues.
Use inline functions rather than macros. When a macro expansion produces control-flow constructs, the resulting code is often not fully exercisable at the point of all invocations. Measuring coverage of such macro expansions results in over-emphasizing the importance of that code, due to it being repeated at every invocation. Additionally, measuring coverage of such macro expansions reduces the overall achievable coverage. For these reasons, BullseyeCoverage does not report coverage of code produced from macro expansions by default.
Do not overload operators && and ||. Defining these operators with operand type or return type other than bool conflicts with condition/decision coverage at a basic level.
In addition to the conflict with condition/decision coverage, overloading operators && and || is not recommended due to these semantic differences:
- The order of operand evaluation is guaranteed left to right with the built-in operators, but not with the overloaded operators.
- The overloaded operators do not short-circuit.
Use the alternative tokens 'and' and 'or' rather than the token spellings '&&' and '||'. The token && is used for multiple purposes, which can be difficult to distinguish. The token && serves as both the logical-and operator as well as the rvalue reference qualifier. With GCC and Clang, && is also a unary operator taking the address of a label. BullseyeCoverage recognizes 'and' as the logical-and operator unambiguously.
To enable the alternative tokens with Microsoft C++, use compiler option /permissive- or use C++20 or later, for example with /std:c++20 .
Declare constant-initialized variables with constinit or constexpr rather than const wherever possible. This helps BullseyeCoverage identify expressions that cannot be modified with run-time instrumentation.
Avoid declaring functions constexpr. Because constexpr functions might evaluate at compile-time, BullseyeCoverage cannot add run-time instrumentation. When a constexpr function evaluates at run-time, code coverage measurement goes unreported.
To address safety-critical code coverage requirements, remove the constexpr keyword from each function. If any invocation of such a function occurs in a constant expression, make a copy of the function with a different name declared consteval for those cases.
With Microsoft C++, use compiler option /Zc:preprocessor to enable the conforming preprocessor. The Microsoft traditional preprocessor is buggy and idiosyncratic. Although we make considerable effort to mimic this incorrect behavior, it is not always possible and in rare instances the differences lead to a failed build. The Microsoft conforming preprocessor behaves properly, eliminating the possibility of a difference in behavior when using BullseyeCoverage.
With Microsoft C++, use compiler option /permissive- or use C++20 or later, for example with /std:c++20 . If this is not feasible, enable covc option --strict. These options ensure that latent errors in uninstantiated templates and other standards violations result in straightforward error messages that are easy to understand and fix. Without these options, errors in uninstantiated templates lead to confusing symptoms.
With Microsoft C++, use compiler option /external:I rather than /I to specify include directories containing code written and tested outside your project. BullseyeCoverage automatically excludes code included via /external:I.
With GCC and Clang, use compiler option -isystem rather than -I to specify include directories containing code written and tested outside your project. BullseyeCoverage automatically excludes code included via -isystem.

Performance

Instrumenting with BullseyeCoverage typically increases execution time to 1.2x, code size to 1.4x and compile time to 1.7x with most Microsoft C++ projects, 3-8x with other compilers. Your results may vary. Compiler optimization settings greatly improve performance of instrumented code.

BullseyeCoverage may modify compiler options and directives affecting performance. These changes may include, but are not limited to, the following.

Inline function expansion may be enabled
The OpenMP for directive (#pragma omp parallel for) is disabled

object code

Updated: 6 Sep 2024