Why We Should Care About Test Execution Output in Safety-Critical Industries

In highly regulated industries such as automotive, aviation, and medical, test artifacts are often essential for further reporting. Automotive SPICE (ASPICE) or AUTOSAR from the automotive field requires detailed, reproducible reporting on executed tests. Companies such as BMW or NVIDIA use Bazel to build their software. However, Bazel does not cache test execution output, but only that a test was executed. To address this, some companies have resorted to forking Bazel and implementing this feature on their own which causes maintenance overhead and impedes collaborative Bazel improvements. Our proposed solution is to treat the test binary as a tool and run it as part of the Bazel build phase. This way, we can depend on the test output for further processing such as skipping integration tests if unit tests fail, collecting output artifacts such as human-readable test reports and machine-readable coverage reports, as well as making coverage collection depend on test outputs. Thanks to Bazel’s caching, the test is not executed again in the test phase which makes this solution an acceptable tradeoff.

Oct 15, 2024 17:00 — 17:10