Continuous Testing in the Build Pipeline

Our development team ships several times a week, but tests are still run manually and only when someone remembers. As a result, broken code sometimes reaches staging, and once it even reached production. We need continuous testing wired into our build pipeline so that every commit automatically triggers the test suite, and a failing run blocks the release before it can do any damage.

The goal is a pipeline stage that runs unit, integration and a subset of end-to-end tests on each push and pull request. It should produce a clear report showing what passed, what failed and why, publish that report where the team can see it, and mark the build red so a broken change cannot be merged or deployed. We would also like flaky tests to be flagged rather than silently retried forever, and total run time kept reasonable through parallelisation and caching.

We use a common Git hosting service and a standard CI runner; please tell us which tools you are comfortable with and how you would structure the stages. Documentation and a short handover session for our engineers are part of the job.

What the finished setup should deliver:
— automatic test runs on every commit and merge request;
— readable pass and fail reports with logs;
— release blocked when tests fail;
— fast feedback through parallel jobs and caching.