How we improved reporting and monitoring of test automation results
As we have moved to a Continuous Delivery approach with our new web application with dozens of daily releases of our main product, it has become more crucial to identify the causes of end-to-end test failures. Every day we fight against test flakiness, i.e. failures unrelated to actual defects in the application under test, and have achieved an average success rate of more than 99% on our Selenium-based tests. This means less than 1% of failed tests on stable main branch in the overnight runs, indicating that there are still inevitable occasional failures that might cause uncertainty. It’s a well known problem of UI automation that can happen frequently, especially on a staging environment where we are not mocking any backend service.