Testing Process

The document describes the testing process in Archipelo.

Policy

1. Testing is a key part of the development process and is the most important part of the quality of our products.

2. We rely on automated testing and take responsibility to develop new ways of ensuring that the software is working as expected.

3. Any change to the codebase should be covered by an adequate amount of automated tests or other testing strategies.

4. Each engineer should include testing in the development activities.

5. We learn from our mistakes. That's why we need to make sure that we will not be surprised by the same errors again. We surround each bugfix with additional tests.

6. We strive for tests quality, not quantity.

Test types

We define following types of tests that ensure the quality in different parts of the product.

Unit tests

Unit tests are the most basic tests that ensure the software is working as expected at the low level by testing individual methods and functions. They check if the output from the given methods is correct, based on the given input, usually without the need to communicate with external components. Their goal is to give the engineers trust that any change they do, does not break any other part of the software. Also, they make the refactoring process easier.

Unit tests are the responsibility of the engineers working with the code. Each new addition or change need to be covered by appropriate amount of the unit tests. We agree that we do not introduce changes that cause the drop in the code coverage metric.

We use Codecov tool for measuring unit tests coverage. Each of our languages (JS, Python, Go) has its own metric that we monitor.

Goal for code coverage for each language:

• 0-60% is risky
• +60% is acceptable
• +75% is commendable
• +90% is exemplary

Environment: these tests run on CI/CD environment for each created pull request and on QA environment right after the PR is merged to main branch. They should also be run locally before pushing the code to remote server.

Our approach to unit testing resembles one that is followed by Google. These are the key notes from the article "Code Coverage Best Practices":

• Code coverage provides significant benefits to the developer workflow
• Efforts in increasing code coverage can often lead to culture changes in engineering excellence that in the long run reduce defects
• A high code coverage percentage does not guarantee high quality in the test coverage...
• ...But a low code coverage number does guarantee that large areas of the product are going completely untested
• There is no “ideal code coverage number” that universally applies to all products
• In general code coverage of a lot of products is below the bar; we should aim at significantly improving code coverage across the board
• More important than the percentage of lines covered is human judgment over the actual lines of code (and behaviors) that aren’t being covered
• Make sure that frequently changing code is covered
• Unit test code coverage is only a piece of the puzzle

Current state

Language	Coverage	Readme
GO		Link
Python		Link
JavaScript		Link
ETL		Link
Combined

Integration tests

Integration tests ensure that the software works as expected with other parts of the product.

Environment: these tests run on CI/CD environment for each created pull request and on QA environment right after the PR is merged to main branch.

Current state

We have only Go tests checking if backend integrates properly with our search engine. The tests are located in go/search/search_service_integ_test.go.

Soon, we are going to add more tests of this type, e.g.:

• contract tests ensuring that the API is compliant with our specification,
• data pipelines tests for checking integration with cloud services.

Instructions how to run integration test can be found here.

End-to-end (E2E) tests

E2E test the entire product from the user perspective. They focus on the optimal flow, without checking any corner cases or errors.

They are the final check before a release, so if any of them fail, the release is not going to be published.

E2E tests are triggered automatically after each merge to the main branch and their results can be found in GitHub Actions output as well as the state for each run is posted to #build Slack channel.

Environment: these tests run on QA environment right after a new PR is merged to main branch and also they trigger for every created Pull Request, blocking it if there are any fails.

Current state

We have E2E tests written for OneBox Chrome Extension and Web Application (see README here).

Test health

main branch

We have several automated tests integrated into our GitHub CI process which are triggered for each pull request and push events. GitHub CI jobs marked as Required will block the PR from being merged, which means the code needs to be fixed or tests need to be adjusted to latest changes. All tests are also run right after the merge to main branch.

Quality of tests

According to our testing policy, we strive for tests quality, not quantity. Of course, more tests mean higher coverage, but does not infer higher code quality. The quality of the software comes from number of tests and their quality combined. When you are reviewing changes, please make sure that the tests bring value and are not added only for coverage reasons. Quantity is easy to measure, but the quality we can assure only by following these practices:

• a developer adding tests need to think if they provide any value and information about the correctness of the tested area,
• a reviewer needs to read the code of the tests carefully and raise any concerns about their usefulness,
• high-level tests (e.g. E2E tests) need to be first defined following the instructions from Automated E2E Tests README.