What Tests to Run in Continuous Integration

A simple introduction to CI and testing

What it is, why use it and how to test

Continuous integration means building your software every time a developer pushes code. This affects your testing and requires careful planning and the use of test automation.

Introduction

Continuous integration (CI) is the process of building a new version of your software every time you push a change. Clearly, CI has a big impact on testing, since it means you are never testing stable software. On the other hand, the changes you are testing are much smaller. This means you have to adopt a different approach to testing than was traditionally the case. In this blog, we will explore CI and look at the challenges it creates for testers.

Continuous Integration

The background and motivation for CI

CI isn't new, but it marked a change from the traditional way of building software. To do CI well requires careful coordination and buy-in from both the developers and testers.

Once upon a time, developers would work individually and in teams on developing their part of the software. Every few weeks they would upload all their new code to the build server, along with the code from all other teams. The next stage was to try and resolve all the merge conflicts. Then you could run your build, at which point large numbers of errors probably showed up. Finally, you would have a working build, ready to pass to the testers.

It doesn't take a genius to see the flaws in this approach. Various tricks were used to try and improve this process. For instance, large pieces of software would be broken into small modules and treated as black boxes. As long as the interface worked, you removed the inter-dependencies. However, that can only go a short way to solving the issue. And so, we come to continuous integration.

What exactly is CI?

CI isn't rocket science. The problem is that merging large amounts of new code caused things to break. So, the solution is to merge smaller amounts of code! Martin Fowler (one of the authors of the Agile manifesto) describes CI as follows:

Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.

CI is a pre-requisite for continuous deployment (which we discussed in a recent blog). In continuous deployment, you release the newly built code to production as soon as it is ready. However, CI and CD are distinct from each other, and many companies only do CI.

Why is CI such a good approach?

CI is a powerful approach that can be applied in most companies. CI will benefit any size of team, so let's look at a few of the key benefits.

Frequent merges reduce merge conflicts. Anyone who has fought with git merge will know the pain of merge conflicts. Sometimes solving these is simple, but other times it needs major changes. At the least, it is a time sink.
Developers benefit from rapid feedback, especially when it's related to an error they have to fix. So, if you build and test the code at least daily, the errors are likely to be in code that has just been committed. This means the developer won't have to reload state about code she worked on weeks ago.
Everyone is always working on the newest codebase. This significantly reduces the risk of problems in complex pieces of software. It also means that developers can see what each other is doing much more easily.
Any changes to the backend environment are instantly available for frontend developers to work with. This removes a key source of problems for many developers.
It enables continuous deployment. Even if you don't practice this as a company, being in a position where your code is always deployable is a distinct advantage.
Product managers can also benefit because they get to see the latest product changes instantly.

There are other benefits too. For instance, developers are learning from each other's mistakes and you know who to blame when things break.

What are the requirements for CI?

Enabling continuous integration requires a few things to be put in place. Let's look at the most important of these in a bit of detail:

Code repo. You need all your code to be in a central repository, ideally something like GitHub, Bitbucket or the like. Where you rely on external libraries you need copies of these too.
Single-action builds. You must be able to launch your build process with a single action, such as a CLI command, mouse click or by automatically monitoring code commits. This is where CI tools like Jenkins come into their own
Automated tests. All your tests must be automated so that as soon as the build is complete it can be verified. More on this later.

The CI process flow

Continuous integration follows a simple flow which we show in the diagram below.

When a developer pushes new code to the commit system, the CI system is notified. It then triggers a build. If the build fails, then the developer is told. If the build succeeds, the test system is updated. The automated test system now runs a suitable set of tests (see later). This may be triggered automatically, periodically or by the CI system. The results of the test are passed to the CI system. This either tells the developer the code is OK or indicates that it failed a test.

How to design your testing for CI

True continuous integration would imply that every single build gets fully tested. However, even with the best test automation system in the world, this simply isn't possible. Some tests take a long time to run, and not all tests are equally important. Let's have a look at both these issues.

Time to test

A full set of tests covers everything from unit testing to full regression and UI testing. Unit tests are, by definition, small. So, unit tests can be completed rapidly. By contrast, a full set of regression tests can easily take hours to complete. This is especially the case if you have to reset the system state between each test (as is often the case). Often your developers will push code several times a day. This means there simply isn't time to run all the tests for every build.

The relative importance of tests

As you will know, not all tests are of equal value. Some may be testing an obscure corner case, while others test the central functionality of your entire system. This factor can be used to prioritize your tests.

Test prioritization

By splitting your tests into three categories you can streamline your CI testing. Firstly, every build should be subjected to a full set of (suitable) unit tests and a smoke test. A smoke test is designed to test the minimal necessary functionality of your system. It should show up the most fundamental issues but may miss some obscure ones. You should design your smoke test to run in a maximum of a couple of minutes.

Every hour you can subject the latest build to a more thorough set of tests, so long as they will complete within the time you have. You should choose the tests that are of highest importance.

Finally, you should run ALL your regression tests overnight or over the weekend if any really take a long time. (I once worked for a company designing storage solutions. Since our tests required reformatting entire RAID arrays, some of them took a day to complete).

Potential issues with CI testing

There are a couple of real gotchas that can cause problems for testing with continuous integration. The first is the test maintenance issue. By definition, your CI system is relying on your tests working. The only failures should be genuine code failures. However, many automated test systems suffer from spurious failures (just speak to anyone that has used Selenium). The second is that new code requires you to create new tests. The combination of both these things can take a lot of time and resources to solve. And while they are being solved your testing may be blocked.

How Functionize helps

The Functionize system simplifies and streamlines automated testing of web applications. We make it easy to integrate with CI systems. Our system is powered by AI, which is why we talk about autonomous testing rather than automated testing. What we mean by that is that our system can recover from most test failures without any human intervention. We call this feature "self-healing". We also make writing new tests as simple as writing plain English. Our ML engine then takes these and automatically converts them into tests for you.

robinsonhispeciam1973.blogspot.com

Source: https://www.functionize.com/blog/continuous-integration-how-it-affects-testing