7 Reasons Why Your CI and Automated Testing Might Actually Slow You Down
Have you ever wondered if implementing CI was actually worth it? Do you feel that the extra automation on builds and tests seems to come at a great cost? That there’s still many bugs in production, but now longer cycle times from developers that need to fix
We analyse many development teams and pipelines and see it all the time. The team has implemented some degree of Continuous Integration, for example adding some automated tests and setting up a service that builds the application and runs the tests. This alone has often taken a lot of time and then the problems start.
- As more tests are added, build times go up significantly
- As builds take longer, developer behavior starts to change, adding to switching time and frustration
- Builds can fail for many different reasons
- Developers start relying heavily on CI, not testing (enough) locally
- Actual total test coverage remains low, letting bugs through and eroding faith in CI practices
- Cheap CI services generally have longer build times, unnecessarily worsening the problems
- Developers Become Lazy
- Expecting full CD benefits from CI
More Tests. Longer Builds.
It makes sense that as more tests are added, build times go up. This seems unavoidable. We want to get the coverage to 100% to get the maximum benefit out of CI and make manual testing a thing of the past, but at the same time we don’t want take the time to run all those tests, either locally or on CI builds, because it starts to take way too long.
We see many teams jump into CI with a bunch of functional tests that basically replace some of the manual testing. This makes sense from a functional standpoint, as we are trying to replace the manual testers, so why not test the same thing they were doing.
Unfortunately it turns out functional tests and e2e (end-to-end) tests are extremely slow, especially when they test the full integrations, including third party calls. They are also very error-prone and require lots of maintenance as every little change in the design or functionality makes you fix these tests.
The alternative? Unit tests. It’s a bit counter-intuitive to start with, but switching from functional tests to writing unit tests instead is the most impactful thing you can do to speed up your automated tests. We’ve improved many pipelines that took 10-60 minutes to under 1 minute using this method, and then we even use some functional tests to test controllers.
Longer Builds Change Developer Behavior
This research shows that when CI builds take longer than 8-12 minutes, it has a profound effect on developers’ behavior. Developers start to group features or switch to other tasks while waiting for CI to pass. Combined with CI failing often, this can lead to developers working on many things at the same time or simply standing at the coffee machine more often as they wait for their builds to pass.
As a developer for many years, the long build times and then still having them fail was a major frustration for me. Imagine having builds taking up to 45 minutes to pass on Jenkins, and then getting a red light. I can tell you it’s even more frustrating if it happens 3 times in a row, and unfortunately that happens way too often.
Builds Fail for Many Different Reasons
CI and your local machine are different. Even if you could test everything locally every time before you commit, how often will CI still fail on your perfectly good code? The answer is unfortunately that this happens too often.
There are some avoidable patterns that cause CI te fail:
- Using different environments. Technologies like Docker make it easy nowadays to use exactly the same environment, to avoid missing a package on CI that you did have installed locally already.
- Using a different database. You can imagine, especially for functional/e2e tests, that having a different database can lead to different results and failing tests. While testing locally, your database may change, easily leading to these discrepancies with CI. You can fix this by either resetting your database before the tests or, even better, by using some sort of in-memory database solution.
- Using random input. For some reason many developers love the idea of providing random input for their tests, so they can test their code with a wider array of variables, ensuring it’s stability. Unfortunately this also leads to CI failures. We advice to avoid using any random input or methods in your automated tests.
- 3rd party problems. Another common issue is doing a functional test that calls a 3rd party api or service. This slows down your test, but also causes random failures with timeouts or other (temporary) problems. This is always an issue for debate, but we choose to mock 3rd party responses and accept that every once in a while some API changes without backwards compatability and warning and causes issues. This does not happen often in our experience.
Developers Stop Testing Locally
There comes a point when implementing CI – usually way too early – when developers stop running full automated tests locally and instead start to rely on CI running the tests for them. 95% of developers using CI say they don’t run all tests locally before pushing their commit.
We all know that programming often has unintentional side-effects and the bigger your application is, the more likely it is that your change broke something on another part of the application you weren’t even working on. It’s an important benefit from CI that it catches these problems, before a user runs into them, but it also makes the CI often fail unexpectedly.
The earlier you find a problem, the less overhead time it takes to fix it. That’s a simple truth. But the problem is that when local testing starts to take many minutes, it often feels like it would be better to outsource it to CI, while you work on something else. This is true only if the failure rate and the extra overhead time those take doesn’t exceed the time of waiting for tests to turn green locally.
Testing locally generally takes a lot less time than the full CI build. Especially if you optimise with unit tests, you can run tests locally in < 1 minute, while the build may still take up to 10 minutes. That’s why we always force developers to test locally before committing, as we feel it saves a tremendous amount of time.
Low Test Coverage Erodes The Faith
It’s way too uncommon to see 100% automated test coverage. When your automated tests don’t cover your full application, it means you cannot rely on the automated tests. This means you still need manual testers.
We’ve found this often goes hand in hand with the developers and management not trusting automated testing all the way. Even if there would be a 100% coverage, they would still use manual testers. Worst of all these manual testers often click through an application and look at the same things a proper functional test can do for them.
If you don’t trust automated tests and keep using manual testing, then what are you slowing down your process for? Some would answer that reducing the number of problems found by manual testers is enough benefit from automated testing. But are you really using less manual testing then? We often feel we are doing the right thing, without the actual statistics to back it up. Can you present numbers on cycle time before and after implementing automated testing? Are you 100% sure it has improved your cycle time, not slowing you down?
We argue that automated tests are only useful if you can rely on them in such a way that manual testers (not talking about product owner acceptation) are not needed. In fact we arrange our development in such a way that product owners can accept stories in production and use Continuous Delivery to have them rely on statistics for this. Using proper monitoring you can quickly fix any issue that does come through for less cost than having a test and acceptation environment.
Not everybody is ready for that level of Continuous Delivery though, but consider at least moving that way by knowing and improving your test coverage.
Cheap CI Increases Build Times
If you get nothing else out of this article, you may consider simply looking at your CI provider or server and upgrading it. This small change can bring build times back as much as 70%. If this brings you below the 8-12 minute threshold, it will help your team keep a fast pace, reducing switching and waiting time.
There’s many other more technical ways to improve build times for the same result. If you give smart developers some time, they may load the database faster, load a pre-compiled image, build in more cache layers or find ways to run tests in parallel. This may well be worth the effort too, but nothing is as simple as upgrading your CI servers to a higher processing power and see the results.
Developers Become Lazy
One of the worst problems we see with CI is what it does with the mentality of developers. It used to be that a developer was responsible for whatever he put into production and had a sweaty finger when deploying his application. This kind of stress is not good, but at least he would think twice before deploying anything and think about all the consequences, including looking at the server directly after deployment if nothing was going wrong.
The more we automate or move down into the pipeline (such as manual testing), the more developers feel it’s not their responsibility anymore. I’ve seen teams deliver feature after feature, seemingly at great speed, only to again and again get red builds, tester feedback and bugs to solve in production.
On top of this we see developers investing less to really understand the feature, relying on the product owner or tester to tell them if they did not understand it properly. And there’s even less involvement from developers to the bottom line of what the company is actually trying to achieve. It becomes production work, like in a factory, where they are just one cog in the wheel, only caring that their code is doing what the requirements say and that the CI succeeds. The rest is not their responsibility.
Expecting Full CD benefits from CI
Continuous Integration and Continuous Deployment or Delivery are terms that are sometimes seen as synonymous by those who don’t know too much about it. CI is most commonly used to describe the process of taking a specific branch or version of the software and automatically executing some processes on it, such as automated testing or static code analysis. It doesn’t say anything about the rest of your process.
Continuous Delivery comes from LEAN practices and properly implemented it aims to bring down cycle times and get features to market quicker, so you can get feedback from actual customers.
Imagine two scenarios:
- A developer creates a new feature. Maybe he creates and runs a few automated tests locally, then lets CI handle the rest. It takes 3 iterations before CI turns green, for which he interrupts another task 3 times before he can finish this one. Then a manual tester or product owner comes in and tells him it’s not working as intended. Again he goes back, and again CI fails. Probably we’re in day 3 by now and finally it passes to production and hopefully there’s no further issue there.
- Another developer in another team creates a new feature. He runs all automated tests locally and only when they all turn green he pushes his branch. He knows there’s no manual testing or acceptation, so he has taken extra care to understand the requirements and perhaps shows a screenshot to his product owner to confirm that’s what he wanted. After a code review from a team mate (mostly for educational purposes), he merges the branch confidently and the whole thing is deployed automatically to production. We still hope there’s no further issue there, but he knows that he has proper monitoring and if there is a bug, he can get it fixed and deployed within 15 minutes.
We know scenario 2 scares the shit out of managers, but can you see how it would greatly reduce the cycle times and therefore time to market? What if the feature was not needed at all, or even though the product owner meant the best, was designed improperly. Customers will help identify the problem much earlier and changes can be made very quickly.
You may have thought, as many do, that your CI has actually improved the lives of your developers and QA team, but perhaps we’ve given you some food for thought if that’s actually true.
If you have any doubts about the effectiveness of your CI or the quality of your developers, let us perform a free audit of your processes to see what more can be improved. CI is just one link in the chain, we can help you find other weak links too.
And if you’re quick you can benefit from our special offer. Until 21 June 2021 our development review is free of charge.