DevOps has changed the testing game
  November 27, 2018


During the past 4 weeks Vincent and I met with 300+ software teams and presented continuous testing at 3 conferences in the US and Canada. We had so many questions and great feedbacks that we decided to publish a blog post that summarises these presentations.

At Hiptest we deploy between 1 to 10 times a day in production and the pace keeps accelerating. Like many software teams, we have embraced DevOps and the impact is far bigger than simply automation or continuous deployment. It has been a game changer for us as we started to adopt the 3 main principles:

  • Emphasize the performance of the entire system, as opposed to the performance of a specific silo (Development, QA, Ops…)
  • Create a continuous flow from idea to production and get quick feedback loops
  • Make experimentations and keep learning about the system using product analytics, user feedbacks…

Define, Test, Learn, the Hiptest process


In this context you have to revolutionize the way you do testing. Anyone can deploy all the time. The questions are: Can you deploy good software? Will it match the needs of the users? Will it work as expected and with a good user experience?

The answer is continuous testing, baking quality into the entire process from idea to production. In this post we share the 3 steps approach to testing used by thousands of highly successful DevOps teams.

Step 1: Test the idea

That’s the Shift left and where BDD comes in, helping the team to discover the problem and surface a solution in collaboration before the code is created.

BDD is a way of describing the application in terms of behavior through examples in plain text. These examples support the conversation and help the cross functional team (marketing, product owner, developer, user…) to create a shared understanding of what should be developed. Examples become part of the definition of done.

There are many ways these can be captured: fixtures, tables, Gherkin syntax (given/when/then).


Test steps in Hiptest


At the end of the discovery session the team is aligned on “Why do we develop this feature?” (business problem) and “what should this feature do?” (different examples). It makes the development process much more efficient.

The examples can be shared with the business users or customers to have early feedbacks and iterate if needed instead of wasting time developing a feature that doesn’t address well the business problem.

Step 2: Test the code

The examples defined earlier are used to drive development and the tests created can be automated, either by a tester or, more likely, by the programmer writing the code. Thus “The code is done [and ready for human exploration] when all the tests are passed.”

BDD should be used together with TDD to ensure a good design of the code and correctness of the feature. Test automation (unit, service, end2end) and continuous integration (CI) provide the safety net to make rapid change possible.


Automated tests at Hiptest

Test pyramid at Hiptest

Step 3: Test in production

Once the tests for a feature are executed by the CI/CD, you can group them together and get both a living documentation and a regression test suite, essentially for free. Unlike a word document, that documentation written in a business readable way will always be up to date by design as it is executed by the CI before deployment in production. The documentation reflects the actual behavior of the system.

For example, the support team can use it, understand the behavior of a particular feature and provide better answers to users...

And the most successful teams extend this approach by adding the product analytics to the feature documentation:

  • Business assumptions related to the feature
  • Current business metrics (usage, activation...)
  • Performance

This way, every stakeholder, marketing, product owner or developer can look at a feature, understand the behavior, get feedback from the production and measure the impact of the feature in real time! That's the value of living documentation.


Idea to production - Hiptest


This is how most performing teams dramatically optimize their development effort. Which feature should be dropped? which one should be tested more thoroughly? Business stakeholders and developers are using the exact same data to support discussion about future development and make decisions that will have the biggest impacts.


In a DevOps context, speed of feedback loop is key. Continuous testing provides a great framework for baking quality into the entire process from idea to production. Testing doesn’t stop at the CI and combining BDD with product analytics really helps mitigate the biggest risk we face in innovation and design: making something that nobody wants.