As you might know, we use Hiptest to write the acceptance scenarios for Hiptest features. Recently, we made a huge refactoring to our tests so they use Gherkin syntax (given, when, then). This was also an opportunity to enhance the way tests were written and ease testing of collaborative features.

A quick word on Hiptest

Some of you might not yet be familiar with Hiptest, so a short introduction might be necessary. Hiptest is the test management tool on the cloud. In enables creating test scenarios, planning their execution and keeping track of the execution results (both manual and automated).

More than that, Hiptest is also a Behaviour Driven Development framework: the test scenarios should not only be meant to ensure the quality of a software, Hiptest allows them to be a base of discussion for all stakeholders of a project (developers, marketing, product owner and all others).

In order to ease the discussion around the tests, Hiptest make them easy to read (no need to be a tester or a developper) and easily accessible (it takes a few seconds to sign-up).

As I said in the excerpt, we use Hiptest to test Hiptest, let's have a look on the existing tests.

An overview of the existing tests

Hiptest development is going on for a few months now and of course, the way we write tests is following the features offered by the tool and our own understanding of the capacities of the tool (yes, we also discover the possibilities offered by the tool we are developing, that's as scary as it's cool). The main problem is that it leads to a completely heterogeneous set of tests. For example:

• some tests are doing everything using the UI (user creation, project creation, writing scenarios ..) while the others have a setup done by an action word (which creates users and project by writing the database)
• tests mixing of "pretty calls" (our internal name for the inlining of parameters in action word title) and classic calls to action words
• tests mixing "BDD" action words ("scenario 'My new scenario' is created in current folder") and assertive names ("Create scenario 'My new scenario')

[gallery size="medium" ids="111,112,109,108"]

This heterogeneity in tests is not a blocking issue (they do the job they were written for: ensure the tool is working). But they cause three problems:

• we are developers and we spend time keeping our code clean ("Code is poetry" ;) ). The same goes for our tests.
• when writing a new test, we are not sure what is the best way to write it (BDD or not ? Create everything using the GUI or use a setup ?)
• some setup action word are doing more magic than expected

The two first problems are pretty simple to fix: the first one by a collective effort, the second one by discussion and agreement in the team on how to write new tests. The third one is a bit more tricky and is due to the way Hiptest is designed.

Entering the magic world of collaboration

There is one fondamental concept behind Hiptest: we want real time collaboration on a project and we need to test it. So most of our scenarios are ensuring it, but maybe not in the best way as shown in the scenario below:

[caption id="attachment_116" align="aligncenter" width="864"] This scenario also tests collaboration but you might not see it[/caption]

The first action word called by this scenario creates more than a simple project, is also create two users (John & Bob) and a browser for each one. John is doing the modifications on the folder and both users check that the updates are correctly applied. Could you have guessed it simply by looking at the scenario ?

The main problem caused by the magic in the action words implementation is that we could not be sure that a feature was also tested for collaboration. Another problem was that we were testing the feature and the collaboration at the same time, so when the test fails we don't know if it is due to the way the feature is implemented or if it is a collaboration problem.

Another problem was the multiple browsers and happened to me a few weeks ago, when implementing the manual ordering of scenarios. The feature was working fine and it was time to implement the integration test describing it. If you used this feature before, you know it's using drag & drop for ordering. When implementing the action word making the drag & drop, nothing was happening on screen, whatever the Watir keyword used. I checked I was using the correct selectors for the elements, everything should have worked. It took me a complete morning of work before I understood the browser I was seeing was the one of the second user, not the one actually performing the drag & drop. Half a day lost that should have not been lost if the scenario had been more explicit (and another day lost due to incompatibilities between Watir and jQuery UI drag & drop, but that's a completely different story ...).

Anyway, it was time to make our tests clearer (and use our pretty calls and BDD syntax everywhere, we should also benefit of the cool features we spend time developing).

Introducing Harry and Hermione to fix magical problems

So, our refactoring of tests has to fix a few problems:

• they must be homogeneous
• they must be explicit about collaboration
• they must enable us to know easily if a feature is failing or if the collaboration is failing

The first problem is easy to fix: we'll rewrite everything so it uses BDD and pretty calls. Also, all tests will use sample projects for setup with an exception when the test is ensuring GUI works as expected (for example checking the creation of a new project).

The two last problems are fixed by using three action words:

• "Harry" that creates a user named Harry and grant him access to the project created in the setup)
• "Hermione" does the same for user named Hermione
• "Harry and Hermione" calls the two previous action words

Now, when implementing those action words, we need to know the list of users available during the test. This is pretty simple to do with Ruby:

def make_user(email)
  # @test_users is a list initialized at each test
  @test_users << User.create(email: email, password: "s3cr3t")
end
def harry
  make_user('harry.potter@hogwarts.com')
end
def hermione
  make_user('hermione.granger@hogwarts.com')
end

When we implement the setup action words, we use this list of users and grant them access to the project:

def project_f1_sc1_and_f2_sc2
  # Project and items creation is not shown here
  @test_users.each_with_index do |user, index|
    create_browser(@browsers) if index > 0
    login_as user, scope: :user
    @browsers[index].goto "/app/#/projects/#{@project.id}"
  end
end

As you can see, the browsers are created after the project is created and not when the user is created. This is due to the fact that we want the users creation as the first step of the scenario.

Now we need to fix the problem of collaboration in the action words implementation. This is more simple than it seems, we just have to follow these rules:

• if an action is done (renaming an object for example) then it is only performed in the first browser
• if anything else is done, all browsers will do it

To help us in this task, we created a simple helper "all_browsers" that will perform the action in all browsers created during the test. Below are two examples of action words implementation, the first one is a renaming action (so only done by the first browser), the second one is navigation so done on all browsers:

def current_scenario_is_renamed_new_name(new_name)
  span(id: 'scenario-name').h1.click
  span(id: 'scenario-name').input.when_present.send_keys new_name
  span(id: 'scenario-name').input.send_keys :enter
  wait_until {!div(:id, 'spinner-xl').visible?}
end

def action_words_page_is_opened
  all_browsers do |b|
    go_to_page('Action words', b)
  end
end

Now you might wonder how this helps us fix our two issues, well that's pretty simple. Let's have a look at the tags management test shown earlier, written with our new action words:

[caption id="attachment_128" align="aligncenter" width="860"] The same scenario, now using BDD, pretty calls and the Harry keyword[/caption]

Here, the test is only used with user Harry. If we run it and it fails, we know we have a problem with the feature (or the way we implemented the action words, that's possible too) and this is not a collaboration problem.

Once the test is passing as expected, we can update it and grant Hermione access to the project:

[caption id="attachment_129" align="aligncenter" width="867"] Now Hermione also access the project and we can ensure collaboration is working[/caption]

Now if the test fails, we know that it is due to a collaboration problem (for example the new information is not sent to all users of the project). And of course, now it is clear when reading the test that there is two users (and two browsers) for this test.

Conclusion

This big refactoring was not that simple, but it helped us getting cleaner and clearer tests (and that's something we believe in). The way we integrated collaboration testing is also way more flexible (we have two users now but we could have has many users as we want instead and almost nothing to do to handle it). It also fixed a few issues we did not discuss for this post (like when we had some half a dozen Chrome windows left opened when the tests were failing ...). A nice side effect of this refactoring is also that we can know which tests are not checking collaboration by opening the action word "Harry" and see in which scenarios it is used.

We know this is not the last refactoring we'll do on our own test set. In the coming days/weeks/months, we'll have another big update on Hiptest, like when we introduced "BDD style" scenarios. We don't know yet when that will be nor what it will be, but we know that we'll have to make our tests use it.

This refactoring also helped us get a better grip on the application, spending lots of time on the application as a normal user and understanding the problem they might encounter when writing dozens of test a day (yes, we plan to make the editor even simpler and user-friendly ;) ).