Living Documentation vs. Test Reports: What’s the Difference?

July 11, 2019

Behavior-driven development, or BDD, is all about improving communication between business and technical teams. The goal is to shorten feedback cycles and eliminate misunderstood or vague requirements using real-world examples. These examples describe how software should behave in the eyes of a user rather than how it should work on a technical level.

The process starts with a deliberate discovery meeting between developers, test engineers and stakeholders. During the meeting, the three amigos discuss a user story and come up with concrete examples. These examples are converted into executable specifications that developers must pass during their test-driven development cycle.

Living documentation is the final product of this process — a human readable record of each feature verified and kept current via an automated test suite.

Let’s take a look at the idea of living documentation, how it differs from test reports and what tools you can use to generate and use it in your organization.

What is Living Documentation?

Living documentation is a way to share the definition of your features and have a common support dialogue for your business and technical teams. While you may already be writing executable specifications that read like a documentation, living documentation consolidates this information into an accessible format so stakeholders don’t have to dig through feature files.

It’s called living documentation because the documentation is generated by an automated test suite each time that the application is built. That way, it’s always kept up-to-date. Developers can spend their time working on new features rather than writing documentation, while stakeholders can be confident that they’re always looking at the latest features.

There are many different reasons to create living documentation:

New team members can quickly get up-to-speed on how a product works before diving into the source code, as well as verify that the existing code works.
Development teams can help maintenance teams quickly understand an application after a handover, as well as confirm that everything works.
Auditors can easily see that relevant regulations have been followed by an application, such as HIPAA requirements or PCI compliance.

Aren’t Test Reports Good Enough?

Test reports are frequently used by development teams to finalize user stories after they’ve been completed.

For example, a quality assurance (QA) engineer may generate a test report to confirm that a user story meets the acceptance criteria before marking the story as complete. These reports are typically very technical in nature and only the result summaries are shared with business teams.

Living documentation is designed to be used throughout the development cycle (including before development starts) and describes features in easy-to-understand terms for non-technical teams. The idea is to provide a roadmap for future features and a user manual of existing features.

John Ferguson Smart provides an excellent example of these differences on his blog post Living Documentation: it’s not just about test reports:

Stakeholders can see a roadmap of what features are coming up and where existing features stand in the development process, while developers have a clear idea of what to do next. In addition, everyone can be sure that they’re on the same page in regards to the business requirements.

How to Generate Living Documentation

There are many different behavior-driven development tools that make it easy to generate living documentation.

We built HipTest to make collaboration between business and technical teams even better. Using a web-based interface, we sought out to make it easier for business teams to login and not only see living documentation but also collaborate on the development of BDD tests following discovery meetings. The goal is to ensure everyone is always on the same page.

HipTest has combined living documentation with several other features critical to the BDD process:

Scenario editor that makes it easy to create scenarios on-the-fly without writing any code.
Reusable step definitions that concentrate test maintenance efforts in a single place.
Support for over 20 testing frameworks, including Cucumber, Specflow, Selenium and TestComplete.
CI/CD integration with Jenkins, Bamboo, TravisCI, Shippable and many others.

"With BDD and Living Documentation features, it is even easier for anyone to understand what the product does,” said Florent Vaution of Ouest France, a HipTest customer. “The design of new executable requirements is done really quickly after the conversation between the PO, the developer and the tester. We went from a few hours to a few minutes!"

Another example, Pickles is a living documentation generator that takes specifications written in Gherkin with Markdown descriptions and converts them into always up-to-date documentation in outputs ranging from HTML to Word docs.

The Bottom Line

Living documentation is a great way to keep business and technical teams on the same page by maintaining an always up-to-date definition of application features.

There are many different tools that you can use to generate living documentation, ranging from Selenium BDD to Pickles, but HipTest incorporates a wider range of features designed to simplify a wide range of behavior-driven development processes — including living documentation.