Confusion will be my epitaph

As I crawl a cracked and broken path

If we make it, we can all sit back and laugh

But I fear, tomorrow, I'll be crying

King Crimson, Epitaph 1969

Cast your mind back to the last time you needed a definitive answer about how your application works. Perhaps you were a considering an upcoming change request or were trying to work out if a customer-reported issue was a defect. Was your first thought “I’ll check the documentation”? My guess is “No.” Instead you either asked someone on the team or explored the relevant behaviour in the running application. That’s because the software is the source of truth – the code determines how the application behaves.

There’s a thought experiment that I ask clients to try: “If you had to choose between unreliable documentation and no documentation, which would you prefer?” It doesn’t take long for most people to arrive at the conclusion that unreliable documentation is worse than having no documentation. If there’s no documentation, then at least you’ll go straight to the source of truth without wasting time with unreliable information.

I then ask them “which do you have?” The answer is almost always “unreliable documentation.” Why? Because we are required to have some sort of documentation and there’s just no way to ensure that is stays in sync with the software. Except that there is. Documentation that stays synchronised with the source of truth (the software) is living documentation.

Living documentation is a big topic – far bigger than I can cover in this blog post. In fact, in 2019 Cyrille Martraire published a whole book on the subject, weighing in at over 400 pages, accurately (if somewhat unimaginatively) called Living Documentation. It covers the core principles of living documentation and explores in detail the practices and patterns that enable your teams to tackle confusion and improve their development process.

Specification by Example

Like cheap wine, long paper documentation ages rapidly and leaves you with a bad headache...

Gojko Azdic, Specification by Example 2011

When Gojko Adzic wrote Bridging the Communication Gap in 2009, techniques like test-driven development (TDD), acceptance-test driven development (ATDD), and behaviour-driven development (BDD) were widely discussed, but less widely used. Gojko identified that each test (or scenario) was an example of how the system was expected to behave, and he went on to coin the umbrella term “specification by example” to describe them all.

The Agile Testing community had long understood the power of examples to aid communication between business and delivery teams. Steps had been taken to present those examples in a format that was accessible to everyone involved in the specification and delivery process – notably FIT/FitNesse, Concordian, JBehave, and Cucumber. These tools provided the ability to write documentation that could also be run as tests.

If the documentation “passes”, then it accurately describes the application. If the documentation “fails”, then it is actively signalling one of the following situations:

• The documented functionality has not been implemented yet
• A defect has been introduced to the application and needs to be fixed
• The documentation is out of date and needs to be updated

The documentation is validating itself against the application. Living documentation had become a reality.

Cucumber Open

Since its launch in 2008, the Cucumber open source project has become the leading tool for automating business-readable documentation (written in a Given/When/Then format called Gherkin). As well as the languages supported by the open source team (Ruby, Java, JavaScript, C, Go) there are countless ports to other popular languages. After Cucumber’s acquisition by SmartBear in 2019, the project has been renamed Cucumber Open, to distinguish it from SmartBear’s commercial products.

Cucumber Open reads the Gherkin documentation, executes relevant test automation code, and displays each line of the documentation in green (if it “passes”) or red (if it “fails). To assist with publishing the living documentation more widely, Cucumber Open can also output the results in HTML or JSON formats, allowing organisations to upload them to shared Wikis or websites.

This fulfils the basic needs of living documentation – a format that anyone can read, which immediately informs the reader whether it accurately describes the current behaviour of the application.

The shape of things to come

Over the years, practitioners have refined their description of BDD. Thanks to the work of Liz Keogh, it’s widely accepted that BDD is comprised of three core practices:

Cucumber Open supports the Automation practice, but its basic living documentation support is often insufficient for real-world projects. Consider these very common situations:

• different parts of the application are written in different programming languages
• functionality is spread across multiple repositories, perhaps in a microservices architecture
• teams need to know the difference between the documentation for the current production release and what's being worked on in development
• team members spread around different geographical locations need to collaborate on writing the specifications

Cucumber Open makes no attempt to manage the results output from multiple runs, leaving that as an exercise for the user and as an opportunity for tool vendors. At SmartBear, we’re working hard to provide tooling that support these use cases and more – helping dispel confusion no matter how complex your product.

I’ll be bringing you more news on this in the near future!