Cucumber Balance and Belonging Survey 2021
  January 18, 2022

One of our big goals for 2021 for the Cucumber community was to get more new contributors. We had noticed that the current set of regular contributors that we have is predominantly super-experienced, almost exclusively white, guys.

Now don’t get me wrong: each of these folks is someone I’m very fond of, and we’re very lucky to have them. But we had a hunch that this demographic isn’t very representative of our users overall, so we wanted to try and understand where and why we’re losing people in the funnel from user to contributor. Because if we can fix that, we’ll have more people making regular contributions!

So back in October we ran a survey. We wanted to try and answer these questions:

  1. Who's using and contributing to Cucumber today?
  2. Is our diversity decreasing as we move along the funnel from user to contributor?
  3. Where the bottlenecks are in that funnel? Where do we lose people?
  4. What specific things could we do to be more inclusive?
  5. Are any particular groups struggling more than others?

Data set

We promoted the survey via Twitter and our mailing list. We had a banner running on the cucumber.io homepage. We also tried to contact everybody who’s ever made a commit to one of the cucumber codebases. Unfortunately, only about 300 of the 1000+ of them have shared their email address on GitHub, but I emailed all of those 300-or-so people personally. In retrospect we could perhaps have used a GitHub issue and tagged them to try and contact more of the contributors.

We ran the survey for about a month, and it was promoted on the homepage for about 10 days.

In the end we had 71 responses.

So it's not a large dataset relative to the number of people we believe are out there using Cucumber, but it’s a start. If we assume that the data is representative of the community, then we can use it to draw some tentative conclusions.

Demographics

Of the 71 people who responded:

  • 75% identified as men, 18% women and 5% non-binary. Although we’re very male, this is better than the average open source project: GitHub’s 2017 survey showed that 95% of all their contributors are male, 3% women and 1% non-binary. We still have a long way to go though! The US bureau of labour statistics shows that about 23% of programmers in the US are women, and even that is still less than half of what it could be.
  • 85% are educated to bachelor's degree or above. Our women and racial minority members are highly educated. Amongst women, 87% are educated to bachelor’s degree or above, and amongst those from a racial minority, it’s 100%.
  • 43% are over 40 years old.
  • Almost half (47%) speak English as their second language
  • 10% identify as being a member of an under-represented racial or ethnic minority.
  • 13% identify as being from the LBGTQ+ community
  • 57% are parents or caretakers

The funnel

There are many more ways you can contribute to an open source project than by writing code! Even just using the thing is a contribution of sorts – the more users we have, the more the maintainers know that their hard work is worthwhile.

We asked each respondent a bunch of questions about various ways they’ve interacted with Cucumber. Here, I’ve tried to organise these into some kind of logical progression from using Cucumber, joining the community forum, through to merging other people’s pull requests:

We can see that, as you would expect, the number of people participating in these activities declines as you move along this funnel. There’s an apparent anomaly with the number of people submitting pull requests, but I wonder if this is down to the audience we pulled in, since I guess a good number of people responded to my personal email to existing code contributors.

It’s useful to see this model, but what we really wanted to learn was how balanced these activities are across different representations like race and gender.

Due to my weak Excel skills, the funnel is flipped upside-down for this next chart, which shows how different minorities are represented in each of those contribution activities:

The “other” category here is basically all the people who are straight, white and male.

Although it’s noisy, it seems that we tend to have less women interacting with GitHub in general than participating in other activities, and we can see that, of the people who responded to the survey, nobody from a racial minority or who’s a woman has merged someone else’s pull request.

Interestingly, the social media and promotional activities (like writing blog posts or giving talks) are much more balanced. Cucumber School also has one of the more balanced representations.

Belonging

Next, we asked some questions to discern how well people felt they belonged in the community.

What strikes me from this chart is how low the scores are from women on each of these questions.

All of us, especially the men in the community, need to reflect on and think about how we're behaving to try and alleviate this. This isn't just about "not being mean" it's about making a proactive effort to ensure people do feel heard, that they do feel they have somewhere to turn when they're stuck or lost. The whole nature of this problem is that people won't speak up, so it's up to us to open up the space where they can.

The suggestion box

The last question asked what specific things we could do to make the community a more welcoming place.

There was a whole load of data here! My colleague Amber and I worked hard to try and discern the major themes and highlight the most actionable feedback.

Here are some of the most representative comments:

I think the comment about people suggesting someone can fix a bug themselves is really interesting, and reflects a cultural shift in open source. I remember when I first started in open source I would often encounter this attitude: “Well, if you want this new feature, send us a pull request”. What I'm learning over the years is that, if we want to build a strong community, we need to be grateful for every effort to engage with the developers of the project, providing people with support, encouragement and coaching to make them into more valuable members of the community. If we fail to do that, we’re only going to get contributions from the most determined, thick-skinned people, and those aren’t necessarily the ones with the most to offer. When you're in the early days of a project and you're feeling rushed and tight for time, it's tempting to be offhand with people because you're worried about having your time wasted, but if you want to build a strong, sustainable community around your project, that's a false economy. Rather than gatekeeping, we need to invest in people who show an interest in the project.

In a similar vein, telling people “this has been asked before” isn’t helpful. If newcomers are all asking similar questions, we can hear that as feedback that we need to provide better mechanisms for sharing the community’s tacit knowledge. If we do that, we at least have somewhere to send people when they ask such questions, and they might even find that answer for themselves. This is one weakness of using Slack over the old Google Groups discussion forum, where it was much easier to find and reference old threads, but it’s not just about tooling. Again, this is about those of us steeped in the community to remember to put ourselves in the shoes of people who are just starting out on their Cucumber journey, and take the time to write up and share these pieces of tacit knowledge.

It’s clear from this feedback that our documentation for new arrivals, whether users or contributors, has significant room for improvement. Someone once described the experience of cloning an open source project’s code as being like arriving in a new city: you need a map! Improving the documentation like this is big project, but something Aurelién and I are keen to get started on for 2022. We also need to provide better examples, which we’ve just started working on.

I love this suggestion about rolling out the welcome wagon, and I think that’s something we’ve really been trying to start doing that this past year. Jo, our community manager, has been great at making sure people who’re interested in contributing feel welcome and can find their feet. We’ve been running a regular beginners ensemble on Fridays for people who want to contribute to the code, and my calendly is always open for people who want a 1:1 session to either figure out what to work on, or to pair on something they’re working on. I’ve met so many wonderful and interesting people, and learned so much this year from these sessions!

Another way we can welcome new contributors is to make sure they reaslise they can get paid for their time. Jo and I have been working on some better documentation and processes around how we pay contributors, and I hope to have something to share in the first few months of 2022.

As we move into 2022, it’s clear we need to continue the work we’ve been doing to improve the experience for new contributors, because they’re the lifeblood of our community. As we make the experience better for them, we make it better for everyone.

Thanks for reading, and thanks to everyone who filled out the survey.