Recently at a meet-up there was a talk on writing Cucumber feature files in Cypress, a question was posed to an audience of around 100 - "Who here has tested their stuff with Cucumber?" to which a mild hubbub felt quite negative including a couple of guffaws and scoffs from the back.
I went to another meetup just days later where 2 of the speakers spoke about Cucumber and had polar views as to its value.
I decided to do some quick Googling:
On What’s your opinion on BDD with Cucumber Eliot Sykes says:
"In the end, for me, the time spent writing the glue code between the plain English specs and driving the UI just wasn’t worth the trade-off in that particular project."
This is perfectly valid given that he didn't have the glue code already written as it is a lot of work, but what if that glue code was already written.
I've seen this first hand at a couple of different companies. Developers reinventing the wheel and spending time having to write extra code only for other colleagues to say "the business won't even look at it".
I then stumbled across: Why Cucumber and the Gherkin Language should not be used for Software Testing which I'll dissect. Gonzalo the author says:
"Other technologies provide more benefits as far as code reuse goes and reduce development time significantly, way over anything Cucumber has to offer."
He completely misses the point behind Cucumber and BDD (the execution of feature documentation written in business-facing text). Granted it can be a fairly long process and learning curve to get to the stage of writing good BDD scenarios, as Cucumber's tools feel fairly unopinionated as to how Gherkin steps should be formed out of the box, and the combination of tools are plentiful.
There is a sensible option though. Keep going!
He continues with:
"Computer Science, and programming, is a specialized skill, which allows engineers to produce well written software."
This is very true but what if manual testers could write automated tests without a huge barrier to entry. Wouldn't that be something to strive for? Engineering could always guide the organisational direction of files as the number of project features grow, however this inclusivity would certainly help bridge gaps between engineers and the business. In my experience, UI developers don't really want to be spending their time automating black box tests if there are other resources available and manual testers certainly do not want regression bugs.
The author then proceeds with an analogy to dentists using plain English instead of technical terms. The analogy doesn't really work, as you wouldn't share your Acceptance Tests with the end user. Imagine users coming to your app or website and you showing off your Gherkin statements in some modal popup window!
He goes on to explain at length that globals are bad.
Indeed Globals should be avoided / reduced by using a framework and modules. This may differ between the different languages Cucumber is written in, but with Cucumber JS you're able to use ES6 modules to encapsulate things and using this which refers to the "World" should be kept to a minimum within your tests.
Referring to cucumber step definitions he says:
"There is an extra layer"
This is similar to what Eliot says. However with pre-written generic step definitions, that extra layer is almost entirely removed. Then...
"Working and matching regular expressions properly is more time consuming since they are hard to understand"
This is a fair point however you can do things here to help yourself out. His example is rather OTT but I have witnessed some particularly unnecessary regexes. Keep them simple, use a Library that already has 90% of them pre-written and has them documented such as Courgette, and put a comment next to anything that takes more than a few seconds to work out what is going on.
"The fact that many arguments arrive as strings deprives the engineer from the benefit of type checking"
This misses the point that you're black-box testing a UI. Everything you see on a webpage is a string, even the value of an input with type="number" is of type string. If you wish to cast to other types for semantics, you're free to.
To paraphrase "Every method produces side effects". This is simply not true and where methods do produce side effects it would be the same case with using any other test framework.
"your system will reach an untouchable-unmaintainable state".
This can certainly be the case if you aren't using a framework in many different applications.
In: WHY BOTHER WITH CUCUMBER TESTING? the author Jack states:
"In my experiences both as a Ruby contractor and as an employer of programmers, most feature files I’ve seen are composed of the web steps included with Cucumber by default."
Annoyingly there's no date on the post, however I do not believe this is the case any more. Regardless of whether the examples are there by default or not, this is usually what tends to happen as developers miss the end goal (to provide the business and themselves with documentation of what is test, not how).
"The Cucumber way says you sit with your client and determine, feature by feature, the business value that each piece of functionality serves. That’s cool, but it isn’t a realistic job description for many programmers. High level analysts and consultants might sometimes do this, but the likelihood is that you aren’t acting as a high level consultant or analyst on your current project, and so Cucumber is inappropriate"
In the many many agile teams I've worked with, developers are pretty much always part of the process when defining stories. The ideal 3 amigos sessions in cross-functional teams happen too seldom but developers do work with product owners very often to define the work they will pick up in the near future.
"The editor’s ability to understand the signature of a method makes tasks such as finding method definitions and uses simple."
This is a valid statement and although the generic step definitions are very well documented in Courgette, this IDE functionality will be a future addition to Courgette.
Some of the subsequent arguments against Cucumber are valid, most of which are related to web steps, which are addressed with Courgette, as Gherkin web steps don't have to be used. Regardless of whether they are used or not though, true business-relevant high-level BDD scenarios can be produced with ease with Courgette.
The conclusion states:
"Swap Cucumber for pure integration tests using Capybara, and you’ll be surprised by how much more productive you can be."
The more you go to configuration over code, as in Courgette's implementation, the less likely are to code yourself into unknown corners and get bogged down into the already difficult task of automating the testing of your UI.
Then I stumbled on Why I recommend against using Cucumber and its reader's comments, and it confirmed what I'd heard at the first meet-up, a fair few developers are seriously turned off by Cucumber.
If we ignore the fact Cucumber in the various languages doesn't provide us with an opinionated way of how to use it and the challenges that poses, and focus on it's reason for existence we ask the question:
Would it be of any value if were instructions in the box and it didn't cost us much development time to use?
The answer would almost certainly be yes!
Even if you're working on your own long term project where actually are the business owner, wouldn't you want to read nice statements summarising the most critical states, actions and their outcomes that actually assert your site or application functions correctly?
The answer again is a yes!
Conclusion
Cucumber is awesome. However I do not believe it should come without a clear instruction manual included and also it does take a long time to set up and work with if you want it to work at scale.
I believe it should be packaged as more of a framework than a tool. Obviously this doesn't mean you cannot use the tools as is if you want to do your own thing but it seems to lack clear guidance for new users.
I quite quickly validated that what I'd heard at the meet-ups were not one-offs and there are issues with Cucumber. Courgette solves those issues and gets us closer to the holy grail of cross-team collaboration and communication.
On top of providing heaps of other tools for reporting, debugging, and efficiency Courgette allows us to use Cucumber to:
- write tests quicker than with Capybara
- create tests that are readable to any layman
- create proper declarative business-readable steps from reusable steps (or the other way around)
As Dan North said on Courgette:
"it comes with batteries included to some extent so you can get up and running quickly"
Check out Courgette Testing at https://courgette-testing.com