What is it that every SDET (Software Development Engineer in Test) should have in their toolkit? What are the essentials?
TLDR: See conclusion.
It is becoming more and more commonplace to see companies only hire developers that have knowledge or interest in testing. It is after all a developer’s role to make sure that the code they’re writing works in the environment it’s being written for (specific platforms, device and browsers) at the most basic level with the acceptance criteria having been satisfied. But what should this involve?
The tooling in software development has become so good for testing that it is no longer acceptable for one to say “I didn’t have time to write any tests”, and it is now easier than ever for developers to loosely cover the test pyramid on the production code they’re writing, without needing to be a dedicated SDET. It just requires a sprinkling of interest and the realisation that you’re a lot more valuable as a developer when you think test-strategy first. Obviously if you’re doing a hackathon or whipping up a small prototype then you probably don’t need a single test. We are talking here about writing maintainable, production-ready code.
Ultimately all of this boils down to long-term maintainability and risk. Nobody wants to work with a house of cards day in, day out, and equally no business wants an unreliable software product. It is part of a developer’s job to educate those less-technical that that prototype you see doesn’t mean it’s a stable product. It’s key to note that one shouldn’t be trying to get to a certain % coverage. After all, getting to 100% test coverage is like having had cement poured over your code. It’s not productive and its only merit is maybe to slow you down and have you think about a problem for longer.
That time is better spent elsewhere dependent on your target market size and the way in which they use your software. For example, if you have a website / app with a large audience that visits only at particular times of day, you’ll want to do load testing. Regardless of the size of the audience, you have a moral duty to make sure you’re not discriminating against certain audiences, by doing some Accessibility (aka A11y) testing. Something that’s incredibly inexpensive to do is to test your UI with Courgette. For example for starters you could just use it to check that your main site/app pages load and display key content, and check that your contact form submits. For now you can just run this from your local machines. It doesn’t necessarily need to be in your CI pipeline from the outset.
None of this is to say that the role of the SDET is dead, but an SDET may not be somebody you require in the beginning phase of a startup say, where you’re rapidly prototyping things. The SDET, or at least a good one, is somebody who:
- prides themselves in trying to prevent bugs from reaching production
- strives to make sure that tests are part of the CI/CD pipeline to enforce good practice, and will take a developer’s tests into more depth in areas of potential risk, including adding regression tests
- endeavours to make sure that their tests only "get in the way" when something legitimately fails because the acceptance criteria is failing or there is an actual bug, rather than because their test is flaky
- enjoys everything test related (Fuzz testing, security testing, load testing, etc)
- helps spread knowledge and gives tips to other developers within the team to help the software build go in the right direction and can help the team with user scenarios
Let’s go through the pyramid now, with some examples of great tools that you can get set up with in minutes, without a massive learning curve (at least to get started), to make yourself more valuable. Where possible, JS (Javascript) tools have been given as examples. Why? Because Courgette is written and extended in JS, front-ends are more commonly architected in JS than any other language; it is a write less / read less language; the save file to see result feedback loop is quicker; the nodeJS ecosystem is huge; and due to its familiarity to more people. This isn’t to say it is better than other languages for performing certain tasks but due to this familiarity, teams can work more cohesively without having to get bogged down learning yet another language. It’s encouraged to learn a number of programming languages to learn new techniques that aren’t possible in others, however it can sometimes become a chore / time sap for an entire team to have to ramp up just to have exposure on a particular testing discipline that has been introduced. Without further ado, the pyramid...
The Pyramid
UI Tests
If you’re looking for something that you can get set up with quickly, something that scales, something that can test on desktop, tablet and mobile (yes natively as well!) and a number of different browsers, and most importantly can spin up a browser and record your steps and write them out in plain English, then you need Courgette. It is THE one stop shop for UI testing. You can integrate it with browserstack and saucelabs and the like to test against other browsers and different mobile devices. It runs test in Parallel to reduce the time it takes to run, produces great reports, a screenshot gallery, and it can run with Appium.
There are a number of proxy injection UI browser testers like TestCafe, Cypress etc but they come with the following problems:
- They very limited to the number of browsers they work with
- Too much Magic. Browsers executed in TestCafe are not aware that they run in test mode. So sometimes automation control can be broken. It’s also quite hard to debug possible issues, as you don’t know how actually a web page is parsed to inject automation scripts.
- No Browser Control. Because TestCafe do not control the browser, you can’t actually automate all users actions. For instance, TestCafe can’t open new tabs or open a new browser window in incognito mode. There can be also some issues running tests on 3rd party servers or inside iframes.
- Simulated Events. Events like click or doubleClick are simulated by JavaScript internally. Inside WebDriver or Puppeteer, where those events are dispatched by a browser, called native events. Native events are closer to real user experience. So in some cases simulated events wouldn’t represent actual user experience, which can lead to false positive results. For instance, a button which can’t be physically clicked by a user, would be clickable inside TestCafe.
- There is less support than Selenium if Chrome for example made breaking changes
A tip is to create a happy path through the application. This isn’t necessarily how a user would use it but guarantees there is a way to get through your application. An example could be a booking system where you have a call center. If there is a bug that’s preventing some people from purchasing, at the very least, the staff can guide the customer over the phone and at least provide a workaround.
Integration Testing
Integration tests involves testing different modules together, for example a checkout module and the stock item module. A simple test could be that once you checkout that you have one less item in stock than before you checked out. They can be done at the API level or via the UI but it is recommended to do at the API level. Going via the UI could lead to false positives as you have more moving parts and also UI tests are expensive as you’re working with more HTTP requests. It is better to spend the time and set up integration testing at the API level which will be run typically before your UI tests so you know where the bug lies and save time in the long run.
If you’re after something similar to Courgette then apickli is the way to go as it has pre-defined step definitions to get you up and running.
If you want to keep everything code-based, these two articles (although somewhat dated) can help out 10 Steps How to Automate your API Testing Effectively and MOCKING A RESTFUL API USING NOCK, SUPERTEST, MOCHA AND CHAI. Ultimately due to the complexity of integration testing APIs, you’re better off creating your own way of working with a collection of mocha, supertest, chai, and the incredibly powerful nock.
Unit Tests
Unit tests are your living, breathing documentation at the function / method level. Whilst your functions should be called what they do, here is your place to put extra information about what it does and show how it is used. These tests are also obviously for asserting that your code works with those cases, that’s the reason for their creation.
Unit tests are not necessarily something an SDET will do much but it’s still good to know what has been covered to reduce the number of bugs.
A favourite tool these days that is widely used is Jest due to being created by Facebook. As painful as it is to say, it has all the features Jasmine has plus a load more, generally focused around people using react, but the tool can be used with any framework and the runner is really really smooth. Alternatively you can use the combination of Mocha, Sinon and Chai which have been around many years and each have plugins.
If you need something really simple for assertions you can use assert which comes with nodeJS, but you may see yourself adding SinonJS for spies and stubs etc later.
A recommendation is to use TDD for pure functions (functions where the same arguments give consistent return values each run) where you’ll write and run the test first that’ll test an empty function, watch it fail, then write your code to satisfy the test to get your assertions to pass. This prevents false positives.
Other areas of testing
Accessibility (aka A11y) testing
Nothing beats doing a manual A11y audit but here are some good automated tools:
Load testing
Fuzz testing
- artillery-plugin-fuzzer plugin - Runnning a quick test with this plugin against your app’s backend can help uncover bugs, security issues and QA problems.
Perf testing
Profilers are definitely a good way to get numbers, however perceived performance is all that matters to the user/client.
- Chrome Devtools
- jslitmus
Layout testing
- Galen - Automated testing of look and feel for your responsive websites
Visual regression testing / Screenshot testing
A/B or split Testing
Typically within the remit of a UX / BA role but thought some tools here might help somebody
Conclusion
To sum up, the following is recommended.
For UI Testing
- Use Courgette
- Create a happy path through the application
For Integration / API Testing
- Use apickli or a combo of nock, supertest, mocha and chai
For Unit Testing
- Use Jest
- Use TDD on pure functions
- Create risk-assessed based unit tests, don’t just blindly go for coverage
See Other areas of testing above for the rest.