Q1 2021 JavaScript End-to-end Testing frameworks report
Intro
JavaScript developers are blessed - they have multiple different tools for End-to-end testing, they have many options to choose from.
Choosing is always betting. How to bet best? How can one know which tools are popular and trendy? Which tools are actively maintained?
I wanted to answer such questions and prepared a report covering Q1 2021 period. The report uses data aggregated from many different authoritive data sources, like ThoughtWorks and Snyk.
If interested, I recommend checking out other reports as well: Frontend Frameworks, State Management Libraries, Testing Frameworks, Build Tools and Module Bundlers, and Static Sites Generators (JAMStack).
Data Sources
Data for the report came from:
- NPM - libraries downloads, releases, types, and dependencies.
- Google - search interest data.
- GitHub - stars, commits, contributors, license, and age.
- State of JavaScript Survey - libraries usage amongst developers.
- ThoughtWorks's Technology Radar data.
- Snyk - security information.
Metrics
- Stars - the total number of GitHub stars as of April 1, 2021.
- New Stars - a number of new stars in Q1 2021.
- New Stars, % - (100 * New Stars) / (Stars - New Stars)
- NPM Downloads monthly - an average monthly npm downloads number in Q1 2021.
- NPM Downloads monthly % (incr.) - an increase of the average monthly npm downloads number compared to Q4 2020.
- Search Interest, % - Google search interest average number compared to other libraries in the 01.10.2020-31.03.2021 period.
- Developer Usage, % - percentage of developers using a library according to the latest StateOfJS 2020 survey.
- ThoughtWorks Tech Radar - a "ring" where a library was put. ThoughtWorks distinguishes four rings - "Adopt", "Trial", "Assess", and "Hold". You can learn more about the Radar here.
- NPM Releases - a number of NPM releases in Q1 2021.
- Commits - a number of commits in Q1 2021.
- Contributors - a number of contributors in Q1 2021.
- Dependencies - a number of NPM packages a library has in its dependencies list.
- Types - the way typings are provided, either bundled with the package (
BUNDLED) or published to the @types organization on npm (SEPARATE). - Security - calculated by Snyk a level of security (from
AtoF) of Npm packages based on the number of vulnerabilities and their severity.A- no vulnerabilities,F- the least secure level. - Age
- License
Report
Hint: scroll horizontally to see values for all the libraries.
Each library has two links - Moiva and GitHub links. Use Moiva link to see more data about a particular library, to see historical data and trends.
Criteria | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
Stars | 70K | 30K | 23K | 21K | 6.6K | 8.9K | 11K | 2.7K | 19K | 7.3K | 3K | 5.5K |
New Stars | 2.1K | 2.9K | 3.2K | 1.1K | 351 | 186 | 151 | 76 | 139 | 6 | 13 | 43 |
New Stars, % | 3.1% | 11% | 16% | 5.6% | 5.7% | 2.2% | 1.4% | 2.9% | 0.7% | 0.1% | 0.4% | 0.8% |
Monthly | 8.9M | 8.4M | 806K | 8M | 3.4M | 964K | 833K | 39K | 50K | 52K | 7.3K | 48K |
Monthly % (incr.) | 21% | 23% | 56% | 7% | 7% | 3% | 4% | 39% | -7% | 31% | 0% | 23% |
Search Interest, % | 2% | 5% | - | - | - | - | - | - | - | - | - | - |
Developer Usage, % | 33.5% | 35.1% | 3.3% | - | 9.9% | - | - | - | - | - | - | - |
Tech Radar | TRIAL | ADOPT | - | - | - | - | - | - | - | - | - | - |
Releases | 8 | 8 | 6 | 0 | 20 | 3 | 2 | 7 | 0 | 0 | 0 | 0 |
Commits | 86 | 457 | 702 | 591 | 270 | 102 | 21 | 94 | 0 | 0 | 0 | 0 |
Contributors | 25 | 32 | 24 | 12 | 28 | 15 | 4 | 8 | 0 | 0 | 0 | 0 |
Dependencies | 12 | 39 | 13 | 4 | 27 | 87 | 19 | 12 | 14 | 0 | 0 | 12 |
Types | BUNDLED | BUNDLED | - | SEPARATE | BUNDLED | BUNDLED | SEPARATE | BUNDLED | SEPARATE | SEPARATE | SEPARATE | - |
Security | ||||||||||||
Age | 4 years | 6 years | 1 year | 8 years | 10 years | 6 years | 9 years | 3 years | 7 years | 10 years | 8 years | 10 years |
License | Apache-2.0 | MIT | Apache-2.0 | Apache-2.0 | MIT | MIT | MIT | MIT | MIT | MPL-2.0 | MIT |
Popularity: GitHub Stars
Maintained by the Chrome DevTools team Puppeteer is the most starred project in this report. It has 70K stars overall.
Cypress with 30K stars goes second.
The new on the scene Playwright, which is maintained by Microsoft and is just a year-old project, goes third and possesses 23K stars.
Playwright and Cypress gained the most number of stars in Q1 2021 - 3.2K and 2.9K stars respectively. 
When looking at a relative growth, Playwright becomes a leader - it grew by 16% in Q1 2021. Cypress goes second with 11% of growth. 
Popularity: NPM Downloads
Surprisingly, when it comes to monthly downloads numbers, there is no much difference among the first three leaders.
Puppeteer, Cypress and Selenium had on average 8-9M monthly downloads in Q1 2021. 
Playwright's monthly downloads increased by 56% (till 806K), which is the biggest increase among other tools.
Taiko and Casperjs also had a significant relative increase in npm downloads numbers - 39% and 31% respectively. But their absolute downloads numbers are much smaller - 39K and 59K (compared to Playwright's 806K).
Nightmare's downloads even decreased, most likely because the project seems to be abondoned and developers started migrating to other solutions. 
Development Activity: commits, releases, and contributors
Most of the tools are in active development and had on average at least one commit per day.
Playwright, the leader here, got 702 commits which amounts to ~8 commits per day on average. 
Webdriverio released 20 versions which is significantly more than any other project. 
Cypress, Webdriverio, Puppeteer, and Playwright had 24-32 contributors which are very good numbers for any open-source project. 