How we made testing 30 times faster without Selenium

When new sites and apps are developed, the biggest headache tends to be in testing in-page transitions, pop-ups, sliders, login windows and so on. A small update might break your page in one or a few browsers and it’s difficult to notice! Right now the easiest way to test behaviors automatically is basically a test automation framework like Selenium or Watir, which requires you to have programming skills. However, people who want to check that “the developer did not brake anything on my site again” often don’t have the necessary skills. Hence bugs make it live. Even recording things with Selenium IDE is not that easy, not to mention the complexity of running recorded scripts later.

90% of test automation projects fail

90% of test automation projects fail

According to Dorothy Graham’s and Mark Fewster’s book “Software Test Automation”, 90% of test automation projects fail because it is a huge effort to maintain the scripts. Test automation takes two efforts – the cost of creating the test and the cost of maintaining the test. Most people invest the time to create the test, but the maintenance often falls behind and most automation scripts become obsolete.

How to lower the cost of creating scripts?

We know that using Recorder needs to be simple. Start recording, do your thing, send to test. Repeat, after bugs are fixed. We think we have gotten pretty close – any product owner should be able now to record and run a test with no preparation. They don’t even have to know how to spell “Selenium”. We designed it to be simple. Recording a simple web checkout process and sending it for a cross-browser check takes less than a minute. Compare that to writing an automation script that has to run in multiple browsers… – we are talking usually hours of work.

A capture-playback tool is nothing new, so we added one more thing – a difference detection algorithm that we already offer for static page testing. The combination of these two will give a potentially very powerful tool.

If you lower the test creation time to the same level of manually running the test, you get a major win. An automation rule of thumb is that you should be able to run an automated test without changing the test code for at least 8-10 times before it pays off. You get a 8-fold speedup from there. If the test creation time is that low then you have very low change cost as well! And that’s the number one reason why automated tests stop running.

The second speedup is running it in multiple browsers simultaneously with the help of the cloud. That will give you a speed up of N, where N is the number of browsers used.

The third speedup is the comparison algorithm. It speeds up the comparison of results by a factor of 30x without sacrificing quality, according to trials.

So let’s summarize the speedup: test creation – 8x, multiple browsers – about 4x, comparison algorithm – 30x. Compared to many other automation tools, this is a significant speedup across the board of test automation activities. The bonus here is that even non-coding people can test!

However, we underestimated by large how hard it actually is to create a test tool. All test tools are known to be buggy, right? Three reasons:

  1. Websites under test are buggy. That’s why you need to test them!
  2. Browsers are buggy. That’s why you need to test your site in many browsers.
  3. Selenium – which we are using to drive these tests – has it’s own quirks.

Combine all three together and you have a challenge; We decided to take it, and the result is here. A very user-friendly way of testing your web app for cross browser testing purposes.

Browserbite Recorder is going to Public Beta

After months in private beta and thanks to more than a hundred trial users who provided valuable feedback, we have decided to release it to the public. We are currently publicly supporting Chrome, Firefox, IE10 and IE9 for now, but we will add browsers as we learn how to tame them.

This is it – we aim to make test automation accessible to everyone!

Try Browserbite Recorder extension for yourself!

Posted by

Comments are closed.