The first goal in our R&D process was to develop a web page screenshot comparison technique. We started with a simple pixel-by-pixel diff comparison, but after some time we had developed a cross-browser testing algorithm that consisted of complex image segmentation and comparison procedures (described in previous blog posts). (more…)
In cross browser testing, comparison based on the Document Object Model (DOM) has so far been more common than image based comparison. The following post will give a short explanation on why Browserbite uses the latter. (more…)
When new sites and apps are developed, the biggest headache tends to be in testing in-page transitions, pop-ups, sliders, login windows and so on. A small update might break your page in one or a few browsers and it’s difficult to notice! Right now the easiest way to test behaviors automatically is basically a test automation framework like Selenium or Watir, which requires you to have programming skills. However, people who want to check that “the developer did not brake anything on my site again” often don’t have the necessary skills. Hence bugs make it live. Even recording things with Selenium IDE is not that easy, not to mention the complexity of running recorded scripts later.
90% of test automation projects fail
According to Dorothy Graham’s and Mark Fewster’s book “Software Test Automation”, 90% of test automation projects fail because it is a huge effort to maintain the scripts. Test automation takes two efforts – the cost of creating the test and the cost of maintaining the test. Most people invest the time to create the test, but the maintenance often falls behind and most automation scripts become obsolete.
How to lower the cost of creating scripts?
We know that using Recorder needs to be simple. Start recording, do your thing, send to test. Repeat, after bugs are fixed. We think we have gotten pretty close – any product owner should be able now to record and run a test with no preparation. They don’t even have to know how to spell “Selenium”. We designed it to be simple. Recording a simple web checkout process and sending it for a cross-browser check takes less than a minute. Compare that to writing an automation script that has to run in multiple browsers… – we are talking usually hours of work.
A capture-playback tool is nothing new, so we added one more thing – a difference detection algorithm that we already offer for static page testing. The combination of these two will give a potentially very powerful tool.
If you lower the test creation time to the same level of manually running the test, you get a major win. An automation rule of thumb is that you should be able to run an automated test without changing the test code for at least 8-10 times before it pays off. You get a 8-fold speedup from there. If the test creation time is that low then you have very low change cost as well! And that’s the number one reason why automated tests stop running.
The second speedup is running it in multiple browsers simultaneously with the help of the cloud. That will give you a speed up of N, where N is the number of browsers used.
The third speedup is the comparison algorithm. It speeds up the comparison of results by a factor of 30x without sacrificing quality, according to trials.
So let’s summarize the speedup: test creation – 8x, multiple browsers – about 4x, comparison algorithm – 30x. Compared to many other automation tools, this is a significant speedup across the board of test automation activities. The bonus here is that even non-coding people can test!
However, we underestimated by large how hard it actually is to create a test tool. All test tools are known to be buggy, right? Three reasons:
Websites under test are buggy. That’s why you need to test them!
Browsers are buggy. That’s why you need to test your site in many browsers.
Selenium – which we are using to drive these tests – has it’s own quirks.
Combine all three together and you have a challenge; We decided to take it, and the result is here. A very user-friendly way of testing your web app for cross browser testing purposes.
Browserbite Recorder is going to Public Beta
After months in private beta and thanks to more than a hundred trial users who provided valuable feedback, we have decided to release it to the public. We are currently publicly supporting Chrome, Firefox, IE10 and IE9 for now, but we will add browsers as we learn how to tame them.
This is it – we aim to make test automation accessible to everyone!
You probably know that feeling when your designer ships a Photoshop design. It gets sliced, coded and finally applied to your web application or landing page. Wouldn’t it be nice to compare what the designer intended first to what got done? Well, we did it. As far as we know it’s the first service in the world to do it automatically and intelligently not pixel-by-pixel. You can call it a “design test” like the customer that requested this specific feature, does.
To use the new feature log on and under “Change baseline and other options” you’ll find a “Upload file” button. Please convert your image to PNG or JPG first – PSD-s tend to be a bit heavy!
As usual, test is against 1024×768 pixel resolution – so look that your design is aimed for that as well. Otherwise our service works as usual! Feedback welcome, as usual!
P.S. We have already received a recommendation that we should enable uploading for multiple addresses as well. It’s in the backlog! Vote it up in our Uservoice forum, if you want it.
The good old days of the Browser wars when Microsoft was battling Netscape come to mind. Looking back – all the battles in browser wars have had one thing in common. Usually there was a dominating browser, Netscape, Internet Explorer. The underdog had to come up with either new features or make use of the market dominance in other areas. When Netscape was bashed – Microsoft made use of the Windows dominance when it bundled IE together with Windows operating system. Later, when Firefox gained popularity – it was due to speed and novel features compared to Internet Explorer at the time.
What’s different now?
The playground now seems to be far more even than before and the war seems to be more about who controls the channels rather than the user. Most new browsers are in fact, indifferent from user perspective. Google, Apple, Amazon and Microsoft are struggling to gather as much data on what their users are doing. Pushing out different devices is part of the game. Looking from the browser perspective – there are many contenders and no-one has a particular edge. Today’s market share is dominated by IE in the desktop, but Google when mobile and desktop market shares are combined.
The browsers are an important part in this game since most devices come with a pre-installed browser. So, the the browser actually will not matter – it’s part of the operating system. Ironic, considering that Microsoft once got huge fines fro bundling IE with Windows.
Who will win this round? My personal opinion which is based on gazing at the crystal ball is the following:
Google is going to be the new Microsoft for the next couple of years. They are really committed and Android is getting to be more and more dominant.
Microsoft will regain some market share. Their new browsers are just as snappy compared to competition and they’ve got a few things right in the mobile landscape.
Apple will start losing market share but keep some mobile browser domination out of inertia. I hope I am wrong, but we have failed to see the Apple’s disruptiveness during the last few years.
Mozilla will fade out, since they still haven’t got a sizeable platform nor partners to play on. I do not believe that FirefoxOS is going to change much during the next couple of years.
The newcomers will make some small ripples, but that’s the outlook for the coming couple of years. But the pull is on and the big fight is going to be between Google and the rest. And Google is clearly showing that it does not want to play with the community, but take the lead in developing rendering engines. We have seen this before – dominant player in the market trying to create a standard on the base of its own product. In short – there will be loads of browsers on the market, each supporting a different set of features in the web.
In short: It looks like Google will win this battle. But I believe it will not bring any relief to the developers nor users.
The developer perspective
Right now, the pragmatic way out seems to be the following:
Use standard libraries like jQuery and Bootstrap. Their developers are accounting for cross browser problems and do cross browser testing on their own.
Test your website after every change – cross browser testing will not go anywhere.
Test your website after every new browser revision comes out – new browser versions do break websites contrary to the common belief.
We updated to the new Rails version following the warnings about the vulnerabilities that had been patched in the new Rails version. Fearing the hackers we performed the updates as well. One can never be too safe.
The result: A new security issue!
One of our private beta testers on the Browserbite Recorder cross browser testing service messaged me privately a day later saying that the search feature is giving results “that should not be there”. I managed to reproduce the issue within seconds. Indeed – just performing an empty search in the requests showed all the requests from all the users. Luckily the breach did not reveal anything that was not public yet.
Financial Times reporters dropped by our office and did an interview. Along with the President of Estonia and some government officials. We’re glad to give a supporting hand back to the country that supports us!
We update our application regularty, but I figure that this release will be worth a separate post since we’ve incorporated quite a few new things that our users have requested. Browserbite is committed to be the simplest to use cross browser testing service available.
Browserbite Recorder beta
We have announced the support for flow testing in a previous post with a video. In short, we have married layout bug detection engine with functional testing. We’re still accepting participants to our beta program. Send an e-mail to firstname.lastname@example.org if you want to participate!
Sharing is now even easier
We changed around the sharing system to make it more straightforward to share cross browser testing results with your colleagues. Now you don’t need to mess around with e-mail programs. Simply throw in the e-mail of your friend and a quick comment. Then it’s shared!
New keyboard shortcuts
You can use up-down and left-right arrows to move around the resultset with ease. Thanks to the new Recorder you can stroll through the steps of your test while keeping the browser that interests you opened in full mode. Just try the arrow keys while browsing the results and you’ll get it!
What’s in the roadmap?
We will be adding more browsers to the Recorder support and roll out Android that has been cooking in the labs for quite a while. After that, behold the responsive tests!