Archive for the ‘Blog’ Category

How we made testing 30 times faster without Selenium

Posted by

When new sites and apps are developed, the biggest headache tends to be in testing in-page transitions, pop-ups, sliders, login windows and so on. A small update might break your page in one or a few browsers and it’s difficult to notice! Right now the easiest way to test behaviors automatically is basically a test automation framework like Selenium or Watir, which requires you to have programming skills. However, people who want to check that “the developer did not brake anything on my site again” often don’t have the necessary skills. Hence bugs make it live. Even recording things with Selenium IDE is not that easy, not to mention the complexity of running recorded scripts later.

90% of test automation projects fail

90% of test automation projects fail

According to Dorothy Graham’s and Mark Fewster’s book “Software Test Automation”, 90% of test automation projects fail because it is a huge effort to maintain the scripts. Test automation takes two efforts – the cost of creating the test and the cost of maintaining the test. Most people invest the time to create the test, but the maintenance often falls behind and most automation scripts become obsolete.

How to lower the cost of creating scripts?

We know that using Recorder needs to be simple. Start recording, do your thing, send to test. Repeat, after bugs are fixed. We think we have gotten pretty close – any product owner should be able now to record and run a test with no preparation. They don’t even have to know how to spell “Selenium”. We designed it to be simple. Recording a simple web checkout process and sending it for a cross-browser check takes less than a minute. Compare that to writing an automation script that has to run in multiple browsers… – we are talking usually hours of work.

A capture-playback tool is nothing new, so we added one more thing – a difference detection algorithm that we already offer for static page testing. The combination of these two will give a potentially very powerful tool.

If you lower the test creation time to the same level of manually running the test, you get a major win. An automation rule of thumb is that you should be able to run an automated test without changing the test code for at least 8-10 times before it pays off. You get a 8-fold speedup from there. If the test creation time is that low then you have very low change cost as well! And that’s the number one reason why automated tests stop running.

The second speedup is running it in multiple browsers simultaneously with the help of the cloud. That will give you a speed up of N, where N is the number of browsers used.

The third speedup is the comparison algorithm. It speeds up the comparison of results by a factor of 30x without sacrificing quality, according to trials.

So let’s summarize the speedup: test creation – 8x, multiple browsers – about 4x, comparison algorithm – 30x. Compared to many other automation tools, this is a significant speedup across the board of test automation activities. The bonus here is that even non-coding people can test!

However, we underestimated by large how hard it actually is to create a test tool. All test tools are known to be buggy, right? Three reasons:

  1. Websites under test are buggy. That’s why you need to test them!
  2. Browsers are buggy. That’s why you need to test your site in many browsers.
  3. Selenium – which we are using to drive these tests – has it’s own quirks.

Combine all three together and you have a challenge; We decided to take it, and the result is here. A very user-friendly way of testing your web app for cross browser testing purposes.

Browserbite Recorder is going to Public Beta

After months in private beta and thanks to more than a hundred trial users who provided valuable feedback, we have decided to release it to the public. We are currently publicly supporting Chrome, Firefox, IE10 and IE9 for now, but we will add browsers as we learn how to tame them.

This is it – we aim to make test automation accessible to everyone!

Try Browserbite Recorder extension for yourself!

Browserbite at IEEE Conference

Posted by

New scientific development is to be presented at 29th IEEE International Conference on Software Maintenance in Eindhoven, The Netherlands. Tõnis Saar will do a presentation under Industry track on Dynamic Analysis. The paper is called “Browserbite: Accurate Cross-Browser Compatibility Testing via Machine Learning Over Image Features“. The paper and corresponding has been developed in Software Technology and Applications Competence Centre where Browserbite is a partner.

The paper describes methods that can be used to further enhance issue detection methods in cross browser testing that is being used in current Browserbite technology using machine learning.

You’re welcome to contact Tõnis at the conference and discuss the challenges in cross browser testing. The benefits of this work will soon be incorporated into our service.

Test Photoshop design vs real browsers automatically

Posted by

You probably know that feeling when your designer ships a Photoshop design. It gets sliced, coded and finally applied to your web application or landing page. Wouldn’t it be nice to compare what the designer intended first to what got done? Well, we did it. As far as we know it’s the first service in the world to do it automatically and intelligently not pixel-by-pixel. You can call it a “design test” like the customer that requested this specific feature, does.

To use the new feature log on and under “Change baseline and other options” you’ll find a “Upload file” button. Please convert your image to PNG or JPG first – PSD-s tend to be a bit heavy!

As usual, test is against 1024×768 pixel resolution – so look that your design is aimed for that as well. Otherwise our service works as usual! Feedback welcome, as usual!

Upload baseline

P.S. We have already received a recommendation that we should enable uploading for multiple addresses as well. It’s in the backlog! Vote it up in our Uservoice forum, if you want it.

Browser wars are here to stay – who will win this battle?

Posted by

Last month has brought in new waves that will affect the future of browsers.

First, Opera announced dumping their original engine in favour of Webkit. Then we heard from Google about forking off from Webkit and calling the new baby Blink. And And there was an interestgin piece where Samsung and Mozilla decided to team up. To stir up the sauce even further – Opera announced that they will actually use Blink in the future. We think that these news mean that the new battle in Browser Wars will bring important changes to cross browser testing in general.

The Browser Wars revisited

The good old days of the Browser wars when Microsoft was battling Netscape come to mind. Looking back – all the battles in browser wars have had one thing in common. Usually there was a dominating browser, Netscape, Internet Explorer. The underdog had to come up with either new features or make use of the market dominance in other areas. When Netscape was bashed – Microsoft made use of the Windows dominance when it bundled IE together with Windows operating system. Later, when Firefox gained popularity – it was due to speed and novel features compared to Internet Explorer at the time.

What’s different now?

The playground now seems to be far more even than before and the war seems to be more about who controls the channels rather than the user. Most new browsers are in fact, indifferent from user perspective. Google, Apple, Amazon and Microsoft are struggling to gather as much data on what their users are doing. Pushing out different devices is part of the game. Looking from the browser perspective – there are many contenders and no-one has a particular edge. Today’s market share is dominated by IE in the desktop, but Google when mobile and desktop market shares are combined.

The browsers are an important part in this game since most devices come with a pre-installed browser. So, the the browser actually will not matter – it’s part of the operating system. Ironic, considering that Microsoft once got huge fines fro bundling IE with Windows.

Who will win this round? My personal opinion which is based on gazing at the crystal ball is the following:

  1. Google is going to be the new Microsoft for the next couple of years. They are really committed and Android is getting to be more and more dominant.
  2. Microsoft will regain some market share. Their new browsers are just as snappy compared to competition and they’ve got a few things right in the mobile landscape.
  3. Apple will start losing market share but keep some mobile browser domination out of inertia. I hope I am wrong, but we have failed to see the Apple’s disruptiveness during the last few years.
  4. Mozilla will fade out, since they still haven’t got a sizeable platform nor partners to play on. I do not believe that FirefoxOS is going to change much during the next couple of years.

The newcomers will make some small ripples, but that’s the outlook for the coming couple of years. But the pull is on and the big fight is going to be between Google and the rest. And Google is clearly showing that it does not want to play with the community, but take the lead in developing rendering engines. We have seen this before – dominant player in the market trying to create a standard on the base of its own product. In short – there will be loads of browsers on the market, each supporting a different set of features in the web.

In short: It looks like Google will win this battle. But I believe it will not bring any relief to the developers nor users.

Cross browser testing

The developer perspective

The real loser in the browser wars has actually been the developer and coincidentally, the user. The wars have introduced a lot of browser-specific prefixes in CSS and behaviours in JavaScript. The lack of a well implemented uniform standard has lead to increased costs in testing and bug fixing. On the other side – there are even bigger costs associated with disgruntled users.

Right now, the pragmatic way out seems to be the following:

  1. Use standard libraries like jQuery and Bootstrap. Their developers are accounting for cross browser problems and do cross browser testing on their own.
  2. Test your website after every change – cross browser testing will not go anywhere.
  3. Test your website after every new browser revision comes out – new browser versions do break websites contrary to the common belief.

Update Rails or not – security issues either way

Posted by

Hackers aheadWe updated to the new Rails version following the warnings about the vulnerabilities that had been patched in the new Rails version. Fearing the hackers we performed the updates as well. One can never be too safe.

The result:  A new security issue!

One of our private beta testers on the Browserbite Recorder cross browser testing service messaged me privately a day later saying that the search feature is giving results “that should not be there”. I managed to reproduce the issue within seconds. Indeed – just performing an empty search in the requests showed all the requests from all the users. Luckily the breach did not reveal anything that was not public yet.

(more…)

New release – Recorder is now in private beta, sharing and keyboard shortcuts

Posted by

New version of cross browser testing tool Browserbite!We update our application regularty, but I figure that this release will be worth a separate post since we’ve incorporated quite a few new things that our users have requested. Browserbite is committed to be the simplest to use cross browser testing service available.

Browserbite Recorder beta

We have announced the support for flow testing in a previous post with a video. In short, we have married layout bug detection engine with functional testing. We’re still accepting participants to our beta program. Send an e-mail to beta@browserbite.com if you want to participate!

Sharing is now even easier

We changed around the sharing system to make it more straightforward to share cross browser testing results with your colleagues. Now you don’t need to mess around with e-mail programs. Simply throw in the e-mail of your friend and a quick comment. Then it’s shared!

New keyboard shortcuts

You can use up-down and left-right arrows to move around the resultset with ease. Thanks to the new Recorder you can stroll through the steps of your test while keeping the browser that interests you opened in full mode. Just try the arrow keys while browsing the results and you’ll get it!

What’s in the roadmap?

We will be adding more browsers to the Recorder support and roll out Android that has been cooking in the labs for quite a while. After that, behold the responsive tests!

Browserbite Recorder private beta

Posted by

Our users have been pushing us towards releasing a tool that would help them mix user interactions with our computer vision difference detection. For now, our tool has only provided capability to test pages that have their own unique URL. Get ready for something different in the video below.  Our goal with this tool is to get out-of-the-way of “normal” testing of new features and give a really fast way to send your actions for cross browser testing. We hope to reduce an hour’s work to mere minutes. The beta will support IE8, IE9, Chrome and Firefox on Windows. We will add more platforms once we are happy with the performance of the “core” browsers.

Most importantly, let us know, what do you think is a fair and simple pricing model for this kind of service? We’re looking for something that works for solo coders and well established businesses alike.

(more…)