Automating visual regression testing for AjaxSwing with Screenster
If you’ve ever tried to find a QA automation tool that would be just right for visual regression testing, you know that the struggle is real;). Catching layout problems, broken CSS and font issues are virtually impossible with frameworks like Selenium or its alternatives.
Manual visual regression testing is painful enough when you're dealing with rich UIs of any kind. But what if you need to routinely test UI updates for a web deployment platform that converts desktop applications into web apps? AjaxSwing generates the UI on the fly. Consequently, the UI is new every time you process a new application, and — because we’re basically talking about desktop programs — each UI is complex, always. In our experience, ensuring that UI is not broken was a painfully daunting task.
AjaxSwing has been around for 18 years (yep, our first release was in 1999🙂), and the first ten years had been challenging for both our devs and our testers. We’ve gone through several QA automation tools, including Selenium, but none of them proved suitable for our visual regression testing needs.
The main issue with these solutions was that none of them was any good at detecting broken CSS and HTML. Now, if you’ve ever dealt with web UIs, you know that multi-browser CSS breaks whenever you tinker with it — and in 99% of cases, it does so in the most unexpected ways.
Besides, we needed a faster solution that would enable us to create new test suites and update old ones within hours, not days. This led our team to a simple conclusion: instead of waiting for someone to launch the perfect tool for visual regression testing, we should do it ourselves.
Introducing Screenster, a UI-driven tool for visual regression testing It actually took us some time to get down to building Screenster, our own tool for UI testing automation. We’ve started working on it in 2013 and officially launched it on May 1, 2016.
In the summer of 2016, we’ve decided that the platform was good enough to do automated visual regression testing for AjaxSwing. Aside from visual regression testing, we’ve been using Screenster for CSS testing and functional testing of our platform and key customer applications.
How it works: record/playback/verify solution on steroids Unlike code-based solutions offered by Selenium, PhantomJS, and the likes, Screenster uses a smart record/playback/verify solution with tons of added functionality. The tool records actual user actions and takes a snapshot of the UI and its DOM each time a user interacts with it. These screenshots become the Visual Baseline of a UI.
Every time someone introduces changes to the UI, QAs can rerun the test suite and compare new screenshots to the Visual Baseline. The platform will inform its users of any change, be it broken layouts, cropped texts, wrong colors, or anything else. If a particular change is expected, you can approve it to the Baseline. Regions with dynamic content can be ignored during comparison to eliminate false positives.
Thanks to its Smart Comparison algorithm, Screenster will automatically ignore minor differences associated with rendering. On the other hand, it will detect layout issues even if the difference comes down to one pixel.
What we’ve done so far So far, we’ve been running more than 50 automated visual tests in our CI, and we are adding new ones every week. It brought sanity and stability to AjaxSwing where engineers are actually not afraid to tweak CSS or change UI behavior.
In addition to using Screenster for our own needs, we’ve made the platform available to anyone — you can try it for free here. Google, Kaspersky, and HomeAway have already signed up for Screenster, so why don’t you try it?
