Testing Functionality, Responsiveness and Rendering on Geico’s Responsive Website
Responsive web development requires detailed attention when incorporating testing practices; the same (hopefully) code is intended to render on many screen sizes and operating systems/versions. In addition, HTML5 functionality may assume access to local sensors to drive a richer and streamlined experience, for example, automatic location detection, further complicating the responsive web test strategy. The testing matrix becomes sizable, while the time-to-test shrinks from increasing competitive pressure and the transition to agile.
In this article I’ll show how to conduct multi-platform parallel testing when focusing on responsive web development, using Geico’s responsive website as an example. In addition to a standard functional test, I will add the ability to validate the responsiveness of the site, as well as leverage Applitools‘ Selenium APIs to ensure proper rendering across different screen sizes. There’s a code sample to complement this article: here.
- In Eclipse, create a new project based on the Perfecto Selenium template (ensure you have the latest plugin)
- Convert this project to Maven, then TestNG, and add TestNG class. Make sure to generate BeforeTest, AfterTest and BeforeSuite functions. Also make sure to select tests to run in parallel when configuring TestNG.
- Add one or more @tests to your java code. In my project it’s in GeicoRWDTest.java. For now focus on the functional aspect of things.
- Fun step #1, platform selection: fill TestNG.XML with the platforms you want to test on. I recommend using device characteristics (OS, version etc.) and avoid device IDs. You can add as many as you want, including browsers, as long as your licenses allow. If you don’t know which devices/browsers to test on, use our digital test coverage index. Platform coverage is key when testing responsive websites.
- At this point your POM.xml is missing a few parts. Best to compare it with the one in the code sample.
- Last piece before the first run: add environment variables to your test
Insert environment variables into TestNG
- Right click from your TestNG.xml file ->Run as ->Run configurations and add the following:
- PERFECTO_CLOUD: your cloud, ex.: mycloud.perfectomobile.com
- PERFECTO_CLOUD_USERNAME: your cloud username, ex.: [email protected]
- PERFECTO_CLOUD_PASSWORD: your cloud password
- (optional, but you’ll want it later) APPLITOOLS_API_KEY. If you don’t have one sign up for free at Applitools.com and you will easily get it. If you don’t want to add it, no worries, the project will work anyway.
- OK, time for first run! right click your TestNG.xml file, select Run as -> TestNG suite. Make sure to turn on Perfecto’s dashboard in the views so you can see the devices/browsers running. Notice in the console on test completion you will get a link to Perfecto’s new report (in my code I highlighted it with “########### ========>>>>>>>> Report URL:”)
- Adding responsiveness measurement to the test: In case you didn’t know, Perfecto offers OCR and image analysis technology to validate correct rendering of specific elements on the page (beyond native objects). You can also measure the time from when you clicked on the previous page until the new one is rendered. The magic is done in PerfectoUtils.Java, specifically in ocrTestCheck and getUXTimer functions. For convenience I put them together into one function, where
- Driver: the remote web driver
- Text: the text you’re looking for. Note don’t use text that appeared on the previous page 😉
- Threshold: percent confidence, 0-100, I usually use 99
- Timeout in seconds
So a complete section would look like:
OCR validate and get UX timer
responsiveness measurement of Geico pages on Android (in milliseconds)
- Adding rendering validation to the test: This is the real fun.
- Let’s start by introducing ApplitToolsHelper.java
- Static “batch” variable to help you collect your tests into a single batch. Note that the batch can be driven from Jenkins or created new from inside the test:
SetBatch from Jenkins or locally
- Local eyes variable, this will help us do the rendering validation. Note: batch is global across all eyes instances.
- The constructor validates you have an API key (remember APPLITOOLS_API_KEY from section 6?), and if so, it will
- Instantiate eyes (for each thread)
- Assign it to the batch
- “Open” the eyes to the driver
- “Close” eyes essentially closes the eyes. What you should know is that eyes are designed to fail the test if Applitools finds material differences in the images. You can decide if that’s what you want to do or just get the Applitools report URL
- Lastly, checkWindow validates the window you’re using. Best to add a tag in there. Applitools correlates images to the baseline by a number of assets, including the tag (“Geico home”), OS, version, screen size etc.
- Main code changes (GeicoRWDTest.java)
- In the “BeforeSuite” you want to initialize the batch so it’s grouping all the eyes instances
- In “BeforeTest” instantiate ApplitoolsHelper
Create ApplitoolsHelper instance before the test is run
Setting the batch before the test and closing the driver at the end
- Hopefully everything went well, you should be able to add applitoolsHelper.checkWindow(tag) everywhere you want an image to be taken
Add visual checkpoints to your script
- In “AfterTest” you want to get the Perfecto report URL, the Applitools report URL and close the eyes 😉
Don’t want to work so hard?
- Clone https://github.com/AmirAtPerfecto/GeicoRWD
- Refresh the project so the POM applies
- Enter environment variables (Step 6 above)
What did I learn?
- Adding render validation and responsiveness to a functional test is real easy. It doesn’t add much time to parallel execution of a test. Totally worth doing.
- I was struggling to select which platforms to use. Perfecto digital index really helps.
- Timing between iOS, Android and Chrome are really different. Worth watching and tracking.
- Well, we’re about to add HAR file creation to mobile. While on browsers it’s easy to generate those, less so on devices. Imagine you can compare HAR files on mobile and desktop browsers and fine tune what’s really bogging down your site load.
- You can also get page load time from Selenium logType.PERFORMANCE. read more here.
- Generally speaking getting stuff from the Chrome developer tab, such as network, timeline, security analysis etc. trust me, lots of interesting stuff there!
Chrome developer toolbar for Geico’s website
- Lastly (on my wish list), Perfecto recently introduced Wave toolbar in Chrome browsers. It’d be great to get the accessibility report straight into the test.
Wave toolbar accessibility analysis of Geico’s responsive website