Skip to content

Wraith

Visual regression testing with Wraith

I’ve had a play with this before, but new machine, new set-up, this time I’ll document it…

First, what is Visual Regression Testing?

Simply, the visual comparison of two versions of a page, and highlighting any differences. It’s spot the difference for web folks.

It’s important to note this compliments other testing, catching the sort of difference easily missed in QA and by functional testing (be that in in CI or elsewhere). It also still requires someone to actually look and the output. Humans are significantly better at spotting things that are wrong, but they first need to be presented with the possibility that something could be wrong.

Okay, what is Wraith?

Wraith is a tool created by the BBC for automating some aspects visual regression testing.

Given two domains, it can compare pages between those domains. Wraith will highlight differences between pages (for example, a page on your local dev or pre-prod environment and a production environment) and save these as a screenshot for you to manually check. It will also flag unexpected percentage of difference, indicating if you have a potential problem.

Screenshot of Wraith output
Wraith output
Wraith output screenshot
Wraith visual output screenshot

Installing Wraith

  • Install HomeBrew. This is will take all the pain away. HomeBrew is a package manager for macOS and saves you having to mess around with installation paths and version management.
  • Install ImageMagick. This is required by Wraith to generate screenshots for comparison. From a terminal:
brew install imagemagick
  • Intall PhantomJS. This is a headless web browser that will load your site
brew install phantomjs
  • Install Wraith. Wraith itself is a Ruby package, and this assumes you have Ruby installed. Ruby ships with macOS, so we’re good to go:
gem install wraith

Now we have all the bits we need to run Wraith, we need to configure it to run.

You could run wraith setup but I prefer a hand-crafted config. You can save this into a config.yaml folder in your project, or wherever you’re doing CI testing. It should be pretty self-explanatory. Here’s the config for one one my sites:

browser: "phantomjs"
domains:
  current:  "https://shellbryson.com"
  dev:   "http://localhost:3001"
paths:
  home:     /
screen_widths:
  - 320
  - 600x768
  - 768
  - 1024
  - 1280
before_capture:
directory: 'wraith'
fuzz: '20%'
Default: 0
threshold: 5
  template: 'slideshow_template'
  thumb_width:  200
  thumb_height: 200
mode: diffs_first

On my particular setup, I have a dev copy running on port 3001 of my local Mac. This could be set to wherever your test environments live (CI environment, for example).

Running Wraith

As we installed a global ruby gem for Wraith and global prerequisites via Homebrew, you can run Wraith from anywhere on your system. It doesn’t have to be in your project folder.

Note: You may actually want to avoid committing your Wraith configuration to your project as it could expose testing URLs and other setup that you don’t want public.

  • Run Wraith with our configuration
wraith capture ./configs/capture.yaml

Wraith terminal screenshot

Wraith terminal screenshot

  • Wraith will provide a summary and will generate a series of screenshots. These can be found (in my case) in ./wraith/home/. These contain screenshots of each URL, a thumbnail, and a 3rd file containing the differences between URL.
  • Wraith also generates a gallery of ./wraith/gallery.html

In my setup I ran Wraith from my project folder, so Wraith’s gallery was served along with that on port 3001

Wraith gallery screenshot

References

Wraith documentation