Skip to content

Testing Guidelines

Jump to bottom
Ryan Johnson edited this page Aug 29, 2019 · 5 revisions

Prerequisites

Please make sure you install the necessary requirements before continuing.

Quick Start

Set up

git clone [email protected]:rackerlabs/helix-ui.git
yarn install
cd test; yarn install; yarn run webdriver:update; cd -;
yarn start

Acceptance Tests

yarn test

Running individual tests

Regression tests:

cd test/
yarn test:regression

Note: You can add more tests to this suite by adding data-visreg="YOUR-SECTION-NAME-HERE" attributes to the html found in docs/components/**/**/index.html. It'll name your artifacts according to what you placed in the html attribute data-visreg for the DOM snapshots, as well as the images that get saved in test/artificats/regressionScreenshots.

Visreg tests:

cd test/
yarn test:visreg

Note: These tests serve as the "behavior"-centric tests in this project. If you find yourself needing to interact with a component in order to trigger state changes, etc., you should put those tests here.

Creating visreg diffs to share

git checkout -b changing-stuff
# change stuff
git commit -am "style(docs): Refactor lots of less"
# get a personal access token from github
cd test/
yarn visreg

Follow the prompts to include your personal access token, as well as your target branch (master is usually what you want).

This concludes the summary of running tests in helix. For a more in-depth view of the above, continue reading!

Visual Regression

Introduction

This project utilizes visual regression testing, a technique that leverages before/after screenshot comparisons to validate that code is behaving as expected. The best introduction to understanding visual regression testing in helix-ui is to study the previous art used in encore-ui, another component-based UI toolkit. This project's visual regression system was created to address situations where the tech stack of your project doesn't align with the tech stack of the testing tool. This was a serious limitation present in the initial visual regression system written, and was the reason we made the switch to this new tool.

Setting Up

Running yarn install in the root of the helix-ui project will automatically navigate to the test/ directory, and install the dependencies for the testing software. Running yarn start in the root of the helix-ui project will automatically start a selenium server in the background, and will stop running when you terminate the development server. It runs on port number 4444.

See: What if yarn run webdriver fails? Go to the FAQ

DOM Snapshots

Part of ensuring that web components work is examining the state of the DOM in the source code and comparing it to the state of the DOM after it's been rendered in a browser. This is what DOM snapshots aim to test in a streamlined fashion -- they are located in the test/regression/ directory,

What's the difference between ./regression tests and ./visreg tests?

Regression tests are automatically generated. All you need to do is to add an attribute, data-visreg="${UNIQUE_TEST_NAME}" to any demo component's html and the development server will notice these changes, and do two things:

  1. Take a DOM snapshot of the state of the component demo DOM post-render.
  2. Take a picture of the browser screen of this section of html.

The DOM snapshots end up here, and the screenshots end up in test/artifacts/regressionScreenshots.

Visual regression tests are a mixture of "visual" tests (taking screenshots of web pages and/or elements on that page) and "regression" tests (interacting with elements on a page and verifying correct behaviors and/or states).

Why are DOM snapshots tracked by version control, but not the screenshots?

DOM snapshots get rendered in a reliable way by the browser, regardless of your platform. Running the test suite using Firefox on Windows will render the same output as running it using Firefox on Linux or Mac. This is not the case with visual regression screenshots! For starters, Windows renders system fonts differently than the system fonts on Mac or Linux, so anything you take a picture of that includes text will render differently on machines running different operating systems.

This core concept appears again for things like screen resolutions, external displays, and timezones. Visual regression testing, in order to be reliable for team-based use, has to be delegated to a dedicated build server. This may be possible in the future using Saucelabs, but for now it's easier to leave these artifacts untracked, and expose a custom command for generating one-off catalogs when the developer needs them.

Sauce Labs

Introduction

Sauce Labs allows users to run tests in the cloud on more than 700 different browser platform, operating system and device combinations, providing a comprehensive test infrastructure for automated and manual testing of desktop and mobile applications using Selenium, Appium and JavaScript unit testing frameworks.

Benefits of using Sauce Labs

Sauce Labs offers automated testing for Continuous Integration (CI) and Continuous Delivery (CD) pipelines, and provides plugin integrations with the many CI platforms including: Jenkins, Bamboo, Travis CI, Circle CI and TeamCity.

Sauce Connect

It is a proxy server that opens a secure connection between a Sauce Labs virtual machine running your browser tests, and an application or website you want to test that's on your local machine or behind a corporate firewall.

Sauce Connect Tunnel

Tunnel is a queue that runs all our selenium tests, and all traffic from each of the test are run under your account automatically and transparently.

Setting up

  1. Open the terminal and navigate to the root directory of the helix-ui project. Export the Sauce Labs username and access key.

        export SAUCE_USERNAME="your username"
    export SAUCE_ACCESS_KEY="your access key"

  2. Go to the password safe and request permission to access the credentials for Sauce Labs:

    After successful login to password safe, the credentials username and access key are located under the folder helix-ui.

  3. Download Sauce Connect from:

    https://wiki.saucelabs.com/display/DOCS/Sauce+Connect+Proxy

  4. Extract the zip file into your desired location. For example, user/Documents/sc.

  5. From the terminal navigate to the bin folder, from the location where you extracted the zip file. For example, user/Documents/sc/bin.

  6. Execute the binary. For example, for Linux users the command is ./sc. For Windows users execute the sc.exe file.

  7. Optional In order to execute the test scripts in batch mode in our local,

    export CI=true

    This makes the execution of test scripts faster in our local.

Running test scripts using Web Driver in Sauce Labs

  1. Complete the setup and run the following command on your local machine:

    yarn test --browsers=chrome --components=buttons/index

    This will execute your test script wriiten for button component in a chrome browser using Sauce Labs.

  2. In order to view the results of the executed test script executes:

    Log in to the Sauce Labs web interface. We can watch the test script run and the results appear in the dashboard. https://saucelabs.com/

How to view the results of the test script in Sauce Labs?

  1. Execute the following command from the root of the helix-ui project.

    yarn test --browsers=chrome,edge --components=search,buttons

    The above command will test the button and search components in chrome and edge browsers respectively in Sauce Labs.

  2. Log in to Sauce Labs by extracting the username and password from the password safe.

  3. Navigate to the Sauce Labs dashboard. https://saucelabs.com/beta/dashboard/builds

  4. We will see all our results for all the test scripts that were executed in the dashboard. Click on the respective result link to view the result of the test that was executed from the terminal.

    For example, for the above test, the results can be obtained from: https://saucelabs.com/beta/builds/6bc29dbdc2d6400c8b5c4fde9d5c290c

  5. For each component and for each browser there is a separate visual of created by Sauce Labs. For example, for the above test script we will have 4 different links.

  6. Navigate to each of the links and click on play to view the visual of each component rendered in the browser.

How to distinguish passed and failed test results?

In Sauce Labs, if all the test cases has passed, you would see Build Success (Green color) along with the test id on the top of the screen.

If any case has failed, then you would see Build Error (Orange color) along with the test id on the top of the screen.

NOTE: The individual test case that passed or failed are marked by Green tick mark and Orange exclamation mark followed by the test case id.

Unit Testing

  • Test Runner: mocha
  • Assertions: chai
  • Mocks/Spies: sinon + sinon-chai
  • Coverage Reporter: istanbul (via nyc)
    • report format: lcov (browser viewable)
  • DOM APIs: jsdom
    • allows us to test functionality dependent on DOM APIs
  • ESM Compatibility: esm
    • allows use of ES Module import/export syntax in CommonJS environment (i.e., NodeJS)
  • Runner: test/run_unit.js

NPM Scripts

yarn test:unit     # run unit tests
yarn test:coverage # generate coverage report

Notes

Location of Specs

Specs should be defined alongside their target source. For example:

Example spec file for src/helix-ui/utils/index.js

/* src/helix-ui/utils/index.spec.js */
const Utils = require('./index');



describe('utils', function () {
    
    it('it should do stuff', () => {

        expect(Utils).to.do.something;
    });
});

Exclude Spec Files in Coverage Report

We ignore *.spec.js files from coverage details using the nyc.exclude configuration in package.json:


"nyc": {
    
    "exclude": ["src/**/*.spec.js", "test/run_unit.js"] 
  
},


FAQ

What if I make a mistake entering my github username?

Whatever you wrote there gets put into test/bin/visreg.config.ts. Edit that file and you should be set.

What if my github token is wrong or I enter the wrong one?

Whatever you entered for your github PAT gets written to test/.github-token. Delete or edit that to fix your problem.

What if yarn run webdriver fails?

platform check

If your system doesn't respond to:

platform_gecko='linux64'
platform_chrome='linux64'
[ `uname` == 'Darwin' ] && platform_gecko='macos' && platform_chrome='mac64'

then the selenium server artifacts we grab won't run on your system. Windows users: consider adding support for yourself in a pull request. Otherwise, send a message to the maintainers of this repo for help.

geckodriver

How do I update a snapshot?

Delete the current snapshot, and rerun regression tests (yarn test:regression). A new snapshot will be generated. Alternatively, you could also navigate to the test directory and run yarn test:regression -u.

Old Notes

  1. WCT is very much biased toward working with the latest browser versions, and the current latest version also required Sun Java 8. I had trouble getting it to run at all on earlier java versions, and trouble getting Selenium to progress past starting the browser until updating to the latest firefox.
  2. WCT seems to favor having your tests in .html files... which felt a little bit weird to me.
  3. I was not able to get the a11ySuite to work, it just kept saying a11ySuite is undefined.
  4. WCT uses Selenium, but very thinly, it seems that the actual tests run in the context of the browser, possibly through selenium's executeScript call.
  5. Because of the above, referencing elements and testing clicks/other events, etc... happen within the browser scope. Which means you use those APIs, and the mouse events will be synthetic/programmatic.
  6. WCT seems slightly biased toward polymer, but not overly so.

Things I didn't try, but would like to if we have more time:

  1. Testing IE on windows
  2. Testing using the browser exclusively instead of the WCT command line
  3. Testing using scripts that aren't HTML scripts

Overall, I think this could be a viable replacement for Protractor in the Web Components project.

Clone this wiki locally