Automated frontend testing with AngularJS, mocha, protractor, SauceLabs & Travis CI

Anyone who has ever tried to get automated frontend testing going will know that it’s not always plain sailing.  We recently put the effort into getting automated frontend tests in place for our latest product offering  and would like to share our experiences with the community. The beauty of the solution that we have employed is that both unit and frontend tests use the same test runner (we have used mocha) and the same CI platform (Travis).  Saucelabs allows us to test on many different browser and operating system combinations.

Here is a video of our tests running locally.

Here is a repository containing the sample code.

Setting up Protractor

First install it:

Then setup a protractor config file (protractor.config.js):

As we are using mocha we will also need to install mocha, as well as our expectation framework of choice:

We now setup our first test (test/test.js):

This is a synthetic test to get us started and let us know if everything has been setup correctly.  To start the test try:

This will probably throw an exception along the lines of ‘Could not find chrome driver at…’ unless you have chrome driver set up somewhere.  To get protractor to download and setup chromedriver and selenium you can execute the following command:

Now your tests should run as expected and output something like:

To make that a little easier we can add a line into the node package.json file under the scripts section.

and now we can use the much shorter  npm test .

Serving Up HTML/AngularJS files

Let’s take a step back before continuing with making our tests somewhat more useful.   In order to test our HTML we need to have something first serve up those files to the browser.  We do this with another node package – express:

Now we create a simple express server:

 static  refers to the directory we will put our angular app into.

The app.use()  sets up a static route so that express will just pass through any file in there (ie. all our html, css, js files) as is.

The app.all()  will handle any file that doesn’t exist and redirect to the index.html file.  We need this to get Angular routes to work – i.e. /members/videos  will go through index.html  and the javascript there will figure out what to show.  We can now run the server by executing node server.js .  Opening a browser to http://localhost:3000/  should show us our angular app.

Setting Up Some Real Tests

We now want to set up some real tests.  To do this we will need a server running so the tests can access our app. The easiest way  to accomplish this is to setup a mocha before()  task to run before everything and an after()  task to clean up and shut the server down.  To do this we created test/00_setup.js :

browser.baseUrl = ... tells protractor where it should be requesting pages.   browser  is a global variable provided by protractor for all basic browser interaction.

You may note that we expect to get passed the server when we require server.js .  This allows to fire up the server on a port of our choosing, in this case any available port (which is what server.listen(0)  means).  To allow this to work we need to modify server.js  by replacing the app.listen() line as follows:

Now we can start creating tests that are a little more useful:

Assuming you have an index page that has a H1  containing ‘Welcome’ this test should succeed.

Why no done()?

Anyone who has written Mocha tests before may have been asking why I haven’t been using the done()  function.   The initial test should probably (in straight Mocha) have looked something like:

The reason we don’t need to do this is because protractor is using the webdriver control flow in order to automagically know when the test has finished and trigger the done()  mechanism for you.   In most cases this isn’t a problem as we are generally executing everything through the webdriver control flow – i.e. we use browser.get()  to request a page be loaded and element(by....)  to find things on the page.  However when we need to execute any non-protractor asynchronous code, this is not going to work.   We won’t go into detail on this topic here but feel free to let us know you’d like to see another article on more advanced topics such as this one.   Other issues we encountered that could merit an follow up posting include clearing session information and testing an app that uses a backend API (a common case for Angular).

Setting Up For Travis (and SauceLabs)

This part took a little bit of troubleshooting.

The first step is to  set up our .travis.yml  file so that Travis knows our setup.   This article covers most of the basics.

The next problem we face is that Travis is headless and hasn’t got selenium installed.  One approach would be to use the   before_script  section of the travis.yml file to install selenium it and set up xvfb.  However, SauceLabs is quicker to set up and provides more options for both the browser and OS used.   This article has the basic information.   There is a bit there where you can specify the sauce_connect  addon from travis however I tried that at one point without success before I moved away from it in an attempt to get everything to work.  It sets up v3 of the sauce connector and they are now up to version 4.3 so I rolled my own script to setup and used that instead (see below).

You now need to add your saucelabs credentials to  travis.yml . You can do so as follows:

This will add your login details as encrypted global env values.

We now create a script to download, and run the latest sauce connector.

Credit: This is a slightly modified version of this gist.

Now we add it to the .travis.yml  file’s before_script :

And finally we need to let protractor know that we want to run things through SauceLabs. This part took  a while to get working in a stable manner.  Travis seem to be having some connection issues with SauceLabs right now so all the tests seemed to stall between the 1st and about the 7th test.  We have been dialoguing with Travis CI on this issue and they are aware of it and working to have it resolved.

Our initial setup was to just use the protractor sauce connector stuff.  I updated protractor.config.js  to look something like:

The above means we still run tests through our own chromedriver, however Travis uses SauceLabs.  To validate the Travis portion we could execute something like this locally:

This assumes you start up the sauce connector with the tunnel identifier set to whatever the job number was (localjob).   This worked perfectly locally machine but, as mentioned earlier, was not reliable on Travis.

The key realisation was that the protractor sauce connector isn’t really using the sauce connector we downloaded (or at least it isn’t using all of it). The sauce connector allows saucelabs to create a tunnel to your system so that they can make localhost requests. The part that wasn’t being used was the side that sends instructions from your machine to SauceLabs to tell them what to do. Those were being sent through http://ondemand.saucelabs.com/ which seems to be where the unreliability comes into play. The solution turns out to be to instead use the http://localhost:4445 proxy which the sauce connector sets up to relay the instructions to SauceLabs. To do that with protractor we need to drop the use of the sauce connector and use their hosted connector instead so the protractor config file now looks more like this:

And now everything (finally) works as we want it.  Do let us know about your experiences in the comments, we’d love to hear from you!