I had hit a wall, my test automation was taking over 5 hours to run and I wasn’t sure what I could do to bring that time down. I knew that there were environmental issues that were slowing down my tests. Around two hours in the number of page timeouts jumps up to nearly 50%. WebDriver has never worked well with test environments that were extremely under powered, but there was more to it than that. This was a sudden jump in time of execution and timeouts.

Eventually we found the problem, but after all the research, I wanted to loose my head.

I have always supported Selenium WebDriver for browser testing as a primary form for test automation. Browser testing made the most since as a user and everyone was using WebDriver, so why did I begin to question that approach?

At the time, I had known about headless testing tools like PhantomJS for a few years and I had used it occasionally. In the past, I  always used it with CasperJS and thought that the two made a great team. I used them for light weight testing around the UI and making sure that pages were up. I introduced Front End developers to them so that they could create UI unit tests.

It was during that week or so of research, that I started trying the Capybara driver. I changed my mind, I wanted to go headless! After tried to find the path of least resistance to convert my test scripts from Selenium WebDriver to PhantomJS or Capybara. After my research,  I discovered was that it was not going to be as easy as I would like.

While I am still proud of what my team and I accomplished during that time, I want to share some important lessons that I learned from that experience and other cases throughout my career.

1) Do your research.

There is more than one way to build test automation and more than one tool. Selenium WebDriver is the most common and it works with many languages, but not all equally. For instance, the APIs for each language are a bit different and that is also true between browsers. You will also have to manage your test browsers and drivers carefully in order to maintain your tests. There are now three main drivers that you need in order to do cross browser testing and this will impact how you write your scripts.

The options that I like best for headless browser test are PhantomJS and Capybara, but there are others to choose from. You should look into execution time, learning curve for development (including language), and types of assertions. I fully believe that you should have more than one tool in your QA tool belt. If you will have to ramp up a new team in test automation, look for languages and tools that are easier to use. One complaint I have heard with headless browser testing is that you don’t see the test execute, therefore you don’t see when it fails. While this could be a valid complaint when you are first starting, it is also true when running on a test server. Read my article “Failed Tests Are Better With Screenshots” to learn about capturing screenshots at the time of failure.

Your research on what tools, frameworks, and languages make sense to use for your automation efforts will yield several options. I recommend building several proofs of concepts builds to find which makes the most sense to use. Some tools might work for specialized tests that you have, but not all of them. You can and should use more than one tool for your test automation needs. Consider having regression test scripts run headless so that they execute faster and take less resources.

2) Socialize the change.

No one loves changes to their workflow or routine. Introducing test automation is a change to workflows and norms in your teams and company. This includes the development team. One big mistake that happens when test automation is brought in for the first time, is socializing the impact that it will have on the development team. Front-End developers should be brought in to discuss naming / identifying standards for elements. You might need to work with them on moving towards IDs and data-attributes.

The testing team will need to be made aware of the changes and what it means to them. The first responses are usually fear of job loss or distain of the tools. No one should fear losing their job because of test automation. Test automation is about freeing up resources to test more rather than replace them. Creating Smoke and Regression test automation allows testers to focus on new development and work in sprint, rather than having to carve out time to execute regression and test tickets.  Distain of the tools, is usually the belief that test automation can’t find enough errors to be useful. Test automation will only find what it is programmed to assert, there is no argument there, but well written test cases help you identify what is important. If you haven’t already, you should read the article about test case writing called “The Key to Lasting Testing Success are in the Test Cases” for more details.

3) Be patient.

We all want to see progress. We want to know that we are moving the ball forward and getting something done. When it comes to building out test automation, you need to slow down and instead focus on the long-term benefits. Building test automation to last is not a race, it’s a team effort. You need to get the development team working to implement QA Hooks on elements, building their own unit tests (back-end and front-end), and cleaning up and identifying the test cases to be automated.

Rushing into a mission accomplished phase of test automation will only create more problems than it solves. Poorly written test automation will mean a lot of maintenance. Developing test scripts for pages that rely on XPath for finding elements means you will always need to go back and update your objects. Work slowly, but united as a cross functional team to make long term progress.

Once a rhythm is developed by the team, you will see progress.

4) If everyone drove off a cliff, would you?

As I mentioned earlier, in my example, it is easy to fall into the majority mindset for developing automation. If everyone else is using Selenium WebDriver, that is not a guarantee it will work for your needs. If your test environment is under powered and your application is heavy, Selenium WebDriver is going to have issues. The same is true if your test server is under powered, Selenium and browsers will eat through your CPU.

Look for a solution that will work for your testing needs and your testing environment. Headless testing is light weight on resources and generally faster to execute. That is one reason why I like it for Smoke and Regression test suite. Now, I run only a few test scripts in Chrome as part of regression testing.

Closing Thoughts

These are four helpful tips to improve the success of your test automation implementation. For test automation development, these are ones that people rarely talk about when starting a test automation project.

What are your favorites and suggestions?

If you like this article, give it a “Like” and a share or a comment! Thanks!