Mistakes – Being Short Sighted?

I’ve been involved in test automation for over ten years now. During that time, I’ve learnt so much from just doing the job, working with some very clever people, and the wealth of knowledge on the internet. But where I’ve learnt most has been from making mistakes, sometimes big mistakes.

And to help you avoid making these same mistakes, I’m going to write regular articles like this, detailing the mistakes I made, and the lesson that was learnt. I do think it’s important that everyone makes their own mistakes, especially when it comes to coding and testing, but reading of others experiences and other peoples mistakes can sometimes be just as useful.

This mistake came from writing my very first framework on my own. It came at a time when I was leaving a job working in a big team where my job was more to maintain the existing framework, and add to the growing test suite. This new role was only my second in software testing, and meant I was working in a one man QA team, so I alone was responsible for creating the framework and getting some tests running.

I knew from working on previous frameworks what was required in terms of designing it and the kind of functionality I’d need, and I also had in my head exactly what it is I wanted to test, and how I wanted to test it.

So straight away I got to work creating my first framework. It was written in C# and was to use Selenium to test the companies customer facing portal. There was nothing complex about the portal, it was already built on well established code that was unlikely to change any time soon, so it meant I didn’t have to worry about UI changes.

At this point, I didn’t really understand much about design patterns or the benefits to using one. As far as I was concerned, my website journeys were already set in stone and unlikely to change, so I’d just code for those journeys. Initially my framework started off quite well, I had my ‘Framework’ project which housed my Driver setup and initialisation code, as well as a few basic utility methods for basic validation on the URL and checking the page has loaded. But that’s about as good as it got.

My ‘Portal’ project that contained all my action methods was truly the epitome of poor design. I had methods that dealt with multiple pages, all in one class. My methods dealt with entire user journeys and were extremely inflexible, and I was declaring and initialising my elements inside of these methods, which meant pages that were using the same elements were being initialised multiple times. Some of these journeys were duplicated rather than writing methods that could handle multiple paths.

Unsurprisingly, in very little time I had created methods to pretty much cover any user journey we were testing manually. I quickly began writing tests, which by the way, also contained WebDriver code as well as my Portal code, so my tests were a mix of both which made for some unnecessarily long tests. However, being young and naive, I was completely oblivious to all this, and began running these tests regularly against our daily builds. Initially, we had very consistent test runs with a 100% pass rate.

I was happy, the dev manager was happy and the project manager was happy. We were getting regular and reliable results. I was feeling pretty chuffed with myself, and already thinking about asking for a pay rise to reward me for my extremely quick turnaround of a fantastic test automation solution.

Then, it all went wrong. Very wrong. We began to implement a shorter sprint cycle, adding in smaller features that would get turned around much more quickly using my test automation to check for issues. Suddenly my green dashboard started looking very red. I was fixing tests, but they were failing faster than I could fix them. Suddenly, all those happy stakeholders were becoming very unhappy, and demanding to know why they weren’t getting any value from the automation tests.

The problem all stemmed from being so short sighted. I was so fixated on getting a solution together and tests running as soon as possible that I neglected the idea that even the most established UI is susceptible to change. So as soon as we began adding small and frequent changes to the UI and the user journeys, my poorly designed classes just couldn’t cope.

They were the complete opposite of being modular. Tests had to be written to accommodate this which meant, any issues that needed to be fixed in the helper classes completely broke the tests. Having multiple methods that dealt with the navigation across the same pages meant that one change to any of those pages meant fixes had to be done in multiple places.

The other issue was the fact my helper classes weren’t designed to deal with every element on a page, so often I’d be compensating for this by having element interaction in my tests themselves as opposed to keeping it wrapped in helper methods. This also meant my tests were extremely fragile, and if element locators changed, my tests not only had to be fixed at a helper level, they also had to be fixed at a test level. When I was quickly approaching a hundred tests, this was very bad.

Soon, the maintenance cost of fixing my short sightedness grew too much, and the return on investment of these tests just fell so low that I had to virtually scrap all of them and start again . Which was hugely embarrassing for myself.

How could I have avoided this? By looking at the long term goals and realising that test automation requires time and patience for it to work properly and provide any real value. This is the fundamental issue with automation in a lot of companies, they want automation because it’s the latest trend and they must have it right now. So they hire someone, or a team to make it happen, but don’t give them the time or resources to get it done right.

Automation, when rushed may provide value short term, but as your system evolves and grows, that value starts to decline and you’re quickly left with a solution that requires constant maintenance, which in turn is giving no measure of quality.

I always tell companies that bring me in that I am more than happy to design them a framework and build it, but first I want to know the roadmap of the project I’m testing. Or if that’s not possible, at least some kind of long term aim for the project. If a project manager says to me, in 1 month we aim to have a skeleton user journey that needs 100% automation, I’ll tell them that that just isn’t worth doing. If however they can tell me that in 1 year, they want a user journey with fully dynamic content based, with different paths planned based on other projects features and that in 6 months, an API will be introduced for retrieving data, then we’re in a much better position.

Then I can write the framework to deal with that initial skeleton journey, but write it in a way that means eventually when that content becomes dynamic and not hardcoded, my framework and my tests can easily be updated to deal with that. You can never completely eliminate the need for maintenance, that’s just impossible. But you can limit that maintenance cost with forward thinking, and less maintenance means more time for writing tests which ultimately is the reason you’re writing the framework in the first place.

There is no need to curb your enthusiasm when it comes to writing a framework, it’s good to have that passion and enthusiasm to get it done. But use that in a positive way to make sure it’s done right. Take that time initially to do things properly, and the dividends later on will be massive. That’s another thing I always say when managers want things to hurry up and start delivering, is that rushing now will only come back to bite you later. That initial cost of spending a few more man days or weeks writing a good, solid framework will far outweigh the cost of having an entire QA team having to maintain poorly written tests that never stay green. That in itself brings issues with bugs not being caught and releases being held back, or even worse, customers being the ones to find the bugs you missed.

I hope this article has been useful, I’m sure some of you are reading this and nodding in agreement having experienced similar yourself. Either from the automation side or the manual side having to pick up the slack left by poorly written automation. If this is a new thing to you, then heed my warning and remember this saying, as it’s pretty fitting…”Fail to prepare then prepare to fail”.