Open Source!!! Free!!! Sounds exciting! Isn’t it? Well, I am not against using open source tools and hence I’m going to say YES along with many of you. But everything has some limitations and open source testing tools are no exception. It is upto the users, aka, “the testers” to understand whether open source testing tools will be a good choice or not? Let’s enlighten ourselves with some of the limitations that are a part and parcel of most open source tools especially the ones for testers.

Today open source tools are an integral part of many organizations’ software testing strategies. When it comes to software testing the budget of most companies are very tight and open source testing tools enables them to achieve their goals in less cost. At least, this is what everybody thinks. Gone are the days when you depended heavily on paid commercial tools to automate your tests; now-a-days there are various tools available, which are open-source and free, which means you’ve complete control over the tool after you’ve downloaded it. The problem is that most of these open-source tools are poorly documented or have no documentation at all. It means that you’ll always get to know what you can do with a particular tool but the “how” part is often tough.

Consider a scenario where a company wants to establish itself in automated testing of mobile apps and wants to evaluate open source options available. The only intent behind going for open source is ZERO cost associated with open source tools (full version) in comparison to commercial tools.

Now, the team responsible for evaluating tools will do at least the following:
1. Prepare a list of all open source tools available
2. Do a comparative study (with other open source tools)
3. Choose top few tools according to their need

With each tool in top few, the team will:
a. Download the tool
b. Install/Configure the tool
c. Do a “Proof of concept”
d. Look for an existing framework or analyze possibility of developing one around it.

Let’s take a closer look at few issues that may arise during some of the above mentioned steps:
Download: The page to download doesn’t mention whether the tool has different versions for different platforms or not? You’re confused and the documentation is of no help at all. Also, the poor documentation doesn’t help you with several external dependencies that you might want to resolve before actual installation.
Install/Configure: Let’s consider the scenario where the team evaluating the tool is facing issues during installation/configuration. The only hope is the petty documentation available or to ask queries in online forums (if they exists). But it’s worth to note that both of these options don’t guarantee 100% working solutions. You may at this point realize the importance of efficient technical support and well maintained documentation that most commercial tools provide, which helps you get going.

POC: During proof of concept, while performing certain action, the team may encounter an error. Again poor documentation only helps you till certain level and after that you’re on your own. The forums are good place to ask queries but it depends on member’s choice if they’ll reply. With no point of contact for show-stopper problems and bugs encountered in middle of an important execution, the project can end up in disaster. This is where commercial tools have an edge because efficient documentation and the presence of multiple forums (backed by organization that owns the tool) makes your life bit easier. Also, you may find lots of books, articles, white papers, etc. focused around that commercial tool and thus life seems much easier with them.

Most open-source tools, if not all, are designed to serve a specific purpose rather than being generic like the available commercial tools. There is a risk that these features which have been evaluated to be suitable during the pilot run might turn out to be insufficient during scale-up. The advantage of having the source code means that customization is faster, although dedicated personnel are required to handle such requirements. This might add further to the overheads.

Another important thing to evaluate is the ability of any tool to integrate with others in family. Software testing tools specially needs to talk to test management tool and defect management tools at least. Integration with CI tools can be the next big thing to look. While commercial tools provide options to integrate with other sets of tools, e.g. for exporting the documentation from a word processor to the tool, the same may be difficult with custom-made applications.

Let’s take example of Calabash, an open source tool for performing acceptance tests on mobile apps. This tool is a fairly good option when it comes to mobile test automation (I personally like this tool), but the lack of documentation makes the life of tester bit difficult. The installation/configuration part is easy yet tricky. An error during installation may leave you frustrated. There exists a Google group for both calabash-android and calabash-iOS but very few responds in those groups. Since the time Xamarin has taken over ownership to maintain calabash, those few have also reduced frequency to respond in those groups. (Not sure if Xamarin plans to make calabash a closed-source tool as their commercial cloud used calabash heavily.)

And this gives us a last, but not the least, point to consider. What if the company or group, responsible for maintaining the open-source tool, decides to make it closed-source? This means no further free downloads (forever), no support on forums, etc. Worst case, what if that company/group decides to end that project? All this means troubles for testers.

So, saying that “open source tools may be a good option to consider” is just a Half-Baked Truth, when we don’t fully know what they offer for real.

This article was published in our May 2014 edition. SaxenaArticlesOpen Source Testing Tools,Testing ArticleOpen Source!!! Free!!! Sounds exciting! Isn’t it? Well, I am not against using open source tools and hence I’m going to say YES along with many of you. But everything has some limitations and open source testing tools are no exception. It is upto the users, aka, “the testers”...