The HALF-BAKED truth about Open Source Testing Tools
Open Source!!! Free!!! Sounds exciting! Isn’t it? Well, I am not against using open source tools and hence I’m going to say YES along with many of you. But everything has some limitations and open source testing tools are no exception. It is upto the users, aka, “the testers” to understand whether open source testing tools will be a good choice or not? Let’s enlighten ourselves with some of the limitations that are a part and parcel of most open source tools especially the ones for testers.
Today open source tools are an integral part of many organizations’ software testing strategies. When it comes to software testing the budget of most companies are very tight and open source testing tools enables them to achieve their goals in less cost. At least, this is what everybody thinks. Gone are the days when you depended heavily on paid commercial tools to automate your tests; now-a-days there are various tools available, which are open-source and free, which means you’ve complete control over the tool after you’ve downloaded it. The problem is that most of these open-source tools are poorly documented or have no documentation at all. It means that you’ll always get to know what you can do with a particular tool but the “how” part is often tough.
Consider a scenario where a company wants to establish itself in automated testing of mobile apps and wants to evaluate open source options available. The only intent behind going for open source is ZERO cost associated with open source tools (full version) in comparison to commercial tools.
Now, the team responsible for evaluating tools will do at least the following:
1. Prepare a list of all open source tools available
2. Do a comparative study (with other open source tools)
3. Choose top few tools according to their need
With each tool in top few, the team will:
a. Download the tool
b. Install/Configure the tool
c. Do a “Proof of concept”
d. Look for an existing framework or analyze possibility of developing one around it.
Let’s take a closer look at few issues that may arise during some of the above mentioned steps:
Download: The page to download doesn’t mention whether the tool has different versions for different platforms or not? You’re confused and the documentation is of no help at all. Also, the poor documentation doesn’t help you with several external dependencies that you might want to resolve before actual installation.
Install/Configure: Let’s consider the scenario where the team evaluating the tool is facing issues during installation/configuration. The only hope is the petty documentation available or to ask queries in online forums (if they exists). But it’s worth to note that both of these options don’t guarantee 100% working solutions. You may at this point realize the importance of efficient technical support and well maintained documentation that most commercial tools provide, which helps you get going.
POC: During proof of concept, while performing certain action, the team may encounter an error. Again poor documentation only helps you till certain level and after that you’re on your own. The forums are good place to ask queries but it depends on member’s choice if they’ll reply. With no point of contact for show-stopper problems and bugs encountered in middle of an important execution, the project can end up in disaster. This is where commercial tools have an edge because efficient documentation and the presence of multiple forums (backed by organization that owns the tool) makes your life bit easier. Also, you may find lots of books, articles, white papers, etc. focused around that commercial tool and thus life seems much easier with them.
Most open-source tools, if not all, are designed to serve a specific purpose rather than being generic like the available commercial tools. There is a risk that these features which have been evaluated to be suitable during the pilot run might turn out to be insufficient during scale-up. The advantage of having the source code means that customization is faster, although dedicated personnel are required to handle such requirements. This might add further to the overheads.
Another important thing to evaluate is the ability of any tool to integrate with others in family. Software testing tools specially needs to talk to test management tool and defect management tools at least. Integration with CI tools can be the next big thing to look. While commercial tools provide options to integrate with other sets of tools, e.g. for exporting the documentation from a word processor to the tool, the same may be difficult with custom-made applications.
Let’s take example of Calabash, an open source tool for performing acceptance tests on mobile apps. This tool is a fairly good option when it comes to mobile test automation (I personally like this tool), but the lack of documentation makes the life of tester bit difficult. The installation/configuration part is easy yet tricky. An error during installation may leave you frustrated. There exists a Google group for both calabash-android and calabash-iOS but very few responds in those groups. Since the time Xamarin has taken over ownership to maintain calabash, those few have also reduced frequency to respond in those groups. (Not sure if Xamarin plans to make calabash a closed-source tool as their commercial cloud used calabash heavily.)
And this gives us a last, but not the least, point to consider. What if the company or group, responsible for maintaining the open-source tool, decides to make it closed-source? This means no further free downloads (forever), no support on forums, etc. Worst case, what if that company/group decides to end that project? All this means troubles for testers.
So, saying that “open source tools may be a good option to consider” is just a Half-Baked Truth, when we don’t fully know what they offer for real.
This article was published in our May 2014 edition.
https://www.testingcircus.com/the-half-baked-truth-about-open-source-testing-tools/https://i0.wp.com/www.testingcircus.com/wp-content/uploads/Half-baked-truth-about-open-source-tools.jpg?fit=600%2C286&ssl=1https://i0.wp.com/www.testingcircus.com/wp-content/uploads/Half-baked-truth-about-open-source-tools.jpg?resize=150%2C143&ssl=1ArticlesOpen Source Testing Tools,Testing ArticleOpen Source!!! Free!!! Sounds exciting! Isn’t it? Well, I am not against using open source tools and hence I’m going to say YES along with many of you. But everything has some limitations and open source testing tools are no exception. It is upto the users, aka, “the testers”...Kapil SaxenaKapil Saxena[email protected]AuthorKapil Saxena is best described as a passionate tester who believes that “everybody tests”. He is currently associated with Magic Software Pvt Ltd. as Test Manager and enjoys working with testers on WEB and MOBILE automated test framework design. In past, he has been associated with organizations like GlobalLogic, HCL, BEL, NTPC. Kapil carries rich testing experience of over 10 years in various domains like Healthcare, Government, E-Commerce, Mobile, E-Learning, etc. He also believes that a good tester must have clear basics and thus imparts weekend sessions to those who need it. He is an active participant in weekend testing and has delivered various talks in conferences. He loves to spend his free time with his family, friends and testers.Testing Circus
I read half of the article and skimmed through the rest, and what you say makes sense; however I have a different philosophy with the open-source testing tools. Most of the testers just give up when they see lot of files that they have downloaded and none of them is an executable. Why is that so? A tester can go ahead and learn to execute it by compiling or using the tool from command line itself. In my experience, I have seen that open-source tools (Some or most) have great potential if a tester doesn’t give up on learning it and getting to a stage where he / she starts using it during the testing activity.
Having said that, in my start-up (It is going to be launched officially to the world in a week from now) we have Research and Development team which focuses on Open-Source contribution and we see that it is not less than a commercial software or a product. We do not want to settle for less just because it is open-source. We want to make easy for software testers to learn our tool or software and see how it can add value to their testing activity. We are game for it and you will see our contribution with usage guide or demo video(s) when we launch.
However, while I want to help those category of testers who want the cakewalk stuff; I still believe in compiling and making the open-source tool work for me or add more features to it with the help of my team instead of going away from open-source.
Last, but not least a nice article! Keep writing whenever you feel like 🙂
Thanks for your effort in writing this and sharing it with the world. Peace!
Hello Kapil,
So I have read the article and am….well…. shocked. Pretty much all the negatives listed for Open Source can be attributed to commercial software just as much. I have worked with open source and commercial testing tools for over a decade and my negative experiences have been next to completely on the commercial side of things. OSS has generally impressed me every step of the way (this does not mean these tools are perfect!).
When looking at commercial tools you can download….ah…..no. You can’t. Please buy (at least the ridiculously expensive ones).
The installation of commercial tools is just as complex as OSS. Actually more complex when you include licensing. Often a specialist has to come in to set everything up anyway ($$$!!!). You can’t have test-versions or just another VM, where you have a play because that will cost $$$. I have had mostly atrocious documentation that barely covers the happy case on commercial software. Forums (if there are any) are hidden behind pay-walls or worse.
Proof of concepts need to be done with any tool! Somehow commercial often are just able to wiggle their way around it causing many test automation projects to fail down the track because of holes in tool functionality. Not doing a proof of concept in your personal & project context is foolish no matter what tool!
If an OSS tool is too complex to install or use you just move on to the next. Apart from learning and time invested there are no extra costs. You as a tester will be the better off because you just learnt something. Usually I note that the problems are not on the computer side of things anyway.
As for purpose of tools I find that OSS covers everything that is an accepted standard (and then some). This is where commercial tools can actually help. If you have some freakish legacy stuff in your software they might just have the answer. But that should be the exception. OSS tools also integrate well to most other applications due to the fact that they do adhere to standards.
I’ve had issues integrating commercial tools from different vendors. Especially where the products are competitors.
Oh and just as OSS tools can vanish, so can companies! With the difference, that when a company does you WON’T have the code in your hands.
I have used OSS for years now and I generally get problems fixed through forums within 24hrs. I challenge any commercial vendor to match that on a regular basis. This includes nightly builds …just for me! The generosity of people in the OSS space is usually just shy of amazing! Yes, there are products that are still struggling to get off the ground and yes, you’d expect documentation to be bad. But if you compare commercial and OSS projects that have the same age you will surely find that OSS is in no way inferior. I admit that there is a gap in factory-style test management tools in the OSS space but that might have a reason too.
I’d say this article is wrong and superficial. Sorry if I am so direct but I think this article too dangerous to let stand without challenge. If we look at the market today OSS testing tools are thriving, where lots of commercial tools are not expanding. If OSS were as bad as you portray, then I couldn’t explain why they’d be so popular.
There are of course some excellent commercial tools too (e.g. Atlassian, SmartBear and others) but I think we need to get deeper into what categories of test tools we are talking about here (commercial & OSS) and differentiate why I’d choose what tool.
Best Regards
Oliver Erlewein
P.S. At this point I want to shout out a BIG thanks to the makers and communities of:
JMeter (Apache),
JMeter-plugins,
Robot Framework,
Selenium crowd,
Watir and
WebDriver
and many others that are putting their passion into these things!
Hello Oliver,
Thanks for your comments, but let me tell you that I am in no way against OSS. In fact, like you, I too prefer using OSS compared to Commercial ones. Like you I am too impressed by them, the way they are developed and I agree with you that all this still doesn’t mean that “these tools are perfect!”. But I am not biased against Commercial Tools. Please understand that through this article, I’ve tried to explain what common testers (or teams) miss to understand while selecting a tool. Most of the time, I’ve seen people blindly following that OSS is good, even when their needs are not fully filled. There are other cases as well where people just neglect the power of Commercial tools.
Said this, I think you’ve too much against Commercial tools. Love for something doesn’t necessarily means hatred for others. Let’s take all points you’ve raised for Commercial Tools one by one:
1. Download: I really don’t think that with this much competition in market, any company not allowing users to download tools. As far as I’ve seen, almost all testing tools are available for download. For exceptions, if they exist, I think they are not doing good to themselves. But still I’d love to know which Commercial Testing Tools are not available for download?
2. Installation: I beg to differ with you on this 1000%. In fact, the beauty of Commercial Tools lies in easy installation. No matter how complex they may be w.r.t usage, but except for clicking few NEXT buttons, you won’t have to do much. QTP, LoadRunner, Test Complete, etc all good tools are very easy to setup.
3. Test-Versions: Almost every Commercial tool have trial versions available, which means you’ve few days with you to explore the tool. Test Complete from Smart Bear gives you 30 days period to evaluate tool with full features. Even you’ll get several emails from technical team during this trial period to help you in case you’re stuck somewhere. Is this possible with any OSS? Everybody knows the answer.
4. You mentioned “commercial often are just able to wiggle their way around it causing many test automation projects to fail down the track because of holes in tool functionality”. I don’t really know about your testing experience but how can you blame any tools functionality as cause of project failure. It’s altogether team’s decision which tool to use? Whether you do a POC or you select tool based on your experience, it’s your choice. You can’t blame others for your decisions. This is the core of this article. You’ve to be SMART while selecting tools as per your needs and environment. Blindly favouring some tool may result in unexpected results.
I won’t mind if you don’t like my article. It’s your choice. My only intention is to spread the awareness that tool selection must be done wisely. Many times, you won’t have time to do POC or explore tool. At other times, you’ve enough time to decide. Whatever may be the case, you’ve to evaluate all available options against all pros & cons before going with any tool.