What Bugs Are You Missing ?
Do you care for information in your context?
Testers might know this situation. There are more things in the software that bug you, than is usually accepted as bugs by development, project management or stakeholders. This article will give you an overview of situations where these bugs are usually to be found and how you can improve your search. And it provides you with some questions to answer for yourself, if your project or process is dealing with all of them.
The two circles model
The model I like to use for explaining the benefit of my testing is based on the intersecting circles diagram. The circles are a bit apart from each other to describe each section better. In reality, I hope for you, the circles should overlap way more.
Let’s kick off with a model with two circles, one for the specification (left) and one for the software (right). In section 1, where both circles overlap, the software meets in general the specification; this area should be covered by regression testing to keep it that way. Section 1 has to be maximized, if the specification is really describing the wanted solution, but more about this later. Where the circles don’t intersect, you have on the left side (section 2) the missing but specified features. Testing against the specification should uncover most of this area. What approach you chose is not that important, as long as you have read and understood the specification.
On the right side (section 3) you find functionality in the software that was not specified. Classical test case driven and specification based approaches tend to miss those areas. You can learn about this area best with exploratory testing and by maintaining some sort of map. You have to test in this area, to find out if the additionally implemented functionality is working correct and serving the solution or if it has to be fixed or even removed.
The three circles model
The first extension of this model comes with a third circle representing the customer. When it comes to customer satisfaction you also have to meet the customer’s expectations. When you know about the problem the customer wants to solve, you can really rate the solution your software is providing.
This model is rather sketchy when your software has not only one but between a few and many customers. In this case usually a role like a product owner takes over the part of defining the requirements, but is not directly representing the customer satisfaction.
Now with a customer in place, we return to section 2. Obviously the specification does not meet the customer’s requirements. Well, who could have thought of that before? With a review process including the customer the content of this section can be moved to section 4, the overlap between customer expectations and specification, and should be maximized. But still, a review is a process that involves human beings, and is therefore fallible. Not everyone can fully imagine a solution from only reading about it. This can result in features localized in section 1.
Section 5 however is the jackpot that needs to be maximized. The software is according to the specification and fulfills the requirements of the customer. This section needs to be checked regularly by regression testing to keep it that way.
So, now let’s take a look at some situations that are not always covered by the common project processes, but should be. Section 1 now meets the specification, but not the customer’s expectations. So, what do you do now? You need to change both the specification and the solution to bring this up to section 5. Or do you leave it the way it is, because the customer was involved in the review and knows that he gets it this way. This is one of the many constellations where context drives the decision. If the stakeholders decide to leave the functionality in section 1, how do you track this decision? Can you follow-up on this later?
Section 6 is a bit tricky, because development implemented something that the customer needs or expects without the specification writing about it. How do you handle situations like this in your projects? How do you find those things? Who takes the responsibility to say, “Good, just what the customer needs, but we forgot about it”?
In section 3 you found something that was not specified and the customer does not obviously need. What now, write a bug and remove it from the solution? Change the specification, so it is described correctly? I would suggest that if you find this, you should at least test it, find out if it works and does no harm to the solution. How do you track such findings in your project context?
In section 7, where the customer’s requirements are not met by specification or solution, we come to a difficult situation. When the software is already in use, those missing requirements are brought up as bugs, and it’s usually too late. Commonly there is a process implemented to track additional requirements or changes in those phases of the project. Bring in the customer early if possible and let him taste the solution. The earlier you find a problem the cheaper it is to fix.
Think of a tester on your team who is able to test from a customer point of view. If this tester finds something that isn’t neither solved by the solution nor specified, does your process enable you to commit such findings and discuss them?
If you let the customer sign-off the specification you face the situation, that you have a legal document to state that the customer saw the specification. And if the solution reflects the specification the customer has nothing to complain about. From a legal point of view, this is correct. But what would you expect about the customer satisfaction? I don’t want to tell you, to do everything the customer wants, even if he didn’t pay for, or if it was his fault that he did not mention it before. No, I only want to tell you, that even if the software fits the specification, the customer still might not be happy. Don’t be surprised, you have been warned.
Expanding the model
But usually the world is not that simple. This model doesn’t go far enough to see the next bigger picture and with that the sources for all sorts of additional problems to deal with. All three parts bring in other components, to give you an idea, where the complexity can hit you. I want to add some hints about how the test team can challenge this complexity or where you have to give an extra thought about your processes to deal with bugs found in those areas.
The first circle to expand is the customer’s circle. When working with people who have a certain (high) degree of expertise in their domain, they tend to assume for some details that everybody knows about them and they don’t mention them. If someone writes the requirements without (much) domain knowledge, he can also miss several details to the fact that he doesn’t know better. So let’s add some domain knowledge around the customer’s circle.
The specification, hopefully written by someone with some degree of domain knowledge and at least a fundamental understanding for development processes, is trying to explain what the customer wants in a language that the developers understand. But, as you might know yourself, a specification is rarely complete, and much too often ambiguous. I want to add this as the implicit part of the specification. This can bring up many problems and fierce discussions between the author of the specification and development, when bugs are brought up for this section.
The solution itself might contain functionality that is not easy to spot, hidden, rarely triggered, of course undocumented, and leaving only minimal traces to the unknowing user. If this unknown feature goes out undiscovered and untested, you might receive rather unexpected bug reports.
Domain Knowledge and customer’s problem
It’s always good to bring some domain knowledge to testing. You don’t have to be a contributory expert, like Harry Collins would define it. But it’s good to at least understand most of the customer’s language, the problems he usually deals with, the processes and the standards of the domain. That way you can understand the problem better and rate the specification and solution from another perspective. When bringing in domain knowledge to reviews or test reports, you as a tester usually get an odd peak. If your domain knowledge is not yet accepted in your project, you need to bring your A game, to convince the others.
For me one of the big objectives of testing is to find out, if the solution fits the customer’s problem. You should know and understand the problem that needs to be solved, even for small changes. What does the customer want to improve, how did he deal with that in the past? Until customer acceptance or even production the customer cannot defend himself, so you have to become the customer’s advocate. And usually, once the specification is signed off, you will be the only one speaking for the customer.
If the system you have to test is the replacement of a legacy system, try to get access to the legacy system. Learn about how the customer used to work with that system, what are the reasons for the replacement. What does the customer expect from the new solution to make the difference?
Or, if you get the chance, talk with the customer or sit in at customer meetings. If you know and understand better the needs of your customer, if you have a chance to experience how he thinks, how he works or what he expects, you gain a big test oracle for your tests.
Another heuristic to the customer’s expectation can be found in the products he produces, sells, or creates. Think about the quality, design, functionality, and robustness of the products and you can imagine what he expects from the product you deliver.
If you’re testing against the specification, like for example the classical test case driven approach, you will find deviations from the specification, deviations that don’t meet the pre-defined expectations. But how do you handle e.g. deviations that exceed the expectation, or in this case meet the customer’s expectation? The software does not behave like the specification describes, it’s even better. If you track this as a bug, do you track it against the specification or the software, or do you track it at all?
If I tell you, that everything you find inside the circle software/solution, that is not described in the specification or requirements has to be documented there. Most of you will probably answer, “Yes, of course”. But really, do you?
In the software there might be unspecified and undocumented features. You have a function where the developer thought that might be good to have, even if no one specified it or asked for it. How do you find those Easter eggs? Three ways that come to my mind are by accident, by exploring the system or by talking with the developer.
How would you handle such a situation in a classical factory and test case-oriented approach? Would you have even found the Easter egg? Yes, if the developer is reviewing the test cases for his part of the system and is actually missing a test case, idea or whatever for this functionality. Well, there is another way. It’s widely known that (at least most) human beings are too intelligent to execute a pre-scripted test case like it is intended to or how a machine would. Most humans register way more information on their way through the test steps than they are asked for. Now, if you have a human being executing a test case and the brain is not yet asleep, the human might start to explore this newly discovered functionality. Is this tester going against the process! The test case didn’t ask for this function, there is nothing in the spec, so what are you doing? Write a new test case, but based on what information? Write a bug, against the software or the specification?
An exploratory tester would note it as an opportunity. It will be reported and tested. Because in my opinion whatever is delivered to the customer should at least work. Specified or not. In the 2-circle model I mentioned earlier, that you need exploratory testing to cover the unspecified parts of the system to test.
Extend your own context model
With this you start introducing context to your testing. For your project just add additional circles, like one circle per customer, operational requirements, laws, standards and all other influences on your project. Don’t accept the specification as the one source of truth. It is not!
Testing without specification
Imagine your test team is big enough and you have a chance to explain the problem to be solved to one tester without giving him the specification. Then let him explore the solution and listen to his report. You have a chance to come up with design flaws, with hard to understand processes, with functionality that doesn’t really solve the problem (even if it perfectly fits the specification and a whole team of reviewers signed it). Or, you get the feedback that the solution is easy to use, fits the problem and everything is fine. Is that a piece of information worth reporting to the stakeholders? Think about it.
Create a culture and an environment that deals with all variations of information
How do you and your project handle findings that are, let’s call it, out of scope. But you still want to mention them to the stakeholders. You don’t have to call them all bugs; that might be confusing. Depending on your project, process and context you can use bugs, defects, errors, changes, issues, items, backlog items, worries, or whatever you want to call them. You can handle them in some sort of backlog, you can track them as issues, or collect them in a list or mind map and go through this collection with your project managers. Go through this list regularly to give them a chance to react to the gained information.
Testing is providing information to the stakeholder, so that he can make an informed decision. So give him ALL the information he needs.
This article was first published in our April 2014 edition. Read more such articles here.https://www.testingcircus.com/what-bugs-are-you-missing/https://i2.wp.com/www.testingcircus.com/wp-content/uploads/3-circles.png?fit=714%2C636&ssl=1https://i2.wp.com/www.testingcircus.com/wp-content/uploads/3-circles.png?resize=150%2C150&ssl=1ArticlesTesting ArticleDo you care for information in your context? Testers might know this situation. There are more things in the software that bug you, than is usually accepted as bugs by development, project management or stakeholders. This article will give you an overview of situations where these bugs are usually to...Patrick PrillPatrick Prill[email protected]AuthorPatrick Prill has over 10 years of experience in software testing. After 4.5 years as a tester he became test manager, coordinating the work of ~50 people for another 5 years in a big test project. His new job as test lead for a software and consulting company for the automotive industry brought him back to a smaller test team and the hands-on experience of testing software again. This experience and following the discussions and articles of the context-driven testing community re-lighted his fire for testing and bug hunting. Patrick is living outside of Munich, Germany and is a proud husband and father of a wonderful daughter. In the little spare time he is a wood turner.Testing Circus