Myths of Software Testing

1. Testers just find bugs
2. Automation will make testers obsolete.
3. Testers and developers are at loggerheads
4. Testers are always testers
5. Anyone can be a tester
6. Complete testing is possible
7. Testers determine quality of the product
8. Just anyone can test.

Different Faces; Different Phases

I have come across a varied personality types in my career in software testing. Here  are a few:

Aggressive

Systematic

Innovative

Exploratory

Domain Expert

Are there more..?

The Perfect Test Suite

A "perfect test suite" would have covered all scenarios possible in testing. Obviously such a test is not humanly possible as the combinations are infinite.The sheer variety of the inputs, the combination of the path and not to mention the environment of usage adds up to a huge number. That is why we are almost surprised by the bugs that we encounter in the field.

The endeavor in test case writing is to reach for this gold standard during the test planning phase.

How could we do it?

Do share in your ideas as well.

Velocity of Software delivery

Time is money and ideas are a plenty and freely available, the market opportunity has a short window of time. In these circumstances, the key factor which would determine the success of an organization  deliver value rapidly.

For an organization developing software, the velocity at which the software is delivered plays a key role in delivering software.

Note that I have used a few terms intentionally and defining them below

Qualification of feature :

While definition of testing is generally limited to finding bugs, we need teams to move towards a more effective qualification system which uses a combination of defect prevention and testing. The more the former the better.

Velocity: The direction in which we run fast is as important as how much we run/


There are two fundamental parts in Software testing:

Test Plan Design

Test design is the craft of converting the user and system requirements to steps which a test executor can execute and check the behaviour against.The key success factor here is to increase the coverage of test which will expose various probable behaviour.

The test plan also acts as a communication to the development engineer on the possible design gotchas.

Test Execution
   Executing the test plan in the most efficient manner is the goal of test execution

The velocity of software delivery can be increased by wearing these two hats by the tester.

Tester Effectiveness Metrics for Performance Appraisals.

Every activity generates a number.The key to excellence in any activity is to find the relevant number and work towards being better at it. Sadly, most test teams focus on number of defects found and hence the huge defect numbers.

So, is measuring defects irrelevant for measuring individual's performance and should it not be counted at all for a teams effectiveness? It should be.Only that it should be tracked by the development team to gauge its effectiveness.The test team which accords undue importance to bug counts plays a limited role of improving the developers skill.

Is there a better test metric?


To answer this question , it is important to ponder about the importance of testing in during product engineering.To analyze this, let us answer the following question.

Why is testing done?
To find all bugs in the engineering phase
Therefore, any bug which was not found during the testing phase is a error in the testing process.So, it is important to track "Defect Escape" as the ultimate in test metric.

However, Defect Escape is a metric which is not useful for managers for the purpose of taking decisions on performance on the individual testers.In business terms, it is a "Lag Indicator".

The corresponding lead indicator for this metric would be the "Test Scenarios Executed".So the defect could have escaped only if there were no corresponding test case OR the test case was not executed.

Hence, for the purpose of Tester effectiveness, it is important to track the following parameters:

• Test coverage for Junior Testers.
• Test Cases/Scenarios Addition for Senior testers.

Engineering's win;Product's Loss.

Software Product: Art , Engineering or Marketing

As with any new discipline, software application development once upon a time an art is fast and definitely becoming engineering. A product of engineering is generally reliable, robust and has the desired quality. It is for this reason that  different disciplines emerged in the industry.Today there are clear demarcations in developer, tester and project managers.In spite of all the structure,we know that if bridges were designed like software then there would be a lot of ferries operating.

Generally,in the software industry,it is  thought "uncool" to follow process. A constraint to creativity is often cited as the reason for not following process.This feeling is more prevalent in the product companies and less in software services where process are followed in letter (if not for the spirit).

QA Engineer or Tester: While the rest of the engineering industry have moved from mere defect finding to finding out the root-cause of the problems, software "engineering" still suffers from this self-image of being more art than engineering.

A Feature development workflow

A process flow diagram of a simple feature workflow would be:

       Requirements -->Development ---> QA/Testing

Product managers is come up with requirements for the next generation product.He is implicitly measured on the number of requirements which is authored.

Developers job is to create software which satisfies the requirement.However, he is implicitly measured on how fast he can churn out the features.

Testers  test the product.He is implicitly measured by how many defects he files.

There is no alignment in each of the measures to the business goal of an organization. I have often found that the importance of the business goal. is completely ignored. Consequently, the product gets caught in interdepartmental warfare.

Product managers are commended for the number of requirements, developers for the number of lines of code and testers for the number of defects.

The customer is still left unhappy.No doubt all work hard, but is it relevant to the customer's need is the question to be asked.

Solution:

As for many problems existing the world today, the solution can often be found in the discipline of marketing.
                                                    "Understand the End Customer"

The solution is simple because it is relevant.However,to change the mindset of the software development practice would need active facilitation.

Measure what you should, not what you can

"What you cannot measure, you cannot improve"---Lord Kelvin

Too often Test Managers measure what they can.One of the most easiest thing to measure is the number of bugs filed. Managers get obsessed with the use of that metric for almost everything.

The metrics generated from the bug filing is used for a variety of scenarios from understanding the stability of the product to measuring the productivity of an employee.

No doubt it has its tactical utility,but more often it can do more harm than good by having bugs filed as the only metric of a testers effectiveness.