Vasu,
Lets assume that in your project, customers are finding more production defects, if you ask me , then i can say
1.I am finding your test strategy is wrong.
2.Whenever production bug is found by user, just analyse whether the bug they found (and you lost) is because of carelessness / negligance of your testers or because of the your nature of testing i.e your test strategy
If first is the case, then we need to make sure it wont happen and we may practice techniques like pair testing (two tester in same test area) or more tight supervision of test execution etc. If second is the case, you have to revisit the strategy.
3. If you keep on getting production bug then you need to get the list of production bug and do the gap analysis with your test cases. It is not simply adding the defect as test cases, but you do the 'gap analysis'. For e.g take X number of defects found in production and see the test cases in that area , the question should be asked is 'Why they havent added as test cases before?'. Finding a pattern between "{Z number of defects found by production users} vs {Y number of (probably ineffective) test cases} will also throw some light.
4.Diversify the testing techniques.This is one of the important area that should be concentrated. Change the testing techniques often during test life cycle.According ot context driven testing school there are four dimensions of testing,
-People oriented (beta testing, UAT etc)
-Risk oriented (Risk based testing)
-Coverage based (like documentation based testing, testing page by page of product documentation, or combination testing etc).
-Activity based (like regression testing, Long sequence testing etc)
There are many techniques in each section. But one of the important technique that i would like to quote is "Map and test all the ways to edit a field". You can change a field value in different ways. Depending upon the context, this technique may prove powerful.You have to test in all the possible ways.
5.Use gray box testing: do the black box testing by applying white box testing techniques like Decision coverage and condition coverage.
6.Clear the old test cases. See what are the existing test cases are obsolete and delete it or make them as non-executable. In this way you can save time & energy which can be used for new powerful attacking test cases.
7. Get powerful test data, it depends upon the project, some project real time data will act as powerful agent.
8.Get the help of end user (subject matter expert / business analyst). Schedule a meeting and go with important questions, like pattern of usage, pattern of real time data, more use cases / user stories etc. based on which we can create more powerful test cases. If it is product, then ask product manager, or marketing person to gain this knowledge on common usage.
9.Get the use of support team, use support team's knowledge on their calls in your testing.
Iteration by iteration, in this way we can reduce the production bugs.
How to increase Test Coverage?
I have often been asked by my clients to increase application test coverage through testcases and reduce customer defects and i wish i have a magical wand to help them.
In my perspective, increasing test coverage depends on various factors. How knowledgable testers are on the application, how effective they are when it comes to understanding of new requirements and creating testcases for that, do they step into the customer's shoes during test design are some of the key aspects which help in increasing test coverage.
Testcase effectiveness is a metric that talks about how effective a testcase is in finding defects and any testcase that is executed repeatedly over a period of time without identifying defects should be reviewed for its effectiveness. And, to design an effective testcase, the tester's knowledge and his approach of the application from the customer's viewpoint definitely matters.
I have seen products that have more than 10000 testcases for the application but repeatedly after every release customers file many defects which should have been ideally caught by QE(Quality Engineering/Testing team). And needless to mention, all these customer defects are being created as testcases after every release and are added to the test execution bucket for the next release to ensure that the same defects are not repeated and for increased test coverage. This eventually results in a lot of testcases being created but somehow the end goal of "coverage " never gets fulfilled.
So, the question is, how do we bring 80/20 rule here? 20 percent of the testcases should find 80 percent of the defects in the application. Because unless we do that, execution will be a laborious task with no goal and at the end of the day testers will still say "I have not tested this part of the application well and hence could be a risk". So, how can testers provide customers with a quality product by testing as much as possible within the given execution timeframe?
p.s: I am definitely not looking at 100% test coverage.
