Developers Reviewing test cases.

Recently I came across one process. Reviewing of test cases by developers. In my team testers are writing test cases (or scenarios) and getting reviewed them by developers, and the net result is developers who are finding test cases are 'suddenly' seeing new conditions, and remembering that they missed these conditions in their unit test or code. Hence, in my company code freeze date is about to change and net result is almost our test cases are passing , without much Bug filing.

Is this good practice? I definitely don't think so. Either the code freeze date should be rigid or Test case review shouldn't be done by developers. It should either inside test team or if it is needed product management should take care of this.

Cheating Tests

Another week goes by and more leading consumer websites crashed and burned, causing losses in both revenue (millions) and, perhaps more importantly, in consumer confidence in ecommerce. This week fail whales were sited swimming in the oceans of one of the world’s largest retailers, a leading children’s toy manufacturer and the industry’s leading online payment processor. PayPal alone (which was down for several hours this past Friday) may have lost hundreds of millions of dollars for itself and the enormous network of retail outlets that rely on them for their financial transaction services.

So why are fail whales continuing to happen so frequently? Don’t these companies test their sites? The answer, for load and performance testing, is of course they do…most of the time. In fact, some companies are spending millions of dollars on people, hardware and tools to load and performance test their websites. So why all the fail whales? One answer may be that organizations that have chosen the wrong testing tools or test service are, in fact, cheating on their tests!

Consumer facing website have become the primary channel of revenue and product information for millions of companies around the globe. So why cheat on testing? The pressure to deliver (business agility) is enormous today, for all IT organizations, and may well be a key reason cheating has become a common practice. Many test subcontractors and test companies cheat on their tests simply because they run out of time. According to PCWeek, this is what happened to AT&T on the pre-registration site for the iPhone4 launch. Due to a last minute feature upgrade they had no time to adequately performance test the site, which, of course, crashed an hour after it was launched, due to a 10x spike in traffic, creating a PR and revenue nightmare! Other organizations cheat because they just can’t afford the resources (people, hardware and tools) to properly test their sites in the first place.

The most common way to defend this cheating is to use semantics to obscure the shortcuts taken. When asked by an enraged business owner “did you test the site before going live?”, the answer is always “of course we did”. The problem is that it’s the wrong question. The right question is “did you test the site by accurately simulating real users performing both normal and unusual tasks, at and above expected volumes?”. For instance, if the goal was to simulate 5,000 concurrent users, a tester may respond that they tested for 5,000 “page views”! This is when language matters, since 5,000 page loads rarely equals the activity of 5,000 real users. In fact, it likely represents only a fraction of the target volume. By simply substituting page views or transactions for accurately simulating the activity of users on the site they almost certainly won’t reach the expected goal set by the business owner.

Another method of cheating the system is to adjust the timings of test scenarios. This practice is widely used by the testing community, primarily due to the cost of hardware and software when using traditional testing tools. For example, if buying a plane ticket online typically takes about 10 minutes, a clever tester may reduce the timing of this process in the test to just 1 minute. This, on the face of it, allows many more “users” into the system, but it doesn’t accurately simulate real world conditions. Finally, many leading edge companies are beginning to realize that the only way to accurately test a site is by including production testing. Testing only in the lab, and then extrapolating the results for the production environment, leaves far too many variables unaccounted for in the complex deployment environment that is the web. Again, cheating the testing system.

Whatever the reason for the cheating (lack of time, people, or resources) we must change this game now to maintain a high level of consumer confidence and continue to expand the growth of online commerce.


Sales and Delivery

This post has nothing to do with testing. Just blogging my frustration.
Prospect calls company and asks to be connected to the sales team. The conversation is as follows:
Prospect: Hello, Can i speak to Mr.Xyz, VP sales?
VP Sales: Yes speaking, what can i do for you?
Prospect: We are interested in your IV & V services. Can you share your offerings?
VP Sales: We offer everything. Name it and you will have it.
Prospect: We have a testing requirement which is slightly complex but
(VP Sales interrupts)
VP Sales: There is nothing complex to us. We have 18000 man years of experience in testing.
Prospect: Ok, we need 2 resources at onsite for this requirement. The requirement is, we have 2 african lions suffering from digestion issues. We need a 2 testers(1 for each lion) to come open the lion's mouth some time after every meal and test if the food has digested properly.
VP Sales: Hmm..this is little challenging but i am sure we can do it. Hold on for a second. Let me pull in my Delivery Manager
VP Sales: So ABC, you are there right..ok..ABC is our Global Delivery Manager taking care of local operations. ABC, they have a simple requirement to test. I am sure we can do it. Can we talk about the rates?
ABC: What's the requirement?
VP Sales: Test if digestion is happening correctly..That's all
ABC: But how do we..
VP Sales: (Interrupts) You remember a similar case we did in Europe 2 years back..The customer was so happy with our "Blackbox Digestion Testing"
Prospect: Can you give us the reference?
VP Sales: No Problem. Once the contract gets signed, we'll give the reference.
Prospect: And can you get onsite resources? We also need substantial evidence on your capabilities.
VP Sales: Just a second..(Puts the line on mute). ABC, can we have 2 testers go to the nearby zoo tomorrow?
ABC: Yeah but why?
VP Sales: (Releases mute) To give you confidence what we can do is have a video shoot of the digestion process done by our testers which we can share over Webex or GoToMeeting. ABC has agreed to do it tomorrow evening itself. What would be your convenient time?
Prospect: Sounds like a plan. We can do it at 9 PST.
VP Sales: Allright cool, looks like we are all on the same page. Also, please note that the video shoot PoC is free of cost for you. Can we talk about the rates?
Prospect: Our current tester from the vendor side has lost 1 hand since he joined and now we have started paying them 2500$. We can start from 1500$ in your case.
VP Sales: I suggest that you don't pay us until we lose both the hands which in a way would prove the value of our services
Prospect: Excellent. Look forward to seeing the video shoot tomorrow.
VP Sales: I will in fact try my best to send it tonight. Convey my regards to the lions. I am sure they will enjoy our services. In case you have any questions or need more discounts anything, please let me know. Thanks. Oh sorry..forgot to tell you something. If you add more than 4 lions, we give a Project Manager free of cost as value add. Keep that in mind.
Prospect: Sure Ok. Have a great day.

Test Environment-Few questions..

Since, I couldnt 'comment' the following due to limitation in no.of characters (for blogspot comments) I am posting this as a separate post for Vasu's problem.

I am trying to answer with my experience, there may be huge no of permutations / questions for Vasu's question about Test environment.
IMHO, like any other QA Problems, there is no 'silver bullet' for this kind of questions.

There are (i) Number of questions to be asked to client (ii) Number of factors to be considered for analysis after getting the answers from client

Also, we may need to bang the client with questions repeatedly

Before going to client , QA person need to conside the nature of the project

Factors/Questions to client

(1) How end customer is going to use this application?
(2) What are the following are supported
    2.3.Server Operating Systems
    2.4.Client (Desktop operating systems)
    2.5.Replica of Databaseses/Webservers for load balancing
    2.6.Router,Firewall (and their required combo of installation, something like having DMZ, webserver inside firewall, outside firewall etc...)
    2.7.So on...............................
above said factors depends upon the project, what I said is in general, based on my experience.
(3) If the application is a project and dependent on heavy database transactions, then
    3.1 Is it possible to cut the production database periodically to QA database?
    3.2 If it is any nightly run, then situations dealing with deadlocks, or issues with integrity   
of data across databases/tables?
    3.3 Optimum setup so that loading the QA environment should take minimum time
    3.4 Size of Database
(4)Factors to consider for VMWares,
    4.1.The VMWare should be used for relatively less performance high functional testing
    4.2.High end base systems with large no.of VMWares are needed if and only if the testing  team  uses all the combinations of OSXBrowsersXServesXDatabase constantly
    4.3.Capacity of base machines are very important since, the performance of VMWares are based on the base system's capacity.
(5)Availability of Softwares including support
    5.1. Since QA deals with all OS/Servers, having a licensed version of each component is important, in one of my projects, we used solaris like OS without proper license, and all of a  sudden it went down, without support, it was a night mare of ITservices (&  QA)
    5.2. In case of evaluation copy of softwares, need to be informed to client beforehand (i.e it will expire any time and there wont be much support)
    5.3. Information about Patches/Service packs should be made clear (especially non-windows OS,Databases, browsers)
(6)Information about customer usage
    This is very important, but we may rarely get full information. Again without nature of project no body can get clear requirements on test environment.
(7)Prioritise combo
    If I.E 8 X windows XP X Oracle X Weblogic in Linux AS combo is most frequently used combo, then need to have additional space for that combo.
(8)Usage of additional softwares if, any
    e.g Share point,XML editor etc.
(9) Relevance of Test environment w.r.t Test data
    Test data is the fulcrum for test environment, if the test data doesn't suit any of your environment combo, eliminate the environment and declare before hand.
(10) Calculation of optimum usage of combinations, switching on and off the VMWares etc.

(11) How we are going to deploy new build in test environment, in different combo, how difficult it would be (e.g installing websphere in Solaris Risc environment, deploying new build in HP Unix etc) ,

corresponding expertise in QA team / IT services team.

(12) SPOC for test environment from IT services.

(13) Howmuch the  QA environment is in sync with dev and production environment (this is very IMPORTANT,depending upon the nature of the project)

(14) Accesability of QA environment by QA team, developers, IT services team from their desktop, from US (client environment) in case of trouble shooting and network setting accordingly.

(15) Security of the environment (like Anti virus softwares,Authentication,Authorisation, internet access from QA environment etc)

(16) Remote Desktop Access Policy, Domain Policy (Test environment is in separate domain? Child domain?, from QA's Dev's Desktop). In case of data transfer from production to QA database, the connection should be possible w.r.t domain policy. Also, admin permission for QA people over machines to install anything they want including automation tools & FTP tools.

(17) A good visio diagram , if needed.

(18) Client Sign off for version 1 of test environment doc :-)

Pheww...................... As you see it, this is very very generalised reply.

Forgot to add, in case of large test environments,

Racks,Switches, no.of monitors Vs VMwares plays a huge role and early planning will avoid nasty word wars!

Test Environment

I was recently involved in a project where the client asked me, "What kind of Test Environment do you think would be ideal for our application?". Frankly, i was speechless for a few minutes and then responded saying, "It depends on how your customers access the application". I know i was not correct but apparently i was not confident about what the right answer was. He again said, when i mean environment, i want to know what kind of Test environment you need for testing. How many servers do you need, is there a need for staging, how many VM's do you need, what kind of infrastructure(hardware + software) you need to replicate customer issues etc. I was not able to confidently answer him and hence asked for some time to revert.

Being in this league(QA) for a while now, i never had the opportunity to setup test environments or rather i never tried to gain knowledge in setting up test environments. After this experience, i realized how important it is for a Senior QA person to help clients in such cases since they completely rely on our experience to help serve their customers better. Request readers to share your thoughts.

Which Generation am I?

Recently when I was conversing with my friend's cousin (who is around 20 years old), I was telling that I couldn't use Facebook, instead I like orkut for social networking. I am of the opinion that  Orkut's UI is better than Facebook's for connecting with friends. His reply was that I belong to previous (web) generation, though not technically (Web 1.0,Web 2.0) but on usage of web.

I was pretty displeased on hearing it.. but on thinking on it, I started listing what are all the sites I do not use or not explored , the list seems to be growing,

Facebook (albeit I am in, not feeling easy)

list is growing.. all the above mentioned are social networking/ news / book mark sharing sites.. Rarely I use ecommerce sites to buy things..

All I use is Orkut, LinkedIn,blogger,youtube,Google maps ...hmm..and obviously gmail.

I read news @ google news, chat via yahoo msnger.

Am I belong to previous gen?? What about you?

Rajnikanth and Test Coverage

"I know that my product lacks test coverage and thats why i have hired you as consultant. Can you increase test coverage in 2 weeks?"

"Don't tell me that i should change my automation framework to get better returns. Can you suggest improvements to the existing framework so that we get better coverage asap?"

"Your audit report says there is lack of communication between manual and automation teams. I disagree with you. How does test coverage suffer if they don't communicate. Can you explain?"

These are comments from some of our customers recently and all they are looking for is a pill that will solve their test coverage issues. Their products suffer from lack of test coverage for 2/3/5 years but the moment they outsource, they want the outsourced vendor to help them increase test coverage the next day after the SOW is signed.

I am reminded of Rajnikanth movies where he becomes rich in just one song. I wish i can do something similar for test coverage improvement. Any Rajnikanth's out there in our industry to do this? Please revert ASAP.

I wandered thinking how to test the Cloud

Does the title line sound like something that you have heard before? Yes, remember the English lessons you took during high school days. Yes, i have slightly modified the line "I wandered lonely as a cloud" from the poem "Daffodils" by William Wordsworth. So, what is this article about? Is it about introducing readers to English poets? No. This article is about "Cloud Computing", around which there is a huge buzz in the software industry today. Analysts predict cloud computing to form the next big wave in technology infrastructure and are also talking about lot of other terms such as SaaS(Software as a Service), PaaS(Platform as a Service) etc in conjunction with Cloud Computing.

Now, let me refrain from discussing about Cloud computing in detail since i am oblivious of the nitty gritties. However, i have limited exposure to SaaS, which i understand as a model in which software will be centrally located somewhere in the internet(cloud) as opposed to storing them in local machines.The basic idea is cut down the expenditure on hardware/software.

To give a simple example, lets say there are 3 star hotels in Chennai and all have their own software/hardware to manage their services. Now, Software as a Service is a model where a vendor comes and offers to manage the hardware/software for all these 3 hotels from his premises and thereby enables the hotels to use the software on a demand basis through virtualization and relieves from all the issues related to managing their own infrastructure. Certainly makes life easy, isn't it?

Now what are the challenges in the cloud world particularly when it comes to testing?

Configuration and Customization
Each of the hotel's software can have a different workflow and hence the vendor needs to make sure that when the user for "Hotel X" logs in, he is able to view only the screens pertaining to him. Customization should also be looked at from an Integration testing viewpoint

The SaaS provider needs to ensure that the data of all his clients are maintained properly. He cannot afford to mess with the data.

Since software will be used on a demand basis, the charges will be made as per the usage(something like electricity). So, testing also needs to take care of this aspect.

Performance Testing
Clear cut goals with respect to Response Time, Throughput, Maximum load supported should be identified and the same needs to be tested carefully.

Paper Submission for ASTA

I am planning to submit a paper in this year's ASTA (Asian Software
Testing Alliance) International Software Conference 2010
which will be held in Vietnam's Hanoi and Ho Chi Minh cities between the 2nd and 5th of November 2010.

There are 3 categories under which papers can be submitted. People, Process and Technology are the 3 categories under which different headers are available to select for paper submission. Under the "People" category, "Testing Skills" is the title on which i intend to submit my paper. As a prelude to the paper submission, the conference expects the participant to send an abstract summarizing the paper content to them.

Under "Testing Skills" i am planning to write about some interesting areas which i have learnt over the years. I will continue to post more information on the abstract, how my paper is shaping up and the obviously about the areas that i plan to cover in the paper under the title "Testing Skills".

Rendezvous with TesterTested

My company's "Talent Nurturing" team were discussing with me about their plans to hire an external consultant and conduct a 1 or 2 day training session for our Testing DU (Delivery Unit) and during the conversation asked me to suggest some good topics that will benefit the attendees. As someone with the experience of being coerced to attend testing training sessions by my employers in the past, i was not keen to recommend a standard training program where you have the trainer talking about concepts that will never help the tester improve his skills. Some examples are, what’s the difference between Sanity and Smoke testing, contents of test plan, what exit/entry criteria exactly mean, bug life cycle, how to have better communication with the developer, how to know if automation is good for your project and finally asking the attendees to write testcases for Yahoo mail as a exercise. Apart from the trainer (he receives a fat pay cheque for the consulting from the company) no one else really derives benefit from such sessions.

So, i suggested that it would be good if we can introduce our testers to the "Context Driven School of Testing" which specializes in approaches such as Exploratory Testing, Rapid Testing etc. The very fact that the context driven school emphasizes on areas like Lateral Thinking made me recommend it as i thought our testers stand a chance improve their skills to "test" better than what we were currently doing. With the idea being accepted, i was not able to think of anyone else apart from Pradeep Soundararajan, who is leading the context driven testing efforts in India.

I sent a mail to Pradeep explaining the need and he in turn consented. Pradeep also compromised a lot on his pricing since he was keen to share his knowledge and experience with the group to help them become better testers. As expected the session focused on puzzles for testers, lateral thinking etc and also questioned our primary understanding of various testing concepts. For example, we were not able to explain the exact meaning of testing despite the fact that we use the term at least 100 times every day. Pradeep also gave us some exercises to help us improve our testing approach and taught us about better defect reporting methods. He also emphasized on the need to use tools when it comes to testing. No, not automation tools but tools like AllPairs, Perlclip etc which can help the tester expand the coverage of testing and subsequently raise the quality bar.

Overall the session helped us to introspect the way we were doing testing and opened our eyes to newer aspects of testing.

ASTA Testing Conference 2010 Vietnam

Please visit for more details

Vietnamese Testing Board(VTB) is considered to be the #1 Software Testing Community in Asia with more than 1,00,000 members.

Getting outlook email metadata.

I want to get access to an msg file (standalone, outlook message file) and read all of its metadata (like To,cc,bcc, size,from,sent time etc..)

Which language will be useful in achieving this in an easy manner?

In addition, will outlook API can be used? If it is , then any code samples please?

Any help,would be appreciated.

A Myth called Test Automation

In the past 6 months, i have seen a lot of prospects asking only for Test Automation from us. Though we clearly say that without manual testing it doesn't make sense for them to move towards automation, they really don't seem to care. As a services company, we also do not stress on the importance of manual testing again and again for the simple fact that we can lose the prospect from being converted into a customer. But personally, i feel bad that we are showing them the wrong direction and also knowing that the automated scripts won't be reused after a period of time, i feel sorry for them since i am aware that they are not going to reap return on their investment. Test Automation is considered to be the "Silver Bullet" by many companies today for all their testing problems.

Product Companies in India...

I am working at a software product company. Many of my friends are working in product companies. Though I am in Product Company, originally I am from Service Company.

I am seeing a model difference (as everybody knows) between service and product companies in India. Here in product companies, there is a collaborative development model, or in other words collaborative business model. The American company is not treating as client but as a parent, the American workers are treated as peers.

Definitely there may be many thought patterns would have ridden over it, but my opinion is, this is not good for both Indian as well as American wings. Indian wing should its American counterpart as its client, which not only increase the internal competency but also greatly justifies its existence.

As a company, either we have to bravely innovate things or should satisfy our clients/customers... Most of the core customer relationship activities are done in US, Indian companies start enjoying the 'non-pressure' of customer satisfaction... Even if it is created, it lags direction...

Then when Indians can treat their American counterpart as peers? In simple words, over the years, i.e. if process, knowledge, maturity of Indian wing is optimum, obviously the 'Equality' will emerge. Attrition plays an important role here.... Say if an ODC in Bangalore/Chennai is having contract with an American company for the past 6 years and with minimum attrition, then it’s the time for treating 'them' as your 'peers'