Exploratory Testing and Session Based Test Management

One of the customers we recently signed had a very tight timeframe for release and was relying on our team to help him deliver the product with good quality. There was not much time for Knowledge Transfer and our team was expected to pitch in for day one. I suggested Exploratory Testing to the customer and to measure the effectiveness of ET recommended SBTM(Session Based Test Management).

Being onsite-offshore model i was not sure how SBTM would work since i had this preconceived image that SBTM works only when our team is sitting with the customer's team at their premises. I prepared a SBTM template to share with the customer which had details of the Charter, Test Notes, Bug Investigation and Reporting, Issues, Opportunity Testing, Charter Vs Opportunity etc.

I shared the template for review with my team before presenting it to the customer. The team felt that such a template is too detailed and the customer may not be interested in that level of detailing. We reworked on the template to include details that would be more relevant to the customer such as Bugs Filed during the day, Clarifications raised etc. Except for the Charter most of the other fields were changed. What started as a SBTM report has now become a Daily status report. :-))

My question to other visitors to this blog are:

Is Session Based Test Management possible in onsite-offshore model or pure offshore model? If yes, how to do it effectively?

Can we convince customers about Charter Vs Opportunity Testing?

I understand that ET(Exploratory Testing) can be effectively performed using skilled testers but unfortunately in many cases you don't get that luxury. How to perform ET using a team that is not very skilled?

How do you show product knowledge acquisition as part of ET to the customer?

Testing for the Cloud

The buzzword seems to be "Cloud Computing" now.

Steve Jobs spoke about iCloud few weeks back. Infosys CEO Kris Gopalakrishnan says "Take Advantage of Cloud Computing". About an year back Larry Ellison asked "What the hell is cloud computing" but he seems to be eating his own words now. Today, Oracle offers two sets of cloud-enabling products and technologies: some of its core technologies like grid computing and middleware, still bearing their traditional names; and a second and newer group of tools to which Oracle has attached the very term—cloud computing.

So cloud computing seems to be the next big thing and i look forward to people calling themselves Cloud testing experts in the near future. Probably, IV&V companies might come up with test automation frameworks designed exclusively to meet the needs of cloud and there could be lot of focus on Performance and Security testing since data sharing on cloud environments would increase the need for performance and security testing. Let's wait and see what cloud has to offer for the Testing community.

Whats happening to Estimation?

There used to be a time when every customer wanted the vendor to give detailed estimates for testing their application. I have seen estimates planned for 2-3 years from the engagement start date and that'll include resource ramp up/down plans, cost planning based on resource plans etc. All this seems to be changing now.

Some of the recent customers we have signed up are not interested in Test Estimation at all. They want us to start with a Proof of Concept, evaluate based on the results of the PoC and if the results are rosy, go ahead and start the engagement. No clear milestones and deliverables defined, no visibility on resource addition/deletion, no Test Plan/Test strategy identified for the application etc. Terms like Function Points, Testcase Points, Cocomo etc are no more heard in the vicinity.

This trend is definitely not good. Test Managers and Leaders must emphasize the need for estimation and planning to prospects/customers.

Point to Ponder

This is a list of approaches, styles, and philosophies in software development. It contains also software development processes, software development methodologies and single practices, principles and laws.
Agent-oriented programming
Agile software development
Agile Unified Process (AUP)
Aspect-oriented Programming
Behavior Driven Development (BDD)
Big Design Up Front (BDUF)
Blind Men And Elephant Approach (BMAEA)
Brooks's law
Cathedral and the Bazaar (see also Release early, release often)
Code and fix
Cone of Uncertainty
Constructionist design methodology (CDM)
Continuous integration
Control tables
Conway's Law
Cowboy coding
Crystal Clear
Dependency injection
Design-driven development (D3)
Design Driven Testing (DDT)
Domain-Driven Design (DDD)
Don't Make Me Think (book by Steve Krug about human computer interaction and web usability)
Don't repeat yourself (DRY) or Duplication is Evil (DIE) or Once and Only Once (OAOO), Single Point of Truth (SPoT), Single Source Of Truth (SSOT)
Dynamic Systems Development Method (DSDM)
Evolutionary prototyping
Extreme Programming (XP)
Feature Driven Development
Good Enough For Now (GEFN)
Hollywood Principle
Inversion of control
Iterative and incremental development
Joint application design, aka JAD or "Joint Application Development"
Kaizen
Kanban
KISS principle original (Keep It Simple and Stupid), derogatory (Keep It Simple, Stupid!)
Lean software development
Literate Programming
Microsoft Solutions Framework (MSF)
Model-driven architecture (MDA)
MoSCoW Method
Open source
Open Unified Process
Parkinson's Law
Quick-and-dirty
Rapid application development (RAD)
Rational Unified Process (RUP)
Release early, release often (see also The Cathedral and the Bazaar)
Responsibility-driven design (RDD)
Scrum
Separation of concerns (SoC)
Service-oriented modeling
Software Craftsmanship
Software System Safety
Solid (object-oriented design)
Spiral model
Structured Systems Analysis and Design Method (SSADM)
SUMMIT Ascendant (now IBM Rational SUMMIT Ascendant)
Team Software Process (TSP)
Test-driven development (TDD)
Two Tracks Unified Process (2TUP)
Unified Process (UP)
Unix philosophy
V-Model
Waterfall model
Wheel and spoke model
When it's ready [1]
Win-Win Model
Worse is better (New Jersey style, as contrasted with the MIT approach)
You Ain't Gonna Need It (YAGNI)

Source: Wikipedia

Wonder how testing happens in each of these?

PDF attachment in hotmail.

I didn't notice this issue for a long time (albeit I am a long term hotmail user, but I seldom use it for 'serious' purpose (like job, getting bills etc). Of late, I tried hotmail for some of my service billings (like broadband, bank etc.)

 Quite interestingly, I am NOT able to open any PDF attachment received via hotmail account. I tried in Firefox 3.6 and I.E 7.I have to forward to my gmail account to open it.

 

Incredulously, I googled and yep that is expected. You cannot open PDF attachment in hotmail for security purpose! Guyz, this is too bad , may be okay in technical perspective but not in user's perspective, across the globe I strongly believe bills and other important documents are coming in PDF format (with or without encryption) and if those cannot be opened from hotmail.... 

Height of security & irrationality !

Test(Data,Case,Environment)<==>Bug

In my experience, I am repeatedly finding that the proper mapping between
Test Case <==> Test Data <==> Test Environment <==> Bug

acting as a great tool for any tester / test team.

In other words, every bug should have its own test case/test data mapped.
In the same way, every test case (or test run) should have its test environment / test data / bug (if applicable) mapped.

It will help tester / test team in

1.High Rate of Reproducible bugs
2.Making the test process as resource independence
3.Transparency in testing
4.Getting a view on test coverage
5.Getting the Test case coverage ratio (No.of test cases Vs Total number of bugs)

Albeit, it seems to be so naive to read, but one of challenging tasks in any test team (inversely proportional to testing team size).

The problem starts when test team enters into hectic schedule of testing during the project. During and After this high concentrated testing, team gets exhausted for maintaining this mapping and concentrates more on 'after live' issues.


But even after two/three short releases when testing team gets little time for doing this exercise, unless team is highly disciplined, it is very difficult to carry this (mapping) exercise. Test Manager can easily use this mapping as a yardstick to measure the 'discipline' of testing team!

Power of Ticketing

I am not sure whether this practice is followed across companies in industry. At least I am sure that few projects are not following this system.

We are aware of filing Bugs against the AUT (Application Under Test). Sometimes the issues filed in Bug tracking system all are not bugs, but some may found to be a Enhancement (Nice to have feature), some are simply 'Task' PR which is assigned against developers.

This Task PR (meaning that particular task has to be done by Assignee of the bug) can be very efficiently used by Test Manager to keep track of the tasks to be done by Testers.

Imagine in a testing module both Manual and Automation testers are working and at the beginning of the testing life cycle many things are discussed as 'to be done' and the way they are tracked is through emails, task requests in Outlook and simply by "Managers". We can use the bug tracking system for this purpose and Manager can raise 'Task' tickets against Testers for all the tasks he wanted tester to do. Or even tester can themselves can create and assign to themselves.

Examples for those task bugs are

(i) Creating new test cases for the new features (and getting sign off from product management and developers). This issue is considered as 'Completed' only when Peer Review/ Product management Review/Developers review/ Second level review is completed. We can create the workflow in our bug tracking system accordingly.

(ii) Select test cases from manual test case repository in order to automate and get sign off from Automation engineer.

(iii) Automate all the selected manual test cases in a feature and get Peer / Client sign off.

(iv) Finish self review for Performance appraisals.

(v) Verify all the bugs for this release.

(vi) Publish the Performance numbers between last release and this release

(vii) Clean the test cases (delete all the obsolete test cases) ..Criteria => Test cases written in past 3 years.

(viii) Complete the knowledge transfer session (this task issue is considered to be completed only if the person who is getting KT has given the reverse presentation and signing off the documents)

(ix) Do 5 interview before 30-Feb




and many more.

All these are treated as open Bugs and considered as important criteria in testing signoff of the particular release.

We can create separate areas in bug tracking systems, such as "Manual Testing Work" , "Automaton Testing work" etc This is analogous to IT help desk ticket but internal to testing team. How diligently we follow this reflect the success.