Blog post series about testing development projects, part 1/11

Articles about planning software testing and managing it.

Point of views:

• Business critical development projects, where IT plays a strong role.
• One or more outside vendors.
• Client’s business personnel as testers. They are business professionals, not software testing professionals.
• Goals for testing:
  1. Make sure that the implemented solution supports the needs of the business.
  2. Make sure the implementation doesn’t endanger the business.

Observations made during testing

During testing, several different observations are logged. For example:

• Software defects
• Master data defects
• Specification defects
• Testers don’t know how to test -> user defects
• Development ideas
• Change requests
• Open issues

Logging every observation, handling them, making decisions and retesting them takes about 2 – 3.5h, depending on methods used. If the processes and tools are good, the handling time is closer to 2 hours. If spreadsheets and emails are the methods used, it takes around 3.5h. And this is only the tasks made in the client’s end. Coding and actual fixing takes always their own time.

Handling defects - how to budget them?

When planning testing, observations may cause trouble: How much time and resources to budget on handling observations?

Estimating the number of defects is quite difficult. If there is history available, you can make predictions. If there’s no history, you can estimate the numbers with vendors.

When I go to a new client, I use these basic formulas as a guideline for my estimate:

• 1.3 observations / coding day
• 2.2 observations / use case, tailoring
• 15 persons, four week testing, there are 157 observations on average.

Based on the project’s complexity, nature and the quality of the management, I give multipliers to these estimates. For example, if there are multiple vendors and lots of integrations in a project, there will be more observations.

An Example:

In this ERP-project, we implemented a new module to a new business area. Before testing started, I estimated that there will be around 157 observations created during testing.

-> Client’s and vendor’s project manager commented: “This is packaged software, there is no way there’s going to be that many observations”.

The final result: We made 204 observations during testing:

• 5 change requests
• 185 defects
• 14 open issues, changes to the business.

I had budgeted 390 hours for handling observations. The actual time used was eventually 500 hours. The reason for this: One area’s execution was not complete when the vendor released it for testing.

Multipliers for the observations

The biggest multipliers on the number of observations I give to the projects that have project managers saying, “It’s all going well”, without any statistics on the statuses. “Development Project Status: Everything is Ready, Except a Few Small Things” If the specifications and tracking executions are not defined, it’s difficult to know what things are ready and what are not -> Some of the testing observations are findings of an incomplete execution.

The second factor that boosts the multipliers is cutting the expenses on training, when the project takes on a new software. If the client’s project group doesn’t know the software that’s being tested, the quality of the testing cannot be great.

It’s not always a bad thing, when there are many observations. During agile development, where the client is close to the people executing the project, the solutions are bounced between them. Whether this is a part of testing or execution, is something that can be decided by the project.