How to measure Quality?Muhammad Zeeshan Ali, PMP, PMI-ACP
When we talk about successful projects, good quality is always assumed to be part of it. Just like Definition of Done, the definition of Quality is important. There are many versions of the definition of Quality but for me two main factors define Quality; First factor is how the output is working compared against the standard industry standards and/or processes. The second factor is how the final product satisfies the requirement of the customer.
For a Project Manager, it is the foremost priority to ensure quality. To ensure quality for the first factor, discussed above, I always recommend “Performance Management” system implementation. For the second factor, I recommend focus on the “Acceptance Criteria”. In the discussion I will be focusing more regarding the fist factor and discussing how we can measure quality.
I always put emphasis on Quality as I believe it to be the main element of “Performance Measurement”. Whenever I am designing the Performance Measurement Matrix, Quality is one element which gets the most weightage in the calculation mechanism. The Quality is further defined by some sub-elements;
Bugs are something, which everyone agrees instantly, that inversely defines Quality. The more bugs (issues) a solution has, mens the more effort is required to fix them. To evaluate the magnitude of any bug, I always consider 2 factors; Severity and Stage. By Severity I mean how much impact an issue has as per the categorization of the tester. Generally the standard categories include Critical, High, Medium, Low etc. By Stage I mean at what point during the development lifecycle the issue was identified. For example, an issue that was identified at UAT stage is more lethal and damaging as compared to the one that was identified at QA stage.
The bug score is calculated by multiplying Severity and Server scores. The bug score is always counted as negative against the developer. It can be calculated as positive for QA if found before moving to the UAT stage and negative for both Developer and QA after that stage.
Using the formula, stated above, means that as the severity or the stage of the bug advances, the bug score is also increased drastically.
Rework is simply all the work carried out to fix any issue. As the “Bug Score” increases, the amount of rework also incurs in the same proportion. For example, if a bug was identified at the QA server, then rework will include time to fix the bug by the developer plus the time taken to test the bug fixing by QA. Similarly, if a bug was identified at a much later stage, like UAT, then all the time of the activities to fix the issue and clear UAT stage will be counted as rework.
Just like the “Bug Score”, the Rework is calculated negatively against the responsible persons. For example, if a bug was found at QA server, then the rework score will be negatively counted against the developer and similarly if the bug was detected at the UAT server, then it will be counted against the QA.
Technical Debt is similar to Bug Score in calculation but the difference is that the Bug Score is calculated for a certain time-frame but the Technical Debt is all the open bugs against a resource and remains there till they are closed.
This is to ensure that the open bugs are not swept under the carpet as we move to the next iteration of quality score calculation. The resources often oppose having this score against them especially if they have moved to some other project but the main concept emphasizes building in the quality and any loopholes should be covered as soon as possible. Any customer will agree with me that even a minor issue can sometimes prove to have a huge magnitude under some circumstances. Therefore quality is considered something of the highest priority for the customer.
The enforcement of this score not only generates more awareness for resources to avoid Bugs and Rework at the earliest stage but also creates a healthy competition between Development and QA teams. This also develops a sense of importance regarding doing work the right way.