In the lean approach, product defects result in quality-level waste due to the time and resources spent to find, fix, and retest them.

To eliminate this kind of waste, principles of global QA and testing organizations, such as ISTQB® (International Software Testing Qualifications Board), should be applied within the product development life cycle:

Testing shows presence of defects.

Testing shows the presence of defects but cannot prove zero defects. Testing efforts reduce the number of undiscovered defects on the product but cannot claim zero defects.

In the lean approach, the balance between time to market and product quality should be well established. Although not desired, a product may even go live with low-priority defects in case time to market is highly critical. This usually happens at the first iteration of a project in order to release a product earlier than competitors. These defects can then be fixed at later iterations.

Exhaustive testing is impossible.

Testing every single part of a product is not possible due to time and resource limitations. In the lean approach, testing efforts should focus on high-risk areas instead of exhaustive testing. Risk prioritization should be done according to potential impact and likelihood levels. This way, the waste due to excessive testing efforts can be prevented.

For most of the projects, requirement documents are used as a basis for test cases. However, test cases should not only contain positive scenarios on requirement documents; they should also cover negative test conditions. These negative test conditions can be identified by applying  risk-based  testing techniques, such as FMEA (Failure Mode and Effect Analysis).

Applying black-box test techniques, such as equivalence partitioning, boundary value, and combinatorial analysis, helps to achieve enough test coverage with minimum test data. These techniques prevent waste by eliminating the need for excessive test data.

Early testing

The product development life cycle is like a river with requirements at its source. If you can’t clean the river at its source, you will have a dirty river flowing down the hill. A reactive rather than a proactive approach to clean the river will increase the costs and risks exponentially.

Due to the gravity of  this situation, quality assurance  teams should be more proactive and start testing as early as possible during the product development life cycle. Resolution of early defects prevents waste due to costly fixes at later stages. Without waiting for the development of the final product, testing should be  started  early  by  reviews  on  requirement  documents  and  user  tests  on

prototypes. By doing this both functionality-and usability-related defects can be easily found and fixed at the early phases of the project.

Pesticide paradox

If the same kinds of tests are repeated again and again, the same set of test cases will no longer help to find any new defects on the product. To overcome this “pesticide paradox,” the test cases should be regularly reviewed and updated. Thus the waste due to ineffective testing efforts will be prevented.

To ensure enough coverage, QA teams should form the test basis according to requirements and business rules on analysis documents and combine them with risk-based test conditions.

During phase one of the CEC mobile application project, the “View and Order a Product” test case was defined as follows based on main, alternative, and exception scenarios of the relevant use case document:

As shown on the above example, use case documents that have detailed scenarios form a good basis for test cases. When agile methodology is applied, however, user stories and their acceptance criteria do not have that level of detail. To ensure enough test coverage, QA teams on agile projects should work in high collaboration with product owners and other business unit representatives during the tests.

Automated Testing

In lean’s iterative approach, adding new components to a live product without impacting its already released parts is like changing the tire of a car while it’s moving.

This iterative development approach requires comprehensive regression testing of the product components that were already released in previous iterations. To ensure enough test coverage, the parts with both direct and indirect integration points should be retested. This brings a huge amount of  extra  testing  effort. Testing the same components manually and repetitively is both time-consuming and impractical. To make this process more efficient, automated regression test suites can be created.

However, automation itself is a challenging project. Project managers should consider the time needed to implement automation tools as a separate risk item in every project. They should not use an automation tool for the first time in a time-sensitive,  high-priority  project.  The  team  should  focus  on  the  project, instead of allocating its limited time for tool implementation. Project managers should remember that upper management always takes account of the score but not how the team played during the game.

In an automated regression-testing process, test procedures are captured as test scripts at the first test cycle and then run automatically in following test cycles. However, test automation is not a magic way of finding defects by pressing a single button. Most of the time technical problems arise on test automation tools during test script generation. Fixing these problems requires advanced technical skills. For this reason test automation should always be the responsibility of technical QA teams rather than business analysts.

In some circumstances manual testing becomes more efficient than automated testing; it takes much more time to generate automated test scripts compared to running test cases manually. Especially in time-sensitive, fast-track projects, this results in a weird situation of coding around bugs instead of finding and fixing them. Project managers and QA managers should consider this issue as a project risk. They should mitigate this risk by determining the right level of test automation.

Shelfware

In recent years, we have started to see a different “ware” category (like hardware and software). This category is called shelfware. Shelfware represents the automation software that sits on the shelves of the company without being used by any single person.

Shelfware causes a huge amount of tool-level waste. At some public companies, high license and support costs paid for useless shelfware has even become an issue investigated during internal audits.

To prevent shelfware situation, managers should know that automation tools are wizards but not magicians. They have limits. They can only help the project team do its work in a more convenient way by automating some of the tasks, but not all of them. If the organization’s QA process maturity is at a good level, automation makes it better; otherwise, automation may even make  it  worse. Hence managers should first focus on improving their QA and testing skills and then give the go-ahead for the automation initiative. If the team has only limited

knowledge of test methods and techniques, automation will only bring extra problems rather than benefits.

UAT Is the Last Filtering Point of Defects.

Even with the existence of a separate QA team, business analysts should be in charge of coordinating and guiding business units during UAT (user acceptance tests). UAT is the final stage for validating requirements and ensuring the fulfillment of business needs of the new product. In case UAT is not conducted effectively, end users will find defects after the release, and this will result in money and reputation loss.

To increase the effectiveness of UAT, users should first conduct experience- based tests without running any test cases. Afterward another UAT cycle should be organized to ensure enough test coverage by running UAT cases. Business analysts can prepare UAT cases by simplifying the test cases generated by QA teams.

Normally user training should be provided after UAT if the new product is replacing an existing one. Users will be able to test the product in a more independent and unbiased way. But, if the product is a new one, user training should be conducted before UAT. Otherwise users will have difficulties in using the product, which is completely new to them, and this will result in lower test coverage ratios and longer UAT durations.