TestingGuidance Wiki & Documentation Rss Feed

Updated Wiki: Documentation

rburte — Tue, 17 Nov 2009 00:08:08 GMT

Acceptance Test Engineering Guide

Volume I: Thinking about Acceptance

Preamble

Part 1 - Thinking About Acceptance

Part 2 - Perspectives on Acceptance

Appendices

Volume II: Managing Acceptance

Test Processes

Acceptance Test Phase (Test Last Acceptance Testing)
Incremental Acceptance Testing
Acceptance Test Driven Development
Regression Testing

Styles of Testing

Exploratory Testing
Script-Driven Testing

Planning Practices

Project Chartering
Customer Proxy Selection
Risk Assessment
Test Strategy
Test Planning
- Planning Test Automation
- Test Estimation - Testing
Test Outsourcing
Done-Done Checklists

Test Management Practices

Plan-Driven (Cycle-Based) Test Management
Session-Based Test Management
Test Status Reporting
Paired/Collaborative Testing
Assessing Test Effectiveness
Test Asset Management
Test Evolution, Refactoring and Maintenance

Bug Management Practices

Bug Management System
Bug Backlog Analysis
Bug Triage

Updated Wiki: Documentation

rburte — Thu, 12 Nov 2009 22:57:44 GMT

Acceptance Test Engineering Guide

Volume I: Thinking about Acceptance

Preamble

Part 1 - Thinking About Acceptance

Part 2 - Perspectives on Acceptance

Appendices

Volume II: Planning and Managing Acceptance

Step1

Updated Wiki: Home

gmelnik — Thu, 12 Nov 2009 04:38:03 GMT

Update: Release Candidate version of Volume I of the Acceptance Test Engineering Guide (Oct 26, 2009) is released

Download the guide

http://testingguidance.codeplex.com/Release/ProjectReleases.aspx?ReleaseId=35058#DownloadId=89573

The Team

Main authors:
- Grigori Melnik (program manager at Microsoft patterns & practices)
- Gerard Meszaros (agile coach, test automation expert and author of xUnit Test Patterns - Refactoring Test Code)

Contributors and production:
- Jon Bach (professional test strategist)
- Michael Puleio (developer at p&p with a passion around testing)
- Rohit Sharma (tester at p&p)
- Hakan Erdogmus (applied researcher at Kalemun Research Inc.)
- RoAnn Corbisier (technical writer and editor at p&p)
- Dennis DeWitt (technical writer with Linda Werner & Associates Inc.)
- Tina Burden (editor)
- Richard Burte (production specialist)
- Veronica Ruiz (graphic designer, CXR Design)
- Members of the Advisory Board and invited reviewers
- You, the community. We want to hear from you via the Discussion forum here, the Issue Tracker, and feedback comments on what we post. We will listen, even if we do not agree with the recommendations or comments.

Project Description

The Acceptance Test Engineering Guide will provide guidance for technology stakeholders (developers, development leads, testers, test leads, architects, etc.) and business stakeholders (managers, customers, end users, etc) on the discipline of acceptance testing.

Why acceptance testing?

patterns & practices has produced just a few guides related to testing (including performance testing, security testing of web apps, and testing of .NET application blocks). However, we hear a lot of requests from our customers for guidance on testing and test strategy in general, and also guidance on every type of testing you can think of. Based on this customer feedback, and a look at what guidance was available, we determined that acceptance testing was the next area to invest in.

What is acceptance testing?

Working definitions for a number of terms are available in our Draft Glossary. The current definition that is framing our work and discussions is:

Acceptance Testing: Planned evaluation of a system by customers/customer proxies to assess to what degree it satisfies their expectations.

We are open to suggestions on this and other terms. Please leave comments on the Draft Glossary page.

What are we producing?

This guide is the first in the series of three dedicated to acceptance testing and requirements engineering:

Acceptance test engineering guide
Acceptance test automation guide
Tool support for acceptance test-driven development.

The first guide will cover the discipline of acceptance testing from several perspectives and contexts. It will provide models, heuristics and a set of actionable job aides rooted in a sample app. The focus is on:

How to Plan for Acceptance Testing
What Kinds of Acceptance Tests to Run
How to Create and Run Acceptance Tests
Defining What “Done” Means
How to Justify Your Approach

What types of things you can learn in the guide?

If any of the following goals apply to you, you will want to check out the the guide.

AcceptanceTestEngineering_Announcement.pdf

Fill out p&p assets usage survey (NEW!!!)

Updated Wiki: Vol1-Appendix-C

rburte — Sat, 07 Nov 2009 02:23:15 GMT

Appendix C - Glossary

Non-Functional (Attributes, Requirements and Tests)

Aliases: Para-functional, extra-functional.

Product Development Team

The team or organization that includes the people who have the technical skills to develop the product. Includes anyone involved in developing the product, not just software developers.

Product Owner

The person responsible for the business success of the product including ensuring that the product is fit for purpose.
Aliases: Customer

Sequential development

Aliases: Waterfall, plan-driven, tayloristic, staged, phased

Whole Team

The team consisting of everyone involved in defining, building and verifying the product or solution. Includes the Product Development Team, the Product Owner and anyone supporting/helping the Product Owner (including interaction designers and acceptance testers) or the Product Development Team (including documentation writers, architects, etc.)

Test Plan

A more detailed (than the Test Strategy) plan for how testing will be performed.

Test Strategy

The overall approach to how the data required to make the readiness and acceptance decisions will be gathered in a cost-effective manner. Includes the decisions around what kinds of testing to perform, what kinds of test automation will be used, where testing will be done, at what point in the project, etc.. Decisions that have large impact on the cost and duration of the acceptance process belong in the test strategy.

Acceptance Test Synonyms

Term	Introduced/Used in
“acceptance tests”	e.g. FDA [121]
“functional tests”	Beck, Extreme Programming Explained, 1/e [11]
“user acceptance tests”\|
“business acceptance tests”\|
“customer tests”	Jeffries [61],Beck, Extreme Programming Explained, 2/e
“customer-inspired tests”	Beck, Extreme Programming Explained, 1/e [11]
“story-tests” and “story-test-driven development”	Kerievsky [91]
“specification by example”	Fowler [41]
“coaching tests”	Marick [111]
“examples”, “business-facing example”, and “example-driven-development”	Marick [101]
“conditions of satisfaction”	Cohn [21]
“scenario tests”	Kaner [81]
“keyword-driven test”	Kaner, Bach, Pettichord [7]]
“formal qualification tests”	e.g. DOD [31]
“system tests”	e.g. IEEE [51]

References

Beck, K. Extreme Programming Explained: Embrace Change, 1/e. Addison-Wesley, Boston, MA, 1999.
Cohn, M. “Do-It-Yourself”, Better Software, 7(9): 18–22, 2005.
Department of Defense. Military Standard Defense System Software Development DOD-STD-2167, section.5.3.3. Online: http://www2.umassd.edu/SWPI/DOD/MIL-STD-2167A/DOD2167A.html.
Fowler, M. “Specification by Example”. Online: http://www.martinfowler.com/bliki/SpecificationByExample.html
Institute of Electrical and Electronics Engineers. IEEE Standard Computer Dictionary: A Compilation of IEEE Standard Computer Glossaries. New York, NY: 1990.
Jeffries, R. “What is XP?” Online: http://www.XProgramming.com/xpmag/whatisXP.htm
Kaner, C., Bach., J., Pettichord, B. Lessons Learnt in Software Testing : A Context-Driven Approach. John Wiley & Sons, New York, NY, 2001.
Kaner, C. “Cem Kaner on Scenario Testing: The Power of ‘What-If…’ and Nine Ways to Fuel Your Imagination”, Better Software, 5(5):16–22, 2003.
Kerievsky, J. “Storytesting”. Online: http://industrialxp.org/storytesting.html
Marick, B. “Example-Driven Development”. Online: http://www.exampler.com, and http://www.testing.com/cgi-bin/blog/2003/09/05#agile-testing-project-4
Marick, B. Agile Acceptance Testing Workshop Report, XP/Agile Universe 2002 Conf. Online: http://www.pettichord.com/XP_Agile_Universe_trip_report.txt
FDA General Principles of Software Validation http://www.fda.gov/cdrh/comp/guidance/938.html#_Toc517237955

Updated Wiki: Vol1-Appendix-B

rburte — Sat, 07 Nov 2009 02:14:21 GMT

Appendix B – Key Points

The chapters in Part II described the process leading up to the acceptance decision from the perspectives of the Business Lead, Product Manager, Test Manager, Development Manager, User Experience Specialist, Solutions Architect, Enterprise Architect, Operations Manager, and Legal. For convenience, we summarized the key points here. There are Key Points common to all roles and others specific to each individual role.
The following table lists Key Points that are role specific and what roles they are specific to.
<The table will be updated and available on testingguidance.codeplex.com>

Figure 1 Key Points specific to individual roles

Key Points Applicable to All Roles

Treat acceptance as a process not an event: Appreciate the fact that acceptance testing will not be a single discrete event, but a process with a distinct start and end.
Define the scope of testing: Have a clear understanding of what will be tested by the supplier and what you are expected to test. Typically, supplier is expected to do thorough (e.g.) testing of all capabilities and quality characteristics (such as performance, reliability, usability etc.) before handing off system for acceptance testing. Does the supplier have the environments for integration testing without in-house systems?
Delegate decisions: If you don’t have enough time or understanding to take part in the project, assign a proxy to represent you and have a clear understanding of the kinds of decisions they can make with/without consulting with you.
Delegate work/testing: Have a clear understanding of your (organization’s) abilities w.r.t. testing skills. Contract other parties to help you test if necessary. But don’t underestimate the effort involved in managing the testing outsourcer as they will need to learn your business domain and requirements before they can be effective.
Understand the test environments: Understand where you will test. Be clear on how you’ll get the new release candidates delivered
View tests as requirements: Acceptance criteria/tests should be articulated and prepared before the software is built and shared with vendor as a way to flesh out the details of the requirements.
Estimate acceptance effort & duration while realizing it is difficult to estimate: Estimating the effort & duration of acceptance testing is hard; time-boxing the final acceptance testing cycle is common but requires past experience to come up with reasonable duration/effort. If in doubt, guess high and allow for several cycles of acceptance testing to give the vendor time to fix bugs.
Use incremental acceptance testing to reduce uncertainty of estimating: Can avoid some of the uncertainty and most of the risk by doing incremental acceptance testing. Bugs found earlier identify ambiguity in the spec while there is still time to address it and can be fixed long before the final acceptance test phase
Understand the decision-making process: Make sure you have a clear understanding of the decision making process. Other potential decision makers include Test Managers, Security Manager, User Experience/Usability Manager, Architect, Legal Department, etc. Ensure you have agreement on whether they make a decision or provide data or a recommendation to you.
Communicate information flow clearly: Ensure anyone who you expect to provide you with data on which to base your decision understands what data they are providing you and when. E.g. The Test Manager provides test results to you and you make the acceptance decision.
Understand the operational requirements: Make sure you understand the operational requirements for your system. Whether you are selling a software package to customers or selling Software-as-a-Service (SaaS), be aware that whoever operates the software will have operational requirements over and above the functional requirements of the end users.
Clarify decision making vs decision supporting: Make sure you have a clear understanding of whether you are the gate-keeper (acceptance decision maker) or a supplier of information to the Product Manager for her to make the decision (preferred)
Maximize learning: Work with the product owner to ensure that everyone on the team has access to the definition of success in the form of acceptance tests before the software is built

Key Points for Business lead

Treat acceptance as a process not an event: Appreciate the fact that acceptance testing will not be a single discrete event, but a process with a distinct start and end.
Recognize there may be many decision makers: The process may require decisions by many parties each supported by data collection activities. Each party likely has very different requirements and interests; you need to be aware of them and communicate them to the supplier(s) to ensure they address them.
Define the scope of testing: Have a clear understanding of what will be tested by the supplier and what you are expected to test. Typically, supplier is expected to do thorough (e.g.) testing of all capabilities and quality characteristics (such as performance, reliability, usability etc.) before handing off system for AT. Does the supplier have the environments for integration testing without in-house systems?
Delegate decisions: If you don’t have enough time or understanding to take part in the project, assign a proxy to represent you and have a clear understanding of the kinds of decisions they can make with/without consulting with you.
Delegate work/testing: Have a clear understanding of your (organization’s) abilities w.r.t. testing skills. Contract other parties to help you test if necessary. But don’t underestimate the effort involved in managing the testing outsourcer as they will need to learn your business domain and requirements before they can be effective.
Understand the test environments: Understand where you will test. Be clear on how you’ll get the new release candidates delivered
View tests as requirements: Acceptance criteria/tests should be articulated/prepared before the software is built and shared with vendor as a way to flesh out the details of the requirements.
Recognize the choice of process impacts everything: Two key styles of AT: Final Test Phase and Incremental AT. The former concentrates most acceptance activities at the end of the project. Incremental acceptance testing spreads out the work and reduces risk but requires all parties to work in different ways.
Estimate acceptance effort & duration while realizing it is difficult to estimate: Estimating the effort & duration of acceptance testing is hard; time-boxing the final acceptance testing cycle is common but requires past experience to come up with reasonable duration/effort. If in doubt, guess high and allow for several cycles of acceptance testing to give the vendor time to fix bugs.
Use incremental acceptance testing to reduce uncertainty of estimating: Can avoid some of the uncertainty and most of the risk by doing incremental acceptance testing. Bugs found earlier identify ambiguity in the spec while there is still time to address it and can be fixed long before the final acceptance test phase.

Key Points for Product Manager

All Business Lead’s Key Points plus:
Understand the decision-making process: Make sure you have a clear understanding of the decision making process. Other potential decision makers include Test Managers, Security Manager, User Experience/Usability Manager, Architect, Legal Department, etc. Ensure you have agreement on whether they make a decision or provide data or a recommendation to you.
Recognize that complex products and organizations have complex decision making processes: There are several different ways to organize the teams to deal with very large products or products that are the amalgamation of several products/product lines (e.g SharePoint=ServerOfficeIE). The choice should consider the resulting decision-making model (e.g. component teams, feature teams.)
Communicate information flow clearly: Ensure anyone who you expect to provide you with data on which to base your decision understands what data they are providing you and when. E.g. The Test Manager provides test results to you and you make the acceptance decision.
Ensure all suppliers understand users and usage model: Make sure you understand the users and communicate this to both supplier and test organizations.
Understand the operational requirements: Make sure you understand the operational requirements for your system. Whether you are selling a software package to customers or selling Software-as-a-Service (SaaS), be aware that whoever operates the software will have operational requirements over and above the functional requirements of the end users.

Key Points for Test Manager

Determine your test mandate: Make sure you understand whether you are doing readiness assessment or acceptance testing or both. For acceptance testing be sure the testers have realistic tests for the users they proxy. Consider using personas. For readiness assessment it is important to have a good relationship with the development team to help them understand the quality of the product and what will improve it.
Clarify decision making vs decision supporting: Make sure you have a clear understanding of whether you are the gate-keeper (acceptance decision maker) or a supplier of information to the Product Manager for her to make the decision (preferred)
Get agreement from the product owner that the test cases reflect the requirements: Aim to write the acceptance test cases first and provide them to the development teams before the software is built to be used as aides to assist in understanding how the system should behave. This is known as Acceptance Test Driven Development.
Plan a multi-pronged test strategy: Plan for a multi-pronged test strategy that includes incremental testing and both scripted and exploratory tests.
Collaborate with the development team: Aim for an incremental delivery of functionality to support incremental acceptance testing. Work with the development team to design for testability to facilitate automated testing and enabling them to be able to run tests on demand.
Do test automation when feasible and in the way it supports (not dictates!) meeting your test objectives. Consider having a dedicated test automation engineer.
Agree on a definition of what done is: Get the various parties involved to agree on a definition of what done is for software before it goes through each of the quality gates of the gating model and include a checklist of requirements.

Key Points for Development Manager

Perform readiness assessments: All software should be adequately tested by the development team before presenting it to the customer(s).
Agree on the minimum credible requirement (MCR) ahead of time: The product owner, test organization and the supplier should all agree on the MCR ahead of time, that is, before the software is built.
Use acceptance tests to define success: Use acceptance tests based on the MQR to define success before the software is built and ensure that everyone on the development team has access to them and runs the tests as they build the software.
Run the tests as part of readiness acceptance: Run all the tests that the product owner will run and some you’ve defined.
Minimize untested code: Test as you go, ideally incremental testing.
Use testing to prevent bugs: Tests can help prevent bugs being introduced during the development cycle. Include functional testing, component/unit tests, and para-function testing as early as possible.

Key Points for User Experience Specialist

Understand your role: Have a clear understanding with the product owner, product manager or business lead what your role shall be in making the acceptance decision.
Maximize learning: Work with the product owner to ensure that everyone on the team has access to the definition of success in the form of acceptance tests before the software is built.
Define the usability component of the acceptance criteria as early as possible: Work with the business lead or product manager to define the usability component of the acceptance criteria as early as possible.
Address the usability risks as early as possible: The areas with highest usability impact should be built and tested first so that any usability-related issues can be addressed as early as possible in the development cycle.

Key Points for Operations Manager

Understand role in acceptance decision: You and everyone concerned should be clear on your role in the making of the acceptance decision. Are you a data provider to the acceptance decision maker or an acceptance decision maker?
Identify ways to reduce the number of test&fix cycles: Work with the development manager to identify ways to reduce the number of test&fix cycles through early delivery of testable software and incremental acceptance testing.
Define acceptance from the operational perspective as early as possible: Ideally you have clearly defined acceptance criteria before the software is built.

Key Points for Solution Architect

Describe the whole solution to everyone: Communicate the solution in its entirety to the whole supplier team to avoid the “My part works just fine (even if the solution doesn’t!)” mentality.
Ensure availability of acceptance tests: Work with the requirements analysts and testers to ensure that the acceptance tests are available to the supplier team before the software is built.
Educate about tradeoffs: Work with the product owner to help them understand what kinds of functionality is easy to build and what is hard. Help them get the most bang for their money.
Unbundle functionality: Work with the product owner to help them identify the lower-value functionality that could potentially be deferred in case there is a need to trade off functionality for quality or delivery date.
Define SLA metrics
Engage enterprise architects: Engage the enterprise-wide architects (responsible for infrastructure, standards, technology decisions, long-term vision, security, data architecture, etc.) early so they can help you understand what requires their approval before it becomes part of the critical path.

Key Points for Enterprise Architect

Publish architectural acceptance criteria: Publish general acceptance criteria so that all supplier teams understand the rules of the game and what kinds of approvals they need to get from you.
Articulate cross system integration requirements and SLAs: Identify applicable integration requirements and service level agreements. Make sure the supplier team is aware of them and that they are acceptance criteria.
Ensure standards compliance: Verify that products comply with applicable standards.
Encourage reuse: Verify that product teams are aware of and capitalize on opportunities to reuse components and data.
Engage supplier teams early: Engage supplier teams early to ensure that inspections prevent defects rather than find them.
Delegate non-critical decisions: If it is not important to be involved in a decision then bow out and let the project team make the decision.
Keep approval process lightweight: Make the process for seeking guidance and approval as lightweight as possible. Emphasize effectiveness of communication over comprehensive documentation. Avoid offloading work (e.g. more documentation) to the project teams to save you effort if that results in increasing the total work.
Be pragmatic: Be pragmatic as to what progress individual project or product teams are expected to make towards long-term architectural objectives. They are responsible to the product owner, not to you.

Key Points for Legal

Include clear stipulations of any contractual implications: Acceptance typically is a condition for rendering a payment and affects the application of any warranty provisions and potentially any remedies available to the customer.
Discourage one-sided contracts: The contract should encourage both parties to get to a mutually agreeable acceptance as quickly as possible. Contracts that penalize one party often lead to dysfunctional behavior that results in lose-lose outcomes.
Clearly define how much time is allowed for acceptance tests: Acceptance testing is always time-boxed. There is a limited amount of time for acceptance and reporting deficiencies.
Include a clear definition of acceptance criteria: Besides testing performed at the location of supplier, and functional and operational tests performed when the software is deployed to the customer, software system often are required to successfully run with current data in a live operating environment for a period of time, with minimum bugs, before the system is formally accepted.
Know the explicit and implicit contractual conditions: Acceptance can be viewed in terms of explicit provisions specified in the software development contract and the terms implied into the contract by law (which also depends on the geographic jurisdiction).

Updated Wiki: Vol1-Appendix-B

rburte — Sat, 07 Nov 2009 02:14:08 GMT

Appendix B – Key Points

The chapters in Part II described the process leading up to the acceptance decision from the perspectives of the Business Lead, Product Manager, Test Manager, Development Manager, User Experience Specialist, Solutions Architect, Enterprise Architect, Operations Manager, and Legal. For convenience, we summarized the key points here. There are Key Points common to all roles and others specific to each individual role.
The following table lists Key Points that are role specific and what roles they are specific to.
<The table will be updated and available on testingguidance.codeplex.com>
Cannot resolve image macro, invalid image name or id.
Figure 1 Key Points specific to individual roles

Key Points Applicable to All Roles

Treat acceptance as a process not an event: Appreciate the fact that acceptance testing will not be a single discrete event, but a process with a distinct start and end.
Define the scope of testing: Have a clear understanding of what will be tested by the supplier and what you are expected to test. Typically, supplier is expected to do thorough (e.g.) testing of all capabilities and quality characteristics (such as performance, reliability, usability etc.) before handing off system for acceptance testing. Does the supplier have the environments for integration testing without in-house systems?
Delegate decisions: If you don’t have enough time or understanding to take part in the project, assign a proxy to represent you and have a clear understanding of the kinds of decisions they can make with/without consulting with you.
Delegate work/testing: Have a clear understanding of your (organization’s) abilities w.r.t. testing skills. Contract other parties to help you test if necessary. But don’t underestimate the effort involved in managing the testing outsourcer as they will need to learn your business domain and requirements before they can be effective.
Understand the test environments: Understand where you will test. Be clear on how you’ll get the new release candidates delivered
View tests as requirements: Acceptance criteria/tests should be articulated and prepared before the software is built and shared with vendor as a way to flesh out the details of the requirements.
Estimate acceptance effort & duration while realizing it is difficult to estimate: Estimating the effort & duration of acceptance testing is hard; time-boxing the final acceptance testing cycle is common but requires past experience to come up with reasonable duration/effort. If in doubt, guess high and allow for several cycles of acceptance testing to give the vendor time to fix bugs.
Use incremental acceptance testing to reduce uncertainty of estimating: Can avoid some of the uncertainty and most of the risk by doing incremental acceptance testing. Bugs found earlier identify ambiguity in the spec while there is still time to address it and can be fixed long before the final acceptance test phase
Understand the decision-making process: Make sure you have a clear understanding of the decision making process. Other potential decision makers include Test Managers, Security Manager, User Experience/Usability Manager, Architect, Legal Department, etc. Ensure you have agreement on whether they make a decision or provide data or a recommendation to you.
Communicate information flow clearly: Ensure anyone who you expect to provide you with data on which to base your decision understands what data they are providing you and when. E.g. The Test Manager provides test results to you and you make the acceptance decision.
Understand the operational requirements: Make sure you understand the operational requirements for your system. Whether you are selling a software package to customers or selling Software-as-a-Service (SaaS), be aware that whoever operates the software will have operational requirements over and above the functional requirements of the end users.
Clarify decision making vs decision supporting: Make sure you have a clear understanding of whether you are the gate-keeper (acceptance decision maker) or a supplier of information to the Product Manager for her to make the decision (preferred)
Maximize learning: Work with the product owner to ensure that everyone on the team has access to the definition of success in the form of acceptance tests before the software is built

Key Points for Business lead

Treat acceptance as a process not an event: Appreciate the fact that acceptance testing will not be a single discrete event, but a process with a distinct start and end.
Recognize there may be many decision makers: The process may require decisions by many parties each supported by data collection activities. Each party likely has very different requirements and interests; you need to be aware of them and communicate them to the supplier(s) to ensure they address them.
Define the scope of testing: Have a clear understanding of what will be tested by the supplier and what you are expected to test. Typically, supplier is expected to do thorough (e.g.) testing of all capabilities and quality characteristics (such as performance, reliability, usability etc.) before handing off system for AT. Does the supplier have the environments for integration testing without in-house systems?
Delegate decisions: If you don’t have enough time or understanding to take part in the project, assign a proxy to represent you and have a clear understanding of the kinds of decisions they can make with/without consulting with you.
Delegate work/testing: Have a clear understanding of your (organization’s) abilities w.r.t. testing skills. Contract other parties to help you test if necessary. But don’t underestimate the effort involved in managing the testing outsourcer as they will need to learn your business domain and requirements before they can be effective.
Understand the test environments: Understand where you will test. Be clear on how you’ll get the new release candidates delivered
View tests as requirements: Acceptance criteria/tests should be articulated/prepared before the software is built and shared with vendor as a way to flesh out the details of the requirements.
Recognize the choice of process impacts everything: Two key styles of AT: Final Test Phase and Incremental AT. The former concentrates most acceptance activities at the end of the project. Incremental acceptance testing spreads out the work and reduces risk but requires all parties to work in different ways.
Estimate acceptance effort & duration while realizing it is difficult to estimate: Estimating the effort & duration of acceptance testing is hard; time-boxing the final acceptance testing cycle is common but requires past experience to come up with reasonable duration/effort. If in doubt, guess high and allow for several cycles of acceptance testing to give the vendor time to fix bugs.
Use incremental acceptance testing to reduce uncertainty of estimating: Can avoid some of the uncertainty and most of the risk by doing incremental acceptance testing. Bugs found earlier identify ambiguity in the spec while there is still time to address it and can be fixed long before the final acceptance test phase.

Key Points for Product Manager

All Business Lead’s Key Points plus:
Understand the decision-making process: Make sure you have a clear understanding of the decision making process. Other potential decision makers include Test Managers, Security Manager, User Experience/Usability Manager, Architect, Legal Department, etc. Ensure you have agreement on whether they make a decision or provide data or a recommendation to you.
Recognize that complex products and organizations have complex decision making processes: There are several different ways to organize the teams to deal with very large products or products that are the amalgamation of several products/product lines (e.g SharePoint=ServerOfficeIE). The choice should consider the resulting decision-making model (e.g. component teams, feature teams.)
Communicate information flow clearly: Ensure anyone who you expect to provide you with data on which to base your decision understands what data they are providing you and when. E.g. The Test Manager provides test results to you and you make the acceptance decision.
Ensure all suppliers understand users and usage model: Make sure you understand the users and communicate this to both supplier and test organizations.
Understand the operational requirements: Make sure you understand the operational requirements for your system. Whether you are selling a software package to customers or selling Software-as-a-Service (SaaS), be aware that whoever operates the software will have operational requirements over and above the functional requirements of the end users.

Key Points for Test Manager

Determine your test mandate: Make sure you understand whether you are doing readiness assessment or acceptance testing or both. For acceptance testing be sure the testers have realistic tests for the users they proxy. Consider using personas. For readiness assessment it is important to have a good relationship with the development team to help them understand the quality of the product and what will improve it.
Clarify decision making vs decision supporting: Make sure you have a clear understanding of whether you are the gate-keeper (acceptance decision maker) or a supplier of information to the Product Manager for her to make the decision (preferred)
Get agreement from the product owner that the test cases reflect the requirements: Aim to write the acceptance test cases first and provide them to the development teams before the software is built to be used as aides to assist in understanding how the system should behave. This is known as Acceptance Test Driven Development.
Plan a multi-pronged test strategy: Plan for a multi-pronged test strategy that includes incremental testing and both scripted and exploratory tests.
Collaborate with the development team: Aim for an incremental delivery of functionality to support incremental acceptance testing. Work with the development team to design for testability to facilitate automated testing and enabling them to be able to run tests on demand.
Do test automation when feasible and in the way it supports (not dictates!) meeting your test objectives. Consider having a dedicated test automation engineer.
Agree on a definition of what done is: Get the various parties involved to agree on a definition of what done is for software before it goes through each of the quality gates of the gating model and include a checklist of requirements.

Key Points for Development Manager

Perform readiness assessments: All software should be adequately tested by the development team before presenting it to the customer(s).
Agree on the minimum credible requirement (MCR) ahead of time: The product owner, test organization and the supplier should all agree on the MCR ahead of time, that is, before the software is built.
Use acceptance tests to define success: Use acceptance tests based on the MQR to define success before the software is built and ensure that everyone on the development team has access to them and runs the tests as they build the software.
Run the tests as part of readiness acceptance: Run all the tests that the product owner will run and some you’ve defined.
Minimize untested code: Test as you go, ideally incremental testing.
Use testing to prevent bugs: Tests can help prevent bugs being introduced during the development cycle. Include functional testing, component/unit tests, and para-function testing as early as possible.

Key Points for User Experience Specialist

Understand your role: Have a clear understanding with the product owner, product manager or business lead what your role shall be in making the acceptance decision.
Maximize learning: Work with the product owner to ensure that everyone on the team has access to the definition of success in the form of acceptance tests before the software is built.
Define the usability component of the acceptance criteria as early as possible: Work with the business lead or product manager to define the usability component of the acceptance criteria as early as possible.
Address the usability risks as early as possible: The areas with highest usability impact should be built and tested first so that any usability-related issues can be addressed as early as possible in the development cycle.

Key Points for Operations Manager

Understand role in acceptance decision: You and everyone concerned should be clear on your role in the making of the acceptance decision. Are you a data provider to the acceptance decision maker or an acceptance decision maker?
Identify ways to reduce the number of test&fix cycles: Work with the development manager to identify ways to reduce the number of test&fix cycles through early delivery of testable software and incremental acceptance testing.
Define acceptance from the operational perspective as early as possible: Ideally you have clearly defined acceptance criteria before the software is built.

Key Points for Solution Architect

Describe the whole solution to everyone: Communicate the solution in its entirety to the whole supplier team to avoid the “My part works just fine (even if the solution doesn’t!)” mentality.
Ensure availability of acceptance tests: Work with the requirements analysts and testers to ensure that the acceptance tests are available to the supplier team before the software is built.
Educate about tradeoffs: Work with the product owner to help them understand what kinds of functionality is easy to build and what is hard. Help them get the most bang for their money.
Unbundle functionality: Work with the product owner to help them identify the lower-value functionality that could potentially be deferred in case there is a need to trade off functionality for quality or delivery date.
Define SLA metrics
Engage enterprise architects: Engage the enterprise-wide architects (responsible for infrastructure, standards, technology decisions, long-term vision, security, data architecture, etc.) early so they can help you understand what requires their approval before it becomes part of the critical path.

Key Points for Enterprise Architect

Publish architectural acceptance criteria: Publish general acceptance criteria so that all supplier teams understand the rules of the game and what kinds of approvals they need to get from you.
Articulate cross system integration requirements and SLAs: Identify applicable integration requirements and service level agreements. Make sure the supplier team is aware of them and that they are acceptance criteria.
Ensure standards compliance: Verify that products comply with applicable standards.
Encourage reuse: Verify that product teams are aware of and capitalize on opportunities to reuse components and data.
Engage supplier teams early: Engage supplier teams early to ensure that inspections prevent defects rather than find them.
Delegate non-critical decisions: If it is not important to be involved in a decision then bow out and let the project team make the decision.
Keep approval process lightweight: Make the process for seeking guidance and approval as lightweight as possible. Emphasize effectiveness of communication over comprehensive documentation. Avoid offloading work (e.g. more documentation) to the project teams to save you effort if that results in increasing the total work.
Be pragmatic: Be pragmatic as to what progress individual project or product teams are expected to make towards long-term architectural objectives. They are responsible to the product owner, not to you.

Key Points for Legal

Include clear stipulations of any contractual implications: Acceptance typically is a condition for rendering a payment and affects the application of any warranty provisions and potentially any remedies available to the customer.
Discourage one-sided contracts: The contract should encourage both parties to get to a mutually agreeable acceptance as quickly as possible. Contracts that penalize one party often lead to dysfunctional behavior that results in lose-lose outcomes.
Clearly define how much time is allowed for acceptance tests: Acceptance testing is always time-boxed. There is a limited amount of time for acceptance and reporting deficiencies.
Include a clear definition of acceptance criteria: Besides testing performed at the location of supplier, and functional and operational tests performed when the software is deployed to the customer, software system often are required to successfully run with current data in a live operating environment for a period of time, with minimum bugs, before the system is formally accepted.
Know the explicit and implicit contractual conditions: Acceptance can be viewed in terms of explicit provisions specified in the software development contract and the terms implied into the contract by law (which also depends on the geographic jurisdiction).

Updated Wiki: Vol1-Appendix-A

rburte — Sat, 07 Nov 2009 02:09:58 GMT

Appendix A – Organization Stereotypes and Reader Personas

This appendix describes two stereotypical organizations and the people involved in making or providing data for the readiness and acceptance decisions. The people are described as personas.

IT Organization of a Business

XYZ Corp is a large transportation company. Computer software has been integral to the company’s business for several decades. The company has several thousand “applications”. At the high end are systems built by the IT department ranging from large mainframe systems written in Cobol and PL-1 to Web-based applications written in ASP.NET. At the low end there are end user applications built in Visual Basic and Access and spreadsheet applications built by end users scattered throughout the business areas.
The IT department will build the software based on the specifications provided by the business and, after doing its own system testing, expects the business to conduct user acceptance testing. There is no separate QA department but each development team has at least one testing specialist who helps the team test their own software. The IT department’s quality gate process also requires signoffs from the IT Architecture group and the Operational support group before software can be deployed into the Acceptance environment. As a result of recently introduced corporate governance initiatives (such as SOX), company policy requires approval by the Corporate Security department before any new application can be deployed into production.
The IT Architecture department includes a team of data architects whose primary responsibility is to own the corporate data model. The company views the data in the applications as a corporate asset and the data architects have been working hard to manage the duplication of data across them. The IT department also manages the lifecycles of the various IT technologies used in the company. New technologies have to be approved by the IT Architecture group before they can be used and old technologies may be marked for replacement. The Corporate Security department takes an active interest in all new applications and must approve the security measures before any new application can be deployed.

A Product Company

Adventure Works is a product company that sells business productivity improvement products that include both hardware and software. Some of the products are sold as hardware with embedded software and some as software packages that leverage existing hardware products. The company has a separate UxD and QA departments. The UxD department is involved in part of the requirements definition activities associated with designing the product as well as ensuring that the products have a consistent look& fell. The QA department tests the software built by the Product Development group and advises the Product Manager on whether the software can be released to the market.

Reader Personas

Personas Often Found in the IT Organization of a Business

Product Manager or Product Owner

Eric Andersen is a product manager who has been asked to work with an agile development team as the Product Owner for one of the software-only products (a new software release that works with existing hardware supplied by the company and third parties.) Because the company’s products are considered high-technology, Eric is fairly tech-savvy. He has a good understanding of the customer’s requirements but does get a bit lost when the discussion descends to the level of networking protocols.

Key questions Eric needs to find answers to:

What kinds of information do I need from the QA department to decide whether or not to accept the software? What quality criteria must be met? How do I know they have been met?Who is responsible for verifying these criteria are met? Who is responsible for ensuring these criteria are met?
How does the UxD team participate in acceptance testing of the software?
How can the release strategy (multiple releases, alphas, betas, etc.) help me?
What is the process for addressing deficiencies prior to release?

Business Person in the Marketing Division

Lisa Andrews is a business person in the marketing division of XYZ Corp. Lisa is in her late twenties (relatively young for her position in the somewhat conservative company) so she has good computer usage skills but is not by any stretch of the imagination considered “technical”. This is Lisa’s first IT project. Lisa has been asked to be the business lead of the project to replace several existing systems used by the staff in the marketing, sales and contracts departments with a single integrated system using the latest technologies. In this role she will have the final say on whether the system is acceptable to be deployed. She will also be responsible for ensuring that the business benefits used to justify the project are actually realized. She will be assisted on this project by several users of the existing applications as well as a business analyst and a test professional. She will need to oversee this team as it defines the detailed requirements for the new system.

Key questions Lisa needs to answer:

What kinds of testing will my team need to do to help me decide whether or not to accept the software?
How can the release strategy (multiple releases, alphas, betas, community technology previews, etc.) help me?
When should I plan on doing AT? How much time should I allow for AT? How much effort will be involved? How much of staff do I need to do it?
What kinds of testing do I expect the IT department to do before asking my team to do acceptance testing?
What information do I need IT to provide to me about the kinds of testing they have done and how will I use that information in deciding whether or not to accept the software (or even start acceptance testing?)
What information do I need to provide to IT prior to me doing AT?

Operations Department Manager

Chatty Cathy is the manager of the operations department. She is responsible for deploying all server-based applications and keeping them running according to each application’s service level agreement (SLA). She’s been with the company 30 years having worked her way up from being an administrative assistant in one of the business units. Her formal training was secretarial school and she’s learned whatever she needed to know on the job. When she was first moved into I.T. it was as a tester for desktop applications but she quickly moved to up manage the desktop support team. When that function was outsourced, she was asked to take over the operations group. That was two years ago and she has only recently become comfortable with her understanding of how the group functions. She has yet to make any changes to the group’s processes.

IT Security Specialist

Fred Spook is the IT Security Specialist in the Corporate Security department. He’s a former spy (“If I told you which agency I worked for I would have to kill you,” he says with a wink whenever he’s asked.) who moved into the private sector when the first major security breeches were publicized. His background has made him rather paranoid and he delights in skewering the development teams with his arsenal of nasty security scenarios. As a result, most applications fail their first security review and require significant architectural refactoring before passing. The better development teams have learned to approach him for an early “off the record” security audit as soon as their design has stabilized.

Personas Often Found in a Product Company

Test Manager

Joseph Krawczak is the manager of the product verification group at Adventure Works. His team of professional testers verifies that the products built by the product development group work as specified by the product management team. His background is in testing telecom systems where he managed a 30 person testing department. When the telecom industry downsized he was given a sizable package and was hired by ABC Corp to head up it’s newly created Independent Test department. His mandate was to improve the quality of the products produced by Adventure Works. His team of professional testers is expected to provide him with the data required to determine whether the product is acceptable. He provides this data, along with a accept/reject recommendation to the Product Manager who makes the final decision. He is always prepared to back up his recommendation with a clear description of the business impact of the bugs that he feels must be fixed before he would be comfortable advising the product be accepted.

User Experience Specialist

Yvette Kirwan is a Fine Art major who got a job as a graphic artist in a media company. When conventional advertising work took a downturn, she was moved into the product design group where she did graphic design for web-based software applications. She quickly realized that the graphics were mainly eye-candy and that the real value was provided by the interaction design so she went back to school part time to get trained as an interaction designer. With this training under her belt she quickly became a key player in the design of software-intensive products at the company. She was hired by Adventure Works. specifically to work on the new product. Her role is to define the interaction design based on market requirements, user models and feedback from the product manager and the development. The designs need to be implementable in a cost efficient manner using the technology stack used by the product development team. While she is technically part of the UxD Team, she sits in the development area with the development team as she has found this to be a more effective way to transition design knowledge to the developers and their manager. She has also found it to be useful to enlist friends and people off the street to do informal usability testing of paper prototypes as well as the actual software. This gives her real data she can use to help influence the test department’s understanding the users.

Updated Wiki: Vol1-Appendix-A

rburte — Sat, 07 Nov 2009 02:08:29 GMT

Appendix A – Organization Stereotypes and Reader Personas

This appendix describes two stereotypical organizations and the people involved in making or providing data for the readiness and acceptance decisions. The people are described as personas.

IT Organization of a Business

A Product Company

Reader Personas

Personas Often Found in the IT Organization of a Business

Product Manager or Product Owner

Key questions Eric needs to find answers to:

What kinds of information do I need from the QA department to decide whether or not to accept the software? What quality criteria must be met? How do I know they have been met?Who is responsible for verifying these criteria are met? Who is responsible for ensuring these criteria are met?
How does the UxD team participate in acceptance testing of the software?
How can the release strategy (multiple releases, alphas, betas, etc.) help me?
What is the process for addressing deficiencies prior to release?

Business Person in the Marketing Division

Key questions Lisa needs to answer:

What kinds of testing will my team need to do to help me decide whether or not to accept the software?
How can the release strategy (multiple releases, alphas, betas, community technology previews, etc.) help me?
When should I plan on doing AT? How much time should I allow for AT? How much effort will be involved? How much of staff do I need to do it?
What kinds of testing do I expect the IT department to do before asking my team to do acceptance testing?
What information do I need IT to provide to me about the kinds of testing they have done and how will I use that information in deciding whether or not to accept the software (or even start acceptance testing?)
What information do I need to provide to IT prior to me doing AT?

Operations Department Manager

IT Security Specialist

Personas Often Found in a Product Company

Test Manager

User Experience Specialist

Updated Wiki: Documentation

rburte — Sat, 07 Nov 2009 02:05:25 GMT

Acceptance Test Engineering Guide

Volume I: Thinking about Acceptance

Preamble

Part 1 - Thinking About Acceptance

Part 2 - Perspectives on Acceptance

Part 3 - Accepting Software

Appendices

Updated Wiki: Documentation

rburte — Sat, 07 Nov 2009 02:04:09 GMT

Acceptance Test Engineering Guide

Volume I: Thinking about Acceptance

Preamble

Part 1 - Thinking About Acceptance

Part 2 - Perspectives on Acceptance

Part 3 - Accepting Software

Appendecies

Updated Wiki: Vol1-3-Chapter-20

rburte — Sat, 07 Nov 2009 02:02:54 GMT

Chapter 20 - Fine-Tuning the Acceptance Process

The previous chapters of Part III introduced the practices involved in planning and executing the acceptance process. By reading those chapters the reader should be able to get a basic understanding of what is involved in accepting software. This chapter describes how we can use the information we obtain while executing the acceptance process as clues for how to improve the product development processes of our organization to reduce defect levels and to minimize the elapsed time and resources consumed while making the acceptance decision.

The acceptance process is a necessary but potentially wasteful exercise. We should strive to keep it as simple and value-adding as possible. The longer it takes, the more it costs in wasted effort and delayed delivery of business value. We can take steps to streamline the acceptance process by reorganizing how we do readiness assessment and acceptance testing. But the acceptance process is just one part of the product development process. The issues we find while executing the acceptance process are consequences of the choices we have made in how we structure our products, markets, and workflows. We can user these symptoms to motivate a better understanding of the underlying problems in how we work.

Debugging the Acceptance Process

Issues encountered during the acceptance process can be valuable clues about problems in how our organization and workflows are structured. These clues provide hints about how we should restructure our organization and its processes to work more efficiently. The following are a list of common symptoms and possible solutions.

Overly Long Duration of Acceptance Process

An acceptance process that takes a long period of time hints at issues with how the organization is structured. It can be caused by:

Too many groups each needing to their own specialized work and too many handoffs between them. Consider using value stream mapping to identify which steps add real value and which could be eliminated entirely (preferable) or done in parallel (second choice.) The root cause is likely the way the organization is structured; changing the structure may be a necessary though high risk/effort endeaver though one with a potentially huge payback.
Too many test&fix cycles being needed. This is described below.

Too Many Defects found during Acceptance Process

If the acceptance process finds many defects, this is likely a sign that The Product Development Teams():

Don’t know how the finished product should behave.
Are not held accountable for delivering a finished product to the Product Owner acceptance testing. A lack of understanding of what “done” looks like is a sign that the Product Owner has not done an effective job of communicating the nature of the product to the Product Development Team. This could be because the product owner doesn’t know either, or it could be ineffective communication such as occurs when the primary means of communication is written requirement specifications. The Product Owner can learn more quickly what they really want by doing Incremental Acceptance Testing. This gives the PDT more time to address any changes than if the acceptance testing were to be done in a final Acceptance Test Phase. In either case, the joint understanding of the acceptance criteria can be improved by including acceptance tests as part of the requirements process. These tests can be provided by the Product Owner or developed jointly through collaboration between the Product Owner and the Product Development Team.

Lack of accountability occurs for several reasons. One is when the product is decomposed into components that are far removed from the end-user functionality. The Product Development Team often doesn’t understand how their component supports the business goals and therefore builds functionality that may actual be at cross purposes to it. This often results in bugs found only after the components are integrated. This is one of the downfalls of the Component Team approach to organizing the work.
Another reason for lack of accountability is when management values schedule over quality. Of course we say quality is important but its management’s actions that really count. When management pressures developers to meet unrealistic timelines we are saying “schedule trumps quality”. A clear definition of the Minimum Quality Requirement (MQR) is crucial. We need to encourage the Product Development Team to improve its processes to deliver better quality by mistake-proofing the processes as much as possible. Test-Driven Develoment and automated tests are both examples of how to do this. Once they can build software with fewer defects, they will be able to deliver faster as well; the inverse is not true.

Too Many Test&Fix Cycles Needed

When the product requirements have been clearly communicated and the Product Development Team has done an effective readiness assessment, the acceptance testing does not find many defects. Those defects that are found should be fairly minor and easily fixed. But what if there are a lot of defects? Then the product will need to go back through the construction, readiness assessment and acceptance testing process another time. When this cycle has to be repeated several times before the product is good enough to consider releasing, it may be due to one of several root causes:

New bugs are being introduced by many of the fixes for the existing bugs. This could be because the software has become brittle due to age and excessive internal coupling (or maybe it wasn’t designed very well even when it was new.) Or it could be due to rushing the work and a lack of a safety net to catch the defects being introduced. The latter can be addressed by incorporating automated unit testing and possibly test-driven development to avoid inserting the defects.
The product Development Team is not doing effective readiness assessment of each release candidate. This may be because they are being rushed or because they are relying primarily on manual regression testing and cannot hope to retest all the affected software. Consider introduceing automated regression testing at various levels (unit, component, system) to reduce the length of time it takes to do an effective regression test cycle.

Lots of Debate Between Bugs vs. Change Requests

When is a bug a defect and when is it a change request? If this discussion is occurring a lot during the acceptance process it may be an indication that the Product Owner and the product Development Team are not striving to solve the same problem. Some possible root causes are:

Fixed price contracts encourage the product Development Team to classify everything as a CR even if it is legimately a bug. Solution is to avoid creating dysfunctional relationships that come out of fixed price contracts. Consider using Target-Price contracts instead; these motivate both customer and supplier to minimize the changes while maximizing value. There is no distinction between a bug and a CR; the only distinction is between changes that are worth making and those that the customer can live without.
Vauge requirements caused by a lack of understanding of the Product Owner as to what they wanted. See
Vague requirements leading to lack of clarity about what PO really asked for. The gap between what the PO thought they asked for and what the product Development Team interpreted the request caused the bug and the subsequent debate. The solution is to improve the communication between the PO and the product Development Team. This is best achieved through collaboration rather than simply writing more copious requirement documentation. The communication can be supported by detailed examples which can be used as sample acceptance tests. Consider including readiness assessors and acceptance testers in the product design discussions as this will usually unearth interesting scenarious that the Product Owner may need to consider.

Debugging the Acceptance Process

Issues encountered during the acceptance process can be valuable clues about problems in how our organization and workflows are structured. These clues provide hints about how we should restructure our organization and its processes to work more efficiently. The following flowchart suggest possible solutions based on common symptoms:
Psuedo-code for a flowchart:

If Acceptance Taking Too Long
If too many Test&fix cycles
If too many regression bugs introduced
See 19.1.111Use Automated Test Execution to Reduce Regression Bugs
else
Else if too long per cycle
Consider streamlining the process using Value Stream Mapping
Else (not taking too long)
If too man new bugs found
If found by end-users
If Users find system hard to use
See 19.1.61Use Incremental Usability Testing to Discover Design Defects Earlier
If bugs found in a few specific areas
See 19.1.131Increase Breadth of Acceptance Testing
Else if bugs found everywhere
See 19.1.141Increase Depth of Acceptance Testing1
If found by acceptance testers
If PDT is not doing through readiness assessment
If PDT is organized around components
See 19.1.121Use Feature Teams to Improve Accountability of PDT for End-User Functionality
Else If PDT has access to acceptance tests
Run them more frequently
See 19.1.111Use Automated Test Execution to Reduce Regression Bugs
Else
See 19.1.101Use Acceptance Test-Driven Develpment to Help PDT Understand Requirements Better
is being rushed by management or PO or deadlines
See 19.1.151Focus on Quality to Get Speed of Delivery
Else if a

If found by Readiness Assessors
If found by independent testers
Use ATTD to improve communication of “done”
Seealso “If found by acceptance testers”
Else if too many accumulated bugs to fix thereby requiring “Bug Triage”
If due to high bug arrival rates
see “If too many new bugs found”
Else if Bugs aren’t being fixed due to lack of time
Educate Product owner on deciding between Bugs vs New Features
If due to fear of introducing new bugs due to fragility of code base
Find ways to get code base under automated test and refactor
Else if disagreements between Bugs and Change Requests
If Fixed Price Contracts
See 19.1.51Use Target Price Contracts to Align Interests of PO and PDT
Else
See 19.1.91Use Acceptance Test-Driven Develpment to Help PDT Understand Requirements Better

Else
Is there a problem?

The solutions referenced in this flowchart are elaborated below.

Possible Remedies

Based on the results of the analysis of acceptance process issues encountered, one or more of the following remedies may be useful:

Use Target Price Contracts to Align Interests of Product Owner and Product Development Team

Use Incremental Usability Testing to Discover Design Defects Earlier

Incorporate usability testing of early version or prototype (including paper prototypes)

Use Incremental Acceptance Testing to Discover Defects Earlier

Use Incremental Acceptance Testing to Help Product Owner Discover Requirements Earlier

Involve Testers During Requirements Defiintion to Ensure Completeness of Requirements

Use Acceptance Test-Driven Develpment to Help Product Development Team Understand Requirements Better

Product Owner should provide acceptance tests to Product Development Team
Collaborate with Product Development to develop them

Use Automated Test Execution to Reduce Regression Bugs

Use Feature Teams to Improve Accountability of Product Development Team for End-User Functionality

Increase Breadth of Acceptance Testing

Include more varied functionality/scope within the scope of testing.

Increase Depth of Acceptance Testing

Use more/different test design (e.g. Scenario-based testing) and execution techniques (e.g. Exploratory Testing) to get better test coverage.

Focus on Quality to Get Speed of Delivery

Focus on Quality; speed will follow due to less time spent finding and fixing bugs.

Streamlining the Acceptance Process

The acceptance process is a necessary but potentially wasteful exercise. We should strive to keep it as simple and value-adding as possible. The longer it takes, the more it costs in wasted effort and delayed delivery of business value. it It can become a serious impediment to being responsive to our users (the product owner’s customers.) Before we can take steps to streamline it we must first understand it.

Use Value Stream Mapping to Understand the Acceptance Process

We can improve our understanding of an existing or proposed acceptance process through an exercise called Value Stream Mapping. This is a form of business process modeling that focuses our attention on the ratio of elapsed time and the amount of value added. Figure 1 is a value stream map of a hypothetical acceptance process.

Figure 1 As-Is Acceptance Process
This process takes an average of 211 days to execute and provides only 60 days of actual value resulting in a cycle efficiency of only 28%. Some of the factors that make this process take so long to execute are:

Sixty days of software development output is sent through the process in a single batch. Therefore, there is a large inventory of untested software which results in many bugs being found during the readiness assessment and acceptance testing activities. The fixing of these bugs is done entirely on the critical path of the project.
There are several cycles in process each of which are typically executed several times. For example, Readiness Assessment sends the code back, on average 4 times for an additional 5 days elapsed time which adds zero days of value because it is pure rework. Each cycle takes 7.5 days (2.5 days fixing and 5 days readiness assessment) therefore this feedback loop adds 30 days of elapsed time.

Some tasks take longer than necessary because the resources are not dedicated. For example, the readiness assessors have other responsibilities and it takes them 5 days to do 2.5 days of testing.

Customer acceptance is done in two separate phases and is followed by three other forms of acceptance decision making, two of which can send the software back all the way to bug fixing. The software needs to be retested each time the software is sent back to fix bugs.
The security team is overworked resulting in an average wait time of 10 days for the security review. The preparation of the security documents takes an average of 10 days as developers discover what is required but only 1 day of that effort adds real value.
The Change Management Board (CMB) meets monthly resulting in an average wait of 10 business days.

Streamline the Acceptance Process

This acceptance process can be streamlined by changing how the work passes through the various readiness and acceptance activities. Figure 2 illustrates the result of having applied a number transformations on the process. These transformations are inspired by Lean Thinking which focuses on eliminating waste. See the sidebar “Waste in Software Development” for a description of the seven common forms of waste. Which forms of waste we try to eliminate first depends on what is important to us. If time-to-market is the key consideration, we may want to tackle the queuing delays first and then look for ways to reduce the amount of processing. If cost/resource constraints are holding us back, we may want to look for ways to reduce the amount of processing first.

Figure 2 Streamlined Acceptance Process
This streamlined version of the process is the result of the following changes:

We have broken the project into 4 two week (10 business day) iterations. Each iteration is sent through readiness assessment, feature acceptance, integration acceptance and operational acceptance. Bugs found in the first three rounds of testing are fixed in the subsequent iteration. After the fourth iteration, the software is retested and the resulting bugs are fixed in a single bug fixing iteration. Contrast this with the 4 rounds of bug fix as a result of readiness assessment and three rounds due to feature acceptance testing. Each round of testing takes roughly 30% of the time it took in the As-Is process because the backlog of untested software is much smaller. This is an example of reducing waste associated with inventory.
The acceptance tests are provided to the development team along with the requirements. This allows the developers and the readiness assessment testers to verify the functionality is working correctly before handing it off to the acceptance testers thereby reducing the number of bugs found in acceptance testing. The readiness assessment shown as occurring after the iteration actually happens continuously (as soon as the developer says the feature is “ready”) therefore any bugs found in readiness assessment are fixed before the developer has shifted their focus on the next fixture. This reduces the cost of fixing the few bugs that are found in readiness assessment by a factor of 4. This is an example of reducing the waste associated with defects.
The acceptance review for security has been changed from an acceptance activity to a readiness activity. This allows it to be done in parallel with development. It has also been changed from being a “push” model where the team prepares a document to be reviewed to a “pull” model. This starts with a consultation with the security specialist who helps the development team understand the appropriate security requirements and design and what needs to be in the security document. This provides input into the software construction process resulting in a security compliant design as well as more appropriate content in the security document. This reduces the turn-back rate from 30% to 10% and the total effort from 10 days to 3 days without any reduction in value. The effort to produce the document is reduced and the effort to read the document is also reduced; definitely a win-win solution. A final security review is held prior to the final iteration to review the finished document as well as any late-breaking security-related design considerations. This is an example of eliminating waste by reducing delays and by reducing extra processing.
The feature acceptance testing, integration acceptance testing and operations acceptance testing are done in parallel with an understanding that any showstopper bugs found in any of the parallel testing activities can result in the software being rejected. This reduces the elapsed time to the longest of the three instead of the sum of the three activities. This eliminates waste in the form of delays incurred while one group waits for another group to finish their work.
Because only 25% of the functionality is being tested anew in each iteration, the acceptance testing can be finished in 1 day. This is a short enough period that the testers can focus on the one project and finish it 1 day elapsed time. This isn’t a form of waste reduction as much as it is an example of improving “flow” by making the outputof the process less bursty via smaller batch sizes. Achieving flow is another of the key principles of lean development.
The Change Management Board documentation is prepared in parallel with the final bug-fixing iteration and can be submitted to the CMB as soon as the three parallel acceptance decisions are positive. To further reduce the delay, the CMB now meets weekly for 1 hour rather than monthly for a half day. This reduces the average wait from 10 business days to 2.5. This is an example of reducing waste associated with waiting. The collected impact of these changes to the acceptance process has reduce the average elapsed time to 94 days to execute 110 days of value-added processing that delivers 58 days of value for a cycle efficiency of 53%.

Summary

A significant portion of the elapsed time between the finish of development of software and when the software can start providing value to the customer is consumed by the acceptance process. The elapsed time can be reduced significantly by building software incrementally and doing incremental readiness assessment and acceptance testing as each increment of software is finished. Different kinds of inspection and testing activities can be done in parallel with development of the later increments reducing the amount of work that needs to be done on the critical path between completion of development and final acceptance. The types of the issues discovered during acceptance process can be used to diagnose organizational, cultural and process issues in the organizations involved. Changing the processes to avoid the issues is usually more beneficial than improving the efficiency of addressing the issues once found.

What’s Next?

Volume I has introduced a number of tools you can use for reasoning about how you accept software. It has described when to use a large number of acceptance-related planning, requirements, inspection, review and testing practices. You may want to research some of these practices in more detail. Volume 2 in this series describes many of these practices in more detail while Volume 3 provides examples of the artefacts that might have been produced on a fictional project.
Note:
Sidebar: Forms of Waste in Software Development
The seven common wastes of manufacturing can be remembered using the acronym TIM WOOD. In software, there is an equivalent of each form waste as exemplified by the list provided by Mary & Tom Poppendieck in their book [Implementing Lean Software Development"]}. Unfortunately, the software-specific names don’t form a nice acronym so we’ve included both names here. The software-specific names have a * next to them.
*T = The waste of Transport (Handoffs{"*)*
Transportation is waste because it doesn’t add any value to the end product but it increases the cost and the elapsed time.
In software, transport corresponds to handoffs between parties usually via documents. The requirements document handed by specifiers to software developers is one example, the design document handed by architects to developers is another. The preparation of these documents takes large amounts of effort – often much more than communicating the same information verbally. Handoffs usually result in loss of information and this is typically worse when the handoff is asynchronous (e.g. documents) rather than face-to-face. See the sidebar Using Whole Teams to Build Products in Chapter 1 for ways to reduce the number of handoffs.
I = Inventory (Partially Done Work*)
Inventory is bad because it costs money to produce the inventory and often costs money to store or manage the inventory. Inventory also masks issues by delaying when defective parts are discovered. Just-in-time manufacturing is all about reducing inventory to the lowest levels possible.
In software, inventory is any artifact that has taken effort to produce but which is not yet providing the customer with the value expected to be provided by having the software in use. Some common forms of inventory include:
- Untested software – Software that has been written but not yet tested
- Software with a long list of bugs untriaged or triaged but not fixed
- Requirements documents – Documents that provide detailed descriptions of functionality that won’t be built right away.
M = Movement (Task Switching*)
Movement is bad because while a worker is moving they can’t be producing. This adds no value to the product but reduces productivity of the worker thereby increasing cost.
In software development, the equivalent of movement is task switching. This is caused by asking people to work on several things at the same time. Every time they switch between one task and another task, time is wasted while they re-establish their working context.
W = Waiting (Delays* or Queuing)
Whenever work stops while waiting for something to happen, it is a form of waste. Common causes of waiting in software include waiting for approvals, waiting for clarification of requirements and waiting for slow computers or tools to finish their processing.
O = Overprocessing (Lost Knowledge* or Process Inefficiency)
Overprocessing is waste caused by doing unnecessary steps or doing a step longer than necessary. This adds no additional value but it does add cost.
In software, overprocessing is any steps in the production process that are required to produce high quality software. The most common example is the production of any documentation that no one will ever read. Another common form is having to rediscover information that was lost somewhere before it could be used.
O = Overproduction (Extra Features*)
Overproduction is producing too much. It is like inventory except that it is finished product while inventory is work-in-progress. In software, overproduction is the development of unnecessary features. It has been reported REF that on average 80% of features developed are rarely or never used. This is clearly overproduction!
D = Defects
Defects, bugs, problem reports, usability issues all require extra work to analyse, understand and address. This extra work is pure waste. It is exactly the same in software.

Resources

[Poppendieck, M] Lean Software Development – An Agile Toolkit, Addison-Wesley Professional # ISBN-10: 0321150783 ISBN-13: 978-0321150783,
[Poppendieck, M., Poppendieck, T.] Implementing Lean Software Development – From Concept to Cash, Addison-Wesley, …. (See ref in another chapter.)

Updated Wiki: Vol1-3-Chapter-20

rburte — Sat, 07 Nov 2009 02:02:26 GMT

Chapter 20 - Fine-Tuning the Acceptance Process

Debugging the Acceptance Process

Overly Long Duration of Acceptance Process

An acceptance process that takes a long period of time hints at issues with how the organization is structured. It can be caused by:

Too many groups each needing to their own specialized work and too many handoffs between them. Consider using value stream mapping to identify which steps add real value and which could be eliminated entirely (preferable) or done in parallel (second choice.) The root cause is likely the way the organization is structured; changing the structure may be a necessary though high risk/effort endeaver though one with a potentially huge payback.
Too many test&fix cycles being needed. This is described below.

Too Many Defects found during Acceptance Process

If the acceptance process finds many defects, this is likely a sign that The Product Development Teams():

Don’t know how the finished product should behave.
Are not held accountable for delivering a finished product to the Product Owner acceptance testing. A lack of understanding of what “done” looks like is a sign that the Product Owner has not done an effective job of communicating the nature of the product to the Product Development Team. This could be because the product owner doesn’t know either, or it could be ineffective communication such as occurs when the primary means of communication is written requirement specifications. The Product Owner can learn more quickly what they really want by doing Incremental Acceptance Testing. This gives the PDT more time to address any changes than if the acceptance testing were to be done in a final Acceptance Test Phase. In either case, the joint understanding of the acceptance criteria can be improved by including acceptance tests as part of the requirements process. These tests can be provided by the Product Owner or developed jointly through collaboration between the Product Owner and the Product Development Team.

Too Many Test&Fix Cycles Needed

New bugs are being introduced by many of the fixes for the existing bugs. This could be because the software has become brittle due to age and excessive internal coupling (or maybe it wasn’t designed very well even when it was new.) Or it could be due to rushing the work and a lack of a safety net to catch the defects being introduced. The latter can be addressed by incorporating automated unit testing and possibly test-driven development to avoid inserting the defects.
The product Development Team is not doing effective readiness assessment of each release candidate. This may be because they are being rushed or because they are relying primarily on manual regression testing and cannot hope to retest all the affected software. Consider introduceing automated regression testing at various levels (unit, component, system) to reduce the length of time it takes to do an effective regression test cycle.

Lots of Debate Between Bugs vs. Change Requests

Fixed price contracts encourage the product Development Team to classify everything as a CR even if it is legimately a bug. Solution is to avoid creating dysfunctional relationships that come out of fixed price contracts. Consider using Target-Price contracts instead; these motivate both customer and supplier to minimize the changes while maximizing value. There is no distinction between a bug and a CR; the only distinction is between changes that are worth making and those that the customer can live without.
Vauge requirements caused by a lack of understanding of the Product Owner as to what they wanted. See
Vague requirements leading to lack of clarity about what PO really asked for. The gap between what the PO thought they asked for and what the product Development Team interpreted the request caused the bug and the subsequent debate. The solution is to improve the communication between the PO and the product Development Team. This is best achieved through collaboration rather than simply writing more copious requirement documentation. The communication can be supported by detailed examples which can be used as sample acceptance tests. Consider including readiness assessors and acceptance testers in the product design discussions as this will usually unearth interesting scenarious that the Product Owner may need to consider.

Debugging the Acceptance Process

Issues encountered during the acceptance process can be valuable clues about problems in how our organization and workflows are structured. These clues provide hints about how we should restructure our organization and its processes to work more efficiently. The following flowchart suggest possible solutions based on common symptoms:
Psuedo-code for a flowchart:

If Acceptance Taking Too Long
If too many Test&fix cycles
If too many regression bugs introduced
See 19.1.111Use Automated Test Execution to Reduce Regression Bugs
else
Else if too long per cycle
Consider streamlining the process using Value Stream Mapping
Else (not taking too long)
If too man new bugs found
If found by end-users
If Users find system hard to use
See 19.1.61Use Incremental Usability Testing to Discover Design Defects Earlier
If bugs found in a few specific areas
See 19.1.131Increase Breadth of Acceptance Testing
Else if bugs found everywhere
See 19.1.141Increase Depth of Acceptance Testing1
If found by acceptance testers
If PDT is not doing through readiness assessment
If PDT is organized around components
See 19.1.121Use Feature Teams to Improve Accountability of PDT for End-User Functionality
Else If PDT has access to acceptance tests
Run them more frequently
See 19.1.111Use Automated Test Execution to Reduce Regression Bugs
Else
See 19.1.101Use Acceptance Test-Driven Develpment to Help PDT Understand Requirements Better
is being rushed by management or PO or deadlines
See 19.1.151Focus on Quality to Get Speed of Delivery
Else if a

If found by Readiness Assessors
If found by independent testers
Use ATTD to improve communication of “done”
Seealso “If found by acceptance testers”
Else if too many accumulated bugs to fix thereby requiring “Bug Triage”
If due to high bug arrival rates
see “If too many new bugs found”
Else if Bugs aren’t being fixed due to lack of time
Educate Product owner on deciding between Bugs vs New Features
If due to fear of introducing new bugs due to fragility of code base
Find ways to get code base under automated test and refactor
Else if disagreements between Bugs and Change Requests
If Fixed Price Contracts
See 19.1.51Use Target Price Contracts to Align Interests of PO and PDT
Else
See 19.1.91Use Acceptance Test-Driven Develpment to Help PDT Understand Requirements Better

Else
Is there a problem?

The solutions referenced in this flowchart are elaborated below.

Possible Remedies

Based on the results of the analysis of acceptance process issues encountered, one or more of the following remedies may be useful:

Use Target Price Contracts to Align Interests of Product Owner and Product Development Team

Use Incremental Usability Testing to Discover Design Defects Earlier

Incorporate usability testing of early version or prototype (including paper prototypes)

Use Incremental Acceptance Testing to Discover Defects Earlier

Use Incremental Acceptance Testing to Help Product Owner Discover Requirements Earlier

Involve Testers During Requirements Defiintion to Ensure Completeness of Requirements

Use Acceptance Test-Driven Develpment to Help Product Development Team Understand Requirements Better

Product Owner should provide acceptance tests to Product Development Team
Collaborate with Product Development to develop them

Use Automated Test Execution to Reduce Regression Bugs

Use Feature Teams to Improve Accountability of Product Development Team for End-User Functionality

Increase Breadth of Acceptance Testing

Include more varied functionality/scope within the scope of testing.

Increase Depth of Acceptance Testing

Use more/different test design (e.g. Scenario-based testing) and execution techniques (e.g. Exploratory Testing) to get better test coverage.

Focus on Quality to Get Speed of Delivery

Focus on Quality; speed will follow due to less time spent finding and fixing bugs.

Streamlining the Acceptance Process

The acceptance process is a necessary but potentially wasteful exercise. We should strive to keep it as simple and value-adding as possible. The longer it takes, the more it costs in wasted effort and delayed delivery of business value. it It can become a serious impediment to being responsive to our users (the product owner’s customers.) Before we can take steps to streamline it we must first understand it.

Use Value Stream Mapping to Understand the Acceptance Process

We can improve our understanding of an existing or proposed acceptance process through an exercise called Value Stream Mapping. This is a form of business process modeling that focuses our attention on the ratio of elapsed time and the amount of value added. Figure 1 is a value stream map of a hypothetical acceptance process.

Cannot resolve image macro, invalid image name or id.
Figure 1 As-Is Acceptance Process
This process takes an average of 211 days to execute and provides only 60 days of actual value resulting in a cycle efficiency of only 28%. Some of the factors that make this process take so long to execute are:

Sixty days of software development output is sent through the process in a single batch. Therefore, there is a large inventory of untested software which results in many bugs being found during the readiness assessment and acceptance testing activities. The fixing of these bugs is done entirely on the critical path of the project.
There are several cycles in process each of which are typically executed several times. For example, Readiness Assessment sends the code back, on average 4 times for an additional 5 days elapsed time which adds zero days of value because it is pure rework. Each cycle takes 7.5 days (2.5 days fixing and 5 days readiness assessment) therefore this feedback loop adds 30 days of elapsed time.

Some tasks take longer than necessary because the resources are not dedicated. For example, the readiness assessors have other responsibilities and it takes them 5 days to do 2.5 days of testing.

Customer acceptance is done in two separate phases and is followed by three other forms of acceptance decision making, two of which can send the software back all the way to bug fixing. The software needs to be retested each time the software is sent back to fix bugs.
The security team is overworked resulting in an average wait time of 10 days for the security review. The preparation of the security documents takes an average of 10 days as developers discover what is required but only 1 day of that effort adds real value.
The Change Management Board (CMB) meets monthly resulting in an average wait of 10 business days.

Streamline the Acceptance Process

This acceptance process can be streamlined by changing how the work passes through the various readiness and acceptance activities. Figure 2 illustrates the result of having applied a number transformations on the process. These transformations are inspired by Lean Thinking which focuses on eliminating waste. See the sidebar “Waste in Software Development” for a description of the seven common forms of waste. Which forms of waste we try to eliminate first depends on what is important to us. If time-to-market is the key consideration, we may want to tackle the queuing delays first and then look for ways to reduce the amount of processing. If cost/resource constraints are holding us back, we may want to look for ways to reduce the amount of processing first.
Cannot resolve image macro, invalid image name or id.
Figure 2 Streamlined Acceptance Process
This streamlined version of the process is the result of the following changes:

We have broken the project into 4 two week (10 business day) iterations. Each iteration is sent through readiness assessment, feature acceptance, integration acceptance and operational acceptance. Bugs found in the first three rounds of testing are fixed in the subsequent iteration. After the fourth iteration, the software is retested and the resulting bugs are fixed in a single bug fixing iteration. Contrast this with the 4 rounds of bug fix as a result of readiness assessment and three rounds due to feature acceptance testing. Each round of testing takes roughly 30% of the time it took in the As-Is process because the backlog of untested software is much smaller. This is an example of reducing waste associated with inventory.
The acceptance tests are provided to the development team along with the requirements. This allows the developers and the readiness assessment testers to verify the functionality is working correctly before handing it off to the acceptance testers thereby reducing the number of bugs found in acceptance testing. The readiness assessment shown as occurring after the iteration actually happens continuously (as soon as the developer says the feature is “ready”) therefore any bugs found in readiness assessment are fixed before the developer has shifted their focus on the next fixture. This reduces the cost of fixing the few bugs that are found in readiness assessment by a factor of 4. This is an example of reducing the waste associated with defects.
The acceptance review for security has been changed from an acceptance activity to a readiness activity. This allows it to be done in parallel with development. It has also been changed from being a “push” model where the team prepares a document to be reviewed to a “pull” model. This starts with a consultation with the security specialist who helps the development team understand the appropriate security requirements and design and what needs to be in the security document. This provides input into the software construction process resulting in a security compliant design as well as more appropriate content in the security document. This reduces the turn-back rate from 30% to 10% and the total effort from 10 days to 3 days without any reduction in value. The effort to produce the document is reduced and the effort to read the document is also reduced; definitely a win-win solution. A final security review is held prior to the final iteration to review the finished document as well as any late-breaking security-related design considerations. This is an example of eliminating waste by reducing delays and by reducing extra processing.
The feature acceptance testing, integration acceptance testing and operations acceptance testing are done in parallel with an understanding that any showstopper bugs found in any of the parallel testing activities can result in the software being rejected. This reduces the elapsed time to the longest of the three instead of the sum of the three activities. This eliminates waste in the form of delays incurred while one group waits for another group to finish their work.
Because only 25% of the functionality is being tested anew in each iteration, the acceptance testing can be finished in 1 day. This is a short enough period that the testers can focus on the one project and finish it 1 day elapsed time. This isn’t a form of waste reduction as much as it is an example of improving “flow” by making the outputof the process less bursty via smaller batch sizes. Achieving flow is another of the key principles of lean development.
The Change Management Board documentation is prepared in parallel with the final bug-fixing iteration and can be submitted to the CMB as soon as the three parallel acceptance decisions are positive. To further reduce the delay, the CMB now meets weekly for 1 hour rather than monthly for a half day. This reduces the average wait from 10 business days to 2.5. This is an example of reducing waste associated with waiting. The collected impact of these changes to the acceptance process has reduce the average elapsed time to 94 days to execute 110 days of value-added processing that delivers 58 days of value for a cycle efficiency of 53%.

Summary

What’s Next?

Resources

Updated Wiki: Documentation

rburte — Sat, 07 Nov 2009 01:46:52 GMT

Updated Wiki: Vol1-3-Chapter-19

rburte — Sat, 07 Nov 2009 01:46:19 GMT

Chapter 19 - Managing the Acceptance Process

The previous chapter introduced the individual test lifecycle and the practices the assessors use for identifying test conditions, designing the tests, executing the tests and assessing the outcomes, and maintaining the tests. This chapter introduces the management practices we use while executing the tests and while making the acceptance decision.
The process of accepting software involves many activities that generate the data we use as input into the acceptance decision. Each of these activities takes time and consumes valuable human and other resources. A well managed acceptance process will use these resources wisely while a poor managed process has the potential to waste these resources, delay the acceptance decision and even compromise the quality of the decision made.
While we are executing our test plans, we want to know the answers to two important and inter-related questions:

First, how is testing progressing? When will we have a high-confidence assessment of the product quality? Will it be in time to make our readiness or acceptance decision?
Second, what is our current assessment of the product quality? Is it good enough to release whether to customers or to the acceptance testers? What actions do we need to take to make it good enough?

Test Scheduling Practices

The overall schedule that defines which tests will be run and when should have already been defined as part of the test plan. That includes the strategic decisions about whether all testing is done during a final test phase or incrementally throughout the project. In this section we focus on the techniques we use for scheduling of test execution. As a reminder, under test execution we include both dynamic testing when test cases are run that involve executing software in the system-under-test and static testing that is performed in the form of reviews or inspections without actually running code in the system-under-test.

Plan-Driven Test Scheduling

The traditional approach to test scheduling involves defining a period of time, sometimes called a test cycle, in which one complete round of testing can be completed. The test plan defines how many test cycles are planned. The details of what happens in each test cycle often emerge as the project unfolds. One approach is to define a detailed project plan for the first test cycle, one which defines the set of testing activities that will be done on a day by day basis. Project management tools such as Pert or Gantt charts may be used to document this detailed test execution plan. (See Plan-Driven Test Management.)

Session-Based Test Scheduling

The alternative to defining a detailed project plan of test activities is to schedule a series of test sessions and create a backlog (prioritized list) of test session charters that are executed in these test sessions. Each session would be of a standard duration, say 90 minutes, and most charters should be executable within one session. This Session-Based Test Management is most typically used for managing exploratory testing but could also be used for execution of scripted testing. When each test session is completed the tester marks the test charter as one of completed or needs one or more additional sessions, or suggests additional test charters for future test session. This provides a good indication of how testing is progressing as illustrated in figure K (Test Charter Burndown) which shows the number of test charters dropping over time but with the occasional upwards spike as new test charters are identified.

Figure K Test Charter Burndown
In this chart, the bars representing the original charters show the charters that were defined before test execution began. The bars labelled Added Charters stacked on top of them represent new charters that were identified while test execution was occurring. The Total Left line indicates how many test charters remain to be fullfiled while the Charters Completed indicates the number of charters already fulfilled. We can get a good idea at any point during test execution of when testing will be completed by projecting the Total Left line out to the right until it intersects zero charters. That is the earliest probably completion date.

Self-Organized Test Scheduling

A third approach is often used by agile or self-organizing teams. All the testing tasks are posted on a wall chart, variously called a “big visible chart” [JeffriesOnBigVisibleCharts] or an “information radiator” [CokburnOnAgileDevelopment], and team members sign up to do specific testing tasks. As they finish one task, they mark it done and pick another task to perform. This is called Self-Organized Test Scheduling.

Event-Triggered Test Scheduling

Some forms of automated testing can be set up to run automatically at certain times (such as every night at 2am) or when certain events occur (such as when someone checks in a change to part of the code base.) The results can be posted on a web site, automatically e-mailed/IM’d to all team members, or communicated by a multi-colored light display or “lava lamps” in the team workspace or by an icon in everyone’s computer’s system tray or side gadget (for one such tool see Team Build Monitor [LambOnTeamBuildMonitor]). This practice is often called Continuous Integration [MartinFowlerOnContinuousIntegration]or CI for short. Teams that use continuous integration typically adopt a “stop the line” mentality to failing tests. The goal is to promote “code health” early. Whenever a test failure is reported by the automated test harness, fixing the failed test becomes the top priority for everyone on the team . This is the software equivalent of the “stop the line” practice used in lean manufacturing. Once the failure is fixed, everyone can go back to working on their respective tasks. This focus on keeping the product code ”healthy” improves quality and and reduces the duration of the acceptance test cycle.
Another approach is to prevent changes that break the build from being checked-in. The Gated Check-in feature of the Team Foundation Server defers a developer’s check-in until it can be merged and validated by an automated build. Passing automated tests or validating results of the static analysis could are examples of the validation policies. For more information on Team Foundation Build, see [MSDNOnTeamFoundationBuild].

Test Progress Reporting Practices

When we first start executing the tests we don’t know whether the quality is high or low but we do know that we don’t have a high degree of confidence yet. As testing progresses, we should be getting a better idea of what the quality level is and how much longer it will take us to get to the required level of confidence. This is very similar to the cone of uncertainty concept that predicts the completion date and/or cost of developing the software. Figure X illustrates the cone of uncertainty for quality for a typical project. The initial guestimate was anywhere between 10 and 100 person-months. As the requirements were better understood, the range reduced somewhat but a sudden discovery of additional scope raised the estimates once again. An effort was made to descope the project to recover the original timelines. Work creep slowly raised both the upper and lower limit and the estimates finally converge when the product is accepted as-is.

Figure X Cone of Uncertainty for Quality
We can, of course, assume that we will have a reasonably accurate assessment of quality when we have finished all our planned testing. This assumption may or may not be correct because it depends on how effectively our planned test activities will find all the important bugs. This also implies a clear understanding of what is important to the product owner. This is very difficult to assess ahead of time. We certainly need to monitor how much testing work remains to be executed in the current test cycle and when we expect to have it all completed. This applies to each test cycle we execute.
Session-based testing introduces a feedback mechanism that explicitly allows us to adjust the test plans as we learn new information both about the product and about the product owner and the product owner’s needs. As each test session is completed the testers add any newly identified areas of concern to the backlog of test charters yet to be executed. Developers may also suggest additional test charters to address the risk associated with the modifications they make to the software as a result of change requests and bug fixes identified by tests. We monitor the size of the backlog of test charters to get a sense of whether we are making headway. As the confidence of the testers and developers improves they will suggest fewer and fewer new test charters and therefore the size of the charter backlog would drop faster.
Predicting when the software will be of good enough quality to deliver is difficult because that involves predicting how many test cycles we will require and how much time the product development team will require between the test cycles to fix the bugs. (See Chapter 16 – Planning for Acceptance for a more detailed discussion on this topic.) This requires monitoring the bug backlog and the arrival rate of new bugs to assess and adjust the predicted delivery date.

Assessing Test Effectiveness

One of the challenges of testing is assessing how effective our tests really are so that we can know how confident we should be in our assessment of the quality we have. There are several ways to calculate the effectiveness of the tests including the use of coverage metrics and using the find rate of intentionally seeded bugs.

Coverage Metrics

We can use metrics like test coverage and code coverage to calculate the theoretical effectiveness of our tests. These metrics can tell us what percentage of the requirements has been tested and what percentage of the code has been executed by the tests but neither of these is a direct measure of what percentage of the bugs we have found. Of course, the primary issue is that we have no idea of how many bugs really exist so it is pretty difficult to say with any certainty what percentage we have found. For a cautious approach to using code coverage, see Brian Marick’s essay [MarickOnCodeCoverageMisuse].

Defect Seeding

One technique the can provide a more direct measure is defect seeding. It involves deliberately placing a known set of defects in the software. As bugs are found during testing we can estimate the percentage of defects found by dividing the number of seeded defects found by the total number of seeded defects as shown in Figure Y.

Figure Y Percentage of Bugs Found
We can project the number of defects yet to be discovered using the formula in Figure Z.

Or,

Figure Z Total Bugs Calculation
These calculations are described in more detail in the Test Status Reporting thumbnail.

Bug Management and Concern Resolution

A key output of testing and reviews is a list of known gaps between what the product owner expects of the product and how it actually behaves. While the progress of testing is usually reported in terms of which tests have been run and which haven’t, the gap between expectations and reality is usually expressed as a list of known bugs or issues that may or may not have to be addressed before the product will be accepted. In Chapter 16 – Planning for Acceptance, we introduced the Concern Resolution Model which describes how concerns, including software bugs, change requests, issues and documentation bugs, can be managed. Concerns that fall into any of these categories could be considered gating (also known as blocking or blockers), that is, would prevent the system from being accepted. Once we have finished our first full pass of testing, presumably at the end of our first test cycle, we can think of the set of gating bugs as being the outstanding work list for the product development team. Our goal is to drive the list of gating bugs down to zero so that we can consider accepting and releasing the product. (Note that just because it reaches zero does not imply that there are no gating bugs, just none we currently know about.) The product owner can influence the gating count in two ways: the product owner can classify newly found bugs as gating or they can reclassify existing gating bugs as non-gating if they decide that the bug can be tolerated because there is a low enough likelihood (reduced probability as described in Chapter 5 – Risk Model) it will be encountered or there are acceptable work-arounds in the event that it is encountered (reduced impact per the risk model.)

Bug Triage

It is common practice to classify the severity of bugs based on their business impact. Usually a Severity 1 (or Sev 1 as it is commonly abbreviated) bug is a complete show stopper while a Sev 5 bug is merely cosmetic and won’t impact the ability of users to use the product effectively. Many product owners insist that all Sev 1 & 2 bugs be fixed before they will accept the product. Some product owners consider Sev 3 bugs critical enough to insist they are resolved before the product can be accepted. Note that the interpretation of the severity scale is merely a vocabulary for communication of the importance of bugs between the product owner and the product development team; it is entirely up to the product owner to make the decision whether the bug needs to be fixed. The product development team may influence that decision by pointing out similarities or difference with other bugs that had the same severity rating or by pointing out the potential impact as part of an argument to increase or decrease the severity. They might also point out potential workarounds or partial fixes that they feel might justify reducing the severity. But ultimately, it is the product owner decisions as to whether the product is acceptable with the bug still present.
The product owner needs to be reasonable about what bugs should be classified as gating. If there are 1,000 bugs and the team can fix 20 bugs per week, it will take the team 50 weeks to fix all the existing bugs assuming that no new bugs are found and no regression bugs are introduced by the bug fixes. Both these assumptions are highly optimistic. Therefore, the customer needs to ask ”Which of these bugs truly need to be fixed before I can accept the product?” This is purely a business decision because every bug has a business impact. Some may have an infinitesimally small business impact while others may have a large business impact. The product owner needs to be opinionated about this and to be prepared to live with the consequences of their decision whether that is to delay acceptance, deployment and usage of the product until a particular bug is fixed or to put the software into use with the known bug present. There is no point in insisting that all bugs must be fixed before acceptance simply to be able to say that there are no known bugs. Doing so will likely delay accruing any benefits of the system unnecessarily. The process of deciding the severity of each bug and whether or not it should ever be fixed is sometimes called Bug Triage.

Bug Backlog Analysis

This list of known bugs is a collection of useful knowledge about the state of the product. Most bug management tools include a set of standard reports that can be used to better understand the bug backlog. These include:

Bug Fix Rate Report – Describes the rate at which bugs are being fixed.
Bug Arrival Rate Report – Describes the rate at which new bugs are being reported. A decreasing arrival rate may be an indication that the return on investment of testing has reached the point of diminishing returns. Or it could just mean that less testing is being done or that the testing is repeating itself. Long-lived products that release on a regular cycle typically find that the shape of the curve is fairly constant from release to release and this can be used to predict the ready-to-deliver date quite accurately. See Figure D – Bug Arrival Rate and Figure E – Cumulative Bugs Found.
Bug Debt or Bug Burn Down Report - Describes the rate at which the number of bugs is being reduced or is increasing. The burn down report aggregates the bug arrival rate and the bug fixing rate to allow us to predict when the number of gating bugs will be low enough to allow accepting the product and releasing it to users.
Bug Aging Report – Classifies bugs by how long it has taken to fix them and how long it has been since unresolved bugs were first reported. The latter will give an indication of potential level of customer dissatisfaction if the average age of bugs is large.
Bug Correlation Report – Classifies bugs based on their relationship with attributes of the system under-test. The product owner is typically most interested in which features have the most bugs because this helps them understand the business impact of accepting the system without resolving them. The product development team, on the other hand, is typically interested in which components, subsystems, or development teams have the most bugs associated with them because this helps them understand where their own internal processes need to be improved most. Figure Z is an example of a Bugs by Area or Bugs by Feature Team report.

Figure D Bug Arrival Rate

Figure E Cumulative Bugs Found.

Figure Z Bug Correlation Report

Figure W Bug Debt/Burn Down Report

Test Asset Management Practices

The artefacts produced while planning and executing the assessment process are assets; these test assets need to be managed. If repeatability of testing is considered important, such as when the same tests need to be used for regression testing subsequent releases, test scripts and the corresponding test data sets need to be stored in a version-controlled test asset repository. This may be a document repository such as SharePoint or a code repository such as the Team Foundation Server repository, Subversion or CVS. This is described in more detail in the Test Asset Management thumbnail.
When test assets are expected to be long lived such as when the same tests will be used to regression test several releases of a product, it is important to have a strategy for the evolving the test assets to ensure maintainability. The Test Evolution, Refactoring and Maintenance thumbnail describes how we can keep our test assets from degrading over time as the product they verify evolves.

Acceptance Process Improvement

The acceptance process consumes a significant portion of a project’s resources. Therefore, it is a good place to look when trying to reduce costs and improve the effectiveness of one’s processes. Two good candidates for process improvement are improving the effectiveness of the test practices and streamlining the acceptance process to reduce the elapsed time. The latter is the subject of the next chapter.

Improving Test Effectiveness

Multi-release projects and long-lived products will go through the acceptance process many times in their lifetime. Each release can benefit from lessons learned in the previous release, if we care to apply the lessons. It is worth conducting one or more retrospectives after each release to better understand which readiness assessment and acceptance testing activities had the most impact on product quality and which had the least. The highly effective activities should be continued in future releases and the ineffective ones should be replaced with something else. Some teams make it a point to try at least one new practice each release to see if it will detect bugs that previously slipped through the readiness assessment and acceptance testing safety nets. It is also worth analysing the list of defect found in the usage phase to identify any shortcomings in the safety net. Bug correlation reports can come in handy in this exercise. See Figure Q – Bugs by How Found Report.

Figure Q Bugs by How Found Report.

Summary

Like any process, execution of the acceptance process needs to be managed. This includes monitoring the execution of the planned testing and ensuring it results in data that allows for a high-confidence acceptance decision. A key aspect of managing acceptance is the managing the bug backlog to get the best possible quality product at the earliest possible time. These two goals are at odds with each other and deciding which takes precedence should be a business decision. When building product is an ongoing goal of the product owner organization, continuous improvement of the acceptance process should also be managed to further reduce time to usage and improve product quality.

What’s Next

The acceptance process typically takes up a significant portion of a project’s resources and elapsed time. The next chapter looks at ways both elapsed time and resource usage can be reduced.

Resources

[MartinFowlerOnContinuousIntegration] Fowler, M. Continuous Integration. http://martinfowler.com/articles/continuousIntegration.html
[MSDNOnTeamFoundationBuild] http://msdn.microsoft.com/en-us/library/bb668958.aspx
[CockburnOnAgileDevelopment]Cockburn, Alistair “Agile Software Development: The Cooperative Game” Addison Wesley Professional
[JeffiresOnBigVisibleCharts] Ron Jeffries, Big Visible Charts, http://xprogramming.com/xpmag/bigvisiblecharts/
[LambOnTeamBuildMonitor] http://blogs.msdn.com/jimlamb/attachment/3467297.ashx
[MarickOnCodeCoverageMisuse] Brian Marick, “How to Misuse Code Coverage“ http://www.exampler.com/testing-com/writings/coverage.pdf

Updated Wiki: Vol1-3-Chapter-19

rburte — Sat, 07 Nov 2009 01:44:36 GMT

Chapter 19 - Managing the Acceptance Process

First, how is testing progressing? When will we have a high-confidence assessment of the product quality? Will it be in time to make our readiness or acceptance decision?
Second, what is our current assessment of the product quality? Is it good enough to release whether to customers or to the acceptance testers? What actions do we need to take to make it good enough?

Test Scheduling Practices

Plan-Driven Test Scheduling

Session-Based Test Scheduling

The alternative to defining a detailed project plan of test activities is to schedule a series of test sessions and create a backlog (prioritized list) of test session charters that are executed in these test sessions. Each session would be of a standard duration, say 90 minutes, and most charters should be executable within one session. This Session-Based Test Management is most typically used for managing exploratory testing but could also be used for execution of scripted testing. When each test session is completed the tester marks the test charter as one of completed or needs one or more additional sessions, or suggests additional test charters for future test session. This provides a good indication of how testing is progressing as illustrated in figure K (Test Charter Burndown) which shows the number of test charters dropping over time but with the occasional upwards spike as new test charters are identified.
Cannot resolve image macro, invalid image name or id.
Figure K Test Charter Burndown
In this chart, the bars representing the original charters show the charters that were defined before test execution began. The bars labelled Added Charters stacked on top of them represent new charters that were identified while test execution was occurring. The Total Left line indicates how many test charters remain to be fullfiled while the Charters Completed indicates the number of charters already fulfilled. We can get a good idea at any point during test execution of when testing will be completed by projecting the Total Left line out to the right until it intersects zero charters. That is the earliest probably completion date.

Self-Organized Test Scheduling

Event-Triggered Test Scheduling

Test Progress Reporting Practices

When we first start executing the tests we don’t know whether the quality is high or low but we do know that we don’t have a high degree of confidence yet. As testing progresses, we should be getting a better idea of what the quality level is and how much longer it will take us to get to the required level of confidence. This is very similar to the cone of uncertainty concept that predicts the completion date and/or cost of developing the software. Figure X illustrates the cone of uncertainty for quality for a typical project. The initial guestimate was anywhere between 10 and 100 person-months. As the requirements were better understood, the range reduced somewhat but a sudden discovery of additional scope raised the estimates once again. An effort was made to descope the project to recover the original timelines. Work creep slowly raised both the upper and lower limit and the estimates finally converge when the product is accepted as-is.
Cannot resolve image macro, invalid image name or id.
Figure X Cone of Uncertainty for Quality
We can, of course, assume that we will have a reasonably accurate assessment of quality when we have finished all our planned testing. This assumption may or may not be correct because it depends on how effectively our planned test activities will find all the important bugs. This also implies a clear understanding of what is important to the product owner. This is very difficult to assess ahead of time. We certainly need to monitor how much testing work remains to be executed in the current test cycle and when we expect to have it all completed. This applies to each test cycle we execute.
Session-based testing introduces a feedback mechanism that explicitly allows us to adjust the test plans as we learn new information both about the product and about the product owner and the product owner’s needs. As each test session is completed the testers add any newly identified areas of concern to the backlog of test charters yet to be executed. Developers may also suggest additional test charters to address the risk associated with the modifications they make to the software as a result of change requests and bug fixes identified by tests. We monitor the size of the backlog of test charters to get a sense of whether we are making headway. As the confidence of the testers and developers improves they will suggest fewer and fewer new test charters and therefore the size of the charter backlog would drop faster.
Predicting when the software will be of good enough quality to deliver is difficult because that involves predicting how many test cycles we will require and how much time the product development team will require between the test cycles to fix the bugs. (See Chapter 16 – Planning for Acceptance for a more detailed discussion on this topic.) This requires monitoring the bug backlog and the arrival rate of new bugs to assess and adjust the predicted delivery date.

Assessing Test Effectiveness

Coverage Metrics

Defect Seeding

One technique the can provide a more direct measure is defect seeding. It involves deliberately placing a known set of defects in the software. As bugs are found during testing we can estimate the percentage of defects found by dividing the number of seeded defects found by the total number of seeded defects as shown in Figure Y.
Cannot resolve image macro, invalid image name or id.
Figure Y Percentage of Bugs Found
We can project the number of defects yet to be discovered using the formula in Figure Z.

Or,
Cannot resolve image macro, invalid image name or id.
Figure Z Total Bugs Calculation
These calculations are described in more detail in the Test Status Reporting thumbnail.

Bug Management and Concern Resolution

Bug Triage

Bug Backlog Analysis

Bug Fix Rate Report – Describes the rate at which bugs are being fixed.
Bug Arrival Rate Report – Describes the rate at which new bugs are being reported. A decreasing arrival rate may be an indication that the return on investment of testing has reached the point of diminishing returns. Or it could just mean that less testing is being done or that the testing is repeating itself. Long-lived products that release on a regular cycle typically find that the shape of the curve is fairly constant from release to release and this can be used to predict the ready-to-deliver date quite accurately. See Figure D – Bug Arrival Rate and Figure E – Cumulative Bugs Found.
Bug Debt or Bug Burn Down Report - Describes the rate at which the number of bugs is being reduced or is increasing. The burn down report aggregates the bug arrival rate and the bug fixing rate to allow us to predict when the number of gating bugs will be low enough to allow accepting the product and releasing it to users.
Bug Aging Report – Classifies bugs by how long it has taken to fix them and how long it has been since unresolved bugs were first reported. The latter will give an indication of potential level of customer dissatisfaction if the average age of bugs is large.
Bug Correlation Report – Classifies bugs based on their relationship with attributes of the system under-test. The product owner is typically most interested in which features have the most bugs because this helps them understand the business impact of accepting the system without resolving them. The product development team, on the other hand, is typically interested in which components, subsystems, or development teams have the most bugs associated with them because this helps them understand where their own internal processes need to be improved most. Figure Z is an example of a Bugs by Area or Bugs by Feature Team report.

Cannot resolve image macro, invalid image name or id.
Figure D Bug Arrival Rate
Cannot resolve image macro, invalid image name or id.
Figure E Cumulative Bugs Found.
Cannot resolve image macro, invalid image name or id.
Figure Z Bug Correlation Report
Cannot resolve image macro, invalid image name or id.
Figure W Bug Debt/Burn Down Report

Test Asset Management Practices

Acceptance Process Improvement

Improving Test Effectiveness

Summary

What’s Next

The acceptance process typically takes up a significant portion of a project’s resources and elapsed time. The next chapter looks at ways both elapsed time and resource usage can be reduced.

Resources

Updated Wiki: Vol1-3-Chapter-18

rburte — Sat, 07 Nov 2009 01:33:16 GMT

Chapter 18 - Assessing Software

In the previous chapters we have introduced many of the concepts around how we plan the assessment of the product against the Minimum Marketable Functionality (MMF) and minimum quality requirement set (MQR) to which we have agreed. In this chapter we will introduce the various techniques that we use as we do the assessment including test conception, test design and test execution.

Assessment is a generic term we can use to describe the activities, testing or otherwise, that we use to evaluate the system-under-test. Some of these activities are focussed on preparing and executing tests while others may be review activities. Some of the activities are done as part of readiness assessment by the product development team while others may be done by or for the product owner under the banner of acceptance testing. Where the same practice is used in both forms of testing, the mechanics of the practices typically don’t change very much although the emphasis of how much each practice is done and the objectives of applying that practice might vary.
We start the discussion with an overview of the lifecycle of an individual tests, something that both functional and non-functional tests do share and then move into a discussion of the practices used in each state of the individual test lifecycle where we introduce the techniques that are unique to specific of kinds of requirements.

The Lifecycle of an Individual Test

Every single test, however simple or complex, whether manual or automated, goes through a number of stages during its lifetime. This lifecycle is illustrated in Figure 1.

Figure 1 Individual Test Lifecycle
The states of the lifecycle are:
Note that test conception and test authoring are often lumped together under the label of test design. Some forms of testing, often called static testing, involve inspecting artefacts that describe the system under test rather than running the actual code. The terminology used for these forms of testing is somewhat different (for example. they are often called reviews rather than tests) but for the purpose of discussing test lifecycle, we shall use the test terminology.
Let’s examine each of these states in a bit more detail.

Test Conception

At some point, someone decides that we need to verify one or more aspects of the system-under-test’s behavior; we call these “things to test” test conditions. At this time the test is just a figment of someone’s imagination. It starts its transition from an implicit requirement to one that is much more explicit when it gets written down or captured in a document. It might appear in a list of test conditions associated with a feature, requirement or user story. Typically, it will just be a phrase or name with no associated detail. Now that the test exists in concept, we can start moving it through its lifecycle.

Test Authoring / Test Design

Test authoring or test design is the transformation of the test or test condition from being just a named item on a list into concrete actions. It may also involve making decisions around how to organize test conditions into test cases which are the sequences of steps we execute to verify them. Note that test authoring/design may happen long before test execution or concurrently with test execution.

Test Scheduling

Once a test case has been identified and authored or a test charter defined, we must decide when it will be executed. The schedule may indicate the one time the test is run or the frequency with which it is run and the triggering mechanism. It may also identify who or what is executing the test and in which test environment(s).

Test Execution

Once authored and scheduled, we need to actually execute the tests. For dynamic tests this involves running the system-under-test; static tests involve inspecting various artefacts that describe the system-under-test but do not involved executing the code . Depending on the kind of test in question the test may be executed manually by a person, by an automated testing tool, or by a person using tools that improve tester productivity through automation.

Result Assessment

Depending on the tools involved, the pass/fail status of the tests may be determined as the tests are executed or there may be a separate step to determine the test results after the test execution has been completed. We determine whether a test passed or failed by inspecting the actual results observed and determining whether they are acceptable.

Test Reporting

Once a suite of tests has been executed and assessed, we can report on the test results. A good test report helps all the project stakeholders understand where the project stands relative to the release gate. Chapter 1 - The Acceptance Process provides details on what information might affect this decision. Test reporting includes both test status reporting to indicate how much test effort remains and test effectiveness reporting which describes our level of confidence in our tests.

Test Actioning

The purpose of executing tests is to learn about the quality of our product so that we can make intelligent decisions about whether it is ready for use or requires further development or testing. The Acceptance Process describes the process for deciding whether or not to accept the software but before we can make that decision we may need to fix some of the defects we have found. The Bug Triaging process is used to make the “Is it good enough?” decision by determining which bugs need to be fixed before we can release. (See the “Doneness Model” for more details.)

Test Maintenance

Some tests are only run once while others may need to be run many times over long periods of time. Tests that will be run more than once may require maintenance between runs as a result of changes to the parts of the system-under-test that they interact with (for example, the database state). These kinds of tests may warrant more of an upfront investment to ensure that they are repeatable and robust. Tests intended for manual execution will need to be updated whenever the parts of the system being tested undergo significant changes in functionality. Whereas human testers can usually work around minor changes, fully automated tests will typically be impacted by even the smallest changes (with tests that interact with the system-under-test through the GUI being the most fragile) and therefore may require significantly more frequent maintenance.

End of Life

Sooner or later a test may no longer be worth executing. Perhaps the functionality it verifies has been removed from the system-under-test or maybe we have determined that the functionality is covered sufficiently well by other tests so that we no longer get much additional value from running this test. At this point the test has reached its end of life and no longer warrants either execution or maintenance.

Variations in Test Lifecycle Traversal

Some tests spend a lot of time in each state of the test lifecycle while others may pass through the states very quickly. For example, in automated functional testing we might spends weeks preparing a complex test, wait several weeks before we can first execute it, and then run it several times a day for many years. In contrast, during a single one hour exploratory manual testing session, the tester may conceive of several test conditions, design a test to explore them, learn something about the system-under-test, conceive several more test conditions and design tests to explore them also, all in the space of a few minutes. The automated tests will spend most of their lifetime in the maintenance state while exploratory tests are very ethereal; there isn’t a concrete representation that needs to be maintained.

Highly Compressed Test Lifecycle - Exploratory Testing

Exploratory testing is summarized by Cem Kaner as “Simultaneous test design and execution with an emphasis on learning”. From this description it should be clear that there is no clear separation of the various stages of the test lifecycle when doing exploratory testing. The tester learns about the system-under-test by using it and forming hypothesis about how it should behave. Based on these hypotheses the tester conceives of one or more test conditions to which they might subject the system-under-test. They rapidly design, in their mind’s eye, a test case they could use to achieve this. They exercise the test case and observe the result thereby forming more hypotheses which in turn lead to more test conditions. There is not an attempt made to have the test persist beyond the test session unless it revealed a bug. This removes the need to document and maintain the tests (though a good exploratory tester keeps a set of notes/journal of key points and discoveries during his or her testing); the obvious consequence is that exploratory testing is not intended to be very repeatable.
This process is very lightweight with very little overhead getting in the way of the tester interacting with the software. This makes it possible for exploratory testers to formulate a lot of hypotheses, test them and find a lot of bugs in a very short period of time. The tester takes notes as they go focussed primarily on the following outputs:

What functionality they have tested and their general impressions of what they have seen.
What bugs they found and what they had done to cause them.
What test conditions they had conceived that they were not able to get to. These may be used as the charter for a subsequent exploratory test session.
How much time was spent actually testing versus how much was spent getting ready. This information is useful when deciding what kind of power tools would make the exploratory tester more efficient in the future.Despite its somewhat chaotic appearance exploratory testing can be quite disciplined and methodical even though it is not very repeatable. Exploratory testing can range from completely unstructured to highly disciplined. The more disciplined forms of exploratory testing use a sequence of time-boxed test sessions to structure the testing activities.

Planning of exploratory testing consists of defining an initial list of high-level test conditions (as in “kinds of things we should test”) for use as test session charters and deciding how many test sessions to budget for executing the charters. Examples of charters might include the following:

Pretend that you are a hacker and try breaking into the system (a persona-based charter)
Try out variations of the invoicing workflow focussing on rejected items (a scenario-based charter)
Try using the user interface using only the keyboard (a device-based or persona-based charter).
Try scenarios where several users try accessing the same account at the same time. (a scenario-based charter often called “tug of war”.)

The test charters are prioritized before being scheduled via assignment to a tester executing a specific test session. Upon completion of the test session, the tester may recommend additional charters be added to the backlog of charters based on what they had learned about the system-under-test, business domain, users’ needs etc. Exploratory testing is often done in an iterative style with many of the test charters for later iterations being discovered during execution of the test sessions in earlier iterations. This allows exploratory testing to focus on the areas of the software that have been found to be the most suspicious rather than providing the same amount of effort for all areas regardless of the quality level actually observed. This, combined with the low overhead nature of the simultaneous test design and execution, is what allows exploratory testing to be such an effective way of finding the bugs we know must lurk in the software.

A Spread Out Test Lifecycle – Scripted Testing

In scripted testing, the average test lifecycle is much longer that in exploratory testing. The tests may be conceived as part of the test planning exercise or in more detailed test design activities. The actual tests are then documented or programmed, and potentially reviewed, often before the software is available for testing. The tests may require maintenance even before they are first executed against system-under-test if the design of the software has evolved since the tests were designed. Eventually, we determine the schedule for executing the tests and the tests are executed at the appropriate time (which may be weeks or even months later.) Any bugs we find are logged and the test results are reported to the stakeholders. The bugs are actioned, often weeks or even months after they were found. If the tests are to be repeated at a later time, usually against a subsequent version of the system-under-test, the tests may require maintenance to track changes in the system they test. Eventually, someone decides that this particular test is no longer adding any value and the test is abandoned.
This cycle could take anywhere for several days or week to many years. The test team for Microsoft Office prepares extensive automated test scripts to verify the behavior of features in applications like MS Word. The tests for a specific generation of the product (e.g. Word 2003) have a lifetime of over 10 years because of Microsoft’s commitment of 5 years of mainstream support and a further 5 years of limited extended support for each business and development software product, followed by a minimum of 1 year of self-help online support via the knowledge base [Ref - http://support.microsoft.com/lifecycle/]. New builds are typically created every week through this product lifetime and the tests are run against each new build. In this example the maintenance phase of the test dominates the individual test lifecycle. Microsoft patterns and practices team use continuous integration that potentially produces several builds a day with continuous automated test execution.

Intermediate Test Lifecycles – Hybrid Test Approaches

The two previous sections described the two extremes of test lifecycle duration. In practice, the test lifecycles can fit anywhere between these two extremes. There could also be a mix of test lifecycle durations even in the same testing session. For example, a manual tester could be following a detailed test script that was written months ago. They notice something odd that isn’t specifically related to the script and decide to go “off-script” to explore the oddity. In this off-script excursion they are doing simultaneous test design and execution (in other words, exploratory testing). At some point they may return to the original script after either confirming that the system is working properly or logging the bugs that they have found.

Practices for Assessing Software

In Chapter 16 – Planning for Acceptance we introduced many practices in the context of planning the readiness assessment and acceptance testing activities. Now it is time to look at the practices that we use while designing, executing and actioning the individual tests. At this point we focus on the practices and not on who does them; it really doesn’t matter whether they are done as part of readiness assessment or acceptance testing as the practices themselves are not changed by when and who does them. Volume II contains a collection of thumbnails and job aides for each practice described here with Volume III presenting sample artefacts of applying those practices in a project.

Test Conception Practices

There are quite a few practices for conceiving tests or test conditions. Some are more structured or formal than others. They all share the goal of creating an extensive to-do list for our subsequent test design and execution efforts. Most start with either requirements, whether functional or non-functional, or risks (concerns about something that might go wrong.) Some of the more common techniques include the following:

Risk-based test identification
Threat modeling
Heuristics or checklists
Use case based testing – Define tests based on specific use cases of the system-under-test (see the Functional Testing thumbnail.)
Business rule testing –Define tests for various combinations of values used as inputs to business rules or business algorithms.
Interface-based testing – Define tests based on the characteristics of the user interface (human-computer interface) or computer-computer interface protocol.
Scenario-based testing – using real-world usage scenarios to inspire the design of test cases.
Soap-opera testing – Using exaggerated real-world usage scenarios to inspire the design of test cases.
Model-based test generation – Building one or more models of key characteristics of the system-under-test and generating tests from the model(s).
Group Brainstorming.
Paired/collaborative testing – Working together to design better test cases.

Risk-Based Test Identification

In risk-based test identification we do risk modeling to identify areas relating to functional or non-functional aspects that we are concerned might not be implemented correctly or might have been adversely affected by changes to the functionality. We use this information to identify test conditions we want to ensure are verified by tests. A good example of a kind of test that might be identified through risk analysis is the fault insertion test. For example, the risk we identified was “The network connection fails.” We ask ourselves “Why would the network connection fail?” and come up with 3 different possible causes: Unplugged cable, network card failure, network card out of service due to maintenance activity in progress. These are three test conditions we would want to exercise against our system-under-test. Other forms of risk might relate to potential mistakes during software development. E.g. “This shipping charge algorithm is very complex.” This might cause us to define a large number of test conditions to verify various aspects of the algorithm based on the kinds of mistakes a developer would be likely to make. For example applying the various surcharges in the wrong order.

Threat Modeling

Another form of risk modeling is threat modeling for security. The potential threats identified by the threat model can lead us to choose from a set of security assurance practices. We might define specific penetration test scenarios to ensure the software repels attempts at penetration. We might conduct security reviews of the code base to ensure safe coding practices have been followed. We can decide to do Fuzz Testing to verify that the software cannot be compromised by injecting specially chosen data values via user input fields.

Use Case Based Test Identification

When we have business requirements defined in the form of use cases we can identify test conditions by enumerating all the possible paths through the use case and determining for each path what input value(s) would cause that path to be executed. Each path constitutes at least one test condition depending on how many distinct combinations of inputs should cause that path to be executed.

Interface-Based Test Identification

Another source of test conditions is the design of the interface though which the use case is exercised whether it be a user interface used by a human or an application programming interface (API) or messaging protocol (such as a web service) used by a computer. The interface may have very detailed design intricacies in addition to the elements required to exercise the use case. These intricacies are a rich source of test conditions. For example, a user interface may have pulldown lists for some input fields. Test conditions for these pull-down lists would include cases where there are no valid entries (empty list), a single valid entry (list of 1 item) and many valid entries (long lists of items.) Each of these test conditions warrants verification.

Business Rules and Algorithms

Business rules and business algorithms are another rich source for test conditions. For a rule that validates a user’s inputs we should identify a test condition for each kind of input that should be rejected. Rules that describe how the system makes decisions about what to do should result in at least one test condition for each possible outcome. Rules that describe how calculations should be done should result in at least one test condition for each form of calculation. For example, when calculating a graduated shipping charge with three different results based on the value of the shipment, we would identify at least one test condition for each of the graduated values.

Scenario-Based Test Identification

Scenario-based testing is the use of real-life scenarios to derive tests. There are various kinds of scenarios. The scenario-based testing thumbnail describes a wide range of scenario types this introduction touches on only a few of them. A common type of scenario-based test is the business workflow test. These tests exercise the system-under-test by identifying common and not-so-common end-to-end sequences of actions by the various users of the system as a particular work item is passed from person to person as it progresses towards successful completion or rejection. We can examine the workflow definitions looking for points in the workflow where decisions are made, and define sufficient workflow scenarios to ensure that each path out of each decision is covered.
A particularly interesting form of scenario test is the soap opera test in which the tester dreams up a particularly torturous scenario that takes the system-under-test from extreme situation to extreme situation. The name comes from its similarity to a soap opera television program which condenses many days, months and potentially years of extraordinary events in peoples’ lives into short melodramatic episodes. This form of test identification is good for thinking outside the box.
Scenarios about how software is installed by a purchaser could lead to identification of potential compatibility issues and the need for compatibility testing.

Model-Based Test Generation

In model-based test generation we build a domain or environmental model of the system-under-test’s desired behaviour (expressed in mathematical terms or in some other abstract notation) and use it to generate all the relevant test cases. For example, when testing a function that takes 4 parameters each with 3 possible values, we could generate 81 test conditions (333*3) by iterating through each value for each parameter. For large and complex systems, the number of such cases will be huge. Various “guiding” or selection techniques are used to reduce the test case space. To fully define the expected result, we would need to have an independent way to determine the expected return value, perhaps a Comparable System Test Oracle or a Heuristic Test Oracle. Generation of the tests from the model may be fully automated or manual.

Identifying Non-functional Tests

The test cases used to assess compliance to the non-functional requirements can be identified in much the same ways as those for functionality requirements with the main difference being how the requirements are discovered and enumerated.

Other Test Identification Practices

All of these practices can be applied by a single person working alone at their desk. But a single person can be biased or blind to certain kinds of test conditions; therefore, it is beneficial to involve several people in any test identification activities. One way to do this is to review the test conditions with at least one other person, a form of test review. Another practice is paired testing in which two people work together to identify the test conditions (or to write or execute tests.) We can also use group techniques like listing, brainstorming or card storming to involve larger groups of people [TabakaOnCollaboration].
All these practices can benefit from the judicious use of checklists and heuristics. These can be used to trigger thought processes that can identify additional tests or entire categories of tests. They are also useful when doing exploratory testing.

Test Authoring / Test Design Practices

Now that we have of list of things we want to test, our to-do list, we can get down to designing the test cases. A key test design decision is whether we will prepare a separate test case for each test condition or address many test conditions in a single test case. There is no single best way as it depends very much on how the tests will be executed. When human testers will be executing the test cases it makes a lot of sense to avoid excessive test environment setup overhead by testing many test conditions in a single test case. The human tester has the intelligence to analyse the impact of a failed test step and decide whether to continue executing the test script, abandon test execution or to work around the failed step. Automated tests are rarely this intelligent therefore it is advisable to test fewer test conditions per test case. In the most extreme case, typical when automating unit tests, each test case includes a single test condition.

Tests as Assets

Tests are assets (not liabilities) that need to be protected from loss or corruption. They should be managed with the same level of care and discipline that is used for managing the product code base. This means they should be stored in a version-controlled repository such as a source code management (SCM) system. The line up of tests that correspond to a particular line up of product code needs to be labelled or tagged in the same way the product code is labelled so that the tests the correspond to a specific product code build can be retrieved easily.

Tests as Documentation

Regardless of whether a test case will be executed by a person or a computer, the test case should be written in such a way that a person can understand it easily. This becomes critical for automated tests when the tests need to be maintained either because they have failed or because we are changing the expected behavior of the system-under-test and we need to modify all the tests for the changed functionality.
Many of the practices used for identifying test conditions carry through to test case design but the emphasis changes to enumerating the steps of the test case and determining what the expected outcome should be for each test. For use case tests we must enumerate the user actions that cause the particular path to be exercised. We also need to include steps to verify any observable outcomes as we execute the steps and a way of assessing whether the outcome matches our expectations. (See the section 17.3.5 on Result Assessment Practices1

Picking an Appropriate Level of Detail for Test Scripts

As we define the steps of our tests it is very important to ensure that the level of detail is appropriate for the kind of test we are writing. For example, in a business workflow test, each step of the test case should correspond to an entire use case. If we were to use the same test vocabulary and level of detail as in a use case test, our workflow tests would become exceedingly long and readers of the test would have a hard time understanding the test. This is a classic example of “not being able to see the forest for the trees.” Soap opera tests are written much like a workflow test except that the steps and circumstance are more extreme. Again, each step in the test should correspond to an entire use case.

Business Rule Testing

Business rules tests can be designed much like use case tests by interacting with the system under test via the user interface. If there are a lot of test conditions, it may make testing proceed much more quickly if we use a data-driven test approach wherein we enumerate each test condition as a row in a table where each column represents one of the inputs or outputs. Then we can simply write a parameterized test case that reads the rows from the table one at a time and exercises the system-under-test with those values. An even more effective approach requires more co-operation with the product development team because it involves interfacing the test case that reads the rows of the table directly to the internal component that implements the business rule (we call them “subcutaneous tests”). This approach results in automated tests that execute much faster than tests that exercise the software through the user interface. The tests can usually be run much earlier in the design phase of the project because they don’t even require the user interface to exist. These tests are much more robust because changes to the user interface don’t affect them. These tests are particularly well suited to test-driven development.

Model-Based Testing

A more sophisticated way of using models is to generate executable test cases that include input values, sequencing of calls and oracle information to check the results. In order to do that, the model must describe the expected behaviour of the system-under-test. Model building is complex but is the key. Once the model is built, a tool (based on some method or notation) is typically used to generate abstract test cases, which later get transformed into executable test scripts. Many such tools allow the tester to guide the test generation process to control the number of test cases produced or to focus on specific areas of the model.

Usability Testing

The design of usability testing requires an understanding of the goals of users who will be using the system-under- test as well as the goals of the usability testing itself and the practices to be used. The goals of testing will change from test session to test session and the practices will evolve as the project progresses. Early rounds of usability testing may be focused on getting the overall design right and will involve paper prototypes, storyboards, and “Wizard of Oz” testing. Later rounds of testing are more likely to involve testing real software with the purpose being to fine tune the details of the user interface and user interaction. All of these tests, however, should be based on the usage models defined as part of User Modeling and Product Design practices.

Operational Testing

Functional requirements tend to focus on the needs of the end users but there are other stakeholders who have requirements for the system. The needs of the operations department, the people who support the software as it is being used, need to be verified as part of the acceptance process. The specific needs may vary from case to case but common forms of operational acceptance testing include:

Testing of installers, uninstallers and software updates.
Testing of batch jobs for initial data loading
Testing of data reformatting for updated software.
Testing of data repair functionality.
Testing of start up scripts and shutdown scripts.
Testing of integration with system monitoring frameworks.
Testing of administrative functionality such as user management.
Testing documentation content.

Reducing the Number of Tests Needed

If we have too many test conditions to be able test each one, we can use various reduction techniques to reduce the number of test conditions we must verify. When we have a large set of possible inputs to verify, we can reduce the number of test cases we need to execute by grouping the input values into equivalence classes based on the expected behavior of the system-under-test. That is, an equivalence class includes all the inputs for which the system-under-test should exhibit the same or equivalent behavior (including both end state and outputs.) We then select a few representative input values from each equivalence class for use in our tests. When the values are numeric and ordered (e.g., integers or reals) we pick the values right at the boundaries between the different behaviors, a technique known as boundary values analysis (BVA). When they are nominal , i.e., they represent classification of behaviours with no natural ordering (e.g., a finite set of artbitrary strings or enumeration types with no meaningful ordering in the context), we can design a single data-driven test for each equivalence class and run the test for each input value in the class.
If we have several inputs that can each vary and we suspect that the behavior of the system based on the individual inputs is not independent, we avoid testing all combinations of input values by using combinatorial test optimization to reduce the number of distinct combinations we test. Examples include:

Many independent variations or exception paths in a use case.
Many different paths through a state model.
Algorithms that take many independent input values that each affect the expected outcome.
Many system configurations that should all behave the same way.

The most common variation of combinatorial test optimization is known as Pairwise or All-Pairs testing ; it involves picking the smallest set of values that ensure that every input value is paired with every other value at least once. This technique is used so frequently that there is a website[PairWiseOnAllPairsTesting] dedicated to listing the many open source and commercial programs that exist to help us pick the values to use. It has been shown that pairwise testing provides better coverage and results in fewer tests than random testing.

Test Execution Practices

The details of how the tests are executed vary greatly across the different kinds of tests but several things are common. A key outcome of test execution is the data that will be used as input to the subsequent readiness and acceptance decisions. Though various project contexts will require much more extensive record-keeping than others, it is reasonable to expect a minimum level of record keeping. This record-keeping consists of the following key pieces of information:

What tests have been run, by whom, when and where?
What results were observed?
How did they compare to what was expected?
What bugs have been found and logged?

The comparison with expectations is described in the section Result Assessment Practices1

Functional Test Execution Practices

The nature of a test determines how we execute it. Dynamic tests involve running the system-under-test while static tests involve inspecting various artefacts that describe the system. Dynamic tests typically fall into one of the following categories:

Automated Functional Test Execution – This involves using computer programs to run the tests without any human intervention. The test automation tool sets up the test environment, runs the tests, assesses the results and reports them. It may even include logging of any bugs detected. The tests may be started by a human or by an automated test scheduler or started automatically when certain conditions are satisfied such as changes to the code base.
Manual test execution – This involves a person executing the test. The person may do all steps manually or may use some automation as power tools to make the testing go faster. The human may adjust the nature of the test as they execute it based on observed behavior or they may execute the steps of a test case exactly as described.
Exploratory test execution – This is a form of manual test execution that gives the tester much more discretion regarding exactly what steps to carry out while testing. Each test session is usually scoped using a test charter.

Non-functional Test Execution Practices

Non-functional tests are somewhat different from functional tests in several ways. As intimated by their name, para-funcational tests span specific functions of the system. Static non-functional tests may involve running tools that analyse the software in question or they many involve reviewers who look at the code or other artefacts to find potential design or coding defects. Dynamic non-functional tests may involve running tools that interact with the system under test to determine its behavior in various circumstances.

Static Non-functional Test Practices

Static analysis is done by examining the code or higher level models of the system to understand certain characteristics. Specific forms of static analysis include the following:
A Design/Architecture Review is used to examine higher-level models of the system to understand certain characteristics. The most common characteristics of interest include capacity/scalability, response time and compliance with standards (internal and external.)
A Security Review is a specialized form of Design or Archiecture review that involves examining the architecture and the code looking for ways a malicious user or program might be able to break into the system.
Static Code Analysis involves examining the source code either manually or using tools to ascertain certain characteristics including:

Reachability of code segments
Correct usage of key language constructs such as type safety

Dynamic Non-functional Test Practices

Performance and stress tests are good examples of non-functional tests that require specialized tools. Sometimes we have complex or long-running test procedures that exercise the system-under-test just to see what will happen; there need not be enumerated expectations per se.

Result Assessment Practices

The value in executing a test case is to determine whether or not some requirement has been satisfied. While a single test cannot prove the requirement has been met, a single failing test can certainly prove that it has not been met completely. Therefore, most test cases include some form of assessment or checking of actual results against what we expect. This assessment can happen in real time as the test is executed or it may be done after the fact. This choice is a purely pragmatic decision based on the relative ease of one approach versus the other. There are the following three basic approaches to verifying whether the actual result is acceptable.

Compare the actual result with a predetermined expected result using a comparator which may either be a person or a computerized algorithm.
Examine the actual result for certain properties that a valid result must exhibit. This is done using a verifier.
Just run the tests and not check the results at all. This may be appropriate when the testing is being conducted expressly to gather data. For example, the purpose of usability testing is to find out what kinds of issues potential users have using the product. We wouldn’t report an individual usability test session as having failed or succeeded. Rather, we aggregate the findings of all the usability test sessions for a specific piece of functionality to determine whether the design of the system-under-test needs to be changed to improve usability.

Using Comparators to Determine Test Results

The most convenient form of assessment is when we can predict what the actual results should be in a highly deterministic fashion. The mechanism that generates the expected result is sometimes called a “true oracle” and there are several ways the test results can be generated. Tests that have a true oracle are very useful when doing Acceptance Test Driven Development because there is a clear definition of “what done looks like.”
When there is an existing system with similar functionality and we expect the new system to produce exactly the same results, it may be convenient to use the existing system as a Comparable System Test Oracle.
When we have a new system for which no comparable system exists, we often have to define the expected results manually. This is known as a Hand-crafted Test Oracle. Once the system is up and running we may also have the option of comparing subsequent releases of the software with previous versions, an approach we call Previous Result Test Oracle. This is the approach that most “record and playback” or “capture / replay” test tools use. In some cases it may be appropriate to forgo the Hand-Crafted Test Oracle and wait for the system to generate results which are then inspected by a person, a Human Test Oracle, before being used as a Previous Result Test Oracle. This approach forgoes the benefits of Acceptance Test Driven Development.

Using Verifiers to Determine Test Results

When we cannot predetermine exactly what the expected result should be, we can instead examine the actual result for certain characteristics. Rather than have an oracle describe what the result should be, we ask the oracle to make a judgement as to whether the result is reasonable given the input(s). This approach has the advantage of not requiring that we predetermine the expected result for each potential input. The main disadvantage is that we may accept some results that satisfy the invariant but which are not actually correct.
For example, a Human Test Oracle could examine a generated graphic to determine whether they can recognize it as an acceptable rendering of an underlying model. Or a computer algorithm could verify that when the actual result is fed into another algorithm the original input value is recovered. The program that implements this algorithm is sometimes called a Heuristic Test Oracle. Heuristic Test Oracles may be able to verify some results are exactly correct while for other results it may only be able to verify they are approximately correct.
For example, we could write a test script that steps through all integers to verify that the square root function returns the right value. Rather than inspect the actual values returned by each function call and compare them to a hand-crafted test oracle or a comparable system oracle, we could instead multiply the result by itself and verify that we get back the original number within a specified tolerance, say, +/- .001. In this example, we should also test against another invariant to ensure that the actual result is not negative which would clearly be a test failure. A computerized heuristic oracle is sometimes called a verifier.

Logging Bugs

One of the key reasons for testing and reviews is to find differences between expectations and reality. When we do find such a difference it is important that it be logged so that it can be further investigated, prioritized and the appropriate action determined. The investigation could reveal it to be a bug, a misunderstanding by testers about how the system was intended to be used, missing documentation, or any of a myriad of other reasons. To avoid presuming the outcome, we prefer to call these concerns. To allow the investigation to be conducted efficiently, it is important to log the key characteristics of each concern. At a minimum, we need to log the following:

The exact steps required to reproduce the problem. This may require rerunning the test a number of times until we can confidently describe exactly what it takes to cause the problem to occur.
What actually occurred.
What we expected to happen. We should provide as much detail as would be necessary for the reader to understand. We should not just refer to a requirement but rather describe exactly what we expected, what happened, and how what actually occurred was different from what we expected.Every potential bug report should be given a clear title that describes specifically what was tested; this avoids confusion between bugs with very similar titles yet completely different expected and actual behaviours.

Refer to [Kaner et al, Chapter. Reporting and Analyzing Bugs.] on bug reporting guidance and http://blogs.msdn.com/productfeedback/archive/2004/09/27/235003.aspx for an additional example.

Test Maintenance Practices

Some tests are only intended to be run once while some are intended to be run many times over a long period of time. Some kinds of tests hold their value longer than others; some kinds of tests deteriorate very quickly because they are so tightly coupled to the SUT that even small changes to the SUT make them obsolete. Tests that are expected to be used more than once may warrant an upfront investment to ensure that they are repeatable and robust.
Useful techniques for making tests more robust include the following:

Build maintainability into the tests. Write the tests at appropriate level of abstraction. Don’t couple the test to any part of the system it isn’t testing. Don’t provide any unnecessary or irrelevant detail in any of the steps of the test. Strive to describe the test steps in business rather than technical terms.
Design the system-under-test for testability. Designing for testability is common practice in computer hardware but is too often neglected in software design. Design the system to make it easy to put it into a specific state. Make it easy for test programs to interface with the system through application programming interfaces (API) rather than forcing programs to use interfaces intended for humans. Writing the tests before the system is designed is a good way to influence the design to support testability.
Refactor the tests to improve maintainability. Tests should be assets, not liabilities. Tests that are hard to understand are hard to maintain when the system-under-test is modified to meet changing requirements. Automated tests in particular should be refactored to avoid unnecessary duplication and irrelevant information. See the Test Evolution, Refactoring and Maintenance thumbnail.

Summary

This chapter introduced the practices we use while defining, executing and maintaining individual tests. While some of these practices are specific to functional testing and others are specific to non-functional testing, the overall lifecycle of a test is consistent for both categories of tests. The practices used for readiness assessment are more or less the same as those used for acceptance testing although the specific tests produced for each will depend on the overall test plan as described in Chapter 16 – Planning for Acceptance.

What’s Next?

The next chapter describes practices related to managing the acceptance process, especially how it relates to monitoring and reporting on test progress and the results and managing the process of deciding which bugs to fix.

Resources

[PairWiseOnAllPairsTesting] Pairwise.Org - A website dedicated to cataloging all the tools available for allpairs testing. http://www.pairwise.org/tools.asp
[TabakaOnCollaboration] Tabaka, Jean “Collaboration Explained” Addison Wesley. NJ Addison Wesley. NJ [Kaner et al] Kaner, C. et al. Testing Computer Software, 2/e. Wiley, 1999.

Updated Wiki: Vol1-3-Chapter-18

rburte — Sat, 07 Nov 2009 01:32:50 GMT

Chapter 18 - Assessing Software

The Lifecycle of an Individual Test

Every single test, however simple or complex, whether manual or automated, goes through a number of stages during its lifetime. This lifecycle is illustrated in Figure 1.
figure:Figure 1
Figure 1 Individual Test Lifecycle
The states of the lifecycle are:
Note that test conception and test authoring are often lumped together under the label of test design. Some forms of testing, often called static testing, involve inspecting artefacts that describe the system under test rather than running the actual code. The terminology used for these forms of testing is somewhat different (for example. they are often called reviews rather than tests) but for the purpose of discussing test lifecycle, we shall use the test terminology.
Let’s examine each of these states in a bit more detail.

Test Conception

Test Authoring / Test Design

Test Scheduling

Test Execution

Result Assessment

Test Reporting

Test Actioning

Test Maintenance

End of Life

Variations in Test Lifecycle Traversal

Highly Compressed Test Lifecycle - Exploratory Testing

What functionality they have tested and their general impressions of what they have seen.
What bugs they found and what they had done to cause them.
What test conditions they had conceived that they were not able to get to. These may be used as the charter for a subsequent exploratory test session.
How much time was spent actually testing versus how much was spent getting ready. This information is useful when deciding what kind of power tools would make the exploratory tester more efficient in the future.Despite its somewhat chaotic appearance exploratory testing can be quite disciplined and methodical even though it is not very repeatable. Exploratory testing can range from completely unstructured to highly disciplined. The more disciplined forms of exploratory testing use a sequence of time-boxed test sessions to structure the testing activities.

Pretend that you are a hacker and try breaking into the system (a persona-based charter)
Try out variations of the invoicing workflow focussing on rejected items (a scenario-based charter)
Try using the user interface using only the keyboard (a device-based or persona-based charter).
Try scenarios where several users try accessing the same account at the same time. (a scenario-based charter often called “tug of war”.)

A Spread Out Test Lifecycle – Scripted Testing

In scripted testing, the average test lifecycle is much longer that in exploratory testing. The tests may be conceived as part of the test planning exercise or in more detailed test design activities. The actual tests are then documented or programmed, and potentially reviewed, often before the software is available for testing. The tests may require maintenance even before they are first executed against system-under-test if the design of the software has evolved since the tests were designed. Eventually, we determine the schedule for executing the tests and the tests are executed at the appropriate time (which may be weeks or even months later.) Any bugs we find are logged and the test results are reported to the stakeholders. The bugs are actioned, often weeks or even months after they were found. If the tests are to be repeated at a later time, usually against a subsequent version of the system-under-test, the tests may require maintenance to track changes in the system they test. Eventually, someone decides that this particular test is no longer adding any value and the test is abandoned.
This cycle could take anywhere for several days or week to many years. The test team for Microsoft Office prepares extensive automated test scripts to verify the behavior of features in applications like MS Word. The tests for a specific generation of the product (e.g. Word 2003) have a lifetime of over 10 years because of Microsoft’s commitment of 5 years of mainstream support and a further 5 years of limited extended support for each business and development software product, followed by a minimum of 1 year of self-help online support via the knowledge base [Ref - http://support.microsoft.com/lifecycle/]. New builds are typically created every week through this product lifetime and the tests are run against each new build. In this example the maintenance phase of the test dominates the individual test lifecycle. Microsoft patterns and practices team use continuous integration that potentially produces several builds a day with continuous automated test execution.

Intermediate Test Lifecycles – Hybrid Test Approaches

Practices for Assessing Software

In Chapter 16 – Planning for Acceptance we introduced many practices in the context of planning the readiness assessment and acceptance testing activities. Now it is time to look at the practices that we use while designing, executing and actioning the individual tests. At this point we focus on the practices and not on who does them; it really doesn’t matter whether they are done as part of readiness assessment or acceptance testing as the practices themselves are not changed by when and who does them. Volume II contains a collection of thumbnails and job aides for each practice described here with Volume III presenting sample artefacts of applying those practices in a project.

Test Conception Practices

Risk-based test identification
Threat modeling
Heuristics or checklists
Use case based testing – Define tests based on specific use cases of the system-under-test (see the Functional Testing thumbnail.)
Business rule testing –Define tests for various combinations of values used as inputs to business rules or business algorithms.
Interface-based testing – Define tests based on the characteristics of the user interface (human-computer interface) or computer-computer interface protocol.
Scenario-based testing – using real-world usage scenarios to inspire the design of test cases.
Soap-opera testing – Using exaggerated real-world usage scenarios to inspire the design of test cases.
Model-based test generation – Building one or more models of key characteristics of the system-under-test and generating tests from the model(s).
Group Brainstorming.
Paired/collaborative testing – Working together to design better test cases.

Risk-Based Test Identification

Threat Modeling

Use Case Based Test Identification

Interface-Based Test Identification

Business Rules and Algorithms

Scenario-Based Test Identification

Model-Based Test Generation

Identifying Non-functional Tests

Other Test Identification Practices

Test Authoring / Test Design Practices

Tests as Assets

Tests as Documentation

Picking an Appropriate Level of Detail for Test Scripts

Business Rule Testing

Model-Based Testing

Usability Testing

Operational Testing

Testing of installers, uninstallers and software updates.
Testing of batch jobs for initial data loading
Testing of data reformatting for updated software.
Testing of data repair functionality.
Testing of start up scripts and shutdown scripts.
Testing of integration with system monitoring frameworks.
Testing of administrative functionality such as user management.
Testing documentation content.

Reducing the Number of Tests Needed

Many independent variations or exception paths in a use case.
Many different paths through a state model.
Algorithms that take many independent input values that each affect the expected outcome.
Many system configurations that should all behave the same way.

Test Execution Practices

What tests have been run, by whom, when and where?
What results were observed?
How did they compare to what was expected?
What bugs have been found and logged?

The comparison with expectations is described in the section Result Assessment Practices1

Functional Test Execution Practices

Automated Functional Test Execution – This involves using computer programs to run the tests without any human intervention. The test automation tool sets up the test environment, runs the tests, assesses the results and reports them. It may even include logging of any bugs detected. The tests may be started by a human or by an automated test scheduler or started automatically when certain conditions are satisfied such as changes to the code base.
Manual test execution – This involves a person executing the test. The person may do all steps manually or may use some automation as power tools to make the testing go faster. The human may adjust the nature of the test as they execute it based on observed behavior or they may execute the steps of a test case exactly as described.
Exploratory test execution – This is a form of manual test execution that gives the tester much more discretion regarding exactly what steps to carry out while testing. Each test session is usually scoped using a test charter.

Non-functional Test Execution Practices

Static Non-functional Test Practices

Reachability of code segments
Correct usage of key language constructs such as type safety

Dynamic Non-functional Test Practices

Result Assessment Practices

Compare the actual result with a predetermined expected result using a comparator which may either be a person or a computerized algorithm.
Examine the actual result for certain properties that a valid result must exhibit. This is done using a verifier.
Just run the tests and not check the results at all. This may be appropriate when the testing is being conducted expressly to gather data. For example, the purpose of usability testing is to find out what kinds of issues potential users have using the product. We wouldn’t report an individual usability test session as having failed or succeeded. Rather, we aggregate the findings of all the usability test sessions for a specific piece of functionality to determine whether the design of the system-under-test needs to be changed to improve usability.

Using Comparators to Determine Test Results

Using Verifiers to Determine Test Results

Logging Bugs

The exact steps required to reproduce the problem. This may require rerunning the test a number of times until we can confidently describe exactly what it takes to cause the problem to occur.
What actually occurred.
What we expected to happen. We should provide as much detail as would be necessary for the reader to understand. We should not just refer to a requirement but rather describe exactly what we expected, what happened, and how what actually occurred was different from what we expected.Every potential bug report should be given a clear title that describes specifically what was tested; this avoids confusion between bugs with very similar titles yet completely different expected and actual behaviours.

Refer to [Kaner et al, Chapter. Reporting and Analyzing Bugs.] on bug reporting guidance and http://blogs.msdn.com/productfeedback/archive/2004/09/27/235003.aspx for an additional example.

Test Maintenance Practices

Build maintainability into the tests. Write the tests at appropriate level of abstraction. Don’t couple the test to any part of the system it isn’t testing. Don’t provide any unnecessary or irrelevant detail in any of the steps of the test. Strive to describe the test steps in business rather than technical terms.
Design the system-under-test for testability. Designing for testability is common practice in computer hardware but is too often neglected in software design. Design the system to make it easy to put it into a specific state. Make it easy for test programs to interface with the system through application programming interfaces (API) rather than forcing programs to use interfaces intended for humans. Writing the tests before the system is designed is a good way to influence the design to support testability.
Refactor the tests to improve maintainability. Tests should be assets, not liabilities. Tests that are hard to understand are hard to maintain when the system-under-test is modified to meet changing requirements. Automated tests in particular should be refactored to avoid unnecessary duplication and irrelevant information. See the Test Evolution, Refactoring and Maintenance thumbnail.

Summary

What’s Next?

Resources

Updated Wiki: Vol1-3-Chapter-18

rburte — Sat, 07 Nov 2009 01:31:23 GMT

Chapter 18 - Assessing Software

The Lifecycle of an Individual Test

Every single test, however simple or complex, whether manual or automated, goes through a number of stages during its lifetime. This lifecycle is illustrated in Figure 1.
Figure 1 Individual Test Lifecycle
The states of the lifecycle are:
Note that test conception and test authoring are often lumped together under the label of test design. Some forms of testing, often called static testing, involve inspecting artefacts that describe the system under test rather than running the actual code. The terminology used for these forms of testing is somewhat different (for example. they are often called reviews rather than tests) but for the purpose of discussing test lifecycle, we shall use the test terminology.
Let’s examine each of these states in a bit more detail.

Test Conception

Test Authoring / Test Design

Test Scheduling

Test Execution

Result Assessment

Test Reporting

Test Actioning

Test Maintenance

End of Life

Variations in Test Lifecycle Traversal

Highly Compressed Test Lifecycle - Exploratory Testing

What functionality they have tested and their general impressions of what they have seen.
What bugs they found and what they had done to cause them.
What test conditions they had conceived that they were not able to get to. These may be used as the charter for a subsequent exploratory test session.
How much time was spent actually testing versus how much was spent getting ready. This information is useful when deciding what kind of power tools would make the exploratory tester more efficient in the future.Despite its somewhat chaotic appearance exploratory testing can be quite disciplined and methodical even though it is not very repeatable. Exploratory testing can range from completely unstructured to highly disciplined. The more disciplined forms of exploratory testing use a sequence of time-boxed test sessions to structure the testing activities.

Pretend that you are a hacker and try breaking into the system (a persona-based charter)
Try out variations of the invoicing workflow focussing on rejected items (a scenario-based charter)
Try using the user interface using only the keyboard (a device-based or persona-based charter).
Try scenarios where several users try accessing the same account at the same time. (a scenario-based charter often called “tug of war”.)

A Spread Out Test Lifecycle – Scripted Testing

In scripted testing, the average test lifecycle is much longer that in exploratory testing. The tests may be conceived as part of the test planning exercise or in more detailed test design activities. The actual tests are then documented or programmed, and potentially reviewed, often before the software is available for testing. The tests may require maintenance even before they are first executed against system-under-test if the design of the software has evolved since the tests were designed. Eventually, we determine the schedule for executing the tests and the tests are executed at the appropriate time (which may be weeks or even months later.) Any bugs we find are logged and the test results are reported to the stakeholders. The bugs are actioned, often weeks or even months after they were found. If the tests are to be repeated at a later time, usually against a subsequent version of the system-under-test, the tests may require maintenance to track changes in the system they test. Eventually, someone decides that this particular test is no longer adding any value and the test is abandoned.
This cycle could take anywhere for several days or week to many years. The test team for Microsoft Office prepares extensive automated test scripts to verify the behavior of features in applications like MS Word. The tests for a specific generation of the product (e.g. Word 2003) have a lifetime of over 10 years because of Microsoft’s commitment of 5 years of mainstream support and a further 5 years of limited extended support for each business and development software product, followed by a minimum of 1 year of self-help online support via the knowledge base [Ref - http://support.microsoft.com/lifecycle/]. New builds are typically created every week through this product lifetime and the tests are run against each new build. In this example the maintenance phase of the test dominates the individual test lifecycle. Microsoft patterns and practices team use continuous integration that potentially produces several builds a day with continuous automated test execution.

Intermediate Test Lifecycles – Hybrid Test Approaches

Practices for Assessing Software

In Chapter 16 – Planning for Acceptance we introduced many practices in the context of planning the readiness assessment and acceptance testing activities. Now it is time to look at the practices that we use while designing, executing and actioning the individual tests. At this point we focus on the practices and not on who does them; it really doesn’t matter whether they are done as part of readiness assessment or acceptance testing as the practices themselves are not changed by when and who does them. Volume II contains a collection of thumbnails and job aides for each practice described here with Volume III presenting sample artefacts of applying those practices in a project.

Test Conception Practices

Risk-based test identification
Threat modeling
Heuristics or checklists
Use case based testing – Define tests based on specific use cases of the system-under-test (see the Functional Testing thumbnail.)
Business rule testing –Define tests for various combinations of values used as inputs to business rules or business algorithms.
Interface-based testing – Define tests based on the characteristics of the user interface (human-computer interface) or computer-computer interface protocol.
Scenario-based testing – using real-world usage scenarios to inspire the design of test cases.
Soap-opera testing – Using exaggerated real-world usage scenarios to inspire the design of test cases.
Model-based test generation – Building one or more models of key characteristics of the system-under-test and generating tests from the model(s).
Group Brainstorming.
Paired/collaborative testing – Working together to design better test cases.

Risk-Based Test Identification

Threat Modeling

Use Case Based Test Identification

Interface-Based Test Identification

Business Rules and Algorithms

Scenario-Based Test Identification

Model-Based Test Generation

Identifying Non-functional Tests

Other Test Identification Practices

Test Authoring / Test Design Practices

Tests as Assets

Tests as Documentation

Picking an Appropriate Level of Detail for Test Scripts

Business Rule Testing

Model-Based Testing

Usability Testing

Operational Testing

Testing of installers, uninstallers and software updates.
Testing of batch jobs for initial data loading
Testing of data reformatting for updated software.
Testing of data repair functionality.
Testing of start up scripts and shutdown scripts.
Testing of integration with system monitoring frameworks.
Testing of administrative functionality such as user management.
Testing documentation content.

Reducing the Number of Tests Needed

Many independent variations or exception paths in a use case.
Many different paths through a state model.
Algorithms that take many independent input values that each affect the expected outcome.
Many system configurations that should all behave the same way.

Test Execution Practices

What tests have been run, by whom, when and where?
What results were observed?
How did they compare to what was expected?
What bugs have been found and logged?

The comparison with expectations is described in the section Result Assessment Practices1

Functional Test Execution Practices

Automated Functional Test Execution – This involves using computer programs to run the tests without any human intervention. The test automation tool sets up the test environment, runs the tests, assesses the results and reports them. It may even include logging of any bugs detected. The tests may be started by a human or by an automated test scheduler or started automatically when certain conditions are satisfied such as changes to the code base.
Manual test execution – This involves a person executing the test. The person may do all steps manually or may use some automation as power tools to make the testing go faster. The human may adjust the nature of the test as they execute it based on observed behavior or they may execute the steps of a test case exactly as described.
Exploratory test execution – This is a form of manual test execution that gives the tester much more discretion regarding exactly what steps to carry out while testing. Each test session is usually scoped using a test charter.

Non-functional Test Execution Practices

Static Non-functional Test Practices

Reachability of code segments
Correct usage of key language constructs such as type safety

Dynamic Non-functional Test Practices

Result Assessment Practices

Compare the actual result with a predetermined expected result using a comparator which may either be a person or a computerized algorithm.
Examine the actual result for certain properties that a valid result must exhibit. This is done using a verifier.
Just run the tests and not check the results at all. This may be appropriate when the testing is being conducted expressly to gather data. For example, the purpose of usability testing is to find out what kinds of issues potential users have using the product. We wouldn’t report an individual usability test session as having failed or succeeded. Rather, we aggregate the findings of all the usability test sessions for a specific piece of functionality to determine whether the design of the system-under-test needs to be changed to improve usability.

Using Comparators to Determine Test Results

Using Verifiers to Determine Test Results

Logging Bugs

The exact steps required to reproduce the problem. This may require rerunning the test a number of times until we can confidently describe exactly what it takes to cause the problem to occur.
What actually occurred.
What we expected to happen. We should provide as much detail as would be necessary for the reader to understand. We should not just refer to a requirement but rather describe exactly what we expected, what happened, and how what actually occurred was different from what we expected.Every potential bug report should be given a clear title that describes specifically what was tested; this avoids confusion between bugs with very similar titles yet completely different expected and actual behaviours.

Refer to [Kaner et al, Chapter. Reporting and Analyzing Bugs.] on bug reporting guidance and http://blogs.msdn.com/productfeedback/archive/2004/09/27/235003.aspx for an additional example.

Test Maintenance Practices

Build maintainability into the tests. Write the tests at appropriate level of abstraction. Don’t couple the test to any part of the system it isn’t testing. Don’t provide any unnecessary or irrelevant detail in any of the steps of the test. Strive to describe the test steps in business rather than technical terms.
Design the system-under-test for testability. Designing for testability is common practice in computer hardware but is too often neglected in software design. Design the system to make it easy to put it into a specific state. Make it easy for test programs to interface with the system through application programming interfaces (API) rather than forcing programs to use interfaces intended for humans. Writing the tests before the system is designed is a good way to influence the design to support testability.
Refactor the tests to improve maintainability. Tests should be assets, not liabilities. Tests that are hard to understand are hard to maintain when the system-under-test is modified to meet changing requirements. Automated tests in particular should be refactored to avoid unnecessary duplication and irrelevant information. See the Test Evolution, Refactoring and Maintenance thumbnail.

Summary

What’s Next?

Resources

Updated Wiki: Vol1-3-Chapter-17

rburte — Sat, 07 Nov 2009 01:26:08 GMT

Chapter 17 - Planning for Acceptance

Part I introduced several models to help us think about the role of acceptance testing in the overall context of the software development lifecycle. It also introduced the abstract roles of the people involved in making the acceptance and readiness decisions. This chapter introduces practices used by one or more of those roles to plan the activities that will provide the data on which the acceptance decision maker(s) can base their decision.
Acceptance is an important part of the life cycle of a product; it is important enough that it should be the result of a carefully thought through process. The test plan is the end result of all this thinking. Like most such documents, it can serve an important role in communicating the plans for testing, but the real value lies in the thinking that went into producing it.
Test planning builds on the work done during project chartering, which defines the initial project scope. In test planning, you we define the scope of the testing that will be done, select the test strategy, and drill down to detailed testing plans that define who will do what, when, and where.
Most projects prepare a test plan that lays out the following, among other things:

The scope of the acceptance process and the breakdown into readiness assessment and acceptance testing phases
The overall test strategy including both manual and automated testing
The activities, testing and otherwise, that will be performed in each phase (the readiness phase and the acceptance phase)
The skills that are required to perform the activities
The other resources that will be utilized to carry out the activities (such as facilities and equipment)
The timeframe and sequence, if relevant, in which each activity will be performed

Defining Test Objectives

We need to have a clear understanding of the scope of the project and the software before we start thinking about how we will accept the software. Most organizations have some kind of project chartering activity that defines the product vision or scope. It may also include a risk assessment activity. One form of risk assessment activity involves brainstorming all the potentially negative events that could cause grief for the project. Some of the commonly occurring risks that we may address through testing are:

The Product Development Team misinterpreted the Product Owners description resulting in a product that behaves differently from expected.
The Product Owners description of the product left out important behaviors resulting in missing functionality.
The Product Development Team implemented the behaviors poorly resulting in frequent crashes of the product.
The product exhibits the right behavior when used by small numbers of users but behaves erratically or responds slowly when the number of users or transactions increases to expected levels.

For each possible event we classify each of the likelihood and the impact as low, medium, or high. Anything ranked medium/high or high/high needs to be addressed. Some risks may cause us to change the way we plan our project, but other risks may cause us to take on specific test planning activities. Together, the vision/scope and risk assessment help drive the test strategy definition and test planning.

Traditional Approaches to Testing

Traditionally, most products are tested from the users’ prespective. That is, once the product is finished the product’s behavior is verified via whatever interfaces the users will typically use. Often, very little if any testing is done of the parts of the system before it is assembled into the final product. This results in a distribution of tests that looks like Figure A – Inverted Test Pyramid.

Figure A Inverted Test Pyramid
This figure shows a high-level view of how various kinds of tests (and consequently, test effort) are typically distributed (based on the level of test granularity – from unit to component/subsystem to system/whole product) when testing is done primarily by testers using the same interfaces as end users. System testing is done at the level of the whole product and can include a range of system sizes: individual systems or integrated workflows through multiple systems that have to interact. Similarly, component tests are used for testing packages, individual executables or subsystems; while the granularity of unit testing can be on little things like methods/functions or bigger things like modules and classes. See the sidebard Testing Terminology Pitfalls for a discussion about what we call various kinds of tests.
The problem with the approach presented in Figure A (where most effort is focused on system testing and very little on unit testing) is that it is hard to verify that the software behaves in all possible circumstances because many of the circumstances are triggered inside the software. Even when this is not the case, much of logic within the system is cumbersome to verify via the end-user interface. This makes verifying the behavior and finding any bugs very expensive and time consuming. Most of the behavior can be verified and many of the bugs can be found much more cheaply through appropriate unit testing, component testing and subcutaneous system testing. These sorts of tests can and should be automated as will be discussed in section 17.2.31Automated Testing1below1
Note:
Sidebar: Testing Terminology PitfallsWhen preparing Figure A (and Figure 4 later in this chapter) we struggled with what to call the tests in the top layer of the upside down pyramid. Some of the alternatives we considered were: Acceptance Tests, Whole Product Tests, Customer Tests, Functional Tests, and Integration Tests. While these names are in common use to describe the tests that occupy the top layer, we weren’t happy with any of these names because they didn’t clearly convey the difference between the top layer and the layers below it. The problem with each of these historical names is that they often exist to help differentiate one kind of tests from another kind. For example, Functional Tests as opposed to what? The obvious answer is Non-functional Tests, but we feel non-functional tests also fit into the top layer of the (inverted) pyramid. If not Acceptance Tests then what? The logical alternative is not Rejection Tests but Regression Tests!! Acceptance tests helpe us verify new functionalty works while regression tests verify that exisiting functionality still works. The acceptance tests for this release are likely added to the regression test suite for the next release. So the terms acceptance and regression refer to points of differentiation on the time horizon.
Examining the potential names with this critical lens helped us eliminate many potential names leaving us only with those names that helped us focus on whether we were testing the fully assembled product or its constituent parts. This led us to Whole Product Test, System Test and Integration Test. Unfortunately, the term Integration Test can be used at various levels depending on whether we are integrating units, components, subsystems or systems (into a system of systems.) Whole product is probably the clearest name but many teams think of themselves as building solutios to business problems rather than products for sale. This led us to focus on System Tests and the contrast with Component Tests and Unit Tests.
There are similar issues in the terminology around how tests are prepared and executed (Exploratory vs Scripted testing) and how tests are automated (Hand-scripted or programmed vs. record/playback, scripted vs. data-driven vs. keyword driven, etc.) The moral of this sidebar is that we need to be careful to make sure we understand what someone means when they use one of these words to describe testing. They may well mean something entirely different from what we would have meant had we used the same word.

Flipping the Pyramid Right Side Up

The upside down test pyramid can be turned right side up by moving much of the testing activity to the unit and component level. But what kinds of testing can be push down?
The agile software development community has shown that it is possible to produce consistently high-quality software without significantly increasing the effort by integrating testing throughout the development life cycle. This has led to a rethinking of the role of testing (the activity) and of test teams.
Brian Marick, has defined a model [MarickOnTestingDimensions] (see Figure 1 ) that helps us understand the purpose behind the different kinds of tests we could execute.

Figure 1 Purpose of Tests
The diagram in Figure 1 classifies various types of testing we can do along two key dimensions:

Whether the tests are business-facing or technology-facing
Whether the tests are intended to support development (by helping them get it right) or to critique the product after it is built

Tests That Support Development

Tests can support development by helping the Product Development Team understand what the product is supposed to do before they build it. These are the tests that can be prepared in advance and run as we build the product. As part of the readiness assessment, the Product Development Team can run these tests to self-assess whether the product implements the necessary functionality.
The tests in this column fall into two categories: the business facing tests that describe what the product should do in terms understandable by the business or Product Owner and the technology-facing tests that describe how the software should work beneath the covers.

Business-Facing Tests

The business facing tests that drive development are the tests (also known as customer tests). These tests elaborate on the requirements and the very act of writing these tests can expose missing or ambiguous requirements. When we (Product Owner, often together with the Product Development Team) prepare these tests before development starts, we can be sure that the Product Development Team understands what they need to build. This is known as Acceptance Test–Driven Development.
If we prepare the system acceptance tests after development is complete or we prepare them in parallel with development and do not share them (this is referred to as "independent verification"), the tests do not help us build the right product; instead, the tests act as an alternative interpretation of the requirements. If they fail when we finally run them, we need to determine which interpretation of the requirements is more accurate: the one implemented by the development team in the code base or the one implemented in the functional tests by the test team. If the latter, time will be consumed while the development team reworks the code to satisfy this interpretation, rework that could have been avoided if we had shared the tests. Either way, we have set up an adversarial relationship between development and testing. It is highly preferable to prepare the tests before the software is built so that testing can help development understand what needs to be built rather than simply criticize what they have built.
These tests may be run manually or they may be automated. The latter allows the Product Development Team to run them throughout the development cycle to ensure that all specified functionality is correctly implemented. The Product Owner will want to run additional acceptance tests to make the final acceptance decision, but supplying a set of tests to the Product Development Team early so they can drive development goes a long way toward building the correct product. This is much more likely to happen when the tests are easy and cheap to run—and that requires automated execution (for more information, see the Automated Functional Test Execution thumbnail). These tests may be implemented as programmatic tests, but they are more typically implemented using Keyword-driven Test Automation.

Technology-Facing Tests

There are many tests used by product development that are not business-facing. Developers may prepare unit tests to verify that the code they wrote has successfully achieved the design intent. This is how they determine that they correctly built the code (as opposed to building the correct product). Test-driven development (TDD) is when developers implement automated unit tests before they build the code the tests verify. This development process has been shown to significantly improve the quality of the software in several ways including better software structure, reduced software complexity, and fewer defects found during acceptance testing [JeffriesMelnikOnTDDEffectiveness]. These tests are ever more frequently automated using members of the xUnit testing framework. For more information, see [MeszarosOnUnitTestPatterns]

Tests That Critique the Product

Assuming that the product has implemented the correct functionality, we need to know whether the product meets the non-functional requirements. These tests support the acceptance decision. We do this by assessing the non-functional attributes of the product after it has been (at least partially) built. These tests critique the product instead of driving the development process. They tell us whether it is good enough from a non-functional perspective. We can divide these tests into the two categories: business-facing and technology-facing.

Technology-Facing Tests

Technology facing tests that critique the product measure how well the product meets technically-oriented quality attributes (scalability, availability, self-consistency, etc.)
These tests provide metrics we can use when deciding whether the product is ready to be shipped. In most cases, these tests will be run as part of readiness assessment because of their technical nature. However, a Product Owner charged with deciding whether to accept a product may be interested in seeing the results and comparing them with the minimum requirement. They may even hire a third-party test lab to conduct the testing on their behalf. For more information, see the Test Outsourcing thumbnail.

Business-Facing Tests

If system acceptance tests are used to drive development to build the product per the requirements, how do we make sure we are building the right product? The business facing tests that critique the product fulfill this role. These tests assess the product (either as built or as proposed) for “fitness for purpose”. Usability tests are examples of tests that critique the product from a business perspective
Typically, these tests cannot be automated because they are highly subjective and some even require us to observe people trying to use the product to achieve their goals.

What Testing Will We Do? And Why?

Now that we have been introduced to a way of reasoning about the kinds of tests, we can decide the types of tests we need to run and which tests to automate. This is an overall test strategy that helps us determine how to best address our testing needs at the lowest cost.

Test Strategy

Defining the test strategy may be considered to be part of the test planning process or a distinct activity. Either way, the purpose of defining a test strategy is to make some high-level decisions about what kinds of testing need to be done and how they will be executed. Test strategy is tightly linked to the test objectives. For example, you would have a different test strategy if your objectives are to verify and accept functionality as opposed to verify compliance.
One of the key decisions is what kinds of tests should be automated and which approach to testing should be used for manual tests. The goal of these decisions is to try to minimize project risk while also minimizing the time and effort spent testing the software.
Previous sections introduced the concepts of functional and non-functional requirements. As part of the test strategy we need to decide where to focus. Clearly, testing cannot prove that software works correctly; it can only prove that it does not work correctly. Therefore, we could spend an infinite amount of time testing and still not prove the software is perfect. The test strategy is about maximizing the return on investment (ROI) of testing by identifying the testing activities that will mitigate the risks most effectively. This implies that some requirements may be tested less thoroughly, by choice.
We also need to decide whether we will do all the acceptance testing at the end of the project (in a separate testing phase, or in a test-last or “big bang testing” style) or incrementally as functionality becomes available. The latter approach, incremental acceptance requires changes in how the project is planned and how the software is developed to ensure a continuous stream of functionality is delivered starting fairly early in the project. This is the sytle that we strongly advocate since big-bang testing carries too many risks and is avoidable in many contexts. The payback of incremental acceptance is early discovery of misunderstood and missed requirements, thereby allowing time for remediation off the critical path of the project.
Another strategic decision may relate to test oracles; our source of truth. How will we define what a correct outcome looks like? Is there a comparable system that we can use as an oracle? (For more information, see Comparable System Test Oracle.) Can we hand-craft expected results? (For more information, see Hand-Crafted Test Oracle.) Or will we need to use a Human Test Oracle? If so, what can we do from a design-for-testability perspective to reduce the dependency on human test oracles?

Manual Testing Freedom

For functional testing, the key strategy decisions relate to how we will execute the tests. When we manually execute the tests, we need to decide how much freedom to grant the testers. Figure 2 contains the "freedom" scale used by Jon and James Bach to describe the choices we have.

Figure 2 "Freedom" scale for testers (Jon Bach, James Bach – used with permission)
At one extreme of the test freedom scale, there is freestyle exploratory testing, in which the testers can test whatever they think is important. At the other end of the scale is scripted testing, in which testers attempt to follow a well-defined test script. In between, there is chartered exploratory testing, which has charters of varying degrees of freedom including scenarios, user roles/personas, and charters. Scripted testing involves having a expert prepare test scripts to be executed much later by someone elseor a computer when automated). here is little opportunity for test design during test execution.
Exploratory testing is an effective and efficient approach to testing that leverages the intelligence of the tester to maximize the bugs found in a fixed amount of time. Unlike with scripted testing, testers are encouraged to conceive new things to try while they are executing tests. Some people describe it as “concurrent test case design and execution with an emphasis on learning”.

For more information on exploratory testing, see [KanerOnExploratoryTesting] and [BachOnExploratoryTesting].

Automated Testing

Automated testing covers a wide range of topics: from automated execution to automated test case generation. Some kinds of non-functional tests require automated execution because of the nature of the testing being performed. A commonly overlooked area for automation is the use of "power tools" while performing manual testing. Tools can also be used to generate test data or to do data inspection. The various uses of test automation need to be determined on a project-by-project basis. For more information about this process, see the Planning Test Automation thumbnail of this guide and [FewsterGrahamOnSoftwareTestAutomation].

Maximizing Automation ROI

An effective test automation strategy strives to maximize the ROI of the investment in automation. Therefore, the tests we automate should cost less, at least in the long run, than we would have spent manually executing the comparable tests. Some tests are so expensive to automate that we will never recoup the investment. These tests should be run manually.

To ensure that we get the best possible ROI for our test automation investment, we need to focus our energies on the following:

Tests that have to be automated by their very nature
Tests that are inherently easier to execute using a computer than a human
Tests that need to be run many times
Tasks (not tests) that can make manual (or automated) testing faster and more effective

Automated Execution of Functional Tests

Automated functional test execution is a powerful way to get rapid feedback on the quality of the software we produce. When used correctly, it can actually prevent defects from being built into the product; when used incorrectly, it can rapidly turn into a black hole into which time and effort are sucked. When automated regression tests are run frequently, such as before every code check-in, they can prevent new defects from being inserted into the product during enhancement or maintenance activities. Ensuring the Product Development Team understands the acceptance tests ahead of time can ensure the Product Development Team builds the correct product the first time instead of as a result of test and fix cycles. For information about how this works, see the Acceptance Test Driven Development thumbnail in Volume II.
A common strategy on projects that have an extensive suite of automated tests is to run these tests first as a form of regression test as the first activity in a test cycle; this is a form of extended smoke test. This ensures that the software functions properly (to the extent of the automated test coverage) before a human tester spends any time doing manual testing.
The key to effective automated functional testing is to use an appropriate tool for each type of test—one size does not fit all. The two most common approaches to automated test preparation are test recording (see the Recorded Test thumbnail) and test scripting. Recorded tests are easy to produce, but they are often hard to maintain. Scripted tests can either be programmatic test automation, which involves technical people writing code to test the code, or keyword-driven test automation, which non-technical people can use to write tests using a much more constrained testing vocabulary. Because keyword-driven tests are typically written in the ubiquitous language defined for the product, they are also much easier to understand than most programmatic tests. Whatever approach we use, we should think beyond the initial test authoring and also consider the life cycle costs of the tests. Recorded test tools do have some valuable uses. They can be used to quickly record throwaway test suites to support the development team while they refactor testability into the product under test. They can also be used in a record and refactor style as a way of quickly building up a collection of keywords or test utility methods to be used in keyword-driven tests or programmatic tests, respectively.
Keyword-driven testing involves specifying test scripts in a non-programmatic style. The steps of the test are data interpreted by a keyword language interpreter. Another style of data-driven test automation is the reuse of a test script with multiple data sets. This is particularly effective when we can generate test data, including inputs and expected outputs, using a comparable system test oracle. Then we run the data-driven test one time for each set of inputs/outputs. Commercial recorded test tools typically provide support for this style of testing and often include minimal support for refactoring of the recorded test scripts into parameterized scripts by replacing the constant values from the recorded test with variables or placeholders to be replaced by values from the data file.

Test Automation Pyramid

The test automation pyramid is a good way to visualize the impact of different approaches to test automation. When test automation is an afterthought, the best we can usually do is to use graphical user interface (GUI)–based test automation tools to drive the product under test. This results in a distribution of tests as shown in Figure A – Inverted Test Pyramid.
Frequently, these tests are very difficult to automate and very sensitive to any changes in the application. Because they run through the GUI, they also tend to take a long time to execute. So if test automation is an afterthought, we end up with a large number of slow, fragile tests.
An important principle when automating tests is to use the simplest possible interface to access the logic we want to verify. Projects that use test-driven development techniques attack this problem at multiple levels. They do detailed unit testing of individual methods and classes. They do automated testing of larger-grained components to verify that the individual units were integrated properly. They augment this with system tests (which include use case or workflow tests) at the whole product level. At each higher level, they try to focus on testing those things that could not be tested at the lower levels. This leaves them with much fewer system tests to automate. This is illustrated in Figure 4 – Proper Test Pyramid

Figure 4 Proper Test Pyramid
One way to reduce the effort involved is to minimize the overlap between the unit tests and component tests with the system tests. A specific example of this is the use of business unit tests to test business logic without having to go through the user interface. Another technique is the use of subcutaneous (literally, “under the skin” meaning behind the GUI) workflow tests to test business workflows without being forced to access the functionality through the user interface. Both of these approaches require the product to be designed for testability.

Automated Testing of Non-functional Requirements

Many types of non-functional tests require the use of automated test tools. Many of these tools are specially crafted for the specific purpose of assessing the product with respect to a particular kind of non-functional requirement. Common examples include performance testing tools that generate load to see how the product copes with high transaction rates or usability testing tools that record test sessions and analyze data on users’ performance (e.g. successful task completion, time on task, navigation paths, error rates etc.)

Automation as Power Tools for Manual Testers

Automated tests provide a high degree of repeatability. This works very effectively as a change detector, but it probably will not find bugs that have always been there. For that we need human testers who are continually looking for ways to break the software. For human testers to be effective, they must be able to focus on the creative task of dreaming up and executing new test scenarios, not the mundane tasks of setting up test environments, comparing output files, or generating or cleansing large amounts of test data. A lot of these tasks can be made fairly painless through appropriate use of automation.
We can use automated scripts and power tools for the following:

To set up test environments
To generate test data
To compare actual output files or databases with test oracles
To generate and analyze test reports
To tear down test environments

If we need a large dataset, we can write a program to generate one with known characteristics. If we need to test how particular transactions behave when the product is stressed, we can write a program that uses up all the memory or disk-space or CPU on command. These are all examples of power tools that make human testers more effective. For specific examples see a series of Visual Studio Team Test walkthroughs [SterlingOnTestTools].

Automated Test Generation

One of the grand objectives of software testing is automated test generation. The industry is still some distance from being able to push one button and have a tool generate and run all the tests we will ever need, but there are some selected situations where automated test generation is practical. One example is combinatorial test optimization. For example, if we have a module we are testing that takes five different parameters, each of which could be any one of four values, each of the values causes the module to behave somewhat differently, but in different way. To test this effectively, we would have to test 1024 (44444) different combinations, which is not very practical. We can use a tool that analyses the five dimensions and generates a minimal set of five-value tuples that will verify each interaction of a particular pair of values at least once (see all pairs test generation and stressful input test generation in [BachOnTestTools] includes all).
Another form of test generation is Model-Based Testing (MBT). In MBT, we build a model of the salient aspects of the product-under-test and use it to derive tests. The tests can be derived manually or we can build a test generation tool that analyses the model and generates the tests need to achieve full test coverage. The tool can simply generate test scripts that will be executed by other tools or human testers, or it can execute each test as it is generated.

Readiness vs Acceptance

As described Chapter 1 - The Acceptance Process -- and Chapter 2 - Decision Making Model , the acceptance of software can be divided, at least logically, into two separate decisions. The readiness decision is made by the Product Development Team before giving the software to the Product Owner who makes the acceptance decision. A key decision is determining which tests are run as part of readiness assessment and which are run as part of acceptance testing. In most cases, the readiness assessment is much more extensive than the acceptance testing. When functional tests are automated, it is likely they run in readiness assessment, and the software is not released to acceptance testing until all the tests pass. This results in a better quality product being presented to the Product Owner for acceptance testing.

Managing Test Development & Automation

It is useful to have a common understanding of how the development of tests relates to the development of the product. In sequential processes, the development of the tests starts well after product development and doesn’t influence it. We prefer to think of test development as occurring in parallel with product development specifically so that it can influence what is built. It is also useful to think about how the tests are defined and how that relates to test automation. If automation is focused on entire test cases, the automation cannot begin until at least some of the test cases are fully defined.
An alternative is to think of test automation at the test step level. Then automation development can begin as soon as some of the steps required by test cases are defined. This also allows the test automation development to be carried out by people with different skills (test automation) than those doing the test design (testing). The automation work can be carried out in parallel with the test definition and the product development. This makes it possible to have automated tests available as soon as the product software is available rather than at some later time.
The test steps or actions should be defined using the Ubiquitous Language adopted by everyone in the project. The test scripts defined using these terms can be executed by human testers or if the ROI is sufficient, automated. If all the steps needed by a test script have automated implementations, the entire test script can be executed automatically. If only some of the steps are automated, a human tester may still be needed but the time required to execute the test may be reduced significantly.

Who Will Accept the Product?

Ultimately, the acceptance decision belongs to the Product Owner. In some cases, the Product Owner may not be a single person. In these cases, we may have a Product Owner Committee that makes the acceptance decision using some sort of democratic or consensus-based process. Or, we may have a set of acceptance decision makers that each can veto acceptance (see Chapter 1 - The Acceptance Process in Part I) In other cases, the Product Owner may be unavailable. In these cases, we may need a Product Owner proxy to act as the "goal donor" who both provides requirements and makes the acceptance decision. The proxy may be either a delegate selected by the Product Owner, a mediator between a group of customers, or a surrogate who acts on behalf of a large group of anonymous customers. The latter role is often referred to as the product manager. For more information about this process, see the Customer Proxy Selection thumbnail in Volume II.
A related question is who will do the acceptance testing? And by extension, who will do the readiness assessment. This very much depends on the business model and the capabilities and skill set of the parties involved. The sidebar "Decision-Making Model Stereotypes" enumerates several common scenarios. When either the Product Development Team or the Product Owner feels they need assistance conducting the readiness assessment or acceptance testing, they may resort to a Test Outsourcing model. The third-party test lab would do the assessment, but the readiness decision and acceptance decision still belong to the Product Development Team and Product Owner respectively.

When Will We Do the Testing?

The test plan needs to address when the testing will be done. Some testing activities will be done by the Product Development Team as part of readiness assessment, while others are the responsibility of the Product Owner who will be deciding whether to accept the product. The test plan needs to include this in further detail to the point where we have an understanding of how much time we need for readiness assessment and acceptance testing and what we will use that time for. There are two main strategies for when testing is done on a project. The sequential approach to product development lifecycle places most testing activities into a separate test phase after all development is completed. The agile approach is to test each increment of functionality as it is completed.

Test Phase

One way to plan testing is to define a testing phase of the project after all new functionality development is complete. This is illustrated in Figure 5 Testing Phase in a Sequential Product Development Lifecycle.

Figure 5 Testing Phase in a Sequential Product Development Lifecycle
This testing phase typically consists of several time-boxed test cycles, each of which contains both readiness assessment and acceptance testing activities as illustrated in Figure 6.

Figure 6 Multiple Test Cycles in Accept Phase of Project
Each test cycle would be focused on one release candidate (RC) version of the software. Within the test cycle the Product Development Team makes a readiness decision before involving the Product Owner Team in the acceptance testing. In the early test cycles, this readiness decision is made by answering the question, "Is it good enough to bother having the Product Owner Team test it?" It is valuable to get Product Owner feedback on the software even if we know there are some defects. The test cycles each result in concerns that need to be investigated. Any concerns that need software changes are then addressed by the Product Development Team, and a new release candidate is built. This sets the stage for the next test cycle. We repeat this process until the release candidate is accepted by the acceptance decision maker(s).
Within each test cycle, it is likely we will have a predefined set of testing activities; these may be laid out as a Pert chart or a Gantt chart to reflect timing and interdependencies. We may decide to reuse the same plan for each of the test cycles, or we could define a unique plan for each cycle. In practice, with good automated regression testing in place, we should find fewer and fewer defects each test cycle. Because of this, a risk-based approach to planning the subsequent cycles could result in shorter cycle times and faster time to market. We may also deliberately choose to defer some kinds of testing activities to later test cycles or to do them in earlier test cycles.
The test resources on a sequential project are typically involved mostly at the back end of the project. Even when they are involved in fleshing out the requirements at the beginning of the project, their involvement while the software is being built tends to be minimal. The staffing profile of a hypothetical 10 week project is illustrated in Figure 7 – Sequential Test Specialist Staffing Profile.

Figure 7 Sequential Test Specialist Staffing Profile
This project involves ten features and each feature requires one person week of requirements, one person week of test design and one person week of test execution followed by a person week of bug fixing by development and one person week of regression testing. Note how the bulk of the activity occurs at the back end of the project and requires ten test specialists for weeks 8 and 10 (week 9 involves primarily development resources.) This bursty nature of testing is usually accommodated by testers splitting their time across several projects. This makes it hard for them to keep up to date on what is being built and how the requirements are evolving because multiplexing (task-switching) is a form of waste. (See the sidebar on Multiplexing vs. Undivided Attention). Also, this requires development to be finished in week 7 to leave time for two 1-week test cycles with a week of bug-fixing in between.
Note:
Sidebar: Mutliplexing vs. Undivided Attention
Having the bulk of testing activity done at the end of the product development life cycle gives the managers the illusion that testers experience slack in their workloads at different times and therefore can be assigned to multiple projects that would enable them to use the time more efficiently and effectively. Not only does this approach place constraints on when developers receive feedback on the product, but also on how much time they have to act on that feedback.
Multiplexing requires context switiching. The cost of context switching includes the effort for understanding each project’s status and re-immersing onself in the other project context as well as finding proper test environments, resetting them, relearning certain requirements of the product, re-establishing contact with the right people, and often regaining focus and rethinking issues that have already been settled. All of these are referred to by Tom DeMarco as immersion [DeMarcoOnSlack]. Multiplexing may also require catching up on the work done in one’s absence.
Gerald Weinberg estimates that the context-switching cost of assigning an additional project to one person is a massive 20% [WeinbergOnContextSwitching]. For three projects, it’s a 40% drop in productivity!
Joel Spolsky’s testimony indicates that with software engineers the cost of context switching even greater – 75% [SpolskyOnContextSwitching]. Spolsky argues that software development is “the kind of intellectual activity where we have to keep a lot of things in our head at once. The more [information] you [can] keep in mind, the more productive you are”. Such information includes variable names, APIs, data structures, requirements, helper functions, test cases, details of stack traces, debugging information, the repository structure, configuration parameters and where they are set.
Researchers in cognitive psychology and organizational behaviour who specifically studied software engineers also came to the same conclusion: multiplexing is harmful. It’s one of the contributing factors to “time famine”, which is destructive to individuals’ lives and not in the best interest of their employers either [PerlowOnTimeFamine] and [OcasioOnWorkFragmentation].

Ooops… the piece of code above clearly doesn’t belong here. It crawled in from a different project. When noticed it, I realized what a travesty this was!! I was working on a demo of the fluent configuration interface for for the Enterprise Library 5.0 project while also working on the Acceptance Test Engineering Guide. This was such a wonderful illustration of another point – how the bugs can easily crawl in while multiplexing and not giving the full attention to one project – that I’ve decided to leave this in here as a case in point!The bottom line is this: full-time involvement in a single project improves individual performance.

Incremental Testing

The alternative to leaving all the testing to the end of a project is to do incremental testing as software is developed. This requires that the Product Development Team organizes its work such that there is a steady stream of testable software available starting early in the project. This is illustrated in Figure 8 – Incremental Product Development Lifecycle.

Figure 8 Testing in an Incremental Product Development Lifecycle
This incremental product development lifecycle allows acceptance testing to be spread out over most of the project duration instead of being jammed into a much shorter testing phase on the critical path near the end of the project. (This is rather different than the sequential approach used in many organizations; the sidebar What it Takes to do Incremental Acceptance describes some of the challenges and solutions.) As each feature or iteration is finished, it is assessed for readiness immediately by the Product Development Team and turned over to the Product Owner for acceptance testing. Ideally, any bugs found are fixed immediately as described in the sidebar Incremental Acceptance and Bug Tracking on Agile Projects in Chapter 8 rather than being stockpiled for the end of the project.
If the same ten features described in the sequential example are built by an agile team using Incremental Acceptance Testing, the testing staff profile looks more like Figure 9 – Agile Test Specialist Staffing Profile.

Figure 9 Agile Test Specialist Staffing Profile
On this project we start out by specifying two features in the first week, design the tests for those two features in the second week as well as specifying a third feature. In the third week we test the first feature and do test design for the third feature. Note how the staffing ramps up to four person weeks of work and stays there for the rest of the duration. Towards the end as new feature testing ramps down the slack is taken up by regression testing of the early features. Because the testers are part of the development team they are able to keep abreast of how the requirements and product design are evolving. Since the same people will be designing and executing the tests the need to write detailed test scripts are reduced and more effort can be spent on exploratory testing and test automation. Since the testing is done soon after the code is developed the developers can fix the bugs immediately so there is no separate “testing and bug fix phase” at the end therefore development can continue closer to the delivery milestone.

Session-Based Test Management

An alternative to using a plan-driven approach within the test cycle is to use a more iterative style known as Session-Based Test Management. We create a prioritized backlog of testing activities that we address in a series of test sessions. As new concerns are identified in test sessions, we may add additional test activities to the test backlog. The key is to keep the backlog prioritized by the value of the testing. This value is typically based on the expected degree of risk reduction. The depth of the backlog gives us an idea of how much testing work we have left and if we are making headway by addressing concerns or if are losing ground ( the backlog increasing in depth). Session-Based Test Management is commonly used exploratory testing.

Where Will We Do the Testing?

The test plan needs to identify where the testing will be performed. When all the testing will be performed in-house, the primary consideration is which physical (or virtual) environments will be used. This is particularly important when new environments need to be created or shared environments need to be booked. If we lack physical resources or the skills to do the testing, we may choose to do test outsourcing to a third-party test lab.
We also need to define the criteria for moving the software between the environments. The transition from the readiness assessment environment to the acceptance environment is governed by the readiness assessment criteria. When developers have their own individual development environments, we also need criteria for when software can be submitted into the team's integration environment where readiness assessment will occur. These criteria are often referred to as the Done-Done Checklist because the definition of "done" is more stringent than what a developer typically refers to as done.

How Long Will the Testing Take?

Because testing is usually on the critical path to delivery of software-intensive-products, project sponsors and project managers usually want to know how long testing will take. The most common answer is "How much time do we have?" Frequently, the time available is not long enough to gather enough data to make a high confidence readiness or acceptance decision. This answer is not quite as flippant as it sounds because of the nature of testing. We cannot prove software works correctly—we can only disprove it by finding bugs. Even if we had an infinite amount of time to test, after a point, diminishing returns will kick in and we will not find many more defects. So, we have to determine what is barely sufficient to get enough confidence about the quality level. This requires at least a minimal level of test estimation to establish the lower bound of the time and effort we need to expend.
Some tests may have dependencies the restrict when they can be run. These could be dependencies on other test environments such as when doing integration testing with other systems or they may be dependencies on date.
We also need to know how long we’ll need to wait for any defects we need addressed to be fixed by the Product Development Team. Will there be “dead” time between test cycles while we wait for fixed software as illustrated in Figure Y – Alternating Test & Fix?

Figure Y Alternating Test & Fix
The dead periods occur if the acceptance decision is the first point at which the Product Development Team is provided the list of “must fix” bugs after which the Product Development Team starts the process of characterizing and fixing the bugs. This may take days or weeks to develop a new release candidate which must then undergo readiness assessment before being presented to the Product Owner for the next round of acceptance testing.
The duration of the acceptance phase (consisting of several test cycles) can be reduced significantly if the Product Development Team is notified of bugs as soon as they are found. This allows them to be fixing defects continuously and delivering a new release candidate on a regular schedule as shown in Figure X – Continuous Test & Fix.

Figure X Continuous Test & Fix
This depends on the Product Owner (or proxy) doing bug triage on all newly found concerns so that the Product Development Team is made aware of all gating bugs (bugs that must be fixed before accepting the software) as soon as practicable.
If we are planning to automate tests, we should have an effort estimate for the automation. In most cases, we want separate estimates for the construction of the automation infrastructure and the preparation of the tests because of the different skills and knowledge needed to do the two jobs. For more information, see the Test Automation Planning thumbnail.

How Will We Determine Our Test Effectiveness?

A learning organization is one that is constantly striving to improve how it works. This involves understanding how well we are doing today and trying new approaches to see whether they make us more effective. Measuring effectiveness requires Test Metrics. These metrics measure two key areas of performance:

They measure how far along are we in executing our test plan. That is, how much work is left before we know enough to make the readiness or acceptance decision? For more information, see the Test Status Reporting thumbnail.
They measure the effectiveness of our testing. For more information, see the Assessing Test Effectiveness thumbnail.

How Will We Manage Concerns?

The purpose of testing is to identify any concerns with the software of those who matter. Many of these concerns will require changes to the software either because something was incorrectly implemented (a bug) or because the Product Owner realized that what they had requested will not satisfy the business need (an enhancement or change request). The test plan needs to lay out how these concerns will be managed and tracked; it also needs to include the process for deciding what needs to be changed and what is acceptable as is.
As we conduct the various readiness assessment and acceptance activities, we note any concerns that come up. Investigating these concerns more closely reveals that, each concern falls into one of the following categories:

Bug or defect
Requirements change
Project issue
Non-concern

Figure 3 illustrates the lifecycle for each of the major types of concerns.

Figure 3 Concern Resolution Model
Part of the advantage of categorizing these concerns is to help identify how they will be addressed. Each category of concerns has its own resolution process and each must be addressed differently.

Bugs or Defects

Bugs are defects found in the software that require a software change. The bugs need to be understood well enough to make decisions about what to do about them. A common process for doing this is known as Bug Triage, which divides the bugs into three categories with respect to the next milestone or release: Must Fix, Would Like to Fix (if we have time), and Will Not Fix. Of course, the software must be retested after the fixes are made, which is why there are typically multiple test cycles. Fixes may also need to be propagated into other branches of the software. If the bug is found to exist in previous versions that are currently in use, a patch may need to be prepared if the bug is serious enough. Security-related bugs are an example of bugs that are typically patched back into all previous versions still in use. Likewise, a bug fixed in the acceptance test branch may need to be propagated into the current development branch if new development has already started in a separate development branch.

Requirements Changes

The Product Owner may have realized that even though the Product Development Team delivered what the Product Owner asked for, it will not provide the expected value. The Product Owner should have the right and responsibility for making the business decision about whether to delay the release to make the change or continue with the less useful functionality. Once we’ve decided to include a change in this release we can treat it more or less the same as a bug from a tracking and retest perspective.

Other Issues

Some concerns that are exposed do not require changes to the software. They may be project issues that need to be tracked to resolution, additional things that should be tested, and so on. These typically do not get tracked in the bug management system because most projects have other means for tracking them. Other concerns may be noted but deemed to not be concerns at all.

Summary

This chapter introduced the activities and practices involved in planning a testing effort. A key activity is the definition of a test strategy because this is what guides us as we strive to maximize the ROI of our efforts. There is a place for both automated testing and manual testing on most projects because the two approaches are complementary. Automated functional tests are highly effective change detectors that go a long way toward preventing new bugs from being introduced during software maintenance activities. They are also repeatable. Care has to be taken to use the appropriate functional test automation tools to avoid the slow, fragile tests quagmire. Exploratory testing manual testing can be effective and efficient ways of finding bugs, especially unusual ones, those that are hard to automate, and those that stem from incomplete, misunderstood, or vague requirements. The use of power tools by human testers can significantly increase the effectiveness of manual testing.

What’s Next?

In this chapter we have described the practices involved in planning for the acceptance of the product. In the next chapter we will examine the practices we use while executing the acceptance process.

References

[MarickOnTestingDimensions] Marick, Brian http://www.exampler.com/old-blog/2003/08/21/
[CrispinGregoryOnAgileTesting] Crispin, Lisa and Janet Gregory. Agile Testing, Addison Wesley, 2008.
[MeszarosOnUnitTestPatterns] Meszaros, Gerard. xUnit Test Patterns: Refactoring Test Code. Addison-Wesley Professional. 2007.
[KanerOnExploratoryTesting] Kaner, Cem, Jack Falk, and Hung Q. Nguyen. Testing Computer Software, 2nd Edition. Wiley. 1999.
[BachOnExploratoryTesting] Bach, James. What is Exploratory Testing? http://www.satisfice.com/articles/et-article.pdf
[FewsterGrahamOnTestAutomation] Fewster, M., Graham, D. Software Test Automation. Addison-Wesley, 1999.
[ItkonenRautiainenOnExploratoryTesting] Juha Itkonen and Kristian Rautiainen, “Exploratory Testing: A Multiple Case Study”, International Symposium on Empirical Software Engineering,” ISESE 2005, 17-18 Nov. 2005, pp. 84-94.
[ItkonenEtAlOnDefectDetection] Itkonen, J.; Mantyla, M.V.; Lassenius, C. “Defect Detection Efficiency: Test Case Based vs. Exploratory Testing”, in Proc. of First International Symposium on the Empirical Software Engineering and Measurement, ESEM 2007. 20-21 Sept. 2007, pp 61 – 70.
[JeffriesMelnikOnTDDEffectiveness] Jeffries, R. and Melnik, G., “TDD: The Art of Fearless Programming”, Guest Editors’ Introduction, IEEE Software Special Issue on Test-Driven Development, May-June 2007, pp.24-30.
[SterlingOnTestTools] Charles Sterling, “Visual Studio Team System 2010 Test Features walk through with screen shots.” http://blogs.msdn.com/charles_sterling/archive/2008/11/05/visual-studio-team-system-2010-test-features-walk-through-with-screen-shots.aspx, October 20, 2009
[BachOnTestTools] James Bach. Repository of Test Tools. http://www.satisfice.com/tools.shtml, October 20, 2009
[WeinbergOnContextSwitching] Gerald Weinberg, Quality Sofwtare Management, Vol.1: Systems Thinking, New York, NY: Dorset House Publishing, 1992.
[SpolskyOnContextSwitching] Joel Spolsky. “Human Task Switches Considered Harmful” http://www.joelonsoftware.com/articles/fog0000000022.html, 2001, visited October 20, 2009
[DeMarcoOnSlack] Tom DeMarco, Slack, Broadway, 2001.
[OcasioOnWorkFragmentation]. William Ocasio. “Towards an attention-based view of the firm.” Strategic Management Journal, 18:187-206, 1997.
[PerlowOnTimeFamine] Leslie Perlow. “The time famine: Toward a sociology of work time”, Administrative Science Quaterly, 44(1):57-81, 1999.

Updated Wiki: Vol1-3-Chapter-17

rburte — Sat, 07 Nov 2009 01:25:11 GMT

Chapter 17 - Planning for Acceptance

The scope of the acceptance process and the breakdown into readiness assessment and acceptance testing phases
The overall test strategy including both manual and automated testing
The activities, testing and otherwise, that will be performed in each phase (the readiness phase and the acceptance phase)
The skills that are required to perform the activities
The other resources that will be utilized to carry out the activities (such as facilities and equipment)
The timeframe and sequence, if relevant, in which each activity will be performed

Defining Test Objectives

The Product Development Team misinterpreted the Product Owners description resulting in a product that behaves differently from expected.
The Product Owners description of the product left out important behaviors resulting in missing functionality.
The Product Development Team implemented the behaviors poorly resulting in frequent crashes of the product.
The product exhibits the right behavior when used by small numbers of users but behaves erratically or responds slowly when the number of users or transactions increases to expected levels.

Traditional Approaches to Testing

Flipping the Pyramid Right Side Up

Whether the tests are business-facing or technology-facing
Whether the tests are intended to support development (by helping them get it right) or to critique the product after it is built

Tests That Support Development

Business-Facing Tests

Technology-Facing Tests

Tests That Critique the Product

Technology-Facing Tests

Business-Facing Tests

What Testing Will We Do? And Why?

Test Strategy

Manual Testing Freedom

Automated Testing

Maximizing Automation ROI

To ensure that we get the best possible ROI for our test automation investment, we need to focus our energies on the following:

Tests that have to be automated by their very nature
Tests that are inherently easier to execute using a computer than a human
Tests that need to be run many times
Tasks (not tests) that can make manual (or automated) testing faster and more effective

Automated Execution of Functional Tests

Test Automation Pyramid

Automated Testing of Non-functional Requirements

Automation as Power Tools for Manual Testers

To set up test environments
To generate test data
To compare actual output files or databases with test oracles
To generate and analyze test reports
To tear down test environments

Automated Test Generation

Readiness vs Acceptance

Managing Test Development & Automation

Who Will Accept the Product?

When Will We Do the Testing?

Test Phase

Incremental Testing

Session-Based Test Management

Where Will We Do the Testing?

How Long Will the Testing Take?

How Will We Determine Our Test Effectiveness?

They measure how far along are we in executing our test plan. That is, how much work is left before we know enough to make the readiness or acceptance decision? For more information, see the Test Status Reporting thumbnail.
They measure the effectiveness of our testing. For more information, see the Assessing Test Effectiveness thumbnail.