Preface: Acceptance Test Engineering Guide
Why We Wrote This Guide
Accepting software is one of the most important decisions for software to take flight and to start delivering value. Yet, there exists little guidance about what this decision should be based on and how to make it effectively.
Over many years in industry and applied research, it was our observation that people often misunderstood the challenges of accepting software, or worse, did not even have a clear notion of who is in charge of acceptance. Many appear not to be aware of the arsenal
of practices and tools available to them to address those challenges. This led to disappointments, failed opportunities, broken business relationships, and litigations. There exists a rich body of knowledge on software testing (with notable works by Boris
Bezier, Bertrand Meyer, Gerald Weinberg, Cem Kaner, James Bach, Brett Pettichord, Lee Copeland, Mark Fewster, Dorothy Graham, Rex Black, Brian Marick, James Whittaker to name a few), but software acceptance in general, and acceptance testing, in particular,
as a process, received no serious, well-reasoned or comprehensive treatment. We set out to fill this gap with two key objectives: devise mental models for thinking about software acceptance and offer actionable guidance on achieving software acceptance.
Who Should Read This Guide
This guide is intended for anyone who is involved in the process of deciding the degree to which a software-intensive product meets the acceptance criteria of those who commissioned its construction. Specifically, this guide will help you if any of the following
apply to you:
- You are involved in deciding whether to accept the software as it has been built. This is the
- You are involved in deciding on what acceptance criteria shall be used to make the acceptance decision.
- You are involved in collecting data that the person making the acceptance decision requires to make that decision. This is
- You are involved in deciding whether the product is ready to be seen by the people involved in the acceptance decision or acceptance testing. This is the
- You are involved in collecting data that the person making the readiness decision requires to make that decision. This is
- You are involved in defining the expectations against which the readiness assessment or acceptance testing activities will be conducted. This is a combination of requirements gathering, test planning
and test design.
- You are involved in managing any of the preceding activities.
This guide describes the practices used by people in the preceding roles, regardless what your job title is. If any of them describes your role, you should find something of interest in this guide. Appendix – Reader Personas describes our target audience in
more detail. These personas are used to:
- remind the authors and reviewers of our primary target audience.
Figure 1. Reader goals and paths through the guide.
- give you, the reader a figure to empathize or associate yourself with. This should help you pick the persona that most closely resembles your own job responsibilities.
This guide is structured as a four volume series (Figure 2):
- Volume I provides an overview of the acceptance process and how acceptance testing and other key practices fit into the process. This volume is intended to be read from beginning to end. It is subdivided into three main parts:
- Volume II covers Acceptance Planning and Management practices and patterns.
- Volume III covers Functional Acceptance Test practices and patterns.
Figure 2. Guide Structure.
- Volume IV covers Operational Acceptance Test practices and patterns. Operational acceptance includes not only performance, recoverability, availability, security but also testing of other non-functional requirements and qualities, such as learnability,
usability, supportability, upgradeability etc.
- Volume I is intended to be read as an introduction to the concepts, terms and practices described in Volumes II-IV. Volumes II-IV are intended to be used as a reference; most readers will not read them from beginning to end.
- Volume I – Thinking About Acceptance consists of 3 parts:
- Part I – Thinking about Acceptance explains six mental models that are useful when thinking about the acceptance process.
- Part II – Perspectives on Acceptance describes the acceptance process from the perspectives of key stakeholders in two different kinds of organizations: the Information Technology Department in a business and the Product Development Company. Most
readers involved in the acceptance process should find some commonality with at least one of the roles describes.
- Part III – Accepting Software introduces the practices that are necessary for planning the acceptance process, for performing acceptance testing and for improving the acceptance process.
- Volumes II through IV contain a collection of thumbnails and samples:
- A Thumbnail is a short overview of a practice that explains what it is, when you may want to use it, the risks that it mitigates, and an overview of how to perform the practice. Thumbnails also include a list of references to papers, books, and other
resources that provide more complete descriptions of the practices in question. The main purpose of a thumbnail is to describe a topic well enough to provide an overview, serve as a mental reminder for someone who has used the practice on how to do it, and
give someone unfamiliar with the practice enough information about the practice and its applicability to determine if they want to learn more about it. Some of these topics and practices have entire books written about them that describe the concepts in greater
detail and depth than this guide could possibly do.
- A Sample is a an artifact generated by applying different practices in a fictional but realistic situation for the fictional company Global Bank. These artifacts are embedded in a series of case studies of what the Global Bank team may have produced
while building the application. The case studies provide some context to the individual artifacts. They also provide cross-references to the thumbnails. The artifacts are intended to be used as way to learn more about how to perform a practice; they can also
be used as templates for your own artifacts.
How to Read This Guide
The way you approach this guide will depend on what role you have and what you want to learn about the process of accepting software. Depending on what you want to do, you will want to apply different strategies.
Get an Overview of Acceptance Practices and Processes
Start by reading Volume I if you want to do any or all of the following:
- Learn general information about acceptance testing.
- Find acceptance testing practices.
- Create a project plan that incorporates software acceptance.
- Justify a project plan that incorporates software acceptance.
- Justify an approach used for acceptance testing.
- Validate that you are on track with your acceptance testing strategy or approach.
- Get your project unstuck when software is unacceptable.
- Determine where there may be gaps in your acceptance testing approach or strategy.
After reading Volume I, you may want to skim particular practices of interest and the corresponding samples in Volumes II through IV.
Decide Which Acceptance Practices to Use on Your Project
Start by reading Volume I to get an overview of possible practices, and then refer to the thumbnails in Volumes II-IV for specific practices you are considering. Each thumbnail includes a section titled "When to Use It," which includes advice about
when the practice should be used, and a section titled "Limitations," which provides guidelines about when the practice should not be applied.
Learn How to Perform a Specific Acceptance Practice
Start by finding a thumbnail in Volumes II-IV if you want to do any of the following:
- Learn a specific acceptance testing practice or strategy.
- Teach a specific acceptance testing practice or strategy to someone else.
- Review a specific acceptance testing practice.
- Find more information and related resources to consult about a particular practice.
- After you locate the thumbnail for the specific practice you want to learn about, read it and any related samples. If you need more detailed information about the practice, see the "References" section in the thumbnail.
Get a Template for a Specific Artifact
Start by finding an sample in Volumes II-IV if you want to do any of the following:
- Find a template for a specific artifact.
- Learn how to fill in a specific artifact.
Find the example you want in Volumes II-IV, remove the sample information, and populate it appropriately. If you need to review the practice that generated the example, the example lists all the appropriate thumbnails in Volumes II-IV.
Plan the Execution of the Practices on Your Project
Start by reading Volume I to get an overview of how the practices fit together and support each other. In particular, the sections on the Acceptance Process, the Decision-Making Model, and the Doneness Model may be of particular interest. After that, review
the specific thumbnails in Volumes II-IV, paying particular attention to the subsection, "Test Life Cycle Applicability" in the section, "When to Use It." Each sample artifact is accompanied by a notation that indicates at what point in
the hypothetical project the artifact was produced. Note that some artifacts appear at several points in the project timeline because they evolve over time. If you find your acceptance process takes too long you may find the Streamlining Acceptance Process
chapter of Volume I to be helpful.
Find Tools for Acceptance Testing
If you are looking for tools to perform acceptance testing, you may use the guide to explore the available space. Although some of the case studies illustrate using specific tools, remember that the primary focus of this guide is on describing practices.
The choice of tools used while producing this guide should not be interpreted as an endorsement of any tool, nor should it be interpreted as an indication that any tool used is the best one for the job. By the time you read this guide, the tools illustrated
in the samples in the guide may have been supplanted by newer and better ones.