Appendix B – Key Points

The chapters in Part II described the process leading up to the acceptance decision from the perspectives of the Business Lead, Product Manager, Test Manager, Development Manager, User Experience Specialist, Solutions Architect, Enterprise Architect, Operations Manager, and Legal. For convenience, we summarized the key points here. There are Key Points common to all roles and others specific to each individual role.
The following table lists Key Points that are role specific and what roles they are specific to.
<The table will be updated and available on testingguidance.codeplex.com>
vol1-App-figure-3.png
Figure 1 Key Points specific to individual roles

Key Points Applicable to All Roles

  • Treat acceptance as a process not an event: Appreciate the fact that acceptance testing will not be a single discrete event, but a process with a distinct start and end.
  • Define the scope of testing: Have a clear understanding of what will be tested by the supplier and what you are expected to test. Typically, supplier is expected to do thorough (e.g.) testing of all capabilities and quality characteristics (such as performance, reliability, usability etc.) before handing off system for acceptance testing. Does the supplier have the environments for integration testing without in-house systems?
  • Delegate decisions: If you don’t have enough time or understanding to take part in the project, assign a proxy to represent you and have a clear understanding of the kinds of decisions they can make with/without consulting with you.
  • Delegate work/testing: Have a clear understanding of your (organization’s) abilities w.r.t. testing skills. Contract other parties to help you test if necessary. But don’t underestimate the effort involved in managing the testing outsourcer as they will need to learn your business domain and requirements before they can be effective.
  • Understand the test environments: Understand where you will test. Be clear on how you’ll get the new release candidates delivered
  • View tests as requirements: Acceptance criteria/tests should be articulated and prepared before the software is built and shared with vendor as a way to flesh out the details of the requirements.
  • Estimate acceptance effort & duration while realizing it is difficult to estimate: Estimating the effort & duration of acceptance testing is hard; time-boxing the final acceptance testing cycle is common but requires past experience to come up with reasonable duration/effort. If in doubt, guess high and allow for several cycles of acceptance testing to give the vendor time to fix bugs.
  • Use incremental acceptance testing to reduce uncertainty of estimating: Can avoid some of the uncertainty and most of the risk by doing incremental acceptance testing. Bugs found earlier identify ambiguity in the spec while there is still time to address it and can be fixed long before the final acceptance test phase
  • Understand the decision-making process: Make sure you have a clear understanding of the decision making process. Other potential decision makers include Test Managers, Security Manager, User Experience/Usability Manager, Architect, Legal Department, etc. Ensure you have agreement on whether they make a decision or provide data or a recommendation to you.
  • Communicate information flow clearly: Ensure anyone who you expect to provide you with data on which to base your decision understands what data they are providing you and when. E.g. The Test Manager provides test results to you and you make the acceptance decision.
  • Understand the operational requirements: Make sure you understand the operational requirements for your system. Whether you are selling a software package to customers or selling Software-as-a-Service (SaaS), be aware that whoever operates the software will have operational requirements over and above the functional requirements of the end users.
  • Clarify decision making vs decision supporting: Make sure you have a clear understanding of whether you are the gate-keeper (acceptance decision maker) or a supplier of information to the Product Manager for her to make the decision (preferred)
  • Maximize learning: Work with the product owner to ensure that everyone on the team has access to the definition of success in the form of acceptance tests before the software is built

Key Points for Business lead

  1. Treat acceptance as a process not an event: Appreciate the fact that acceptance testing will not be a single discrete event, but a process with a distinct start and end.
  2. Recognize there may be many decision makers: The process may require decisions by many parties each supported by data collection activities. Each party likely has very different requirements and interests; you need to be aware of them and communicate them to the supplier(s) to ensure they address them.
  3. Define the scope of testing: Have a clear understanding of what will be tested by the supplier and what you are expected to test. Typically, supplier is expected to do thorough (e.g.) testing of all capabilities and quality characteristics (such as performance, reliability, usability etc.) before handing off system for AT. Does the supplier have the environments for integration testing without in-house systems?
  4. Delegate decisions: If you don’t have enough time or understanding to take part in the project, assign a proxy to represent you and have a clear understanding of the kinds of decisions they can make with/without consulting with you.
  5. Delegate work/testing: Have a clear understanding of your (organization’s) abilities w.r.t. testing skills. Contract other parties to help you test if necessary. But don’t underestimate the effort involved in managing the testing outsourcer as they will need to learn your business domain and requirements before they can be effective.
  6. Understand the test environments: Understand where you will test. Be clear on how you’ll get the new release candidates delivered
  7. View tests as requirements: Acceptance criteria/tests should be articulated/prepared before the software is built and shared with vendor as a way to flesh out the details of the requirements.
  8. Recognize the choice of process impacts everything: Two key styles of AT: Final Test Phase and Incremental AT. The former concentrates most acceptance activities at the end of the project. Incremental acceptance testing spreads out the work and reduces risk but requires all parties to work in different ways.
  9. Estimate acceptance effort & duration while realizing it is difficult to estimate: Estimating the effort & duration of acceptance testing is hard; time-boxing the final acceptance testing cycle is common but requires past experience to come up with reasonable duration/effort. If in doubt, guess high and allow for several cycles of acceptance testing to give the vendor time to fix bugs.
  10. Use incremental acceptance testing to reduce uncertainty of estimating: Can avoid some of the uncertainty and most of the risk by doing incremental acceptance testing. Bugs found earlier identify ambiguity in the spec while there is still time to address it and can be fixed long before the final acceptance test phase.

Key Points for Product Manager

  • All Business Lead’s Key Points plus:
  • Understand the decision-making process: Make sure you have a clear understanding of the decision making process. Other potential decision makers include Test Managers, Security Manager, User Experience/Usability Manager, Architect, Legal Department, etc. Ensure you have agreement on whether they make a decision or provide data or a recommendation to you.
  • Recognize that complex products and organizations have complex decision making processes: There are several different ways to organize the teams to deal with very large products or products that are the amalgamation of several products/product lines (e.g SharePoint=ServerOfficeIE). The choice should consider the resulting decision-making model (e.g. component teams, feature teams.)
  • Communicate information flow clearly: Ensure anyone who you expect to provide you with data on which to base your decision understands what data they are providing you and when. E.g. The Test Manager provides test results to you and you make the acceptance decision.
  • Ensure all suppliers understand users and usage model: Make sure you understand the users and communicate this to both supplier and test organizations.
  • Understand the operational requirements: Make sure you understand the operational requirements for your system. Whether you are selling a software package to customers or selling Software-as-a-Service (SaaS), be aware that whoever operates the software will have operational requirements over and above the functional requirements of the end users.

Key Points for Test Manager

  • Determine your test mandate: Make sure you understand whether you are doing readiness assessment or acceptance testing or both. For acceptance testing be sure the testers have realistic tests for the users they proxy. Consider using personas. For readiness assessment it is important to have a good relationship with the development team to help them understand the quality of the product and what will improve it.
  • Clarify decision making vs decision supporting: Make sure you have a clear understanding of whether you are the gate-keeper (acceptance decision maker) or a supplier of information to the Product Manager for her to make the decision (preferred)
  • Get agreement from the product owner that the test cases reflect the requirements: Aim to write the acceptance test cases first and provide them to the development teams before the software is built to be used as aides to assist in understanding how the system should behave. This is known as Acceptance Test Driven Development.
  • Plan a multi-pronged test strategy: Plan for a multi-pronged test strategy that includes incremental testing and both scripted and exploratory tests.
  • Collaborate with the development team: Aim for an incremental delivery of functionality to support incremental acceptance testing. Work with the development team to design for testability to facilitate automated testing and enabling them to be able to run tests on demand.
  • Do test automation when feasible and in the way it supports (not dictates!) meeting your test objectives. Consider having a dedicated test automation engineer.
  • Agree on a definition of what done is: Get the various parties involved to agree on a definition of what done is for software before it goes through each of the quality gates of the gating model and include a checklist of requirements.

Key Points for Development Manager

  • Perform readiness assessments: All software should be adequately tested by the development team before presenting it to the customer(s).
  • Agree on the minimum credible requirement (MCR) ahead of time: The product owner, test organization and the supplier should all agree on the MCR ahead of time, that is, before the software is built.
  • Use acceptance tests to define success: Use acceptance tests based on the MQR to define success before the software is built and ensure that everyone on the development team has access to them and runs the tests as they build the software.
  • Run the tests as part of readiness acceptance: Run all the tests that the product owner will run and some you’ve defined.
  • Minimize untested code: Test as you go, ideally incremental testing.
  • Use testing to prevent bugs: Tests can help prevent bugs being introduced during the development cycle. Include functional testing, component/unit tests, and para-function testing as early as possible.

Key Points for User Experience Specialist

  • Understand your role: Have a clear understanding with the product owner, product manager or business lead what your role shall be in making the acceptance decision.
  • Maximize learning: Work with the product owner to ensure that everyone on the team has access to the definition of success in the form of acceptance tests before the software is built.
  • Define the usability component of the acceptance criteria as early as possible: Work with the business lead or product manager to define the usability component of the acceptance criteria as early as possible.
  • Address the usability risks as early as possible: The areas with highest usability impact should be built and tested first so that any usability-related issues can be addressed as early as possible in the development cycle.

Key Points for Operations Manager

  • Understand role in acceptance decision: You and everyone concerned should be clear on your role in the making of the acceptance decision. Are you a data provider to the acceptance decision maker or an acceptance decision maker?
  • Identify ways to reduce the number of test&fix cycles: Work with the development manager to identify ways to reduce the number of test&fix cycles through early delivery of testable software and incremental acceptance testing.
  • Define acceptance from the operational perspective as early as possible: Ideally you have clearly defined acceptance criteria before the software is built.

Key Points for Solution Architect

  • Describe the whole solution to everyone: Communicate the solution in its entirety to the whole supplier team to avoid the “My part works just fine (even if the solution doesn’t!)” mentality.
  • Ensure availability of acceptance tests: Work with the requirements analysts and testers to ensure that the acceptance tests are available to the supplier team before the software is built.
  • Educate about tradeoffs: Work with the product owner to help them understand what kinds of functionality is easy to build and what is hard. Help them get the most bang for their money.
  • Unbundle functionality: Work with the product owner to help them identify the lower-value functionality that could potentially be deferred in case there is a need to trade off functionality for quality or delivery date.
  • Define SLA metrics
  • Engage enterprise architects: Engage the enterprise-wide architects (responsible for infrastructure, standards, technology decisions, long-term vision, security, data architecture, etc.) early so they can help you understand what requires their approval before it becomes part of the critical path.

Key Points for Enterprise Architect

  • Publish architectural acceptance criteria: Publish general acceptance criteria so that all supplier teams understand the rules of the game and what kinds of approvals they need to get from you.
  • Articulate cross system integration requirements and SLAs: Identify applicable integration requirements and service level agreements. Make sure the supplier team is aware of them and that they are acceptance criteria.
  • Ensure standards compliance: Verify that products comply with applicable standards.
  • Encourage reuse: Verify that product teams are aware of and capitalize on opportunities to reuse components and data.
  • Engage supplier teams early: Engage supplier teams early to ensure that inspections prevent defects rather than find them.
  • Delegate non-critical decisions: If it is not important to be involved in a decision then bow out and let the project team make the decision.
  • Keep approval process lightweight: Make the process for seeking guidance and approval as lightweight as possible. Emphasize effectiveness of communication over comprehensive documentation. Avoid offloading work (e.g. more documentation) to the project teams to save you effort if that results in increasing the total work.
  • Be pragmatic: Be pragmatic as to what progress individual project or product teams are expected to make towards long-term architectural objectives. They are responsible to the product owner, not to you.

Key Points for Legal

  • Include clear stipulations of any contractual implications: Acceptance typically is a condition for rendering a payment and affects the application of any warranty provisions and potentially any remedies available to the customer.
  • Discourage one-sided contracts: The contract should encourage both parties to get to a mutually agreeable acceptance as quickly as possible. Contracts that penalize one party often lead to dysfunctional behavior that results in lose-lose outcomes.
  • Clearly define how much time is allowed for acceptance tests: Acceptance testing is always time-boxed. There is a limited amount of time for acceptance and reporting deficiencies.
  • Include a clear definition of acceptance criteria: Besides testing performed at the location of supplier, and functional and operational tests performed when the software is deployed to the customer, software system often are required to successfully run with current data in a live operating environment for a period of time, with minimum bugs, before the system is formally accepted.
  • Know the explicit and implicit contractual conditions: Acceptance can be viewed in terms of explicit provisions specified in the software development contract and the terms implied into the contract by law (which also depends on the geographic jurisdiction).

Last edited Nov 7, 2009 at 1:14 AM by rburte, version 2

Comments

No comments yet.