Chapter 8 - The Business Lead's Perspective
An organization may commission the construction or customization of software for its own use. The construction would be done by another organization which may be an internal department (such as the Information Technology department) or another company (an
outsourced development company.) In such an organization, the Business Lead is the person who brings to the project team the vision of what the software is intended to do. The Business Lead may also be known as the Project Sponsor, Business Sponsor, Business
Owner, or Business Partner, and acts in the capacity of the Product Owner.
See the Business Person in the Marketing Division persona in Appendix A – Organization Stereotypes and Reader Personas.
The decision to commission custom
as it is called in some locales) software is a business decision made based on the expected business benefits of the investment - the return on investment (ROI). The benefits may be increased income
through additional sales or services, cost reduction through fewer staff, more efficient work processes or reduced errors – essentially, these benefits are intended to contribute to the economic bottom-line of the organization. So now that you’ve made the
decision to commission construction of the software, now what? Before you can realize the benefits you will be asked to make the decision to accept the software as built or provide a list of deficiencies that must be addressed before you will accept it. Acceptance
is a significant stage of any contractual process (See Legal Perspectives on Acceptance). How you approach this acceptance decision can have a big influence on how painful or painless the process will be. It will also impact your post-acceptance activities
(including actual usage, operational support, maintenance etc.)
The wrong approach can be summarized with the famous quote “I cannot describe good art but I’ll recognize it when I see it.” Taking this approach to the acceptance decision is sure to result in late delivery, cost overruns, delivery of software that does not
bring value to your organization and a strained relationship with your Product Development Team because the they will have a poor chance of meeting your needs if they don’t know what they are. When they deliver the software you wanted, you’ll be sure to find
all sorts of issues with how it works. You’ll report these “bugs” to the Product Development Team and expect most of them to be fixed before you accept the software and put it into production. That will take additional time and resources thereby increasing
costs and deferring the benefits. In the worst case, the vendor may be unable to fix all the defects to your satisfaction and you’ll either have to accept software you are not happy with or write off the system entirely.
To quote Ludwig Wittgenstein, one of the leading philosophers of the 200th century, “there is a gulf between an order and its execution. It has to be filled by the act of understanding” [WittgensteinOnPhilosophicalInvestigations]. To be successful, do not underestimate
this “act of understanding”. Successful projects typically have a clear agreement between the Product Owner and the Product Development Team on what constitutes a successful outcome for the project. This project charter provides a high-level view of the business
lead’s expectations of the project and how each party will behave throughout the project. It also lays out any key constraints regarding schedule, cost and any other relevant factors. Don’t focus the charter (or contract, Statement of Work, etc.) on what remedies
you will have if the Product Development Team fails to deliver; by then the project has already failed!! Instead, focus on the behaviors that will ensure project success. All parties should treat the project as a miniature joint venture and strive for a win-win
outcome. If any of the parties is trying to win at the other’s expense, all parties will lose. Dysfunctional relationships lead to dysfunctional products. See the sidebar “Target cost contracting” for description of an alternative to traditional fixed price
or time & materials contracts. Refer to Section Responsbilities of the Product Owner in Chapter 1 for a summary of the responsibilities.
Sidebar: Target Cost Contracting : from adversarial contracts towards partnering contracts
Lean Software Development visionaries Tom and Mary Poppendieck advocate the “partnering” or “relational” approach to contracting, in which parties agree to work
in a collaborative way to carry out the project [PoppendiecksOnLean, Ch.9]. Target Cost Contracts are a type of incentives-contracts that can be used to support sharing of both financial risks and rewards. This helps to align the best interest of each party
with the best interests of the joint venture. The Poppendiecks encourage forging synergistic relationship across company boundaries. They envision that those who do will see at least the 25 percent to 30 percent cost reduction of the cost of doing business
predicted by Peter Drucker in [DruckerOnManagementChallenges].Target Cost Contracts are an approach to identifying and constraining uncertainty. It’s done in the following way. A target cost for the project is agreed at the beginning and both parties are proactively
working on achieving the target cost. Upon project’s completion, the actual cost is measured against the originally set target cost. If the actual cost is great than the target cost, the supplier and the customer each bear an agreed element of that cost overrun.
If the actual cost is lower than the target cost, the supplier shares the savings with the customer. The share of the cost overrun/underrun may be a constant proportion or may vary depending on the size of the delta.The analysis of Perry & Barnes reveals
a strong case for setting the supplier’s share of cost overrun or underrun at a value that is not less than 50% [PerryEtAlOnTargetCostContracts].
The key characteristics of the Target Cost Contracts are as follows:
- Target cost includes all changes
- Target is the joint responsibility of both parties
- Target cost is clearly communicated to workers
- Negotiations occur if target cost is exceeded with neither party benefiting.
- This entices workers at all level to work collaboratively, compromise, and meet the target.For additional case study of using Target-Cost Contracts in software development, see this experience report [EckfeldtEtAlOnTargetCostContracts].
Accepting software sounds like a discrete event but there is much to be done leading up to it. As a Product Owner you need to be prepared to participate in the development of your software. Anyone who has had a house built specifically for them knows that there
are a lot of decisions to be made and inspections to be done throughout the entire construction process. Software should not be any different. Expect to get as much out of your software as you are prepared to put in. The absentee Product Owner gets the results
they deserve; an unsuitable system designed for someone else.
To avoid this fate you certainly must have a plan for making the acceptance decision. But you must also plan for regular communication with the Product Development Team about the criteria for that decision. One of the major acceptance criteria will be successful
implementation of the requirements you have specified. These requirements may take many different forms including:
Functional specifications written in plain language (e.g. “The system shall…”)
Use case models and documents
Protocol specifications for interfaces to other systems
User interface mockups, prototypes, cognitive walkthroughs, or existing systems to emulate.
Descriptions of business rules and algorithms
These requirements documents form an important basis both for the vendor building the system and your acceptance testing plans. But recognize that these documents and models will never answer all the questions the vendor may have. Nor will they exhaustively
describe all your expectations. In fact, you may not even be aware of some of your expectations until they fail to be satisfied by the software. No amount of up-front requirements planning will flush out all your expectations therefore it is crucial to interact
with the software early and often to give yourself time to discover these unrealized expectations early enough to afford the Product Development Team time to implement them.
Expect the Product Development Team to discover many ambiguities and unforeseen circumstances in whatever requirements and expectations you have described to them. Plan to be available to the Product Development Team on a regular basis, the more frequent the
better. If you cannot participate personally, assign someone to act as your Product Owner proxy and ensure that they have full understanding of what you want to achieve and a clear mandate on what they can decide themselves and what needs your approval. On
agile projects, expect to provide at least one person full time for the entire duration of the project to clarify requirements and do continuous incremental acceptance testing.
Planning When to Test
A key part of the agreement with the Product Development Team should be the frequency with which you are supplied working software on which you can perform acceptance testing. In the pure sequential or classic waterfall model, you will likely need to wait until
near the end of the project before you receive any software. This may be a single “final” release that the vendor believes is complete, high quality code, or there may be a series of releases with names such as Alpha, Beta or Release Candidate that are provided
early enough to allow revision of how the software works based on the results of acceptance testing of each release. In most cases, the preliminary releases are either of lower quality or contain less functionality than the final release. Make your incremental
acceptance testing plans accordingly; don’t expect everything to be available in the early releases. What is to be available in each release should be the subject of discussions with the Product Development Team.
In the “agile” model, you should expect to receive working, potentially deployable software on a regular schedule throughout the entire project timeline. You’ll be asked to decide what functionality the Product Development Team should build in each iteration
(aka sprint) and you’ll need to be prepared to provide the detailed acceptance criteria for each feature or user story to the Product Development Team early in the iteration. This process is known as
Acceptance Test Driven Development
. When the software feature is finished, you’ll be asked to try out the newly built functionality and decide whether it works to your satisfaction or requires changes before you will accept it. In this
incremental acceptance testing
model you will need to test additional functionality each iteration. You may also need to dedicate one or more iterations at the end of the project to the final end-to-end acceptance testing and release activities.
Regardless of the process model you should expect the Product Development Team to do adequate testing as part of their readiness assessment. The expectation should be clearly understood by all parties and you should plan to provide the acceptance criteria and
detailed acceptance tests to the Product Development Team so that they can run the tests during their readiness assessment activities.
You will also need to have plans in place for how you will assess the software when it becomes available. At a minimum, you will need to know how many people and how much time will be required to do the assessment. The details of what tests they would run may
also be included or this may be determined nearer to the actual testing.
Assuming you will receive a new version of the software more than once, and this is a very safe assumption regardless of whether you use a sequential or agile process model, you’ll need to retest not only the newly introduced or changed functionality but also
all the previously built functionality. This can turn out to be a lot of effort. One strategy is to choose the tests to be run based on what was changed but this increases the risk of missing newly introduced bugs. Another strategy is to have a suite of automated
tests (see Automated Testing1regression testing
. Therefore make sure to ask your Product Development Team to include automated tests as part of their costing to ensure you don’t have an ever-increasing workload as more and more functionality is delivered.
The automated tests are really for the Product Development Team’s benefit as it helps them ensure that they are not introducing defects into the existing software. Also, you may want to request the regression tests suites to be shipped as part of the project
deliverables. Tests are assets and therefore you should plan on keeping them current for as long as you plan to use the product. Having the acceptance tests in place may reduce the cost of your future maintenance/extension projects (for which you may choose
to use a different supplier). But the acceptance tests may themselves require
test maintenance or refactoring
as the behavior of the system under test evolves.
Test Resourcing and Outsourcing
The testing you plan to do may be influenced by your knowledge of what testing the Product Development Team will have done prior to giving you access to the software and by the skills and experience of the people you have available to test. When you don’t have
the necessary expertise and you cannot or choose not to bring it in house you can elect to use a third-party organization to do your acceptance testing on your behalf. Be aware, however, that very few
have perfected mind-reading!! You will need to provide them with detailed direction regarding the nature and objectives of the testing you want them to do and the expected behavior (i.e. requirements) of the system to be tested. You’ll
also want to be very clear about the acceptance criteria and whether they are making the acceptance decision on your behalf or providing you with data to support your acceptance decision. Test outsourcing is common for specialized type of testing (e.g. security
testing). Test outsourcing may also be mandated by a certification body/regulatory authority that requires an independent verification/audit to be performed. For example, all tax preparation software packages may need to be tested for compliance to the tax
authorities’ standards for tax return information.
Estimating the amount of time and effort to allocate to acceptance testing is difficult without a good understanding of the nature of the functionality involved and the thoroughness of the readiness assessment that will be done by the Product Development Team.
A common strategy is to time-box the final acceptance test phase
and guess the number of cycles of acceptance testing plus bug fixing that will be required. This requires past experience to come up with reasonable guesses. If in doubt, guess high and
allow for several cycles of testing to give the vendor time to fix bugs. Warning: There is always a lot of pressure to guess low to allow earlier delivery of the software and the lower cost that this implies. But guessing too low will typically result in having
to choose between accepting the software on schedule with many known deficiencies or delaying the release to allow additional test&fix cycles to be performed until all important deficiencies are fixed and retested. The testing phase is often squeezed by
late delivery of the software by the Product Development Team and the testers are often asked to make up the delay by reducing the elapsed time for testing.
The alternative strategy involves incremental acceptance testing
; acceptance testing that is done frequently throughout the project. This improves the predictability of the delivery date in two ways. First, it finds bugs earlier allowing them to be fixed
long before the final acceptance test cycle. Second, it provides you the opportunity to learn how much testing is really required and how much time and effort it will take to do it. This learning happens early enough in the project that you can adjust your
plans for later iterations and the final test cycle so that the project can be delivered on time. Adjustments could include increasing or decreasing the planned number of test cycles, add staff or skills, increasing or decreasing the amount of test automation,
contracting a third party for specialized testing and so on.
What Do You Need to Test?
Ensure that you have a clear understanding of what will be tested by the Product Development Team and what you are expected to test. Typically, the Product Development Team is expected to do thorough testing of all capabilities and
(such as performance, reliability, usability etc.) before handing off system for acceptance testing. Of course, this is based on the Product Development Team’s understanding of your expectations which may vary from yours if the
communication is less than perfect which is almost always the case. Your responsibility during acceptance testing is ensuring the system is suitable for your own purpose.
The test cases you run during acceptance testing should cover everything you care about. This can include business functionality, operations functionality and non-functional characteristics of the system. Each of these areas may require different expertise
to test and possibly different stakeholder signoffs. Deciding who will sign-off based on what testing, done by whom, is an important part of acceptance test planning. The business lead may need to negotiate with other parties to determine the acceptance decision
process and the testing that provides the information.
Functional testing should include business workflow tests
that verify that all reasonable business scenarios or workflows function properly. More detailed
use case tests
can be utilized to verify that each business transaction or use case is implemented correctly. The happy path or main success scenario of each use case is not where bugs will typically be found; they tend to lurk in the dark corners of
the less commonly exercised scenarios. Business rule tests
can focus on verifying that the business rules and algorithms are implemented correctly. Applications with rich user interfaces typically require extensive testing of the user interface itself.
This testing can either be bundled with the use case test or you can have separate test sessions dedicated to
the behavior of the user interface.
The degree to which you need to test should be based on the level of confidence you have in your Product Development Team and the complexity of the deliverable. This can be based on past experience on previous projects or on what they have demonstrated on this
project. A Product Development Team that has provided full visibility of their internal processes and especially their readiness assessment activities may inspire much more confidence in the quality of the product. This could change the focus of your acceptance
testing activities from looking for obvious mis-implementations of the business logic or rules to higher value-adding activities like usability testing and business workflow testing both of which improve the product design and therefore the fitness for purpose
rather than just verifying the quality of the construction. In particular, the presence of an extensive suite of business-readable automated workflow, use case and business rule acceptance tests could change the emphasis from conducting extensive manual testing
of this functionality to spot checking. See Automated Execution of Functional Tests1
Non-functional testing (aka extra-functional testing) is often done by the Product Development Team during the readiness assessment or by a separate test organization. It should include verifying any non-functional characteristics of the system that are deemed
critical for acceptance as per the System Requirements Model described in Chaper 5, such as:
- Performance (capacity and response time) and stress (behavior when overloaded) tests,
- Security testing (enforcement of privacy policies, resistance to attack by hackers, etc.)
- Scalability (the ability to handle additional users or transaction loads in the future)
- Usability (the efficiency, effectiveness and satisfaction with which users can employ the software system in order to achieve their particular goals)
Ensure that you have a common understanding with the Product Development Team as to what Product Development Team functional testing they will do and how much information you expect to receive as input into the acceptance decision.
The people charged with operating and sustaining the system (herein called the operations department) will have very specific requirements that need to be addressed. You’ll need to be aware of these as you work with the Product Development Team exploring and
defining the requirements for the system. In a larger organization you may have a counterpart in the operations department who can provide the requirements and act as the acceptance decision maker from the operations perspective. See Operations Manager’s Perspective
in Part II.
At a minimum you’ll need to be aware of the time and resources complying with these requirements will entail. In some cases you’ll need to give up new functionality in order to ensure that the operational requirements are met. Some examples of operational requirements
- The system needs to integrate with the specific systems monitoring software in use at your company.
- Data needs to be migrated from a previous version of the system or from a system this one replaces.
- The system needs to have upgrade, startup and shutdown scripts that can be used during automated server maintenance windows.
- The system needs to be built using an approved technology stack. The operations department may not have the skills to support some technologies. E.g. A .NET shop may not have the skills to support an application built in Java or PHP. Some technologies may
be incompatible with technologies used at your company.
- The operations group may have specific windows during which they cannot install or test new versions of software. This may affect the release schedule you have devised and it may make on-time delivery of software by the Product Development Team even more
critical as small schedule slips could result in long in-service delays.
- There could be requirements for performing such service operations without degrading the current system
- Reliability requirements dictate that the system meets the agreed upon service level agreements (SLAs) such as uninterrupted uptime, transaction latency, .
- The system must provide self-monitoring, error detection, and system logging capabilities.
- The system must provide for robust recoverability (including the ability to verify multiple failovers and recoveries), disaster recovery, data backup & recovery, and all other aspects of system recovery.
- A full range of documentation must be provided including support documentation, both pre and post roll-out, training manuals, and other reference documentation.
- The system must be migrated to the new version of the platform.
- Multiple versions of the system must run side-by-side.
Other Parties Involved in Acceptance Testing
If there are other parties involved in the acceptance decision ensure that you have a common understanding about what they will test and how that feeds into the acceptance decision. Common parties include the operations department who would have a say in whether
the system can be installed and managed, usability practitioners who might need to ensure that the system complies with any usability (including accessibility) standards, and corporate security departments who need to ensure the system is secure before it
can be accepted.
Where Will You Test?
A key part of test planning is deciding what environments will be used for what kinds of activities and how the software is moved between environments.
Expect to have many activities happening concurrently as you do acceptance testing. The Product Development Team will likely be fixing the bugs that you report while you continue to look for further issues. This requires a separate environment. When the Product
Development Team has fixed enough bugs to warrant delivering a new version of the software they will need to do readiness assessment. This may require yet another environment if bug fixing is to continue in parallel. If a test organization needs to test the
software before passing it on to you for acceptance testing they will need to have their own test environment as well.
Software Delivery Process
Given that different groups will be working in separate environments, how does a new version of the software get installed in the environment? For example, you’ve finished acceptance testing on release candidate 1 (RC1) of the software, reported a number of
bugs, and the Product Development Team has fixed enough bugs to propose delivering a new version, RC2, to your test environment. What is the process for replacing the RC1 software in the acceptance testing environment with RC2? Does the Product Development
Team push the new version with or without any advance notice? Does the Product Development Team notify you of the availability of RC2 and wait for you to ask for it to be installed in the acceptance testing environment when you are ready to receive it? And
how long will the acceptance testing environment be unavailable while the upgrade is being installed?
When the software is delivered for acceptance testing, acceptance test execution must begin in earnest. It is important to know where you stand at all times with respect to the progress of testing and your current understanding of the suitability of the software.
As acceptance testing is invariably time-boxed by either contracts or customer expectations, it is important to ensure that the testing activities are progressing in a time fashion. The nature of the test plan determines the nature of progress monitoring. Agile
projects have acceptance testing spread out over the duration of the project and the results of testing are highly visible in the form of feedback to the Product Development team. There is plenty of time for the acceptance testers to learn from their mistakes
and misassumptions in time for the next round of acceptance testing.
Traditional sequential projects, on the other hand, have all testing highly concentrated at the end of the project. It is critical that these test activities be very closely monitored to ensure that any deviations from the test plan are quickly identified and
rectified. is fraught with many dangers and should avoid at all.
Bug Tracking & Record Keeping
The nature of the product being built and the context in which it will be used has a large bearing on the nature and degree of record keeping for both test execution and any bugs found. It may be sufficient to simply run tests and track any bugs found in an
informal manner as described in the sidebar Incremental Acceptance Testing and Bug Tracking on Agile Projects. In other environments, records need to be kept about what tests were run, against which version of the software, when, by whom and the actual results
that were observed.
At a minimum, any defects that were found during acceptance testing need to be recorded and triaged. Each bug should include which version of the software it was found in, whether it will prevent acceptance of the software (severity level) and who found it.
Bugs that must be fixed before the software will be accepted must be clearly communicated to the Product Development Team. Less severe bugs should also be recorded but should also be clearly labeled as not gating acceptance. Keep in mind that in many contexts
anything label non-gating will likely never get fixed so the real decision is “can you live with the product exhibiting this bug, on not?”
Each time the software is changed, whether to add new functionality or to fix bugs found during acceptance testing, new bugs may have been introduced. Therefore, any tests performed before the software was changed
need to be executed again on the new version of the software. This can be very time consuming if the regression tests are not automated.
Some Product Development Teams automate their unit tests as well as the functional tests that you will use to make the acceptance decision. Other Product Development Teams don’t know how to do this or don’t feel justified in including the cost of automated
tests in their cost estimates. They may need to be pushed into doing this by you, the Product Owner, insisting that automated regression tests as just as important as user manuals and built-in help. If you plan to use this product for any length of time, expect
the product to need to evolve. The evolution easier if the tests are automated save you large amounts of effort and help avoid having the quality degrade as the product evolves.
An important consideration in regression testing is the choice of data to be used in the tests. Regression tests can be executed with the same snapshot of production data for genuine repeatability. An approach that may be more representative of a real, evolving
production environment is to refresh the regression testing data over time. This more dynamic approach to regression testing sacrifices repeatability in return for the potential to find new problems.
Maintaining Multiple Versions of the Software
There will be times that you may choose to maintain several versions of the software at the same time. A common situation is after release, during the warranty period, if you are also developing a subsequent release at the same time. Another situation is when
you choose to support multiple version of software in production and need to apply any bug fixes to all supported versions. Be forewarned that there is an extra cost to maintaining versions in parallel as any bug fixes done in the support/warranty stream will
need to be propagated into the development stream at some point and retested there.
Sidebar: Incremental Acceptance and Bug Tracking on Agile Projects – Case Study
A recent IT project I worked on was pretty typical of how bugs are dealt with on agile projects. The developers would consult with the business lead (acting as the Product Owner) as they developed the functionality to make sure they understood what she (the
business lead) wanted built. They would jointly define the accetance tests for the functionality before or while the functionality was developed. When the developer had finished writing the code and had fully unit tested it, they would run all the previously
defined acceptance tests. Assuming the tests all passed, they would deem the user story ready for acceptance testing by the business lead. They would walk over the her (the 15 person team all sat in the same team room) and ask for a few minutes of her time.
They would demonstrate the functionality to her by executing the main acceptance tests in front of her. She would try a few more things that occurred to her. If everything worked as expected thus far, she would say something like “This looks good thus far.
Let me do some more detailed testing and get back to you with the results later in the day.” The developers would then go back to their desks to catch on their e-mail or pick and start researching the next user story while they waited for the final acceptance
decision. If the business lead found something she didn’t like during this preliminary testing, she would discuss the issue with the developers and they would jointly come up with options and decided how to proceed. If this meant changing the software, the
developers would go back to their desks to make the changes and this whole process would be repeat when they were done.
If the business lead found any bugs while doing the more detailed acceptance testing she would go over to the developers desk to discuss them. They would then agree discuss what to do. The business lead had several choices. Some bugs were serious enough that
she wanted them fixed before she would accept the user story; the developers would address these problems right away dropping anything else they were working on to do so. The key incentive was that any user stories that were almost finished but which hadn’t
been accepted by the Product Owner wouldn’t count towards the team’s velocity for the iteration. (The team was very proud of its velocity and was always looking for ways to improve it.) Another option she had was to address the bug in an already planned user
story that would be implemented in a future iteration. She would choose this option if the functionality was otherwise working and the particular test case was borderline as to which story’s scope it truly belonged to.
A third option was to turn the bug into a full blown user story. She chose this option if it didn’t fit into any of the other stories already in the product backlog.The final option was to track the bug on a separate bug list. Each bug was written on a 5x7
card and added to the “bug management system” which was simply a designated area of the whiteboard. Because most bugs were fixed within a few hours of being found, this list of bugs on the whiteboard rarely exceeded a dozen or so. During our biweekly iteration
planning meetings the business lead would pick the stories from the product backlog she wanted built and the bugs from the bug list she wanted addressed. Sometimes developers would suggest that a particular bug be included in the plans because development
of a user story would affect the same code.This approach to bug management worked very well for our team because we were colocated. We didn’t need a database or fancy reports because we never had more bugs on the board than the team could keep track of at
any one time. And it ensured that the quality of the product always met our MQR because we didn’t have a place to hide the bugs. This whiteboard-based approach won’t work on large, distributed projects but the philosophy of fixing bugs right away rather than
letting them build up to unmanageable numbers will. Gerard Meszaros
The Business Lead is the person nominated by the business to play the role of Product Owner for the duration of a project to develop a software-intensive solution to a business problem. They are responsible for making the decision about how to best utilize
the allocated money or resources in building or extending the system functionality.
A summary of the responsibilities of the Product Owner can be found in Chapter 1 The Basic Acceptance Process.