A Cautionary Tale
The following is the story of a project where the person in charge of defining and accepting a product fails to rise to the challenge. It describes what goes wrong and why. It also provides an alternate outcome of how the project could have worked out if
proper acceptance practices had been used.
Bob Kelly is a mid-level manager in the marketing department at Acme Products. He is in charge of the Mid Sized Markets product group and is the product manager of its core product, the XCelRator. He’s just finished a bunch of market research and has come up
with a great idea for a new module for the product that he believes will double the revenue of the product. The Product Development division has a team that is just winding down its current project and Bob is keen to get the team started on building the new
module. He calls a meeting and runs through the PowerPoint slide deck he’s been using when talking with focus groups and potential clients. He concludes the presentation by laying out the key delivery dates which must be met to allow the company to showcase
the product at the annual trade show. He asks the project manager to draw up the project plan and get the team working on the new module.
The Dev Manager meets with his team to collect some information and then he defines the project plan. Based on the delivery date he defines intermediate milestones for Requirements Complete, Design Complete, Code Complete and Test Complete.
Team starts writing requirements documents. At Requirements Complete, dev manager declares requirements complete on time. The product development team (dev team) knows full well that some areas of the requirements are still too vague to implement. They will
have to get clarification on the details as they do the design. Hopefully the product manager, Bob, will be more available to answer questions about the requirements as his lack of availability is at least partially to blame for the requirements still being
At Design Complete, ditto. Dev team knows it hasn’t designed some parts of the functionality except at a very cursory level. They will have to fill in the details as they code. Meanwhile, Bob, the product manager thinks everything is proceeding according to
plan and that he’ll have a great product to demo at the trade show. He starts thinking about how he’ll spend the bonus he’s sure to get for completing this project on time. He calls his architect and asks her to start drawing up plans for the extension to
his house, complete with indoor/outdoor swimming pool.
At code complete the team requests an extra month to finish coding parts of the functionality they haven’t had time to do yet. The Test Manager asks what parts of the software are complete so that testers can start testing and the team responds “None, we are
each working on a different part of the system and it will all be finished at about the same time.”
To ensure the product is ready for the trade show, Bob asks the test manager to make up the schedule by compressing the duration of the test phase. He reduces the number of planned test cycles to reduce the elapsed time based on assurances that the code will
be in great shape due to the extra time the dev team is being given. Bob still believes the product can be ready for the trade show (but nonetheless he asks the architect to scale back the extension to his house by removing the enclosure for the pool thinking
to himself “I’ll still be able to use it most of the year and I can always add the enclosure with my release 2 bonus.”)
As the new Code Complete deadline approaches, the team asks for another month. Bob reluctantly gives them 2 weeks. “Two weeks later will make it tight for the trade show but we can demo the beta,” he hopes.
The dev team finally delivers the code two months late. The test team starts testing but finds significant problems that prevent completion of the first pass of the test cases. Testing halts after less than a week. The dev team takes a month to fix all the
major show-stopper bugs before handing the code back to test. Test team makes it through the full test cycle this time but finds several hundred defects. Development starts fixing the bugs as they are found.
The test team finally receives a new build that the dev team says has all the Sev 1 and 2 bug fixes. Almost immediately they find several new Sev 1 regression bugs. Development disagrees that some of the bugs are even valid requirements. “The requirements never
said anything about …” and “Why would anyone ever do that?” they exclaim. Bob, the product manager, has to be brought in to settle the dispute. He agrees that some of the bugs aren’t real requirements but most are valid scenarios that he never considered but
which need to be supported. Test tells Bob that there is no way in … that the original schedule can be met.
After 4 test&fix cycles that took 50% longer than the original schedule (let alone “making up the schedule”), most of the Sev 1 and 2 bugs appear to have been fixed. There is still several hundred Sev 3 and 4 bugs and test team has stopped bothering to
log the Sev 5’s (poorly worded messages, field labels or button names, etc.). Test team says the product is not ready to ship and will require at least 2 more test&fix cycles before they will agree to release it.
The product is now several months late and won’t be ready for the trade show even in alpha release. “We can still show the box and do some very controlled demo’s.” Bob assures his boss. Bob is seeing his bonus diminish with each week of reduced sales in this
fiscal year. He decides to ship the product, overruling the Test department. He revises the sales forecasts based on the late launch caused by the “underperformance of the development and test teams.”
The manager of the operations department hears about this too late and comes storming into Bob’s office. “I hear you are planning to release the new module with 300 known Severity 3 defects. Do you have any idea what this will do to our support costs?” A counterproductive
argument ensues because there is no turning back at this point; the announcements have been made and to change plans now would be hugely embarrassing to the company. “I sure hope this works out OK.” thinks Bob to himself; at this point he doesn’t feel like
he’s in charge of his own destiny.
Bob’s marketing machine has been in high gear for quite a while and has generated quite a bit of pent up demand, especially since some users were hoping to be using the product months ago. Users try using the new product in droves. They run into all manner
of problems and call the help line which is overwhelmed by the call volumes. Many users get busy signals; the lucky ones wait listening to recorded announcements for long periods of time. The help desk has to bring on extra staff who need to be trained very
hastily and therefore cannot provide very good service to the customers. Some customers give up trying to use the product because of long waits or poor support.
Many of the user problems are related to usability issues that were not detected during the rushed testing because it was so focused on the Sev 1 & 2 bugs (and the usability stuff was usually rated 3 or below, many of which weren’t even logged due to the
focus on the 1’s and 2’s.) At peak usage times the system slows to a crawl; the dev team is called in to figure out why it is so slow distracting them from working on some of the improvements identified during testing. Users have trouble importing data from
prior versions of the software or from competitors’ products.
A large percentage of the users abandon the product after the free trial is over; the conversion rate is less than half of the projected rate. Revenues are running at 40% of the revised projections and less than 20% of the original projections. Support costs
are 50% over original projections due to the hiring binge in the user support centre.
The capital cost is 35% over the original budget and has eaten into the planned budget for a second module with additional must-have functionality. Instead, the 2nd release has to focus on improving the quality. The new module will have to wait until 2nd half
of next year. The product manager revises the revenue projections yet again (and calls the contractor to cancel the addition to his house). Shortly after sending in his monthly status report his phone rings. It is the VP, his boss, “requesting” he come to
his office immediately...
What Went Wrong
- Product manager Bob decided scope, timeframes and resources thereby leaving only quality as the derivable variable. This resulted in the dev team being overcommitted at the product manager’s insistence. (Sometimes the dev team will overcommit out of optimism
but in this case the product manager did it to them.)
- Sequential process is inherently opaque from a progress perspective. The first milestone that isn’t easy to fudge is Code Complete. The first realistic assessment of progress is well into the test cycle.
- Product manager wasn’t available to clarify vague and missing requirements. Testers were not involved soon enough to identify the test scenarios that would have highlighted the missing requirements. But no one could prove the requirements were incomplete
so RC was declared on time.
- Product development team couldn’t prove design wasn’t done (because it is a matter of opinion as to how detailed the design needs to be) so Design Complete was declared on time. The sequential process doesn’t give the development team a good way to estimate
its true velocity until late in the project, when the code is mostly written and debugged.
- Dev team cut corners to make the new (already late) Code Complete deadline. The code was written but much of it wasn’t properly unit tested. The team was deep in unit test debt, they knew it, and would have told anyone who asked but no one wanted to hear
- The quality was awful when delivered to Test. So it had to be redone (we never have time to do it right but we always make time to do it over!)
- Test was asked to recover the schedule (typical!) but testing took longer because the poor quality code required more test&fix cycles to get it into shape.
- No clear definition of “done” so decision is made on the fly when emotions get in the way of clear thinking. The product manager let his attachement to his commitments override any sensibility about quality.
- The operations acceptance criteria were never solicited and by the time they were known it was too late to address them.
- Sequential process hid true progress (velocity) until it was too late to recover. There was no way to scale back functionality by the time it became undeniable that it won’t all be done on time. There was no way to reduce scope to fit the timeline because
the sequential-style approach to the project plan (Requirements Complete, Design Complete, Code Complete, Testing Complete milestones) caused all features to be at roughly the same stage of development. Therefore cutting any scope would result in a large waste
of effort and very little savings of elapsed time. (Only the effort of bug fixing would have been saved.)
- Development was rushed in fixing the Sev 1 problems so that testing could get through at least one full test cycle. This caused them to make mistakes and introduce regression bugs. It took several test&fix cycles just to fix the original Sev 1&2’s
and the resulting regression bugs. In the meantime the Sev 3’s piled up and up and up. This resulted in several more test&fix cycles to fix and even then more than half were still outstanding.
- Lack of planning for the usage phase of the project resulted in a poor customer support experience which exacerbated the poor product quality.
How it Could Have Gone
Bob, the product manager comes to product development team with requirements that are representative of the customers’ expectations.
Dev team estimates requirements as being 50% over teams capability based on demonstrated development velocity. The product manager is not happy about this. Dev team proposes incremental development & acceptance testing. Instead of 4 sequential milestones
based on phases of development they suggest 4 incremental internal releases of functionality where each increment can be tested properly. This will allow them to determine their development capacity (or“velocity”) which will help the product manager plan subsequent
milestones more accurately.
Bob selects first increment of functionality to develop. With the help of the dev team he defines the usage model consisting of user personas and tasks. The Dev team whips up some sketches or paper prototypes and helps Bob run some
Wizard of Oz
tests on the paper prototypes that reveal some usability issues. Bob adjusts the requirements and dev team adjusts the UI design. Bob, the product manager works with dev team and the testers to define the acceptance tests. The dev team automates
the tests so they can be run on demand. They also break down the features into user stories that each take just a few days to design and test.
Team designs software and writes code using test-driven development. All code is properly unit-tested as it is written. All previously defined automated tests are re-run several times a day to make sure no regression bugs are introduced as the new feature is
implemented. As each feature is finished, as defined by the feature-level “done-done” checklist , the developer demos to product manager and tester who can point out any obviously missing functionality that needs to be fixed before the software is considered
“ready for acceptance testing”.
As part of incremental acceptance testing they do identify a few new usage scenarios that were not part of the requirements and provide these to Bob as suggestions for potential inclusion in the subsequent increments. Bob adjusts the content of the next increment
by including a few of the more critical items and removing an equivalent amount of functionality. He also contacts the operations manager to validate some of the operational usage scenarios identified by the dev team and testers. The operations manager suggests
a few additional operational user stories which Bob, the product manager, adds to the feature backlog for the next increment.
At the end of first increment of functionality (which took 5 iterations to develop, one more than expected) dev team runs the non-functional qualities tests to verify the software performs up to expectations even with 110% of the rated numbers of users. The
first test results indicate it can only handle roughly 50% of the expected users and gets even slower as the database starts to accumulate records. They add some work items to the schedule for the second increment to address these issues and warn testing about
the limitations to avoid their wasting time stumbling onto them. Testing finds only a few minor bugs during execution of the functional test scripts (the major stuff was all caught during incremental acceptance testing.) They move on to doing some exploratory
testing using soap operas and scenarios as their test session charters. These tests identify several potential scenarios Bob never thought of; he adds them to the feature backlog.
Bob, as the product manager arranges to do some usability testing of the first increment with some friendly users based on some of the usage scenarios identified in the user model. The results of the testing identify several enhancements that would improve
the user satisfaction. Bob adds these to the things to do in the next increment of functionality.
Bob calculates that demonstrated development velocity is 25% less than original estimates. Based on this he adjusts his expectations for the functionality to be delivered in the release by removing some of the less critical features and “thinning” some of the
critical features by removing some nice-to-have glitz. “Better to deliver good quality, on time than to try to cram in extra functionality and risk everything” he thinks to himself.
The development team delivers the 2nd increment of functionality with similar results as the first. The work they did to improve performance results in the performance tests passing with flying colors with acceptable response times at 120% of rated capacity
and no degradation as the database fills up with transactional data. They add some tests for penetration testing and schedule a security review with the security department. Bob makes some adjustments to the functionality planned for the 3rd increment of functionality.
He includes some functionality to address operational requirements such as migrating data from earlier versions of the software and importing data from competitor’s products; he wants to make it real easy for users to adopt his product. He’s happy with how
the project is progressing and confident that they will be able to deliver excellent quality and on schedule.
In the third increment, the development team delivers 20% more functionality than originally planned. The product manager had to scramble to provide the acceptance tests for the extra features brought forward from the fourth increment. Based on this, the product
manager is able to plan for some extra functionality for the fourth increment. He considers reviving some of the functionality that he’d cut after the first increment but decides it really wasn’t that useful; a much better use of the dev teams efforts would
be some of the usability enhancements suggested by the last round of usability testing. He also adds functionality to make it easy to upgrade to the next (yet unplanned) release without having to take down the server. That will help reduce the operational
costs of the software. “Yes, this is going to a great product!” he says to himself.
As the development team is working on Increment 4, the product manager discusses the Acceptance Test Phase of the project. “We had originally planned 3 full test&fix cycles each of 2 weeks duration with a week for fixes in between for a total of 8 weeks
of testing.” recounts the Test Manager. “But based on the results of testing Increments 1, 2 and 3 I’m predicting that we’ll only need one full test cycle of 2 weeks plus a 1 week mini-cycle of regression testing for any fixes that need to be done (and I’m
not expecting many.) The automated regression testing the dev team is doing as part of readiness assessment has been preventing the introduction of many regression bugs and the automated story tests we’ve been co-authoring with you and the dev team has prevented
misunderstandings of the requirements. This is the most confident I’ve ever felt about a product at this point in the project!”
There are good ways to effectively define, build and accept products and there are ways that may result in disappointment. The rest of this guide is dedicated to helping you understand which is which.