Chapter 13 - Operations Manager's Perspective
As an operations manager you likely have a completely different perspective on the deployment of a new or upgraded software system. Other parties care about whether the new functionality works properly and whether it will provide sufficient business benefit
in the near future; you care about what it will do to your support costs and whether it will be sustainable in the longer term. Therefore, your idea of acceptability will be quite different and it should influence the acceptance decision.
Role in Acceptance Decision
It is crucial that your role in the making of the acceptance decision is clear to everyone concerned. Do you provide information to the acceptance decision maker or are you one of several acceptance decision makers who operate by consensus, voting or veto?
Or perhaps you participate in the readiness decision before the software is made available for acceptance testing?
As with other forms of acceptance criteria it is highly desirable for these criteria to be clearly defined before the software is built so as to avoid unnecessary test&fix cycles before the software can be deployed. It is also desirable to being able to
conduct the testing incrementally throughout the project rather than just at the end. Incremental Acceptance Testing decreases project risks by uncovering issues early enough to have them fixed before the final acceptance test phase which is then much more
likely to pass without uncovering any show-stopper problems.
Operational Acceptance Criteria
The actual criteria that an operations group will use to determine acceptability will be dependent on the nature of the organization and the services it provides as well as the service level agreement for the software application in question. The base level
requirements typically include:
- Installer and uninstaller testing
- Auto-update or patch installation
- Compatibility with your Systems monitoring framework
- Performance under expected loads
- Systems management scripts for startup, restart and shutdown.
Additional acceptance criteria may include:
- Frequently-asked-questions and answers
- Troubleshooting scripts for the helpdesk
- Failover and recovery testing for high availability applications (including application shutdown, OS failure, hardware failure, occasional connectivity etc.)
- Testing of how system handles bad (ill-formated) data
- Penetration testing for public facing applications
- Stress testing for applications with potentially large numbers of users
- Data migration from a previous version of the system or from a system this one replaces
- Transactional data preservation and archiving
- IT Architecture approval of the technology stack.
- Simultaneous maintenance (performing service operations with constrained or no performance degradation of the current systm) Reliability, availability, meeting certain SLAs (uptime etc.)
- Self-monitoring, error detection, system logging
- Recoverability (must verify multiple failover and recovery), disaster recovery, data backup & recovery etc
- Support documentation (pre and post roll-out), training
- The system must be migrated to the new version of the platform
- Multiple versions of the system must run side-by-side.
- As an operations team:
- We should contribute to the requirements discussions as early as possible to ensure that realistic targets are being proposed for the product. We should help the product owner and the product development team understand operational contexts.
- We recognize criticality of our system under test and understand how significantly operational outages affect our business and reputation.
- We understand usage patterns and fluctuations based on the time of day, season, special events. The projected usage will most likely change over time.
- Managing test environment is important. Our testers would likely need the flexibility to make tuning and configuration changes. For example, many operability scenarios often require testers to purposefully configure the system wrong to observe the outcome.
- The choices we make for test environment will reflect a cost-benefit analysis based on your risk tolerance.
- Test lab procedures are crucial. It is imperative to keep a log of all changes that are made to the environment in order for us to be confident that when acceptance testing is done, the configuration and state of the product in the test environment is aligned
with the intended production configuration.
- We communicate regularly with the deployment team and the product development team.
Operational acceptance practices are discussed in Volume IV of this Guide. For a serious treatment and discussion of the patterns for performance and operability, see