Configuration Management Database – CMDB

• Definitions of planned Releases, including the constituent hardware and software CIs together with a reference to the original Change requests
• Records of the CIs impacted by planned and past Releases, covering both hardware and software
• Information about the target destination for the Released components (e.g. the physical location for hardware and the servers that will receive the software changes)

The software and or hardware components that compromise a new Release of an IT service should be assembled in a controlled manner to ensure a reproducible process. For Software, the standard approach is to receive new sources code from developers and then to generate the executables using controlled procedures on dedicated build hardware.

This process is called build management and is the responsibility of Release Management. It is quite common to automate the build procedures and automation should be controlled as additional CIs themselves. These may be generic or specific to the release.  Hardware components may also need assembling and configuring. This should be performed in a controlled and documented way. It is quite common to write scripts to automate the installation of systems and application software onto servers and workstations. depending on the implementation plan, this may be able to be performed in advance (for example, if the equipment is being replaced) or it may have occurred in situ in the live environment. Build management the responsibility of release management from the controlled test environments onwards.

Testing

Before a Release can be rolled out to the live environment, it should undergo stringent testing and User Acceptance. This should include functional testing, operational testing, performance testing and integration testing. Change Management should ensure that there is formal user acceptance and sign-off before Release Management can continue to roll out the Release. Insufficient testing is the most common single cause of failure of all Changes and Releases.

Back-Out Plans

A back-out plan should be produced to document the actions to be taken to restore the service should the rollout of a Release fail, either partially or totally. The production of back-out plans for each Change is the responsibility of Change Management, but Release Management has a role in ensuring that the back-out plans for each change within a release operate together to create a Release back-out plan.

These are two approaches and a combination

• A failed roll-out may be completely reversed to allow the complete restoration of the IT service to its previously known state. This is critical for a full release and preferable for a delta Release.
• Contingency measures may need to be taken to restore as much as possible of the IT service if it is not possible to restore it fully. This may be considered for a delta Release in the events that backout and recovery are not practical.

Examples:

• A release may have only partially been rolled out because the disk space on the target server is too small and so only some of the software changes have been implemented. In this case, it is necessary to document the procedure for backing out the failed rollout and restoring the application to the state prior to that failure. This would require the rollout plan to include a step to take a back-up of the application files being changed (or possibly the whole system) prior to the rollout of the new release.
• A rollout plan may include replacing some critical hardware or software component (Such as a mainframe computer or its operating system) and there may not be time to reinstate the old components, should something go wrong. The back-out plan may document how a spare hardware component, or mobile recovery, can be used to provide the service required.
• A back-out plan may also have to be invoked if the implementation of the change is taking longer than expected and will impact normal services to customers. In this instance, the back-out plan should contain timing details specifying the latest point at which implementation can continue before the back-out plan has to be invoked. In this instance, it is important to try to determine the length of the back-out, so that it can be completed before the customer is impacted.

A backout plan should be verified as part of the risk assessment of the overall rollout plans, and it should be agreed with the end-users as sufficient. For example, a service might not be critical, in which case it might be agreed that manual procedures can be used whilst the rollout is completed successfully. Back out plans should be tested as part of the process of verifying the rollout process.

Benefits of Release Management

As organisations become more dependent on IT, the effective control and security of their computer systems assume greater importance. Organisations should be able to cope with a high frequency of software and hardware Releases across the organisations without sacrificing IT service quality. The controls and mechanisms within release management help support this requirement in an efficient and economical manner.

The principal benefits of Release Management, when combined with the effective Configuration Management, Change Management and operational testing functions are:

• A greater success rate in the Release of hardware and software and therefore an improved quality of service delivered to the business
• Consistency in the release process of the hardware platforms or software environments
• The minimization of the disruption  of the service to the business through synchronization of releases within packages involving hardware and software components that have been constructed under Change Management
• Stable test and live environments, because changes are normally combined into Releases and so there should be fewer individual implementations
• Better use of user resources because of combined efforts when testing new Releases – this also means that it will be easier to justify the cost of system-wide regression testing
• The minimization of regression-testing requirements, offering greater coverage than is possible with small Changes that occur frequently or too close together
• Better expectations setting within the organization on the publication of a release schedule in advance
• Error reduction through the Controlled Release of hardware and software to the live environment is maintained both of software distributions and of hardware changes
• A complete record (or audit trail) of changes to the live environment is maintained, both software distributions and of hardware changes.
• Proper control and safeguarding of hardware and software assets, upon which an organization may be heavily dependent
• An ability to absorb high rates of change to the live systems, effectively and without adversely affecting IT service quality – this is achieved by releasing a large number of changes together as a single, controlled and well-understood release
• The ability to build and control the software used at remote sites from a central location
• Savings in support costs through the ability to maintain consistent software over a large number of locations
• Reduced likelihood of there being illegal copies of the software in use at any location
• Easier detection of wrong versions and unauthorized copies of the software
• Reduced risk of unnoticed of viruses or other malicious software
• reduced time to release to be rolled out to customers
• The smoother transition of releases from the development activities (projects) to the customer’s business environment

As the efficiency of Release Management grows, the productivity of IT services staff is likely to increase. More importantly, productivity benefits are likely to be realised amongst end-users, as releases should be fewer and better planned, with appropriate training and documentation of higher quality.

Potential Problems of Release Management

The potential problems in implementing Release Management are:

• There may be some initial resistance from staff who are familiar with the old procedures and who may not welcome change. To overcome this, the benefits of the new procedures should be carefully explained during an awareness campaign, and there should be a close working relationship with the team implementing the change.
• Experience has shown that those teams that are in the most need for help with release management often have the latest time to adopt it. One way around this is to look for minimal-impact ways to gain a foothold and to provide them with some hands-on assistance at the beginning.
• Circumvention of the release Management procedures may be attempted. This needs to be dealt with firmly, particularly if it involves the installation of unauthorized software, as this is the most likely cause of viruses or untested software that make support very difficult and costly.
• Staff may also be tempted to bypass the standard procedures to install an emergency fix. This should be banned and disallowed by software security rules as far as possible.
• There may be some reluctance to carry out a controlled build in the test environment, relying instead on copying over software from the development environment.
• In a distributed system, difficulties may arise if new versions of software or hardware are not installed and activated on time at remote locations. The use of software distribution tools, backed up by regular audits, can help avoid this problem.
• Some people (including IT management) may view the Release Management procedures ac cumbersome and expensive. Nevertheless, they are almost invariably necessary to cope effectively with software changes.
• Unclear ownership and responsibilities between operational groups and development team (project teams) may exist – for instance, there may be a lack of understanding about who is responsible for managing components of a release at different points in the release cycle.
• Insufficient resources available for adequate testing will reduce the effectiveness of these procedures or a high number of variants in the live environment may limit complete testing
• If sufficient machine and network resources are not available, then it may be impossible to build and test new releases and equipment adequately, or in a timely fashion. Time needs to be allowed in testing for a release to fail the first time and be reworked. Similarly, back-out procedures should be proven in a controlled test environment.
• A lack of understanding of the release contents, build and installation components, can lead to mistakes.
• Testing in one area may be acceptable but may fail in another area – e.g. different infrastructure or parameters not set the same values.
• Staff may be reluctant to back out from a release, and there may be pressure to transfer inadequately tested software and hardware into the live environment.
• Poor, restricted or non-representative testing environments and procedures may exist.

In all cases, the Release Management plan and the principles and policies on which it is based may be subject to a publicity campaign at the outset and periodically thereafter in order to set expectations and goals.