I was intrigued by a recent article I read on InfoQ about a process known as Continuous Production. As the name implies, the idea is that rather than using a controlled series of incremental deployments to your production platform every month, or every week, you build a process that enables you to deploy software to production daily, or even continuously. What intrigued me wasn't so much that the idea was novel. It struck a raw nerve that made me consider what processes and controls must be in place for a company to streamline its processes sufficiently to enable high-quality deployments of high-quality quality software continuously. Releasing software to production every 30 minutes (as in the case of Flickr) is a compelling challenge to any software development process. Paul Duvall's article on testearly.com referenced by the InfoQ article above shed some light on the details of this process.
Over the past year, I’ve been advocating a much more intensive criteria for Continuous Integration with my peers and customers, which is more like “Continuous Production” meaning you can produce production-ready software on a daily or even a continuous basis. Effectively, we’re talking about extending the Continuous Integration model into all environments (Test, Stage and Production), not just the development environment.
Before discussing the process itself, consider the barriers to such a process. First, I believe any barrier that would keep you from adopting such a process probably stems from an unhealthy set of, or lack of, existing processes. The ability to adopt a process of continuous production while maintaining a high quality product is the ultimate measure of a software development process. It requires a high degree of maturity in all the processes from compilation, testing, deployment, to final deployment verification. Even in development shops where continuous production isn't desirable, the goal of being able to support continuous production is a worthy one; it provides a framework to develop a healthy set of processes for software development.
Obviously, applications deployed on the web are the best candidates for continuous production. The software is deployed to a controlled environment and a rolling upgrade can normally prevent any noticeable downtime. However, in cases where software is downloaded and installed by end-users, or worse, in cases where it is centrally deployed and undergoes traditionally long deployment and test cycles, it isn't practical to deploy updates frequently. Even in these cases, however, it can still enable healthier internal processes if the compilation, testing, and building of final installation packages was completely automated.
Consider the some of the process requirements for continuous production.
Automated Source Code Compilation and Unit Test Execution
After checking a change into source control (you use a good source control tool, right?), the source should be compiled automatically and the appropriate unit tests should be executed providing immediate feedback to everyone involved on the status of the build.
Quality Assurance
To make this system, your QA group needs to work seamlessly with your development group. They should understand how to test the software without complex descriptions of how to test the specific change that was made. Better yet the QA group should use an automated regression test to verify the correctness of the software. All manner of tests, including unit tests, regression tests, load tests, etc. should be run either automatically or very easily by the QA group.
Automated Deployments
I work in a web application environment, so my perspective is skewed toward server-side deployments. However, many of the same principles are equally applied to other project types. Once the testing has verified the quality of the software, you should be able to have the software automatically deployed to the appropriate servers. This step is considered very risky in many environments - great care should be taken to ensure a quality deployment. Many unit tests, some regression tests, and other forms of deployment verification are available to verify that the software was deployed successfully. Same story - these verifications should be automated to reduce the chance of the task being forgotten or performed improperly.
I suggest that it is advantageous for development groups to adopt continuous production processes, especially when it seems more difficult to do so. The barriers to adopting these processes are probably symptoms of broken or inefficient processes. Even if continuous production isn't desirable, consider the changes required to make it easier to update the software seamlessly, automatically, and often. In cases where there is concern for negative user reaction to frequent changes to the software, consider establishing an expectation of frequent updates and providing the ability to keep users abreast of the changes. In cases where there is concern that such a process cannot produce quality software, consider the changes to your process to increase the quality in such an environment. In all cases, even if you don't plan to release the software to customers continuously, I suggest that your process (overall) will mature and enable you to produce higher quality software.
What is the biggest barrier that might keep your development group from adopting a policy of Continuous Production? Make a list of those barriers, and then label it "Areas for Process Improvement".
