Technical Debt Meets Continuous Deployment
As you would expect in a conference entitled velocity, and in a follow-on devops day, speeding up things was an overarching theme. In the context of devops, the theme primarily manifested itself in lively discussions about the number of deploys per day. Comments such as the following reply to my post Ops Driven Dev were typical:
Conceptually, I move the whole business application configuration into the source code…
The theme that was missing for me in many of the presentations and discussions on the subject was the striking of a balance between velocity and quality. The classical trade-off in process control is between production rate and product quality (and safety, but that aspect [safety] is beyond the scope of this post). IMHO this trade-off applies to software just as it applies to mechanical or chemical processes.
The heart of the “deploy early and often” strategy hailed by advocates of continuous deployment is known deployment state to known deployment state. You don’t let the deployment evolve from one state to another before it has stabilized to a robust state. The power of this incremental deployment is in dealing with single-piece (or as small number of pieces as possible) flow rather than dealing with the effects of multiple-piece flow. When the deployment increments are small enough, rollback, root cause analysis and recovery are relatively straightforward if a deployment turns sour. It is a similar concept to Agile development, extending continuous integration to continuous deployment.
While I am wholeheartedly behind this devops strategy, I believe it needs to be reinforced through rigorous quality criteria the code must satisfy prior to deployment. The most straightforward way for so doing is through embedding technical debt criteria in the release/deploy process. For example:
- The code will not be deployed unless the overall technical debt per line of code is lower than $2.
- To qualify for deployment, code duplication levels must be kept under 8%.
- Code whose Cyclomatic complexity per Java class is higher than 15 will not be accepted for deployment.
- 50% unit test coverage is the minimal level required for deployment.
- Many others…
I have no doubt whatsoever that code which does not satisfy these criteria might be successfully deployed in a short-term manner. The problem, however, is the accumulative effect over the long haul of successive deployments of code increments of inadequate quality. As Figure 1 demonstrates, a Java file with Cyclomatic complexity of 38 has a probability of 50% to be error-prone. If you do not stop it prior to deployment through technical debt criteria, it is likely to affect your customers and play havoc with your deployment quite a few times in the future. The fact that it did not do so during the first hour of deployment does not guarantee that such a file will be “well-behaved” in the future.
Figure 1: Error-proneness as a Function of Cyclomatic Complexity (Source: http://www.enerjy.com/blog/?p=198)
To attain satisfactory long-term quality and stability, you need both the right process and the right code. Continuous deployment is the “right process” if you have developed the deployment infrastructure to support it. The “right code” in this context is code whose technical debt levels are quantified and governed prior to deployment.
The Agile Executive 
The point of continuous deployment is not just to deploy code more frequently.
No one who is advocating continuous deployment is not also advocating a commitment to quality code in both the process and culture.
There is a word for choosing to implement continuous deployment without balancing it with metric driven decisions: downtime.
I’ll post more on this topic later…
Regards,
Andrew
Andrew Clay Shafer
July 1, 2010 at 10:54 am
Andrew, I am sure your thoughts will be much read and appreciated. Can’t wait to see your post.
Thanks so much for contributing!
Israel
israelgat
July 1, 2010 at 3:51 pm
[…] debt criteria that could form the core of a devops boundary object are mentioned in the post Technical Debt Meets Continuous Deployment. Corresponding criteria could be used in the boundary object to satisfy operational requirements […]
Boundary Objects in DevOps « The Agile Executive
July 6, 2010 at 6:44 am
I think more important than specific metrics is the culture. Culture determines how people approach a problem and how they react when things go wrong. I remember years ago Alistair Cockburn describes retrospectives as the core agile process — because you could derive the other practices from retrospectives — I think starting with the right culture you’d end up with the right level of quality.
Paul Julius wrote up a cultural maturity model for continuous integration last year that I think could be easily extended for continuous deployment:
http://www.anthillpro.com/blogs/anthillpro-blog/2009/09/11/enterprise_ci_cultural_maturity.html
Jtf
Jeffrey Fredrick
July 10, 2010 at 6:54 am
No doubt, Jeffrey, culture is critically important. As a matter of fact, my Agile 2010 workshop How We Do Things Around Here In Order to Succeed will explore the cultural aspects with special emphasis on the way these aspects manifest themselves on the cusp between dev and ops.
I really do not know whether culture is more important than metrics or the other way around. The important thing is striking a good balance between the two as well as other ‘ingredients’ of the ‘secret sauce.’ For example, organization and alignment are as important as culture and metrics IMHO.
The Enterprise CI Cultural Maturity Model by Paul Julius is indeed very interesting and I am thankful to you for referencing it. Having said that, it seems to me that the model Paul discusses is more about behavior than culture. I believe cultural changes take a very long time – ten years is a good magic number to use. The advice I repeatedly give to my clients is to focus on behavioral changes, not on cultural changes. The reason for so doing is very pragmatic: nobody gets such a long period (~10 years if you accept my premise) to introduce change.
With the distinction between culture and behavior noted, extending Paul’s model to continuous deployment is certainly of interest to me. May I ask what are your thoughts along these lines?
Thanks!
Israel
israelgat
July 10, 2010 at 5:16 pm
Perhaps Casper Jones’ updated article on Getting Quality Right would make a good companion to this one. Good stuff, glad I found this site.
Curtis
July 10, 2010 at 9:17 am
Hi Curtis:
Indeed, the Capers Jones article you recommend makes an excellent companion. Here is a tiny URL for it: http://bit.ly/aWZ2tc.
Thanks so much for both the contribution and the kind words!
Israel
israelgat
July 10, 2010 at 11:04 am
Hey Israel
Interesting post. I agree that continuous deployment relies on high quality code, and that it can only be achieved if you’re rigorous about the quality of the app. However I think that automated testing is the key to ensuring high quality. That means >70% unit test coverage, automated acceptance test coverage of at least happy path scenarios, and preferably automated capacity testing as well.
As Andrew points out, it also requires cultural change: excellent collaboration between developers, testers and systems administrators from the start of every project. All this stuff is the focus of my book (see the website link in this post).
I do like your focus on metrics – some other interesting ones are efferent and afferent coupling. However ultimately I think the metrics that really matter are cycle time, MTBF and MTTR – it would be interesting to correlate code metrics with these operational level metrics. John Allspaw did an interesting talk on these kind of metrics that is worth checking out: http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change
Thanks,
Jez.
Jez Humble
July 10, 2010 at 9:53 am
Thanks for the kind post, Jez. I believe we are in total agreement on just about everything you touched on – Andrew, John, correlation of code metrics with operational metrics, etc.
One other factor worth mentioning in this {code –> ops –> biz} context is forthcoming changes to the CMDB. See, for example, Is the CMDB Irrelevant in a Virtual and Cloud Based World?
And, needless to say, best of luck with your forthcoming book. I am anxious to read it.
Israel
israelgat
July 10, 2010 at 12:05 pm
Having a speedy deployment is certainly part of the agile practice, and I think your criteria for IT deployment teams include important methods to prevent errors. The cumulative effect of codes with technical difficulties is worrisome, but automated deployment will speed up this process. Once a code’s bugs are worked out, automated deployment can help avoid the backlog and keep up with the faster deliverables agile promotes.
XebiaLabs
July 15, 2010 at 8:02 am
[…] https://theagileexecutive.com/2010/07/01/technical-debt-meets-continuous-deployment/ Posted by neilj at 8:48 pm Tagged with: Continuous deployment, kanban, lean […]
In Praise of Continuous Deployment | fragile
February 13, 2011 at 2:49 pm