Posts Tagged ‘Continuous Deployment’
A New Chasm is Forming
http://www.flickr.com/photos/96483949@N00/192144459/
Recently I had frustrating experiences with the bureaucracies of two companies: a company who is bringing me in to consult on continuous value delivery and another company who is interested in a seminar on the consumerization of enterprise software. While the two companies could not be more different, good-hearted engagement managers in both companies cautioned me in advance that their bureaucracies would drive me nuts. Based on my experience to date they were not kidding…
It is a striking contrast between the ease with which I can get precious data from my social network anytime I need it versus the unbelievable waste of time answering the very same question in two or three redundant forms used to screen me as a supplier. This contrast leads me to conclude we are witnessing the formation of a new chasm. It is between companies who stick to their old bureaucratic patterns with respect to suppliers versus those that realize that a supplier these days is a “prosumer.” He/she might provide services one day, consume (other) services the other day.
The business opportunity this chasm presents is providing efficient marketplace infrastructures. Anyone who can collect my data once and provide it as needed to multiple companies I interact with as a supplier will be doing me, and countless number of social networking aficionado, a huge service. Time is simply too precious to be wasted typing in the maiden name of my mother multiple times.
The distinguished economist Ronald Coase perceived reduction of transaction costs as the essence of the firm. His thoughts of more than 70 years ago are right on these days. The bar for transaction costs is “fill in the details only once”. Once in this context means “once in your lifetime.”
Recommendation: Examine the way your company acquires new customers versus the way it brings aboard suppliers. Something is wrong if your company’s procurement folks routinely tell suppliers “we know you have already given this information, but ‘they’ would not accept it from ‘us’ if you don’t fill this extra form as well.” This being the case, you need to rethink your approach to composite value chains.
The Punched Cards in the Middle of Your Devops
Source: http://www.flickr.com/photos/pjen/1070766105/
In her foreword to Gender Codes, Linda Shafer vividly describes the flow of programming work at NASA in 1965:
Following a design, we wrote – by hand – computer program instructions on large coding pads (80 columns per instruction, the same width as a Hollerith punched card). A Courier came by twice each day, picking up the coding pads and delivering yesterday’s instructions that had been magically translated into a different physical medium – card decks. Put some paper on a cart one day and presto, the next day, a stack of 7 and 3/8 inch by 3 and 1/4 inch, stiff paper sheets with holes punched in them were delivered. These cards constituted the program, which was sent to the machine room where operators fed the decks through the card reader.
Fast forward to 2010. If you have not yet moved to Continuous Deployment, metaphorically speaking you are still punching card. Not only are you falling behind on value delivery, you are missing up on the following great point made by colleague Josh Kerievsky:
What fascinates me most about #continuousdelivery is how it changes the way we design and collaborate on code.
‘Super-Fresh’ Code
Source: http://www.flickr.com/photos/21560098@N06/3636921327/
Misty Belardo published a great post/video clip on Social Media in which she describe the effect of the ‘Super-Fresh’ web on brands:
… millions of people are creating content for the social web… the next 3 billion consumers will access the Internet from a mobile device. Imagine what that means for bad customer experiences! The ‘super-fresh’ web will force brands to engage with its customers…
I would contend we are also going to experience ‘Super-Fresh’ code in not too long a time. Such code is likely to emerge as the convergence of two overarching trends:
- Continuous Integration –> Continuous Deployment –> ‘Super Fresh’ Code. Sophisticated companies are already translating velocity in dev to competitive advantage through Continuous Deployment. ‘Super-fresh’ code is a natural next step.
- Open-sourcing –> Crowd-sourcing –> Expert-sourcing. Marketplaces for knowledge work expertise are becoming both effective and efficient. For example, uTest indicates “… 25,000+ testers in more than 160 countries.” A marketplace for mobile application developers could probably be organized along fairly similar lines.
No doubt, complex software systems of various kinds will continue to be produced through more conventional processes for many years to come. However, ‘Super-Fresh’ code will establish itself as a new category. Code in this category will owe its robustness (and creativity!) to millions of people creating software and fixing it in extremely short time, not to process rigor.
In case you are still wondering about the premise, I would like to point out two corroborative facts:
- It is a small step from content to code.
- Various mobile applications are already developed and tested today in a different manner from the way web applications have been done.
A fascinating link exists between ‘Super-Fresh’ web and ‘Super-Fresh’ code. The dynamics (“Imagine what that means for bad customer experiences!”) Misty discusses in her blog post are a major driver for the evolution of knowledge work marketplaces and for the production of ‘Super-Fresh’ code.
A Devops Case Study
An outline of my forthcoming Agile 2010 workshop was given in the post “A Recipe for Handling Cultural Conflicts in Devops and Beyond” earlier this week. Here is the case study around which the workshop is structured:
NotHere, Inc. Case Study
NotHere, Inc. is a $500M company based in Jerusalem, Israel. The company developed an eCommerce platform for small to medium retailers. Through a combination of this platform and its hosting data center, NotHere provides online store fronts, shopping carts, order processing, inventory, billing and marketing services to tens of thousands of retailers in a broad spectrum of verticals. For these retailers, NotHere is a one-stop “shopping” for all their online needs. In particular, instead of partnering with multiple companies like Amazon, Ebay, PayPal and Shopzilla, a retailer merely needs to partner with NotHere (who partners with these four companies and many others).
The small to medium retailers that use the good services of NotHere are critically dependent on the availability of its data center. For all practical purposes retailers are (temporarily) dead when the NotHere data center is not available. In recognition of the criticality of this aspect of its IT operations, NotHere invested a lot of effort in maturing its ITIL[i] processes. Its IT department successfully implements the ITIL service support and service delivery functions depicted in the figure below. From an operational perspective, an overall availability level of four nines is consistently attained. The company advertises this availability level as a major market differentiator.
In response to the accelerating pace in its marketplace, NotHere has been quite aggressive and successful in transitioning to Agile in product management, dev and test. Code quality, productivity and time-to-producing-code have been much improved over the past couple of years. The company measures those three metrics (quality, productivity, time-to-producing-code) regularly. The metrics feed into whole-hearted continuous improvement programs in product management, dev and test. They also serve as major components in evaluating the performance of the CTO and of the EVP of marketing.
NotHere has recently been struggling to reconcile velocity in development with availability in IT operations. Numerous attempts to turn speedy code development into fast service delivery have not been successful on two accounts:
- Technical: Early attempts to turn Continuous Integration into Continuous Deployment created numerous “hiccups” in both availability and audit.
- Cultural: Dev is a competence culture; ops is a control culture.
A lot of tension has arisen between dev and ops as a result of the cultural differences compounding the technical differences. The situation deteriorated big time when the “lagging behind” picture below leaked from dev circles to ops.
The CEO of the company is of the opinion NotHere must reach the stage of Delivery over Development. She is not too interested in departmental metrics like the time it takes to develop code or the time it takes to deploy it. From her perspective, overall time-to-delivery (of service to the retailers) is the only meaningful business metric.
To accomplish Delivery over Development, the CEO launched a “Making Cats Work with Dogs[ii]” project. She gave the picture above to the CTO and CIO, making it crystal clear that the picture represents the end-point with respect to the relationship she expects the two of them and their departments to reach. Specifically, the CEO asked the CTO and the CIO to convene their staffs so that each department will:
- Document its Outmodel (in the sense explored in the “How We Do Things Around Here In Order to Succeed” workshop) of the other department.
- Compile a list of requirements it would like to put on the other group “to get its act together.”
The CEO also indicated she will convene and chair a meeting between the two departments. In this meeting she would like each department to present its two deliverables (world view of the other department & and the requirements to be put on it) and listen carefully to reflections and reactions from the other department. She expects the meeting will be the first step toward a mutual agreement between the two departments how to speed up overall service delivery.
[i] “Information Technology Infrastructure library – a set of concepts and practices for Information Technology Services Management (ITSM), Information Technology (IT) development and IT operations” [Wikipedia].[ii] I am indebted to Patrick DeBois for suggesting this title.
© Copyright 2010 Israel Gat
Technical Debt Meets Continuous Deployment
As you would expect in a conference entitled velocity, and in a follow-on devops day, speeding up things was an overarching theme. In the context of devops, the theme primarily manifested itself in lively discussions about the number of deploys per day. Comments such as the following reply to my post Ops Driven Dev were typical:
Conceptually, I move the whole business application configuration into the source code…
The theme that was missing for me in many of the presentations and discussions on the subject was the striking of a balance between velocity and quality. The classical trade-off in process control is between production rate and product quality (and safety, but that aspect [safety] is beyond the scope of this post). IMHO this trade-off applies to software just as it applies to mechanical or chemical processes.
The heart of the “deploy early and often” strategy hailed by advocates of continuous deployment is known deployment state to known deployment state. You don’t let the deployment evolve from one state to another before it has stabilized to a robust state. The power of this incremental deployment is in dealing with single-piece (or as small number of pieces as possible) flow rather than dealing with the effects of multiple-piece flow. When the deployment increments are small enough, rollback, root cause analysis and recovery are relatively straightforward if a deployment turns sour. It is a similar concept to Agile development, extending continuous integration to continuous deployment.
While I am wholeheartedly behind this devops strategy, I believe it needs to be reinforced through rigorous quality criteria the code must satisfy prior to deployment. The most straightforward way for so doing is through embedding technical debt criteria in the release/deploy process. For example:
- The code will not be deployed unless the overall technical debt per line of code is lower than $2.
- To qualify for deployment, code duplication levels must be kept under 8%.
- Code whose Cyclomatic complexity per Java class is higher than 15 will not be accepted for deployment.
- 50% unit test coverage is the minimal level required for deployment.
- Many others…
I have no doubt whatsoever that code which does not satisfy these criteria might be successfully deployed in a short-term manner. The problem, however, is the accumulative effect over the long haul of successive deployments of code increments of inadequate quality. As Figure 1 demonstrates, a Java file with Cyclomatic complexity of 38 has a probability of 50% to be error-prone. If you do not stop it prior to deployment through technical debt criteria, it is likely to affect your customers and play havoc with your deployment quite a few times in the future. The fact that it did not do so during the first hour of deployment does not guarantee that such a file will be “well-behaved” in the future.
Figure 1: Error-proneness as a Function of Cyclomatic Complexity (Source: http://www.enerjy.com/blog/?p=198)
To attain satisfactory long-term quality and stability, you need both the right process and the right code. Continuous deployment is the “right process” if you have developed the deployment infrastructure to support it. The “right code” in this context is code whose technical debt levels are quantified and governed prior to deployment.
Devops: It is Not About ITIL, It is About Proficiency
As you would expect, the Information Technology Infrastructure Library (ITIL) topic was brought up in the devops day held last Friday in a LinkedIn facility in Mountain View, CA. We, of course, had the expected spectrum of opinions about ITIL in the context of devops – from “ITIL will never work for a true continuous development shop” to “well, you can make ITIL work under such circumstances.” Needless to say, a noticeable level of passion accompanied these two statements…
IMHO the heart of the issue is not ITIL per se but system management proficiency. If your system management proficiency is high, you are likely to be able to effectively respond to 10, 20 or 50 deploys per day. Conversely, if your system management proficiency is low, ops is not likely to be able to cope with high velocity in dev. The critical piece is alignment of velocities between dev and ops, not the method used to manage IT systems and services. Whether you use ITIL, COBIT or your own home-grown set of best practices is irrelevant. Achieving alignment of velocities between dev and ops is a matter of proficiency in system management.
Every Nine Minutes
The post Every Thirty Minutes highlighted Flickr’s deployment frequency – literally every thirty minutes.
IMVU is reported to do so every nine minutes. The report has created a fair amount of controversy. Whether you are or are not in favor of such continuous deployment, one point is worthy of mention. IMVU is the very same company featured in The Lean Startup presentation. The way they do customer development has been discussed in our recent post Why Agile Matters.
Between what they do in iterative Customer Development and their work on Continuous Deployment IMVU is well worth paying attention to.
The Agile Executive 