Posts Tagged ‘Velocity’
How to Break the Vicious Cycle of Technical Debt
The dire consequences of the pressure to quickly deliver more functions and features to the market have been described in detail in various posts in this blog (see, for example, Toxic Code). Relentless pressure forces the development team to take technical debt. The very same pressure stands in the way of paying back the debt in a timely manner. The accrued technical debt reduces the velocity of the development team. Reduced development velocity leads to increased pressure to deliver, which leads to taking additional technical debt, which… It is a vicious cycle that is extremely difficult to break.
Figure 1: The Vicious Cycle of Technical Debt
The post Using Credit Limits to Constrain “Development on Margin” proposed a way of coping with the vicious cycle of technical debt – placing a limit on the amount of technical debt a development team is allowed to accrue. Such a limit addresses the demand side of the software development process. Once a team reaches the pre-determined technical debt limit (such as $3 per line of code) it cannot continue piling on new functions and features. It must attend to reducing the technical debt.
A complementary measure can be applied to the supply side of the software development process. For example, one can dynamically augment the team by drawing upon on-demand testing. uTest‘s recent announcement about securing Series C financing explains the rationale for the on-demand paradigm:
“The whole ‘appification’ of software platforms, whether it’s for social platforms like Facebook or mobile platforms like the iPhone or Android or Palm, or even just Web apps, creates a dramatically more complex user-testing matrix for software publishers, which could mean media companies, retailers, enterprise software companies,” says Wienbar. “Anybody who has to interact with consumers needs a service to help with that testing. You can’t cover that whole matrix with your in-house test team.”
Likewise, on-demand development can augment the development team whenever the capacity of the in-house team is insufficient to satisfy demand. IMHO it is only a matter of little time till marketplaces for on-demand development will evolve. All the necessary ‘ingredients’ for so doing – Agile, Cloud, Mobile and Social – are readily available. It is merely a matter of putting them together to offer on-demand development as a commercial service.
Whether you do on-demand testing, on-demand development or both, you will soon be able to address the supply side of software development in a flexible and cost-effective manner. Between curtailing demand through technical debt limits and expanding supply through on-demand testing/development, you will be better able to cope with the relentless pressure to deliver more and quicker than the capacity of your team allows.
A Devops Case Study
An outline of my forthcoming Agile 2010 workshop was given in the post “A Recipe for Handling Cultural Conflicts in Devops and Beyond” earlier this week. Here is the case study around which the workshop is structured:
NotHere, Inc. Case Study
NotHere, Inc. is a $500M company based in Jerusalem, Israel. The company developed an eCommerce platform for small to medium retailers. Through a combination of this platform and its hosting data center, NotHere provides online store fronts, shopping carts, order processing, inventory, billing and marketing services to tens of thousands of retailers in a broad spectrum of verticals. For these retailers, NotHere is a one-stop “shopping” for all their online needs. In particular, instead of partnering with multiple companies like Amazon, Ebay, PayPal and Shopzilla, a retailer merely needs to partner with NotHere (who partners with these four companies and many others).
The small to medium retailers that use the good services of NotHere are critically dependent on the availability of its data center. For all practical purposes retailers are (temporarily) dead when the NotHere data center is not available. In recognition of the criticality of this aspect of its IT operations, NotHere invested a lot of effort in maturing its ITIL[i] processes. Its IT department successfully implements the ITIL service support and service delivery functions depicted in the figure below. From an operational perspective, an overall availability level of four nines is consistently attained. The company advertises this availability level as a major market differentiator.
In response to the accelerating pace in its marketplace, NotHere has been quite aggressive and successful in transitioning to Agile in product management, dev and test. Code quality, productivity and time-to-producing-code have been much improved over the past couple of years. The company measures those three metrics (quality, productivity, time-to-producing-code) regularly. The metrics feed into whole-hearted continuous improvement programs in product management, dev and test. They also serve as major components in evaluating the performance of the CTO and of the EVP of marketing.
NotHere has recently been struggling to reconcile velocity in development with availability in IT operations. Numerous attempts to turn speedy code development into fast service delivery have not been successful on two accounts:
- Technical: Early attempts to turn Continuous Integration into Continuous Deployment created numerous “hiccups” in both availability and audit.
- Cultural: Dev is a competence culture; ops is a control culture.
A lot of tension has arisen between dev and ops as a result of the cultural differences compounding the technical differences. The situation deteriorated big time when the “lagging behind” picture below leaked from dev circles to ops.
The CEO of the company is of the opinion NotHere must reach the stage of Delivery over Development. She is not too interested in departmental metrics like the time it takes to develop code or the time it takes to deploy it. From her perspective, overall time-to-delivery (of service to the retailers) is the only meaningful business metric.
To accomplish Delivery over Development, the CEO launched a “Making Cats Work with Dogs[ii]” project. She gave the picture above to the CTO and CIO, making it crystal clear that the picture represents the end-point with respect to the relationship she expects the two of them and their departments to reach. Specifically, the CEO asked the CTO and the CIO to convene their staffs so that each department will:
- Document its Outmodel (in the sense explored in the “How We Do Things Around Here In Order to Succeed” workshop) of the other department.
- Compile a list of requirements it would like to put on the other group “to get its act together.”
The CEO also indicated she will convene and chair a meeting between the two departments. In this meeting she would like each department to present its two deliverables (world view of the other department & and the requirements to be put on it) and listen carefully to reflections and reactions from the other department. She expects the meeting will be the first step toward a mutual agreement between the two departments how to speed up overall service delivery.
[i] “Information Technology Infrastructure library – a set of concepts and practices for Information Technology Services Management (ITSM), Information Technology (IT) development and IT operations” [Wikipedia].[ii] I am indebted to Patrick DeBois for suggesting this title.
© Copyright 2010 Israel Gat
Technical Debt Meets Continuous Deployment
As you would expect in a conference entitled velocity, and in a follow-on devops day, speeding up things was an overarching theme. In the context of devops, the theme primarily manifested itself in lively discussions about the number of deploys per day. Comments such as the following reply to my post Ops Driven Dev were typical:
Conceptually, I move the whole business application configuration into the source code…
The theme that was missing for me in many of the presentations and discussions on the subject was the striking of a balance between velocity and quality. The classical trade-off in process control is between production rate and product quality (and safety, but that aspect [safety] is beyond the scope of this post). IMHO this trade-off applies to software just as it applies to mechanical or chemical processes.
The heart of the “deploy early and often” strategy hailed by advocates of continuous deployment is known deployment state to known deployment state. You don’t let the deployment evolve from one state to another before it has stabilized to a robust state. The power of this incremental deployment is in dealing with single-piece (or as small number of pieces as possible) flow rather than dealing with the effects of multiple-piece flow. When the deployment increments are small enough, rollback, root cause analysis and recovery are relatively straightforward if a deployment turns sour. It is a similar concept to Agile development, extending continuous integration to continuous deployment.
While I am wholeheartedly behind this devops strategy, I believe it needs to be reinforced through rigorous quality criteria the code must satisfy prior to deployment. The most straightforward way for so doing is through embedding technical debt criteria in the release/deploy process. For example:
- The code will not be deployed unless the overall technical debt per line of code is lower than $2.
- To qualify for deployment, code duplication levels must be kept under 8%.
- Code whose Cyclomatic complexity per Java class is higher than 15 will not be accepted for deployment.
- 50% unit test coverage is the minimal level required for deployment.
- Many others…
I have no doubt whatsoever that code which does not satisfy these criteria might be successfully deployed in a short-term manner. The problem, however, is the accumulative effect over the long haul of successive deployments of code increments of inadequate quality. As Figure 1 demonstrates, a Java file with Cyclomatic complexity of 38 has a probability of 50% to be error-prone. If you do not stop it prior to deployment through technical debt criteria, it is likely to affect your customers and play havoc with your deployment quite a few times in the future. The fact that it did not do so during the first hour of deployment does not guarantee that such a file will be “well-behaved” in the future.
Figure 1: Error-proneness as a Function of Cyclomatic Complexity (Source: http://www.enerjy.com/blog/?p=198)
To attain satisfactory long-term quality and stability, you need both the right process and the right code. Continuous deployment is the “right process” if you have developed the deployment infrastructure to support it. The “right code” in this context is code whose technical debt levels are quantified and governed prior to deployment.
Ops Driven Dev
In The Agile Flywheel, colleague Ray Riescher describes how velocity in dev drove corresponding velocity in ops:
Scrum set the flywheel in motion and caused the rest of the IT process life cycle to respond. ITIL’s processes still form the solid core of service support and we’ve improved the processes’ capability to handle intense work velocity. The organization adapted by developing unprecedented speed in the ability to deliver production fixes and to solve root cause problems with agility.
From what I gleaned yesterday in the O’Reilly Velocity conference I believe the tables are turning. Ops, or at least web ops, will soon drive development.
The reason for my saying so is quite simple: the breadth and depth of forthcoming web analytics unveiled in the conference. This is not “just” about Google making website performance part of their ranking algorithm. Everything related to web performance will soon be analyzed mercilessly under the “make the web faster” mantra. Dev will need to respond to analytics from operations with an unprecedented speed. For most practical purposes analytics run in ops will dictate the speed for dev.
The phenomenon actually goes beyond performance aspects. To be able to implement changes quickly, dev will need to be very good in ensuring the quality of fast changes. While quality has many dimensions to it, the most applicable one is test coverage. There is no way to change the code quickly without a comprehensive automated test suite.
The first step toward dev meeting the required speed is described in the post How to Initiate a Devops Project:
For a devops project, start by establishing the technical debt of the software to be released to operations. By so doing you build the foundations for collaboration between development and operations through shared data. In the devops context, the technical debt data form the basis for the creation and grooming of a unified backlog which includes various user stories from operations.
I would actually go one step further and suggest including technical debt criteria in the release process. The code is not accepted unless the technical debt per line of code is below a certain pre-set level such as $2. The criteria, of course, can be refined to include specific criteria for the various components of technical debt such as coverage, complexity or duplication. For example, unit test coverage in excess of 70% could be established as a technical debt criterion.
Once such release criteria are established, the metaphorical flywheel starts turning in an opposite direction to that described in The Agile Flywheel. With technical debt criteria embedded in the release process, the most straightforward way for dev to meet these criteria is to use the very same criteria as integral part of the build process. The scheme for so doing in given in the following chart:
One last recommendation: don’t wait till Velocity 2011 to start on the path described above. Velocity 2010 already provides plenty of actionable insights to warrant starting now. Just take a look at the web site.
The Agile Executive 