Archive for July 2010
Schedule Constraints in the Devops Triangle
Last week’s post “The Devops Triangle” demonstrated the extension of Jim Highsmith‘s Agile Triangle to devops. The extension relied on adding compliance to the three traditional constraints of software development: scope, schedule, cost. A graphical representation of this extension is given in Figure 1.
Figure 1: Compliance as the Fourth Constraint in Devops Projects
This blog post examines how time/schedule should be governed in the devops context. It does so by building on the concluding observation in the previous post:
The Devops Triangle and the corresponding Tradeoff Matrix demonstrate how governance a la Agile can be extended to devops projects as far as compliance goes. The proposed governance framework however is incomplete in the following sense: schedule in devops projects can be a much more granular and stringent constraint than schedule in “dev only” projects.
For the schedule constraint in devops, I propose a schedule set. It consists of four components:
- Lead Time or Engineering Time
- Time to change
- Time to deploy
- Time to roll back
Lead Time/Engineering Time: These are customary metrics used in Kanban software development, as demonstrated in Figure 3.
Figure 3: The Engineering Time Metric Used by the BBC (David Joyce in the LSSC10 Conference)
Time to change: The amount of time it takes for the various stakeholders (e.g., dev, test, ops, customer support) to review the code to be deployed, approve its deployment and assign a time window for the deployment.
Time to deploy: The amount of time from (metaphorically speaking) pushing the Deploy “button” to completion of deployment.
Time to roll back: The amount of time to undo a deployment. (Rigorous that the engineering practices and IT processes might be, the time to roll back a deployment can’t be ignored – it is a critical risk parameter).
A graphical representation of these four schedule metrics together with the Devops Triangle is given in the figure below:
Figure 4: The Devops Triangle with a Schedule Set
Using hours as the common unit of measure, a typical schedule set could be {100, 48, 3, 2}. In this hypothetical example, it takes a little over 4 days to carry out the development of the code increment; 2 days to get approval for the change; 3 hours to deploy the code; and, 2 hours to roll back.
Whatever your specific schedule numbers might be, it is highly recommended you apply value stream mapping (see Figure 5 below) to your schedule set. Based on the findings of the value stream mapping, apply statistical process control methods like those illustrated in Figure 3 to continuously improving both the mean and the variances of the four schedule components.
Figure 5: An Example of Value Stream Mapping (Source: Wikipedia entry on the subject)
Extending the Scope of The Agile Executive
For the past 18 months Michael Cote and I focused The Agile Executive on software methods, processes and governance. Occasional posts on cloud computing and devops have been supplementary in nature. Structural changes in the industry have generally been left to be covered by other blogs (e.g. Cote’s Redmonk blog).
We have recently reached the conclusion that The Agile Executive needs to cover structural changes in order to give a forward-looking view to its readers. Two reasons drove us to this conclusion:
- The rise of software testing as a service. The importance of this trend was summarized in Israel’s recent Cutter blog post “Changing Playing Fields“:
Consider companies like BrowserMob (acquired earlier this month by NeuStar), Feedback Army, Mob4Hire, uTest (partnered with SOASTA a few months ago), XBOSoft and others. These companies combine web and cloud economics with the effectiveness and efficiency of crowdsourcing. By so doing, they change the playing fields of software delivery…
- The rise of devops. The line between dev and ops, or at least between dev and web ops, is becoming fuzzier and fuzzier.
As monolithic software development and delivery processes get deconstructed, the structural changes affect methods, processes and governance alike. Hence, discussion of Agile topics in this blog will not be complete without devoting a certain amount of “real estate” to these two changes (software testing as a service and devops) and others that are no doubt forthcoming. For example, it is a small step from testing as a service to development as a service in the true sense of the word – through crowdsourcing, not through outsourcing.
I asked a few friends to help me cover forthcoming structural changes that are relevant to Agile. Their thoughts will be captured through either guest posts or interviews. In these posts/interviews we will explore topics for their own sake. We will connect the dots back to Agile by referencing these posts/interviews in the various posts devoted to Agile. Needless to say, Agile posts will continue to constitute the vast majority of posts in this blog.
We will start the next week with a guest post by Peter McGarahan and an interview with Annie Shum. Stay tuned…
Round Two: Can Technical Debt Constitute a Breach of Implied Warranties?
In a previous post I discussed whether technical debt could under some conditions constitute a breach of implied warranties. Examining the subject with respect to intent, I made the following observation:
It is a little tricky (though not impossible – see Using Credit limits to Constrain Development on Margin) to define the precise point where technical debt becomes “unmanaged.” One needs to walk a fine line between technical/methodical incompetence and resource availability to determine technical fraud. For example, if your code has 35% coverage, is it or is not unmanaged? Does the answer to this question change if your cyclomatic complexity per class exceeds 30? I would think the courts might be divided for a very long time on the question when does hidden technical debt represent a fraudulent misrepresentation.
One component of technical debt deserves special attention in the context of this post. I am referring to the conscious decision not to do unit testing at all… Such a conscious decision IMHO indicates no intention to pay back this category of technical debt – unit test coverage. It is therefore quite incompatible with the nature of an implied warranty:
Responses to my post were mixed. Various readers who are much more knowledgeable in the law than I am pointed out various legal defenses a software vendor could use. Click here for a deeper understanding of these subtle legal points.
Imagine my delight reading yesterday’s uTest interview with Cem Kaner. Cem makes the following statement in his interview:
ALI [American Law Institute] … started writing the Principles of the Law of Software Contracts. One of its most important rules is one that I advocated: a seller of software who knows about a defect of the software but does not disclose the defect to the customer will be held liable for damages caused to that customer by that defect. Note that this does not apply to free software (not sold). And if the seller discloses the defect, it becomes part of the product’s specification (it’s a feature). And if the seller doesn’t know about the defect there is no liability (once customers tell you about a defect, you have knowledge, so you cannot avoid knowledge for long by not testing). ALI adopted it unanimously last year. This is not law, but until the legislatures pass statutes, the Principles will be an important guide for judges. Even though I am a minor contributor to this work, I think the defect-disclosure requirement might be my career’s most important contribution to software quality. [Highlights by IG].
While not (yet?) a law, the Principles could indeed lead us where IMHO we as an industry should be. Technical debt manifests itself as bugs. By making the specific bugs part of the spec the software evolves from a quality standpoint. Moreover, the defect-disclosure requirement practically force software vendors to address their bugs within a “reasonable” amount of time. Customer bugs become an antidote to the relentless pressure to add functions and features without paying due attention to software quality.
I guess I missed the opportunity to include technical debt in the 2010 revision of the Principles. As we always do in software, I will target the next rev…
The Devops Triangle
The Agile Triangles was introduced by Jim Highsmith as an antidote to the Iron Triangle. Instead of balancing development between cost, schedule and scope, the Agile Triangle strives to strike a balance between value, quality and constraints:
Figure 1 – The Agile Triangle (based on Figure 1-3 in Agile Project Management: Creating Innovative Products.)
Consider the Iron Triangle in the context of devops. Value, quality and constraints apply to IT operations as meaningfully as they apply to software development. IT can go beyond cost, schedule and scope to focus on value and quality just as the Agile software development team does. Between development and operations the specific tasks to be carried out change, but the principles embodies in the triangle remain invariant.
In addition to cost, schedule and scope, devops projects must cope with another constraint: compliance. For example, a bank that implements a ‘follow the sun’ strategy with respect to trading must finish reconciling transaction that took place in London before the start of trade in Wall Street. From the bank’s point of view, its IT department needs to be mindful of four constraints: compliance, cost, schedule and scope. This view is represented in Figure 2 below.
Figure 2 – The Devops Triangle
Balancing the four constraints – compliance, cost, schedule, and scope – is not a trivial task. However, just like the Agile Triangle, the Tradeoff Matrix used in Agile software development applies to IT. In its software development variant, the Tradeoff matrix is an effective tool to decide between conflicting constraints, as follows:
Table 1 – Tradeoff Matrix (based on Table 6-1 in Agile Project Management: Creating Innovative Products.)
For devops, the matrix is extended to include a compliance row and a Reluctantly Accept column as follows:
Table 2 – Tradeoff Matrix for Devops
The Devops Triangle and the corresponding Tradeoff Matrix demonstrate how governance a la Agile can be extended to devops projects as far as compliance goes. The proposed governance framework however is incomplete in the following sense: schedule in devops projects can be a much more granular and stringent constraint than schedule in “dev only” projects. The subject of schedule constraints in devops projects will be addressed in a forthcoming post.
The Things They Say
Here is a compilation of summaries of Amazon reviews on the Kindle edition of The Concise Executive Guide to Agile:
If you are such a decision maker, and like most are hard-pressed for time, then you must read this book. Dr. Gat cuts right to the chase and gives pearls of wisdom – that come from having “done it himself” when it comes to the daunting task of TRANSFORMING an organization into successfully applying Agile methods. If you’re a team leader and you need executive support and buy-in, then buy this book for the key people whom you value the most. Give it to them as a gift. They will thank you for it!
“The Concise Executive Guide to Agile” by Israel Gat is a must-read for enthusiasts of agile methods at all levels of the organization. Israel has well over 30 years of industry experience, has his PhD in Computer Science, and has helped manage an agile rollout involving 1,000 software developers. While there are many technical and managerial treatises on agile methods emerging in increasing frequency all of the time, Israel’s new guide is meant to explain the essence of agile methods to R&D, Marketing, Sales, Program Management, Professional Services, Customer Support, Finance, or IT folks.
Dr. Gat goes beyond the traditional software development process literature and broadens the case for the Agile organization, with effectiveness, efficiency, and integrity – much like the Agile development process described by Israel.
A must read for any organization leader dependant on software whether it be for software for internal operations or software features built for customer sales. Israel’s breadth and depth of both knowledge and experience shine through as he provides practical advice to help you understand what Agile can and can not do for your organization.
For me, this book provided something unique, which is a look into the way high level executives think about Agile.
This book rapidly cuts through the layers of confusion surrounding Agile development and focuses on what you “NEED” to know. Dr. Gat’s vast experience shines through as he elegantly simplifies the process.
Israel’s experience and wisdom in transforming a software development culture in a very large multi-site virtual team environment is well captured… Israel has done a great job in transforming his experience into a guide. That is the beauty of the book.
This is the “must-read” book if you’re an executive charged with Agile transformation, product development, or business strategy. The insights from Gat cover all three. Most companies talk a good Agile game. If you want results, however, please pay heed to the advice in this book.
The book will help you to flexibly navigate the hurdles inherent in selling and succeeding with Agile in your business: i.e. existing methodologies, outsourcing, diverse geographies, resistance to change, and the rest. They are all explained here by someone who has apparently already successfully fought the battles. A great read!
He has been there, and this book shares the unique insights of a senior manager who guided a large organization through the change process. If you are in this seat in your organization then this book is for you.
Israel distills his broad experience and many years of wisdom rolling out Agile into a succinct overview that is deep in knowledge, but accessible.
I’ve been a fan of Israel’s “The Agile Executive” blog for some time and had high hopes for this book… and the book didn’t disappoint!
While I was looking for answers; what I found instead was much more interesting. Israel was able to collect all the wisdom of doing large scale agile rollouts into one document, highlighting the geographic features along the way. That is a lot harder to do than simply proclaiming one correct path, and, speaking from experience, more helpful to the reader than they know.
You can read the full reviews here.
First IEEE Computer Society Guidebook on Kindle
A Kindle edition of The Concise Executive Guide to Agile is available now through Amazon. It is the first IEEE Computer Society guidebook designed for and issued on an electronic reader. My sentiments about being the IEEE pilot project for Kindle are expressed in the following quote from yesterday’s press release:
“How appropriate it is that a book on Agile software methods was chosen as the pilot Kindle project by the prestigious IEEE Computer Society,” said Israel Gat. “The reach and richness of Kindle make it an ideal vehicle for effectively disseminating the Agile message to audiences that so far have not been touched by it.”
I am indebted to Kate Guillemette and Linda Shafer who made this IEEE pilot project happen.
Boundary Objects in DevOps
Source: Flickr; Chrice‘s Photostream
The following recommendation was given in the post How to Initiate a Devops Project:
For a DevOps project, start by establishing the technical debt of the software to be released to operations. By so doing you build the foundations for collaboration between development and operations through shared data. In the devops context, the technical debt data form the basis for the creation and grooming of a unified backlog which includes various user stories from operations.
I would like to augment this recommendation with a suggestion with respect to the mindset during the initiation phase. Chances are the IT folks feel outnumbered by the dev folks. It might or might not be a matter of optics, but recognizing and appreciating this mindset is will help a lot in getting the devops project on track.
Here is a simple example I heard from a participant in the June 25 devops day in Mountain View, CA. The participant with whom I talked is an IT ops person who tries to get ops aligned with fairly proficient Agile development teams. She is, however, constrained with respect to the IT ops resources available to her. She simply does not have the resources required to attend each and every Scrum meeting as 25 such meetings take place every day. She strongly feels “outnumbered.”
Various schemes could be devised to enable meaningful participation of ops in the Agile process. The more important thing though is to be fully sensitized to the “outnumbered” feeling. The extension of Agile principles to ops will not succeed at the face of such a feeling.
Discussing the subject with my friend Andrew Shafer, he mentioned the effectiveness of boundary objects in such cross-organizational situations:
Boundary objects are objects which are both plastic enough to adapt to local needs and constraints of the several parties employing them, yet robust enough to maintain a common identity across sites. They are weakly structured in common use, and become strongly structured in individual-site use. They may be abstract or concrete. They have different meanings in different social worlds but their structure is common enough to more than one world to make them recognizable means of translation. The creation and management of boundary objects is key in developing and maintaining coherence across intersecting social worlds. [Source: Wikipedia].
As an example, the boundary object for the situation described in this post could be a set of technical debt criteria that make the code eligible for deployment from a product life cycle perspective. By so doing, it shifts the dialog from the process to the outcome of the process. Instead of working on generating IT resources in an “outnumbered” mode, the energy shifts toward developing a working agreement on the intrinsic quality of the code to be deployed.
Some technical debt criteria that could form the core of a devops boundary object are mentioned in the post Technical Debt Meets Continuous Deployment. Corresponding criteria could be used in the boundary object to satisfy operational requirements which are critical to the proper functioning of the code. For example, a ceiling on configuration drift in IT could be established to ensure an adequate operating environment for the code. A boundary object that contains both technical debt criteria and configuration drift criteria satisfies different concerns – those of dev and those of ops – simultaneously.
Technical Debt Meets Continuous Deployment
As you would expect in a conference entitled velocity, and in a follow-on devops day, speeding up things was an overarching theme. In the context of devops, the theme primarily manifested itself in lively discussions about the number of deploys per day. Comments such as the following reply to my post Ops Driven Dev were typical:
Conceptually, I move the whole business application configuration into the source code…
The theme that was missing for me in many of the presentations and discussions on the subject was the striking of a balance between velocity and quality. The classical trade-off in process control is between production rate and product quality (and safety, but that aspect [safety] is beyond the scope of this post). IMHO this trade-off applies to software just as it applies to mechanical or chemical processes.
The heart of the “deploy early and often” strategy hailed by advocates of continuous deployment is known deployment state to known deployment state. You don’t let the deployment evolve from one state to another before it has stabilized to a robust state. The power of this incremental deployment is in dealing with single-piece (or as small number of pieces as possible) flow rather than dealing with the effects of multiple-piece flow. When the deployment increments are small enough, rollback, root cause analysis and recovery are relatively straightforward if a deployment turns sour. It is a similar concept to Agile development, extending continuous integration to continuous deployment.
While I am wholeheartedly behind this devops strategy, I believe it needs to be reinforced through rigorous quality criteria the code must satisfy prior to deployment. The most straightforward way for so doing is through embedding technical debt criteria in the release/deploy process. For example:
- The code will not be deployed unless the overall technical debt per line of code is lower than $2.
- To qualify for deployment, code duplication levels must be kept under 8%.
- Code whose Cyclomatic complexity per Java class is higher than 15 will not be accepted for deployment.
- 50% unit test coverage is the minimal level required for deployment.
- Many others…
I have no doubt whatsoever that code which does not satisfy these criteria might be successfully deployed in a short-term manner. The problem, however, is the accumulative effect over the long haul of successive deployments of code increments of inadequate quality. As Figure 1 demonstrates, a Java file with Cyclomatic complexity of 38 has a probability of 50% to be error-prone. If you do not stop it prior to deployment through technical debt criteria, it is likely to affect your customers and play havoc with your deployment quite a few times in the future. The fact that it did not do so during the first hour of deployment does not guarantee that such a file will be “well-behaved” in the future.
Figure 1: Error-proneness as a Function of Cyclomatic Complexity (Source: http://www.enerjy.com/blog/?p=198)
To attain satisfactory long-term quality and stability, you need both the right process and the right code. Continuous deployment is the “right process” if you have developed the deployment infrastructure to support it. The “right code” in this context is code whose technical debt levels are quantified and governed prior to deployment.
The Agile Executive 
