Posts Tagged ‘Technical Debt’
A New Arithmetic for the Backlog
The heart of the matter in this engagement was ensuring that technical debt stories would not become ‘second citizens.’ We proposed treating technical debt as a strategic investment theme. To our way of thinking, technical debt is no different from customary budget allocations to growing market segments, tactical sales opportunities, cost reduction and the like.
Click here for details in the Cutter blog including guidance how to work through the Data Structure of the Enterprise figure below.
Allocation Flows in the Data Structure of the Enterprise
Beyond Devops
Based on feedback from participants in my Agile 2010 workshop “How We Do Things Around Here In Order To Succeed,” I am planning to offer the workshop as a one-day seminar. A tentative agenda for the seminar is as follows:
- Introduction to Cultural Framework
- Exercise #1: Determine Your Culture
- Exercise #2: Strengths and Weaknesses of Your Culture
- Change Behavior, Not Culture
- When Cultures Meet
- Exercise #3: Conflicts in Devops
- The Agile Flywheel
- Exercise #4: Using Technical Debt as a Boundary Object
- Bringing Individuals and Organizations Together
- Exercise #5: It is About Sharing the Process, Not Just Sharing the Information
- Exercise #6: From success in devops to end-to-end success
Until I publish a full-fledged outline for the seminar, here is the central theme:
Beyond Devops
Inter-departmental flow in a corporate setting is often envisioned as the inner workings of a swiss watch. Wheels turn other wheels in a precise manner. Not only is effectiveness maintained, it is maintained in an efficient manner.
Problem is, many individuals and most departments hold distorted views of the departments they interact with. Reasonable distortions can be mitigated as long as the operational balance between departments is maintained. Once the operational balance is broken the “swiss watch” stops to function as the inter-departmental distortions block any attempt to restore the balance.
The most effective way to get dev and ops on a path of collaboration is for the two departments to jointly construct a boundary object. As dev and ops are joined in the hip through the code, and even more so through its quality, technical debt is well suited to serve as the core of a boundary object around which the two department share meaning while retaining operational autonomy.
Similar boundary objects can be constructed between dev and other departments – customer support, professional services, marketing, sales and finance. When conceived and implemented in a manner that links numerous boundary objects together, Agile success in dev can be extended to both upstream and downstream functions.
A Recipe for Handling Cultural Conflicts in Devops and Beyond
My Agile 2010 workshop “How We Do Things Around Here In Order To Succeed” will weave together four trends that I am witnessing in my practice:
- The ascendance of Agile portfolio management in a world characterized by loosely coupled processes
- Devops dynamics are becoming more and more characteristic of end-to-end Agile/Kanban patterns
- Viral spread of technical debt metrics in software governance
- Increasing use of boundary objects in the enterprise context
The workshop is structured around three case studies/exercises that will take about two-thirds of the allotted time (the morning of August 9). The other third provides the theory and tools to be used in the three workshop exercises and (hopefully) in many future engagements participants in the workshop will carry out. Deep technical knowledge is not required – the workshop targets any Agile practitioner who has conceptual grasp of culture, software development, IT operations and portfolio management.
The #1 takeaway from the presentation is the details you need to know about creation and capture of lasting value through end-to-end Agile initiatives.
Here is the workshop agenda (still subject to some minor tweaking):
- Introduction to Cultural Framework
- Exercise #1: Strengths and Weaknesses of Your Culture
- Change Behavior, Not Culture
- When Organizations Clash
- Exercise #2: Conflicts in Devops
- The Agile Flywheel
- Exercise #3: Using Technical Debt as a Boundary Object in Devops
- Bringing Organizations Together Through Enlightened Governance Loops
I look forward to meeting you in the workshop and learning from your experiences and insights!
Israel
Technical Debt Meets Continuous Deployment
As you would expect in a conference entitled velocity, and in a follow-on devops day, speeding up things was an overarching theme. In the context of devops, the theme primarily manifested itself in lively discussions about the number of deploys per day. Comments such as the following reply to my post Ops Driven Dev were typical:
Conceptually, I move the whole business application configuration into the source code…
The theme that was missing for me in many of the presentations and discussions on the subject was the striking of a balance between velocity and quality. The classical trade-off in process control is between production rate and product quality (and safety, but that aspect [safety] is beyond the scope of this post). IMHO this trade-off applies to software just as it applies to mechanical or chemical processes.
The heart of the “deploy early and often” strategy hailed by advocates of continuous deployment is known deployment state to known deployment state. You don’t let the deployment evolve from one state to another before it has stabilized to a robust state. The power of this incremental deployment is in dealing with single-piece (or as small number of pieces as possible) flow rather than dealing with the effects of multiple-piece flow. When the deployment increments are small enough, rollback, root cause analysis and recovery are relatively straightforward if a deployment turns sour. It is a similar concept to Agile development, extending continuous integration to continuous deployment.
While I am wholeheartedly behind this devops strategy, I believe it needs to be reinforced through rigorous quality criteria the code must satisfy prior to deployment. The most straightforward way for so doing is through embedding technical debt criteria in the release/deploy process. For example:
- The code will not be deployed unless the overall technical debt per line of code is lower than $2.
- To qualify for deployment, code duplication levels must be kept under 8%.
- Code whose Cyclomatic complexity per Java class is higher than 15 will not be accepted for deployment.
- 50% unit test coverage is the minimal level required for deployment.
- Many others…
I have no doubt whatsoever that code which does not satisfy these criteria might be successfully deployed in a short-term manner. The problem, however, is the accumulative effect over the long haul of successive deployments of code increments of inadequate quality. As Figure 1 demonstrates, a Java file with Cyclomatic complexity of 38 has a probability of 50% to be error-prone. If you do not stop it prior to deployment through technical debt criteria, it is likely to affect your customers and play havoc with your deployment quite a few times in the future. The fact that it did not do so during the first hour of deployment does not guarantee that such a file will be “well-behaved” in the future.
Figure 1: Error-proneness as a Function of Cyclomatic Complexity (Source: http://www.enerjy.com/blog/?p=198)
To attain satisfactory long-term quality and stability, you need both the right process and the right code. Continuous deployment is the “right process” if you have developed the deployment infrastructure to support it. The “right code” in this context is code whose technical debt levels are quantified and governed prior to deployment.
Beautiful Quality
Figure 1: Agile Assessment – Quality (Source: QSMA)
Colleague and friend Michael Mah has kindly shared with me the figure above – quality assessment for a sample of Agile projects in the QSMA metrics database of more than 8000 software projects. The two red squares in this figure represent the recent results Mah measured on two projects carried out by Quick Solutions (QSI) – a Westerville, OH company offering a broad range IT services.
One to one-and-a-half standard deviation better than the mean might not seem like much to six sigma black belts. However, in the context of typical results we see in the software industry the QSI results are outstanding. I have not done the exact math whether those results are superior to 95%, 97% or 98% of software projects in the QSMA database as the very exact figure almost does not matter when you achieve this level of excellence.
I asked Bart Murphy – QSI’s Vice President of Delivery and Operations – for the ‘secret sauce.’ Here is the QSI recipe:
For the projects referenced in Michael’s evaluation, our primary focus was quality… Our team was tasked with not only a significant Innovation effort, but we also managed all aspects of supporting and stabilizing the application (production support, triage, infrastructure, database support, etc.). Our ‘secret sauce’ was assembling a world class team and executing our Agile methodology. We organized the team efforts to focus on the three major initiatives; Innovation, Stabilization (Technical Debt), and Production Support. We developed a release plan and coordinated efforts to deploy releases that resolved a significant number of defects, introduced market differentiating features, and addressed massive amount of technical debt. The team was able to accomplish this without introducing additional defects into the production system. Our success can be attributed to the commitment from the business to understand our Agile methodology and being highly engaged throughout the project (co-located with the team). In addition, the focus of quality that is integrated into our process through the use of Test Driven Development, Continuous Integration, Test Automation, Quality Assurance, and Show & Tells. Lastly, this would not have been possible without the co-location of the entire team given the significant issues and time constraints for delivery.
Bart also provided me with a table ‘a la Capers Jones’ in which he elaborates on the factors that helped QSI achieve these results and those that stood in the way. I will publish and discuss Bart’s table in a forthcoming post. As part of this post I will also compare the factors identified by Bart with those reported by Capers Jones.
Technical Debt at Cutter
No, this post is not about technical debt we identified in the software systems used by the Cutter Consortium to drive numerous publications, events and engagements. Rather, it is about various activities carried out at Cutter to enhance the state of the art and make the know-how available to a broad spectrum of IT professionals who can use technical debt engagements to pursue technical and business opportunities.
The recently announced Cutter Technical Debt Assessment and Valuation service is quite unique IMHO:
- It is rooted in Agile principles and theory but applicable to any software method.
- It combines the passion, empowerment and collaboration of Agile with the rigor of quantified performance measures, process control techniques and strategic portfolio management.
- It is focused on enlightened governance through three simple metrics: net present value, cost and technical debt.
Here are some details on our current technical debt activities:
- John Heintz joined the Cutter Consortium and will be devoting a significant part of his time to technical debt work. I was privileged and honored to collaborate with colleagues Ken Collier, Jonathon Golden and Chris Sterling in various technical debt engagements. I can’t wait to work with them, John and other Cutter consultants on forthcoming engagements.
- John and I will be jointly presenting on the subject Toxic Code in the Agile Roots conference next week. In this presentation we will demonstrate how the hard lesson learned during the sub-prime loans crisis apply to software development. For example, we will be discussing development on margin…
- My Executive Report entitled Revolution in Software: Using Technical Debt Techniques to Govern the Software Development Process will be sent to Cutter clients in the late June/early July time-frame. I don’t think I had ever worked so hard on a paper. The best part is it was labor of love….
- The main exercise in my Agile 2010 workshop How We Do Things Around Here in Order to Succeed is about applying Agile governance through technical debt techniques across organizations and cultures. Expect a lot of fun in this exercise no matter what your corporate culture might be – Control, Competence, Cultivation or Collaboration.
- John and I will be doing a Cutter webinar on Reining in Technical Debt on Thursday, August 19 at 12 noon EDT. Click here for details.
- A Cutter IT Journal (CITJ) on the subject of technical debt will be published in the September-October time-frame. I am the guest editor for this issue of the CITJ. We have nine great contributors who will examine technical debt from just about every possible perspective. I doubt that we have the ‘real estate’ for additional contributions, but do drop me a note if you have intriguing ideas about technical debt. I will do my best to incorporate your thoughts with proper attribution in my editorial preamble for this issue of the CITJ.
- Jim Highsmith and I will jointly deliver a seminar entitled Technical Debt Assessment: The Science of Software Development Governance in the forthcoming Cutter Summit. This is really a wonderful ‘closing of the loop’ for me: my interest in technical debt was triggered by Jim’s presentation How to Be an Agile Leader in the Agile 2006 conference.
Standing back to reflect on where we are with respect to technical debt at Cutter, I see a lot of things coming nicely together: Agile, technical debt, governance, risk management, devops, etc. I am not certain where the confluence of all these threads, and possibly others, might lead us. However, I already enjoy the adrenaline rush this confluence evokes in me…
How Many Metrics do You Need to Effectively Govern the Software Process?
A Simple Metrics-Driven Software Governance Framework Based on Jim Highsmith’s Agile Triangle Framework
In my recent Cutter Blog post entitled Three Governance Metrics I recommended using just three metrics:
- Value
- Cost
- Technical debt
The heart pf this recommendation is that all three can be expressed in dollar terms as depicted in the figure above. An apples-to-apples comparison is made through the common denominator – $$. For example, something is likely to be either technically, methodically or governance-wise wrong if the technical debt figure exceeds the cost figure for a prolonged period of time. One can actually characterize such a situation as accruing debt faster than building equity.
I am often asked about adding metrics to this simple governance framework. For example, should not productivity be included in the framework?
‘Less is more’ is my usual response to such questions. IMHO value, cost and technical debt address the most important high level governance considerations:
- Value –> Why are we doing the project?
- Cost –> Can we afford the project?
- Technical debt –> Is the execution risk acceptable?
Please pay special attention to the unit of measure of any metric you might add to this simple governance framework. As long as the metric is a dollar-based metric, the cohesion of the governance framework can be maintained. However, metrics which are not expressed in dollars will probably superimpose other frameworks on top of the simple governance framework. For example, you introduce a programming framework if you add a productivity metric which is measured in function points per man month. Sponsors who govern using value, cost, technical debt and productivity will need to mentally alternate between the simple governance framework and the programming framework whenever they try to combine the productivity metric with any of the other three metrics.
Toward An Event Driven Scrum Process
Recent posts in this blog and elsewhere recommended using technical debt (TD) criteria as an integral part of the build process. One could fail the build under conditions such as:
- TD(current build) > TD(previous build)
- TD per line of code > $1
- TD as % of cost > 50%
- Many others…
If you follow this recommendation while integrating continuously, chances are some fixing might need to be done multiple times a day. This being the case, the relationship between the daily stand-up meeting and the build fail event needs to be examined. If the team anyway re-groups whenever the build fails, what is the value add of the daily stand-up meeting?
Figure 1 below, taken from Agile Software Development with Scrum, examines the feedback loop of Scrum from a process control perspective. It shows the daily stand-up meeting providing the process control function.
Figure 1: Process Control View of Scrum
Figure 2 below shows the feedback loop being driven by the build process instead of by the daily stand-up meeting. From a process control standpoint it is essentially identical to Figure 1. Control through the regularly scheduled daily stand-up meeting has simply been replaced by event-driven control. The team re-groups whenever the build fails.
Figure 2: Build Driven Process Control
Some groups, e.g. the Scrum/Kanban teams under Erik Huddleston and Stephen Chin at Inovis have been practicing Event Driven Scrum for some time now. A lot more data is needed in order to validate the premise that Event Driven Scrum is indeed an improvement over vanilla Scrum. Until the necessary data is collected and examined, two intuitive observations are worth mentioning:
- Event Driven Scrum preserves the nature of Scrum as an empirical process.
- A control unit that is triggered by events (build fail) is in the very spirit of adaptive Agile methods.
Toxic Code
“[The 100% loan-to-value subprime loan is] the most dangerous product in existence and there can be nothing more toxic…”
This quip by Angelo Mozilo, founder of Countrywide Financial, led me to coin the term “toxic code.” The code stops being an asset when it accumulates technical debt equal to the value it is expected to generate. As matter of fact, the code could become a liability when the cost to “pay back” the technical debt exceeds the revenues the code would generate. Such situations have been known to happen more often than we might realize. They occur, for example, when numerous customers develop elaborate business processes around poor quality enterprise software.
I will be exploring the topic in my May 27, 2010 presentation at The Path to Agility conference in Columbus, OH. Here is the abstract of my talk:
Technical debt had originally been conceived as an expediency measure – “a little debt speeds development so long as it is paid back promptly with a rewrite.” However, like financial debt, unrestrained borrowing can lead to a broad spectrum of difficulties, from collapsed roadmaps to inability to respond to customer problems in a timely manner, and anything in between.
Recent advances in source code analysis enable us to quantify technical debt and express it in $$ terms. By so doing, the software development process can be governed with unprecedented effectiveness. It is possible to constrain the “development on margin” mal-practice and avoid the toxic code phenomenon: technical-debt-to-value ratio of 100%. Moreover, even toxic code can ultimately be “marked-to-market” by reducing/eliminating technical debt.
The combination of Agile refactoring practices with technical debt analytics brings rigor to the software development process through:
- Providing quantifiable data about the overall state of the code and its value
- Highlighting error-prone modules in the code and offering guidance to fixing them in a biggest-bang for-the-buck manner
- Pinpointing technical issues all the way down to the individual line of code level
- Balancing the technical debt work stream vis-a-vis other work streams
In the course of managing software development in this manner, your team will improve its design, coding, testing and project management skills. Ultimately, these improvements are the best antidote against accrual of technical debt in the future.
The Agile Executive 
Cutter’s Technical Debt Assessment and Valuation Service
with 3 comments
The Cutter Consortium has announced the availability of the Technical Debt Assessment and Valuation Service. The service combines static code analytics with dynamic program analytics to give the client “x-rays” of the software being examined at any desired granularity – from the whole project portfolio to a single instruction. It breaks down technical debt into the areas of coverage, complexity, duplication, violations and comments. Clients get an aggregate dollar figure for “paying back” debt that they can then plug into their financial models to objectively analyze their critical software assets. Based on these metrics, they can make the best decisions about their ongoing strategy for the software development effort under scrutiny.
This new service is an important addition to the enlightened software governance framework that Jim Highsmith, Michael Mah and I have been thinking about and contributing to for sometime now (see Beyond Scope, Schedule and Cost: Measuring Agile Performance and Quantifying the Start Afresh Option). The heart of both the technical debt service and the enlightened governance framework is captured by the following words from the press release:
By boiling down technical debt to dollars and tying it to cost and value, the service enables a metrics-driven governance framework for the use of five major constituencies, as follows:
It should finally be pointed out that the technical debt assessment service and the governance framework it enables are applicable to any software method. They can be used to:
Forthcoming Cutter Executive Reports, Executive Updates and Email Advisors on the technical debt service are restricted to Cutter clients. As appropriate, I will publish the latest and greatest news on the subject in the Cutter Blog (which is an open forum I highly recommend).
Acknowledgements: I would like to wholeheartedly thank the following colleagues for inspiring, enlightening and supporting me during the preparation of the service:
Written by israelgat
May 5, 2010 at 4:40 am
Posted in Agile Performance Management, Companies, Events, Performance Measurement, Software Costs
Tagged with Anne Mullaney, CEO, Chris Sterling, Cindy Swain, CIO, Comments, Complexity, Coverage, CTO, Cutter Consortium, Duplication, Governance Framework, Industry Norms, Israel Gat, IT Operations, Jennifer Flaxman, Jim Highsmith, John Heintz, Jonathon Golden, Kara Letourneau, Karen Coburn, Ken Collier, Kim Leonard, M&A, Michael Mah, NPV, Paying Back, Software Method, Technical Debt, Valuation, Venture Capitalist, Violation