The Agile Executive

Making Agile Work

Posts Tagged ‘Productivity

Technical Debt: Assessment and Reduction

with one comment

Below is the detailed outline for my August 8, 1:30-5:00PM Technical Debt Workshop in Agile 2011. I look forward to meeting you and interacting with you in the conference before, during and after this workshop!

Best,

Israel

Technical Debt: Assessment and Reduction

Part I: Technical Debt in the Overall Context of the Software Process

  • A Holistic Model of the Software Process
  • Two Aspects of Output
  • Three Aspects of Technical Debt
  • Six Aspects of Software

Part II: What Really is Technical Debt?

  • What’s in a Metaphor?
  • Code Analysis
  • Time is Money
  • Monetizing Technical Debt
  • Typical Stakeholder Dialog Around Technical Debt
  • Analysis of the Cassandra Code
  • Project Dashboard

Part III : Case Study – NotMyCompany, Inc.

  • NotMyCompany Highlights
  • Modernizing Legacy Code
  • Error Proneness

Part IV: The Tricky Nature of Technical Debt

  • The Explicit Form of Technical Debt
  • The Implicit Form of Technical Debt
  • The Strategic Impact of Technical Debt
  • No Good Strategy Following Prolonged Neglect

Part V: Unified Governance

  • How We View Success
  • Three Core Metrics
  • Productivity, Affordability, Risk
  • What is the Real ROI?

Part VI: Process Control Models

  • A Typical Technical Debt Pattern
  • Process Control View of Scrum
  • Integration of Technical Debt in the Agile Process
  • Using Statistical Process Control Methods

Part VII: Reducing Technical Debt

  • A Framework for Thinking about and Acting on Technical Debt Issues
  • Portfolio Governance

Part VIII: Takeaways

  • Nine Simple Takeaway
  • Connecting the dots
Advertisements

The Real Cost of One Trillion Dollars in IT Debt: Part II – The Performance Paradox

with 7 comments

Some of the business ramifications of the $1 trillion in IT debt have been explored in the first post of this two-part analysis. This second post focuses on “an ounce of prevention is worth a pound of cure” aspects of IT debt. In particular, it proposes an explanation why prevention was often neglected in the US over the past decade and very possibly longer. This explanation is not meant to dwell on the past. Rather, it studies the patterns of the past in order to provide guidance for what you could do and should do in the future to rein in technical debt.

The prevention vis-a-vis cure trade-off  in software was illustrated by colleague and friend Jim Highsmith in the following figure:

Figure 1: The Technical Debt Curve

As Jim astutely points out, “once on far right of curve all choices are hard.” My experience as well as those of various Cutter colleagues have shown it is actually very hard. The reason is simple: on the far right the software controls you more than you control it. The manifestations of technical debt [1] in the form of pressing customer problems in the production environment force you into a largely reactive mode of operation. This reactive mode of operation is prone to a high error injection rate – you introduce new bugs while you fix old ones. Consequently, progress is agonizingly slow and painful. It is often characterized by “never-ending” testing periods.

In Measure and Manage Your IT Debt, Gartner’s Andrew Kyte put his finger on the mechanics that lead to the accumulation of technical debt – “when budget are tight, maintenance gets cut.” While I do not doubt Andrew’s observation, it does not answer a deeper question: why would maintenance get cut in the face of the consequences depicted in Figure 1? Most CFOs and CEOs I know would get quite alarmed by Figure 1. They do not need to be experts in object-oriented programming in order to take steps to mitigate the risks associated with slipping to the far right of the curve.

I believe the deeper answer to the question “why would maintenance get cut in the face of the consequences depicted in Figure 1?” was given by John Seely Brown in his 2009 presentation The Big Shift: The Mutual Decoupling of Two Sets of Disruptions – One in Business and One in IT. Brown points out five alarming facts in his presentation:

  1. The return on assets (ROA) for U.S. firms has steadily fallen to almost one-quarter of 1965 levels.
  2. Similarly, the ROA performance gap between corporate winners and losers has increased over time, with the “winners” barely maintaining previous performance levels while the losers experience rapid performance deterioration.
  3. U.S. competitive intensity has more than doubled during that same time [i.e. the US has become twice as competitive – IG].
  4. Average Lifetime of S&P 500 companies [declined steadily over this period].
  5. However, in those same 40 years, labor productivity has doubled – largely due to advances in technology and business innovation.

Discussion of the full-fledged analysis that Brown derives based on these five facts is beyond the scope of this blog post [2]. However, one of the phenomena he highlights –  “The performance paradox: ROA has dropped in the face of increasing labor productivity” – is IMHO at the roots of the staggering IT debt we are staring at.

Put yourself in the shoes of your CFO or your CEO, weighing the five facts highlighted by Brown in the context of Highsmith’s technical debt curve. Unless you are one of the precious few winner companies, the only viable financial strategy you can follow is a margin strategy. You are very competitive (#3 above). You have already ridden the productivity curve (#5 above). However, growth is not demonstrable or not economically feasible given the investment it takes (#1 & #2 above). Needless to say, just thinking about being dropped out of the S&P 500 index sends cold sweat down your spine. The only way left to you to satisfy the quarterly expectations of Wall Street is to cut, cut and cut again anything that does not immediately contribute to your cashflow. You cut on-going refactoring of code even if your CTO and CIO have explained the technical debt curve to you in no uncertain terms. You are not happy to do so but you are willing to pay the price down the road. You are basically following a “survive to fight another day” strategy.

If you accept this explanation for the level of debt we are staring at, the core issue with respect to IT debt at the individual company level [3] is how “patient” (or “impatient”) investment capital is. Studies by Carlota Perez seem to indicate we are entering a phase of the techno-economic cycle in which investment capital will shift from financial speculation toward (the more “patient”) production capital. While this shift is starting to happens, you have the opportunity to apply “an ounce of prevention is worth a pound of cure” strategy with respect to the new code you will be developing.

My recommendation would be to combine technical debt measurements with software process change. The ability to measure technical debt through code analysis is a necessary but not sufficient condition for changing deep-rooted patterns. Once you institute a process policy like “stop the line whenever the level of technical debt rose,” you combine the “necessary” with the “sufficient” by tying the measurement to human behavior. A possible way to do so through a modified Agile/Scrum process is illustrated in Figure 2:

Figure 2: Process Control Model for Controlling Technical Debt

As you can see in Figure 2, you stop the line and convene an event-driven Agile meeting whenever the technical debt of a certain build exceeds that of the previous build. If ‘stopping the line’ with every such build is “too much of a good thing” for your environment, you can adopt statistical process control methods to gauge when the line should be stopped. (See Using 3σ  Control Limits in Software Engineering for a discussion of the settings appropriate for your environment.)

An absolutely critical question this analysis does not cover is “But how do we pay back our $1 trillion debt?!I will address this most important question in a forthcoming post which draws upon the threads of this post plus those in the preceding Part I.

Footnotes:

[1] Kyte/Gartner define IT Debt as “the costs for bringing all the elements [i.e. business applications] in the [IT] portfolio up to a reasonable standard of engineering integrity, or replace them.” In essence, IT Debt differs from the definition of Technical Debt used in The Agile Executive in that it accounts for the possible costs associated with replacing an application. For example, the technical debt calculated through doing code analysis on a certain application might amount to $500K. In contrast, the cost of replacement might be $250K, $1M or some other figure that is not necessarily related to intrinsic quality defects in the current code base.

[2] See Hagel, Brown and Davison: The Power of Pull: How Small Moves, Smartly Made, Can Set Big Things in Motion.

[3] As distinct from the core issue at the national level.

A Devops Case Study

leave a comment »

An outline of my forthcoming Agile 2010 workshop was given in the post “A Recipe for Handling Cultural Conflicts in Devops and Beyond” earlier this week. Here is the case study around which the workshop is structured:

NotHere, Inc. Case Study

NotHere, Inc. is a $500M company based in Jerusalem, Israel. The company developed an eCommerce platform for small to medium retailers. Through a combination of this platform and its hosting data center, NotHere provides online store fronts, shopping carts, order processing, inventory, billing and marketing services to tens of thousands of retailers in a broad spectrum of verticals. For these retailers, NotHere is a one-stop “shopping” for all their online needs. In particular, instead of partnering with multiple companies like Amazon, Ebay, PayPal and Shopzilla, a retailer merely needs to partner with NotHere (who partners with these four companies and many others).

The small to medium retailers that use the good services of NotHere are critically dependent on the availability of its data center. For all practical purposes retailers are (temporarily) dead when the NotHere data center is not available. In recognition of the criticality of this aspect of its IT operations, NotHere invested a lot of effort in maturing its ITIL[i] processes. Its IT department successfully implements the ITIL service support and service delivery functions depicted in the figure below. From an operational perspective, an overall availability level of four nines is consistently attained. The company advertises this availability level as a major market differentiator.

In response to the accelerating pace in its marketplace, NotHere has been quite aggressive and successful in transitioning to Agile in product management, dev and test. Code quality, productivity and time-to-producing-code have been much improved over the past couple of years. The company measures those three metrics (quality, productivity, time-to-producing-code) regularly. The metrics feed into whole-hearted continuous improvement programs in product management, dev and test. They also serve as major components in evaluating the performance of the CTO and of the EVP of marketing.

NotHere has recently been struggling to reconcile velocity in development with availability in IT operations. Numerous attempts to turn speedy code development into fast service delivery have not been successful on two accounts:

  • Technical:  Early attempts to turn Continuous Integration into Continuous Deployment created numerous “hiccups” in both availability and audit.
  • Cultural: Dev is a competence culture; ops is a control culture.

A lot of tension has arisen between dev and ops as a result of the cultural differences compounding the technical differences. The situation deteriorated big time when the “lagging behind” picture below leaked from dev circles to ops.

The CEO of the company is of the opinion NotHere must reach the stage of Delivery over Development. She is not too interested in departmental metrics like the time it takes to develop code or the time it takes to deploy it. From her perspective, overall time-to-delivery (of service to the retailers) is the only meaningful business metric.

To accomplish Delivery over Development, the CEO launched a “Making Cats Work with Dogs[ii]” project. She gave the picture above to the CTO and CIO, making it crystal clear that the picture represents the end-point with respect to the relationship she expects the two of them and their departments to reach. Specifically, the CEO asked the CTO and the CIO to convene their staffs so that each department will:

  • Document its Outmodel (in the sense explored in the “How We Do Things Around Here In Order to Succeed” workshop) of the other department.
  • Compile a list of requirements it would like to put on the other group “to get its act together.”

The CEO also indicated she will convene and chair a meeting between the two departments. In this meeting she would like each department to present its two deliverables (world view of the other department & and the requirements to be put on it) and listen carefully to reflections and reactions from the other department. She expects the meeting will be the first step toward a mutual agreement between the two departments how to speed up overall service delivery.


[i] “Information Technology Infrastructure library – a set of concepts and practices for Information Technology Services Management (ITSM), Information Technology (IT) development and IT operations” [Wikipedia].

[ii] I am indebted to Patrick DeBois for suggesting this title.

© Copyright 2010 Israel Gat

Technical Debt at Cutter

leave a comment »

No, this post is not about technical debt we identified in the software systems used by the Cutter Consortium to drive numerous publications, events and engagements. Rather, it is about various activities carried out at Cutter to enhance the state of the art and make the know-how available to a broad spectrum of IT professionals who can use technical debt engagements to pursue technical and business opportunities.

The recently announced Cutter Technical Debt Assessment and Valuation service is quite unique IMHO:

  1. It is rooted in Agile principles and theory but applicable to any software method.
  2. It combines the passion, empowerment and collaboration of Agile with the rigor of quantified performance measures, process control techniques and strategic portfolio management.
  3. It is focused on enlightened governance through three simple metrics: net present value, cost and technical debt.

Here are some details on our current technical debt activities:

  1. John Heintz joined the Cutter Consortium and will be devoting a significant part of his time to technical debt work. I was privileged and honored to collaborate with colleagues Ken Collier, Jonathon Golden and Chris Sterling in various technical debt engagements. I can’t wait to work with them, John and other Cutter consultants on forthcoming engagements.
  2. John and I will be jointly presenting on the subject Toxic Code in the Agile Roots conference next week. In this presentation we will demonstrate how the hard lesson learned during the sub-prime loans crisis apply to software development. For example, we will be discussing development on margin…
  3. My Executive Report entitled Revolution in Software: Using Technical Debt Techniques to Govern the Software Development Process will be sent to Cutter clients in the late June/early July time-frame. I don’t think I had ever worked so hard on a paper. The best part is it was labor of love….
  4. The main exercise in my Agile 2010 workshop How We Do Things Around Here in Order to Succeed is about applying Agile governance through technical debt techniques across organizations and cultures. Expect a lot of fun in this exercise no matter what your corporate culture might be – Control, Competence, Cultivation or Collaboration.
  5. John and I will be doing a Cutter webinar on Reining in Technical Debt on Thursday, August 19 at 12 noon EDT. Click here for details.
  6. A Cutter IT Journal (CITJ) on the subject of technical debt will be published in the September-October time-frame. I am the guest editor for this issue of the CITJ. We have nine great contributors who will examine technical debt from just about every possible perspective. I doubt that we have the ‘real estate’ for additional contributions, but do drop me a note if you have intriguing ideas about technical debt. I will do my best to incorporate your thoughts with proper attribution in my editorial preamble for this issue of the CITJ.
  7. Jim Highsmith and I will jointly deliver a seminar entitled Technical Debt Assessment: The Science of Software Development Governance in the forthcoming Cutter Summit. This is really a wonderful ‘closing of the loop’ for me: my interest in technical debt was triggered by Jim’s presentation How to Be an Agile Leader in the Agile 2006 conference.

Standing back to reflect on where we are with respect to technical debt at Cutter, I see a lot of things coming nicely together: Agile, technical debt, governance, risk management, devops, etc. I am not certain where the confluence of all these threads, and possibly others, might lead us. However, I already enjoy the adrenaline rush this confluence evokes in me…

How Many Metrics do You Need to Effectively Govern the Software Process?

with 4 comments

A Simple Metrics-Driven Software Governance Framework Based on Jim Highsmith’s Agile Triangle Framework

In my recent Cutter Blog post entitled Three Governance Metrics I recommended using just three metrics:

  • Value
  • Cost
  • Technical debt

The heart pf this recommendation is that all three can be expressed in dollar terms as depicted in the figure above. An apples-to-apples comparison is made through the common denominator – $$. For example, something is likely to be either technically, methodically or governance-wise wrong if the technical debt figure exceeds the cost figure for a prolonged period of time. One can actually characterize such a situation as accruing debt faster than building equity.

I am often asked about adding metrics to this simple governance framework. For example, should not productivity be included in the framework?

‘Less is more’ is my usual response to such questions. IMHO value, cost and technical debt address the most important high level governance considerations:

  • Value –> Why are we doing the project?
  • Cost –> Can we afford the project?
  • Technical debt –> Is the execution risk acceptable?

Please pay special attention to the unit of measure of any metric you might add  to this simple governance framework. As long as the metric is a dollar-based metric, the cohesion of the governance framework can be maintained. However, metrics which are not expressed in dollars will probably superimpose other frameworks on top of the simple governance framework. For example, you introduce a programming framework if you add a productivity metric which is measured in function points per man month. Sponsors who govern using value, cost, technical debt and productivity will need to mentally alternate between the simple governance framework and the programming framework whenever they try to combine the productivity metric with any of the other three metrics.

How to Combine Development Productivity Data with Software Quality Metrics

with 2 comments

Consider the situation described in Should You Invest in This Software:

  • One of your portfolio companies expects to ship 500K lines of code in 6 months.
  • The company asks for additional $2M to complete development and bring the product to market.
  • Using technical debt quantification techniques you find the technical debt amounts to $1M.

You are not at all comfortable “paying back” the technical debt in addition to funding the requested $2M. You wonder whether you should start afresh instead of trying to complete and fix the code.



Photo credit: @muntz (Flickr)

A good starting point for assessing the fresh start option is Michael Mah‘s studies of software productivity. Based on the QSMA SLIM metrics database of more than 8,000 projects, Michael will probably bracket the productivity per person in a team consisting of product management, development and test at 10-15K lines of code per year. If you use the 15K lines of code per year figure for the purposes of the analysis, 500K lines of code could theoretically be delivered with an investment of about 33.3 (500/15) man years. Assuming average loaded cost of $99,000 per man-year,  the software represents a programming effort of $3.3M. Not much is left if you deduct $3M ($2M+1M) from $3.3M…

Five considerations are of paramount importance in evaluating the start afresh option:

  • The comparison above ($3.3M versus $3.0M) is timeless. It is a snapshot at a certain point in time which does not take into account the value of time. To factor in the time dimension, the analysis needs to get into value (as distinct from cost) considerations. See the note on Intrinsic Quality v. Extrinsic Quality at the bottom of this post.
  • Your “mileage” may vary. For example, best in class teams in large software projects have reported productivity of 20K lines of code per team member per year. As another example, productivity in business applications is very different from productivity in real-time software.
  • If you decide to start with a brand new team, remember Napoleon’s quip: “Soldiers have to eat soup together for a long time before they are ready to fight.”
  • If you decide to start afresh with the same team plus some enhancements to the headcount, be mindful of  ‘Mythical Man-Month‘ effects. Michael Mah’s studies of the BMC BPM projects indicate that such effects might not hold for proficient Agile teams. Hence, you might opt to go Agile if you plan to enhanced the team in an aggressive manner.
  • Starting afresh is not an antidote to accruing technical debt (yet again…) over time. But, it gives you the opportunity to aggressively curtail technical debt by applying the techniques described in Using Credit Limits to Constrain Development on Margin. For example, you might run source code analytics every two weeks and go over the results in the bi-weekly demo.

As long as you are mindful of these five aspects (timeless analysis, your mileage may vary, Napoleon’s quip, mythical man-month effects and credit limits on technical debt), combining technical debt figures with productivity data is an effective way to consider the pros and cons of “fix it” versus starting afresh. The combination of the two simplifies a complex  investment decision by reducing all considerations to a single common denominator – $$.

Note: This is not a discussion from a value perspective. The software, warts and everything, might (or might not)  be valuable to the target customers. The reader is referred to Jim Highsmith‘s analysis of Intrinsic quality versus Extrinsic Quality in Agile Project Management: Creating Innovative Products. See the Cutter Blog post entitled Beyond Scope, Schedule and Cost: Measuring Agile Performance for a short summary of the distinction between the two.

Turning the “Law of Software Physics” Upside Down

leave a comment »

Agile Event

Next week, Michael Mah will be presenting new quantitative data on software productivity, cost, time-to-market and quality. Here is an excerpt from the announcement of his talk:

Many companies are adopting Agile practices in an effort to increase project throughput, reduce cost, and improve quality. But are they working? Drawing from industry statistics, Michael answers vital questions about Agile’s effectiveness, which may be turning the “law of software physics” upside down. Until now, there have been predictable relationships among schedule, staffing, and quality; industry data indicates Agile may be changing all this. See productivity findings at 5 Agile companies, and the results for time-to-market, productivity, and quality. Learn the right practices for your environment, including characteristics of successful measurement. See how metrics reveal insights into Agile approaches that are becoming mainstream.

Knowing Michael (and, well, knowing a bit about his latest and greatest findings…) I have every reason to believe Michael will be breaking new grounds in his presentation. Click here and here for details about this exciting presentation.

Written by israelgat

March 5, 2010 at 12:34 pm