The Agile Executive

Making Agile Work

Archive for October 2010

The Nine Transformative Aspects of the Technical Debt Metric

with 3 comments

No, the technical debt metric will not improve your tennis game. However, using it could help you generate time for practicing the game due to its nine transformative aspects:
  1. The technical debt metric enables Continuous Inspection of the code through ultra-rapid feedback to the software process (see Figure 1 below).
  2. It shifts the emphasis in software development from proficiency in the software process to the output of the process.
  3. It changes the playing fields from qualitative assessment to quantitative measurement of the quality of the software.
  4. It is an effective antidote to the relentless function/feature pressure.
  5. It can be used with any software method, not “just” Agile.
  6. It is applicable to any amount of code.
  7. It can be applied at anypoint in time in the software life-cycle.
  8. These seven characteristics of the technical debt metric enable effective governance of the software process.
  9. The above  characteristics of the technical debt metric enable effective governance of the software product portfolio.

Figure 1: Continuous Inspection

Written by israelgat

October 28, 2010 at 8:40 am

Code2Cloud: Bigger than a Disruption in ALM

with 2 comments

 

Update, October 22, 2010: Watch this excellent demo of Code2Cloud!

 

 

http://en.wikipedia.org/wiki/File:ALM.svg

Figure 1: A Representation of the Application Lifecycle Management Concepts

VMware’s Code2Cloud announcement a couple of days ago is intriguing. According to this announcement, the whole development infrastructure is delivered as a service with no setup, no hardware or software to manage. The tedious and time consuming task of setting (and as appropriate modifying) the environment within which coding is carried out is done by Code2Cloud, not by the programming/testing team. As pointed out by colleague and friend Michael Cote, Code2Cloud might have the potential to be quite a disruption in Application Lifecycle Management (ALM):

“The software development tool chain has always been tedious to setup and integrate,” said Red Monk analyst Michael Cote. “While cloud-based development promises to make application delivery, deployment, and use easier, I haven’t seen excellent unified application management approaches that take full advantage of cloud. VMware’s SpringSource Code2Cloud is an ambitious step towards moving much of the development management stack into the cloud and hopefully vacuuming up those tedious application management tasks. It’ll be fun to watch this idea evolve as more and more people and applications start taking advantage of cloud computing.”

Important that such a disruption in the ALM space might be, I believe the main significance of the Code2Cloud announcement is in demonstrating so vividly how powerful the Everything as a Service (EaaS) paradigm could be and probably will be. IMHO Code2Cloud is another proof point to the power of the confluence of Agile, Cloud, Mobile and Social. It is a virtuous cycle of unprecedented impact – in technology delivery, in the structure of markets, in society and in the patterns of living we are accustomed to.

© Copyright 2010 Israel Gat

Figure 2: The Virtuous Cycle of Agile, Cloud, Mobile and Social

The Code2Cloud announcement is primarily about the {Agile –> Cloud} link in Figure 2. The {Cloud –> Mobile}, {Mobile –> Social} and {Social –> Agile} are just as powerful. For example, the {Social –> Agile} link, in conjunction with Cloud and Mobile, opens the door for highly efficient Testing as a Service.

Think of the Code2Cloud as a great example of Everything as a Service. Many other examples of such services are forthcoming. The common denominator of all these examples to come is their transformative power. Not in the tactical sense of “transformative”, but in the deep strategic meaning of the word.

Action item: Start a pilot to evaluate Code2Cloud. Expand rapidly if it meets your development needs. Tie at the earliest point in time to your plans for application delivery, deployment and use in the cloud.

____________________________________________________________________________________________________

Wrestling with governing your software from a lifcycle perspective? Let me know if you would like assistance in implementing a simple yet highly effective software governance framework that can be used by both technical and non-technical members of your staff. Click Services for details and contact information.

____________________________________________________________________________________________________

 

Written by israelgat

October 21, 2010 at 7:19 am

How to Use Technical Debt Data in the M&A Process

leave a comment »

http://www.flickr.com/photos/brajeshwar/266749872/

As a starting point, please read Implication of Technical Debt Uncertainty for Software Licensing Negotiations. Everything stated there holds for negotiating M&A deals. In particular:

  • You (as the buyer) should insist on conducting a Technical Debt Assessment as part of the due diligence process.
  • You should be able to deduct the monetized technical debt figure from the price of the acquisition.
  • You should be able to quantify the execution risk (as far as software quality is concerned).

An important corollary holds with respect to acquiring a company who is in the business of doing maintenance on an open source project, helping customers deploy it and training them in its use. You can totally eliminate uncertainty about the quality of the open source project without needing to negotiate permission to conduct technical debt assessment. Actually, you will be advised to conduct the assessment of the software prior to approaching the target company. By so doing, you start negotiations from a position of strength, quite possibly having at your disposal (technical debt) data that the company you consider acquiring does not possess.

Action item: Supplement the traditional due diligence process with a technical debt assessment. Use the monetized technical debt figure to assess execution risk and drive the acquisition price down.

http://www.flickr.com/photos/tantek/254940135/

____________________________________________________________________________________________________

Negotiating a major M&A deal? Let me know if you would like assistance in conducting a technical debt assessment and bringing up technical debt issues with the target company. I will help you with negotiating the acquisition price down. Click Services for details and contact information.

____________________________________________________________________________________________________

Written by israelgat

October 20, 2010 at 5:37 am

Implications of Technical Debt Uncertainty for Software Licensing Negotiations

with 2 comments

A few years ago, my friend Sebastian Hassinger characterized the state of affairs in enterprise software by the following chart a la Christensen:

The key point this charts gets across is that Open Source Software is becoming “good enough”. It has already met or will soon be meeting the minimum requirements of the enterprise customer. By so doing, open source software will steadily gain ground from traditional enterprise software vendors.

Consider this chart from a buyer’s perspective. Functionality (the vertical axis in the chart) can be thought of as value. Whatever the value might be, it is diminished by technical debt in the software as the debt manifests itself as application crashes, degradation of  performance and possible corruption of customer data. Everything else being equal, an application with lower technical debt per line of code is preferable to an application with a higher technical debt per line of code.

Traditional enterprise software vendors do not typically provide the technical debt data for the applications they sell/license. In contrast, a customer can carry out his/her assessment of technical debt straight off the open source code. For example, colleague and friend John Heintz carried out the following technical debt analysis on the Cassandra open source project:



As demonstrated in this chart, any customer can measure the level of technical debt in an open source software he/she considers. For better or worse, there is no uncertainty about the amount of technical debt the customer will need to live with in an open source software. In contrast, a customer will usually need to live with  uncertainty about the level of technical debt in proprietary software.

Uncertainty has economical consequences. For example, testing a product increases its value because it decreases operational uncertainty. The economical value of uncertainty about technical debt is conceptually depicted in the figure below in which value is adjusted in accord with the knowledge or lack thereof of the amount of technical debt. Please note that the following equation holds for the various intersection points on the Enterprise Customer Requirements line: {T3-T2} < {T1-T0}. What this equation means is that under conditions of uncertain technical debt open source software is becoming more attractive than proprietary software faster than it would without taking technical debt uncertainty into account.

Action Item: Before licensing an enterprise application or renewing an existing license, ask the vendor for technical debt data for the application and the plans to reduce the debt. If the vendor refuses to disclose this data or can’t generate it within a reasonable amount of time, ask for the number of open bugs against this application in the vendor’s bug data base. Use either kind of data to drive down the price. Consider  an open source solution (even if it provides less functionality than the proprietary software product) if the vendor you are dealing with refuses to disclose either the technical debt data or the number of open bugs in the enterprise application.

____________________________________________________________________________________________________

Negotiating a major enterprise software deal? Let me know if you would like assistance in bringing up technical debt issues with the vendor to help with negotiating the price down. Click Services for details and contact information.

____________________________________________________________________________________________________

Y2K vis-a-vis IT Debt

leave a comment »

http://www.flickr.com/photos/plural/4279707276/

Andrew Dailey of MGI Research and Andy Kyte of Gartner Group kindly did some digging for me on the total amount of money that was spent on Y2K. Here is the bottom line from Andy concluding our email thread on the subject of Y2K expenditures:

I have remained comfortable with our estimate of $300B to $600B.

In other words, it will take an effort comparable to the Y2K effort at the turn of the century to ‘pay back’ the current IT Debt.

____________________________________________________________________________________________________

Considering modernization of your legacy code? Let me know if you would like assistance in monetizing your technical debt, devising plans to reduce it and governing the debt reduction process. Click Services for details.

____________________________________________________________________________________________________

Written by israelgat

October 18, 2010 at 6:13 am

Fresh Perspectives on Technical Debt

leave a comment »

Update, October 15: The issue has been posted on the Cutter website (Cutter IT Journal subscription privileges required).

Cutter is just about ready to post the October issue of the IT Journal for which I am the guest editor. Print subscribers should receive it by the last week of the month. Jim Highsmith and I will be reflecting on it in our forthcoming seminar on technical debt in the Cutter Summit.

This issue sheds light on three noteworthy aspects of technical debt techniques:

  1. Their pragmatic use as an integral part of Governance, Risk and Compliance (GRC).
  2. Extending the techniques to shed light on various nuances of technical debt that have alluded us so far.
  3. Applying the techniques in new domains such as devops.

Here is the Table of Contents for this exciting issue:

Opening Statement

by Israel Gat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  3

Modernizing the DeLorean System: Comparing Actual and Predicted Results of a Technical Debt Reduction Project

by John Heintz . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

The Economics of Technical Debt

by Stephen Chin, Erik Huddleston, Walter Bodwell, and Israel Gat . . . . . . . . . . . . . . . . . 11

Technical Debt: Challenging the Metaphor

by David Rooney . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Manage Project Portfolios More Effectively by Including Software Debt in the Decision Process

by Brent Barton and Chris Sterling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

The Risks of Acceptance Test Debt

by Ken Pugh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Transformation Patterns for Curing the Human Causes of Technical Debt

by Jonathon Michael Golden . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  30

Infrastructure Debt: Revisiting the Foundation

by Andrew Shafer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

 

Action Item: Apply the techniques recommended in this issue to govern your software assets in an effective manner.

____________________________________________________________________________________________________

Overwhelmed by a “mountain” of technical debt? Let me know if you would like assistance in devising and carrying out plans to reduce the debt in a biggest-bang-for-the-buck manner. Click Services for details.

____________________________________________________________________________________________________

A New Chasm is Forming

leave a comment »

http://www.flickr.com/photos/96483949@N00/192144459/

Recently I had frustrating experiences with the bureaucracies of two companies: a company who is bringing me in to consult on continuous value delivery and another company who is interested in a seminar on the consumerization of enterprise software. While the two companies could not be more different, good-hearted engagement managers in both companies cautioned me in advance that their bureaucracies would drive me nuts. Based on my experience to date they were not kidding…

It is a striking contrast between the ease with which I can get precious data from my social network anytime I need it versus the unbelievable waste of time answering the very same question in two or three redundant forms used to screen me as a supplier. This contrast leads me to conclude we are witnessing the formation of a new chasm. It is between companies who stick to their old bureaucratic patterns with respect to suppliers versus those that realize that a supplier these days is a “prosumer.” He/she might provide services one day, consume (other) services the other day.

The business opportunity this chasm presents is providing efficient marketplace infrastructures. Anyone who can collect my data once and provide it as needed to multiple companies I interact with as a supplier will be doing me, and countless number of social networking aficionado, a huge service. Time is simply too precious to be wasted typing in the maiden name of my mother multiple times.

The distinguished economist Ronald Coase perceived reduction of transaction costs as the essence of the firm. His thoughts of more than 70 years ago are right on these days. The bar for transaction costs is “fill in the details only once”. Once in this context means “once in your lifetime.”

Recommendation: Examine the  way your company acquires new customers versus the way it brings aboard suppliers. Something is wrong if your company’s procurement folks routinely tell suppliers “we know you have already given this information, but ‘they’ would not accept it from ‘us’ if you don’t fill this extra form as well.” This being the case, you need to rethink your approach to composite value chains.

Written by israelgat

October 13, 2010 at 7:40 am

Apropos is Going Places

with one comment

Pictured above is a screen shot from the forthcoming Rally implementation of Apropos – the end-to-end Kanban system unveiled by Erik Huddleston, Stephen Chin, Walter Bodwell and me in the Lean Software and Systems conference last April.

Pictured below is Stephen Chin presenting the forthcoming product in the recent JavaOne conference:

The commercial version by Rally builds on the four pillars of the original implementation of Apropos at Inovis and the subsequent open source version:

  • Stakeholder Based Investment Themes
  • Business Case Management
  • Upstream and Downstream WIP Limits
  • Dynamic Allocations

These four pillars enable Apropos users to dynamically adjust their plans as needed in accord with the realities of end-to-end execution. Agile portfolio planning and actual execution truly run alongside each other as depicted in the following figure:

Adjustments to allocations can take place in either in the plan or in execution. Here are two typical examples of stakeholders’ dialogs:

  • In planning: “In response to the quick growth of the sales funnel, we decide to increase the % of time allotted to tactical sales opportunities from 35% of the total R&D budget to 40%.”
  • In execution: “The introduction of product Pj will be delayed by three months due to lack of qualified professional services resources. During this period, the affected R&D resources will be reassigned to help with multi-tenant aspects of a SaaS version of product Pk.”

Recommendations: Consider using the open source version of Apropos for a small-scale pilot as part of your 2011 planning/budget cycle. If the pilot proves a good fit with your needs,  switch over to the commercial version in the 2012 planning/budget cycle.

____________________________________________________________________________________________________

Considering end-to-end Agile/Kanban roll-out? Let me know if you would like assistance in planning and implementing a roll-out which focuses on continuous value delivery. Click Services for details.

____________________________________________________________________________________________________

How Technical Debt Ties to Cloud, Mobile and Social

leave a comment »

http://en.wikipedia.org/wiki/File:West_Side_Highway_collapsed_at_14th.jpg

Many years ago, when I came to the US, I was shocked to the core seeing the collapsed West Side Highway in New York City. I simply could not believe that a highway would be neglected to that extent amidst all the affluence of the city. The contrast was too much for me.

Nowadays I often have a deja vu sensation in various technical debt engagements in which I find the code crumbling. This sensation is not so much about what happened (see The Real Cost of a One Trillion Dollars in IT Debt: Part II – The Performance Paradox for an explanation of the economics of the neglect of software maintenance during the past decade), but about the company for which I do the assessment giving up on immense forthcoming opportunities.

Whether you do or do not fully subscribe to the vision of the Internet-of-Things depicted in the figure below, it is fairly safe to assume that your business in the years to come will be much more connected to the outside world than it is now. The enhanced connectivity might come through mobile applications, through social networks or through the cloud. As a matter of fact, it is quite likely to come through a confluence of the three: Cloud, Mobile and Social.

http://en.wikipedia.org/wiki/File:Internet_of_Things.png

In the context of current trends in cloud, mobile and social, your legacy software is like the West Side Highway in New York City. If you maintain it to an acceptable level, it can become the core of two major benefits of much higher connectivity and connectedness in the not-too-far future:

  • Through mobile and social your legacy software will enable you to flexibly produce, market and distribute small quantities of whatever your products might need to be in niche markets.
  • Through cloud it will enable you to offer these very same products and many others as services.

Conversely, if you consistently neglect to pay back your technical debt, your legacy code is likely to collapse due to the effects of software decay. You certainly will not be able to get it to interoperate with mobile and social networking applications, let alone offer it in the form of cloud services. Nor would you be able to wrap additional services around decaying legacy code. Take a look at the warehousing and distribution services offered by Amazon to get a sense of what this kind of additional services could do for your core business: they will enable you to transform your current business design by adding an Online-to-Offline (O2O) component to it.

What is the fine line differentiating “acceptably maintained” code from toxic code? I don’t think I have conducted a large enough sample of technical debt assessments to provide a statistically significant answer. My hunch is that the magic ceiling for software development in the US is somewhere around $10 per line of code in technical debt. As long as you are under this ceiling you could still pay back your technical debt (or a significant portion of it) in an economically viable manner. Beyond $10 per line of code the decay might prove too high to fix.

Why $10 and not $1 or $100 per line of code? It is a matter of balancing investment versus debt. An average programmer (in the US) with a $100,000 salary would probably be able to produce about 10K lines of Java code per year. The cost of a line of code under these simplistic assumptions is $10. Something is terribly wrong if the technical debt exceeds the cost per line. They call it living on margin.

Action item: CIOs should conduct a technical debt assessment on a representative sample of their legacy code. A board level discussion on the strategic implications for the company is called for if technical debt per line of code exceeds $10. The board discussion should focus on the ability of the company (or lack thereof) to participate in the business tsunami that cloud, mobile and social are likely to unleash.

____________________________________________________________________________________________________

Considering modernization of your legacy code? Let me know if you would like assistance in monetizing your technical debt, devising plans to reduce it and governing the debt reduction process. Click Services for details.

____________________________________________________________________________________________________

The Real Cost of One Trillion Dollars in IT Debt: Part II – The Performance Paradox

with 7 comments

Some of the business ramifications of the $1 trillion in IT debt have been explored in the first post of this two-part analysis. This second post focuses on “an ounce of prevention is worth a pound of cure” aspects of IT debt. In particular, it proposes an explanation why prevention was often neglected in the US over the past decade and very possibly longer. This explanation is not meant to dwell on the past. Rather, it studies the patterns of the past in order to provide guidance for what you could do and should do in the future to rein in technical debt.

The prevention vis-a-vis cure trade-off  in software was illustrated by colleague and friend Jim Highsmith in the following figure:

Figure 1: The Technical Debt Curve

As Jim astutely points out, “once on far right of curve all choices are hard.” My experience as well as those of various Cutter colleagues have shown it is actually very hard. The reason is simple: on the far right the software controls you more than you control it. The manifestations of technical debt [1] in the form of pressing customer problems in the production environment force you into a largely reactive mode of operation. This reactive mode of operation is prone to a high error injection rate – you introduce new bugs while you fix old ones. Consequently, progress is agonizingly slow and painful. It is often characterized by “never-ending” testing periods.

In Measure and Manage Your IT Debt, Gartner’s Andrew Kyte put his finger on the mechanics that lead to the accumulation of technical debt – “when budget are tight, maintenance gets cut.” While I do not doubt Andrew’s observation, it does not answer a deeper question: why would maintenance get cut in the face of the consequences depicted in Figure 1? Most CFOs and CEOs I know would get quite alarmed by Figure 1. They do not need to be experts in object-oriented programming in order to take steps to mitigate the risks associated with slipping to the far right of the curve.

I believe the deeper answer to the question “why would maintenance get cut in the face of the consequences depicted in Figure 1?” was given by John Seely Brown in his 2009 presentation The Big Shift: The Mutual Decoupling of Two Sets of Disruptions – One in Business and One in IT. Brown points out five alarming facts in his presentation:

  1. The return on assets (ROA) for U.S. firms has steadily fallen to almost one-quarter of 1965 levels.
  2. Similarly, the ROA performance gap between corporate winners and losers has increased over time, with the “winners” barely maintaining previous performance levels while the losers experience rapid performance deterioration.
  3. U.S. competitive intensity has more than doubled during that same time [i.e. the US has become twice as competitive – IG].
  4. Average Lifetime of S&P 500 companies [declined steadily over this period].
  5. However, in those same 40 years, labor productivity has doubled – largely due to advances in technology and business innovation.

Discussion of the full-fledged analysis that Brown derives based on these five facts is beyond the scope of this blog post [2]. However, one of the phenomena he highlights –  “The performance paradox: ROA has dropped in the face of increasing labor productivity” – is IMHO at the roots of the staggering IT debt we are staring at.

Put yourself in the shoes of your CFO or your CEO, weighing the five facts highlighted by Brown in the context of Highsmith’s technical debt curve. Unless you are one of the precious few winner companies, the only viable financial strategy you can follow is a margin strategy. You are very competitive (#3 above). You have already ridden the productivity curve (#5 above). However, growth is not demonstrable or not economically feasible given the investment it takes (#1 & #2 above). Needless to say, just thinking about being dropped out of the S&P 500 index sends cold sweat down your spine. The only way left to you to satisfy the quarterly expectations of Wall Street is to cut, cut and cut again anything that does not immediately contribute to your cashflow. You cut on-going refactoring of code even if your CTO and CIO have explained the technical debt curve to you in no uncertain terms. You are not happy to do so but you are willing to pay the price down the road. You are basically following a “survive to fight another day” strategy.

If you accept this explanation for the level of debt we are staring at, the core issue with respect to IT debt at the individual company level [3] is how “patient” (or “impatient”) investment capital is. Studies by Carlota Perez seem to indicate we are entering a phase of the techno-economic cycle in which investment capital will shift from financial speculation toward (the more “patient”) production capital. While this shift is starting to happens, you have the opportunity to apply “an ounce of prevention is worth a pound of cure” strategy with respect to the new code you will be developing.

My recommendation would be to combine technical debt measurements with software process change. The ability to measure technical debt through code analysis is a necessary but not sufficient condition for changing deep-rooted patterns. Once you institute a process policy like “stop the line whenever the level of technical debt rose,” you combine the “necessary” with the “sufficient” by tying the measurement to human behavior. A possible way to do so through a modified Agile/Scrum process is illustrated in Figure 2:

Figure 2: Process Control Model for Controlling Technical Debt

As you can see in Figure 2, you stop the line and convene an event-driven Agile meeting whenever the technical debt of a certain build exceeds that of the previous build. If ‘stopping the line’ with every such build is “too much of a good thing” for your environment, you can adopt statistical process control methods to gauge when the line should be stopped. (See Using 3σ  Control Limits in Software Engineering for a discussion of the settings appropriate for your environment.)

An absolutely critical question this analysis does not cover is “But how do we pay back our $1 trillion debt?!I will address this most important question in a forthcoming post which draws upon the threads of this post plus those in the preceding Part I.

Footnotes:

[1] Kyte/Gartner define IT Debt as “the costs for bringing all the elements [i.e. business applications] in the [IT] portfolio up to a reasonable standard of engineering integrity, or replace them.” In essence, IT Debt differs from the definition of Technical Debt used in The Agile Executive in that it accounts for the possible costs associated with replacing an application. For example, the technical debt calculated through doing code analysis on a certain application might amount to $500K. In contrast, the cost of replacement might be $250K, $1M or some other figure that is not necessarily related to intrinsic quality defects in the current code base.

[2] See Hagel, Brown and Davison: The Power of Pull: How Small Moves, Smartly Made, Can Set Big Things in Motion.

[3] As distinct from the core issue at the national level.