The Agile Executive

Making Agile Work

Posts Tagged ‘Technical Debt

Is There Something Inherently un-Agile About ERP Software?

with one comment

A reader of the post Make the Hairs on the Back of Your Neck Stand Up posed the following question:

I wonder if there’s something inherently un-Agile (and thus, unable to change fast enough to meet new business demands) about older enterprise software, or just ERP software?

The answer IMHO is size. To quote Capers Jones:

Since defect potentials tend to rise with the overall size of the application, and since defect removal efficiency levels tend to decline with the overall size of the application, the overall volume of latent defects delivered with the application rises with size. This explains why super-large applications in the range of 100,000 function points, such as Microsoft Windows and many enterprise resource planning (ERP) applications may require years to reach a point of relative stability. These large systems are delivered with thousands of latent bugs or defects. 

The phenomenon described by Jones is often exacerbated through the “ship more infrequently” syndrome. IMVU’s Timothy Fritz describes it as follows:

While this may decrease downtime (things break and you roll back), the cost on development time from work and rework will be large, and mistakes will continue to slip through. The natural tendency will be to ship even more infrequently, until you aren’t shipping at all. Then you’ve gone and forced yourself into a total rewrite. Which will also be doomed.  

You might choose to reduce your technical debt instead of trying total rewrite. Chances are you will struggle to find Elbow Room for Handling Technical Debt. My hunch is that once >50% of development resources are assigned to maintaining the software on an on-going basis, it is time to get into refactoring big time. If you don’t, sooner or later you are likely to find you can’t afford the software you developed.

Make the Hairs on the Back of Your Neck Stand Up

with 9 comments

Cote sent me the recent CIO Magazine article entitled ERP’s Paralysis Problem and the Repercussions for Business Everywhere. The article discusses the findings from a December 2009 study conducted by IDC and sponsored by ERP vendor Agresso, as follows:

A couple of verbatim responses from respondents should make the hairs on the back of your neck stand up: “Capital expenditure priorities are shifted into IT from other high-payback projects” just to perform necessary ERP changes, noted one respondent. Said another: “Change to ERP paralyzes the entire organization in moving forward in other areas that can bring more value.”

To make doubly certain the message gets across, the article finishes with the following “nocturnal” paragraph:

As the sun finally sets on the first decade in the new millennium, it’s high time we say good night to ERP. A new day will be starting soon, and the blemished legacy and failings of ERP’s nearly four-decade-long reign will be a distant memory.

Maybe. While ERP systems no doubt have their own particular twists, the sorry state of affairs described above is true of various industries that have developed complex software systems over prolonged periods of time. Just in the past few months I have witnessed such situations in banking and health care. In previous life I had been exposed to more of the same in other industries. The software decayed and decayed but technical debt had never been reduced. Consequently, the cost of change, any change, today is horrendous. As Jim Highsmith‘s chart below indicates, “once on the right of the curve, all choices are hard.”

in-can-you-afford-the-software-you-are-developing

In Estimating Software Costs, author Capers Jones quantifies five-year cost of software application ownership (for the vendor). He examines three similar applications, each of nominal size of 1000 function points, as a function of the sophistication of the corresponding projects. The respective life cycle costs are as follows:

  • Lagging projects: $2,316,000
  • Average projects: $1,860,000
  • Leading projects: $1,312,000

Jones goes on to issue the following stern warning:

All known compound objects decay and become more complex with the passage of time unless effort is exerted to keep them repaired and updated. Software is no exception… Indeed, the economic value of lagging applications is questionable after about three to five years. The degradation of initial structure and the increasing difficulty of making updates without “bad fixes” tends towards negative returns on investment (ROI) within a few years.

Enough, indeed, to make the hairs on the back of your neck stand up…

The Headlong Pursuit of Growth, and Its Aftermath

with 4 comments

The December 12-18, 2009 issues of The Economist features a couple of fascinating articles on Toyota Motor Corporation. According to The Economist, Toyota’s President  reached the following dire conclusion on the situation Toyota is facing:

Mr Toyoda’s alarm call last month appears partly to have been prompted by reading “How the Mighty Fall”, a book by Jim Collins, an American management writer, which identifies five stages of corporate decline. Mr Toyoda reckons that Toyota may already be at the fourth stage. Companies at this point, says Mr Collins, frequently still have their destinies in their own hands, but often flit from one supposed “silver bullet” strategy to another, thus accelerating towards the fate they are trying to avoid.

In the litany of things that went wrong, an interesting point is made about the Toyota Production System:

… the recalls continued and Toyota started slipping in consumer-quality surveys. A year later Consumer Reports, an influential magazine, dropped three Toyota models from its recommended list. The magazine added that it would “no longer recommend any new or redesigned Toyota-built models without reliability data on a specific design”. People within the company believe these quality problems were caused by the strain put on the fabled Toyota Production System by the headlong pursuit of growth.

Whatever Agile method you practice – Scrum, Lean, Kanban, Crystal, etc. – you need to be cognizant of three touch points with the Toyota experience reported above:

  • Just like the Toyota Production System, your software method is a “vehicle” which is subject to policy decisions from above. It cannot, however, compensate for policy failures.
  • If your company relentlessly pursues growth, the quality/technical debt liability it is likely to incur coud easily outweigh the benefits of growth. Consider the upside potential of growth vis-a-vis the downside of the resultant technical debt. When appropriate, monetize technical debt using the technique described in Technical Debt on Your Balance Sheet.
  • In addition to monetizing the technical debt, evaluate the various kinds of risks indicated in The View From The Executive Suite. A sense of how devastating those might be is given by Toyota’s own experience:

Just as Cadillac used to be synonymous with luxury and BMW with sportiness, Toyota was a byword for quality and reliability… The danger in all of this for Toyota is that its loyal (and mostly satisfied) customers in America have long believed that the firm was different from others and thus hold it to a higher standard. The moment that Toyota is seen as just another big carmaker, a vital part of the mystique that has surrounded the brand will have been rubbed away.

Please remember – unless you work for Toyota Motor Corporation, chances are your company would not be able to take the kind of risk Toyota can.

It Won’t Work Here

with 3 comments

Two major obstacles to vetting Agile topics effectively with executives were identified in the post entitled The Business Value of Agile Software Methods:

  1. Lack of hard quantitative data.
  2. The “It won’t work here” syndrome.

As indicated in the post, the data provided in the study How Agile Projects Measure Up, and What This Means to You and the book The business Value of Agile Software Methods address the first obstacle. This follow-on post is about the second of the two obstacles – the resistance to Agile transformation in the face of hard data on its benefits to other companies.

Resistance in the form of “it won’t work here” is typically anchored in one or more of the following five beliefs:

  1. Uniqueness: “Some very unique elements exist in our company. These elements render industry data inapplicable.”
  2. Secret sauce: “Something very special element existed in the companies reporting great success with Agile. Our company does not possess nor have access to the ‘secret sauce’ that enabled success elsewhere.”
  3. Cultural change: “For the Agile initiative to succeed, our corporate culture needs to change. The required cultural change takes a lot of time and involves a great deal of pain. All in all, the risk of rolling Agile is unacceptably high.”
  4. Affordability: “The company is strapped to the degree that investment in another software method is a luxury it can’t afford.”
  5. Software is not core to us: “We are not a software company, nor is software engineering our core competency. Software is merely one of the many elements we use in our business.”

Various other reasons for not going Agile in the context of a specific company are, of course, cited at some frequency. The five reasons listed above seem to be encountered most often by Agile champions.

Use the following counter-arguments to turn around these beliefs:

  1. Uniqueness: Very rare occurence. Companies use similar business designs, apply fairly standard operating procedures, utilize common technology, are subject to the same regulatory constraints that their competitors are, have offshore sites in places like India, etc. Discussion of your company vis-a-vis its direct competitor usually suffices to overcome the uniqueness claim. 
  2. Secret sauce: The ‘secret sauce’ is neither secret nor difficult to concoct. For example, the secret sauce used by BMC Software in its successful Agile initiative  had four simple ingredient: intentionality, know-how, flexibility and patience. Based on insights by colleague and friend Alan Atlas, I have recently added mutuality as the fifth ingredient. Your own secret sauce might be somewhat different, but I very much doubt that an extravagantly exotic sauce will be needed.
  3. Cultural change: Myth has it that Agile would only work in the Collaborative culture. Reality is it will work in any of the four core cultures identified by Schneider: Control, Competence, Cultivation or Collaboration. See Four Principles, Four Cultures, One Mirror for an approach to building Agile on the strength of whatever culture prevails in your company/organization.
  4. Affordability: The question to ask is whether you can afford not to improve your software. Tools are readily available to quantify your company’s technical debt. Monetize the technical debt and include it as a liability line item in a pro forma balance sheet. Doing so is likely to shift the discussion from affordability to how to create elbow room for handling the technical debt.
  5. Software is not core to us: Indeed, it might not be but it is likely to become so in just about any industry. Use an analogy like the record industry vis-a-vis the publishing industry. The record industry has been decimated by software over the past decade. Chances are a similar decimation is likely to occur in publishing unless the industry transforms itself. (Some of the decimation that already took place in publishing has become quite visible recently. For example, last week Bloomberg LP announced completion of the acquisition of BusinessWeek for a paltry $5M).

You will need to be realistically patient with respect to the time it takes for the considerations listed above to sink in. It could easily take six months just to forge a consensus on the subject in the executive team. It might then take another six month to operationalize the consensus. Chances are there is an elephant hidden somewhere in the “room” if you don’t carry the day with within a one year period of diligently vetting Agile with your executives.

Your Investment in Enterprise Software – Guidelines to CIOs and CFOs

with 6 comments

The overall investment associated with implementing and maintaining a suite of enterprise software products could be significant. A 1:4 ratio between product investment and the corresponding investment over time in related services is not uncommon. In other words, an initial $2M in licensing a suite of enterprise software products might easily balloon to $10M in total life-cycle costs (initial investment in perpetual license plus the ongoing investment in associated services).

I offer the following rule-of-a-thumb guidelines to assessing whether the terms quoted by a vendor for an enterprise software suite of products are right:

  1. Standard maintenance costs: Insist on a 1:1 ratio between license and standard maintenance over a 5 year period. If standard maintenance costs over this period exceed the corresponding license costs, chances are: A) the vendor is quite greedy; or, B) the vendor’s software accrued a non-negligible amount of technical debt. Ask the vendor to quantify the technical debt in monetary terms. See Technical Debt on Your Balance Sheet for an example how to conduct such quantification.

  2. Premium customer support costs: Certain premium customer support services could be quite appropriate for your business parameters. However, various “premium services” could actually address deficits or defects in the enterprise software products you license. If the technical debt figure is high, the vendor you are considering might not be able to afford the software he has developed. Under such circumstances, “premium services” could simply be a vehicle the vendor uses to recoup his investment in software development.
  3. Professional services costs: Something is wrong if the costs of professional services exceed licensing cost. Either the suite of products you are considering is not a good fit for your business parameters or the initiative you are aspiring to implement through the software is overly ambitious.

To summarize, the grand total of license fees, customer support fees and professional services fees over a 5 year period should not be higher than 3X license fees. Something is out of balance if you are staring at  a 4X or 5X ratio for the software you are considering.

One final point: please do not forget to add End-of-Life costs to the economic calculus. Successful enterprise software  initiatives can be very sticky.

Technical Debt: Refactoring vis-a-vis Starting Afresh

with 2 comments

Only three options are available to you once your technical debt overwhelms the development and customer support teams:

  1. Continue as is.
  2. Start refactoring in earnest to pay back your debt.
  3. Switch to a new code base, either by launching an unencumbered development project or through acquiring new software from another company.

The first option, obviously, does not solve any problem – you simply continue a struggle that becomes more and more difficult over time. As indicated in the post Elbow Room for Handling Technical Debt, the second option is quite difficult to carry out if it is exercised at a late point in the software’s life cycle. This post examines the third option – starting afresh.

The attraction of starting afresh is always tempting. The sins of the past, hidden in layers over layers of technical debt and manifesting themselves in low customer satisfaction and poor customer retention, are (sort of) forgiven. The shiny new software holds the promise of better days to come.

Trouble is, your old sins are there to get you in three nasty ways:

  1. New technical debt: Even if the shiny new software is very good indeed, it will decay over time. Unless you transformed your software engineering practices to ensure refactoring is carried out on an ongoing basis, sooner or later you will accrue new technical debt.
  2. Migration: Good that the new software might be, it is not likely to provide all the functionality the old software had. Migrating your customers will require a lot of hard work, particularly if your customers developed elaborate business processes on top of your old software.
  3. Critical race between old and new: For a period of time (say a couple of years) the old software is likely to run alongside the new software. Any significant new regulatory requirement will force you to update both the old software and the new software. Ditto for security. Any nifty new technology you might want to embed in your new software is likely to be equally desirable to customers who still use your old software. You will be pressed to incorporate the new technology in the old software, not just in the new software.

Before opting for new software (in preference to refactoring the old software), I would recommend you compare the economics of the two option. My rule of a thumb for the calculus of introducing new enterprise software to replace legacy software is straightforward:

For a period of about two years, assume the run rate for dev/test/support will be 150% of your current investment in development, test and support of the old software.

Please note that the 150% figure is just the expected run rate for running the new software alongside the legacy software. You will need, of course, to add the cost of acquiring/producing the new software to the economic calculus.

You might be able to reduce the ‘150% for two years’ investment by applying some draconian migration measures. Such measures, of course, will affect your customers. In the final analysis, your decision to acquire new software must be viewed as the following simple question:

What value do you place on your customer base?!

Technical Debt Goes Generic

with 3 comments

Rally’s Richard Leavitt mentioned “his” technical debt in a conversation the two of us had last evening. As Richard is the head of marketing for Rally, I was expecting to hear about some deficit in the functionality, design, coding or testing of one of the market and customer facing websites his department deploys. I was dead wrong.

Richard was actually using technical debt in a generic sense. Anything in his department that they had to rush through and now plan to go back to and revisit/improve/fix is categorized as technical debt. The term applies to (say) laying the foundations for a marketing campaign as much as it does to re-architecting an application in order to improve its performance.

I don’t really know how wide spread the use of “technical debt” in this generic sense is. I am, however, impressed: another term of art is starting to get into the English language! How appropriate that such use of the term starts at a company that applies Agile values and practice to most of its operational and business processes.

Written by israelgat

November 4, 2009 at 2:58 pm

Elbow Room for Handling Technical Debt

with 5 comments

It has become something of a pattern recently. Somebody contacts me about software that has become extremely difficult to maintain. Irrespective of the domain in which the software is applied, the situation is usually characterized by an overwhelming amount of technical debt accompanied by an unaccptably high error feedback ratio. Between those “twins”, both customer support and development are thrashing to the extent that development of new functionality has pretty much ceased. “Just” maintaining the software consumes 90-100% of the cycles of the development teams (and >100% of the cycles of the customer support team).

I do not really mind being considered kind of “Software 911” service. What I find fustraing, however, is that I (and other consultants) typically get called too late. The technical debt when we get called is so overwhelming that it is extremely difficult to generate the cycles required for refactoring the code and establishing solid software engineering  practices. The refactoring “medicine” can’t be taken because customer crises leave no time for learning how to refactor nor for carrying out refactoring in a thoughtful manner. Folks trying to refactor the code get interrupted so often to deal with crises that any attempt to establish flow gets in trouble. The elbow room required for systemic refactoring work simply does not exist anymore.

I am not quite certain where the fine line between “Software 911” and “Pathology 911” lies. My hunch is that once >50% of development resources are assigned to maintaining the software on an on-going basis, it is time to get into refactoring big time. If you don’t, sooner or later you are likely to find you can’t afford the software you developed.

Written by israelgat

October 7, 2009 at 6:56 pm

Sign of the Times

leave a comment »

As I learned the hard way, the post Technical Debt on Your Balance Sheet missed an important risk associated with technical debt.

I got an email today advising me as follows:

Debt Con Expert is now following you on Twitter!

The good services provided by Debt Con Expert are debt consolidation loans, help with bad credit situations, how to tips on filing bankruptcy, etc.  As I do not owe a penny to anyone in this solar system, I am inclined to conclude I am being followed as a result of yesterday’s post on technical debt.

Please be prudent with technical debt. In addition to elevating the error feedback ratio of your software to an unacceptable level,  it might also lead to your mail box being flooded with debt consolidation offers.

Sigh.

Written by israelgat

September 30, 2009 at 11:44 pm

Technical Debt on Your Balance Sheet

with 17 comments

Colleague Jonathon Golden introduced me to a new plug in to Sonar. The plug in calculates the cost to fix the technical debt accrued in a product. For example, you might find an accrued technical debt amounting to $1M in a 500KLOC application. Obviously, you will need to spend $2 per each line of code to “pay back” your debt.

The expression of technical debt in monetary terms is intriguing. Unlike financial debt, there is no credit limit on technical debt. Hence, unless a team is proficient at refactoring on an ongoing basis, technical debt tends to grow over time as the underlying software decays. Beyond a certain level of debt, no good option is available. The code decayed to the point in which fixing anything in a hazardous proposition – every fix is likely to break something else. Under such circumstances, most/all of the development team gets sucked into maintaining the software instead of developing new features and functions.

Monetizing technical debt can have two far reaching implications, as follows:

  • A credit limit on technical debt can be established.  For example, when the technical debt reaches a certain level (say 25 cents per line of code), new functionality is put on hold. The team applies itself to aggressive refactoring to reduce the debt to an acceptable level.
  • For companies who capitalize software, technical debt could become a line item on the balance sheet. It will simply be listed as a liability.

From a customer perspective, the monetized technical debt on the balance sheet of a software vendor is a proxy for the technical risk involved in licensing software from this vendor. Such monetization could be easily extended to report technical debt per product family. With such reporting in place, the technical risk associated with licensing a specific product can be assessed.

Software vendors might frown at the requirement to monetize technical debt. I would contend that such a reporting requirement is absolutely consistent with the spirit of the Agile Manifesto:

Customer collaboration over contract negotiations

In other words, if you are reluctant to list your monetized technical debt you can’t really claim you practice Agile.

Written by israelgat

September 29, 2009 at 8:18 pm