Archive for the ‘Risk Assessment and Mitigation’ Category
Round Two: Can Technical Debt Constitute a Breach of Implied Warranties?
In a previous post I discussed whether technical debt could under some conditions constitute a breach of implied warranties. Examining the subject with respect to intent, I made the following observation:
It is a little tricky (though not impossible – see Using Credit limits to Constrain Development on Margin) to define the precise point where technical debt becomes “unmanaged.” One needs to walk a fine line between technical/methodical incompetence and resource availability to determine technical fraud. For example, if your code has 35% coverage, is it or is not unmanaged? Does the answer to this question change if your cyclomatic complexity per class exceeds 30? I would think the courts might be divided for a very long time on the question when does hidden technical debt represent a fraudulent misrepresentation.
One component of technical debt deserves special attention in the context of this post. I am referring to the conscious decision not to do unit testing at all… Such a conscious decision IMHO indicates no intention to pay back this category of technical debt – unit test coverage. It is therefore quite incompatible with the nature of an implied warranty:
Responses to my post were mixed. Various readers who are much more knowledgeable in the law than I am pointed out various legal defenses a software vendor could use. Click here for a deeper understanding of these subtle legal points.
Imagine my delight reading yesterday’s uTest interview with Cem Kaner. Cem makes the following statement in his interview:
ALI [American Law Institute] … started writing the Principles of the Law of Software Contracts. One of its most important rules is one that I advocated: a seller of software who knows about a defect of the software but does not disclose the defect to the customer will be held liable for damages caused to that customer by that defect. Note that this does not apply to free software (not sold). And if the seller discloses the defect, it becomes part of the product’s specification (it’s a feature). And if the seller doesn’t know about the defect there is no liability (once customers tell you about a defect, you have knowledge, so you cannot avoid knowledge for long by not testing). ALI adopted it unanimously last year. This is not law, but until the legislatures pass statutes, the Principles will be an important guide for judges. Even though I am a minor contributor to this work, I think the defect-disclosure requirement might be my career’s most important contribution to software quality. [Highlights by IG].
While not (yet?) a law, the Principles could indeed lead us where IMHO we as an industry should be. Technical debt manifests itself as bugs. By making the specific bugs part of the spec the software evolves from a quality standpoint. Moreover, the defect-disclosure requirement practically force software vendors to address their bugs within a “reasonable” amount of time. Customer bugs become an antidote to the relentless pressure to add functions and features without paying due attention to software quality.
I guess I missed the opportunity to include technical debt in the 2010 revision of the Principles. As we always do in software, I will target the next rev…
Can Technical Debt Constitute a Breach of Implied Warranties?
Photo credit: Dancing Lemur (Flickr)
Cunningham’s quip “A little debt speeds development so long as it is paid back promptly with a rewrite” is intuitively very clear. We are talking about short-term debt which will be reduced, and hopefully eliminated in entirety, at the earliest possible time.
The question this post addresses is what happens when the expected short-term technical debt becomes a significant long-term debt? Specifically, can technical debt under some conditions constitute a breach of implied warranties?
In his InformIT article Don’t “Enron” Your Software Project, Aaron Erickson coined the term “Technical Fraud” and connected it to Lemmon Laws:
As a reaction to seeing this condition and its deleterious effects, I coined the term technical fraud to refer to the practice of incurring unmanaged and hidden technical debt. Many U.S. states have “lemon laws” that make it illegal to knowingly sell someone a car that has undisclosed maintenance problems. Selling a “lemon” is a fraudulent practice in the world of cars, and it should be considered as such in the world of software.
It is a little tricky (though not impossible – see Using Credit limits to Constrain Development on Margin) to define the precise point where technical debt becomes “unmanaged.” One needs to walk a fine line between technical/methodical incompetence and resource availability to determine technical fraud. For example, if your code has 35% coverage, is it or is not unmanaged? Does the answer to this question change if your cyclomatic complexity per class exceeds 30? I would think the courts might be divided for a very long time on the question when does hidden technical debt represent a fraudulent misrepresentation.
One component of technical debt deserves special attention in the context of this post. I am referring to the conscious decision not to do unit testing at all.
Best I understand it, the rationale for not “bothering” with unit testing is a variant of the old ploy “we do not have time for testing here.” It is a resource allocation strategy that bets on the code being miraculously bug-free. Some amount of functional testing is done out of necessity – the code in customers hands needs to function as proclaimed. But, the pieces of code from which functionality is constructed are not subject to direct rigorous testing. The individual units of code will be indirectly exercised in some manner through functional testing, but not in a systemic manner to verify and validate correctness of the units of code per se.
Such a conscious decision IMHO indicates no intention to pay back this category of technical debt – unit test coverage. It is therefore quite incompatible with the nature of an implied warranty:
An implied warranty is as an unstated promise, assumed by the law in most sales transactions, that the product will be of at least average quality and will do what the average customer would expect it to do [The Reader’s Digest Legal Questions & Answers Book]
To #1 defense open to a software vendor who gets sued over lack of unit testing is that a fair average quality of software can be attained without any unit testing. As a programmer, I would think such defense would fly at the teeth of the availability since 1987 of the IEEE Standard for Software Unit Testing.
It is fascinating to note the duality between contracts and programming. For the programmer who follows the tenets of design by contract, “a unit test provides a strict, written contract that the piece of code must satisfy…”
Disclaimer: I am not an expert in the law. The opinion expressed in this post merely represents my layman’s understanding of principles of contract law that might be applicable to technical debt situations.
Should You Ship This Code Before Reducing Technical Debt?!
Source: JulesH, Wikipedia, A control flow graph of a simple function
Technical debt is usually perceived as a measure of expediency. You borrow a little (time) with the intent of paying it back as soon as possible. To quote Ward Cunnigham:
Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite… I thought that rushing software out the door to get some experience with it was a good idea, but that of course, you would eventually go back and as you learned things about that software you would repay that loan by refactoring the program to reflect your experience as you acquired it.
As is often the case with financial debt, technical debt accrues with compound interest. Once it reaches a certain level (e.g. $1 per line of code) you stare at a difficult question:
Should I ship this code before reducing the accrued technical debt?!
The Figure below, taken from An Objective Measure of Code Quality by Mark Dixon, answers the question with respect to one important component of technical debt – cyclomatic complexity. Once complexity per source code file exceeds 74, the file is for most practical purposes guaranteed to contain errors. Some of the errors in such a file might be trivial. However, a 2007 study by Capers Jones indicates about a third of the errors found in released code are likely to be serious enough to stop an application from running or create erroneous outputs.
To answer the question cited above – Should You Ship This Software Before Reducing Technical Debt?! – examine both cost and risk for the number of error-prone files you are about to unleash:
- The economics of defect removal clearly favor early defect removal over late defect removal. The cost of removal grows exponentially as function of time.
- Brand risk should be first and foremost on your mind. If complexity figures higher than 74 per file are more of the norm than the exception, you are quite likely to tarnish your image due to poor quality.
If you decide to postpone the release date until the technical debt has been reduced, you can apply yourself to technical debt reduction in a biggest-bang-for-the-buck manner. The analysis of complexity can identify the hot spots in your code, giving you a de-facto roadmap you would be wise to follow.
Conversely, if you opt to ship the code without reducing technical debt, you might lose this degree of freedom to prioritize your “fix it” work. Customer situations and pressures might force you to attend to fixing modules that do not necessarily provide as much bang for the buck.
Postscript: Please note that the discussion in this post is strictly limited to intrinsic quality. It does not address at all extrinsic quality. In other words, reducing/eliminating technical debt does not guarantee that the customer will find the code valuable. I would suggest reading Beyond Scope, Schedule and Cost: Measuring Agile Performance in the Cutter Blog for a more detailed analysis of the distinction between the two.
Erratum: The figure above is actually taken from a blog post on the Mark Dixon paper cited in my post. See McCabe Cyclomatic Complexity: the proof is in the pudding. My apology for the error.
How to Initiate a DevOps Project
Source: 17th/21st Lancers c. 1922-1929 “THE FIGHTING SPIRIT!”
Agile consultants on a development project often start by helping the team construct a backlog. The task is sufficiently concrete to get all stakeholders (product management, project management, development, test, any others) on a collaborative track through the creation of a key artifact. The backlog establishes a base line for the tasks to be carried out in the project.
For a DevOps project, start by establishing the technical debt of the software to be released to operations. By so doing you build the foundations for collaboration between development and operations through shared data. In the DevOps context, the technical debt data form the basis for the creation and grooming of a unified backlog which includes various user stories from operations.
Apply the same approach when you are fortunate to be able to include folks from operations in the Agile team from the very beginning. You start with zero technical debt, but you track it on an ongoing basis and include the corresponding “fix-it” stories in the backlog as you accrue the debt. Running technical debt analytics on the source code every two weeks is a good practice to follow.
As the head of development, you might not be comfortable sharing technical debt data. This being the case, you are not ready for DevOps.
Using Credit Limits to Constrain “Development on Margin”
Buying (stocks) on margin is broadly recognized as a risky investment strategy. Funding long-term investments with short-term debt exposes the investor to margin calls as he/she might not be able to secure more financing when needed. The resultant margin call is never pleasant.
The accrual of technical debt in the course of aggressively developing functions and features is quite a similar phenomenon. The CTO is betting the functionality he/she is developing will pay off before the need to “pay back” the technical debt becomes imperative. The temptation to do so is particularly strong due to the lack of credit limits on technical debt. For all practical purposes the CTO is “developing on margin.”
In his comprehensive studies of the economics of software, Capers Jones has actually put a 3-5 year ceiling on the economical viability of developing on margin:
Indeed, the economic value of lagging applications is questionable after about three to five years. The degradation of initial structure and the increasing difficulty of making updates without “bad fixes” tends towards negative returns on investment (ROI) within a few years.
As the CEO leading a company, or the venture capitalist funding it, you can restrain development on margin by establishing credit limits. Use a combination of static code analysis with dynamic program analysis to calculate the amount of accrued technical debt in $$ terms. (An illustration of such calculation as well as a breakdown of the technical debt is given in the Sonar chart above). Set a limit (say $0.25 per line of code) on the amount of permitted technical debt. Once the limit is reached, developers are not allowed to continue developing new functionality – they have to first reduce (and hopefully eliminate) their technical debt.
A very simple “Lacmus test” is available to the CEO/VC until the code is instrumented and the analytics illustrated above generated. Ask your CTO about unit test coverage. If the coverage is low (say <30%), chances are the technical debt is high. Whether the CTO realizes it or not, low unit test coverage is a good indicator of technical debt of all kinds. Moreover, the investment required to develop a full-fledged suite of unit tests is often the largest component of the technical debt to be paid back.
The Headlong Pursuit of Growth, and Its Aftermath
The December 12-18, 2009 issues of The Economist features a couple of fascinating articles on Toyota Motor Corporation. According to The Economist, Toyota’s President reached the following dire conclusion on the situation Toyota is facing:
Mr Toyoda’s alarm call last month appears partly to have been prompted by reading “How the Mighty Fall”, a book by Jim Collins, an American management writer, which identifies five stages of corporate decline. Mr Toyoda reckons that Toyota may already be at the fourth stage. Companies at this point, says Mr Collins, frequently still have their destinies in their own hands, but often flit from one supposed “silver bullet” strategy to another, thus accelerating towards the fate they are trying to avoid.
In the litany of things that went wrong, an interesting point is made about the Toyota Production System:
… the recalls continued and Toyota started slipping in consumer-quality surveys. A year later Consumer Reports, an influential magazine, dropped three Toyota models from its recommended list. The magazine added that it would “no longer recommend any new or redesigned Toyota-built models without reliability data on a specific design”. People within the company believe these quality problems were caused by the strain put on the fabled Toyota Production System by the headlong pursuit of growth.
Whatever Agile method you practice – Scrum, Lean, Kanban, Crystal, etc. – you need to be cognizant of three touch points with the Toyota experience reported above:
- Just like the Toyota Production System, your software method is a “vehicle” which is subject to policy decisions from above. It cannot, however, compensate for policy failures.
- If your company relentlessly pursues growth, the quality/technical debt liability it is likely to incur coud easily outweigh the benefits of growth. Consider the upside potential of growth vis-a-vis the downside of the resultant technical debt. When appropriate, monetize technical debt using the technique described in Technical Debt on Your Balance Sheet.
- In addition to monetizing the technical debt, evaluate the various kinds of risks indicated in The View From The Executive Suite. A sense of how devastating those might be is given by Toyota’s own experience:
Just as Cadillac used to be synonymous with luxury and BMW with sportiness, Toyota was a byword for quality and reliability… The danger in all of this for Toyota is that its loyal (and mostly satisfied) customers in America have long believed that the firm was different from others and thus hold it to a higher standard. The moment that Toyota is seen as just another big carmaker, a vital part of the mystique that has surrounded the brand will have been rubbed away.
Please remember – unless you work for Toyota Motor Corporation, chances are your company would not be able to take the kind of risk Toyota can.
Your Investment in Enterprise Software – Guidelines to CIOs and CFOs
The overall investment associated with implementing and maintaining a suite of enterprise software products could be significant. A 1:4 ratio between product investment and the corresponding investment over time in related services is not uncommon. In other words, an initial $2M in licensing a suite of enterprise software products might easily balloon to $10M in total life-cycle costs (initial investment in perpetual license plus the ongoing investment in associated services).
I offer the following rule-of-a-thumb guidelines to assessing whether the terms quoted by a vendor for an enterprise software suite of products are right:
- Standard maintenance costs: Insist on a 1:1 ratio between license and standard maintenance over a 5 year period. If standard maintenance costs over this period exceed the corresponding license costs, chances are: A) the vendor is quite greedy; or, B) the vendor’s software accrued a non-negligible amount of technical debt. Ask the vendor to quantify the technical debt in monetary terms. See Technical Debt on Your Balance Sheet for an example how to conduct such quantification.
- Premium customer support costs: Certain premium customer support services could be quite appropriate for your business parameters. However, various “premium services” could actually address deficits or defects in the enterprise software products you license. If the technical debt figure is high, the vendor you are considering might not be able to afford the software he has developed. Under such circumstances, “premium services” could simply be a vehicle the vendor uses to recoup his investment in software development.
- Professional services costs: Something is wrong if the costs of professional services exceed licensing cost. Either the suite of products you are considering is not a good fit for your business parameters or the initiative you are aspiring to implement through the software is overly ambitious.
To summarize, the grand total of license fees, customer support fees and professional services fees over a 5 year period should not be higher than 3X license fees. Something is out of balance if you are staring at a 4X or 5X ratio for the software you are considering.
One final point: please do not forget to add End-of-Life costs to the economic calculus. Successful enterprise software initiatives can be very sticky.
Elbow Room for Handling Technical Debt
It has become something of a pattern recently. Somebody contacts me about software that has become extremely difficult to maintain. Irrespective of the domain in which the software is applied, the situation is usually characterized by an overwhelming amount of technical debt accompanied by an unaccptably high error feedback ratio. Between those “twins”, both customer support and development are thrashing to the extent that development of new functionality has pretty much ceased. “Just” maintaining the software consumes 90-100% of the cycles of the development teams (and >100% of the cycles of the customer support team).
I do not really mind being considered kind of “Software 911” service. What I find fustraing, however, is that I (and other consultants) typically get called too late. The technical debt when we get called is so overwhelming that it is extremely difficult to generate the cycles required for refactoring the code and establishing solid software engineering practices. The refactoring “medicine” can’t be taken because customer crises leave no time for learning how to refactor nor for carrying out refactoring in a thoughtful manner. Folks trying to refactor the code get interrupted so often to deal with crises that any attempt to establish flow gets in trouble. The elbow room required for systemic refactoring work simply does not exist anymore.
I am not quite certain where the fine line between “Software 911” and “Pathology 911” lies. My hunch is that once >50% of development resources are assigned to maintaining the software on an on-going basis, it is time to get into refactoring big time. If you don’t, sooner or later you are likely to find you can’t afford the software you developed.
Technical Debt on Your Balance Sheet
Colleague Jonathon Golden introduced me to a new plug in to Sonar. The plug in calculates the cost to fix the technical debt accrued in a product. For example, you might find an accrued technical debt amounting to $1M in a 500KLOC application. Obviously, you will need to spend $2 per each line of code to “pay back” your debt.
The expression of technical debt in monetary terms is intriguing. Unlike financial debt, there is no credit limit on technical debt. Hence, unless a team is proficient at refactoring on an ongoing basis, technical debt tends to grow over time as the underlying software decays. Beyond a certain level of debt, no good option is available. The code decayed to the point in which fixing anything in a hazardous proposition – every fix is likely to break something else. Under such circumstances, most/all of the development team gets sucked into maintaining the software instead of developing new features and functions.
Monetizing technical debt can have two far reaching implications, as follows:
- A credit limit on technical debt can be established. For example, when the technical debt reaches a certain level (say 25 cents per line of code), new functionality is put on hold. The team applies itself to aggressive refactoring to reduce the debt to an acceptable level.
- For companies who capitalize software, technical debt could become a line item on the balance sheet. It will simply be listed as a liability.
From a customer perspective, the monetized technical debt on the balance sheet of a software vendor is a proxy for the technical risk involved in licensing software from this vendor. Such monetization could be easily extended to report technical debt per product family. With such reporting in place, the technical risk associated with licensing a specific product can be assessed.
Software vendors might frown at the requirement to monetize technical debt. I would contend that such a reporting requirement is absolutely consistent with the spirit of the Agile Manifesto:
Customer collaboration over contract negotiations
In other words, if you are reluctant to list your monetized technical debt you can’t really claim you practice Agile.
How Hard Should the Agile Champion Push?
A question which often comes up in the course of the Rally Agile Success Tour is about the balance to strike between conviction, passion and politics. By virtue of his/her interest in the topic, the Agile champion is often more knowledgeable than his/her superiors on what Agile really is and which strategies are best suited to roll it out. A thoughtful push toward Agile values, principles and practices may lead to major improvements in the way an organization practices Agile. In contrast, an overly hard push might lead to regression in the state of Agile affairs. Moreover, it could easily lead to a breach of the psychoilogical contract between the Agile champion and employer.
Reading The Bureaucratic Phenomenon – a book recommended to me by Forrester’s Tom Grant – I found the following nugget:
The power of decision making within a bureaucratic system of organization is located exactly at the points where the stability of the internal “political” system is preferred to the achievement of the functional goals of the organization.
The specific nature of the bureaucracy with which one has to cope changes, of course, from one company to another. No matter what kind of bureaucracy the Agile champion has to cope with, the observation cited above applies. The Agile champion can successfully push hard as long as he/she does not cross the fine line between achievement of functional goals and “political” stability.
The Agile Executive 