Measuring Agile Success Rate the Right Way
Much has been said recently about the success/failure rate of Agile projects. In particular, a debate arose around the success rate of Scrum vis-a-vis Kanban. For example, in a post entitled Some Day Kanban will fail 75% of the Time, colleague Jurgen Appelo states as follows:
Unfortunately, some people arguing against Scrum include these ScrumBut teams in their evaluations of the “high failure rate” of Scrum. They love quoting that “at least 75 percent of Scrum implementations fail.” And I think “Yes of course, 75% fails when that includes the teams that don’t understand what they’re doing.”
I would like to add one other “dimension” to the discussion: boundary conditions.
Any Agile initiative – Crystal, Scrum, Kanban, etc. – typically starts from a certain state of affairs of the code that has already been developed using a Waterfall method or no method at all. Even brand new projects produce code that invariably interacts with other software components that are already deployed, warts and everything. Pristine environments with no technical debt for the Agile initiative to deal with are rare.
Like it or not, the Agile initiative is saddled from the outset with a certain amount of technical debt. Code has been duplicated, rules violated, complexity ran amuck, etc. A typical enterprise software team starts with hundreds of thousands $$ in technical debt, if not millions. This debt needs to be “paid back.” Probably not over night, but certainly over a period of time. As illustrated by the following figure from Jim Highsmith, things get ugly if the debt is not paid back over an extended period of time.
The evaluation of success or failure of the Agile initiative needs to take technical debt into account. A team of 50 with an accrued technical debt of $100,000 has a much easier job on its hands transitioning to Agile than a similar size team starting with $1M in technical debt on its hands.
Whatever criteria you use to determine whether an Agile initiative has been successful, I would suggest the following boundary condition needs to be satisfied:
Technical debt at the end of the project/initiative must be significantly lower than technical debt at the start of the project.
Use the techniques outlined in Using Credit Limits to Constrain Development on Margin to calculate technical debt before and after. In addition to qualifying your Agile success, quantifying technical debt will do a lot towards improving the quality of your software.
The Agile Executive 
I’ve had many organizations tell me that things weren’t working because of a really bad code base.
It was often technically true, but only because they planned based upon nearly perfect conditions and then their plan collided with reality. “We could have completed all this in a few months, if they had done a better job on the last release!”
I’ll argue that technical debt is irrelevant. You plan based upon having technical debt. If you deliver on time with good quality (for the features/changes you promised) then you have a success. If the product still has hundreds or thousands of existing issues – that you did not insert – that is not relevant to the measure of success.
It is always good to reduce technical debt, but making it a goal is just another requirement added to the effort. If you had the requirement, you want to achieve it. If not, it is irrelevant.
I’ve had quality and test organizations beat us up in front of more senior management because we delivered software that still had existing issues from previous releases. The good news was it was because we delivered on time with good quality what we promised. The bad news, for QA and test, was they felt threatened by a sudden increase in quality.
Bruce Benson
http://PMToolsThatWork.com
Bruce Benson
March 13, 2010 at 8:03 pm
Technical debt is deceptive – it does not get you immediately, but it sure will over time. In his comprehensive studies of the economics of software, Capers Jones has actually put a 3-5 year ceiling on the economical viability of applications with a high level of debt:
“Indeed, the economic value of lagging applications is questionable after about three to five years. The degradation of initial structure and the increasing difficulty of making updates without “bad fixes” tends towards negative returns on investment (ROI) within a few years.”
The practice I recommend for dealing with technical debt is capturing the the work that needs to be carried out to reduce/eliminate technical debt as part of a unified backlog. A user story addressing technical debt could look as follows:
“As the general manager in charge of the business unit I want to reduce complexity per class below 15 so that we can respond more quickly and effectively to customer requests.”
Best,
Israel
israelgat
March 15, 2010 at 9:24 am
The 2009 Standish report says that 68% of all software projects fail. However, their definition of failure is poor at best. 75% by itself without definition is fairly useless.
Jim Highsmith
March 19, 2010 at 3:15 pm
I am truly baffled by Scrum/Lean and other ridiculous methodologies created to satisfy the hope that we can build complex software systems using infantile football strategies and blind hope. The current author like the majority of practitioners miss the point completely; the fact that teams do not what they are doing is EXACTLY why we need a formal well-defined process such as Open UP or RUP. Scrum does NOT work because it assumes that somehow developers/architects/BAs instinctively do the “right thing” by the project and everything works out as expected – seriously are you guys delusional?? You need a structured, formal and WELL_DOCUMENTED process for the SDLC. Nothing else works. Get over it, accept it and start building software that actually works. There is no quick fix, software development is COMPLEX. End of discussion.
John Smith
September 10, 2012 at 6:45 pm
The two are not inconsistent IMHO.
– Israel
israelgat
December 26, 2012 at 10:20 am