The Agile Executive

Making Agile Work

Posts Tagged ‘Executive Coaching

Should You Ship This Code Before Reducing Technical Debt?!

with 8 comments

File:Control flow graph of function with loop and an if statement without loop back.svg

Source: JulesH, Wikipedia, A control flow graph of a simple function

Technical debt is usually perceived as a measure of expediency. You borrow a little (time) with the intent of paying it back as soon as possible. To quote Ward Cunnigham:

Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite… I thought that rushing software out the door to get some experience with it was a good idea, but that of course, you would eventually go back and as you learned things about that software you would repay that loan by refactoring the program to reflect your experience as you acquired it.

As is often the case with financial debt, technical debt accrues with compound interest. Once it reaches a certain level (e.g. $1 per line of code) you stare at a difficult question:

Should I ship this code before reducing the accrued technical debt?!

The Figure below, taken from An Objective Measure of Code Quality by Mark Dixon, answers the question with respect to one important component of technical debt – cyclomatic complexity. Once complexity per source code file exceeds 74, the file is for most practical purposes guaranteed to contain errors. Some of the errors in such a file might be trivial. However, a 2007 study by Capers Jones indicates about a third of the errors found in released code are likely to be serious enough to stop an application from running or create erroneous outputs.

mccabegraph.jpg

To answer the question cited above – Should You Ship This Software Before Reducing Technical Debt?! –  examine both cost and risk for the number of error-prone files you are about to unleash:

  • The economics of defect removal clearly favor early defect removal over late defect removal. The cost of removal grows exponentially as function of time.
  • Brand risk should be first and foremost on your mind. If complexity figures higher than 74 per file are more of the norm than the exception, you are quite likely to tarnish your image due to poor quality.

If you decide to postpone the release date until the technical debt has been reduced, you can apply yourself to technical debt reduction in a biggest-bang-for-the-buck manner. The analysis of complexity can identify the hot spots in your code, giving you a de-facto roadmap you would be wise to follow.

Conversely, if you opt to ship the code without reducing technical debt, you might lose this degree of freedom to prioritize your “fix it” work.  Customer situations and pressures might force you to attend to fixing modules that do not necessarily provide as much bang for the buck.

Postscript: Please note that the discussion in this post is strictly limited to intrinsic quality. It does not address at all extrinsic quality. In other words, reducing/eliminating technical debt does not guarantee that the customer will find the code valuable. I would suggest reading Beyond Scope, Schedule and Cost: Measuring Agile Performance in the Cutter Blog for a more detailed analysis of the distinction between the two.

Erratum: The figure above is actually taken from a blog post on the Mark Dixon paper cited in my post. See McCabe Cyclomatic Complexity: the proof is in the pudding. My apology for the error.

Should You Invest in This Software?!

with 2 comments

Martin Fowler - Technical debt quadrant by Kalle Hoppe.

Source: martinfowler.com/bliki/TechnicalDebtQuadrant.html

Consider the following scenario: You are a venture capitalist. One of your portfolio companies has been working for a few years on a promising software application. Various surprises with respect to schedule and functionality have been sprung on you along the way. The company now asks for one last shot-in-the-arm in order to get the product out the door, market and sell it. Should you open your wallet one more time to fund this alleged last push?

It is a familiar scenario not only for venture capitalists, but for CEOs, CFOs, general managers and M&A executives. A renowned CEO once told me the following when I pushed my luck with respect to project funding:

Israel, I have a warehouse of software products that never generated a dime for me.

Believe me, this CEO was neither amused nor philosophical…

Code analysis techniques have progressed to the point that the answer to the software investment question for object-oriented code can to a certain extent be determined  through quantifying technical debt. For example, assume the following circumstances:

  • A company expects to ship 500K lines of code in 6 months.
  • The company asks for additional $2M to complete development and make a significant resound in the market.

To assess the investment decision, apply the code analysis techniques described in Using Credit Limits to Constrain Development on Margin to quantify the technical debt.  Assuming a debt of $2 per line of code has been identified, the overall technical debt amounts to $1M (2X500K).

The investment decision then is not an incremental $2M decision. It is actually a $3M ($2M+1M) investment decision when the technical debt is taken into account.  The technical debt might not need to be paid overnight, but it will have to be paid back over a period of time. The team might not hire additional resources to reduce/eliminate the technical debt, but the team resources dedicated to reducing technical debt will not be available  to carry out other assignments. Hence, the opportunity cost ($1M) is real, relevant and should be taken into account.

If you are hesitant to continue investing in this software/team, you stare at a tricky question:

  • What will it take to start afresh?

If you decide to make the $3M investment, two operational questions pose themselves:

  • How should work on reducing/eliminating technical debt be interleaved with other pressing work such as new functions and features?
  • Given a $1M debt on 500K lines of code, can the company indeed ship as expected in 6 months?

We will address these three questions in forthcoming posts in The Agile Executive.

Written by israelgat

March 4, 2010 at 5:40 am

Using Credit Limits to Constrain “Development on Margin”

with 9 comments

Buying (stocks) on margin is broadly recognized as a risky investment strategy. Funding long-term investments with short-term debt exposes the investor to margin calls as he/she might not be able to secure more financing when needed. The resultant margin call is never pleasant.

The accrual of technical debt in the course of aggressively developing functions and features is quite a similar phenomenon. The CTO is betting the functionality he/she is developing will pay off before the need to “pay back” the technical debt becomes imperative. The temptation to do so is particularly strong due to the lack of credit limits on technical debt. For all practical purposes the CTO is “developing on margin.”

In his comprehensive studies of the economics of software, Capers Jones has actually put a 3-5 year ceiling on the economical viability of developing on margin:

Indeed, the economic value of lagging applications is questionable after about three to five years. The degradation of initial structure and the increasing difficulty of making updates without “bad fixes” tends towards negative returns on investment (ROI) within a few years.

As the CEO leading a company, or the venture capitalist funding it, you can restrain development on margin by establishing credit limits. Use a combination of static code analysis with dynamic program analysis to calculate the amount of accrued technical debt in $$ terms. (An illustration of such calculation as well as a breakdown of the technical debt is given in the Sonar chart above). Set a limit (say $0.25 per line of code) on the amount of permitted technical debt. Once the limit is reached, developers are not allowed to continue developing new functionality – they have to first reduce (and hopefully eliminate) their technical debt.

A very simple “Lacmus test” is available to the CEO/VC until the code is instrumented and the analytics illustrated above generated. Ask your CTO about unit test coverage. If the coverage is low (say <30%), chances are the technical debt is high. Whether the CTO realizes it or not, low unit test coverage is a good indicator of technical debt of all kinds. Moreover, the investment required to develop a full-fledged suite of unit tests is often the largest component of the technical debt to be paid back.

The System Assumes the System is Right

with 2 comments

The post It Won’t Work Here proposed tactics for dealing with a specific class of arguments why a company should not adopt Agile:

  • Uniqueness: “Some very unique elements exist in our company. These elements render industry data inapplicable.”
  • Secret sauce: “Something very special element existed in the companies reporting great success with Agile. Our company does not possess nor have access to the ‘secret sauce’ that enabled success elsewhere.”
  • Cultural change: “For the Agile initiative to succeed, our corporate culture needs to change. The required cultural change takes a lot of time and involves a great deal of pain. All in all, the risk of rolling Agile is unacceptably high.”
  • Affordability: “The company is strapped to the degree that investment in another software method is a luxury it can’t afford.”
  • Software is not core to us: “We are not a software company, nor is software engineering our core competency. Software is merely one of the many elements we use in our business.”

This posts augments It Won’t Work Here by shedding light on the reflexive behavior of the system the Agile champion is likely to run into while applying the recommended tactics.

The fundamental distinction the Agile champion needs to keep in mind is between individuals and organizations. To quote from Charles Perrow‘s work on the subject of complex organizations:

One cannot explain organizations by explaining the attitudes and behavior of individuals or even small groups within them. We learn a great deal about psychology and social psychology but little about organizations per se in this fashion.

To successfully function as an entity, an organization must assume that the system put in place to carry out its tasks is right. If the system is not right, the order and integration required for the proper functioning of the organization can’t be maintained. This assumption about being right tends to become all-inclusive. It is applied to both essential tasks and non-essential trivia.

The key to the success of Agile adoption in the face of the “system is right” reflex, is to budget your battles. As an Agile champion you fight only for things that are core to Agile methods, not for the myriad of contextual details about which the system assumes it is right.

For example, I would not spend a lot of energy on determining the unit of measure to be used in the Agile initiative. As an Agilist I prefer stories as the standard unit, e.g. release 2.0 was 500 stories while releases 3.0 constitutes 1,000 stories. But, I would accept lines of code or functions points as the standard unit of measure if the system so prefers.

Conversely, I would not accept system convictions when they violate core principles of Agile. For example, The Agile Triangle advocated by Jim Highsmith views scope, schedule and cost as constraints. This way of viewing Agile software is non-negotiable in my book. The reason for this strongly held conviction of mine is simple – the Agile initiative is likely to fail if the three (scope, schedule, cost) are considered as goals.

Source: based on Figure 1-3 in Jim Highsmith‘s Agile Project Management: Creating Innovative Products.

Definition: Agile Methodology

leave a comment »

Agile Methodology is actually a bit of a controversial termVarious authors consider Agile a method, as distinct from a methodology. Others, prefer methodology over method. For example, using the Merriam-Webster dictionaries, Alistair Cockburn makes the following distinction between methodology and method in Agile Software Development: The Cooperative Game :

  • Methodology: A series of related methods or techniques
  • Method: Systematic procedure

Alistair views Agile as a methodology in the sense defined above. For example, he discusses Crystal as a family of methodologies. The reader is referred to Alistair’s book for a an excellent analysis of the various aspects of methodologies. As a matter of fact, Alistair tracks down the confusion between method and methodology to certain inconsistencies between various versions of the Oxford English Dictionary.

On the other hand, best I can tell from various conversations with him, Jim Highsmith seems to prefer the term Agile Method. This preference is reflected in Agile Project Management: Creating Innovative Products. It is possible that Jim’s preference is due to writing his book from a project management perspective.

Rather than getting in-depth to the method versus methodology controversy, I would simply cite two definitions I find useful in capturing the essence of Agile methodology, or method if you prefer.

An interesting metaphor for Agile has been used by Jim Highsmith in a 2009 Cutter Advisory:

Visualize a house structure with a roof, a foundation, and three pillars… The roof is business goals — the rationale for implementing agile methods and scaling to larger agile projects. The foundation is agile values or principles — principles that need careful interpretation as to how to apply them to larger teams. And finally, the three pillars: organization, product backlog, and process/practice.

The simplicity of the metaphor makes it quite effective in communicating what Agile is in a concise way without losing the richness of the various elements in Agile.

Using Scrum as an example, colleague David Spann gives the following down-to-earth summary of the key structural components of Agile in a 2008 Cutter Executive Report:

Scrum, as a management methodology, is elegant in its design, requiring only three roles (i.e., product owner, ScrumMaster, and self-organized team), three ceremonies (sprint/iteration planning, daily Scrum/debrief, and sprint review meetings), and three artifacts (product and sprint backlogs and the burndown chart) — just-enough practical advice so agile teams do not overcomplicate the development lifecycle with too much ceremony and documentation.

Needless to say, the structural elements will change from one Agile methodology to another. However, examining an Agile methodology through the {roles, ceremonies, artifacts} “lens” is an excellent way to summarize an Agile methodology. Furthermore, it enables easy comparison between the ‘usual suspects’ of Agile – Crystal Methods, Dynamic Systems Development, Extreme Programming, Feature Driven Development, and Kanban. The reader is referred to The Business Value of Agile Software Methods: Maximizing ROI with Just-in-Time Processes and Documentation for detailed comparisons between the various methods/methodologies.

The Friction of Agile

with 2 comments

Everything is very simple in War, but the simplest thing is difficult. These difficulties accumulate and produce a friction which no man can imagine exactly who has not seen War… This enormous friction, which is not concentrated, as in mechanics, at a few points, is therefore everywhere brought into contact with chance, and thus incidents take place upon which it was impossible to calculate…

IMHO, this excerpt from On War applies exceptionally well to Agile roll-outs these days:

  • Simplicity: The four principles of the Agile Manifesto are intuitively compelling. You could (and probably should) use them as the core definition of what Agile is all about. Likewise, you do not need more than a single hand-drawn matrix to illustrate how WIP limits in Kanban work. In contrast to various other terms used in development and IT – e.g. SOA – the conceptual power of Agile methods is easy to grasp.
  • Friction: Assume you were building a company from scratch without any pre-conceived notions of the organizational constructs you would put in place. Assume as well that you were developing your organization with end-to-end Agile effectiveness in mind. You would probably devise a holistically integrated organization. For example, you might opt for an organizational design in which each level of the organization will include all functions relevant to Agile – R&D, IT, Marketing, Support, Sales etc. In other words, ideally you will not go for the traditional organizational design: a vertical R&D stove pipe, a vertical Marketing stove pipe, a vertical Sales stove pipe, etc.  As in reality you are unlikely to get the charter to start with a clean sheet of paper, the friction arises in each and every point in which your end-to-end organizational design for Agile deviates from the traditional organizational constructs your company uses.
  • Not concentrated: As Clausewitz points out, the friction of war is not mechanical friction – you can’t address it by pouring in a ‘organizational lubricant’ in just a few places. Flooding the whole organization with the lubricant is likely to create a morass in which all agility will be lost.

I recommend four principles to minimize the organizational friction of Agile, as follows:

  • Acknowledge you accrued organizational debt: It is conceptually quite similar to accruing technical debt – various organizational decisions and compromises taken along the way were rushed to the extent that they leave much to be desired. The organizational constructs and practices that sprang out of these decisions need to be refactored.
  • Carry out the organizational refactoring work from the outside to the inside:  A truly holistic Agile design will incorporate customers and partners. Start with the way you will integrate them, thence apply this very same way to the integration of  the organizational entities within your company.
  • Build on the strengths of your core corporate culture: As pointed out by Drucker:

… culture is singularly persistent… changing [organizational] behavior works only if it can be based on the existing ‘culture’… [Drucker, 1991]

Since the end of the Cold War, a fair amount of debate has taken place around the applicability of the friction of war principles to armed conflicts in the information age. The conclusion is of interest to both military personnel, Agile practitioners and IT professionals:

… while technological advances might temporarily mitigate general friction, they could neither eliminate it nor substantially reduce its potential magnitude.

Definition: Agile Development

with 2 comments

The difficulty to concisely define the term Agile Development stems from the very nature of the Agile Manifesto:

  • The manifesto is a statement of values. By the very nature of values, people share them in a loose manner. Both definition and adherence (“But do they really practice Agile development?”) are qualitative and open to interpretation.
  • The manifesto values are relative. The manifesto is quite explicit in stating “… while there is value in the items on the right, we value the items on the left more:”

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiations

Responding to change over following a plan

Agile development is often described in terms of the software method in use. For example, in his foreword to Agile Software Development with Scrum, Bob Martin summarizes Agile methods as “… people oriented software processes that work without getting in the way,” Martin Fowler emphasizes another aspect of Agile methods in his own foreword to the very same book:

… a new breed of software processes which are based on an empirical approach to controlling a project.

A more detailed definition is given by authors Rico, Sayani and Sone in their October 2009 book The Business Value of Agile Software Methods: Maximizing ROI with Just-in-time Processes and Documentation:

Agile methods are contemporary approaches for creating new software based on customer collaboration, teamwork, iterative development, and response to change. Combining communication and interpersonal trust with a flexible management and development framework, they contain just enough process to capture customer needs in the form of user stories and to rapidly create working software. However, the key to Agile methods are rich, high-context communications with customers along with cohesive teamwork.

On the other hand, such an authority (and signatory to the Manifesto) as Jim Highsmith does not seem to define the term Agile Development per se in the second edition of Agile Project Management: Creating Innovative Products. Instead, Jim defines Agility through two statements:

Agility is the ability to both create and respond to change in order to profit in a turbulent business environment.

Agility is the ability to balance flexibility and stability.

Likewise, in Agile Software Development, Alistair Cockburn focuses on discussing what is core to Agile, emphasizing the properties of Agility through the following citation:

Agility is dynamic, context specific, aggressively change-embracing and growth-oriented. It is not about improving efficiency, cutting costs or battening down the business hatches to ride out fearsome competitive ‘storms.’ It is about succeeding and about winning: about succeeding in emerging competitive arenas, and about winning profits, market share and customers in the very center of the competitive storms many companies now fear.

Rather than trying to reconcile all these worthy definitions, I would suggest five context-dependent approaches to the definition, as follows:

  • For the reader who tries to understand what Agile is all about: It is the mindset that really matters. Read the Agile Manifesto and the corresponding History.
  • For the reader who is anxious to put his/her hands around an Agile implementation: Pick a specific Agile method – any method – and study it with special emphasis on the roles, process and artifacts of the method. It could be Crystal, Extreme Programming (XP), Scrum, Dynamic Systems Development Method (DSDM), Kanban or any other method that shows promise as a good fit  for your specific environment. Consult 10 Steps for Starting an Agile Start-up for a down-to-earth blueprint for implementation.
  • For the reader who tries to assess whether a project team is really Agile: It is a maturity curve issue that manifests itself in quite a few disciplines. For example, see the various kinds of maturity models surveyed in the BSM Review blog. You will probably need to determine the maturity model that suits your environment and apply it to the method you are practicing.
  • For the reader who needs to explore Agile in a business context: You need not worry about the technical aspects of Agile. See the category Benefits of Agile in this blog.
  • For the reader interested in applying Agile beyond development: Extending Agile changes its definition. See the various posts on the subject by Eric Ries in Lessons Learned.

Please remember: when it comes to defining Agile Development, you have a problem of choosing, not of choice. It is the use to which you put the definition that determines the choosing.

Prosperity without Growth

leave a comment »

Readers of this blog might recall the ‘secret sauce’ proposed in The Mindset for talking about Agile with executives:

Success, however, depends on a certain kind of mindset of the executive you are talking to.  This mindset is nicely described in H. Thomas Johnson‘s article Manage a Living System, Not a Ledger:

…a business organization cannot improve its long-run financial results by working to improve its financial results. But the only way to ensure satisfactory and stable long-term financial results is to work on improving the system from which those results emerge.

In a perceptive CQI  article on the recently reported problems at Toyota,  Johnson offers the following analysis:

Toyota avoided this fate until the last decade because it did not regard results as outcomes that a business achieves by requiring managers to drive people to meet financial targets. It saw that results emerge from a process in which people carefully nurture a web of relationships. These relationships, strikingly enough, emulate the behaviour in natural living systems.

The reversal of Toyota’s fortunes in the past decade suggests that many of its top managers lost the habit of thought that had previously shaped the company’s policies and actions. They lost the habit of thought that caused the company, perhaps unconsciously, to act like a living system. Toyota adopted the finance-oriented mechanistic thinking that had spawned the inferior management practices and the poor performance shown by most of its competitors after the 1970s. And because it abandoned living-system thinking for mechanistic thinking, Toyota began to embrace a virtual world of finance, not a concrete world of humans in cooperative relationships.

Johnson concludes his analysis with a broad warning:

Efforts of companies to reduce that waste by “going green” are not likely to be any more effective than efforts to improve performance by “going lean”. In neither case do these efforts change the thinking that produces excess growth. The efforts might reduce the rate of growth for a time, but they will never reverse it as long as companies adhere to the conventional wisdom from the virtual world of finance that says prosperity is not possible without growth. [Highlights by IG]

The hazards of the virtual world of finance have been conclusively demonstrated during the macro-economic crisis of 2008-2009. One must wonder what it would take to learn the applicable lessons at the micro level of individual companies.

Wrestling with Scaling Software Agility

with 2 comments

Software Development Times has just published Guest View: Wrestling with Scaling Software Agility by Ryan Martens and me. This Guest View is a little unique in that Ryan and I actually try to wrestle each other to the ground… Here is why we try to do so:

Agile champions spend a lot of time trying to communicate the agile premise to the executives in their organization. The difference in context between the champion and the executive often makes it a difficult conversation. A Scrum Master versed in behavior-driven design is not always able to relate to the frustrations of a sales executive who gets free advice on how to sell from everyone and his grandmother.

Conversely, a CFO does not necessarily understand why unit test coverage on the company’s legacy code is still inadequate after a full year of investment in agile methods that embrace refactoring as a core practice.

To bridge the chasm through this article, we resort to role-playing. Ryan Martens plays the Agile Champion; Israel Gat plays the Skeptical Executive. Metaphorically speaking, each one tries to wrestle the other to the ground.

Before you get into this Guest View, I would like to reinforce an important disclaimer:

A note of caution before Ryan and Israel make irreparable damage to their long-standing relationship: The two actually understand each other extremely well and rarely are they of different opinions on the fundamentals of agile in real life…

Enjoy the article!

Socializing Kanban with Your Executives

leave a comment »

The topic Socializing Agile with Your Executives has been a major thread in this blog. A convenient to browse compilation of posts on the subject can be found in the Starting Agile category. In particular, two recent posts – The Business Value of Agile Software Methods and It Won’t Work Here – are quite specific on the data to use and the line of reasoning to follow in such discussions.

When it comes to socializing Kanban with your executives, you might choose to start the conversation by looking at the defect tracking system your company is using. Chances are your executive will “discover” the all important aspect of flow simply by examining the system with respect to some natural questions such as:

  • Have more defects been closed than opened over the past month?
  • What is the average time to close a reported defect?
  • How many defects have been open (in one stage or another) in the system for more than a year?
  • When a defect moves from one stage in the system to another, how does it get aligned with the various activities that need to take place in the release management system?
  • If development and QA were to stop everything they do and just work exclusively on closing defects that have already been captured, could they clean slate in six months?

The power of this straightforward approach lies in the ease of making the mental jump from defect to Kanban in the context of the tracking system. The breaking down of an epic or a story to granular components that can be pushed to members of the Agile team is not always an easy concept to grasp (and often times a technique teams struggle with in the initial phases of an Agile roll-out). In contrast, one can simply visualize defects entered into the tracking system as inputs to a de-facto Kanban system. Obviously, the defect/Kanban maintains its identity as it “flows” through the system all the way from being reported by a customer to communication of its resolution to the customer. 

If your defect tracking system does not easily lend itself to answering the questions listed above, you might try one of the public domain data sets from Mining Software Archives. The specific data, of course is not likely to be applicable to your company. The criticality of flow, however, could probably be demonstrated by making a few fairly straightforward assumptions on the operating environment behind the data.

Written by israelgat

December 8, 2009 at 5:35 am