Archive for January 2010
Use the Agile Triangle Instead of the Balanced Scorecard
As the name implies, the Balanced Scorecard strives to strike a balance between various performance measures. When Financial, Customer, Business Processes and Learning and Growth measures are presented together, as in Figure 1 below, the Balanced Scorecard allows managers to view the company from several perspectives at once.
Figure 1 – The Balanced Scorecard (source: Trump University)
Likewise, the Agile Triangle depicted in Figure 2, presents in a single “dashboard” the three dimensions critical to Agile performance measurement – Value, Quality and Constraints. Just as in the Balanced Scorecard, it is easy to see imbalances between the three, to respond to them and to restore balance. For example, the tendency to produce more and more lines of code is held in check through the quality metrics.
Figure 2 – The Agile Triangle (based on Figure 1-3 in Jim Highsmith‘s Agile Project Management: Creating Innovative Products.)
My recommendation to clients who do Agile as a strategic initiative is to drop the Balanced Scorecard and use the Agile Triangle instead. There is precious little, if any, to be gained by using the two in parallel. As a matter of fact, one could easily interfere with the other.
The Learning and Growth dimension of the Balanced Scorecard, which does not explicitly show in the Agile Triangle, is, of course, important. As part of an Agile initiative I would expect Agile proficiency to be closely observed. However, I would not include it explicitly in a system based on the Agile Triangle. Agile proficiency is not and end to itself. If the outputs and outcomes we measure through the Agile Triangle are unsatisfactory over a prolonged period of time, a close examination of the way Agile is practiced is called for.
The Urgency of Now – Guest Post by Annie Shum
Failure to learn, failure to anticipate, and failure to adapt are the three generic causes of military disasters. Each one of these three failures is bad enough. In combination, they can be catastrophic. Germany swiftly defeated and conquered France in 1940 due to the utter failure of the French army to grasp the nature of future war, to conceive the probable action of the German forces and to adequately react to the German initiative once it unfolded through the Ardennes. The patterns leading to the catastrophe suffered by the French are similar in some ways to the eco-meltdowns described by Jared Diamond in Collapse: How Societies Choose to Succeed or Fail.
In this guest post, colleague and friend Annie Shum poses disturbing questions with respect to our willingness and ability as IT professionals to learn, anticipate and adapt to the imperatives of Cloud Computing. Between shockingly low (15%) server capacity utilization on the one hand, and dramatic changes in the needs of the business on the other hand, companies who continue to use industrial-era IT models are at peril. Annie weaves theses and other related threads together, and makes a resounding call-to-action to re-think IT.
It is remarkable that Annie’s analysis herein of the root causes of a possible meltdown in IT identifies worrisome patterns similar to those that the Agile movement has pointed out to with respect to arcane methods of software development. The very same core problems that afflict software development manifest themselves in the IT paradigm as well as in the corresponding business design. Painful and wasteful that this repeated manifestation is, it actually creates the opportunity to manage software, IT, and the business in unison. To do so, we need to embrace a data-driven version of the economics of IT, to grasp the true nature of Cloud Computing without the hype that currently surrounds it, and to adapt software development, IT operations and business design accordingly. As the title of this post states, we need to start carrying out these three tasks now.
Here is Annie:
The Urgency of Now: The Edge of Chaos and A “Strategic Inflection Point” for IT
“It was the worst of times. It may be the best of times.” – IBM
Consider the following table. It contains a list of statistics pertaining to the enterprise datacenter index compiled by Peter Mell and Tim Grance, NIST. Overall, the statistics are sobering, perhaps even alarming, and do not bode well for the long-term sustainability of traditional on-premises datacenters. Prudent IT organizations – whether big or small, stalwart or startup – should consider this as a wake-up call. In particular, out of the almost twelve million servers in US datacenters today, the typical server capacity utilization is only around fifteen percent. Although not explicitly shown in this table, the average utilization of the mainframe z/OS servers is typically over eighty percent. However, mainframe z/OS server utilization is only a minor component of the overall average server utilization.
Statistics Enterprise Datacenter Index 11,800,000 Servers in US datacenters 15% Typical server capacity utilization $800,000,000,000/year Purchasing & maintaining enterprise software 80% Software costs spent on maintenance: the “80-20” ratio 100x Power consumption/sq ft compared to office building 4x Increase in server power consumption, 2001 to 2006 2x Increase in number of servers, 2001 to 2006 $21,300,000 Datacenter construction cost, 9000 sq ft $1,000,000/year Annual cost to power the datacenter 1.5% Portion of national power generation 50% Potential power reduction from green technologies 2% Portion of global carbon emissions
Over the years, organizations have accepted such skewed levels of server inefficiency and escalating maintenance costs of IT infrastructure as the norm. Even as organizations continue to express concerns, many seem resigned to the status quo tacitly: akin to what Bob Evans of InfoWeek described as “insurmountable laws of physics.” Looking ahead, however, the status quo may no longer be a viable option for most organizations. Due to soaring electricity/power costs compounded by the recent global financial meltdown with a near collapse of the financial system that triggered a prolonged (and for now, apparently indefinite) credit crunch, these are unparalleled strident and chaotic times for businesses. Pressured by business decision-makers who are under a heightened level of anxiety, enterprise IT is now confronting a transformative dilemma whether to preserve the status quo or to re-think IT.
On one hand, the current global recessionary down cycle is a particularly powerful (albeit rooted in fear) and instinctive deterrent to challenging the status quo. For risk-adverse organizations, it is only understandable why status quo, fundamental flaws notwithstanding, may trump disruptive change during these challenging times. On the other hand, forward-thinking decision-makers may make the bold but disruptive (radical) choice to view status quo as the fundamental problem: acknowledge the growing “urgency of now” by resolving to overcome and correct the entrenched shortcomings of enterprise IT.
“You never want a serious crisis to go to waste”. That quote (or its many variations) has been attributed alike to economists and politicians. The same could be said for IT. Indeed a growing number of IT industry observers believe the profound impact of the on-going economic crisis could offer a rare window of opportunity for organizations to rethink traditional capital-intensive, command-control, on-premises IT operations and invest in new and more flexible self-service IT delivery/deployment models. Think of this defining moment as what Andy Grove, co-founder of Intel, described as the “strategic inflection point”. He was referring to the point in the dynamic when the fundamentals of a business are about to change and “that change can mean an opportunity to rise to new heights.” Nonetheless, the choices will be hard decisions because the options are stark: either counter-intuitively invest in a down cycle by focusing on a more sustainable but disruptive trajectory or hunker down and risk irreversible shrinking business.
As one considers how to address the challenges of today’s enterprise IT, perhaps the following two observations should be taken into account. First, despite the quantum leap in technology advancements, generally the basic design and delivery models of existing IT applications/services are variations of traditionally insular, back-office automation business tools. Second, the organizational structure and business models of most companies are deeply rooted in models of yesteryear, in many instances dating back to the Industrial Revolution. In theory, adhering to the traditional organizational model of top-down command-control can maximize predictability, efficiency and order. Heretofore, this has been the modus operandi for most organizations that Umair Haque succinctly characterized as “ industrial-era companies that make industrial-era stuff — and play by industrial-era rules.” In today’s exponential times, however, the velocity of change and the rapidly growing need of interconnecting to other organizations and automating value chains inevitably lead to an increase in uncertainty and disorder. Strategically, forward-thinking organizations should consider seeking alternative models to address the interdependent and shifting new world order.
In their book, “Presence – Human Purpose and the Field of the Future”, authors Peter Senge, Otto Schramer, Joe Jaworski and Betty Sue Flower observe that many of the practices of the Industrial Age appear to be largely unaffected by the changing reality of today’s society and continue to expand in today’s business organizations. They conclude with this advice: “As long as our thinking is governed by industrial ‘machine age’ metaphors such as control, predictability, and faster is better, we will continue to re-create organizations as we have had – for the last 100 years – despite their increasing disharmony with the world and the science of the 21st century.” Likewise, the traditional top-down command-control modus operandi of enterprise IT today does not reflect adequately and hence likely is unable to accommodate fully the transformational shift of business from silo organizations to “all thing’s digital all the time”, hyper-interconnected and hyper-interdependent ecosystems.
The System Assumes the System is Right
The post It Won’t Work Here proposed tactics for dealing with a specific class of arguments why a company should not adopt Agile:
- Uniqueness: “Some very unique elements exist in our company. These elements render industry data inapplicable.”
- Secret sauce: “Something very special element existed in the companies reporting great success with Agile. Our company does not possess nor have access to the ‘secret sauce’ that enabled success elsewhere.”
- Cultural change: “For the Agile initiative to succeed, our corporate culture needs to change. The required cultural change takes a lot of time and involves a great deal of pain. All in all, the risk of rolling Agile is unacceptably high.”
- Affordability: “The company is strapped to the degree that investment in another software method is a luxury it can’t afford.”
- Software is not core to us: “We are not a software company, nor is software engineering our core competency. Software is merely one of the many elements we use in our business.”
This posts augments It Won’t Work Here by shedding light on the reflexive behavior of the system the Agile champion is likely to run into while applying the recommended tactics.
The fundamental distinction the Agile champion needs to keep in mind is between individuals and organizations. To quote from Charles Perrow‘s work on the subject of complex organizations:
One cannot explain organizations by explaining the attitudes and behavior of individuals or even small groups within them. We learn a great deal about psychology and social psychology but little about organizations per se in this fashion.
To successfully function as an entity, an organization must assume that the system put in place to carry out its tasks is right. If the system is not right, the order and integration required for the proper functioning of the organization can’t be maintained. This assumption about being right tends to become all-inclusive. It is applied to both essential tasks and non-essential trivia.
The key to the success of Agile adoption in the face of the “system is right” reflex, is to budget your battles. As an Agile champion you fight only for things that are core to Agile methods, not for the myriad of contextual details about which the system assumes it is right.
For example, I would not spend a lot of energy on determining the unit of measure to be used in the Agile initiative. As an Agilist I prefer stories as the standard unit, e.g. release 2.0 was 500 stories while releases 3.0 constitutes 1,000 stories. But, I would accept lines of code or functions points as the standard unit of measure if the system so prefers.
Conversely, I would not accept system convictions when they violate core principles of Agile. For example, The Agile Triangle advocated by Jim Highsmith views scope, schedule and cost as constraints. This way of viewing Agile software is non-negotiable in my book. The reason for this strongly held conviction of mine is simple – the Agile initiative is likely to fail if the three (scope, schedule, cost) are considered as goals.
Source: based on Figure 1-3 in Jim Highsmith‘s Agile Project Management: Creating Innovative Products.
The Hole in the Soul and the Legitimacy of Capitalism
In a January 13, 2010 post entitled The Hole in the Soul of Business, Gary Hamel offers the following perspective on success, happiness and business:
I believe that long-lasting success, both personal and corporate, stems from an allegiance to the sublime and the majestic… Viktor Frankl, the Austrian neurologist, held a similar view, which he expressed forcefully in “Man’s Search for Meaning:” “For success, like happiness, cannot be pursued; it must ensue, and it only does so as the unintended consequence of one’s personal dedication to a cause greater than oneself . . ..”
Which brings me back to my worry. Given all this, why is the language of business so sterile, so uninspiring and so relentlessly banal? Is it because business is the province of engineers and economists rather than artists and theologians? Is it because the emphasis on rationality and pragmatism squashes idealism? I’m not sure. But I know this—customers, investors, taxpayers and policymakers believe there’s a hole in the soul of business. The only way for managers to change this fact, and regain the moral high ground, is to embrace what Socrates called the good, the just and the beautiful.So, dear reader, a couple of questions for you: Why do you believe the language of beauty, love, justice and service is so notably absent in the corporate realm? And what would you do to remedy that fact?
Hamel’s call for action is echoed in the analysis of the double bubble at the turn of the century by Carlota Perez:
The current generation of political and business leaders has to face the task of reconstituting finance and bringing the world out of recession. It is crucial that they widen their lens and include in their focus a much greater and loftier task: bringing about the structural shift within nations and in the world economy. Civil society through its many new organisations and communications networks is likely to have a much greater role to play in the outcome on this occasion. Creating favourable conditions for a sustainable global knowledge society is a task waiting to be realized. When – or if – it is done we should no longer measure growth and prosperity by stock market indices but by real GDP, employment and well being, and by the rate of global growth and reduction of poverty (and violence) across and within countries.
As if these words were not arousing enough, Perez adds one final piercing observation:
The legitimacy of capitalism rests upon its capacity to turn individual quest for profit into collective benefit.
IMHO Hamel and Perez identified the very same phenomenon (“hole in the soul”). The only difference is at the level they discuss the phenomenon. Hamel makes his observation at the business/corporate level; Perez at the socio-economic level.
And Now the Bottle-neck is in Operations
In his forthcoming Agile Austin presentation, colleague and friend Michael Cote will be discussing velocity in Agile development vis-a-vis velocity in IT operations. To quote Cote:
Technologies used by public web companies and now cloud computing are looking to offer a new way to deliver applications by addressing deployment and provisioning concerns. Agile software development has sped up the actual development of software, and now the bottle-neck is in operations who’re not always able to deploy software at the same velocity that Agile teams ship code. What do these technologies look like, are they realistic, and how might they affect your organization?
The topic is important from a few perspectives, such as the new business models it enables. With Agile infrastructure, a closed loop is formed between vendor and customer. This loop operates on the basis of close to real-time feedback. The new functionality in the software deployed in the afternoon could be in response to a specific need that was brought up in the morning. Hence, the business focus and the business design change from software that has already been developed and tested (‘done done’) but not yet delivered, to one that has been developed, tested and deployed (‘done done done’) in ultra fast way.
It should also be pointed out that the line between developing content and developing software gets really blurry nowadays. From a company perspective both software and contents are entities that are being made available for dissemination. If you accept the premise that the generation of content and development of the corresponding software should be done under a unified Agile model, the desirability, the power and the benefits of managing development and delivery in unison become obvious. When applied to both content and software, an agile infrastructure paradigm could easily transform the publishing industry, and others.
In short, the business benefits Agile Infrastructure begets trump the (very significant) operational benefits it enables.
Definition: Agile Methodology
Agile Methodology is actually a bit of a controversial term. Various authors consider Agile a method, as distinct from a methodology. Others, prefer methodology over method. For example, using the Merriam-Webster dictionaries, Alistair Cockburn makes the following distinction between methodology and method in Agile Software Development: The Cooperative Game :
- Methodology: A series of related methods or techniques
- Method: Systematic procedure
Alistair views Agile as a methodology in the sense defined above. For example, he discusses Crystal as a family of methodologies. The reader is referred to Alistair’s book for a an excellent analysis of the various aspects of methodologies. As a matter of fact, Alistair tracks down the confusion between method and methodology to certain inconsistencies between various versions of the Oxford English Dictionary.
On the other hand, best I can tell from various conversations with him, Jim Highsmith seems to prefer the term Agile Method. This preference is reflected in Agile Project Management: Creating Innovative Products. It is possible that Jim’s preference is due to writing his book from a project management perspective.
Rather than getting in-depth to the method versus methodology controversy, I would simply cite two definitions I find useful in capturing the essence of Agile methodology, or method if you prefer.
An interesting metaphor for Agile has been used by Jim Highsmith in a 2009 Cutter Advisory:
Visualize a house structure with a roof, a foundation, and three pillars… The roof is business goals — the rationale for implementing agile methods and scaling to larger agile projects. The foundation is agile values or principles — principles that need careful interpretation as to how to apply them to larger teams. And finally, the three pillars: organization, product backlog, and process/practice.
The simplicity of the metaphor makes it quite effective in communicating what Agile is in a concise way without losing the richness of the various elements in Agile.
Using Scrum as an example, colleague David Spann gives the following down-to-earth summary of the key structural components of Agile in a 2008 Cutter Executive Report:
Scrum, as a management methodology, is elegant in its design, requiring only three roles (i.e., product owner, ScrumMaster, and self-organized team), three ceremonies (sprint/iteration planning, daily Scrum/debrief, and sprint review meetings), and three artifacts (product and sprint backlogs and the burndown chart) — just-enough practical advice so agile teams do not overcomplicate the development lifecycle with too much ceremony and documentation.
Needless to say, the structural elements will change from one Agile methodology to another. However, examining an Agile methodology through the {roles, ceremonies, artifacts} “lens” is an excellent way to summarize an Agile methodology. Furthermore, it enables easy comparison between the ‘usual suspects’ of Agile – Crystal Methods, Dynamic Systems Development, Extreme Programming, Feature Driven Development, and Kanban. The reader is referred to The Business Value of Agile Software Methods: Maximizing ROI with Just-in-Time Processes and Documentation for detailed comparisons between the various methods/methodologies.
Three Criteria for Qualifying as Agile
Agile methods have been gaining popularity to the extent that one sees the term Agile used beyond the domain of software methods. Agile Infrastructure and Agile Business Service Management were used in this blog and elsewhere. Recently I have seen the term used in the domain of Business Process Management (BPM). For example, a presentations entitled Best Practices for Agile BPM will be delivered in the forthcoming Gartner Group Business Process Management Summit 2010.
I have no doubt the term Agile will be adopted in various fields. Using BPM as an example, I propose the following three criteria to differentiate between agile (small A) and Agile (capital A):
- Beyond software: A software team carrying out a BPM initiative might use Agile methods. This fact to itself does not suffice to make the initiative Agile BPM.
- Methodical specificity: Roles, forums/ceremonies and artifacts for the BPM initiative must be specified. Folks might be already applying Lean, TOC or other approaches to BPM, but a definitive Agile BPM method has not crystalized yet.
- Values: Adherence in spirit to the four principles of the Agile Manifesto. Replace the word “software” with “product” in the manifesto (just two occurences!) and you get a universal value statement that is not restricted to “just” software. It applies to BPM as well as to any other field in which products are produced and used.
You might be impressively agile in what you do but it does not necessarily make you Agile. The pace by which you do things must be anchored in a broader perspective that incorporates customers and employees. A forthcoming post entitled Indivisibility of the Principles of Operation will explore the connection between the Agile values (plural) you hold and the business value (singular) you generate.
Standish Group Chaos Reports Revisited
The Standish Group “Chaos” reports have been mentioned in various posts in this blog and elsewhere. The following figure from the 2002 study is quite representative of the data provided in the Standish annual surveys of the state of software projects:
The January/February 2010 issue of IEEE Software features an article entitled The Rise and Fall of the Chaos Report Figures. The authors – J. Laurenz Eveleens and Chris Verhoef of the VU University, Amsterdam – give the following summary of their findings:
In 1994, Standish published the Chaos report that showed a shocking 16 percent project success. This and renewed figures by Standish are often used to indicate that project management of application software development is in trouble. However, Standish’s definitions have four major problems. First, they’re misleading because they’re based solely on estimation accuracy of cost, time, and functionality. Second, their estimation accuracy measure is one-sided, leading to unrealistic success rates. Third, steering on their definitions perverts good estimation practice. Fourth, the resulting figures are meaningless because they average numbers with an unknown bias, numbers that are introduced by different underlying estimation processes. The authors of this article applied Standish’s definitions to their own extensive data consisting of 5,457 forecasts of 1,211 real-world projects, totaling hundreds of millions of Euros. The Standish figures didn’t reflect the reality of the case studies at all.
I will leave it to the reader to draw his/her conclusion with respect to the differences between the Standish Group and the authors. I would, however, quote Jim Highsmith‘s deep insight on the value system within its context we measure performance. Following excerpt is from Agile Project Management: Creating Innovative Products:
It we are ultimately to gain the full range of benefits of agile methods, if we are ultimately to grow truly agile, innovative organizations, then, as these stories show, we will have to alter our performance management systems…. We have to be as innovative with our measurement systems as we are with our development methodology.
See pp. 335-358 of Jim’s book for details on transforming performance management systems. His bottom line is elusively simple:
The Standish data are NOT a good indicator of poor software development performance. However, they ARE an indicator of systemic failure of our planning and measurement processes.
Jim is referring to the standard definition of project “success” – on time, on budget, all specified features.
I will be working with a client to carry out the performance management ideas articulated by Jim later this month. Jim indicated he has a customer engagement in February where he expects to learn about interesting ways in which the client is using the Agile Triangle (which is conceptually quite related to the fundamental question what to measure). Client confidentiality permitting, I am confident we will soon be able to brief readers of The Agile Executive on our progress.
Between Agile and ITIL – Part II
The July 2009 post Between Agile and ITIL introduced the application of Agile principles to system management with the following words:
You do not need to be an expert in Value Stream Mapping to appreciate the power of speeding up deployment to match the pace of Agile development. By aligning development with deployment, you streamline “production” with “consumption.” The rationale for so doing is aptly captured in the first bullet of the Declaration of Interdependence: “We increase return on investment by making continuous flow of value our focus.”
Yesterday’s press release about the acquisition of Phurnace by BMC validates the projection given in the afore-listed post. Colleague and friend Michael Cote puts his finger on the heart of the acquisition in his post in People Over Process:
The interesting part is also that this is automation – I’m assuming – at the application layer, where as most automation talk in past and present is at the infrastructure layer. Of course, the thought leaders in this area – folks like Reductive Labs (Puppet),OpsCode (Chef), and in a more general sense cloud management outfits – are doing a helpful job of blurring the distinction between traditional IT layers like application and infrastructure with their dev/ops angled automation. Check out this white paper done by Reductive Labs and dto solutions on the topic for a nice toe-dip. And, I’d expect to see more application layer automation from VMWare/SpringSource. Older automation portfolios like BMC’s Blade Logic line need to keep a close eye on these developments, hopefully, taking in the proven parts of that work.
One can, of course, automate IT tasks without embracing Agile. The fundamental question to be answered is whether one considers ITIL as an “empirical” process control model or as a “defined” process control model (or possibly a hybrid).
The Friction of Agile
Everything is very simple in War, but the simplest thing is difficult. These difficulties accumulate and produce a friction which no man can imagine exactly who has not seen War… This enormous friction, which is not concentrated, as in mechanics, at a few points, is therefore everywhere brought into contact with chance, and thus incidents take place upon which it was impossible to calculate…
IMHO, this excerpt from On War applies exceptionally well to Agile roll-outs these days:
- Simplicity: The four principles of the Agile Manifesto are intuitively compelling. You could (and probably should) use them as the core definition of what Agile is all about. Likewise, you do not need more than a single hand-drawn matrix to illustrate how WIP limits in Kanban work. In contrast to various other terms used in development and IT – e.g. SOA – the conceptual power of Agile methods is easy to grasp.
- Friction: Assume you were building a company from scratch without any pre-conceived notions of the organizational constructs you would put in place. Assume as well that you were developing your organization with end-to-end Agile effectiveness in mind. You would probably devise a holistically integrated organization. For example, you might opt for an organizational design in which each level of the organization will include all functions relevant to Agile – R&D, IT, Marketing, Support, Sales etc. In other words, ideally you will not go for the traditional organizational design: a vertical R&D stove pipe, a vertical Marketing stove pipe, a vertical Sales stove pipe, etc. As in reality you are unlikely to get the charter to start with a clean sheet of paper, the friction arises in each and every point in which your end-to-end organizational design for Agile deviates from the traditional organizational constructs your company uses.
- Not concentrated: As Clausewitz points out, the friction of war is not mechanical friction – you can’t address it by pouring in a ‘organizational lubricant’ in just a few places. Flooding the whole organization with the lubricant is likely to create a morass in which all agility will be lost.
I recommend four principles to minimize the organizational friction of Agile, as follows:
- Acknowledge you accrued organizational debt: It is conceptually quite similar to accruing technical debt – various organizational decisions and compromises taken along the way were rushed to the extent that they leave much to be desired. The organizational constructs and practices that sprang out of these decisions need to be refactored.
- Carry out the organizational refactoring work from the outside to the inside: A truly holistic Agile design will incorporate customers and partners. Start with the way you will integrate them, thence apply this very same way to the integration of the organizational entities within your company.
- Build on the strengths of your core corporate culture: As pointed out by Drucker:
… culture is singularly persistent… changing [organizational] behavior works only if it can be based on the existing ‘culture’… [Drucker, 1991]
- Use modern corporate for the fundamental organizing principles. See Relational Networks, Strategic Advantage: New Challenges for Collaborative Control by Hagel, Brown and Jelinek for a good recent article on the subject.
Since the end of the Cold War, a fair amount of debate has taken place around the applicability of the friction of war principles to armed conflicts in the information age. The conclusion is of interest to both military personnel, Agile practitioners and IT professionals:
… while technological advances might temporarily mitigate general friction, they could neither eliminate it nor substantially reduce its potential magnitude.
The Agile Executive 
