Archive for April 2010
Figure 1: End-to-end flow slide from Reformulating the Product Delivery Process
Erik Huddleston, Walter Bodwell, Stephen Chin and I delivered a presentation at LSSC10 on the design and implementation of the end-to-end Kanban system Apropos at Inovis. The presentation highlights four key ingredients of the ‘secret sauce’ that makes Apropos so powerful:
- Stakeholder Based Investment Themes
- Business Case Management
- Upstream and Downstream WIP Limits
- Dynamic Allocations
Erik Huddleston, Walter Bodwell, Stephen Chin and I will present and demo an end-to-end Kanban system that addresses the #1 challenge modern software methods pose – reformulation of the product delivery process. We will do so the coming Friday, April 23, 10:45AM at the Lean Software and Systems Conference. Here is the abstract for our presentation/demo:
Software methods can be viewed as the glue that holds the product development process together. With Kanban, the glue is melting on both sides of the process. Traditional portfolio management systems and organizations have difficulty coping with the granularity of Kanban. Likewise, today’s product release and delivery systems and the corresponding organizational constructs are ill-equipped to effectively handle the Kanban flow.
We present a field-tested system for implementing Kanban on an end-to-end basis – from product ideation through continuous delivery. This system reformulates the deconstructed product delivery process to strike an optimal balance between planning, development and operations.
A fascinating difference exists between Agile and Business Service Management (BSM). Agile emphasizes continuous flow of value to the customer. In contrast, BSM focuses on the business – it aligns the deliverables of IT to the enterprise’s business goals. Subtle that the difference might be, the two methods evolved along quite different lines in spite of the common denominator – dealing with software.
In this guest post, Alan Shalloway – Founder and CEO of NetObjectives – discusses the implications of focusing on the business as distinct from focusing on the customer. His discussion is part of a few thought-provoking threads he weaves around the Agile Manifesto. Alan perceives the Manifesto a product of the times. He thinks aloud whether today’s circumstances require a revised manifesto.
Alan is a man of passion. While I do not always agree with him, I have a lot of respect for his quest to find the deeper truths. Furthermore, I always learn from him. Whether you agree or disagree with the opinion Alan articulates in this post, “listening” to his thoughts is well worth your time.
Here is Alan:
The Agile Manifesto was a watershed event that has forever changed the landscape of software development. So profound a positive impact of it has had, that few challenge whether it was actually correct. Manifestos are often a statement in reaction to something prevalent that needs changing. This makes them very topical and temporal – and the exact intention needs to be restated when the landscape against which it was drawn has changed. The Agile Manifesto states:
“Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on
the right, we value the items on the left more.”
At the time of its edict, this was profound and well stated. Too many software teams were:
- Following a process dictated to them from outside their team
- Managing according to an extensive set of artifacts that recorded where they supposedly were well before software existed
- Were given a set of requirements to meet with little opportunity to discuss real needs (note, points 3 and 4 in the manifesto address this point in two ways – first recognizing that customer collaboration is necessary to discover their true needs and second that it is essential to take advantage of newly discovered information)
The manifesto represented a new paradigm from which to work – one in which the team would have better control over its destiny and where it was recognized that one had to make incremental, iterative movement towards one’s goals – both in discovering the true goal and in implementing it.
Unfortunately, the perspective from which the manifesto was created, or at least the methods which first followed the manifesto, have been extremely team centric. Not a surprise, given the paradigm at the time gave development teams too little say in their own methods. The impact of this has been, not surprisingly, success at the teams and difficult beyond the team. It is almost axiomatic now that companies will have successful team pilots only to bog down in their enterprise agile adoption efforts or even revert back to earlier methods.
I say “not surprisingly” because several things have been left out of the agile manifesto. These are:
- The role of management
- The role of process
- The role of planning
- And indirectly, the role of guiding principles
It could also be said that the driver for agile development is misplaced. I do not believe “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.” This makes software groups customer driven, not business driven.
There is a subtle, but important difference. Basically, conscious or not, the Agile Manifesto is driven with a team-centric view of satisfying customers – business and management play mostly secondary roles. Unfortunately, there is a significant difference between customer driven and business driven (see the figure entitled Alignment with Vision of Business to the right). This is not apparent at the team level (the team is supposed to satisfy the customer). But definitely at the business level. Not surprising, however, since the manifesto is a team perspective it states things in terms of customer value.
Ironically, it is the over-focus on the team, however, that is robbing teams of one of their greatest tools. Clearly any method must have respecting people and providing those doing the work the ability to choose how they do their work. But this does not mean that process isn’t essential or that attending to certain laws of software development is optional. Rather it means that process can’t be imposed on teams since to do so would both rob them of respect and almost certainly be the wrong way to do things – who knows more about how to get work done than the people doing the work themselves?
But “Individuals over process” as the first line has come to be called, makes it sound like people are it. I do not think so – and I think this mindset has caused a lot of damage in many ways.
There is great evidence that the best approach is not merely get smart people onto a team and have them figure out how to solve their problems. They must be properly equipped to do so. Just being smart doesn’t mean you can solve the challenges facing you. This should be readily apparent, but in many ways, the Agile community has mostly ignored it. Actually, there is not really an Agile community any more – there are factions that have significantly different beliefs. For example, XP has long recognized the need for technical practices in Agile while the Scrum community is only just starting to get into what these are.
However, except for the Lean/Kanban community, few Agilists seem to espouse discovering and following the laws of product development flow (or even recognize their existence). This, in my mind, has led to the low rate of success in scaling Scrum to the enterprise*. Ironically, it is the over-focus on people that leads many in the Scrum community to assert this lack of success is a lack willingness to take the effort to improve. This is not surprising – if it’s up to the team to succeed, then when they don’t it must be something wrong with the team or their management when they fail. I think not. I think it is the lack of understanding of the principles of software development flow.
These laws are not new. Don Reinertsen, in his iconic book Managing the Design Factory lays out much of the rules of product development. His more recent book The Principles of Product Development Flow: Second Generation Lean Product Development he lays out 175 of these principles.
To me, true respect for people means that one must equip them with what they need to get their job done. Our thinking in the Agile community should change from “People over process” to one of “People times process.” This phrase emphasizes that if either are low, you get a low productivity. Process does not ensure success. But a poor process requires heroes to succeed. We’d like good, motivated, well-intentioned people to be able to succeed.
Our new agile perspective needs to include an understanding of what teams need to know to do their work. This opens up a role for managers to actively help teams get their job done and to coach them when they have challenges or lose their way. While I will always be thankful for the Agile Manifesto, I am looking for a Business Agile Manifesto that will expand the focus from the team to the entire enterprise.
 Ken Schwaber, iconic leader of the Scrum community has said that “I estimate that 75% of those organizations using Scrum will not succeed in getting the benefits that they hope for from it.”
 I first heard this from Don Reinertsen. Before that I used to say “People plus process.”
“[The 100% loan-to-value subprime loan is] the most dangerous product in existence and there can be nothing more toxic…”
This quip by Angelo Mozilo, founder of Countrywide Financial, led me to coin the term “toxic code.” The code stops being an asset when it accumulates technical debt equal to the value it is expected to generate. As matter of fact, the code could become a liability when the cost to “pay back” the technical debt exceeds the revenues the code would generate. Such situations have been known to happen more often than we might realize. They occur, for example, when numerous customers develop elaborate business processes around poor quality enterprise software.
I will be exploring the topic in my May 27, 2010 presentation at The Path to Agility conference in Columbus, OH. Here is the abstract of my talk:
Technical debt had originally been conceived as an expediency measure – “a little debt speeds development so long as it is paid back promptly with a rewrite.” However, like financial debt, unrestrained borrowing can lead to a broad spectrum of difficulties, from collapsed roadmaps to inability to respond to customer problems in a timely manner, and anything in between.
Recent advances in source code analysis enable us to quantify technical debt and express it in $$ terms. By so doing, the software development process can be governed with unprecedented effectiveness. It is possible to constrain the “development on margin” mal-practice and avoid the toxic code phenomenon: technical-debt-to-value ratio of 100%. Moreover, even toxic code can ultimately be “marked-to-market” by reducing/eliminating technical debt.
The combination of Agile refactoring practices with technical debt analytics brings rigor to the software development process through:
- Providing quantifiable data about the overall state of the code and its value
- Highlighting error-prone modules in the code and offering guidance to fixing them in a biggest-bang for-the-buck manner
- Pinpointing technical issues all the way down to the individual line of code level
- Balancing the technical debt work stream vis-a-vis other work streams
In the course of managing software development in this manner, your team will improve its design, coding, testing and project management skills. Ultimately, these improvements are the best antidote against accrual of technical debt in the future.
In the podcasts around here and Israel’s posts we often talk about using cloud computing as a tool to deliver better software – indeed, to better the software delivery process. Those two angle on cloud computing are what I tackled in a recent presentation (given as a keynote at the Philly Emerging Technologies for the Enterprise conference last week). As I said in the talk Agile development thinking is a great start for figuring out how to take advantage of cloud computing.
The good folks at the IEEE Computer Society are just about ready to publish my e-book The Concise Executive Guide to Agile. It will be available for purchase in the Computer Society store in May. A Kindle version will follow in June.
Here is the preface from the guide:
Preface: Connecting the Agile Dots
“The closer one listens to it, the more distantly one hears it.” These insightful words about Schubert’s Unfinished Symphony apply equally well to the art of Agile software methods. The nuts and bolts of Agile methods are not likely to be very relevant to the executive. Instead, he or she needs to stand back and focus on the mindset, values, and principles that make Agile methods so powerful, and on harnessing their power to create business value.
It is the objective of this guide to provide the know-how for approaching Agile in a concise manner that requires minimal investment of time and effort by the reader. It does so by summarizing most Agile topics in a page or two with minimal use of geek jargon. Detailed coverage of a topic is left for follow-on reading in the selected references that accompany each topic and in the Further Reading appendix.
The guide targets executives as the primary audience. It gives them the principles they need to comprehend and apply in order to become effective with an Agile initiative. These executives are not necessarily software engineering experts. They come from any function that Agile affects — R&D, Marketing, Sales, Program Management, Professional Services, Customer Support, Finance, or IT. Nor are the executives restricted to companies whose business is software; they are as likely to reside in companies that embed software in their products or utilize software in implementing business initiatives. Nowadays one can hardly think of a company that would not be included in at least one of these three categories.
Four broad topics are covered in The Concise Executive Guide to Agile: rationale for Agile, implementing it, fitting it into your company, and scaling it to the enterprise level. The rationale explains why Agile is so appropriate for our time, summarizes the state of the art in Agile, and sets realistic expectations with respect to its business value. Implementing addresses critical “real life” issues such as risk assessment and mitigation, off-shoring and outsourcing, governance, and sustainability of the Agile initiative. Fitting connects Agile to the hard realities of introducing a new software method into an environment in which various processes already exist — multiple software methods as well as planning and budgeting processes. Scaling is primarily about the numerous benefits to be attained through end-to-end implementation, such as enabling new business designs that fully utilize the power of Agile.
In the course of covering these four topics, the guide puts special emphasis on the operational, financial, and business benefits of Agile methods. The overarching message is clear and simple: Agile is the most productive technology your business is not using.
 Giuseppe Sinopoli. “Dream and Memory in Schubert’s ‘Unfinished.’” Program Notes to his recording of the symphony with the Philharmonia Orchestra, Deutsche Grammophon 410 862-2, 1984.
Photo credit: Dancing Lemur (Flickr)
Cunningham’s quip “A little debt speeds development so long as it is paid back promptly with a rewrite” is intuitively very clear. We are talking about short-term debt which will be reduced, and hopefully eliminated in entirety, at the earliest possible time.
The question this post addresses is what happens when the expected short-term technical debt becomes a significant long-term debt? Specifically, can technical debt under some conditions constitute a breach of implied warranties?
In his InformIT article Don’t “Enron” Your Software Project, Aaron Erickson coined the term “Technical Fraud” and connected it to Lemmon Laws:
As a reaction to seeing this condition and its deleterious effects, I coined the term technical fraud to refer to the practice of incurring unmanaged and hidden technical debt. Many U.S. states have “lemon laws” that make it illegal to knowingly sell someone a car that has undisclosed maintenance problems. Selling a “lemon” is a fraudulent practice in the world of cars, and it should be considered as such in the world of software.
It is a little tricky (though not impossible – see Using Credit limits to Constrain Development on Margin) to define the precise point where technical debt becomes “unmanaged.” One needs to walk a fine line between technical/methodical incompetence and resource availability to determine technical fraud. For example, if your code has 35% coverage, is it or is not unmanaged? Does the answer to this question change if your cyclomatic complexity per class exceeds 30? I would think the courts might be divided for a very long time on the question when does hidden technical debt represent a fraudulent misrepresentation.
One component of technical debt deserves special attention in the context of this post. I am referring to the conscious decision not to do unit testing at all.
Best I understand it, the rationale for not “bothering” with unit testing is a variant of the old ploy “we do not have time for testing here.” It is a resource allocation strategy that bets on the code being miraculously bug-free. Some amount of functional testing is done out of necessity – the code in customers hands needs to function as proclaimed. But, the pieces of code from which functionality is constructed are not subject to direct rigorous testing. The individual units of code will be indirectly exercised in some manner through functional testing, but not in a systemic manner to verify and validate correctness of the units of code per se.
Such a conscious decision IMHO indicates no intention to pay back this category of technical debt – unit test coverage. It is therefore quite incompatible with the nature of an implied warranty:
An implied warranty is as an unstated promise, assumed by the law in most sales transactions, that the product will be of at least average quality and will do what the average customer would expect it to do [The Reader’s Digest Legal Questions & Answers Book]
To #1 defense open to a software vendor who gets sued over lack of unit testing is that a fair average quality of software can be attained without any unit testing. As a programmer, I would think such defense would fly at the teeth of the availability since 1987 of the IEEE Standard for Software Unit Testing.
It is fascinating to note the duality between contracts and programming. For the programmer who follows the tenets of design by contract, “a unit test provides a strict, written contract that the piece of code must satisfy…”
Disclaimer: I am not an expert in the law. The opinion expressed in this post merely represents my layman’s understanding of principles of contract law that might be applicable to technical debt situations.