Posts Tagged ‘Kanban’
“Big Agile”
On January 30, 2012 12:00 pm EST, colleague and friend Hubert Smits and I will be doing a Cutter webinar entitled “Big Agile” is More than Just a Software Method. We will follow on in February with a “Big Agile” issue of the Cutter IT Journal [CITJ] for which I am the guest editor. Coming April we are likely to discuss the topic some more in the Cutter Summit.
The heart of the webinar can be summarized in the following words:
Small is beautiful in software. While big software might not be beautiful, more often than not, it’s in the nature of what needs to be accomplished. This contrast between the beauty of small and the requirements of the big generates systemic tension in many software projects, organizations, and companies. Resolving this conflict is the focus of this webinar.
Discussing the webinar and the follow-on CITJ issue with various folks in the Agile, Lean and Kanban movement(s), I became painfully aware of the very many interpretation folks associate with the the term “Big Agile.” Hence, I would like to say a few words on what it means to me. I am taking the very pragmatic view of a client on the subject. A VP of one department or another is chartered to implement an Agile roll-out at scale. The roll-out might not include all the teams. Nor might it include all functions within the company. The roll-out, however, affects a significant number of folks. Focusing the roll-out at the team level is not sufficient.
The typical VP that I run into under such circumstances is not an expert on software methods (and usually acknowledges it). He/she, however, is smart enough and experienced enough to understand that scale matters. He/she knows or feels intuitively that Big Agile is akin to Big Data. Data is data is data, but when it is Big Data you need to address various aspects that do not manifest themselves in dealing with ordinary amounts of data. Likewise for Agile IMHO.
We plan to make the webinar very interactive. Anne Mullaney will start with a few ‘warm up’ questions to enable us to ‘dig’ into the subject, thence turn it over to questions from the audience. We plan to take the discussion to wherever the participants might want to.
I hope you will join us whether you love, hate or indifferent to the term “Big Agile.” We are expecting a lively discussion…
Beyond Devops
Based on feedback from participants in my Agile 2010 workshop “How We Do Things Around Here In Order To Succeed,” I am planning to offer the workshop as a one-day seminar. A tentative agenda for the seminar is as follows:
- Introduction to Cultural Framework
- Exercise #1: Determine Your Culture
- Exercise #2: Strengths and Weaknesses of Your Culture
- Change Behavior, Not Culture
- When Cultures Meet
- Exercise #3: Conflicts in Devops
- The Agile Flywheel
- Exercise #4: Using Technical Debt as a Boundary Object
- Bringing Individuals and Organizations Together
- Exercise #5: It is About Sharing the Process, Not Just Sharing the Information
- Exercise #6: From success in devops to end-to-end success
Until I publish a full-fledged outline for the seminar, here is the central theme:
Beyond Devops
Inter-departmental flow in a corporate setting is often envisioned as the inner workings of a swiss watch. Wheels turn other wheels in a precise manner. Not only is effectiveness maintained, it is maintained in an efficient manner.
Problem is, many individuals and most departments hold distorted views of the departments they interact with. Reasonable distortions can be mitigated as long as the operational balance between departments is maintained. Once the operational balance is broken the “swiss watch” stops to function as the inter-departmental distortions block any attempt to restore the balance.
The most effective way to get dev and ops on a path of collaboration is for the two departments to jointly construct a boundary object. As dev and ops are joined in the hip through the code, and even more so through its quality, technical debt is well suited to serve as the core of a boundary object around which the two department share meaning while retaining operational autonomy.
Similar boundary objects can be constructed between dev and other departments – customer support, professional services, marketing, sales and finance. When conceived and implemented in a manner that links numerous boundary objects together, Agile success in dev can be extended to both upstream and downstream functions.
A Recipe for Handling Cultural Conflicts in Devops and Beyond
My Agile 2010 workshop “How We Do Things Around Here In Order To Succeed” will weave together four trends that I am witnessing in my practice:
- The ascendance of Agile portfolio management in a world characterized by loosely coupled processes
- Devops dynamics are becoming more and more characteristic of end-to-end Agile/Kanban patterns
- Viral spread of technical debt metrics in software governance
- Increasing use of boundary objects in the enterprise context
The workshop is structured around three case studies/exercises that will take about two-thirds of the allotted time (the morning of August 9). The other third provides the theory and tools to be used in the three workshop exercises and (hopefully) in many future engagements participants in the workshop will carry out. Deep technical knowledge is not required – the workshop targets any Agile practitioner who has conceptual grasp of culture, software development, IT operations and portfolio management.
The #1 takeaway from the presentation is the details you need to know about creation and capture of lasting value through end-to-end Agile initiatives.
Here is the workshop agenda (still subject to some minor tweaking):
- Introduction to Cultural Framework
- Exercise #1: Strengths and Weaknesses of Your Culture
- Change Behavior, Not Culture
- When Organizations Clash
- Exercise #2: Conflicts in Devops
- The Agile Flywheel
- Exercise #3: Using Technical Debt as a Boundary Object in Devops
- Bringing Organizations Together Through Enlightened Governance Loops
I look forward to meeting you in the workshop and learning from your experiences and insights!
Israel
Toward An Event Driven Scrum Process
Recent posts in this blog and elsewhere recommended using technical debt (TD) criteria as an integral part of the build process. One could fail the build under conditions such as:
- TD(current build) > TD(previous build)
- TD per line of code > $1
- TD as % of cost > 50%
- Many others…
If you follow this recommendation while integrating continuously, chances are some fixing might need to be done multiple times a day. This being the case, the relationship between the daily stand-up meeting and the build fail event needs to be examined. If the team anyway re-groups whenever the build fails, what is the value add of the daily stand-up meeting?
Figure 1 below, taken from Agile Software Development with Scrum, examines the feedback loop of Scrum from a process control perspective. It shows the daily stand-up meeting providing the process control function.
Figure 1: Process Control View of Scrum
Figure 2 below shows the feedback loop being driven by the build process instead of by the daily stand-up meeting. From a process control standpoint it is essentially identical to Figure 1. Control through the regularly scheduled daily stand-up meeting has simply been replaced by event-driven control. The team re-groups whenever the build fails.
Figure 2: Build Driven Process Control
Some groups, e.g. the Scrum/Kanban teams under Erik Huddleston and Stephen Chin at Inovis have been practicing Event Driven Scrum for some time now. A lot more data is needed in order to validate the premise that Event Driven Scrum is indeed an improvement over vanilla Scrum. Until the necessary data is collected and examined, two intuitive observations are worth mentioning:
- Event Driven Scrum preserves the nature of Scrum as an empirical process.
- A control unit that is triggered by events (build fail) is in the very spirit of adaptive Agile methods.
Balancing Agile – Guest Post by Alan Shalloway
A fascinating difference exists between Agile and Business Service Management (BSM). Agile emphasizes continuous flow of value to the customer. In contrast, BSM focuses on the business – it aligns the deliverables of IT to the enterprise’s business goals. Subtle that the difference might be, the two methods evolved along quite different lines in spite of the common denominator – dealing with software.
In this guest post, Alan Shalloway – Founder and CEO of NetObjectives – discusses the implications of focusing on the business as distinct from focusing on the customer. His discussion is part of a few thought-provoking threads he weaves around the Agile Manifesto. Alan perceives the Manifesto a product of the times. He thinks aloud whether today’s circumstances require a revised manifesto.
Alan is a man of passion. While I do not always agree with him, I have a lot of respect for his quest to find the deeper truths. Furthermore, I always learn from him. Whether you agree or disagree with the opinion Alan articulates in this post, “listening” to his thoughts is well worth your time.
Here is Alan:
The Agile Manifesto was a watershed event that has forever changed the landscape of software development. So profound a positive impact of it has had, that few challenge whether it was actually correct. Manifestos are often a statement in reaction to something prevalent that needs changing. This makes them very topical and temporal – and the exact intention needs to be restated when the landscape against which it was drawn has changed. The Agile Manifesto[1] states:
“Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a planThat is, while there is value in the items on
the right, we value the items on the left more.”At the time of its edict, this was profound and well stated. Too many software teams were:
- Following a process dictated to them from outside their team
- Managing according to an extensive set of artifacts that recorded where they supposedly were well before software existed
- Were given a set of requirements to meet with little opportunity to discuss real needs (note, points 3 and 4 in the manifesto address this point in two ways – first recognizing that customer collaboration is necessary to discover their true needs and second that it is essential to take advantage of newly discovered information)
The manifesto represented a new paradigm from which to work – one in which the team would have better control over its destiny and where it was recognized that one had to make incremental, iterative movement towards one’s goals – both in discovering the true goal and in implementing it.
Unfortunately, the perspective from which the manifesto was created, or at least the methods which first followed the manifesto, have been extremely team centric. Not a surprise, given the paradigm at the time gave development teams too little say in their own methods. The impact of this has been, not surprisingly, success at the teams and difficult beyond the team. It is almost axiomatic now that companies will have successful team pilots only to bog down in their enterprise agile adoption efforts or even revert back to earlier methods.
I say “not surprisingly” because several things have been left out of the agile manifesto. These are:
- The role of management
- The role of process
- The role of planning
- And indirectly, the role of guiding principles
It could also be said that the driver for agile development is misplaced. I do not believe “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.” This makes software groups customer driven, not business driven.
There is a subtle, but important difference. Basically, conscious or not, the Agile Manifesto is driven with a team-centric view of satisfying customers – business and management play mostly secondary roles. Unfortunately, there is a significant difference between customer driven and business driven (see the figure entitled Alignment with Vision of Business to the right). This is not apparent at the team level (the team is supposed to satisfy the customer). But definitely at the business level. Not surprising, however, since the manifesto is a team perspective it states things in terms of customer value.
Ironically, it is the over-focus on the team, however, that is robbing teams of one of their greatest tools. Clearly any method must have respecting people and providing those doing the work the ability to choose how they do their work. But this does not mean that process isn’t essential or that attending to certain laws of software development is optional. Rather it means that process can’t be imposed on teams since to do so would both rob them of respect and almost certainly be the wrong way to do things – who knows more about how to get work done than the people doing the work themselves?
But “Individuals over process” as the first line has come to be called, makes it sound like people are it. I do not think so – and I think this mindset has caused a lot of damage in many ways.
There is great evidence that the best approach is not merely get smart people onto a team and have them figure out how to solve their problems. They must be properly equipped to do so. Just being smart doesn’t mean you can solve the challenges facing you. This should be readily apparent, but in many ways, the Agile community has mostly ignored it. Actually, there is not really an Agile community any more – there are factions that have significantly different beliefs. For example, XP has long recognized the need for technical practices in Agile while the Scrum community is only just starting to get into what these are.
However, except for the Lean/Kanban community, few Agilists seem to espouse discovering and following the laws of product development flow (or even recognize their existence). This, in my mind, has led to the low rate of success in scaling Scrum to the enterprise*[2]. Ironically, it is the over-focus on people that leads many in the Scrum community to assert this lack of success is a lack willingness to take the effort to improve. This is not surprising – if it’s up to the team to succeed, then when they don’t it must be something wrong with the team or their management when they fail. I think not. I think it is the lack of understanding of the principles of software development flow.
These laws are not new. Don Reinertsen, in his iconic book Managing the Design Factory lays out much of the rules of product development. His more recent book The Principles of Product Development Flow: Second Generation Lean Product Development he lays out 175 of these principles.
To me, true respect for people means that one must equip them with what they need to get their job done. Our thinking in the Agile community should change from “People over process” to one of “People times process.[3]” This phrase emphasizes that if either are low, you get a low productivity. Process does not ensure success. But a poor process requires heroes to succeed. We’d like good, motivated, well-intentioned people to be able to succeed.
Our new agile perspective needs to include an understanding of what teams need to know to do their work. This opens up a role for managers to actively help teams get their job done and to coach them when they have challenges or lose their way. While I will always be thankful for the Agile Manifesto, I am looking for a Business Agile Manifesto that will expand the focus from the team to the entire enterprise.
Footnotes:
[1] http://www.agilemanifesto.org
[2] Ken Schwaber, iconic leader of the Scrum community has said that “I estimate that 75% of those organizations using Scrum will not succeed in getting the benefits that they hope for from it.”
[3] I first heard this from Don Reinertsen. Before that I used to say “People plus process.”
Measuring Agile Success Rate the Right Way
Much has been said recently about the success/failure rate of Agile projects. In particular, a debate arose around the success rate of Scrum vis-a-vis Kanban. For example, in a post entitled Some Day Kanban will fail 75% of the Time, colleague Jurgen Appelo states as follows:
Unfortunately, some people arguing against Scrum include these ScrumBut teams in their evaluations of the “high failure rate” of Scrum. They love quoting that “at least 75 percent of Scrum implementations fail.” And I think “Yes of course, 75% fails when that includes the teams that don’t understand what they’re doing.”
I would like to add one other “dimension” to the discussion: boundary conditions.
Any Agile initiative – Crystal, Scrum, Kanban, etc. – typically starts from a certain state of affairs of the code that has already been developed using a Waterfall method or no method at all. Even brand new projects produce code that invariably interacts with other software components that are already deployed, warts and everything. Pristine environments with no technical debt for the Agile initiative to deal with are rare.
Like it or not, the Agile initiative is saddled from the outset with a certain amount of technical debt. Code has been duplicated, rules violated, complexity ran amuck, etc. A typical enterprise software team starts with hundreds of thousands $$ in technical debt, if not millions. This debt needs to be “paid back.” Probably not over night, but certainly over a period of time. As illustrated by the following figure from Jim Highsmith, things get ugly if the debt is not paid back over an extended period of time.
The evaluation of success or failure of the Agile initiative needs to take technical debt into account. A team of 50 with an accrued technical debt of $100,000 has a much easier job on its hands transitioning to Agile than a similar size team starting with $1M in technical debt on its hands.
Whatever criteria you use to determine whether an Agile initiative has been successful, I would suggest the following boundary condition needs to be satisfied:
Technical debt at the end of the project/initiative must be significantly lower than technical debt at the start of the project.
Use the techniques outlined in Using Credit Limits to Constrain Development on Margin to calculate technical debt before and after. In addition to qualifying your Agile success, quantifying technical debt will do a lot towards improving the quality of your software.
Definition: Agile Methodology
Agile Methodology is actually a bit of a controversial term. Various authors consider Agile a method, as distinct from a methodology. Others, prefer methodology over method. For example, using the Merriam-Webster dictionaries, Alistair Cockburn makes the following distinction between methodology and method in Agile Software Development: The Cooperative Game :
- Methodology: A series of related methods or techniques
- Method: Systematic procedure
Alistair views Agile as a methodology in the sense defined above. For example, he discusses Crystal as a family of methodologies. The reader is referred to Alistair’s book for a an excellent analysis of the various aspects of methodologies. As a matter of fact, Alistair tracks down the confusion between method and methodology to certain inconsistencies between various versions of the Oxford English Dictionary.
On the other hand, best I can tell from various conversations with him, Jim Highsmith seems to prefer the term Agile Method. This preference is reflected in Agile Project Management: Creating Innovative Products. It is possible that Jim’s preference is due to writing his book from a project management perspective.
Rather than getting in-depth to the method versus methodology controversy, I would simply cite two definitions I find useful in capturing the essence of Agile methodology, or method if you prefer.
An interesting metaphor for Agile has been used by Jim Highsmith in a 2009 Cutter Advisory:
Visualize a house structure with a roof, a foundation, and three pillars… The roof is business goals — the rationale for implementing agile methods and scaling to larger agile projects. The foundation is agile values or principles — principles that need careful interpretation as to how to apply them to larger teams. And finally, the three pillars: organization, product backlog, and process/practice.
The simplicity of the metaphor makes it quite effective in communicating what Agile is in a concise way without losing the richness of the various elements in Agile.
Using Scrum as an example, colleague David Spann gives the following down-to-earth summary of the key structural components of Agile in a 2008 Cutter Executive Report:
Scrum, as a management methodology, is elegant in its design, requiring only three roles (i.e., product owner, ScrumMaster, and self-organized team), three ceremonies (sprint/iteration planning, daily Scrum/debrief, and sprint review meetings), and three artifacts (product and sprint backlogs and the burndown chart) — just-enough practical advice so agile teams do not overcomplicate the development lifecycle with too much ceremony and documentation.
Needless to say, the structural elements will change from one Agile methodology to another. However, examining an Agile methodology through the {roles, ceremonies, artifacts} “lens” is an excellent way to summarize an Agile methodology. Furthermore, it enables easy comparison between the ‘usual suspects’ of Agile – Crystal Methods, Dynamic Systems Development, Extreme Programming, Feature Driven Development, and Kanban. The reader is referred to The Business Value of Agile Software Methods: Maximizing ROI with Just-in-Time Processes and Documentation for detailed comparisons between the various methods/methodologies.
The Friction of Agile
Everything is very simple in War, but the simplest thing is difficult. These difficulties accumulate and produce a friction which no man can imagine exactly who has not seen War… This enormous friction, which is not concentrated, as in mechanics, at a few points, is therefore everywhere brought into contact with chance, and thus incidents take place upon which it was impossible to calculate…
IMHO, this excerpt from On War applies exceptionally well to Agile roll-outs these days:
- Simplicity: The four principles of the Agile Manifesto are intuitively compelling. You could (and probably should) use them as the core definition of what Agile is all about. Likewise, you do not need more than a single hand-drawn matrix to illustrate how WIP limits in Kanban work. In contrast to various other terms used in development and IT – e.g. SOA – the conceptual power of Agile methods is easy to grasp.
- Friction: Assume you were building a company from scratch without any pre-conceived notions of the organizational constructs you would put in place. Assume as well that you were developing your organization with end-to-end Agile effectiveness in mind. You would probably devise a holistically integrated organization. For example, you might opt for an organizational design in which each level of the organization will include all functions relevant to Agile – R&D, IT, Marketing, Support, Sales etc. In other words, ideally you will not go for the traditional organizational design: a vertical R&D stove pipe, a vertical Marketing stove pipe, a vertical Sales stove pipe, etc. As in reality you are unlikely to get the charter to start with a clean sheet of paper, the friction arises in each and every point in which your end-to-end organizational design for Agile deviates from the traditional organizational constructs your company uses.
- Not concentrated: As Clausewitz points out, the friction of war is not mechanical friction – you can’t address it by pouring in a ‘organizational lubricant’ in just a few places. Flooding the whole organization with the lubricant is likely to create a morass in which all agility will be lost.
I recommend four principles to minimize the organizational friction of Agile, as follows:
- Acknowledge you accrued organizational debt: It is conceptually quite similar to accruing technical debt – various organizational decisions and compromises taken along the way were rushed to the extent that they leave much to be desired. The organizational constructs and practices that sprang out of these decisions need to be refactored.
- Carry out the organizational refactoring work from the outside to the inside: A truly holistic Agile design will incorporate customers and partners. Start with the way you will integrate them, thence apply this very same way to the integration of the organizational entities within your company.
- Build on the strengths of your core corporate culture: As pointed out by Drucker:
… culture is singularly persistent… changing [organizational] behavior works only if it can be based on the existing ‘culture’… [Drucker, 1991]
- Use modern corporate for the fundamental organizing principles. See Relational Networks, Strategic Advantage: New Challenges for Collaborative Control by Hagel, Brown and Jelinek for a good recent article on the subject.
Since the end of the Cold War, a fair amount of debate has taken place around the applicability of the friction of war principles to armed conflicts in the information age. The conclusion is of interest to both military personnel, Agile practitioners and IT professionals:
… while technological advances might temporarily mitigate general friction, they could neither eliminate it nor substantially reduce its potential magnitude.
Definition: Agile Development
The difficulty to concisely define the term Agile Development stems from the very nature of the Agile Manifesto:
- The manifesto is a statement of values. By the very nature of values, people share them in a loose manner. Both definition and adherence (“But do they really practice Agile development?”) are qualitative and open to interpretation.
- The manifesto values are relative. The manifesto is quite explicit in stating “… while there is value in the items on the right, we value the items on the left more:”
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiations
Responding to change over following a plan
Agile development is often described in terms of the software method in use. For example, in his foreword to Agile Software Development with Scrum, Bob Martin summarizes Agile methods as “… people oriented software processes that work without getting in the way,” Martin Fowler emphasizes another aspect of Agile methods in his own foreword to the very same book:
… a new breed of software processes which are based on an empirical approach to controlling a project.
A more detailed definition is given by authors Rico, Sayani and Sone in their October 2009 book The Business Value of Agile Software Methods: Maximizing ROI with Just-in-time Processes and Documentation:
Agile methods are contemporary approaches for creating new software based on customer collaboration, teamwork, iterative development, and response to change. Combining communication and interpersonal trust with a flexible management and development framework, they contain just enough process to capture customer needs in the form of user stories and to rapidly create working software. However, the key to Agile methods are rich, high-context communications with customers along with cohesive teamwork.
On the other hand, such an authority (and signatory to the Manifesto) as Jim Highsmith does not seem to define the term Agile Development per se in the second edition of Agile Project Management: Creating Innovative Products. Instead, Jim defines Agility through two statements:
Agility is the ability to both create and respond to change in order to profit in a turbulent business environment.
Agility is the ability to balance flexibility and stability.
Likewise, in Agile Software Development, Alistair Cockburn focuses on discussing what is core to Agile, emphasizing the properties of Agility through the following citation:
Agility is dynamic, context specific, aggressively change-embracing and growth-oriented. It is not about improving efficiency, cutting costs or battening down the business hatches to ride out fearsome competitive ‘storms.’ It is about succeeding and about winning: about succeeding in emerging competitive arenas, and about winning profits, market share and customers in the very center of the competitive storms many companies now fear.
Rather than trying to reconcile all these worthy definitions, I would suggest five context-dependent approaches to the definition, as follows:
- For the reader who tries to understand what Agile is all about: It is the mindset that really matters. Read the Agile Manifesto and the corresponding History.
- For the reader who is anxious to put his/her hands around an Agile implementation: Pick a specific Agile method – any method – and study it with special emphasis on the roles, process and artifacts of the method. It could be Crystal, Extreme Programming (XP), Scrum, Dynamic Systems Development Method (DSDM), Kanban or any other method that shows promise as a good fit for your specific environment. Consult 10 Steps for Starting an Agile Start-up for a down-to-earth blueprint for implementation.
- For the reader who tries to assess whether a project team is really Agile: It is a maturity curve issue that manifests itself in quite a few disciplines. For example, see the various kinds of maturity models surveyed in the BSM Review blog. You will probably need to determine the maturity model that suits your environment and apply it to the method you are practicing.
- For the reader who needs to explore Agile in a business context: You need not worry about the technical aspects of Agile. See the category Benefits of Agile in this blog.
- For the reader interested in applying Agile beyond development: Extending Agile changes its definition. See the various posts on the subject by Eric Ries in Lessons Learned.
Please remember: when it comes to defining Agile Development, you have a problem of choosing, not of choice. It is the use to which you put the definition that determines the choosing.
Predicting the Year Ahead
Cutter Consortium has published predictions for 2010 by about a dozen of its experts. My own prediction, which examines the crash of 1929, the burst of the “dot-com bubble” in 2000 and the financial collapse in 2008, is actually quite bullish:
I expect 2010 to be the first year of a prolonged golden age. Serious as the various problems we all are wrestling with after the 2008-2009 macro-economic crisis are, they should be viewed as systemic to the way a new generation of revolutionary infrastructure gets assimilated in economy and society.
In addition to the techno-economic view expressed in the Cutter prediction, here are my Agile themes for 2010:
- Agile moves “downstream” into Release Management.
- Agile breaks out of Development into IT (and beyond) in the form of Agile Infrastructure and Agile Business Service Management.
- SOA and Agile start to be linked in enterprise architecture and software/hardware/SaaS organizations.
- Kanban starts an early adoption cycle similar to Scrum in 2006.
Acknowledgements: I am thankful to my colleagues Walter Bodwell, Sebastian Hassinger, Erik Huddleston, Michael Cote and Annie Shum who influenced my thinking during 2009 and contributed either directly and indirectly to the themes listed above.
The Agile Executive 