Archive for the ‘Kanban’ Category
Apropos is Going Places
Pictured above is a screen shot from the forthcoming Rally implementation of Apropos – the end-to-end Kanban system unveiled by Erik Huddleston, Stephen Chin, Walter Bodwell and me in the Lean Software and Systems conference last April.
Pictured below is Stephen Chin presenting the forthcoming product in the recent JavaOne conference:
The commercial version by Rally builds on the four pillars of the original implementation of Apropos at Inovis and the subsequent open source version:
- Stakeholder Based Investment Themes
- Business Case Management
- Upstream and Downstream WIP Limits
- Dynamic Allocations
These four pillars enable Apropos users to dynamically adjust their plans as needed in accord with the realities of end-to-end execution. Agile portfolio planning and actual execution truly run alongside each other as depicted in the following figure:
Adjustments to allocations can take place in either in the plan or in execution. Here are two typical examples of stakeholders’ dialogs:
- In planning: “In response to the quick growth of the sales funnel, we decide to increase the % of time allotted to tactical sales opportunities from 35% of the total R&D budget to 40%.”
- In execution: “The introduction of product Pj will be delayed by three months due to lack of qualified professional services resources. During this period, the affected R&D resources will be reassigned to help with multi-tenant aspects of a SaaS version of product Pk.”
Recommendations: Consider using the open source version of Apropos for a small-scale pilot as part of your 2011 planning/budget cycle. If the pilot proves a good fit with your needs, switch over to the commercial version in the 2012 planning/budget cycle.
____________________________________________________________________________________________________
Considering end-to-end Agile/Kanban roll-out? Let me know if you would like assistance in planning and implementing a roll-out which focuses on continuous value delivery. Click Services for details.
____________________________________________________________________________________________________
Schedule Constraints in the Devops Triangle
Last week’s post “The Devops Triangle” demonstrated the extension of Jim Highsmith‘s Agile Triangle to devops. The extension relied on adding compliance to the three traditional constraints of software development: scope, schedule, cost. A graphical representation of this extension is given in Figure 1.
Figure 1: Compliance as the Fourth Constraint in Devops Projects
This blog post examines how time/schedule should be governed in the devops context. It does so by building on the concluding observation in the previous post:
The Devops Triangle and the corresponding Tradeoff Matrix demonstrate how governance a la Agile can be extended to devops projects as far as compliance goes. The proposed governance framework however is incomplete in the following sense: schedule in devops projects can be a much more granular and stringent constraint than schedule in “dev only” projects.
For the schedule constraint in devops, I propose a schedule set. It consists of four components:
- Lead Time or Engineering Time
- Time to change
- Time to deploy
- Time to roll back
Lead Time/Engineering Time: These are customary metrics used in Kanban software development, as demonstrated in Figure 3.
Figure 3: The Engineering Time Metric Used by the BBC (David Joyce in the LSSC10 Conference)
Time to change: The amount of time it takes for the various stakeholders (e.g., dev, test, ops, customer support) to review the code to be deployed, approve its deployment and assign a time window for the deployment.
Time to deploy: The amount of time from (metaphorically speaking) pushing the Deploy “button” to completion of deployment.
Time to roll back: The amount of time to undo a deployment. (Rigorous that the engineering practices and IT processes might be, the time to roll back a deployment can’t be ignored – it is a critical risk parameter).
A graphical representation of these four schedule metrics together with the Devops Triangle is given in the figure below:
Figure 4: The Devops Triangle with a Schedule Set
Using hours as the common unit of measure, a typical schedule set could be {100, 48, 3, 2}. In this hypothetical example, it takes a little over 4 days to carry out the development of the code increment; 2 days to get approval for the change; 3 hours to deploy the code; and, 2 hours to roll back.
Whatever your specific schedule numbers might be, it is highly recommended you apply value stream mapping (see Figure 5 below) to your schedule set. Based on the findings of the value stream mapping, apply statistical process control methods like those illustrated in Figure 3 to continuously improving both the mean and the variances of the four schedule components.
Figure 5: An Example of Value Stream Mapping (Source: Wikipedia entry on the subject)
Using 3σ Control Limits in Software Engineering
Source: Wikipedia; Control Chart
The July/August 2010 issue of IEEE Software features an article entitled “Monitoring Software Quality Evolution for Defects” by Hongyu Zhang and Sunghun Kim. The article is of interest to the software developer/tester/manager in quite a few ways. In particular, the authors report on their successful use of 3σ control limits in c-charts used to plot defects in software projects.
To put things in perspective, consider my recent assessment of the results accomplished by Quick Solutions (QSI) in two of their projects:
One to one-and-a-half standard deviation better than the mean might not seem like much to six-sigma black belts. However, in the context of typical results we see in the software industry the QSI results are outstanding. I have not done the exact math whether those results are superior to 95%, 97% or 98% of software projects in Michael Mah‘s QSMA database as the very exact figure almost does not matter when you achieve this level of excellence.
A complementary perspective is provided by Capers Jones in Estimating Software Costs: Bringing Realism to Estimating:
Another way of looking at six-sigma in a software context would be to achieve a defect-removal efficiency level of about 99.9999 percent. Since the average defect-removal efficiency level in the United States is only about 85 percent, and less than one project in 1000 has ever topped 98 percent, it can be seen that actual six-sigma results are beyond the current state of the art.
The setting of control limits is, of course, quite a different thing from the actual defect-removal efficiency numbers reported by Jones for the US and the very low number of defects reported by Mah for QSI. Having said that, driving a continuous improvement process through using 3σ control limits is the best recipe toward eventually reaching six-sigma results. For example, one could drive the development process by using Cyclomatic complexity per Java class as the quality characteristic in the figure at the top of this post. In this figure, a Cyclomatic complexity reading higher than 10.860 (the Upper Control Limit) will indicate a need to “stop the line” and attend to reducing complexity before resuming work on functions and features.
Coming on the heels of the impressive results reported by David Joyce on the use of statistical process control (SPC) techniques by the BBC, the article by Zhang and Kim is another encouraging report on the successful application of manufacturing techniques to software (and to knowledge work in general). I am not at liberty to quote from this just published IEEE article, but here is the abstract:
Quality control charts, especially c-charts, can help monitor software quality evolution for defects over time. c-charts of the Eclipse and Gnome systems showed that for systems experiencing active maintenance and updates, quality evolution is complicated and dynamic. The authors identify six quality evolution patterns and describe their implications. Quality assurance teams can use c-charts and patterns to monitor quality evolution and prioritize their efforts.
Apropos has been Open Sourced
Erik Huddleston, Walter Bodwell, Stephen Chin and I unveiled Apropos – the Agile Project Portfolio Scheduler – a month ago in the LSSC10 conference in Atlanta, GA. The system is now available as open source. Click here to go to the home page of the project and download the software. It will enable you to:
- Synergies R&D with downstream organizations such as Operations, Professional Services, and Sales
- Increase delivery value through organization-wide alignment of priorities
- Achieve continuous improvement by whole process feedback loops
- Gain realtime visibility into delivery status and potential blockages
The core concept of Apropos – multiple parallel feedback loops – is demonstrated by the following process control diagram:
Figure 1: Process Control View of Apropos
Enjoy Apropos, benefit from it and please give us feedback!
Open-Sourcing the Inovis End-to-End Kanban System
Source: Gat, Huddleson, Bodwell and Chin, “Reformulating the Product Delivery Process“
Colleague and “partner in crime” Stephen Chin has published a post on the Inovis End-to-End Kanban System (aka Apropos) we presented at the LSSC10 conference on April 23. As readers of this blog might recall, the system tracks features through their full life-cycle from proposal to validation, ensuring actionable feedback cycles. By so doing it firmly anchors the software method in the overall business context with special attention to operational aspects such as deployment, monitoring and support.
Stephen outlines details of the forthcoming open-sourcing of Apropos as follows:
The plan for this tool is to do the initial launch of a BSD-licensed open-source version on May 22nd. This will include support for the Rally Community Edition, which is free for up to 10 users. In future releases we plan to support other Agile Lifecycle Management tools, both commercial and open-source, but will need assistance from the community to do this.
If you are interested in helping out with this project, please contact me. I will have limited bandwidth until after the initial launch, but after that would love to scale up this project with interested parties.
I really can’t wait till the 22nd. IMHO Apropos has the potential to become the leading Kanban system by the community for the community.
Apropos – The Inovis End-to-End Kanban System
Figure 1: End-to-end flow slide from Reformulating the Product Delivery Process
Erik Huddleston, Walter Bodwell, Stephen Chin and I delivered a presentation at LSSC10 on the design and implementation of the end-to-end Kanban system Apropos at Inovis. The presentation highlights four key ingredients of the ‘secret sauce’ that makes Apropos so powerful:
- Stakeholder Based Investment Themes
- Business Case Management
- Upstream and Downstream WIP Limits
- Dynamic Allocations
You can read the slides here. A recording of the presentation will soon be posted by InfoQ. A commercial friendly open-source license of the code will be available on May 22, 2010.
Reformulating the Product Delivery Process
Erik Huddleston, Walter Bodwell, Stephen Chin and I will present and demo an end-to-end Kanban system that addresses the #1 challenge modern software methods pose – reformulation of the product delivery process. We will do so the coming Friday, April 23, 10:45AM at the Lean Software and Systems Conference. Here is the abstract for our presentation/demo:
Software methods can be viewed as the glue that holds the product development process together. With Kanban, the glue is melting on both sides of the process. Traditional portfolio management systems and organizations have difficulty coping with the granularity of Kanban. Likewise, today’s product release and delivery systems and the corresponding organizational constructs are ill-equipped to effectively handle the Kanban flow.
We present a field-tested system for implementing Kanban on an end-to-end basis – from product ideation through continuous delivery. This system reformulates the deconstructed product delivery process to strike an optimal balance between planning, development and operations.
Measuring Agile Success Rate the Right Way
Much has been said recently about the success/failure rate of Agile projects. In particular, a debate arose around the success rate of Scrum vis-a-vis Kanban. For example, in a post entitled Some Day Kanban will fail 75% of the Time, colleague Jurgen Appelo states as follows:
Unfortunately, some people arguing against Scrum include these ScrumBut teams in their evaluations of the “high failure rate” of Scrum. They love quoting that “at least 75 percent of Scrum implementations fail.” And I think “Yes of course, 75% fails when that includes the teams that don’t understand what they’re doing.”
I would like to add one other “dimension” to the discussion: boundary conditions.
Any Agile initiative – Crystal, Scrum, Kanban, etc. – typically starts from a certain state of affairs of the code that has already been developed using a Waterfall method or no method at all. Even brand new projects produce code that invariably interacts with other software components that are already deployed, warts and everything. Pristine environments with no technical debt for the Agile initiative to deal with are rare.
Like it or not, the Agile initiative is saddled from the outset with a certain amount of technical debt. Code has been duplicated, rules violated, complexity ran amuck, etc. A typical enterprise software team starts with hundreds of thousands $$ in technical debt, if not millions. This debt needs to be “paid back.” Probably not over night, but certainly over a period of time. As illustrated by the following figure from Jim Highsmith, things get ugly if the debt is not paid back over an extended period of time.
The evaluation of success or failure of the Agile initiative needs to take technical debt into account. A team of 50 with an accrued technical debt of $100,000 has a much easier job on its hands transitioning to Agile than a similar size team starting with $1M in technical debt on its hands.
Whatever criteria you use to determine whether an Agile initiative has been successful, I would suggest the following boundary condition needs to be satisfied:
Technical debt at the end of the project/initiative must be significantly lower than technical debt at the start of the project.
Use the techniques outlined in Using Credit Limits to Constrain Development on Margin to calculate technical debt before and after. In addition to qualifying your Agile success, quantifying technical debt will do a lot towards improving the quality of your software.
The Friction of Agile
Everything is very simple in War, but the simplest thing is difficult. These difficulties accumulate and produce a friction which no man can imagine exactly who has not seen War… This enormous friction, which is not concentrated, as in mechanics, at a few points, is therefore everywhere brought into contact with chance, and thus incidents take place upon which it was impossible to calculate…
IMHO, this excerpt from On War applies exceptionally well to Agile roll-outs these days:
- Simplicity: The four principles of the Agile Manifesto are intuitively compelling. You could (and probably should) use them as the core definition of what Agile is all about. Likewise, you do not need more than a single hand-drawn matrix to illustrate how WIP limits in Kanban work. In contrast to various other terms used in development and IT – e.g. SOA – the conceptual power of Agile methods is easy to grasp.
- Friction: Assume you were building a company from scratch without any pre-conceived notions of the organizational constructs you would put in place. Assume as well that you were developing your organization with end-to-end Agile effectiveness in mind. You would probably devise a holistically integrated organization. For example, you might opt for an organizational design in which each level of the organization will include all functions relevant to Agile – R&D, IT, Marketing, Support, Sales etc. In other words, ideally you will not go for the traditional organizational design: a vertical R&D stove pipe, a vertical Marketing stove pipe, a vertical Sales stove pipe, etc. As in reality you are unlikely to get the charter to start with a clean sheet of paper, the friction arises in each and every point in which your end-to-end organizational design for Agile deviates from the traditional organizational constructs your company uses.
- Not concentrated: As Clausewitz points out, the friction of war is not mechanical friction – you can’t address it by pouring in a ‘organizational lubricant’ in just a few places. Flooding the whole organization with the lubricant is likely to create a morass in which all agility will be lost.
I recommend four principles to minimize the organizational friction of Agile, as follows:
- Acknowledge you accrued organizational debt: It is conceptually quite similar to accruing technical debt – various organizational decisions and compromises taken along the way were rushed to the extent that they leave much to be desired. The organizational constructs and practices that sprang out of these decisions need to be refactored.
- Carry out the organizational refactoring work from the outside to the inside: A truly holistic Agile design will incorporate customers and partners. Start with the way you will integrate them, thence apply this very same way to the integration of the organizational entities within your company.
- Build on the strengths of your core corporate culture: As pointed out by Drucker:
… culture is singularly persistent… changing [organizational] behavior works only if it can be based on the existing ‘culture’… [Drucker, 1991]
- Use modern corporate for the fundamental organizing principles. See Relational Networks, Strategic Advantage: New Challenges for Collaborative Control by Hagel, Brown and Jelinek for a good recent article on the subject.
Since the end of the Cold War, a fair amount of debate has taken place around the applicability of the friction of war principles to armed conflicts in the information age. The conclusion is of interest to both military personnel, Agile practitioners and IT professionals:
… while technological advances might temporarily mitigate general friction, they could neither eliminate it nor substantially reduce its potential magnitude.
The Agile Executive 