Posts Tagged ‘Michael Mah’
This is Not a “Me Too” Product
I had the distinct pleasure to discuss AgileEVM over dinner yesterday with its two “fathers” – Chris Sterling and Brent Barton. I was much impressed by the originality of the concept, the clear differentiation of the product and the laser focus on Earned Value Management (EVM) aspects.
Governance of the software development process is a topic that has been much discussed in various Cutter publications – see for example recent Cutter blog posts by Stephen Andriole, Robert Charette, Jim Highsmith, Vince Kellen, Masa Maeda, Michael Mah and me. Chris and Brent add a layer of EVM – both in terms of theoretical foundations and actual tool – to our spectrum of thoughts on governance. IMHO AgileEVM has the potential to induce a disruption in the way companies go about the business of governing software.
AgileEVM is particularly intriguing as a {product + professional services} combo. Chris and Brent are well-known as awesome Agile consultants. The opportunity to pick their brains twice – once through the product, a second time through a related consulting engagement with them – is most attractive.
The two “pages” below give a sense of the power of the product. Even better, simply download the product through the AgileEVM website.
Grab Chris or Brent for a 1-1 demo if you are at Agile 2010 this week. You will not be disappointed…
Using 3σ Control Limits in Software Engineering
Source: Wikipedia; Control Chart
The July/August 2010 issue of IEEE Software features an article entitled “Monitoring Software Quality Evolution for Defects” by Hongyu Zhang and Sunghun Kim. The article is of interest to the software developer/tester/manager in quite a few ways. In particular, the authors report on their successful use of 3σ control limits in c-charts used to plot defects in software projects.
To put things in perspective, consider my recent assessment of the results accomplished by Quick Solutions (QSI) in two of their projects:
One to one-and-a-half standard deviation better than the mean might not seem like much to six-sigma black belts. However, in the context of typical results we see in the software industry the QSI results are outstanding. I have not done the exact math whether those results are superior to 95%, 97% or 98% of software projects in Michael Mah‘s QSMA database as the very exact figure almost does not matter when you achieve this level of excellence.
A complementary perspective is provided by Capers Jones in Estimating Software Costs: Bringing Realism to Estimating:
Another way of looking at six-sigma in a software context would be to achieve a defect-removal efficiency level of about 99.9999 percent. Since the average defect-removal efficiency level in the United States is only about 85 percent, and less than one project in 1000 has ever topped 98 percent, it can be seen that actual six-sigma results are beyond the current state of the art.
The setting of control limits is, of course, quite a different thing from the actual defect-removal efficiency numbers reported by Jones for the US and the very low number of defects reported by Mah for QSI. Having said that, driving a continuous improvement process through using 3σ control limits is the best recipe toward eventually reaching six-sigma results. For example, one could drive the development process by using Cyclomatic complexity per Java class as the quality characteristic in the figure at the top of this post. In this figure, a Cyclomatic complexity reading higher than 10.860 (the Upper Control Limit) will indicate a need to “stop the line” and attend to reducing complexity before resuming work on functions and features.
Coming on the heels of the impressive results reported by David Joyce on the use of statistical process control (SPC) techniques by the BBC, the article by Zhang and Kim is another encouraging report on the successful application of manufacturing techniques to software (and to knowledge work in general). I am not at liberty to quote from this just published IEEE article, but here is the abstract:
Quality control charts, especially c-charts, can help monitor software quality evolution for defects over time. c-charts of the Eclipse and Gnome systems showed that for systems experiencing active maintenance and updates, quality evolution is complicated and dynamic. The authors identify six quality evolution patterns and describe their implications. Quality assurance teams can use c-charts and patterns to monitor quality evolution and prioritize their efforts.
Beautiful Quality
Figure 1: Agile Assessment – Quality (Source: QSMA)
Colleague and friend Michael Mah has kindly shared with me the figure above – quality assessment for a sample of Agile projects in the QSMA metrics database of more than 8000 software projects. The two red squares in this figure represent the recent results Mah measured on two projects carried out by Quick Solutions (QSI) – a Westerville, OH company offering a broad range IT services.
One to one-and-a-half standard deviation better than the mean might not seem like much to six sigma black belts. However, in the context of typical results we see in the software industry the QSI results are outstanding. I have not done the exact math whether those results are superior to 95%, 97% or 98% of software projects in the QSMA database as the very exact figure almost does not matter when you achieve this level of excellence.
I asked Bart Murphy – QSI’s Vice President of Delivery and Operations – for the ‘secret sauce.’ Here is the QSI recipe:
For the projects referenced in Michael’s evaluation, our primary focus was quality… Our team was tasked with not only a significant Innovation effort, but we also managed all aspects of supporting and stabilizing the application (production support, triage, infrastructure, database support, etc.). Our ‘secret sauce’ was assembling a world class team and executing our Agile methodology. We organized the team efforts to focus on the three major initiatives; Innovation, Stabilization (Technical Debt), and Production Support. We developed a release plan and coordinated efforts to deploy releases that resolved a significant number of defects, introduced market differentiating features, and addressed massive amount of technical debt. The team was able to accomplish this without introducing additional defects into the production system. Our success can be attributed to the commitment from the business to understand our Agile methodology and being highly engaged throughout the project (co-located with the team). In addition, the focus of quality that is integrated into our process through the use of Test Driven Development, Continuous Integration, Test Automation, Quality Assurance, and Show & Tells. Lastly, this would not have been possible without the co-location of the entire team given the significant issues and time constraints for delivery.
Bart also provided me with a table ‘a la Capers Jones’ in which he elaborates on the factors that helped QSI achieve these results and those that stood in the way. I will publish and discuss Bart’s table in a forthcoming post. As part of this post I will also compare the factors identified by Bart with those reported by Capers Jones.
The Concise Executive Guide to Agile
a
a
The IEEE Computer Society Press published today a ReadyNote[1] that I authored entitled The Concise Executive Guide to Agile. It is available through the IEEE Computer Society Store. A Kindle version will be published in June.
I had two main objectives in writing the guide:
- Provide the know-how for approaching Agile in a concise manner that requires minimal investment of time and effort by the reader. The ReadyNote does so by summarizing most Agile topics in a page or two. Detailed coverage of a topic is left for follow-on reading in the selected references that accompany each topic and in the Further Reading appendix.
- Be accessible to any executive — R&D, Marketing, Sales, Program Management, Professional Services, Customer Support, Finance, or IT. The only assumption I make is that the reader has a conceptual grasp of software and software engineering as well as an interest in learning about Agile. No deep knowledge, let alone technical knowledge, about software engineering is required for comprehending the guide.
There is no fluff in the guide. Every paragraph has been written to satisfy the “And therefore what?!” criterion. It is my intent to drive a point home and make it clear to the reader what he/she could do with the information in as few words as possible.
A simple acid test for the guide is your successful assimilation of it in entirety during a flight in the continental US. Something has not quite worked if you need to fly all the way from the US to Europe or vice versa in order to comprehend the guide…
I would like to express my sincere thanks to Michael Cote, Michael Mah, Hubert Smits and to the fourth reviewer (whose identity I don’t know) for their many helpful insights and suggestions. I am also deeply indebted to Linda Shafer and Kate Guillemette of the IEEE Computer Society who got me to write the guide and supported the writing and editing process along the way.
Enjoy reading!
Footnotes:
- “ReadyNotes are short e-books that are tightly focused on specific topics” [IEEE Computer Society Press].
How to Combine Development Productivity Data with Software Quality Metrics
Consider the situation described in Should You Invest in This Software:
- One of your portfolio companies expects to ship 500K lines of code in 6 months.
- The company asks for additional $2M to complete development and bring the product to market.
- Using technical debt quantification techniques you find the technical debt amounts to $1M.
You are not at all comfortable “paying back” the technical debt in addition to funding the requested $2M. You wonder whether you should start afresh instead of trying to complete and fix the code.
Photo credit: @muntz (Flickr)
A good starting point for assessing the fresh start option is Michael Mah‘s studies of software productivity. Based on the QSMA SLIM metrics database of more than 8,000 projects, Michael will probably bracket the productivity per person in a team consisting of product management, development and test at 10-15K lines of code per year. If you use the 15K lines of code per year figure for the purposes of the analysis, 500K lines of code could theoretically be delivered with an investment of about 33.3 (500/15) man years. Assuming average loaded cost of $99,000 per man-year, the software represents a programming effort of $3.3M. Not much is left if you deduct $3M ($2M+1M) from $3.3M…
Five considerations are of paramount importance in evaluating the start afresh option:
- The comparison above ($3.3M versus $3.0M) is timeless. It is a snapshot at a certain point in time which does not take into account the value of time. To factor in the time dimension, the analysis needs to get into value (as distinct from cost) considerations. See the note on Intrinsic Quality v. Extrinsic Quality at the bottom of this post.
- Your “mileage” may vary. For example, best in class teams in large software projects have reported productivity of 20K lines of code per team member per year. As another example, productivity in business applications is very different from productivity in real-time software.
- If you decide to start with a brand new team, remember Napoleon’s quip: “Soldiers have to eat soup together for a long time before they are ready to fight.”
- If you decide to start afresh with the same team plus some enhancements to the headcount, be mindful of ‘Mythical Man-Month‘ effects. Michael Mah’s studies of the BMC BPM projects indicate that such effects might not hold for proficient Agile teams. Hence, you might opt to go Agile if you plan to enhanced the team in an aggressive manner.
- Starting afresh is not an antidote to accruing technical debt (yet again…) over time. But, it gives you the opportunity to aggressively curtail technical debt by applying the techniques described in Using Credit Limits to Constrain Development on Margin. For example, you might run source code analytics every two weeks and go over the results in the bi-weekly demo.
As long as you are mindful of these five aspects (timeless analysis, your mileage may vary, Napoleon’s quip, mythical man-month effects and credit limits on technical debt), combining technical debt figures with productivity data is an effective way to consider the pros and cons of “fix it” versus starting afresh. The combination of the two simplifies a complex investment decision by reducing all considerations to a single common denominator – $$.
Note: This is not a discussion from a value perspective. The software, warts and everything, might (or might not) be valuable to the target customers. The reader is referred to Jim Highsmith‘s analysis of Intrinsic quality versus Extrinsic Quality in Agile Project Management: Creating Innovative Products. See the Cutter Blog post entitled Beyond Scope, Schedule and Cost: Measuring Agile Performance for a short summary of the distinction between the two.
Turning the “Law of Software Physics” Upside Down
Next week, Michael Mah will be presenting new quantitative data on software productivity, cost, time-to-market and quality. Here is an excerpt from the announcement of his talk:
Many companies are adopting Agile practices in an effort to increase project throughput, reduce cost, and improve quality. But are they working? Drawing from industry statistics, Michael answers vital questions about Agile’s effectiveness, which may be turning the “law of software physics” upside down. Until now, there have been predictable relationships among schedule, staffing, and quality; industry data indicates Agile may be changing all this. See productivity findings at 5 Agile companies, and the results for time-to-market, productivity, and quality. Learn the right practices for your environment, including characteristics of successful measurement. See how metrics reveal insights into Agile approaches that are becoming mainstream.
Knowing Michael (and, well, knowing a bit about his latest and greatest findings…) I have every reason to believe Michael will be breaking new grounds in his presentation. Click here and here for details about this exciting presentation.
An Omen in Chicago
Amazing how things come together. A gentleman introduces himself at the conclusion of my breakout session (Socializing Agile with your Executives) in yesterday’s Rally Agile Success Tour (AST) event in Chicago. I am pleasantly surprised to learn he is Cutter Consortium colleague Scott Stribrny. Within a few sentences I discover he was actually the Cutter consultant to Follett Software. As readers of this blog are well aware of, Follett Software was prominently featured in the landmark study of Agile quality, productivity and time-to market by Michael Mah. To put the icing on the cake (so to speak), Rachel Weston – Rally’s Director of Professional Services – uses this very study by Michael Mah in her keynote presentation at the end of the event…
Symphono’s Robert Schmitt started the day with a quote from one of his developers:
I don’t want to deliver just twice a year; I love to deliver!
The power of this kind of craftsman’s pride in his/her software was nicely illustrated by hard numbers Robert cited. For example, on one of their projects, Symphono observed a cost of $12K instead of the $72K they would have expected under traditional software methods.
Playboy’s Mark Row highlighted the intricacies of project managing contents alongside project managing software. In Mark’s experience, contents developers tend to be visually oriented. Writing requirements does not quite cut it for folks of such orientation. As Mark needs to manage software development priorities across all contents initiatives (and many owners), the balance to be struck between the two is quite tricky. The non-formalistic nature of Agile has proven quite effective in bringing things together. As a matter of fact, Mark indicated Playboy’s marketing teams are now doing daily Scrum-like stand-up meeting. The bottom line from the perspective of his executive management is crystal clear:
Night and day since going Agile
Pariveda’s Jim West kept all of us honest with respect to how bad the starting point for Agile often is. According to Jim, they did not start Agile from square zero – they actually started from minus two (-2)…. In spite of this far from optimal starting conditions, Pariveda been successful on two noteworthy accounts:
- Productivity improved by 15-20%
- The managed to satisfy the needs of other processes by incorporating them in their Agile process. For example, SOX work items are represented as story cards in their backlog
Last but not least, ShopLocal’s Brendan Flynn highlighted the progress they made with Agile contracts. They incorporate both user stories and acceptance criteria in the contract. Furthermore, they pay special attention to specifying what is not included in the contract. To paraphrase the French proverb, Shop Local’s experience is that “good accounts make good (customer) relationships.” Remarkably, they achieve good customer relationships through Agile contracts at the scale of 5+ Billion page views annually through just one of their products!
Expressive quips were brought up in the lively Q&A sessions that followed the presentations. Here are a few gems:
Make Agile your flavor [tailoring Agile to the needs of the organization]
Make database decisions [data-driven decisions in Agile]
A cube empire [working environments in the 80’s and 90’s]
Exchange requests, not change requests [Agile contract policy]
In two week the Agile Success Tour “train” will cross the channel to London. (Please, do not enquire now how the train will make its way from Boulder, CO to Paris, France – we are delaying the decision on that leg of the trip to the last responsible minute). I suspect some of the Agile topics to be discussed in London might give a mild heart-burn to UK-based ITIL aficionados. But, how appropriate it is to conclude a year of great Agile success tours with an event in the grand city London!
How Agile Projects Measure Up, and What This Means to You
The Cutter Consortium Executive Report How Agile Projects Measure Up, and What This Means to You by Michael Mah and Mike Lunt is now available to anyone who is interested in software metrics. This is a rigorous data-driven report. Click here and use the promotion code MEASUREUP in the registration forum.
The Agile Executive 
Cutter’s Technical Debt Assessment and Valuation Service
with 3 comments
The Cutter Consortium has announced the availability of the Technical Debt Assessment and Valuation Service. The service combines static code analytics with dynamic program analytics to give the client “x-rays” of the software being examined at any desired granularity – from the whole project portfolio to a single instruction. It breaks down technical debt into the areas of coverage, complexity, duplication, violations and comments. Clients get an aggregate dollar figure for “paying back” debt that they can then plug into their financial models to objectively analyze their critical software assets. Based on these metrics, they can make the best decisions about their ongoing strategy for the software development effort under scrutiny.
This new service is an important addition to the enlightened software governance framework that Jim Highsmith, Michael Mah and I have been thinking about and contributing to for sometime now (see Beyond Scope, Schedule and Cost: Measuring Agile Performance and Quantifying the Start Afresh Option). The heart of both the technical debt service and the enlightened governance framework is captured by the following words from the press release:
By boiling down technical debt to dollars and tying it to cost and value, the service enables a metrics-driven governance framework for the use of five major constituencies, as follows:
It should finally be pointed out that the technical debt assessment service and the governance framework it enables are applicable to any software method. They can be used to:
Forthcoming Cutter Executive Reports, Executive Updates and Email Advisors on the technical debt service are restricted to Cutter clients. As appropriate, I will publish the latest and greatest news on the subject in the Cutter Blog (which is an open forum I highly recommend).
Acknowledgements: I would like to wholeheartedly thank the following colleagues for inspiring, enlightening and supporting me during the preparation of the service:
Written by israelgat
May 5, 2010 at 4:40 am
Posted in Agile Performance Management, Companies, Events, Performance Measurement, Software Costs
Tagged with Anne Mullaney, CEO, Chris Sterling, Cindy Swain, CIO, Comments, Complexity, Coverage, CTO, Cutter Consortium, Duplication, Governance Framework, Industry Norms, Israel Gat, IT Operations, Jennifer Flaxman, Jim Highsmith, John Heintz, Jonathon Golden, Kara Letourneau, Karen Coburn, Ken Collier, Kim Leonard, M&A, Michael Mah, NPV, Paying Back, Software Method, Technical Debt, Valuation, Venture Capitalist, Violation