Archive for June 2010
Devops: It is Not About ITIL, It is About Proficiency
As you would expect, the Information Technology Infrastructure Library (ITIL) topic was brought up in the devops day held last Friday in a LinkedIn facility in Mountain View, CA. We, of course, had the expected spectrum of opinions about ITIL in the context of devops – from “ITIL will never work for a true continuous development shop” to “well, you can make ITIL work under such circumstances.” Needless to say, a noticeable level of passion accompanied these two statements…
IMHO the heart of the issue is not ITIL per se but system management proficiency. If your system management proficiency is high, you are likely to be able to effectively respond to 10, 20 or 50 deploys per day. Conversely, if your system management proficiency is low, ops is not likely to be able to cope with high velocity in dev. The critical piece is alignment of velocities between dev and ops, not the method used to manage IT systems and services. Whether you use ITIL, COBIT or your own home-grown set of best practices is irrelevant. Achieving alignment of velocities between dev and ops is a matter of proficiency in system management.
Velocity 2010 – Where Have All The Business Executives Gone?
By now I have “touched” and been touched by dozens and dozens of participants in the O’Reilly Velocity conference. I did not meet any business executive (other than various CEOs/CMOs who pitched their companies from the podium).
No doubt, the O’Reilly Velocity conferences (this is the third one) are geek events. Having said that, IMHO these conferences are extremely important to the business executive. If you don’t attend you miss up on four value propositions:
- Getting a sense of what the future in web operation holds.
- Grasping what forthcoming advances in web operations mean to your business design. See for example my post Ops Driven Dev from earlier today.
- Understanding the needs of a growing demographic sector that your business might not be able to access today.
- Getting to know the kind of developers and sysadmins that probably work in the trenches of your company.
You owe it to yourself to consider attending Velocity 2011 if terms like “sysadmin 2.0”, “darkmode” and “devops” and do not resonate with you. I can’t give you “your money back” guarantee if you are not satisfied with the conference, but I will gladly pick the bar tab when we meet there.
Ops Driven Dev
In The Agile Flywheel, colleague Ray Riescher describes how velocity in dev drove corresponding velocity in ops:
Scrum set the flywheel in motion and caused the rest of the IT process life cycle to respond. ITIL’s processes still form the solid core of service support and we’ve improved the processes’ capability to handle intense work velocity. The organization adapted by developing unprecedented speed in the ability to deliver production fixes and to solve root cause problems with agility.
From what I gleaned yesterday in the O’Reilly Velocity conference I believe the tables are turning. Ops, or at least web ops, will soon drive development.
The reason for my saying so is quite simple: the breadth and depth of forthcoming web analytics unveiled in the conference. This is not “just” about Google making website performance part of their ranking algorithm. Everything related to web performance will soon be analyzed mercilessly under the “make the web faster” mantra. Dev will need to respond to analytics from operations with an unprecedented speed. For most practical purposes analytics run in ops will dictate the speed for dev.
The phenomenon actually goes beyond performance aspects. To be able to implement changes quickly, dev will need to be very good in ensuring the quality of fast changes. While quality has many dimensions to it, the most applicable one is test coverage. There is no way to change the code quickly without a comprehensive automated test suite.
The first step toward dev meeting the required speed is described in the post How to Initiate a Devops Project:
For a devops project, start by establishing the technical debt of the software to be released to operations. By so doing you build the foundations for collaboration between development and operations through shared data. In the devops context, the technical debt data form the basis for the creation and grooming of a unified backlog which includes various user stories from operations.
I would actually go one step further and suggest including technical debt criteria in the release process. The code is not accepted unless the technical debt per line of code is below a certain pre-set level such as $2. The criteria, of course, can be refined to include specific criteria for the various components of technical debt such as coverage, complexity or duplication. For example, unit test coverage in excess of 70% could be established as a technical debt criterion.
Once such release criteria are established, the metaphorical flywheel starts turning in an opposite direction to that described in The Agile Flywheel. With technical debt criteria embedded in the release process, the most straightforward way for dev to meet these criteria is to use the very same criteria as integral part of the build process. The scheme for so doing in given in the following chart:
One last recommendation: don’t wait till Velocity 2011 to start on the path described above. Velocity 2010 already provides plenty of actionable insights to warrant starting now. Just take a look at the web site.
Using 3σ Control Limits in Software Engineering
Source: Wikipedia; Control Chart
The July/August 2010 issue of IEEE Software features an article entitled “Monitoring Software Quality Evolution for Defects” by Hongyu Zhang and Sunghun Kim. The article is of interest to the software developer/tester/manager in quite a few ways. In particular, the authors report on their successful use of 3σ control limits in c-charts used to plot defects in software projects.
To put things in perspective, consider my recent assessment of the results accomplished by Quick Solutions (QSI) in two of their projects:
One to one-and-a-half standard deviation better than the mean might not seem like much to six-sigma black belts. However, in the context of typical results we see in the software industry the QSI results are outstanding. I have not done the exact math whether those results are superior to 95%, 97% or 98% of software projects in Michael Mah‘s QSMA database as the very exact figure almost does not matter when you achieve this level of excellence.
A complementary perspective is provided by Capers Jones in Estimating Software Costs: Bringing Realism to Estimating:
Another way of looking at six-sigma in a software context would be to achieve a defect-removal efficiency level of about 99.9999 percent. Since the average defect-removal efficiency level in the United States is only about 85 percent, and less than one project in 1000 has ever topped 98 percent, it can be seen that actual six-sigma results are beyond the current state of the art.
The setting of control limits is, of course, quite a different thing from the actual defect-removal efficiency numbers reported by Jones for the US and the very low number of defects reported by Mah for QSI. Having said that, driving a continuous improvement process through using 3σ control limits is the best recipe toward eventually reaching six-sigma results. For example, one could drive the development process by using Cyclomatic complexity per Java class as the quality characteristic in the figure at the top of this post. In this figure, a Cyclomatic complexity reading higher than 10.860 (the Upper Control Limit) will indicate a need to “stop the line” and attend to reducing complexity before resuming work on functions and features.
Coming on the heels of the impressive results reported by David Joyce on the use of statistical process control (SPC) techniques by the BBC, the article by Zhang and Kim is another encouraging report on the successful application of manufacturing techniques to software (and to knowledge work in general). I am not at liberty to quote from this just published IEEE article, but here is the abstract:
Quality control charts, especially c-charts, can help monitor software quality evolution for defects over time. c-charts of the Eclipse and Gnome systems showed that for systems experiencing active maintenance and updates, quality evolution is complicated and dynamic. The authors identify six quality evolution patterns and describe their implications. Quality assurance teams can use c-charts and patterns to monitor quality evolution and prioritize their efforts.
It is the Passion, Stupid!
If you harbored some doubts about having good time in Salt Lake City, please take a look at the picture above: Colleague Alex Pukinskis and I “caught” in a minute of pure joy during the Agile Roots conference. I don’t recall the subject we were discussing, but the picture speaks for itself even without a subject.
The relevance of the picture to Agile goes beyond “and here is a picture from the Agile Roots conference.” I actually believe that there is something special in Agile conferences of small to medium scale. These conferences are all about the experience. The program, the presenters and the venue are, of course, important. But, more than anything else these conferences are about savoring the community experience. The value is in the experience.
Over the past year I presented in four small to medium Agile conferences:
My sense is that in each of these conferences the passion of a small group of organizers was the key to success. This passion proved infectious – it affected presenters, participants and volunteers to form an experience that can only be expressed in the words “I wish I were there.”
While I meet many passionate folks in larger Agile/Software conferences, the experience is different. My guess is that it is a matter of “density of passion.” A few passionate organizers suffice to “ignite” a small/medium conference. This core passion does not seem to scale beyond a certain number of participants.
Beautiful Quality
Figure 1: Agile Assessment – Quality (Source: QSMA)
Colleague and friend Michael Mah has kindly shared with me the figure above – quality assessment for a sample of Agile projects in the QSMA metrics database of more than 8000 software projects. The two red squares in this figure represent the recent results Mah measured on two projects carried out by Quick Solutions (QSI) – a Westerville, OH company offering a broad range IT services.
One to one-and-a-half standard deviation better than the mean might not seem like much to six sigma black belts. However, in the context of typical results we see in the software industry the QSI results are outstanding. I have not done the exact math whether those results are superior to 95%, 97% or 98% of software projects in the QSMA database as the very exact figure almost does not matter when you achieve this level of excellence.
I asked Bart Murphy – QSI’s Vice President of Delivery and Operations – for the ‘secret sauce.’ Here is the QSI recipe:
For the projects referenced in Michael’s evaluation, our primary focus was quality… Our team was tasked with not only a significant Innovation effort, but we also managed all aspects of supporting and stabilizing the application (production support, triage, infrastructure, database support, etc.). Our ‘secret sauce’ was assembling a world class team and executing our Agile methodology. We organized the team efforts to focus on the three major initiatives; Innovation, Stabilization (Technical Debt), and Production Support. We developed a release plan and coordinated efforts to deploy releases that resolved a significant number of defects, introduced market differentiating features, and addressed massive amount of technical debt. The team was able to accomplish this without introducing additional defects into the production system. Our success can be attributed to the commitment from the business to understand our Agile methodology and being highly engaged throughout the project (co-located with the team). In addition, the focus of quality that is integrated into our process through the use of Test Driven Development, Continuous Integration, Test Automation, Quality Assurance, and Show & Tells. Lastly, this would not have been possible without the co-location of the entire team given the significant issues and time constraints for delivery.
Bart also provided me with a table ‘a la Capers Jones’ in which he elaborates on the factors that helped QSI achieve these results and those that stood in the way. I will publish and discuss Bart’s table in a forthcoming post. As part of this post I will also compare the factors identified by Bart with those reported by Capers Jones.
Technical Debt at Cutter
No, this post is not about technical debt we identified in the software systems used by the Cutter Consortium to drive numerous publications, events and engagements. Rather, it is about various activities carried out at Cutter to enhance the state of the art and make the know-how available to a broad spectrum of IT professionals who can use technical debt engagements to pursue technical and business opportunities.
The recently announced Cutter Technical Debt Assessment and Valuation service is quite unique IMHO:
- It is rooted in Agile principles and theory but applicable to any software method.
- It combines the passion, empowerment and collaboration of Agile with the rigor of quantified performance measures, process control techniques and strategic portfolio management.
- It is focused on enlightened governance through three simple metrics: net present value, cost and technical debt.
Here are some details on our current technical debt activities:
- John Heintz joined the Cutter Consortium and will be devoting a significant part of his time to technical debt work. I was privileged and honored to collaborate with colleagues Ken Collier, Jonathon Golden and Chris Sterling in various technical debt engagements. I can’t wait to work with them, John and other Cutter consultants on forthcoming engagements.
- John and I will be jointly presenting on the subject Toxic Code in the Agile Roots conference next week. In this presentation we will demonstrate how the hard lesson learned during the sub-prime loans crisis apply to software development. For example, we will be discussing development on margin…
- My Executive Report entitled Revolution in Software: Using Technical Debt Techniques to Govern the Software Development Process will be sent to Cutter clients in the late June/early July time-frame. I don’t think I had ever worked so hard on a paper. The best part is it was labor of love….
- The main exercise in my Agile 2010 workshop How We Do Things Around Here in Order to Succeed is about applying Agile governance through technical debt techniques across organizations and cultures. Expect a lot of fun in this exercise no matter what your corporate culture might be – Control, Competence, Cultivation or Collaboration.
- John and I will be doing a Cutter webinar on Reining in Technical Debt on Thursday, August 19 at 12 noon EDT. Click here for details.
- A Cutter IT Journal (CITJ) on the subject of technical debt will be published in the September-October time-frame. I am the guest editor for this issue of the CITJ. We have nine great contributors who will examine technical debt from just about every possible perspective. I doubt that we have the ‘real estate’ for additional contributions, but do drop me a note if you have intriguing ideas about technical debt. I will do my best to incorporate your thoughts with proper attribution in my editorial preamble for this issue of the CITJ.
- Jim Highsmith and I will jointly deliver a seminar entitled Technical Debt Assessment: The Science of Software Development Governance in the forthcoming Cutter Summit. This is really a wonderful ‘closing of the loop’ for me: my interest in technical debt was triggered by Jim’s presentation How to Be an Agile Leader in the Agile 2006 conference.
Standing back to reflect on where we are with respect to technical debt at Cutter, I see a lot of things coming nicely together: Agile, technical debt, governance, risk management, devops, etc. I am not certain where the confluence of all these threads, and possibly others, might lead us. However, I already enjoy the adrenaline rush this confluence evokes in me…
Apropos has been Open Sourced
Erik Huddleston, Walter Bodwell, Stephen Chin and I unveiled Apropos – the Agile Project Portfolio Scheduler – a month ago in the LSSC10 conference in Atlanta, GA. The system is now available as open source. Click here to go to the home page of the project and download the software. It will enable you to:
- Synergies R&D with downstream organizations such as Operations, Professional Services, and Sales
- Increase delivery value through organization-wide alignment of priorities
- Achieve continuous improvement by whole process feedback loops
- Gain realtime visibility into delivery status and potential blockages
The core concept of Apropos – multiple parallel feedback loops – is demonstrated by the following process control diagram:
Figure 1: Process Control View of Apropos
Enjoy Apropos, benefit from it and please give us feedback!
The Agile Executive 