The Agile Executive

Making Agile Work

Archive for the ‘Technical Debt’ Category

Boundary Objects in DevOps

with one comment

Boundary Object by Cherice.

Source: Flickr; Chrice‘s Photostream

The following recommendation was given in the post How to Initiate a Devops Project:

For a DevOps project, start by establishing the technical debt of the software to be released to operations. By so doing you build the foundations for collaboration between development and operations through shared data. In the devops context, the technical debt data form the basis for the creation and grooming of a unified backlog which includes various user stories from operations.

I would like to augment this recommendation with a suggestion with respect to the mindset during the initiation phase. Chances are the IT folks feel outnumbered by the dev folks. It might or might not be a matter of optics, but recognizing and appreciating this mindset is will help a lot in getting the devops project on track.

Here is a simple example I heard from a participant in the June 25 devops day in Mountain View, CA. The participant with whom I talked is an IT ops person who tries to get ops aligned with  fairly proficient Agile development teams. She is, however, constrained with respect to the IT ops resources available to her. She simply does not have the resources required to attend each and every Scrum meeting as 25 such meetings take place every day. She strongly feels “outnumbered.”

Various schemes could be devised to enable meaningful participation of ops in the Agile process. The more important thing though is to be fully sensitized to the “outnumbered” feeling. The extension of Agile principles to ops will not succeed at the face of such a feeling.

Discussing the subject with my friend Andrew Shafer, he mentioned the effectiveness of boundary objects in such cross-organizational situations:

Boundary objects are objects which are both plastic enough to adapt to local needs and constraints of the several parties employing them, yet robust enough to maintain a common identity across sites. They are weakly structured in common use, and become strongly structured in individual-site use. They may be abstract or concrete. They have different meanings in different social worlds but their structure is common enough to more than one world to make them recognizable means of translation. The creation and management of boundary objects is key in developing and maintaining coherence across intersecting social worlds. [Source: Wikipedia].

As an example, the boundary object for the situation described in this post could be a set of technical debt criteria that make the code eligible for deployment from a product life cycle perspective. By so doing, it shifts the dialog from the process to the outcome of the process. Instead of working on generating IT resources in an “outnumbered” mode, the energy shifts toward developing a working agreement on the intrinsic quality of the code to be deployed.

Some technical debt criteria that could form the core of a devops boundary object are mentioned in the post Technical Debt Meets Continuous Deployment. Corresponding criteria could be used in the boundary object to satisfy operational requirements which are critical to the proper functioning of the code. For example, a ceiling on configuration drift in IT could be established to ensure an adequate operating environment for the code. A boundary object that contains both technical debt criteria and configuration drift criteria satisfies different concerns – those of dev and those of ops – simultaneously.

Written by israelgat

July 6, 2010 at 6:44 am

Technical Debt Meets Continuous Deployment

with 11 comments

As you would expect in a conference entitled velocity, and in a follow-on devops day, speeding up things was an overarching theme. In the context of devops, the theme primarily manifested itself in lively discussions about the number of deploys per day. Comments such as the following reply to my post Ops Driven Dev were typical:

Conceptually, I move the whole business application configuration into the source code…

The theme that was missing for me in many of the presentations and discussions on the subject was the striking of a balance between velocity and quality. The classical trade-off in process control is between production rate and product quality (and safety, but that aspect [safety] is beyond the scope of this post). IMHO this trade-off applies to software just as it applies to mechanical or chemical processes.

The heart of the “deploy early and often” strategy hailed by advocates of continuous deployment is known deployment state to known deployment state. You don’t let the deployment evolve from one state to another before it has stabilized to a robust state. The power of this incremental deployment is in dealing with single-piece (or as small number of pieces as possible) flow rather than dealing with the effects of multiple-piece flow. When the deployment increments are small enough, rollback, root cause analysis and recovery are relatively straightforward if a deployment turns sour. It is a similar concept to Agile development, extending continuous integration to continuous deployment.

While I am wholeheartedly behind this devops strategy, I believe it needs to be reinforced through rigorous quality criteria the code must satisfy prior to deployment. The most straightforward way for so doing is through embedding technical debt criteria in the release/deploy process. For example:

  • The code will not be deployed unless the overall technical debt per line of code is lower than $2.
  • To qualify for deployment, code duplication levels must be kept under 8%.
  • Code whose Cyclomatic complexity per Java class is higher than 15 will not be accepted for deployment.
  • 50% unit test coverage is the minimal level required for deployment.
  • Many others…

I have no doubt whatsoever that code which does not satisfy these criteria might be successfully deployed in a short-term manner. The problem, however, is the accumulative effect over the long haul of successive deployments of code increments of inadequate quality. As Figure 1 demonstrates, a Java file with Cyclomatic complexity of 38 has a probability of 50% to be error-prone. If you do not stop it prior to deployment through technical debt criteria, it is likely to affect your customers and play havoc with your deployment quite a few times in the future. The fact that it did not do so during the first hour of deployment does not guarantee that such a  file will be “well-behaved” in the future.

mccabegraph.jpg

Figure 1: Error-proneness as a Function of Cyclomatic Complexity (Source: http://www.enerjy.com/blog/?p=198)

To attain satisfactory long-term quality and stability, you need both the right process and the right code. Continuous deployment is the “right process” if you have developed the deployment infrastructure to support it. The “right code” in this context is code whose technical debt levels are quantified and governed prior to deployment.

Ops Driven Dev

with 6 comments

In The Agile Flywheel, colleague Ray Riescher describes how velocity in dev drove corresponding velocity in ops:

Scrum set the flywheel in motion and caused the rest of the IT process life cycle to respond.  ITIL’s processes still form the solid core of service support and we’ve improved the processes’ capability to handle intense work velocity. The organization adapted by developing unprecedented speed in the ability to deliver production fixes and to solve root cause problems with agility.

From what I gleaned yesterday in the O’Reilly Velocity conference I believe the tables are turning. Ops, or at least web ops, will soon drive development.

The reason for my saying so is quite simple: the breadth and depth of forthcoming web analytics unveiled in the conference. This is not “just” about Google making website performance part of their ranking algorithm. Everything related to web performance will soon be analyzed mercilessly under the “make the web faster” mantra. Dev will need to respond to analytics from operations with an unprecedented speed. For most practical purposes analytics run in ops will dictate the speed for dev.

The phenomenon actually goes beyond performance aspects. To be able to implement changes quickly, dev will need to be very good in ensuring the quality of fast changes. While quality has many dimensions to it, the most applicable one is test coverage. There is no way to change the code quickly without a comprehensive automated test suite.

The first step toward dev meeting the required speed is described in the post How to Initiate a Devops Project:

For a devops project, start by establishing the technical debt of the software to be released to operations. By so doing you build the foundations for collaboration between development and operations through shared data. In the devops context, the technical debt data form the basis for the creation and grooming of  a unified backlog which includes various user stories from operations.

I would actually go one step further and suggest including technical debt criteria in the release process. The code is not accepted unless the technical debt per line of code is below a certain pre-set level such as $2. The criteria, of course, can be refined to include specific criteria for the various components of technical debt such as coverage, complexity or duplication. For example, unit test coverage in excess of 70% could be established as a technical debt criterion.

Once such release criteria are established, the metaphorical flywheel starts turning in an opposite direction to that described in The Agile Flywheel. With technical debt criteria embedded in the release process, the most straightforward way for dev to meet these criteria is to use the very same criteria as integral part of the build process. The scheme for so doing in given in the following chart:

One last recommendation: don’t wait till Velocity 2011 to start on the path described above. Velocity 2010 already provides plenty of actionable insights to warrant starting now. Just take a look at the web site.

Written by israelgat

June 24, 2010 at 7:55 am

Beautiful Quality

leave a comment »

Figure 1: Agile Assessment – Quality (Source: QSMA)

Colleague and friend Michael Mah has kindly shared with me the figure above – quality assessment for a sample of Agile projects in the QSMA metrics database of more than 8000 software projects. The two red squares in this figure represent the recent results Mah measured on two projects carried out by Quick Solutions (QSI) – a Westerville, OH company offering a broad range IT services.

One to one-and-a-half standard deviation better than the mean might not seem like much to six sigma black belts. However, in the context of typical results we see in the software industry the QSI results are outstanding.  I have not done the exact math whether those results are superior to 95%, 97% or 98% of software projects in the QSMA database as the very exact figure almost does not matter when you achieve this level of excellence.

I asked Bart Murphy – QSI’s Vice President of Delivery and Operations – for the ‘secret sauce.’ Here is the QSI recipe:

For the projects referenced in Michael’s evaluation, our primary focus was quality…  Our team was tasked with not only a significant Innovation effort, but we also managed all aspects of supporting and stabilizing the application (production support, triage, infrastructure, database support, etc.).  Our ‘secret sauce’ was assembling a world class team and executing our Agile methodology.  We organized the team efforts to focus on the three major initiatives; Innovation, Stabilization (Technical Debt), and Production Support.  We developed a release plan and coordinated efforts to deploy releases that resolved a significant number of defects, introduced market differentiating features, and addressed massive amount of technical debt.  The team was able to accomplish this without introducing additional defects into the production system.   Our success can be attributed to the commitment from the business to understand our Agile methodology and being highly engaged throughout the project (co-located with the team).  In addition, the focus of quality that is integrated into our process through the use of Test Driven Development, Continuous Integration, Test Automation, Quality Assurance, and Show & Tells.  Lastly, this would not have been possible without the co-location of the entire team given the significant issues and time constraints for delivery.

Bart also provided me with a table  ‘a la Capers Jones’ in which he elaborates on the factors that helped QSI achieve these results and those that stood in the way. I will publish and discuss Bart’s table in a forthcoming post. As part of this post I will also compare the factors identified by Bart with those reported by Capers Jones.

Technical Debt at Cutter

leave a comment »

No, this post is not about technical debt we identified in the software systems used by the Cutter Consortium to drive numerous publications, events and engagements. Rather, it is about various activities carried out at Cutter to enhance the state of the art and make the know-how available to a broad spectrum of IT professionals who can use technical debt engagements to pursue technical and business opportunities.

The recently announced Cutter Technical Debt Assessment and Valuation service is quite unique IMHO:

  1. It is rooted in Agile principles and theory but applicable to any software method.
  2. It combines the passion, empowerment and collaboration of Agile with the rigor of quantified performance measures, process control techniques and strategic portfolio management.
  3. It is focused on enlightened governance through three simple metrics: net present value, cost and technical debt.

Here are some details on our current technical debt activities:

  1. John Heintz joined the Cutter Consortium and will be devoting a significant part of his time to technical debt work. I was privileged and honored to collaborate with colleagues Ken Collier, Jonathon Golden and Chris Sterling in various technical debt engagements. I can’t wait to work with them, John and other Cutter consultants on forthcoming engagements.
  2. John and I will be jointly presenting on the subject Toxic Code in the Agile Roots conference next week. In this presentation we will demonstrate how the hard lesson learned during the sub-prime loans crisis apply to software development. For example, we will be discussing development on margin…
  3. My Executive Report entitled Revolution in Software: Using Technical Debt Techniques to Govern the Software Development Process will be sent to Cutter clients in the late June/early July time-frame. I don’t think I had ever worked so hard on a paper. The best part is it was labor of love….
  4. The main exercise in my Agile 2010 workshop How We Do Things Around Here in Order to Succeed is about applying Agile governance through technical debt techniques across organizations and cultures. Expect a lot of fun in this exercise no matter what your corporate culture might be – Control, Competence, Cultivation or Collaboration.
  5. John and I will be doing a Cutter webinar on Reining in Technical Debt on Thursday, August 19 at 12 noon EDT. Click here for details.
  6. A Cutter IT Journal (CITJ) on the subject of technical debt will be published in the September-October time-frame. I am the guest editor for this issue of the CITJ. We have nine great contributors who will examine technical debt from just about every possible perspective. I doubt that we have the ‘real estate’ for additional contributions, but do drop me a note if you have intriguing ideas about technical debt. I will do my best to incorporate your thoughts with proper attribution in my editorial preamble for this issue of the CITJ.
  7. Jim Highsmith and I will jointly deliver a seminar entitled Technical Debt Assessment: The Science of Software Development Governance in the forthcoming Cutter Summit. This is really a wonderful ‘closing of the loop’ for me: my interest in technical debt was triggered by Jim’s presentation How to Be an Agile Leader in the Agile 2006 conference.

Standing back to reflect on where we are with respect to technical debt at Cutter, I see a lot of things coming nicely together: Agile, technical debt, governance, risk management, devops, etc. I am not certain where the confluence of all these threads, and possibly others, might lead us. However, I already enjoy the adrenaline rush this confluence evokes in me…