The Agile Executive

Making Agile Work

Archive for the ‘devops’ Category

Technical Debt Assessment, Sterling Barton LLC and the Moussaka

with one comment

A few month ago Chris Sterling and I were carrying out a Cutter Technical Debt Assessment and Valuation engagement for a venture capitalist who was considering a certain company. We discovered various things in the code of this company. More noteworthy, my deep domain expertise led to Chris discovering the great Greek dish Moussaka.

I have eaten a lot of good Moussakas over the years. Even against this solid gastronomic background I can’t forget how the eyes of Chris lit up when he took the first bite. It took him only a tiny little time to get on his iPhone and tweet on the culinary aspects of our engagement. I then knew it was going to be a very successful engagement…

The relationship with Chris deepened since this episode. For example, in collaboration with Brent Barton Chris contributed a great article to the forthcoming issue of the Cutter IT Journal on Technical Debt. In this article Chris and Brent  demonstrate how technical debt techniques can be applied at the portfolio level. They make the reader step into the shoes of the project portfolio planner and walk him through their approach to enhancing the decision-making process by using the software debt dashboard.

Chris has just published an excellent post entitled “Using Sonar Metrics to Assess Promotion of Builds to Downstream Environments” in Getting Agile and was kind enough to suggest I cross-post it in The Agile Executive. Here it is (please note that the examples given below by Chris have nothing to do with the engagement described above):

“For those of you that don’t already know about Sonar you are missing an important tool in your quality assessment arsenal. Sonar is an open source tool that is a foundational platform to manage your software’s quality. The image below shows one of the main dashboard views that teams can use to get insights into their software’s health.

The dashboard provides rollup metrics out of the box for:

  • Duplication (probably the biggest Design Debt in many software projects)
  • Code coverage (amount of code touched by automated unit tests)
  • Rules compliance (identifies potential issues in the code such as security concerns)
  • Code complexity (an indicator of how easy the software will adapt to meet new needs)
  • Size of codebase (lines of code [LOC])

Before going into how to use these metrics to assess whether to promote builds to downstream environments, I want to preface the conversation with the following note:

Code analysis metrics should NOT be used to assess teams and are most useful when considering how they trend over time

Now that we have this important note out-of-the-way and, of course, nobody will ever use these metrics for “evil”, lets discuss pulling data from Sonar to automate assessments of builds for promotion to downstream environments. For those that are unfamiliar with automated promotion, here is a simple, happy example:

A development team makes some changes to the automated tests and implementation code on an application and checks their changes into source control. A continuous integration server finds out that source control artifacts have changed since the last time it ran a build cycle and updates its local artifacts to incorporate the most recent changes. The continuous integration server then runs the build by compiling, executing automated tests, running Sonar code analysis, and deploying the successful deployment artifact to a waiting environment usually called something like “DEV”. Once deployed, a set of automated acceptance tests are executed against the DEV environment to validate that basic aspects of the application are still working from a user perspective. Sometime after all of the acceptance tests pass successfully (this could be twice a day or some other timeline that works for those using downstream environments), the continuous integration server promotes the build from the DEV environment to a TEST environment. Once deployed, the application might be running alongside other dependent or sibling applications and integration tests are run to ensure successful deployment. There could be more downstream environments such as PERF (performance), STAGING, and finally PROD (production).

The tendency for many development teams and organizations is that if the tests pass then it is good enough to move into downstream environments. This is definitely an enormous improvement over extensive manual testing and stabilization periods on traditional projects. An issue that I have still seen is the slow introduction of software debt as an application is developed. Highly disciplined technical practices such as Test-Driven Design (TDD) and Pair Programming can help stave off extreme software debt but these practices are still not common place amongst software development organizations. This is not usually due to lack of clarity about these practices, excessive schedule pressure, legacy code, and the initial hurdle to learning how to do these practices effectively. In the meantime, we need a way to assess the health of our software applications beyond just tests passing and in the internals of the code and tests themselves. Sonar can be easily added into your infrastructure to provide insights into the health of your code but we can go even beyond that.

The Sonar Web Services API is quite simple to work with. The easiest way to pull information from Sonar is to call a URL:

http://nemo.sonarsource.org/api/resources?resource=248390&metrics=technical_debt_ratio

This will return an XML response like the following:

  248390
  com.adobe:as3corelib
  AS3 Core Lib
  AS3 Core Lib
  PRJ
  TRK
  flex
  1.0
  2010-09-19T01:55:06+0000

    technical_debt_ratio
    12.4
    12.4%

Within this XML, there is a section called  that includes the value of the metric we requested in the URL, “technical_debt_ratio”. The ratio of technical debt in this Flex codebase is 12.4%. Now with this information we can look for increases over time to identify technical debt earlier in the software development cycle. So, if the ratio to increase beyond 13% after being at 12.4% 1 month earlier, this could tell us that there is some technical issues creeping into the application.

Another way that the Sonar API can be used is from a programming language such as Java. The following Java code will pull the same information through the Java API client:

Sonar sonar = Sonar.create("http://nemo.sonarsource.org");
Resource commons = sonar.find(ResourceQuery.createForMetrics("248390",
        "technical_debt_ratio"));
System.out.println("Technical Debt Ratio: " +
        commons.getMeasure("technical_debt_ratio").getFormattedValue());

This will print “Technical Debt Ratio: 12.4%” to the console from a Java application. Once we are able to capture these metrics we could save them as data to trend in our automated promotion scripts that deploy builds in downstream environments. Some guidelines we have used in the past for these types of metrics are:

  • Small changes in a metric’s trend does not constitute immediate action
  • No more than 3 metrics should be trended (the typical 3 I watch for Java projects are duplication, class complexity, and technical debt)
  • The development should decide what are reasonable guidelines for indicating problems in the trends (such as technical debt +/- .5%)

In the automated deployment scripts, these trends can be used to stop deployment of the next build that passed all of its tests and emails can be sent to the development team regarding the metric culprit. From there, teams are able to enter the Sonar dashboard and drill down into the metric to see where the software debt is creeping in. Also, a source control diff can be produced to go into the email showing what files were changed between the successful builds that made the trend go haywire. This might be a listing per build and the metric variations for each.

This is a deep topic that this post just barely introduces. If your organization has a separate configuration management or operations group that managed environment promotions beyond the development environment, Sonar and the web services API can help further automate early identification of software debt in your applications before they pollute downstream environments.”

Thank you, Chris!

Advertisements

The Punched Cards in the Middle of Your Devops

leave a comment »

Source: http://www.flickr.com/photos/pjen/1070766105/

In her foreword to Gender Codes, Linda Shafer vividly describes the flow of programming work at NASA in 1965:

Following a design, we wrote – by hand – computer program instructions on large coding pads (80 columns per instruction, the same width as a Hollerith punched card). A Courier came by twice each day, picking up the coding pads and delivering yesterday’s instructions that had been magically translated into a different physical medium – card decks. Put some paper on a cart one day and presto, the next day, a stack of 7 and 3/8 inch by 3 and 1/4 inch, stiff paper sheets with holes punched in them were delivered. These cards constituted the program, which was sent to the machine room where operators fed the decks through the card reader.

Fast forward to 2010. If you have not yet moved to Continuous Deployment, metaphorically speaking you are still punching card. Not only are you falling behind on value delivery, you are missing up on the following great point made by colleague Josh Kerievsky:

What fascinates me most about #continuousdelivery is how it changes the way we design and collaborate on code.

Written by israelgat

September 8, 2010 at 6:39 am

Changing Culture to Enable Devops

leave a comment »

InfoQ has posted the video recording of the panel on Cultural Change in Devops from DevOps Day US 2010. Under the skillful moderation of Andrew Shafer, panelists John Allpsaw, Lee Thompson, Lloyd Taylor and I shed light on the fascinating cultural dynamics that devops teams go through. The four of us and Andrew are not necessarily in complete agreement on every point, but we all emphasize one key lesson:

Defining learning and readiness in technical terms is inadequate in the devops context.

Click here for the recording of the panel on Changing Culture to Enable Devops.

Written by israelgat

August 23, 2010 at 12:20 pm

A Good Start Point for Devops – Guest Post by Peter McGarahan

with one comment

Source: http://www.flickr.com/photos/stevepj2009/3461077400/

Many of the devops posts in this blog were written from a dev perspective. Today’s guest post by Peter McGarahan examines the topic from the ops perspective. It is inspired by the following eloquent quip about change:

Assume we’re starting from scratch. Assume that we actually are a startup that doesn’t have over a hundred years of experience and sub-optimized IT legacy.

A few biographical details for readers who might not know Peter or know of him. Peter J. McGarahan is the founder and president of McGarahan & Associates, an IT Service Management consulting and training organization.  Peter offers 27 years of IT and Business Service Management experience in optimizing and aligning the service and support organizations of the Fortune 1000 to deliver value against business objectives. His thought leadership has influenced the maturity and image of the service and support industry. His passion for customer service led the Taco Bell support organization to achieve the Help Desk Institute Team Excellence Award in 1995. IT Support News named him one of the “Top 25 Professionals in the Service and Support Industry” in 1999.  Support professionals voted McGarahan “The Legend of the Year” in 2002 and again in 2004 at the Service Desk Professionals conference for his endless energy, mentoring and leadership coaching. As a practitioner, product manager and support industry analyst and expert, McGarahan has left his service signature on the support industry / community.

Here is Peter:

As a former Director of Infrastructure & Operations (I&O), I found it beneficial to establish a respectful working relationship with my Development Colleagues. It was important for the accountable leaders to better understand the objectives, workings and success metrics of each team. It was also critical for the leader to establish the ‘rules of engagement’ for how each team would assist each other in achieving their stated objectives (success metrics). It certainly helped to have an IT / Business leader who established a cooperative / collaborative teamwork culture. She also supported it with shared IT / Business objectives and performance goals for all accountable IT leaders. The I&O team certainly benefited from a CIO who understood the importance of customer service, the value of support and the business impact (negative IT perception) caused by repetitive incidents, problems and service disruptions. It was a game changing day for I&O when the CIO announced that all accountable leaders would have half of their performance objectives (bonus compensation) based on the success metrics of the I&O team.

In working with Infrastructure & Operations organizations, it has become apparent that as we continue to implement, measure and continuously improve the IT Infrastructure Library v3 (ITIL) processes, we must simultaneously address how we focus on all things new! In a recent Cutter Executive Update entitled IT’s Change Imperative, I relate lessons learned from my conversations with Geir Ramleth, CIO of Bechtel and Ron Griffin, Senior VP of Applications for Hewlett-Packard. Their leadership, vision and courage inspired me to think differently about how IT can better work together for the benefit of the business. In the end, the only success that matters – is the continued growth and profitability of the business. A summary of their change success stories:

  • Hewlett-Packard CIO Randy Mott hired the right people to implement his IT strategy and change plan that included building, consolidating and automating its data centers; transferred work in-house from contractors; standardized on only a quarter of its apps; and built one central data warehouse — all while cutting spending in half.
  • Geir Ramleth, CIO of Bechtel described how he used cloud computing principles to transform IT and make Bechtel’s computing environment more agile. He had a vision of allowing Bechtal’s global employees access to the right resources at any place at any time with any device – delivered securely and cost-effectively. He encouraged his IT people to step outside their comfort zones and do things in a different way. He resisted modifying the current state and went with the transformational change fearing they would only wind-up incrementally better. In targeting a desired end state, he gave his team guiding instructions to “Assume we’re starting from scratch. Assume that we actually are a startup that doesn’t have over a hundred years of experience and sub-optimized IT legacy.”

In the spirit of change, we should challenge ourselves to develop shared ‘devops’ goals / objectives. In the end, these should help us identify, link and realize how to translate IT objectives / metrics into tangible business benefits / value.

I have listed some shared ‘devops’ goals / objectives that I believe are a good starting point. I encourage and invite your thoughts, opinions and ideas around these and any others that you feel would aid ‘devops’ in working to establish measurable business value credibility.

  1. Lower the total cost of ownership of all services (best way to achieve this is build them with serviceability, usability and maintainability in the design of all new applications, systems and services).
  2. Increase business value – achieve business benefits (lower operational costs, increased revenues, improved customer experience)
  • Simplified navigation
  • Productivity enhancing capabilities /functionality
  • Plug ‘n play integration
  • Personalization
  • Training / On-line Self-help features

3. Minimize business impact

  • Reduce change-related outages / incidents.
  • Reduce number of problems / incidents / calls.
  • Reduce the number of requests / training-related calls / inquiries.
    • Provide insights and tracking to the number of Known errors / workarounds / knowledge articles (solutions).
  • Speed to resolution based on business prioritization model
    • Operating Level Agreement / Commitment between Single Point of Contact (SPOC) Service Desk and internal IT Service Providers based on response / resolution times / commitments.
    • Bug-fix Process:

– Provide insights into the ‘bug/fix/enhancement’ list and process with transparent visibility to business prioritization (needs / requirements / quantifiable benefits).

4. Improved and frequent Communication

  • A marketing / product launch / status update and awareness campaign.
    • Especially around rollout / enhancement time.

Beyond Devops

leave a comment »

Based on feedback from participants in my Agile 2010 workshop “How We Do Things Around Here In Order To Succeed,” I am planning to offer the workshop as a one-day seminar. A tentative agenda for the seminar is as follows:

  • Introduction to Cultural Framework
  • Exercise #1: Determine Your Culture
  • Exercise #2: Strengths and Weaknesses of Your Culture
  • Change Behavior, Not Culture
  • When Cultures Meet
  • Exercise #3: Conflicts in Devops
  • The Agile Flywheel
  • Exercise #4: Using Technical Debt as a Boundary Object
  • Bringing Individuals and Organizations Together
  • Exercise #5: It is About Sharing the Process, Not Just Sharing the Information
  • Exercise #6: From success in devops to end-to-end success

Until I publish a full-fledged outline for the seminar, here is the central theme:

Beyond Devops

Inter-departmental flow in a corporate setting is often envisioned as the inner workings of a swiss watch. Wheels turn other wheels in a precise manner. Not only is effectiveness maintained, it is maintained in an efficient manner.

Problem is, many individuals and most departments hold distorted views of the departments they interact with. Reasonable distortions can be mitigated as long as the operational balance between departments is maintained. Once the operational balance is broken the “swiss watch” stops to function as the inter-departmental distortions block any attempt to restore the balance.

The most effective way to get dev and ops on a path of collaboration is for the two departments to jointly construct a boundary object. As dev and ops are joined in the hip through the code, and even more so through its quality, technical debt is well suited to serve as the core of a boundary object around which the two department share meaning while retaining operational autonomy.

Similar  boundary objects can be constructed between dev and other departments – customer support, professional services, marketing, sales and finance. When conceived and implemented in a manner that links numerous boundary objects together, Agile success in dev can be extended to both upstream and downstream functions.

Written by israelgat

August 11, 2010 at 5:11 am

A Devops Case Study

leave a comment »

An outline of my forthcoming Agile 2010 workshop was given in the post “A Recipe for Handling Cultural Conflicts in Devops and Beyond” earlier this week. Here is the case study around which the workshop is structured:

NotHere, Inc. Case Study

NotHere, Inc. is a $500M company based in Jerusalem, Israel. The company developed an eCommerce platform for small to medium retailers. Through a combination of this platform and its hosting data center, NotHere provides online store fronts, shopping carts, order processing, inventory, billing and marketing services to tens of thousands of retailers in a broad spectrum of verticals. For these retailers, NotHere is a one-stop “shopping” for all their online needs. In particular, instead of partnering with multiple companies like Amazon, Ebay, PayPal and Shopzilla, a retailer merely needs to partner with NotHere (who partners with these four companies and many others).

The small to medium retailers that use the good services of NotHere are critically dependent on the availability of its data center. For all practical purposes retailers are (temporarily) dead when the NotHere data center is not available. In recognition of the criticality of this aspect of its IT operations, NotHere invested a lot of effort in maturing its ITIL[i] processes. Its IT department successfully implements the ITIL service support and service delivery functions depicted in the figure below. From an operational perspective, an overall availability level of four nines is consistently attained. The company advertises this availability level as a major market differentiator.

In response to the accelerating pace in its marketplace, NotHere has been quite aggressive and successful in transitioning to Agile in product management, dev and test. Code quality, productivity and time-to-producing-code have been much improved over the past couple of years. The company measures those three metrics (quality, productivity, time-to-producing-code) regularly. The metrics feed into whole-hearted continuous improvement programs in product management, dev and test. They also serve as major components in evaluating the performance of the CTO and of the EVP of marketing.

NotHere has recently been struggling to reconcile velocity in development with availability in IT operations. Numerous attempts to turn speedy code development into fast service delivery have not been successful on two accounts:

  • Technical:  Early attempts to turn Continuous Integration into Continuous Deployment created numerous “hiccups” in both availability and audit.
  • Cultural: Dev is a competence culture; ops is a control culture.

A lot of tension has arisen between dev and ops as a result of the cultural differences compounding the technical differences. The situation deteriorated big time when the “lagging behind” picture below leaked from dev circles to ops.

The CEO of the company is of the opinion NotHere must reach the stage of Delivery over Development. She is not too interested in departmental metrics like the time it takes to develop code or the time it takes to deploy it. From her perspective, overall time-to-delivery (of service to the retailers) is the only meaningful business metric.

To accomplish Delivery over Development, the CEO launched a “Making Cats Work with Dogs[ii]” project. She gave the picture above to the CTO and CIO, making it crystal clear that the picture represents the end-point with respect to the relationship she expects the two of them and their departments to reach. Specifically, the CEO asked the CTO and the CIO to convene their staffs so that each department will:

  • Document its Outmodel (in the sense explored in the “How We Do Things Around Here In Order to Succeed” workshop) of the other department.
  • Compile a list of requirements it would like to put on the other group “to get its act together.”

The CEO also indicated she will convene and chair a meeting between the two departments. In this meeting she would like each department to present its two deliverables (world view of the other department & and the requirements to be put on it) and listen carefully to reflections and reactions from the other department. She expects the meeting will be the first step toward a mutual agreement between the two departments how to speed up overall service delivery.


[i] “Information Technology Infrastructure library – a set of concepts and practices for Information Technology Services Management (ITSM), Information Technology (IT) development and IT operations” [Wikipedia].

[ii] I am indebted to Patrick DeBois for suggesting this title.

© Copyright 2010 Israel Gat

A Recipe for Handling Cultural Conflicts in Devops and Beyond

with 4 comments

My Agile 2010 workshop “How We Do Things Around Here In Order To Succeed”  will weave together four trends that I am witnessing in my practice:

  • The ascendance of Agile portfolio management in a world characterized by loosely coupled processes
  • Devops dynamics are becoming more and more characteristic of end-to-end Agile/Kanban patterns
  • Viral spread of technical debt metrics in software governance
  • Increasing use of boundary objects in the enterprise context

The workshop is structured around three case studies/exercises that will take about two-thirds of the allotted time (the morning of August 9). The other third provides the theory and tools to be used in the three workshop exercises and (hopefully) in many future engagements participants in the workshop will carry out. Deep technical knowledge is not required – the workshop targets any Agile practitioner who has conceptual grasp of culture, software development, IT operations and portfolio management.

The #1 takeaway from the presentation is the details you need to know about creation and capture of lasting value through end-to-end Agile initiatives.

Here is the workshop agenda (still subject to some minor tweaking):

  • Introduction to Cultural Framework
  • Exercise #1: Strengths and Weaknesses of Your Culture
  • Change Behavior, Not Culture
  • When Organizations Clash
  • Exercise #2: Conflicts in Devops
  • The Agile Flywheel
  • Exercise #3: Using Technical Debt as a Boundary Object in Devops
  • Bringing Organizations Together Through Enlightened Governance Loops

I look forward to meeting you in the workshop and learning from your experiences and insights!

Israel