The Agile Executive

Making Agile Work

Posts Tagged ‘IT Operations

The Gat/Highsmith Joint Seminar on Technical Debt and Software Governance

leave a comment »

Jim and I have finalized the content and the format for our forthcoming Cutter Summit seminar. The seminar is structured around a case study which includes four exercise. We expect the case study/exercises will take close to two-thirds of the allotted time (the morning of October 27). In the other third we will provide the theory and practices to be used in the seminar exercises and (hopefully) in many future technical debt engagements participants in the workshop will oversee.

The seminar does not require deep technical knowledge. It targets participants who possess conceptual grasp of software development, software governance and IT operations/ITIL. If you feel like reading a little about technical debt prior to the Summit, the various posts on technical debt in this blog will be more than sufficient.

We plan to go with the following agenda (still subject to some minor tweaking):

Agenda for the October 27, 9:30AM to 1:00PM Technical Debt Seminar

  • Setting the Stage: Why Technical Debt is a Strategic Issue
  • Part I: What is Technical Debt?
  • Part II : Case Study – NotMyCompany, Inc.
    • Exercise #1 – Modernizing NotMyCompany’s Legacy Code
  • Part III: The Nature of Technical Debt
  • Part IV: Unified Governance
    • Exercise #2 – The acquisition of SocialAreUs by NotMyCompany
  • Part V: Process Control Models
    • Exercise #3 – How Often Should NotMyCompany Stop the Line?
  • (Time Permitting – Part VI: Using Technical Debt in Devops
    • Exercise #4 – The Agile Versus ITIL Debate at NotMyCompany)

By the end of the seminar you will know how to effectively apply technical debt techniques as an integral part of software governance that is anchored in business realities and imperatives.

Written by israelgat

September 30, 2010 at 3:20 pm

What 108M Lines of Code do not Tell Us

leave a comment »

Source: Nemo

Coming on the heels of Gartner’s research note projecting $1 trillion in IT Debt by 2015, CAST’s study provided a more granular view of the debt, estimating an average of over $1 million in technical debt per application in a sample of 288 applications. Between these two studies, the situation examined at the micro-level seems to be quite consistent with the state of affairs estimated and projected at the macro-level.

My hunch is that the gravity of the situation from a software quality and maintenance perspective is actually masked by efforts of IT staffs to compensate for programming problems through operational excellence. For example, carefully staged deployment and quick rollback often enable coping with defects that could/should have been handled through higher test coverage, lesser complexity or a more acceptable level of code duplication.

Part of the reason that the masking effects of IT staffs are not always fully appreciated is that they are embedded in the business design of IT Outsourcing companies. The company to which you outsourced your IT is ‘making a bet’ it can run your IT better than you can. It often succeeds in so doing. The unresolved defects in your old code plus those that evolved over time through software decay have not necessarily been fixed. Rather, the manifestations of these defects are  handled operationally in a more efficient manner.

Think again if your visceral reaction to the technical debt situation described in the Gartner research note and the CAST study is of the “This can’t possibly be true” variety. It is what it is – just take a quick look at Nemo to see representative technical debt data with your own eyes. And, as indicated in this post, it might even be worse than what it looks. As Gartner puts it:

The results of such [IT Debt] an assessment will be, at best, unsettling and, at worst, truly shocking.

Technical Debt Assessment, Sterling Barton LLC and the Moussaka

with one comment

A few month ago Chris Sterling and I were carrying out a Cutter Technical Debt Assessment and Valuation engagement for a venture capitalist who was considering a certain company. We discovered various things in the code of this company. More noteworthy, my deep domain expertise led to Chris discovering the great Greek dish Moussaka.

I have eaten a lot of good Moussakas over the years. Even against this solid gastronomic background I can’t forget how the eyes of Chris lit up when he took the first bite. It took him only a tiny little time to get on his iPhone and tweet on the culinary aspects of our engagement. I then knew it was going to be a very successful engagement…

The relationship with Chris deepened since this episode. For example, in collaboration with Brent Barton Chris contributed a great article to the forthcoming issue of the Cutter IT Journal on Technical Debt. In this article Chris and Brent  demonstrate how technical debt techniques can be applied at the portfolio level. They make the reader step into the shoes of the project portfolio planner and walk him through their approach to enhancing the decision-making process by using the software debt dashboard.

Chris has just published an excellent post entitled “Using Sonar Metrics to Assess Promotion of Builds to Downstream Environments” in Getting Agile and was kind enough to suggest I cross-post it in The Agile Executive. Here it is (please note that the examples given below by Chris have nothing to do with the engagement described above):

“For those of you that don’t already know about Sonar you are missing an important tool in your quality assessment arsenal. Sonar is an open source tool that is a foundational platform to manage your software’s quality. The image below shows one of the main dashboard views that teams can use to get insights into their software’s health.

The dashboard provides rollup metrics out of the box for:

  • Duplication (probably the biggest Design Debt in many software projects)
  • Code coverage (amount of code touched by automated unit tests)
  • Rules compliance (identifies potential issues in the code such as security concerns)
  • Code complexity (an indicator of how easy the software will adapt to meet new needs)
  • Size of codebase (lines of code [LOC])

Before going into how to use these metrics to assess whether to promote builds to downstream environments, I want to preface the conversation with the following note:

Code analysis metrics should NOT be used to assess teams and are most useful when considering how they trend over time

Now that we have this important note out-of-the-way and, of course, nobody will ever use these metrics for “evil”, lets discuss pulling data from Sonar to automate assessments of builds for promotion to downstream environments. For those that are unfamiliar with automated promotion, here is a simple, happy example:

A development team makes some changes to the automated tests and implementation code on an application and checks their changes into source control. A continuous integration server finds out that source control artifacts have changed since the last time it ran a build cycle and updates its local artifacts to incorporate the most recent changes. The continuous integration server then runs the build by compiling, executing automated tests, running Sonar code analysis, and deploying the successful deployment artifact to a waiting environment usually called something like “DEV”. Once deployed, a set of automated acceptance tests are executed against the DEV environment to validate that basic aspects of the application are still working from a user perspective. Sometime after all of the acceptance tests pass successfully (this could be twice a day or some other timeline that works for those using downstream environments), the continuous integration server promotes the build from the DEV environment to a TEST environment. Once deployed, the application might be running alongside other dependent or sibling applications and integration tests are run to ensure successful deployment. There could be more downstream environments such as PERF (performance), STAGING, and finally PROD (production).

The tendency for many development teams and organizations is that if the tests pass then it is good enough to move into downstream environments. This is definitely an enormous improvement over extensive manual testing and stabilization periods on traditional projects. An issue that I have still seen is the slow introduction of software debt as an application is developed. Highly disciplined technical practices such as Test-Driven Design (TDD) and Pair Programming can help stave off extreme software debt but these practices are still not common place amongst software development organizations. This is not usually due to lack of clarity about these practices, excessive schedule pressure, legacy code, and the initial hurdle to learning how to do these practices effectively. In the meantime, we need a way to assess the health of our software applications beyond just tests passing and in the internals of the code and tests themselves. Sonar can be easily added into your infrastructure to provide insights into the health of your code but we can go even beyond that.

The Sonar Web Services API is quite simple to work with. The easiest way to pull information from Sonar is to call a URL:

http://nemo.sonarsource.org/api/resources?resource=248390&metrics=technical_debt_ratio

This will return an XML response like the following:

  248390
  com.adobe:as3corelib
  AS3 Core Lib
  AS3 Core Lib
  PRJ
  TRK
  flex
  1.0
  2010-09-19T01:55:06+0000

    technical_debt_ratio
    12.4
    12.4%

Within this XML, there is a section called  that includes the value of the metric we requested in the URL, “technical_debt_ratio”. The ratio of technical debt in this Flex codebase is 12.4%. Now with this information we can look for increases over time to identify technical debt earlier in the software development cycle. So, if the ratio to increase beyond 13% after being at 12.4% 1 month earlier, this could tell us that there is some technical issues creeping into the application.

Another way that the Sonar API can be used is from a programming language such as Java. The following Java code will pull the same information through the Java API client:

Sonar sonar = Sonar.create("http://nemo.sonarsource.org");
Resource commons = sonar.find(ResourceQuery.createForMetrics("248390",
        "technical_debt_ratio"));
System.out.println("Technical Debt Ratio: " +
        commons.getMeasure("technical_debt_ratio").getFormattedValue());

This will print “Technical Debt Ratio: 12.4%” to the console from a Java application. Once we are able to capture these metrics we could save them as data to trend in our automated promotion scripts that deploy builds in downstream environments. Some guidelines we have used in the past for these types of metrics are:

  • Small changes in a metric’s trend does not constitute immediate action
  • No more than 3 metrics should be trended (the typical 3 I watch for Java projects are duplication, class complexity, and technical debt)
  • The development should decide what are reasonable guidelines for indicating problems in the trends (such as technical debt +/- .5%)

In the automated deployment scripts, these trends can be used to stop deployment of the next build that passed all of its tests and emails can be sent to the development team regarding the metric culprit. From there, teams are able to enter the Sonar dashboard and drill down into the metric to see where the software debt is creeping in. Also, a source control diff can be produced to go into the email showing what files were changed between the successful builds that made the trend go haywire. This might be a listing per build and the metric variations for each.

This is a deep topic that this post just barely introduces. If your organization has a separate configuration management or operations group that managed environment promotions beyond the development environment, Sonar and the web services API can help further automate early identification of software debt in your applications before they pollute downstream environments.”

Thank you, Chris!

Why Spend the Afternoon as well on Technical Debt?

with 2 comments

Source: http://www.flickr.com/photos/pinksherbet/233228813/

Yesterday’s post Why Spend a Whole Morning on Technical Debt? listed eight characteristics of the technical debt metric that will be discussed during the morning of October 27 when Jim Highsmith and I deliver our joint Cutter Summit seminar. This posts adds to the previous post by suggesting a related topic for the afternoon.

No, I am not trying to “hijack” the Summit agenda messing with the afternoon sessions by colleagues Claude Baudoin and Mitchell Ummel. I am simply pointing out a corollary to the morning seminar that might be on your mind in the afternoon. Needless to say, thinking about it in the afternoon of the 28th instead of the afternoon of the 27th is quite appropriate…

Yesterday’s post concluded with a “what it all means” statement, as follows:

Technical debt is a meaningful metric at any level of your organization and for any department in it. Moreover, it is applicable to any business process that is not yet taking software quality into account.

If you accept this premise, you can use the technical debt metric to construct boundary objects between various departments in your company/organization. The metric could serve as the heart of boundary objects between dev and IT ops, between dev and customer support, between dev and a company to which some development tasks are outsourced, etc. The point is the enablement of working agreements between multiple stakeholders through the technical debt metric. For example, dev and IT ops might mutually agree that the technical debt in the code to be deployed to the production environment will be less than $3 per line of code. Or, dev and customer support might agree that enhanced refactoring will commence if the code decays over time to more than $4 per line of code.

You can align various departments by by using the technical debt metric. This alignment is particularly important when the operational balance between departments has been disrupted. For example, your developers might be coding faster than your ITIL change managers can process the change requests.

A lot more on the use of the technical debt metric to mitigate cross-organizational dysfunctions, including some Outmodel aspects, will be covered in our seminar in Cambridge, MA on the morning of the 27th. We look forward to discussing this intriguing subject with you there!

Israel

Consumerization of Enterprise Software

with 7 comments

Source: http://www.flickr.com/photos/ross/3055802287/

Figure 1: Consumerization of IT

The devastation in traditional Publishing needs precious little mentioning. Just think about a brand like BusinessWeek selling for a meager cash offer in the $2 million to $5 million range, McGraw Hill getting into interactive text books through Inkling or Flipboard delivering “… your personalized social magazine” to your iPad. This devastation might not have gotten the attention that the plight of the ‘big three’ automobile manufacturers got, but in its own way it is as shocking as a visit to the abandoned properties in Detroit is.

As most of my clients do enterprise software, many of my discussions with them is about the consumerization of IT. From a day-to-day perspective this consumerization is primarily about six aspects:

  • Use of less expensive/consumer-focused components as infrastructure
  • ‘Pay as you go’ pricing (through Cloud pricing mechanisms/policies)
  • Use of web application interfaces to monitor IT infrastructure
  • Use of mobile and consumer based devices for accessing IT alerts and interfacing with systems
  • Use of the fast growing number of mobile applications to enhance productivity
  • Application of enterprise social networks and social software in the data center

From a strategic perspective, IT consumerization IMHO is all about the transformation toward “everything as a service” [1]. The virtuous cycle driven by Cloud, Mobile and Social manifests itself at three levels:

  • It obviously affects the IT folks with whom I discuss the subject. Immense changes are already taking place in many IT departments.
  • It affects their company. For example, the company might need to change the business design in order to optimize its supply chain.
  • It affects the clients of their company. Their definition of value changes these days faster than the time it takes the CIO I speak with to say “value.”

© Copyright 2010 Israel Gat

Figure 2: The Virtuous Cycle of Cloud, Mobile and Social

Sometimes I get a push-back from my clients on this topic. The push-back is usually rooted in the immense complexity (and fragility) of the enterprise software systems that had been built over the past ten, twenty or thirty years. The folks who push back on me point out that consumerization of IT will not scale big time until enterprise software gets “consumerized” or at least modernized.

I agree with this good counter-point but only up to a point. I believe two factors are likely to accelerate the pace toward “consumerization” of enterprise software:

  1. Any department/business unit that can get a service in entirety from an outside source is likely to do so without worrying about enterprise software and/or data center considerations. This is already happening in Marketing. As other functions start doing so, more and more links in the value chain of enterprise software will be “consumerized.” In other words, these services will be carried out without the involvement of the IT department.
  2. Once the switch-over costs from legacy code to state-of-the-art code are less than the steady state costs (to maintain and update legacy code), the “consumerization” of enterprise software is going to happen with ferocious urgency.

If you are in enterprise software you need to start modernizing your applications today. The reason is the imperative need to mitigate risk prior to reaching the end-point, almost irrespective of how far down the road the end-point might be.  See Llewellyn Falco‘s excellent video clip Rewriting Vs Refactoring for a crisp articulation of the risk involved in rewriting and why starting to refactor now is the best way to mitigate the risk.

Footnotes:

[1] The phrase “Everything as a Service” has been coined by Russ Daniels.

The Punched Cards in the Middle of Your Devops

leave a comment »

Source: http://www.flickr.com/photos/pjen/1070766105/

In her foreword to Gender Codes, Linda Shafer vividly describes the flow of programming work at NASA in 1965:

Following a design, we wrote – by hand – computer program instructions on large coding pads (80 columns per instruction, the same width as a Hollerith punched card). A Courier came by twice each day, picking up the coding pads and delivering yesterday’s instructions that had been magically translated into a different physical medium – card decks. Put some paper on a cart one day and presto, the next day, a stack of 7 and 3/8 inch by 3 and 1/4 inch, stiff paper sheets with holes punched in them were delivered. These cards constituted the program, which was sent to the machine room where operators fed the decks through the card reader.

Fast forward to 2010. If you have not yet moved to Continuous Deployment, metaphorically speaking you are still punching card. Not only are you falling behind on value delivery, you are missing up on the following great point made by colleague Josh Kerievsky:

What fascinates me most about #continuousdelivery is how it changes the way we design and collaborate on code.

Written by israelgat

September 8, 2010 at 6:39 am

Changing Culture to Enable Devops

leave a comment »

InfoQ has posted the video recording of the panel on Cultural Change in Devops from DevOps Day US 2010. Under the skillful moderation of Andrew Shafer, panelists John Allpsaw, Lee Thompson, Lloyd Taylor and I shed light on the fascinating cultural dynamics that devops teams go through. The four of us and Andrew are not necessarily in complete agreement on every point, but we all emphasize one key lesson:

Defining learning and readiness in technical terms is inadequate in the devops context.

Click here for the recording of the panel on Changing Culture to Enable Devops.

Written by israelgat

August 23, 2010 at 12:20 pm

A Devops Case Study

leave a comment »

An outline of my forthcoming Agile 2010 workshop was given in the post “A Recipe for Handling Cultural Conflicts in Devops and Beyond” earlier this week. Here is the case study around which the workshop is structured:

NotHere, Inc. Case Study

NotHere, Inc. is a $500M company based in Jerusalem, Israel. The company developed an eCommerce platform for small to medium retailers. Through a combination of this platform and its hosting data center, NotHere provides online store fronts, shopping carts, order processing, inventory, billing and marketing services to tens of thousands of retailers in a broad spectrum of verticals. For these retailers, NotHere is a one-stop “shopping” for all their online needs. In particular, instead of partnering with multiple companies like Amazon, Ebay, PayPal and Shopzilla, a retailer merely needs to partner with NotHere (who partners with these four companies and many others).

The small to medium retailers that use the good services of NotHere are critically dependent on the availability of its data center. For all practical purposes retailers are (temporarily) dead when the NotHere data center is not available. In recognition of the criticality of this aspect of its IT operations, NotHere invested a lot of effort in maturing its ITIL[i] processes. Its IT department successfully implements the ITIL service support and service delivery functions depicted in the figure below. From an operational perspective, an overall availability level of four nines is consistently attained. The company advertises this availability level as a major market differentiator.

In response to the accelerating pace in its marketplace, NotHere has been quite aggressive and successful in transitioning to Agile in product management, dev and test. Code quality, productivity and time-to-producing-code have been much improved over the past couple of years. The company measures those three metrics (quality, productivity, time-to-producing-code) regularly. The metrics feed into whole-hearted continuous improvement programs in product management, dev and test. They also serve as major components in evaluating the performance of the CTO and of the EVP of marketing.

NotHere has recently been struggling to reconcile velocity in development with availability in IT operations. Numerous attempts to turn speedy code development into fast service delivery have not been successful on two accounts:

  • Technical:  Early attempts to turn Continuous Integration into Continuous Deployment created numerous “hiccups” in both availability and audit.
  • Cultural: Dev is a competence culture; ops is a control culture.

A lot of tension has arisen between dev and ops as a result of the cultural differences compounding the technical differences. The situation deteriorated big time when the “lagging behind” picture below leaked from dev circles to ops.

The CEO of the company is of the opinion NotHere must reach the stage of Delivery over Development. She is not too interested in departmental metrics like the time it takes to develop code or the time it takes to deploy it. From her perspective, overall time-to-delivery (of service to the retailers) is the only meaningful business metric.

To accomplish Delivery over Development, the CEO launched a “Making Cats Work with Dogs[ii]” project. She gave the picture above to the CTO and CIO, making it crystal clear that the picture represents the end-point with respect to the relationship she expects the two of them and their departments to reach. Specifically, the CEO asked the CTO and the CIO to convene their staffs so that each department will:

  • Document its Outmodel (in the sense explored in the “How We Do Things Around Here In Order to Succeed” workshop) of the other department.
  • Compile a list of requirements it would like to put on the other group “to get its act together.”

The CEO also indicated she will convene and chair a meeting between the two departments. In this meeting she would like each department to present its two deliverables (world view of the other department & and the requirements to be put on it) and listen carefully to reflections and reactions from the other department. She expects the meeting will be the first step toward a mutual agreement between the two departments how to speed up overall service delivery.


[i] “Information Technology Infrastructure library – a set of concepts and practices for Information Technology Services Management (ITSM), Information Technology (IT) development and IT operations” [Wikipedia].

[ii] I am indebted to Patrick DeBois for suggesting this title.

© Copyright 2010 Israel Gat

As If Another Proof Point Was Needed

leave a comment »

Annie Shum’s interview earlier this week gave readers of this blog a multi-dimensional view of imminent changes in IT. If you needed independent validation, it came yesterday through EMC’s Chuck Hollis words in the national solution provider GreenPages Technology Solutions’ 14th annual summit:

Vice President Global Marketing CTO Chuck Hollis Monday said the changes resulting from the storage giant’s own no-holds barred journey to the private cloud led to a decline in IT employee job satisfaction…

Hollis said the internal IT satisfaction drop came in the second phase of the EMC cloud revolution focused squarely on mission critical applications. That second phase — which EMC is in the midst of now — has sparked major changes in IT jobs as the company has replaced IT management, security staff and backend IT staff.

“During this phase, this is where org (organizational) chart issues started to come in,” Hollis said. “People’s jobs started to change. Younger people in the organization were being promoted over older people.”

As if another proof point to add to Annie‘s rigorous data was needed…

Written by israelgat

August 4, 2010 at 7:30 am

A Recipe for Handling Cultural Conflicts in Devops and Beyond

with 4 comments

My Agile 2010 workshop “How We Do Things Around Here In Order To Succeed”  will weave together four trends that I am witnessing in my practice:

  • The ascendance of Agile portfolio management in a world characterized by loosely coupled processes
  • Devops dynamics are becoming more and more characteristic of end-to-end Agile/Kanban patterns
  • Viral spread of technical debt metrics in software governance
  • Increasing use of boundary objects in the enterprise context

The workshop is structured around three case studies/exercises that will take about two-thirds of the allotted time (the morning of August 9). The other third provides the theory and tools to be used in the three workshop exercises and (hopefully) in many future engagements participants in the workshop will carry out. Deep technical knowledge is not required – the workshop targets any Agile practitioner who has conceptual grasp of culture, software development, IT operations and portfolio management.

The #1 takeaway from the presentation is the details you need to know about creation and capture of lasting value through end-to-end Agile initiatives.

Here is the workshop agenda (still subject to some minor tweaking):

  • Introduction to Cultural Framework
  • Exercise #1: Strengths and Weaknesses of Your Culture
  • Change Behavior, Not Culture
  • When Organizations Clash
  • Exercise #2: Conflicts in Devops
  • The Agile Flywheel
  • Exercise #3: Using Technical Debt as a Boundary Object in Devops
  • Bringing Organizations Together Through Enlightened Governance Loops

I look forward to meeting you in the workshop and learning from your experiences and insights!

Israel