The Agile Executive

Making Agile Work

Search Results

The Punched Cards in the Middle of Your Devops

leave a comment »

Source: http://www.flickr.com/photos/pjen/1070766105/

In her foreword to Gender Codes, Linda Shafer vividly describes the flow of programming work at NASA in 1965:

Following a design, we wrote – by hand – computer program instructions on large coding pads (80 columns per instruction, the same width as a Hollerith punched card). A Courier came by twice each day, picking up the coding pads and delivering yesterday’s instructions that had been magically translated into a different physical medium – card decks. Put some paper on a cart one day and presto, the next day, a stack of 7 and 3/8 inch by 3 and 1/4 inch, stiff paper sheets with holes punched in them were delivered. These cards constituted the program, which was sent to the machine room where operators fed the decks through the card reader.

Fast forward to 2010. If you have not yet moved to Continuous Deployment, metaphorically speaking you are still punching card. Not only are you falling behind on value delivery, you are missing up on the following great point made by colleague Josh Kerievsky:

What fascinates me most about #continuousdelivery is how it changes the way we design and collaborate on code.

Written by israelgat

September 8, 2010 at 6:39 am

From Vivek Kundra to Devops and Compounding Interest: Cutter’s Forthcoming Special Issue on Technical Debt

leave a comment »

Source: http://www.flickr.com/photos/wallyg/152453473/

In a little over a month the Cutter Consortium will publish a special issue of the Cutter IT Journal (CITJ) on Technical Debt. As the guest editor for this issue I had the privilege to set the direction for it and now have early exposure to the latest and greatest in research and field work from the various authors. This short post is intended to share with you some of the more exciting findings you could expect in this issue of the CITJ.

The picture above of the debt clock is a common metaphor that runs through all articles. The various authors are unanimously of the opinion that one must measure his/her technical debt, embed the measurements in the software governance process and relentlessly push hard to reduce technical debt. One can easily extrapolate this common thread to conjecture an initiative by Vivek Kundra to assess technical debt and its ramification at the national level.

Naturally, the specific areas of interest with respect to technical debt vary from one author to another. From the broad spectrum of topics addressed in the journal, I would like to mention two that are quite representative:

  • One of the authors focuses on the difference between the manifestation of technical debt in dev versus its manifestation in devops, reaching the conclusion that the change in context (from dev to devops) makes quite a difference. The author actually doubts that the classical differentiation between “building the right system” and “building the system right” holds in devops.
  • Another author derives formulas for calculating  Recurring Interest and Compounding Interest in technical debt. The author uses these formulas to demonstrate two scenario: Scenario A in which technical debt as % of total product revenue is 12% and Scenario B in which technical debt as % of total product revenue is 280%. The fascinating thing is that this dramatic difference (12% v. 280%) is induced through much smaller variances in the Recurring Interest and the Compounding Interest.

I will blog much more on the subject when the CITJ issue is published in October. In addition, Jim Highsmith and I will discuss the findings of the various authors as part of our joint seminar on the subject in the forthcoming Cutter Summit.

Stay tuned!

Changing Culture to Enable Devops

leave a comment »

InfoQ has posted the video recording of the panel on Cultural Change in Devops from DevOps Day US 2010. Under the skillful moderation of Andrew Shafer, panelists John Allpsaw, Lee Thompson, Lloyd Taylor and I shed light on the fascinating cultural dynamics that devops teams go through. The four of us and Andrew are not necessarily in complete agreement on every point, but we all emphasize one key lesson:

Defining learning and readiness in technical terms is inadequate in the devops context.

Click here for the recording of the panel on Changing Culture to Enable Devops.

Written by israelgat

August 23, 2010 at 12:20 pm

A Good Start Point for Devops – Guest Post by Peter McGarahan

with one comment

Source: http://www.flickr.com/photos/stevepj2009/3461077400/

Many of the devops posts in this blog were written from a dev perspective. Today’s guest post by Peter McGarahan examines the topic from the ops perspective. It is inspired by the following eloquent quip about change:

Assume we’re starting from scratch. Assume that we actually are a startup that doesn’t have over a hundred years of experience and sub-optimized IT legacy.

A few biographical details for readers who might not know Peter or know of him. Peter J. McGarahan is the founder and president of McGarahan & Associates, an IT Service Management consulting and training organization.  Peter offers 27 years of IT and Business Service Management experience in optimizing and aligning the service and support organizations of the Fortune 1000 to deliver value against business objectives. His thought leadership has influenced the maturity and image of the service and support industry. His passion for customer service led the Taco Bell support organization to achieve the Help Desk Institute Team Excellence Award in 1995. IT Support News named him one of the “Top 25 Professionals in the Service and Support Industry” in 1999.  Support professionals voted McGarahan “The Legend of the Year” in 2002 and again in 2004 at the Service Desk Professionals conference for his endless energy, mentoring and leadership coaching. As a practitioner, product manager and support industry analyst and expert, McGarahan has left his service signature on the support industry / community.

Here is Peter:

As a former Director of Infrastructure & Operations (I&O), I found it beneficial to establish a respectful working relationship with my Development Colleagues. It was important for the accountable leaders to better understand the objectives, workings and success metrics of each team. It was also critical for the leader to establish the ‘rules of engagement’ for how each team would assist each other in achieving their stated objectives (success metrics). It certainly helped to have an IT / Business leader who established a cooperative / collaborative teamwork culture. She also supported it with shared IT / Business objectives and performance goals for all accountable IT leaders. The I&O team certainly benefited from a CIO who understood the importance of customer service, the value of support and the business impact (negative IT perception) caused by repetitive incidents, problems and service disruptions. It was a game changing day for I&O when the CIO announced that all accountable leaders would have half of their performance objectives (bonus compensation) based on the success metrics of the I&O team.

In working with Infrastructure & Operations organizations, it has become apparent that as we continue to implement, measure and continuously improve the IT Infrastructure Library v3 (ITIL) processes, we must simultaneously address how we focus on all things new! In a recent Cutter Executive Update entitled IT’s Change Imperative, I relate lessons learned from my conversations with Geir Ramleth, CIO of Bechtel and Ron Griffin, Senior VP of Applications for Hewlett-Packard. Their leadership, vision and courage inspired me to think differently about how IT can better work together for the benefit of the business. In the end, the only success that matters – is the continued growth and profitability of the business. A summary of their change success stories:

  • Hewlett-Packard CIO Randy Mott hired the right people to implement his IT strategy and change plan that included building, consolidating and automating its data centers; transferred work in-house from contractors; standardized on only a quarter of its apps; and built one central data warehouse — all while cutting spending in half.
  • Geir Ramleth, CIO of Bechtel described how he used cloud computing principles to transform IT and make Bechtel’s computing environment more agile. He had a vision of allowing Bechtal’s global employees access to the right resources at any place at any time with any device – delivered securely and cost-effectively. He encouraged his IT people to step outside their comfort zones and do things in a different way. He resisted modifying the current state and went with the transformational change fearing they would only wind-up incrementally better. In targeting a desired end state, he gave his team guiding instructions to “Assume we’re starting from scratch. Assume that we actually are a startup that doesn’t have over a hundred years of experience and sub-optimized IT legacy.”

In the spirit of change, we should challenge ourselves to develop shared ‘devops’ goals / objectives. In the end, these should help us identify, link and realize how to translate IT objectives / metrics into tangible business benefits / value.

I have listed some shared ‘devops’ goals / objectives that I believe are a good starting point. I encourage and invite your thoughts, opinions and ideas around these and any others that you feel would aid ‘devops’ in working to establish measurable business value credibility.

  1. Lower the total cost of ownership of all services (best way to achieve this is build them with serviceability, usability and maintainability in the design of all new applications, systems and services).
  2. Increase business value – achieve business benefits (lower operational costs, increased revenues, improved customer experience)
  • Simplified navigation
  • Productivity enhancing capabilities /functionality
  • Plug ‘n play integration
  • Personalization
  • Training / On-line Self-help features

3. Minimize business impact

  • Reduce change-related outages / incidents.
  • Reduce number of problems / incidents / calls.
  • Reduce the number of requests / training-related calls / inquiries.
    • Provide insights and tracking to the number of Known errors / workarounds / knowledge articles (solutions).
  • Speed to resolution based on business prioritization model
    • Operating Level Agreement / Commitment between Single Point of Contact (SPOC) Service Desk and internal IT Service Providers based on response / resolution times / commitments.
    • Bug-fix Process:

- Provide insights into the ‘bug/fix/enhancement’ list and process with transparent visibility to business prioritization (needs / requirements / quantifiable benefits).

4. Improved and frequent Communication

  • A marketing / product launch / status update and awareness campaign.
    • Especially around rollout / enhancement time.

Beyond Devops

leave a comment »

Based on feedback from participants in my Agile 2010 workshop “How We Do Things Around Here In Order To Succeed,” I am planning to offer the workshop as a one-day seminar. A tentative agenda for the seminar is as follows:

  • Introduction to Cultural Framework
  • Exercise #1: Determine Your Culture
  • Exercise #2: Strengths and Weaknesses of Your Culture
  • Change Behavior, Not Culture
  • When Cultures Meet
  • Exercise #3: Conflicts in Devops
  • The Agile Flywheel
  • Exercise #4: Using Technical Debt as a Boundary Object
  • Bringing Individuals and Organizations Together
  • Exercise #5: It is About Sharing the Process, Not Just Sharing the Information
  • Exercise #6: From success in devops to end-to-end success

Until I publish a full-fledged outline for the seminar, here is the central theme:

Beyond Devops

Inter-departmental flow in a corporate setting is often envisioned as the inner workings of a swiss watch. Wheels turn other wheels in a precise manner. Not only is effectiveness maintained, it is maintained in an efficient manner.

Problem is, many individuals and most departments hold distorted views of the departments they interact with. Reasonable distortions can be mitigated as long as the operational balance between departments is maintained. Once the operational balance is broken the “swiss watch” stops to function as the inter-departmental distortions block any attempt to restore the balance.

The most effective way to get dev and ops on a path of collaboration is for the two departments to jointly construct a boundary object. As dev and ops are joined in the hip through the code, and even more so through its quality, technical debt is well suited to serve as the core of a boundary object around which the two department share meaning while retaining operational autonomy.

Similar  boundary objects can be constructed between dev and other departments – customer support, professional services, marketing, sales and finance. When conceived and implemented in a manner that links numerous boundary objects together, Agile success in dev can be extended to both upstream and downstream functions.

Written by israelgat

August 11, 2010 at 5:11 am

A Devops Case Study

leave a comment »

An outline of my forthcoming Agile 2010 workshop was given in the post “A Recipe for Handling Cultural Conflicts in Devops and Beyond” earlier this week. Here is the case study around which the workshop is structured:

NotHere, Inc. Case Study

NotHere, Inc. is a $500M company based in Jerusalem, Israel. The company developed an eCommerce platform for small to medium retailers. Through a combination of this platform and its hosting data center, NotHere provides online store fronts, shopping carts, order processing, inventory, billing and marketing services to tens of thousands of retailers in a broad spectrum of verticals. For these retailers, NotHere is a one-stop “shopping” for all their online needs. In particular, instead of partnering with multiple companies like Amazon, Ebay, PayPal and Shopzilla, a retailer merely needs to partner with NotHere (who partners with these four companies and many others).

The small to medium retailers that use the good services of NotHere are critically dependent on the availability of its data center. For all practical purposes retailers are (temporarily) dead when the NotHere data center is not available. In recognition of the criticality of this aspect of its IT operations, NotHere invested a lot of effort in maturing its ITIL[i] processes. Its IT department successfully implements the ITIL service support and service delivery functions depicted in the figure below. From an operational perspective, an overall availability level of four nines is consistently attained. The company advertises this availability level as a major market differentiator.

In response to the accelerating pace in its marketplace, NotHere has been quite aggressive and successful in transitioning to Agile in product management, dev and test. Code quality, productivity and time-to-producing-code have been much improved over the past couple of years. The company measures those three metrics (quality, productivity, time-to-producing-code) regularly. The metrics feed into whole-hearted continuous improvement programs in product management, dev and test. They also serve as major components in evaluating the performance of the CTO and of the EVP of marketing.

NotHere has recently been struggling to reconcile velocity in development with availability in IT operations. Numerous attempts to turn speedy code development into fast service delivery have not been successful on two accounts:

  • Technical:  Early attempts to turn Continuous Integration into Continuous Deployment created numerous “hiccups” in both availability and audit.
  • Cultural: Dev is a competence culture; ops is a control culture.

A lot of tension has arisen between dev and ops as a result of the cultural differences compounding the technical differences. The situation deteriorated big time when the “lagging behind” picture below leaked from dev circles to ops.

The CEO of the company is of the opinion NotHere must reach the stage of Delivery over Development. She is not too interested in departmental metrics like the time it takes to develop code or the time it takes to deploy it. From her perspective, overall time-to-delivery (of service to the retailers) is the only meaningful business metric.

To accomplish Delivery over Development, the CEO launched a “Making Cats Work with Dogs[ii]” project. She gave the picture above to the CTO and CIO, making it crystal clear that the picture represents the end-point with respect to the relationship she expects the two of them and their departments to reach. Specifically, the CEO asked the CTO and the CIO to convene their staffs so that each department will:

  • Document its Outmodel (in the sense explored in the “How We Do Things Around Here In Order to Succeed” workshop) of the other department.
  • Compile a list of requirements it would like to put on the other group “to get its act together.”

The CEO also indicated she will convene and chair a meeting between the two departments. In this meeting she would like each department to present its two deliverables (world view of the other department & and the requirements to be put on it) and listen carefully to reflections and reactions from the other department. She expects the meeting will be the first step toward a mutual agreement between the two departments how to speed up overall service delivery.


[i] “Information Technology Infrastructure library – a set of concepts and practices for Information Technology Services Management (ITSM), Information Technology (IT) development and IT operations” [Wikipedia].

[ii] I am indebted to Patrick DeBois for suggesting this title.

© Copyright 2010 Israel Gat

A Recipe for Handling Cultural Conflicts in Devops and Beyond

with 4 comments

My Agile 2010 workshop “How We Do Things Around Here In Order To Succeed”  will weave together four trends that I am witnessing in my practice:

  • The ascendance of Agile portfolio management in a world characterized by loosely coupled processes
  • Devops dynamics are becoming more and more characteristic of end-to-end Agile/Kanban patterns
  • Viral spread of technical debt metrics in software governance
  • Increasing use of boundary objects in the enterprise context

The workshop is structured around three case studies/exercises that will take about two-thirds of the allotted time (the morning of August 9). The other third provides the theory and tools to be used in the three workshop exercises and (hopefully) in many future engagements participants in the workshop will carry out. Deep technical knowledge is not required – the workshop targets any Agile practitioner who has conceptual grasp of culture, software development, IT operations and portfolio management.

The #1 takeaway from the presentation is the details you need to know about creation and capture of lasting value through end-to-end Agile initiatives.

Here is the workshop agenda (still subject to some minor tweaking):

  • Introduction to Cultural Framework
  • Exercise #1: Strengths and Weaknesses of Your Culture
  • Change Behavior, Not Culture
  • When Organizations Clash
  • Exercise #2: Conflicts in Devops
  • The Agile Flywheel
  • Exercise #3: Using Technical Debt as a Boundary Object in Devops
  • Bringing Organizations Together Through Enlightened Governance Loops

I look forward to meeting you in the workshop and learning from your experiences and insights!

Israel

Schedule Constraints in the Devops Triangle

leave a comment »

Last week’s post “The Devops Triangle” demonstrated the extension of Jim Highsmith‘s Agile Triangle to devops. The extension relied on adding compliance to the three traditional constraints of software development: scope, schedule, cost. A graphical representation of this extension is given in Figure 1.

Figure 1: Compliance as the Fourth Constraint in Devops Projects

This blog post examines how time/schedule should be governed in the devops context. It does so by building on the concluding observation in the previous post:

The Devops Triangle and the corresponding Tradeoff Matrix demonstrate how governance a la Agile can be extended to devops projects as far as compliance goes. The proposed governance framework however is incomplete in the following sense: schedule in devops projects can be a much more granular and stringent constraint than schedule in “dev only” projects.

For the schedule constraint in devops, I propose a schedule set.  It consists of  four components:

  • Lead Time or Engineering Time
  • Time to change
  • Time to deploy
  • Time to roll back

Lead Time/Engineering Time: These are customary metrics used in Kanban software development, as demonstrated in Figure 3.

Figure 3: The Engineering Time Metric Used by the BBC (David Joyce in the LSSC10 Conference)

Time to change: The amount of time it takes for the various stakeholders (e.g., dev, test, ops, customer support) to review the code to be deployed, approve its deployment and assign a time window for the deployment.

Time to deploy: The amount of time from (metaphorically speaking) pushing the Deploy “button” to completion of deployment.

Time to roll back: The amount of time to undo a deployment. (Rigorous that the engineering practices and IT processes might be, the time to roll back a deployment can’t be ignored – it is a critical risk parameter).

A graphical representation of these four schedule metrics together with the Devops Triangle is given in the figure below:

Figure 4: The Devops Triangle with a Schedule Set

Using hours as the common unit of measure, a typical schedule set could be {100, 48, 3, 2}. In this hypothetical example, it takes a little over 4 days to carry out the development of the code increment; 2 days to get approval for the change; 3 hours to deploy the code; and, 2 hours to roll back.

Whatever your specific schedule numbers might be, it is highly recommended you apply value stream mapping (see Figure 5 below) to your schedule set. Based on the findings of the value stream mapping, apply statistical process control methods like those illustrated in Figure 3 to continuously improving both the mean and the variances of the four schedule components.

Figure 5: An Example of Value Stream Mapping (Source: Wikipedia entry on the subject)

The Devops Triangle

with one comment

The Agile Triangles was introduced by Jim Highsmith as an antidote to the Iron Triangle. Instead of balancing development between cost, schedule and scope, the Agile Triangle strives to strike a balance between value, quality and constraints:

Figure 1 – The Agile Triangle (based on Figure 1-3 in Agile Project Management: Creating Innovative Products.)

Consider the Iron Triangle in the context of devops. Value, quality and constraints apply to IT operations as meaningfully as they apply to software development. IT can go beyond cost, schedule and scope to focus on value and quality just as the Agile software development team does. Between development and operations the specific tasks to be carried out change, but the principles embodies in the triangle remain invariant.

In addition to cost, schedule and scope, devops projects must cope with another constraint: compliance. For example, a bank that implements a ‘follow the sun’ strategy with respect to trading must finish reconciling transaction that took place in London before the start of trade in Wall Street. From the bank’s point of view, its IT department needs to be mindful of four constraints: compliance, cost, schedule and scope. This view is represented in Figure 2 below.

Figure 2 – The Devops Triangle

Balancing the four constraints – compliance, cost, schedule, and scope – is not a trivial task. However, just like the Agile Triangle, the Tradeoff Matrix used in Agile software development applies to IT. In its software development variant, the Tradeoff matrix is an effective tool to decide between conflicting constraints, as follows:

Table 1 – Tradeoff Matrix (based on Table 6-1 in Agile Project Management: Creating Innovative Products.)

For devops, the matrix is extended to include a compliance row and a Reluctantly Accept column as follows:

Table 2 – Tradeoff Matrix for Devops

The Devops Triangle and the corresponding Tradeoff Matrix demonstrate how governance a la Agile can be extended to devops projects as far as compliance goes. The proposed governance framework however is incomplete in the following sense: schedule in devops projects can be a much more granular and stringent constraint than schedule in “dev only” projects. The subject of schedule constraints in devops projects will be addressed in a forthcoming post.

Boundary Objects in DevOps

with one comment

Boundary Object by Cherice.

Source: Flickr; Chrice‘s Photostream

The following recommendation was given in the post How to Initiate a Devops Project:

For a DevOps project, start by establishing the technical debt of the software to be released to operations. By so doing you build the foundations for collaboration between development and operations through shared data. In the devops context, the technical debt data form the basis for the creation and grooming of a unified backlog which includes various user stories from operations.

I would like to augment this recommendation with a suggestion with respect to the mindset during the initiation phase. Chances are the IT folks feel outnumbered by the dev folks. It might or might not be a matter of optics, but recognizing and appreciating this mindset is will help a lot in getting the devops project on track.

Here is a simple example I heard from a participant in the June 25 devops day in Mountain View, CA. The participant with whom I talked is an IT ops person who tries to get ops aligned with  fairly proficient Agile development teams. She is, however, constrained with respect to the IT ops resources available to her. She simply does not have the resources required to attend each and every Scrum meeting as 25 such meetings take place every day. She strongly feels “outnumbered.”

Various schemes could be devised to enable meaningful participation of ops in the Agile process. The more important thing though is to be fully sensitized to the “outnumbered” feeling. The extension of Agile principles to ops will not succeed at the face of such a feeling.

Discussing the subject with my friend Andrew Shafer, he mentioned the effectiveness of boundary objects in such cross-organizational situations:

Boundary objects are objects which are both plastic enough to adapt to local needs and constraints of the several parties employing them, yet robust enough to maintain a common identity across sites. They are weakly structured in common use, and become strongly structured in individual-site use. They may be abstract or concrete. They have different meanings in different social worlds but their structure is common enough to more than one world to make them recognizable means of translation. The creation and management of boundary objects is key in developing and maintaining coherence across intersecting social worlds. [Source: Wikipedia].

As an example, the boundary object for the situation described in this post could be a set of technical debt criteria that make the code eligible for deployment from a product life cycle perspective. By so doing, it shifts the dialog from the process to the outcome of the process. Instead of working on generating IT resources in an “outnumbered” mode, the energy shifts toward developing a working agreement on the intrinsic quality of the code to be deployed.

Some technical debt criteria that could form the core of a devops boundary object are mentioned in the post Technical Debt Meets Continuous Deployment. Corresponding criteria could be used in the boundary object to satisfy operational requirements which are critical to the proper functioning of the code. For example, a ceiling on configuration drift in IT could be established to ensure an adequate operating environment for the code. A boundary object that contains both technical debt criteria and configuration drift criteria satisfies different concerns – those of dev and those of ops – simultaneously.

Written by israelgat

July 6, 2010 at 6:44 am

Follow

Get every new post delivered to your Inbox.

Join 38 other followers