The Agile Executive

Making Agile Work

Posts Tagged ‘Continuous Integration

‘Super-Fresh’ Code

with 2 comments

Source: http://www.flickr.com/photos/21560098@N06/3636921327/

Misty Belardo published a great post/video clip on Social Media in which she describe the effect of the ‘Super-Fresh’ web on brands:

…  millions of people are creating content for the social web… the next 3 billion consumers will access the Internet from a mobile device. Imagine what that means for bad customer experiences! The ‘super-fresh’ web will force brands to engage with its customers…

I would contend we are also going to experience ‘Super-Fresh’ code in not too long a time. Such code is likely to emerge as the convergence of two overarching trends:

  1. Continuous Integration –> Continuous Deployment –> ‘Super Fresh’ Code. Sophisticated companies are already translating velocity in dev to competitive advantage through Continuous Deployment. ‘Super-fresh’ code is a natural next step.
  2. Open-sourcing –> Crowd-sourcing –> Expert-sourcing. Marketplaces for knowledge work expertise are becoming both effective and efficient. For example, uTest indicates “… 25,000+ testers in more than 160 countries.” A  marketplace for mobile application developers could probably be organized along fairly similar lines.

No doubt, complex software systems of various kinds will continue to be produced through more conventional processes for many years to come. However, ‘Super-Fresh’ code will establish itself as a new category. Code in this category will owe its robustness (and creativity!) to millions of people creating software and fixing it in extremely short time, not to process rigor.

In case you are still wondering about the premise, I would like to point out two corroborative facts:

  1. It is a small step from content to code.
  2. Various mobile applications are already developed and tested today in a different manner from the way web applications have been done.

A fascinating link exists between ‘Super-Fresh’ web and ‘Super-Fresh’ code. The dynamics (“Imagine what that means for bad customer experiences!”) Misty discusses in her blog post are a major driver for the evolution of knowledge work marketplaces and for the production of ‘Super-Fresh’ code.

A Devops Case Study

leave a comment »

An outline of my forthcoming Agile 2010 workshop was given in the post “A Recipe for Handling Cultural Conflicts in Devops and Beyond” earlier this week. Here is the case study around which the workshop is structured:

NotHere, Inc. Case Study

NotHere, Inc. is a $500M company based in Jerusalem, Israel. The company developed an eCommerce platform for small to medium retailers. Through a combination of this platform and its hosting data center, NotHere provides online store fronts, shopping carts, order processing, inventory, billing and marketing services to tens of thousands of retailers in a broad spectrum of verticals. For these retailers, NotHere is a one-stop “shopping” for all their online needs. In particular, instead of partnering with multiple companies like Amazon, Ebay, PayPal and Shopzilla, a retailer merely needs to partner with NotHere (who partners with these four companies and many others).

The small to medium retailers that use the good services of NotHere are critically dependent on the availability of its data center. For all practical purposes retailers are (temporarily) dead when the NotHere data center is not available. In recognition of the criticality of this aspect of its IT operations, NotHere invested a lot of effort in maturing its ITIL[i] processes. Its IT department successfully implements the ITIL service support and service delivery functions depicted in the figure below. From an operational perspective, an overall availability level of four nines is consistently attained. The company advertises this availability level as a major market differentiator.

In response to the accelerating pace in its marketplace, NotHere has been quite aggressive and successful in transitioning to Agile in product management, dev and test. Code quality, productivity and time-to-producing-code have been much improved over the past couple of years. The company measures those three metrics (quality, productivity, time-to-producing-code) regularly. The metrics feed into whole-hearted continuous improvement programs in product management, dev and test. They also serve as major components in evaluating the performance of the CTO and of the EVP of marketing.

NotHere has recently been struggling to reconcile velocity in development with availability in IT operations. Numerous attempts to turn speedy code development into fast service delivery have not been successful on two accounts:

  • Technical:  Early attempts to turn Continuous Integration into Continuous Deployment created numerous “hiccups” in both availability and audit.
  • Cultural: Dev is a competence culture; ops is a control culture.

A lot of tension has arisen between dev and ops as a result of the cultural differences compounding the technical differences. The situation deteriorated big time when the “lagging behind” picture below leaked from dev circles to ops.

The CEO of the company is of the opinion NotHere must reach the stage of Delivery over Development. She is not too interested in departmental metrics like the time it takes to develop code or the time it takes to deploy it. From her perspective, overall time-to-delivery (of service to the retailers) is the only meaningful business metric.

To accomplish Delivery over Development, the CEO launched a “Making Cats Work with Dogs[ii]” project. She gave the picture above to the CTO and CIO, making it crystal clear that the picture represents the end-point with respect to the relationship she expects the two of them and their departments to reach. Specifically, the CEO asked the CTO and the CIO to convene their staffs so that each department will:

  • Document its Outmodel (in the sense explored in the “How We Do Things Around Here In Order to Succeed” workshop) of the other department.
  • Compile a list of requirements it would like to put on the other group “to get its act together.”

The CEO also indicated she will convene and chair a meeting between the two departments. In this meeting she would like each department to present its two deliverables (world view of the other department & and the requirements to be put on it) and listen carefully to reflections and reactions from the other department. She expects the meeting will be the first step toward a mutual agreement between the two departments how to speed up overall service delivery.


[i] “Information Technology Infrastructure library – a set of concepts and practices for Information Technology Services Management (ITSM), Information Technology (IT) development and IT operations” [Wikipedia].

[ii] I am indebted to Patrick DeBois for suggesting this title.

© Copyright 2010 Israel Gat

Technical Debt Meets Continuous Deployment

with 11 comments

As you would expect in a conference entitled velocity, and in a follow-on devops day, speeding up things was an overarching theme. In the context of devops, the theme primarily manifested itself in lively discussions about the number of deploys per day. Comments such as the following reply to my post Ops Driven Dev were typical:

Conceptually, I move the whole business application configuration into the source code…

The theme that was missing for me in many of the presentations and discussions on the subject was the striking of a balance between velocity and quality. The classical trade-off in process control is between production rate and product quality (and safety, but that aspect [safety] is beyond the scope of this post). IMHO this trade-off applies to software just as it applies to mechanical or chemical processes.

The heart of the “deploy early and often” strategy hailed by advocates of continuous deployment is known deployment state to known deployment state. You don’t let the deployment evolve from one state to another before it has stabilized to a robust state. The power of this incremental deployment is in dealing with single-piece (or as small number of pieces as possible) flow rather than dealing with the effects of multiple-piece flow. When the deployment increments are small enough, rollback, root cause analysis and recovery are relatively straightforward if a deployment turns sour. It is a similar concept to Agile development, extending continuous integration to continuous deployment.

While I am wholeheartedly behind this devops strategy, I believe it needs to be reinforced through rigorous quality criteria the code must satisfy prior to deployment. The most straightforward way for so doing is through embedding technical debt criteria in the release/deploy process. For example:

  • The code will not be deployed unless the overall technical debt per line of code is lower than $2.
  • To qualify for deployment, code duplication levels must be kept under 8%.
  • Code whose Cyclomatic complexity per Java class is higher than 15 will not be accepted for deployment.
  • 50% unit test coverage is the minimal level required for deployment.
  • Many others…

I have no doubt whatsoever that code which does not satisfy these criteria might be successfully deployed in a short-term manner. The problem, however, is the accumulative effect over the long haul of successive deployments of code increments of inadequate quality. As Figure 1 demonstrates, a Java file with Cyclomatic complexity of 38 has a probability of 50% to be error-prone. If you do not stop it prior to deployment through technical debt criteria, it is likely to affect your customers and play havoc with your deployment quite a few times in the future. The fact that it did not do so during the first hour of deployment does not guarantee that such a  file will be “well-behaved” in the future.

mccabegraph.jpg

Figure 1: Error-proneness as a Function of Cyclomatic Complexity (Source: http://www.enerjy.com/blog/?p=198)

To attain satisfactory long-term quality and stability, you need both the right process and the right code. Continuous deployment is the “right process” if you have developed the deployment infrastructure to support it. The “right code” in this context is code whose technical debt levels are quantified and governed prior to deployment.

Beautiful Quality

leave a comment »

Figure 1: Agile Assessment – Quality (Source: QSMA)

Colleague and friend Michael Mah has kindly shared with me the figure above – quality assessment for a sample of Agile projects in the QSMA metrics database of more than 8000 software projects. The two red squares in this figure represent the recent results Mah measured on two projects carried out by Quick Solutions (QSI) – a Westerville, OH company offering a broad range IT services.

One to one-and-a-half standard deviation better than the mean might not seem like much to six sigma black belts. However, in the context of typical results we see in the software industry the QSI results are outstanding.  I have not done the exact math whether those results are superior to 95%, 97% or 98% of software projects in the QSMA database as the very exact figure almost does not matter when you achieve this level of excellence.

I asked Bart Murphy – QSI’s Vice President of Delivery and Operations – for the ‘secret sauce.’ Here is the QSI recipe:

For the projects referenced in Michael’s evaluation, our primary focus was quality…  Our team was tasked with not only a significant Innovation effort, but we also managed all aspects of supporting and stabilizing the application (production support, triage, infrastructure, database support, etc.).  Our ‘secret sauce’ was assembling a world class team and executing our Agile methodology.  We organized the team efforts to focus on the three major initiatives; Innovation, Stabilization (Technical Debt), and Production Support.  We developed a release plan and coordinated efforts to deploy releases that resolved a significant number of defects, introduced market differentiating features, and addressed massive amount of technical debt.  The team was able to accomplish this without introducing additional defects into the production system.   Our success can be attributed to the commitment from the business to understand our Agile methodology and being highly engaged throughout the project (co-located with the team).  In addition, the focus of quality that is integrated into our process through the use of Test Driven Development, Continuous Integration, Test Automation, Quality Assurance, and Show & Tells.  Lastly, this would not have been possible without the co-location of the entire team given the significant issues and time constraints for delivery.

Bart also provided me with a table  ‘a la Capers Jones’ in which he elaborates on the factors that helped QSI achieve these results and those that stood in the way. I will publish and discuss Bart’s table in a forthcoming post. As part of this post I will also compare the factors identified by Bart with those reported by Capers Jones.

How to Use Observations From Outside the Agile Process

with 2 comments

a9723 The Whorl of Architecture by tengtan.

Photo credit: tengtan (Flickr)

Most posts on technical debt in this blog emphasize the use of technical debt for strategic decision-making. In this post we will point out the use of technical debt in Agile teams at the tactical level. Specifically:

  • Every two weeks; and/or,
  • With every build.

Taking a close look at the various components of technical debt during the  bi-weekly iteration review meeting provides plenty of useful information to the process. For example, you might look for insights to explain the following:

  • Why is the unit test coverage figure going down?
  • Any particular reason the cyclomatic complexity figure has gone up?
  • Why is the figure of merit for design lower than the figure indicated in the previous iteration review meeting?
  • Many others…

The emphasis in this mode of operation is on guiding the retrospection. Plenty of good and valid reasons might exist for any of the trends mentioned above. However, observing the trends helps you ask the right questions, focusing on what happened during the iteration just completed. In conjunction with technical debt data from previous iteration review meetings, trends that characterize your software development project become visible. You may or may not need to change anything you are doing, but you become very conscious of any “let’s not change” decision.

An intriguing practice suggested by colleague and friend Erik Huddleston is to make technical debt a criterion for the build to pass. The build automatically fails if the technical debt figure has gone up. Or, if you are very focused on a specific aspect of technical debt such as complexity, you fail the build whenever the complexity figure of merit rises above  a certain pre-determined threshold. For example, you might fail a build in which the cyclomatic complexity per method has exceeded 4.

The power of failing a build whenever the technical debt arises is in utilizing the build as an exceptionally effective influence point. You instill the discipline of reducing technical debt one build at a time. If your team aggressively practices continuous integration, it will address technical debt issues multiple times a day. Instead of staring at a “mountain” of technical debt towards the release of a product, you chunk it to really small increments that get addressed “real-time.” For instance, a build that failed due to lack of comments can usually be fixed very quickly by the developer who “upset the apple cart” while the logic embedded in the code is fresh on his/her mind.

A good insight to the way the tactical use of technical debt techniques adds value is provided by the following observation: the technical debt data is observed from outside the Agile process. Hence, technical debt data  is nicely suited to guiding the process. If you think of the software engineering fabric as a virtual stack, the technical debt “layer” could be considered a layer above the Agile process.

10 Steps for Setting up an Agile Start-up

with 4 comments

Mapping the Agile thinking, theory and practices to the realities of the target company is a tricky part of making Agile happen in a sustainable manner. HubOne’s Nick Beaugeard, known to readers of this blog from his post Enterprise Product: $50,000 and 8 months – You Must be Kidding, shares with us his recipe for so doing in a start-up environment. He manages to weave the pragmatic details with the core principles behind software development in general and Agile methods in particular. For example, consider the following insight provided by Nick:

In fact, when developing an API, the unit tests are your clients!

Readers of this blog might want to compare and contrast the thoughts Nick expresses in this post with those of:

  1. Ryan Martens on prescriptive versus adaptive rollout of Agile (click here).
  2. Eric Ries on iterating on the problem definition and developing the customer base in parallel with iterating on the solution (Click here).

Here is Nick:

After my last post, where I discussed the concept of implementing the tools and process before you get people on board, I though it appropriate to provide some prescriptive guidance on how we achieved the process. This post is primarily aimed at start-ups where you have total control over your infrastructure, computers, network and internet connection. If there is enough interest, I’ll produce another post of how to perform the same, but in a corporate environment.

I believe, and my experience has shown, that preparing your work environment, tools and process before the team starts coding helps eliminate costly and lengthy discussions about tools and process. In fact, in my experience, most developers are pleasantly surprised to find the environment ready and working and slip into the processes extremely quickly.

So, please find below my ten steps. Following this process really helped us get up and running quickly. Whilst we used Microsoft Development tools, this equally applies to their open source equivalents, so feel free to substitute specific tools, just not the requirements and process.

  1. Authentication, Network and VPN. Setting up the core of your environment is critical. As you are more than likely working on secret software at the outset, you need good, auditable mechanisms for authentication and logon. In addition, we don’t want our developers to have to do anything except start their PC or laptop and login, and we really want them to be able to work remotely. If you don’t feel qualified or able to complete the steps below, any good local IT Pro should be able to set this all up for you. To perform this, we implemented the following:
    1. Network Connection – we are in Australia so our networks are not fantastic. We subscribed to a good ADSL 2+ plan (24Mb) with a 80Gb limit. We implemented a modern ADSLwireless modem/router and configured it correctly. This gave us acceptable internet connectivity.
    2. Domain – We implemented a domain controller running on Windows Server 2008. This gave us corporate authentication, auditing and identification. The domain controller was hosted on our private network (see 1.c)
    3. Routing and VPN –our internal development network needed to be protected from internet connectivity so we implemented a Windows Server 2008 machine with two network cards (called multihomed). One card was connected to the router and one connected to an Ethernet switch. We used a private IP subnetfor our development Local Area Network (LAN) and enabled Microsoft Routing and Remote Access. This gave us the ability to authenticate domain users to VPN into the private network for remote working. We then configured our router to allow access to the server for VPN Access.
    4. DNS –one of the issues you face with ADSL is that your Internet IP Address changes often. The solution for this was to use a solution from DYNDNS (www.dyndns.com) which allowed us to register a host name, coupled with a client application which ensured our host name for VPN always pointed at the correct IP Address.
    5. DHCP –it’s a real pain, especially when using VPN when your client machines are configured for static IP addresses. We used Microsoft Dynamic Host Configuration Protocol (DHCP) to ensure every machine had a unique IP Address and that our networking became just “plug-and-play”
  2. Version Control and issue tracking – in my experience, there are four critical systems needed for any software development team. These are:
    1. Version Control – Also known as source repositories, these systems allow control over check-in, check-out and versioning of source files. I cannot recall the number of times we backed out an individual source file to a previous “working version”. Without this in place, I can guarantee you’re going to struggle maintaining a good code base!
    2. Bill of Materials– the way I use these is to highlight each of the key deliverables in a project. When building Enterprise Profile Management, our BOM had 345 separate items, everything from the corporate website, to graphic design, to core components of the API. The Bill of materials is a way to track the overall progress of the project. Each item in the Bill of Materials should have at a minimum a title, description, owner, due date, % complete and %tested. We also use the bill of materials to determine our release criteria (more on this later).
    3. Build System – the biggest mistake a development manager can make is to not build the software regularly. We build our software system every time there is a new check-in (called continuous integration). Even with a very small team, the ripple effect of changes could go un-noticed for ages without building regularly. As an aside, we also have a Nabaztag (www.nabaztag.com) bunny which tells us whenever there is a build, whose check-in caused it and whether it succeeded or failed. While this is really annoying, it focuses the developers on good check-ins. We might also make the person who broke the builds buy us a round of beer, but that’s a secret!
    4. Issue Management – More important than email, IM, or indeed any other form of communication in the development team is issue management. I believe that in  a project of any size, you’ll be hard pressed to ever finish if you don’t have good issue management. Issues should contain a title, description, history, assignee, status and indicate which version/iteration of which product the issue applies to. I don’t actually think a spreadsheet will cut it here. If there’s one investment you make, make sure it’s issue management.
      We used Microsoft Visual Studio Team System for all of the above. Being an ex-microsoft product team member, I am familiar with the way the product works, but there are lots of plug-ins available for scrum, agile (MSF) and CMMI. To do this on a startup budget, we were able to join the Microsoft BizSpark program (http://www.bizspark.com) which gave us instant access to Microsoft’s developer tools. I’d highly recommend taking a look at that program!
  3. Backup – Now is the time to do a backup and recovery operation. You have no real data in the system and how you installed it all is probably fresh in your mind. Trust me, every first recovery operation fails. You need to imagine your office has been hit by lightning and you have no servers, and just a backup. If you can successfully recover your environment in under 24 hours, you’re in a good place. Document how to do it and test it regularly.
    When we were developing Enterprise Profile Management, our server with all issue management, reporting, builds and version control failed (the processor fried). We were taking backups, but the restore failed. It took me 22 hours to perform a forensic recovery of our production platform. Luckily the developers could work offline, but we still introduced a ton of integration bugs. Don’t skimp on backup.
  4. Email, IM and Web Conferencing – You’re going to be working remotely at some time, whether you think you will or not. We quickly implemented the following tools:
    1. Email – Google Gmail for your domain (www.google.com/apps)
    2. IM – Windows Live Messenger and Skype (www.live.com, http://www.skype.com)
    3. Web Conferencing – Dim Dim (www.dimdim.com)
      Note: there are lots of other tools out there, we just chose these (with little science, but they’ve worked well for us)
  5. Coding Standards – Getting at least some coding standards in place before you write any code is really important. Good Version Control Systems should be able to validate code against your standards on check-in. My key coding standards are:
    1. No String Literals – All strings must be externalised in resource files for later localization (a process called globalization)
    2. Commenting – I insist that every class, method, property, event and interface be commented. There’s method in my madness here; obviously code commenting is important but with proper commenting (especially in C#) and tools like sandcastle, you can automatically create documentation like ours at http://api.hubone.com.
    3. No Short Cuts – This one was a little contentious – Our code will in the future (hopefully) be worked on by people who had no idea of the concept. Writing out If…then constructs and property accessors in long-hand makes the code much more readable for novices, junior programmers, support teams and architects alike.
    4. Unit Test Everything –I insist that every method, property, event etc, etc has an associated unit test. These we execute as a part of the build. My goal is to have over 80% of any code written covered by unit tests. In fact, when developing an API, the unit tests are your client! Although this increases the programming effort up front, it actually reduces the total time taken to ship the product. I havelost count of how many times our massive unit test library has saved us from nasty regressions that we could never have found from UI testing.
  6. Write some code –Don’t worry, you don’t need to be the world’s best developer to write some code. All you need to do is think carefully about the different items in the Bill of Materials and start prototyping the methods, properties and events that will make it all work. Ensure you adhere to your coding standards. The goal of this exercise is to effectively build a skeleton of your application before the developers get on board. This can make them hyper productive when they start. They’ll probably end up getting rid of every line of your code, but it will get them in the rhythm of how you want the platform to look. As the architect of our platform, I used this period to prototype and test all of my assumptions of how things would work. By the time the developers got on board, I had a passable working prototype of an API with unit tests and coding standards, although there is almost none of my code in the final product, this enabled us to get running really quickly.
  7. Don’t write a test plan – Write the user documentation instead, and use this to test your application. That way you’ll know your application does everything the user guide says it does and you won’t double up on a test plan and the user guide (which should end up being pretty similar anyway). I wrote our user guide because I knew what I wanted the product to do and it’s also a fantastic way (sometimes they only way) you get to communicate with your customers.
  8. Give ownership, responsibility and praise – You will have no choice. As you get developers, testers, writers and others on board, you must give them ownership of components. I find that if I give someone total ownership, they always deliver. People are proud when they do great work and if they feel and really have ownership over something, it will be their best work. Not only that, but they’ll pull out every stop to impress you, the team and your customers. If you hire correctly, you’ve got great professionals in the team. Respect them as professionals and the results will be awesome.
  9. The best wire framing tools are not wire framing tools – You’re a start-up, right? Then you’ve discussed your ideas with potential customers and you deeply understand their needs. You could sit down with any number of wire framing tools and attempt to design components. How about you take a different approach? Write functional code which does what you (and therefore your customers want) – Developers find it far easier to code from a working model and they can re-use portions of your code. All of this streamlines the process and you can take your models to customers (with caveats on your poor coding skills and lack of stylistic ability).
  10. Constantly review and drive – We spend some time every day in front of a whiteboard discussing options, vision, checklists and almost everything else, but in a start-up, it’s not really a democracy. I run my start-ups like a benevolent dictatorship – I hold the final decision, but let everyone have input. I’m the one talking to customers, and sometimes I bring customers in to explain a requirement, but at the end of the day, how it works is up to me (I have the most to lose if it doesn’t work!)

Notice here, we never had a specification, requirements specification or design, we build the code a piece at a time and made it work. We’ve gone back and documented some core functionality, flowcharts and features, but at the outset, it’s more important to make it work. I’ll share a caveat, though, I’m also the software architect so I can hold the requirements for scalability, security and reliability, and guide the team to accomplish those goals.

Hiring the team can happen sometime between points 7 and 8, I don’t recommend you do it before!

If you’d like more information, please feel free to get in touch at nick@hubone.com, and if you’d like a copy of the software this built, we can be found at http://www.hubone.com.