Posts Tagged ‘Agile’
On January 30, 2012 12:00 pm EST, colleague and friend Hubert Smits and I will be doing a Cutter webinar entitled “Big Agile” is More than Just a Software Method. We will follow on in February with a “Big Agile” issue of the Cutter IT Journal [CITJ] for which I am the guest editor. Coming April we are likely to discuss the topic some more in the Cutter Summit.
The heart of the webinar can be summarized in the following words:
Small is beautiful in software. While big software might not be beautiful, more often than not, it’s in the nature of what needs to be accomplished. This contrast between the beauty of small and the requirements of the big generates systemic tension in many software projects, organizations, and companies. Resolving this conflict is the focus of this webinar.
Discussing the webinar and the follow-on CITJ issue with various folks in the Agile, Lean and Kanban movement(s), I became painfully aware of the very many interpretation folks associate with the the term “Big Agile.” Hence, I would like to say a few words on what it means to me. I am taking the very pragmatic view of a client on the subject. A VP of one department or another is chartered to implement an Agile roll-out at scale. The roll-out might not include all the teams. Nor might it include all functions within the company. The roll-out, however, affects a significant number of folks. Focusing the roll-out at the team level is not sufficient.
The typical VP that I run into under such circumstances is not an expert on software methods (and usually acknowledges it). He/she, however, is smart enough and experienced enough to understand that scale matters. He/she knows or feels intuitively that Big Agile is akin to Big Data. Data is data is data, but when it is Big Data you need to address various aspects that do not manifest themselves in dealing with ordinary amounts of data. Likewise for Agile IMHO.
We plan to make the webinar very interactive. Anne Mullaney will start with a few ‘warm up’ questions to enable us to ‘dig’ into the subject, thence turn it over to questions from the audience. We plan to take the discussion to wherever the participants might want to.
I hope you will join us whether you love, hate or indifferent to the term “Big Agile.” We are expecting a lively discussion…
A few month ago Chris Sterling and I were carrying out a Cutter Technical Debt Assessment and Valuation engagement for a venture capitalist who was considering a certain company. We discovered various things in the code of this company. More noteworthy, my deep domain expertise led to Chris discovering the great Greek dish Moussaka.
I have eaten a lot of good Moussakas over the years. Even against this solid gastronomic background I can’t forget how the eyes of Chris lit up when he took the first bite. It took him only a tiny little time to get on his iPhone and tweet on the culinary aspects of our engagement. I then knew it was going to be a very successful engagement…
The relationship with Chris deepened since this episode. For example, in collaboration with Brent Barton Chris contributed a great article to the forthcoming issue of the Cutter IT Journal on Technical Debt. In this article Chris and Brent demonstrate how technical debt techniques can be applied at the portfolio level. They make the reader step into the shoes of the project portfolio planner and walk him through their approach to enhancing the decision-making process by using the software debt dashboard.
Chris has just published an excellent post entitled “Using Sonar Metrics to Assess Promotion of Builds to Downstream Environments” in Getting Agile and was kind enough to suggest I cross-post it in The Agile Executive. Here it is (please note that the examples given below by Chris have nothing to do with the engagement described above):
“For those of you that don’t already know about Sonar you are missing an important tool in your quality assessment arsenal. Sonar is an open source tool that is a foundational platform to manage your software’s quality. The image below shows one of the main dashboard views that teams can use to get insights into their software’s health.
The dashboard provides rollup metrics out of the box for:
- Duplication (probably the biggest Design Debt in many software projects)
- Code coverage (amount of code touched by automated unit tests)
- Rules compliance (identifies potential issues in the code such as security concerns)
- Code complexity (an indicator of how easy the software will adapt to meet new needs)
- Size of codebase (lines of code [LOC])
Before going into how to use these metrics to assess whether to promote builds to downstream environments, I want to preface the conversation with the following note:
Code analysis metrics should NOT be used to assess teams and are most useful when considering how they trend over time
Now that we have this important note out-of-the-way and, of course, nobody will ever use these metrics for “evil”, lets discuss pulling data from Sonar to automate assessments of builds for promotion to downstream environments. For those that are unfamiliar with automated promotion, here is a simple, happy example:
A development team makes some changes to the automated tests and implementation code on an application and checks their changes into source control. A continuous integration server finds out that source control artifacts have changed since the last time it ran a build cycle and updates its local artifacts to incorporate the most recent changes. The continuous integration server then runs the build by compiling, executing automated tests, running Sonar code analysis, and deploying the successful deployment artifact to a waiting environment usually called something like “DEV”. Once deployed, a set of automated acceptance tests are executed against the DEV environment to validate that basic aspects of the application are still working from a user perspective. Sometime after all of the acceptance tests pass successfully (this could be twice a day or some other timeline that works for those using downstream environments), the continuous integration server promotes the build from the DEV environment to a TEST environment. Once deployed, the application might be running alongside other dependent or sibling applications and integration tests are run to ensure successful deployment. There could be more downstream environments such as PERF (performance), STAGING, and finally PROD (production).
The tendency for many development teams and organizations is that if the tests pass then it is good enough to move into downstream environments. This is definitely an enormous improvement over extensive manual testing and stabilization periods on traditional projects. An issue that I have still seen is the slow introduction of software debt as an application is developed. Highly disciplined technical practices such as Test-Driven Design (TDD) and Pair Programming can help stave off extreme software debt but these practices are still not common place amongst software development organizations. This is not usually due to lack of clarity about these practices, excessive schedule pressure, legacy code, and the initial hurdle to learning how to do these practices effectively. In the meantime, we need a way to assess the health of our software applications beyond just tests passing and in the internals of the code and tests themselves. Sonar can be easily added into your infrastructure to provide insights into the health of your code but we can go even beyond that.
The Sonar Web Services API is quite simple to work with. The easiest way to pull information from Sonar is to call a URL:
This will return an XML response like the following:
248390 com.adobe:as3corelib AS3 Core Lib AS3 Core Lib PRJ TRK flex 1.0 2010-09-19T01:55:06+0000 technical_debt_ratio 12.4 12.4% Within this XML, there is a section called that includes the value of the metric we requested in the URL, “technical_debt_ratio”. The ratio of technical debt in this Flex codebase is 12.4%. Now with this information we can look for increases over time to identify technical debt earlier in the software development cycle. So, if the ratio to increase beyond 13% after being at 12.4% 1 month earlier, this could tell us that there is some technical issues creeping into the application.
Another way that the Sonar API can be used is from a programming language such as Java. The following Java code will pull the same information through the Java API client:
Sonar sonar = Sonar.create("http://nemo.sonarsource.org"); Resource commons = sonar.find(ResourceQuery.createForMetrics("248390", "technical_debt_ratio")); System.out.println("Technical Debt Ratio: " + commons.getMeasure("technical_debt_ratio").getFormattedValue());
This will print “Technical Debt Ratio: 12.4%” to the console from a Java application. Once we are able to capture these metrics we could save them as data to trend in our automated promotion scripts that deploy builds in downstream environments. Some guidelines we have used in the past for these types of metrics are:
- Small changes in a metric’s trend does not constitute immediate action
- No more than 3 metrics should be trended (the typical 3 I watch for Java projects are duplication, class complexity, and technical debt)
- The development should decide what are reasonable guidelines for indicating problems in the trends (such as technical debt +/- .5%)
In the automated deployment scripts, these trends can be used to stop deployment of the next build that passed all of its tests and emails can be sent to the development team regarding the metric culprit. From there, teams are able to enter the Sonar dashboard and drill down into the metric to see where the software debt is creeping in. Also, a source control diff can be produced to go into the email showing what files were changed between the successful builds that made the trend go haywire. This might be a listing per build and the metric variations for each.
This is a deep topic that this post just barely introduces. If your organization has a separate configuration management or operations group that managed environment promotions beyond the development environment, Sonar and the web services API can help further automate early identification of software debt in your applications before they pollute downstream environments.”
Thank you, Chris!
An outline of my forthcoming Agile 2010 workshop was given in the post “A Recipe for Handling Cultural Conflicts in Devops and Beyond” earlier this week. Here is the case study around which the workshop is structured:
NotHere, Inc. Case Study
NotHere, Inc. is a $500M company based in Jerusalem, Israel. The company developed an eCommerce platform for small to medium retailers. Through a combination of this platform and its hosting data center, NotHere provides online store fronts, shopping carts, order processing, inventory, billing and marketing services to tens of thousands of retailers in a broad spectrum of verticals. For these retailers, NotHere is a one-stop “shopping” for all their online needs. In particular, instead of partnering with multiple companies like Amazon, Ebay, PayPal and Shopzilla, a retailer merely needs to partner with NotHere (who partners with these four companies and many others).
The small to medium retailers that use the good services of NotHere are critically dependent on the availability of its data center. For all practical purposes retailers are (temporarily) dead when the NotHere data center is not available. In recognition of the criticality of this aspect of its IT operations, NotHere invested a lot of effort in maturing its ITIL[i] processes. Its IT department successfully implements the ITIL service support and service delivery functions depicted in the figure below. From an operational perspective, an overall availability level of four nines is consistently attained. The company advertises this availability level as a major market differentiator.
In response to the accelerating pace in its marketplace, NotHere has been quite aggressive and successful in transitioning to Agile in product management, dev and test. Code quality, productivity and time-to-producing-code have been much improved over the past couple of years. The company measures those three metrics (quality, productivity, time-to-producing-code) regularly. The metrics feed into whole-hearted continuous improvement programs in product management, dev and test. They also serve as major components in evaluating the performance of the CTO and of the EVP of marketing.
NotHere has recently been struggling to reconcile velocity in development with availability in IT operations. Numerous attempts to turn speedy code development into fast service delivery have not been successful on two accounts:
- Technical: Early attempts to turn Continuous Integration into Continuous Deployment created numerous “hiccups” in both availability and audit.
- Cultural: Dev is a competence culture; ops is a control culture.
A lot of tension has arisen between dev and ops as a result of the cultural differences compounding the technical differences. The situation deteriorated big time when the “lagging behind” picture below leaked from dev circles to ops.
The CEO of the company is of the opinion NotHere must reach the stage of Delivery over Development. She is not too interested in departmental metrics like the time it takes to develop code or the time it takes to deploy it. From her perspective, overall time-to-delivery (of service to the retailers) is the only meaningful business metric.
To accomplish Delivery over Development, the CEO launched a “Making Cats Work with Dogs[ii]” project. She gave the picture above to the CTO and CIO, making it crystal clear that the picture represents the end-point with respect to the relationship she expects the two of them and their departments to reach. Specifically, the CEO asked the CTO and the CIO to convene their staffs so that each department will:
- Document its Outmodel (in the sense explored in the “How We Do Things Around Here In Order to Succeed” workshop) of the other department.
- Compile a list of requirements it would like to put on the other group “to get its act together.”
The CEO also indicated she will convene and chair a meeting between the two departments. In this meeting she would like each department to present its two deliverables (world view of the other department & and the requirements to be put on it) and listen carefully to reflections and reactions from the other department. She expects the meeting will be the first step toward a mutual agreement between the two departments how to speed up overall service delivery.
[i] “Information Technology Infrastructure library – a set of concepts and practices for Information Technology Services Management (ITSM), Information Technology (IT) development and IT operations” [Wikipedia].
[ii] I am indebted to Patrick DeBois for suggesting this title.
© Copyright 2010 Israel Gat
- The ascendance of Agile portfolio management in a world characterized by loosely coupled processes
- Devops dynamics are becoming more and more characteristic of end-to-end Agile/Kanban patterns
- Viral spread of technical debt metrics in software governance
- Increasing use of boundary objects in the enterprise context
The workshop is structured around three case studies/exercises that will take about two-thirds of the allotted time (the morning of August 9). The other third provides the theory and tools to be used in the three workshop exercises and (hopefully) in many future engagements participants in the workshop will carry out. Deep technical knowledge is not required – the workshop targets any Agile practitioner who has conceptual grasp of culture, software development, IT operations and portfolio management.
The #1 takeaway from the presentation is the details you need to know about creation and capture of lasting value through end-to-end Agile initiatives.
Here is the workshop agenda (still subject to some minor tweaking):
- Introduction to Cultural Framework
- Exercise #1: Strengths and Weaknesses of Your Culture
- Change Behavior, Not Culture
- When Organizations Clash
- Exercise #2: Conflicts in Devops
- The Agile Flywheel
- Exercise #3: Using Technical Debt as a Boundary Object in Devops
- Bringing Organizations Together Through Enlightened Governance Loops
I look forward to meeting you in the workshop and learning from your experiences and insights!
Here is a compilation of summaries of Amazon reviews on the Kindle edition of The Concise Executive Guide to Agile:
If you are such a decision maker, and like most are hard-pressed for time, then you must read this book. Dr. Gat cuts right to the chase and gives pearls of wisdom – that come from having “done it himself” when it comes to the daunting task of TRANSFORMING an organization into successfully applying Agile methods. If you’re a team leader and you need executive support and buy-in, then buy this book for the key people whom you value the most. Give it to them as a gift. They will thank you for it!
“The Concise Executive Guide to Agile” by Israel Gat is a must-read for enthusiasts of agile methods at all levels of the organization. Israel has well over 30 years of industry experience, has his PhD in Computer Science, and has helped manage an agile rollout involving 1,000 software developers. While there are many technical and managerial treatises on agile methods emerging in increasing frequency all of the time, Israel’s new guide is meant to explain the essence of agile methods to R&D, Marketing, Sales, Program Management, Professional Services, Customer Support, Finance, or IT folks.
Dr. Gat goes beyond the traditional software development process literature and broadens the case for the Agile organization, with effectiveness, efficiency, and integrity – much like the Agile development process described by Israel.
A must read for any organization leader dependant on software whether it be for software for internal operations or software features built for customer sales. Israel’s breadth and depth of both knowledge and experience shine through as he provides practical advice to help you understand what Agile can and can not do for your organization.
For me, this book provided something unique, which is a look into the way high level executives think about Agile.
This book rapidly cuts through the layers of confusion surrounding Agile development and focuses on what you “NEED” to know. Dr. Gat’s vast experience shines through as he elegantly simplifies the process.
Israel’s experience and wisdom in transforming a software development culture in a very large multi-site virtual team environment is well captured… Israel has done a great job in transforming his experience into a guide. That is the beauty of the book.
This is the “must-read” book if you’re an executive charged with Agile transformation, product development, or business strategy. The insights from Gat cover all three. Most companies talk a good Agile game. If you want results, however, please pay heed to the advice in this book.
The book will help you to flexibly navigate the hurdles inherent in selling and succeeding with Agile in your business: i.e. existing methodologies, outsourcing, diverse geographies, resistance to change, and the rest. They are all explained here by someone who has apparently already successfully fought the battles. A great read!
He has been there, and this book shares the unique insights of a senior manager who guided a large organization through the change process. If you are in this seat in your organization then this book is for you.
Israel distills his broad experience and many years of wisdom rolling out Agile into a succinct overview that is deep in knowledge, but accessible.
I’ve been a fan of Israel’s “The Agile Executive” blog for some time and had high hopes for this book… and the book didn’t disappoint!
While I was looking for answers; what I found instead was much more interesting. Israel was able to collect all the wisdom of doing large scale agile rollouts into one document, highlighting the geographic features along the way. That is a lot harder to do than simply proclaiming one correct path, and, speaking from experience, more helpful to the reader than they know.
You can read the full reviews here.
A fascinating difference exists between Agile and Business Service Management (BSM). Agile emphasizes continuous flow of value to the customer. In contrast, BSM focuses on the business – it aligns the deliverables of IT to the enterprise’s business goals. Subtle that the difference might be, the two methods evolved along quite different lines in spite of the common denominator – dealing with software.
In this guest post, Alan Shalloway – Founder and CEO of NetObjectives – discusses the implications of focusing on the business as distinct from focusing on the customer. His discussion is part of a few thought-provoking threads he weaves around the Agile Manifesto. Alan perceives the Manifesto a product of the times. He thinks aloud whether today’s circumstances require a revised manifesto.
Alan is a man of passion. While I do not always agree with him, I have a lot of respect for his quest to find the deeper truths. Furthermore, I always learn from him. Whether you agree or disagree with the opinion Alan articulates in this post, “listening” to his thoughts is well worth your time.
Here is Alan:
The Agile Manifesto was a watershed event that has forever changed the landscape of software development. So profound a positive impact of it has had, that few challenge whether it was actually correct. Manifestos are often a statement in reaction to something prevalent that needs changing. This makes them very topical and temporal – and the exact intention needs to be restated when the landscape against which it was drawn has changed. The Agile Manifesto states:
“Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on
the right, we value the items on the left more.”
At the time of its edict, this was profound and well stated. Too many software teams were:
- Following a process dictated to them from outside their team
- Managing according to an extensive set of artifacts that recorded where they supposedly were well before software existed
- Were given a set of requirements to meet with little opportunity to discuss real needs (note, points 3 and 4 in the manifesto address this point in two ways – first recognizing that customer collaboration is necessary to discover their true needs and second that it is essential to take advantage of newly discovered information)
The manifesto represented a new paradigm from which to work – one in which the team would have better control over its destiny and where it was recognized that one had to make incremental, iterative movement towards one’s goals – both in discovering the true goal and in implementing it.
Unfortunately, the perspective from which the manifesto was created, or at least the methods which first followed the manifesto, have been extremely team centric. Not a surprise, given the paradigm at the time gave development teams too little say in their own methods. The impact of this has been, not surprisingly, success at the teams and difficult beyond the team. It is almost axiomatic now that companies will have successful team pilots only to bog down in their enterprise agile adoption efforts or even revert back to earlier methods.
I say “not surprisingly” because several things have been left out of the agile manifesto. These are:
- The role of management
- The role of process
- The role of planning
- And indirectly, the role of guiding principles
It could also be said that the driver for agile development is misplaced. I do not believe “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.” This makes software groups customer driven, not business driven.
There is a subtle, but important difference. Basically, conscious or not, the Agile Manifesto is driven with a team-centric view of satisfying customers – business and management play mostly secondary roles. Unfortunately, there is a significant difference between customer driven and business driven (see the figure entitled Alignment with Vision of Business to the right). This is not apparent at the team level (the team is supposed to satisfy the customer). But definitely at the business level. Not surprising, however, since the manifesto is a team perspective it states things in terms of customer value.
Ironically, it is the over-focus on the team, however, that is robbing teams of one of their greatest tools. Clearly any method must have respecting people and providing those doing the work the ability to choose how they do their work. But this does not mean that process isn’t essential or that attending to certain laws of software development is optional. Rather it means that process can’t be imposed on teams since to do so would both rob them of respect and almost certainly be the wrong way to do things – who knows more about how to get work done than the people doing the work themselves?
But “Individuals over process” as the first line has come to be called, makes it sound like people are it. I do not think so – and I think this mindset has caused a lot of damage in many ways.
There is great evidence that the best approach is not merely get smart people onto a team and have them figure out how to solve their problems. They must be properly equipped to do so. Just being smart doesn’t mean you can solve the challenges facing you. This should be readily apparent, but in many ways, the Agile community has mostly ignored it. Actually, there is not really an Agile community any more – there are factions that have significantly different beliefs. For example, XP has long recognized the need for technical practices in Agile while the Scrum community is only just starting to get into what these are.
However, except for the Lean/Kanban community, few Agilists seem to espouse discovering and following the laws of product development flow (or even recognize their existence). This, in my mind, has led to the low rate of success in scaling Scrum to the enterprise*. Ironically, it is the over-focus on people that leads many in the Scrum community to assert this lack of success is a lack willingness to take the effort to improve. This is not surprising – if it’s up to the team to succeed, then when they don’t it must be something wrong with the team or their management when they fail. I think not. I think it is the lack of understanding of the principles of software development flow.
These laws are not new. Don Reinertsen, in his iconic book Managing the Design Factory lays out much of the rules of product development. His more recent book The Principles of Product Development Flow: Second Generation Lean Product Development he lays out 175 of these principles.
To me, true respect for people means that one must equip them with what they need to get their job done. Our thinking in the Agile community should change from “People over process” to one of “People times process.” This phrase emphasizes that if either are low, you get a low productivity. Process does not ensure success. But a poor process requires heroes to succeed. We’d like good, motivated, well-intentioned people to be able to succeed.
Our new agile perspective needs to include an understanding of what teams need to know to do their work. This opens up a role for managers to actively help teams get their job done and to coach them when they have challenges or lose their way. While I will always be thankful for the Agile Manifesto, I am looking for a Business Agile Manifesto that will expand the focus from the team to the entire enterprise.
 Ken Schwaber, iconic leader of the Scrum community has said that “I estimate that 75% of those organizations using Scrum will not succeed in getting the benefits that they hope for from it.”
 I first heard this from Don Reinertsen. Before that I used to say “People plus process.”