The Agile Executive

Making Agile Work

Cloud Computing: Agile Deployment for Agile QA Testing

with 9 comments

Annie Shum‘s original thinking has often been quoted in this blog. Her insights are always characterized by seeing the world through the prism of fractals principles.  And, she always relentlessly pursues the connecting of the dots. In this guest post, she examines in an intriguing manner both the tactical and the strategic aspects of large scale testing in the cloud.

Here is Annie:

Cloud Computing: Agile Deployment for Agile QA Testing
Annie Shum twitter@insightspedia
Invariably, the underlying questions at the heart of every technology or business initiative are less about technology but more about the people (generally referred to as the users and consumers in the IT industry). For example, “How does this technology/initiative impact the lives and productivity of people?” or “What happens to the uses/consumers when they are offered new power or a new vehicle of empowerment?” Remarkably, very often the answers to these questions will directly as well as indirectly influence whether the technology/initiative will succeed or fail; whether its impact will be lasting or fleeting ; and whether it will be a strategic game-changer (and transform society) or a tactical short-term opportunity.
One can approach some of the Cloud-friendly applications, e.g. large scale QA and load stress testing in the Cloud, either from a tactical or from a strategic perspective. As aforementioned, the answer to the question “What happens to the uses/consumers when they are offered new power or a new vehicle of empowerment?” can influence whether a new technology initiative will be a strategic game-changer (and transform society) or a tactical short-term opportunity. In other words, think about the bacon-and-eggs analogy where the chicken is involved but the pig is committed. Look for new business models and innovation opportunities by leveraging Cloud Computing that go beyond addressing tactical issues (in particular, trading CapEx for OpEx). One example would be to explore transformative business possibilities stemming from Cloud Computing’s flexible, service-based delivery and deployment options.
Approaching Large-scale QA and Load Stress Testing in the Cloud from a Tactical Perspective
Nowadays, an enterprise organization is constantly under pressure to demonstrate ROI of IT projects. Moreover, they must be able to do this quickly and repeatedly. So as they plan for the transition to the Cloud, it is only prudent that they start small and focus on a target area that can readily showcase the Cloud potential. One of the oft-touted low hanging fruit of Cloud Computing is large scale QA (usability and functionality) testing and application load stress testing in the Cloud. Traditionally, one of the top barriers and major obstacles to comprehensive and high quality (iterative) QA testing is the lack of adequate computing resources. Not only is the shortfall due to budget constraint but also staff scheduling conflicts and the long lead time to procure new hardware/software. This can cause significant product release delays, particularly problematic with new application development under Scrum. An iterative incremental development/management framework commonly used with Agile software development, Scrum requires rapid successive releases in chunks, commonly referred to as splints. Sophisticated Agile users leverage this chunking technique as an affordable experimentation vehicle that can lead to innovationi. However, the downside is each iteration can lead to new testing needs and further compounding the QA woes.
By providing virtually unlimited computing resources on-demand and without up-front CapEx or long-term commitment, QA/load stress testing in the Cloud is a good starting point. More than likely it will turn out to be one of the least risky but quick ROI pilot Cloud projects for enterprise IT. In addition, the flexibility and on-demand elasticity of Cloud Computing meet the iterative nature of Agile on an on-going basis. Case in point, Franz Inc, opted for the Cloud solution when confronted with the dilemma of either abandoning their critical software product testing plan across dozens of machines and databases or procuring new hardware and software that would have been cost-prohibitive. Staging the stress testing study in Amazon’s S3, Franz completed its mission within a few days. Instead of the $100K capital expense for new hardware as well as additional soft costs (such as IT staff and other maintenance costs), the cost of the Amazon’s Cloud services was under $200 and without the penalty of delays in acquisition and configuration.
Approaching Large-scale QA and Load Stress Testing in the Cloud from a Strategic Perspective
While Franz Inc. leverages the granular utility payment model, the avoidance of upfront CapEx and long-term commitment for a one-off project, other entrepreneurs have decided to harness the power of on-demand QA testing in the Cloud as a new business model. Several companies, e.g. SOASTA, LoadStorm and Browsermob are now offering “Testing as a Service” also known as “Reliability as a Service” to enable businesses to test the real-world performance of their Web applications based on a utility-based, on-demand Cloud deployment model. Compared to traditional on-premises enterprise testing tool such as LoadRunner, the Cloud offerings promise to reduce complexity without any software download and up-front licensing cost. In addition, unlike conventional outsourcing models, enterprise IT can retain control of their testing scenarios. This is important because comprehensive QA testing typically requires an iterative process of test-analyze-fix-test cycle that spans weeks if not months.
Notably, all three organizations built their service offerings on Amazon EC2 infrastructure. LoadStorm launched in January 2009 and Browsermob (open source) currently in beta, each enable users to run iterative and parallel load tests directly from its Website. SOASTA, more established than the aforementioned two startups, recently showcases the viability of “Testing as a Service” business model by spawning 650 EC2 Servers to simulate load from two different availability zones to stress test a music-sharing website QTRAX. As reported by Amazon, after a 3-month iterative process of test-analyze-fix-test cycle, QTRAX can now serve 10M hits/hour and handle 500K concurrent users.
The bottom line is there are effectively two different perspectives: tactical (“involved”) versus the strategic (“committed”) and both can be successful. Moreover, the consideration of tactical versus strategic is not a discrete binary choice but a granularity spectrum that accommodates amalgamations of short term and long-term thinking. Every business must decide the best course to meet its goals.
i A shout out to Israel Gat for his insightful comment on chunking as a vehicle for innovation.

Invariably, the underlying questions at the heart of every technology or business initiative are less about technology but, as Clive Thompson of Wired Magazine observed, more about the people (generally referred to as the users and consumers in the IT industry). For example, “How does this technology/initiative impact the lives and productivity of people?” or “What happens to the uses/consumers when they are offered new power or a new vehicle of empowerment?” Remarkably, very often the answers to these questions will directly as well as indirectly influence whether the technology/initiative will succeed or fail; whether its impact will be lasting or fleeting ; and whether it will be a strategic game-changer (and transform society) or a tactical short-term opportunity.

One can approach some of the Cloud-friendly applications, e.g. large scale QA and load stress testing in the Cloud, either from a tactical or from a strategic perspective. As aforementioned, the answer to the question “What happens to the uses/consumers when they are offered new power or a new vehicle of empowerment?” can influence whether a new technology initiative will be a strategic or tactical. In other words, think about the bacon-and-eggs analogy where the chicken is involved but the pig is committed. Look for new business models and innovation opportunities by leveraging Cloud Computing that go beyond addressing tactical issues (in particular, trading CapEx for OpEx). One example would be to explore transformative business possibilities stemming from Cloud Computing’s flexible, service-based delivery and deployment options.

Approaching Large-scale QA and Load Stress Testing in the Cloud from a Tactical Perspective

Nowadays, an enterprise organization is constantly under pressure to demonstrate ROI of IT projects. Moreover, they must be able to do this quickly and repeatedly. So as they plan for the transition to the Cloud, it is only prudent that they start small and focus on a target area that can readily showcase the Cloud potential. One of the oft-touted low hanging fruit of Cloud Computing is large scale QA (usability and functionality) testing and application load stress testing in the Cloud. Traditionally, one of the top barriers and major obstacles to conducting comprehensive, iterative and massively parallel QA test cases is the lack of adequate computing resources. Not only is the shortfall due to budget constraint but also staff scheduling conflicts and the long lead time to procure new hardware/software. This can cause significant product release delays, particularly problematic with new application development under Scrum. An iterative incremental development/management framework commonly used with Agile software development, Scrum requires rapid successive releases in chunks, commonly referred to as splints. Advanced Agile users leverage this chunking technique as an affordable experimentation vehicle that can lead to innovation. However, the downside is the rapid accumulation of new testing needs.

By providing virtually unlimited computing resources on-demand and without up-front CapEx or long-term commitment, QA/load stress and scalability testing in the Cloud is a good starting point. Especially, the flexibility and on-demand elasticity of the Cloud Computing meet the iterative requirements of Agile on an on-going basis. More than likely it will turn out to be one of the least risky but quick ROI pilot Cloud projects for enterprise IT. Case in point, Franz Inc, opted for the Cloud solution when confronted with the dilemma of either abandoning their critical software product testing plan across dozens of machines and databases or procuring new hardware and software that would have been cost-prohibitive. Staging the stress testing study in Amazon’s S3, Franz completed its mission within a few days. Instead of the $100K capital expense for new hardware as well as additional soft costs (such as IT staff and other maintenance costs), the cost of the Amazon’s Cloud services was under $200 and without the penalty of delays in acquisition and configuration.

Approaching Large-scale QA and Load Stress Testing in the Cloud from a Strategic Perspective

While Franz Inc. leverages the granular utility payment model, the avoidance of upfront CapEx and long-term commitment for a one-off project, other entrepreneurs have decided to harness the power of on-demand QA testing in the Cloud as a new business model. Several companies, e.g. SOASTA, LoadStorm and Browsermob are now offering “Testing as a Service” also known as “Reliability as a Service” to enable businesses to test the real-world performance of their Web applications based on a utility-based, on-demand Cloud deployment model. Compared to traditional on-premises enterprise testing tool such as LoadRunner, the Cloud offerings promise to reduce complexity without any software download and up-front licensing cost. In addition, unlike conventional outsourcing models, enterprise IT can retain control of their testing scenarios. This is important because comprehensive QA testing typically requires an iterative process of test-analyze-fix-test cycle that spans weeks if not months.

Notably, all three organizations built their service offerings on Amazon EC2 infrastructure. LoadStorm launched in January 2009 and Browsermob (open source) currently in beta, each enable users to run iterative and parallel load tests directly from its Website. SOASTA, more established than the aforementioned two startups, recently showcases the viability of “Testing as a Service” business model by spawning 650 EC2 Servers to simulate load from two different availability zones to stress test a music-sharing website QTRAX. As reported by Amazon, after a 3-month iterative process of test-analyze-fix-test cycle, QTRAX can now serve 10M hits/hour and handle 500K concurrent users.

The bottom line is there are effectively two different perspectives: tactical (“involved”) versus the strategic (“committed”) and both can be successful. Moreover, the consideration of tactical versus strategic is not a discrete binary choice but a granularity spectrum that accommodates amalgamations of short term and long-term thinking. Every business must decide the best course to meet its goals.

P.S.  A shout out to Israel Gat for not only allowing me to post my piece today but for his always insightful comments in our daily email exchanges.

Advertisements

9 Responses

Subscribe to comments with RSS.

  1. I have always listened to Annie’s advice and here is another fine example. She was the first to enlighten me to the power of SOA over a decade ago, and was first again with Cloud Computing. While I must admit to being a bit of a hard sale at the outset from a concern for data security, now I realize she was right again. And this post has really sealed the deal for me. Scalability testing is a current concern of mine from a cost perspective and now Annie has put the solution right in front of me. Cheers once again!!!

    Mic Maner

    October 22, 2009 at 12:15 pm

  2. […] a key sentence fell between my editing fingers while publishing Annie Shum’s splendid post Cloud Computing: Agile Deployment for Agile QA Testing. Here is the corrected paragraph with the missing sentence highlighted: By providing virtually […]

  3. Israel,
    Great to see you and Annie talking about something so close to my heart. As the founder of BrowserMob, I’d like to correct a small error: BrowserMob was launched in 2008, shortly after SOASTA really turned it’s full attention on the cloud as well.

    We’re certainly not in beta; in fact we just launched a second product that is seeing amazing uptake: cloud-based website monitoring and we have thousands of users. Also, we are not opensource, though we do depend on opensource (namely Selenium, as does SOASTA.)

    Our big differentiation is our market focus (SMB) and our product approach (using real web browsers instead of HTTP simulations). The real browser approach requires about 50X more EC2 machines, which means we’re regularly launching hundreds or even thousands of machines in the cloud, but the pay-by-the-hour economics of the cloud it still possible for us to be very profitable.

    But I agree, Annie is spot on: these services are opening up new ways for agile teams to think about testing, as much of the traditional setup costs and overhead are removed.

    Patrick Lightbody

    October 23, 2009 at 7:10 am

  4. Thanks so much for correcting us, Patrick. I apologize for the inaccuracies.

    Please keep the two of us and the readers of this blog posted on your progress. It is heart warming to see companies like yours thrive.

    Best,

    Israel

    Israel Gat

    October 23, 2009 at 9:14 am

  5. No worries at all – thanks for the great blog!

    Patrick Lightbody

    October 23, 2009 at 9:16 am

  6. Labor of love…

    Israel

    Israel Gat

    October 23, 2009 at 9:29 am

  7. […] How appropriate it is that Annie has recently posted in this blog on the two threads coming together – Cloud Computing: Agile Deployment for Agile QA Testing. […]

  8. I really like this approach. Another significant win is that I can give the dev teams more development environments if they constrain the designs to cloud platforms and so start to reduce the complexity of my estate (even if the production systems run in a private cloud).

    With suitable test points, I can decouple the various layers of my application stacks so that the different half lifespans don’t interfere with each other and I can continuously reduce the unit cost of transactions for my applications.

    Who knows; when the FUD settles, I can just deploy the whole lot into a public cloud and no one will notice.

    Tim Coote

    June 30, 2010 at 11:31 am

    • Makes two of us, Tim… Annie Shum, the author of this guest post, is a real thought leader. In addition to guest posts here, you can read her musings on the Amdocs CTO blog.

      Best,

      Israel

      israelgat

      June 30, 2010 at 5:19 pm


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: