For decades, developers and project managers took comfort in the waterfall method of application development.

Resource allocation seemed intuitive, management had an easily recognisable progress path, and eventually, customers were delivered new products and services significantly better than they previously used.

But there were a few problems along the way. Projects often ran over time and over budget, management often complained that the final outcomes failed to deliver the goals described in the business plan, and by the time the new products or services arrived in the market- sometimes years later, the market – and the customers – had moved on.

These days, new approaches like Agile, Lean, DevOps and Continuous Development have collapsed development into a more iterative and rapid cycle.

Now, the smartest organisations are leveraging these new approaches to introduce a culture of experimentation. This is an important means of responding to the ever-growing expectations of customers who take their best experience in any context and apply it to every context.

The issue is explored in a white paper developed by Optimizely called “Getting started with product experimentation”.

It’s a theme that will be developed in more detail in the upcoming Optimizely Modernize conference in Sydney next week.

According to Optimizely, ”Some of the most revolutionary new products and features were originally controversial, or counter-intuitive. Experimentation transforms the need for debate into an opportunity for discovery. With it, teams can now evaluate ideas and make decisions based on real, usable data.”

A starting assumption of the white paper is that it is not enough these days to simply optimise the performance and reliability of products.

Instead, Optimizely argue, organisations must optimise the actual benefits of their software as well.

However, in the white paper they caution, “Without sufficient experimentation product developers won’t have adequate information at hand to make the best product decisions possible.”

The best way forward, they say, is to expand experimentation capabilities and strengthen software development process.

To set the scene, many teams already embrace principles such as Agile, Lean, DevOps, and Continuous Development to ship quality software faster. In fact, Optimizely note that agile is now the norm with 94 per cent of companies saying that all or some of their development teams currently use agile methods.

By breaking applications into multiple services and shipping smaller changes more frequently, software teams are looking to mitigate risks and more rapidly to customer needs.

Many organisations have also migrated from monolithic applications to microservices architectures. Yet while all these changes are important and certainly lead to greater efficiency, they are not enough in themselves to confer genuine competitive advantage.

Instead, argue Optimizely, “[The approach] must also deliver real value to end users and customers faster and better than any other product. That’s where experimentation-driven product development comes in.”


The shift to experimentation is a trail already blazed by some of the world’s most successful technology companies such as Google, Facebook, Microsoft, and Amazon.

These companies all run thousands of experiments each year and have scaled and managed their businesses by using experimentation as a core part of their product development process.

While not every aspect of their success is replicable, happily their adoption of an experimental approach contains lessons for all companies who build software to deliver better experiences.

Teams that successfully develop a culture of experimentation and execute effectively typically prepare around three principles.

  1. Choose the right metrics: The most important step before launching an experiment is agreeing on the metrics to measure. Metrics allow teams to define the success of their experiments.
  2. Your feature roadmap is also your testing roadmap: This is precisely what is meant by “experimentation-driven product development”. A core part of every sprint process should be to decide when to attach experiments to new features and feature updates. For riskier features, it’s helpful to experiment with a small portion of users before rolling the feature out for the entire customer base.
  3. Test small and test often: Progressive teams are releasing smaller, incremental software iterations at a more frequent pace. While it’s useful to have some bigger, riskier experiments in the pipeline, running successive, smaller experiments helps diversify risk. Plus, these small bets sometimes pay huge dividends.

According to the Optimizely, “Adopting any new system will require some learning and additional work. But the upside is undeniable. Embracing experimentation-driven product development will make your team more innovative, your business more agile, and your customers more satisfied.”

They write, “By adopting experimentation, teams eliminate uncertainty and guesswork from their software development process. By exposing new versions of software to a portion of customers while the remainder continue with the original experience, teams can quickly assess if their new updates are improvements or regressions. Experimentation, in essence, becomes the feedback loop through which teams determine if their products are impacting customers in a positive way.”

About the Author

Andrew Birmingham is the director of the Which-50 Digital Intelligence Unit of which Optimizely is a member. DIU clients provide their insights for commentary for the benefit of our readers. Membership fees apply.


Previous post

Silo busting essential to delivering personalised experiences

Next post

Digital is Helping UK Banks Regain Consumer Trust: Accenture