Observations

In most teams I've been working, though different missions (building a tool, a new product, or improving performance of an existing one), the one thing all teams share in common is the need to validate different hypotheses as fast as they can. Despite whatever the methodology team uses (Agile, Scrum, Kanban, etc), if they do not realise the importance of shorten hypotheses validation lifecycle, it's very common to fall into the pitchfall of so-called Parkinson's law. Work will naturally expands to fill the time available, and members will tend to iterate over and over on the same item without closing it. Few common mistakes that teams make:

• Prioritisation: Different workstreams are allocated resources on a first-come, first-served basis. As a way to hold individual accountable, everyone owns their own streams and try to grab resources needed for them without thinking about coordinating team's prioritisation. This could only work if resources are illimited, which rarely (if not never) be the case.

• Unclear hypothesis: Sometimes hypotheses are unclear or unaligned before execution. This leads to back-and-forth discussions when products are built or results are available, making it difficult to reach alignment or determine next steps.

• Big batch: Sometimes team wants to validate a strong hypothesis with high uncertainty, requiring many data points to prove. Instead, they could break down the hypothesis into smaller ones and validate each individually. Validating one hypothesis can inform subsequent development works. Other times, team wants to test a hypothesis that requires significant changes. Instead, they could simplify the design, use available resources, or even "fake it until make it" and validate the hypothesis before investing in significant development.

Speeding up, on the one hands, will help to redefine future roadmap and build best products quickly. On the other hand, it reduces wasted efforts, so team can operate in high efficiency and intensity, which is important in our time hyper volatile market and fast-paced technology change.

"Lean" Engineering

Lean manufacturing, originating from Toyota, then later Lean Startup proposed by Eric Ries, is proven to be a powerful methodology to speed up iterations toward validating hypotheses and building products. At its core, the methodology advocates for a systematic, scientific approach that startups should focus on learning what their customers really want and then iterate their product quickly based on feedback, rather than spending years perfecting a product without understanding market needs. Defined a startup as "a human institution designed to create a new product or service under conditions of extreme uncertainty", this methodology is so compelling that every teams in big corporates could also adopt.

My proposal to help every teams moving fast is to adopt Kanban in their hypotheses validating and combining with a small-batch mindset:

A pool of hypotheses

Team should actively maintain a pool of hypotheses which serve its mission. The hypotheses could be anything from building a new features, fixing a bug, improving the performance of an ML model. A hypothesis should be consisted of following element:

• Hypothesis: What we want to prove/disprove? How it helps making a closer step to the vision?
• MVP: What’s minimal change to the current system to get this hypothesis tested ? If needed to build a new product, which is the shortest way to test out in real environment and how long does it take?
• Metrics: Which metrics to prove/disprove this hypothesis? What’re expected behaviours of listed metrics?
• Validation: Who in the team are needed to get a signoff of this validated hypothesis? How does it influence the hypothesis pool?

Backlog

A list of testing hypotheses, which should have consisted of no more than N prioritised at the time (suggested N=3). As most team is more than 3, they need to come up with how to hold both group and individuals accountable of validating these hypotheses (aka innovation accounting).

In progress

Any preparation to be able to build the new features or experiments needed to validate a hypothesis.

Built

At this stage, a hypothesis is ready to be tested, most time in a real A/B test in small group of customers. This step also includes analysing the data, building reports and setup enough time with reviewers.

There’re scenarios when follow-ups are required and the hypothesis should remain in the Built stage but required further effort. As it remains in the bucket, it has the highest priority for the team or group of accountable individuals to get it done and validated, either accepting or rejecting, with the primary objective is to shorten its validation lifecycle.

Validated

A hypothesis is considered validated all reviewers signoff and no more follow-up to address. A validated hypothesis hence directing any adjustment needed for our pool of hypotheses.

Conclusions

In my view, by adopting "Lean" Engineering, teams could significantly improve the efficiency and effectiveness of our hypothesis validation process. By embracing the Build-Measure-Learn cycle and leveraging Kanban principles with small batches, teams can take advantage of available resources and provide more generous sets of validated hypotheses.

Let end by a quote I like from Eric Ries in his book:

"The only way to win is to learn faster than anyone else."