Product Clinic #6: Releasing, Fast and Slow

At Lindus Health we do a product release every 6 to 8 weeks. Depending on your background this will evoke one of two reactions. If you work in the clinical space you might think: woah, what an incredibly fast release cycle! If you’re at a typical SaaS startup you might instead think: how quaint, I release multiple times per day!‍

We’re a startup in the clinical space where these two worlds collide. We pride ourselves on our speed in constantly improving our product to make running clinical studies as simple, safe and effective as possible. At the same time, we are operating in a regulatory environment governed by Good Clinical Practice (GCP) which implies a set of release practices that can slow down a team. The most prominent of those practices is formal Computer Software Validation (CSV). In this post I will explain how we use automation to make validation both faster and more thorough.

‍

‍Thinking, Fast and Slow

Daniel Kahnemann laid out two different ways our brains form thoughts in his bestseller Thinking, Fast and Slow:

The two systems of thinking are complementary as well as opposing forces. We utilise both systems in many situations, for example, to drive a car successfully from A to B. On the other hand, we might rely too much on System 1, rooted in heuristics and bias, when System 2 could improve our decision-making. (I’m only scratching the surface of Kahneman’s ideas here - read the book! Or for an introduction watch this video.)

Just like our minds find themselves torn between these two systems of thinking, our product team finds itself torn between two traditions of releasing:

We do a lot of thinking - of the conscious System 2 kind! - about how we could effectively marry aspects of both kinds of release cycles. We realised that the key to planned releases that feel more like frequent spontaneous releases is to make the process really fast and easy. So we set out to automate as much of it as possible, including the validation.

‍

Continuous Validation

This is how I would break down the kind of Computer Software Validation (CSV) that is usually adopted under Good Clinical Practice:

1. Describe what you want to do (also called User Requirements Specification - URS)

2. Do it

3. Validate that you did it correctly

4. … all while recording evidence that you did these steps in this order

A traditional approach might require carefully planning and weeks to validate a release. We have now adopted a process that allows us to initiate a release at will and validate it in a largely automated way within less than a day. Borrowing ideas from the best practice of Continuous Integration we call this Continuous Validation.

Traditionally, validation would be performed as a distinct step during a release process for the entirety of the work that went into a release. But if you closely observe software engineers during their work, you will notice that they constantly perform little acts of validation. They are naturally already engaged in a loop of making code changes and then manually verifying that these changes are working as intended. What if we could capture this ad-hoc manual testing and automate it?

This is the simple idea behind Continuous Validation. When an engineer develops a new feature, they write unit tests as a general best practice. But in addition to that they write a different type of test: automated validation tests. These capture expected behaviour at the user requirement level and are akin to a traditional validation script intended for manual execution by a QA person - except they can be executed programmatically.

‍

Anatomy of a Validation Test

The underlying tooling is fairly simple. Our product consists of web apps hosted in the cloud. Our validation tests are written in a powerful framework called Cypress, which lets us express user interactions in those web apps, such as filling out a text input box or clicking on a button. The tests run in a real browser environment against a real installation of the product. This is as close as it gets to simulating a real user - effectively a manual QA script has been turned into a programmatic test.

Watch this video of a validation test (truncated and slowed down) going through a patient survey. You can see how Cypress is quite literally navigating the web app in a browser as a real patient would.

Caption: Extract from a simplified validation test covering aspects of patient surveys.

The suite of all validation tests runs automatically on a regular schedule, at least twice per day, so we know our product is continuously in a validated state. When the time comes to publish a new release, every 6 to 8 weeks, we only need to reference the test results in a formal validation report (complemented with manual testing in critical areas identified by risk assessments).

‍

Above and Beyond Validation

We took inspiration from Kahneman as well as practices in the clinical space and in modern SaaS startups. The result is a release cycle that is measured, but processes that are fast. Because the release process relies so much on automation it can be handled easily by a single release manager within less than a day.

Our approach goes far beyond what's achievable with traditional clinical software validation which is constrained by the amount of manual labour put into it. These constraints would often mean that only new features or changes would be validated for each release, neglecting the revalidation of earlier features. In contrast, we comprehensively validate the full product for every release. This gives sponsors and patients the peace of mind that their data will be captured securely and accurately.

Our own product team benefits in ways that are maybe less obvious but just as important. Continuous Validation infuses the whole team with the confidence to go fast when they might otherwise shy away from impactful new features. Because all engineers get instant feedback on how their changes affect the validated state of the product, they can act with high agency. They can take on responsibility for bold choices and don't have to rely on others to perform testing for them.

I just touched on two of our company values: go fast, and be high-agency! (The other four values are: be transparent, stay curious, benefit patients, and have fun.) That’s not quite a coincidence. We choose practices informed by and in support of our values. Ultimately, we believe this will serve us best in our mission to accelerate clinical trials for healthtech and biotech pioneers.

‍

Nik is the CTO and a Co-Founder at Lindus Health.

‍