Never mistake data collection activity for an achievement

Recruiting and caring for patients (whether at home or on site) is a research site responsibility.

Capturing pages (note the paper paradigm!) is one of the key billable metrics for a site.

While data collection, detection of exceptions and action to close issues are all activities essential to the study, pages of data collection are not a achievement in their own right.

The achievement is proving or disproving the experiment – does the drug or device work and is it safe?

Never mistake activity for achievement — John Wooden

John Wooden was the legendary UCLA basketball coach who won NCAA national championships in a 12-year period as head coach for the UCLA Bruins, including a record seven in a row. Wooden coached players at UCLA who would later become stars — players like layers Kareem Abdul-Jabbar and Bill Walton. Wooden understood and trained his players that just being busy on the court doesn’t count.

We need to compensate sites for clinical data observability, not pages of data entry.

Why clinical protocol compliance matters

Compliance to a clinical protocol matters to drug companies that want to bring good products to patients. Non-compliance to the dosing protocol when discovered by FDA will result in the drug not being approved. When discovered in an interim analysis, it can result in trashing the study. When discovered in use by patients, it can result in avoidable hospitalizations.

Compliance to a clinical protocol is both a feeling and a reality.

We feel compliance when we feel protected from process issues, free from data quality issues and safe from adverse events. In this way, protocol compliance is merely a state of mind. None of these threats to the success require unblinding your clinical trial. But there’s the reality of protocol compliance, a reality that has nothing to do how we feel.

We need to ground that feeling of compliance in the reality of compliance and not merely in placebos of dashboards.

Monitoring clinical operations

There is an archaic monitoring activity still conducted in most clinical trials today called SDV — source document verification. SDV involves study monitors manually comparing pieces of paper with digital records in the EDC system. 30 years ago, when we were transcribing from paper forms, this made sense.

Central monitoring operations is actually a step backwards is it shifts costs from fairly-inexpensive CRAs to expensive data managers at the additional price of not having boots on the ground.

With electronic source, wearable devices and mobile patient apps, SDV is no longer needed. That’s the good news. The bad news is that the industry is still using a batch paradigm where data managers extract data and produce periodic reports and dashboards in order to monitor the clinical operations.

Distributed systems cannot be understood with dashboards

Monitoring clinical trials employs a traditional paradigm of monitoring clinical ops activities. With COVID, there is a growing trend to patient-centric and decentralized trials where patients check in from home using a mobile and/or wearable devices. More good news and bad news. Good news because we’re collecting data directly or passively from patients using digital channels.

Bad news, because monitoring clinical operations activities sidesteps the problem that 90–95% of the data in a decentralized trial is generated by a distributed, unmanaged system of patients and connected devices.

As anyone who has ever tried to personally comply to a treatment and/or manage a distributed system with over 1000 dimensions and 100s of components can testify:

High cardinality and high-dimensional systems of unreliable moving parts cannot be managed with dashboards.

Enough for the connected device clocks to drift for your patient alerts to lose meaning.

Observability for clinical data

What is observability?

Observability is a measure of how well the internal states of a system can be inferred from knowledge of its external outputs. 

In control theory, the observability and controllability of a linear system are mathematical duals. The concept of observability was introduced by  Rudolf E. Kálmán for linear dynamic systems.

A system is said to be observable if, for any possible evolution of state and control vectors, the current state can be estimated using only the information from outputs (physically, this generally corresponds to information obtained by sensors). In other words, one can determine the behavior of the entire system from the system’s outputs.

Let’s apply observability to clinical data.

There are 3 data outputs in clinical trials:  data from people who run the study, outcomes reported by patients and data from devices worn/used by patients.

These 3 classes of outputs  are then used to control 3 classes of GCP constraints for the study:  Data integrity,  patient safety and protocol compliance.

As control theory defines – the GCP controls are the duals of observing the clinical data.

Complying with ICH/GCP is an exercise in observability.

To paraphrase Charity Majors — cofounder/CTO of @honeycombio

Monitoring is for clinical operations (the ‘operating system’ of clinical trials)
Instrumentation is for clinical end-points (humans and code in clinical trials)
Observability is for understanding systems of patients in a clinical trial

Summary

The industry has over 25 years of GCP and clinical monitoring best practices. We have analytics and reports and systems of queries and discrepancies that are optimized for detecting data entry problems at clinical research sites.

This seems quaint and almost absurd.

The next step is to instrument the clinical end points and achieve observability for clinical data towards understanding these highly complex systems of humans and devices called clinical trials.

Originally published in Medium