A freeway paradigm helps explain why onsite visits by study monitors don’t work and helps us plan and implement an effective system for protocol compliance monitoring of all sites, all data, all the time that saves time and money.
But first – let’s consider some special aspects of clinical trial data:
Clinical trial data is highly dimensional data.
Clinical trial data is not “big data” but it is highly-dimensional in terms of variables or features of a particular subject.
Highly dimensional data is often found in biology; a common example of highly dimensional data in biology is gene sequencer output. There are often tens of thousands of genes (features), but only tens of hundreds of samples.
In medical device clinical trial, there may be thousands of features but only tens of subjects.
Traditional protocol compliance monitoring uses on-site visits and SDV (source document verification) that requires visual processing of the information at the “scene”. Since the amount of visual information available at the scene is enormous, a person processes only a subset of the scene.
Humans focus on the interesting facets of a scene ignoring the rest. This is explained by selective attention theory.
Selective attention.
Selective attention is a cognitive process in which a person attends to one or a few sensory inputs while ignoring the other ones.
Selective attention can be likened to the manner by which a bottleneck restricts the flow rate of a fluid.
The bottleneck doesn’t allow the fluid to enter into the body of the bottle all at once; rather, it lets the fluid to enter in certain amounts depending on the flow rate, until all of it has entered the bottle’s body.
Selective attention is necessary for us to attend consciously to sensory stimuli in such a way that we will not experience sensory overload. See the article in Wikipedia on Attenuation theory.
However, the challenge to study monitoring does not end with selective attention; it only begins with it.
Study monitors visit a site once every 4-6 weeks.
A good study monitor will use a best practice of repeatable process, doing the same thing each time and looking at the same interesting facets of the scene each time.
Repeating the same subset of a scene at a site, is the study monitoring analog of the person with 1 year of experience repeated 20 times, which is not equivalent to 20 years of experience in a variety of tasks and jobs.
In our freeway paradigm – it is equivalent to to standing on the same overpass and counting cars at the same time once every 4 weeks which is hardly comparable to using cameras that count license plates 24 hours / day.
The consequences of selective attention in clinical trial monitoring.
For example, in the Second European Stroke Prevention Study, data on 438 patients were fabricated at one site.
This was first detected by statistical anomalies in the data and later confirmed by a central review of blood results: A visit to the site had failed to identify any transgression.[5,6]
Quis custodiet ipsos custodes?
Consider that study data is available in real time from the EDC system. Using statistical methods we may trace discrepancies in the data and identify outliers.
If not automated, then statistics may turn into a tool for manipulation in the presence of greed, as witnessed by the sad story of Andromeda Biotech who colluded with a bio-statistics firm in Israel to cook the results.
In 2014, Hyperion Therapeutics Inc. (Nasdaq: HPTX) cancelled its acquisition of Israeli company Andromeda Biotech, which was developing a Type 1 (juvenile) diabetes drug DiaPep277.
According to Hyperion, Andromeda colluded “with a third-party bio-statistics firm in Israel to improperly receive un-blinded DIA-AID 1 trial data and to use such data in order to manipulate the analyses to obtain a favorable result. See Hyperion cancels Andromeda acquisition.
The freeway model of study monitoring.
Imagine an observer standing on an overpass over a freeway watching cars that are moving at a fairly uniform pace.
After a very short period of time, the observer will generally ignore cars traveling at the same speed (“consistent events”).
The observer will however notice slow-moving vehicles or cars speeding and weaving in and out of traffic (“novel events”).
Consistent events are bad news for people who audit a process
Consistent events however, may be bad news events for patient safety or protocol compliance.
Consider the case of the psychiatric study where 55 percent of the subjects were consistently under-dosed – both a protocol violation and potentially a safety issue for the treatment.
The CRO study monitors did not detect these consistent under-dosing events for the same reason that our casual observer on the freeway overpass doesn’t notice that the majority of cars are speeding, bumper to bumper.
6 things you must do in order to achieve very effective study monitoring.
1. Protocol monitoring is like central labs – both highly benefit from automation.
Your monitoring operations, should not be dependent on manual export and import and manual processing of data from the EDC system.
Manual data extracts, transformation and load are highly vulnerable to human error and human greed as we see from Hyperion Andromeda debacle.
2. An automated protocol monitoring system should behave in a manner similar to our human freeway observer.
It will first learn the normal, consistent events and then detect novel or inconsistent events. We see capabilities like this in so-called smart highways and every day in traffic light management systems that adapt to traffic patterns.
A key point in an automated monitoring system is that the software learns the novel or inconsistent events over time.
Life is dynamic. Plan for change.
This is in contrast to the traditional approach of developing risk signatures before the study starts and using them throughout the study ignoring changes and difficulties that develop.
3. The third part of assuring the success of clinical trial monitoring is to know what you don’t know.
There is the story of the medical device clinical trial that did not collect the primary end-point – which is I suppose taking this principle to the extreme.
In fact – the process of protocol compliance monitoring is dynamic just like freeway traffic.
The tools you need to monitor traffic on the San Diego freeway at 3AM are different than those that work at 930 AM since the weather and road conditions can change unpredictably.
A visit by a US President to a meeting of the DNC in Beverly Hills can disrupt the entire system.
Rapidly iterate to improve
Issues can popup anywhere.
The users of automated monitoring technology need to be able to rapidly iterate in order to improve the performance of their tools using search and visualization to discover questions they never thought of asking at the kickoff meetings with the sites.
Move quickly in a slow-moving situation
4. The fourth part is rapid response in a slow moving situation.
Medical device clinical trial usually moves slowly.
However – protocol violations due to mis-dosing or poor data quality due to CRO mistakes need to be addressed immediately by the sponsor and not brushed under the table.
Total ownership
5. Getting back to Quis custodiet ipsos custodes? (the a Latin phrase in the work of the Roman poet Juvenal from his Satires is literally translated as “Who will guard the guards themselves?” Monitor your CRO.
6. Last but not least is, do not forget that patient compliance automation technology is a tool used by people.
The technology must be reliable, fast, friendly and communicative and allow the sponsor to do much more with his monitoring dollar which accounts for 30% – 50% of the budget in Phase 2-3 trials.
In summary:
We have seen that a freeway model describes the challenges of remote risk-based monitoring and also provides a framework for meeting those challenges.
