What does taking off your shoes and belt in the airport have in common with risk assessment in clinical trials?

Today we talk about the drawbacks of traditional risk assessment and propose an alternative approach to clinical trial risk assessment that is based on data and considering plausible attacks on your trial as opposed to fixed protocols and human monitoring processes.

The FDA draft guidance on central, risk-based monitoring of clinical trials devotes a chapter on risk assessment. The guidance discusses the risk assessment, a component of risk management, as applied in the context of clinical monitoring. Risk assessment generally involves identifying risks, analyzing risks, and then determining whether risks need to be modified by implementing controls (e.g., processes, policies, or practices). The guidance does not go into particular methodologies of risk assessment for clinical trials and instead refers the reader to the literature, citing that risk management is a well-understood discipline.

Traditional risk management

A traditional risk management approach will start with members of the team looking at what risks exist for patient safety and data quality. From a systems perspective, those risks are then translated into so-called risk signatures that can be used in a central, risk-based monitoring activity of the sites. Risk signatures enable us to operationalize a risk management policy into actions – defining what data we need to collect and what data characteristics we need to measure.

Taking off your shoes in the airport is a control for the risk signature of the shoe bomber which as we will know, was an event that only happened once.

There are 3 challenges with risk management in clinical trials:

1. Risk mitigation is a dynamic process. Except for study parameters like primary endpoints, we need to manage data from the sites in order to write risk signatures to monitor the sites. In other words, we need a feedback loop.

2. Risk signatures are hit or miss.  If you base them on historical experience of the clinical study management team you can have some hits but you might have some really crazy misses like discovering 12 months into the study that sites were entering Roman numerals into date fields and closing all the queries as “date valid”. In other words, you need a dynamic feedback loop.

3. People are involved. It seems to me that assessment of risk is a difficult process to understand and execute, primarily because the people who are tasked with assessing the risk in their clinical trial may not understand the basic concepts very well. Study monitors may also find it challenging to connect the dots of “risk management” (which is a very fuzzy sort of idea) with their work on the ground in the site. Let’s face it – risk is difficult to understand, since it is a function of many different, often-interdependent variables.

In view of the above 3 challenges, we suggest using an attack modeling approach, that collects clinical data continuously and adapts risk signatures to the realities of site and CRO performance.

An attack modeling approach for remote monitoring of clinical trials

A traditional risk management approach assesses risks based on the study protocol, before we have collected any clinical data from the field. We’ve already established that this is problematic. An attack modeling approach dives beneath the surface of risk, using a model of attacks on patient safety and data quality. In the course of the clinical trial, as new attack patterns emerge, we adjust our risk signatures accordingly in order to detect repeat incidents.

Structured questions is a keystone of attack modeling

At your meeting with clinical and regulatory people, ask 4 structured questions; for each question prepare an example of 2 answers in advance:

1. What are  the valuable assets in the study ? (patient safety, recruitment rate)

2. What are the vulnerabilities? (sites that are poorly trained)

3. What are the threats that exploit vulnerabilities? (data entry long after the visit)

4. What kind of controls can we implement? (central monitoring, CRO oversight, site performance oversight)

Every person I’ve ever met gets the idea immediately. For starters – ask 7 questions and give an example of an answer to each question:

1. What is the single most important asset in study in your opinion? (patient safety, dosing compliance)

2. What do you think is the single biggest threat to that asset? (over dosing)

3. How do you think attackers will cause damage to the asset? (over dosing, killing a patient)

4. Can you give me one example of something really disastrous happening in the study, what would it be? (Discovering that 50% of the patients were under-dosed after the study completed)

5. If you could give the VP clinical one suggestion, what would it be? (don’t assume anything)

6. If you had to give the CEO one suggestion, what would it be? (be patient)

7. If you had to give President Trump one suggestion on how to reduce the threat of global terror, what would it be? (beyond my pay grade).

Collecting clinical data from the study across all sites enables us to measure CRO and site performance, focus our risk signatures and response on problem areas and manage to fix problems (i.e. mitigate threats) in real-time instead of waiting for data lock to start cleaning up.