Neurological medical devices are transforming patient care. The underlying diseases and clinical symptoms neurological devices can treat are broad, and safety considerations can cover a wide array of risk levels, from minimal risk, temporary, non-invasive devices to high-risk implantable or other invasive treatments. Additionally, evaluation of neurological interventions often involves subjective endpoints or multi-dimensional outcomes, potentially complicating benefit-risk considerations.
A robust safety strategy is not just a regulatory checkbox or a one-size fits all solution. It is an essential and tailored part of a development strategy. Poor planning can cause safety issues, produce hard to interpret data, cause regulatory delays, and require costly rescue studies. In this article, we will explore how to build a comprehensive safety framework for neurological device trials, with key activities occurring prior to study initiation and extending past completion.
Step 1: Start with a Strong Product Design
Starting to think about safety at the study design stage is too late – safety needs to be integrated into the product development process from beginning. This includes all aspects of a device: materials, construction, the target population, claims, reimbursement, etc. Safety evaluations are also informed by risk assessments as well as bench and non-clinical data. All this needs to be integrated from the beginning to avoid needing to start over.
Step 2: Get the Right Regulatory Strategy
Safety considerations for a device are dictated by the details of the indication, intended use, and associated claims.
Know the Regulations
Reasonable assurance that a device is safe is defined in the Federal Code of Regulations in terms of “probable benefits to health.” This is considered against the ability of the device to “outweigh any probable risks” and is evaluated in the context of “intended uses and conditions of use” and “directions and warnings against unsafe use”.
Small tweaks to an indication or instructions for use, or desired claims, can have large consequences either in preventing or causing safety issues. A careful regulatory strategy that thoughtfully defines these can prevent safety issues from derailing your plans. Throughout a development program, it is helpful to ask, “are there modifications to the device use or instructions that can have a meaningful impact on safety?”.
Consider Safety in a Benefit-risk Context
The definition of safety also requires considering benefit-risk calculations since risks are not considered in a vacuum; how do the probable benefits to health balance risks? One place this balance is illustrated is in the FDA’s Safer Technologies Program for Medical Devices. This program was designed specifically for products expected to significantly improve the safety of currently available treatments or diagnostics. This demonstrates that improved safety is a priority, and it is worth considering potential safety benefits relative to currently approved therapies, instead of safety as just a source for increased risk.
Step 3: Plan for a Strong Evidence Package
For many devices, the bulk of evidence will be based on a clinical study. A weak study design can lead to IDE approval challenges, protocol amendments, ambiguous results, and delays in marketing approval. Here’s how to build a robust design:
Define the Right Comparator
Proper interpretation of data requires understanding of the context, in particular what are we comparing against? The specific choice of control can have a dramatic impact on the details of the study design, the sample size, and interpretation of the results. Each choice will also have implications in terms of the clinical impact and potential reimbursement considerations.
A randomized controlled trial (RCT) will have a different ability to support causal inference than a single arm trial. However, if safety events are defined in terms of procedure or device related events, comparisons of treatment vs. a “usual care” or non-device medical therapy may be unreasonable since the control group is not at risk for such events.
Examples of common comparators:
- For a lower-risk 510(k) product, it may be sufficient to compare to published data from the predicate device.
- In some cases, comparison to literature may be appropriate for higher risk devices depending on the strength of relevant scientific literature, or if randomization is not feasible.
- An RCT may compare an investigational treatment with a current standard of care treatment, in either a non-inferiority or superiority setting, depending on the expected performance and desires for claims.
Increasingly, the FDA is looking to real-world data to support regulatory decision making. As examples, the FDA has used large real-world registries to support labeling modifications, literature reviews supplemented with clinical data to support expanded indications, and even smaller post-market retrospective studies to support de novo classifications for neurology products.
Define the Right Endpoints
Endpoints are the backbone of your trial; they determine how safety (and effectiveness) is measured. For neurological medical devices, often, safety endpoints are defined broadly in terms of procedure or devices related to adverse events (AEs).
A safety endpoint may or may not be subject to a formal hypothesis test. Sometimes the question is not about answering whether a device causes adverse events, but rather determining what events occur and how often. This is especially true for a novel device that is evaluated in a single arm study; descriptive analyses of endpoints may be most appropriate to capture a broad picture of safety. The lack of literature for an appropriate formal statistical comparison may render a performance goal impractical and unnecessary.
In other cases, there is interest in a composite of specific serious adverse events. This may occur in a setting where alternative similar devices are available, and their safety profile is well known. In these cases, a comparison against a performance goal (for a single arm trial), or a formal statistical test against a control group (for a randomized trial) may be best. Still, interpretation of the composite will depend on a carefully considered definition of the composite endpoint. Ideally, all the components of a composite would have equal clinical weight.
Step 4: Implement Robust Data Collection
While safety is largely based on adverse events, it is valuable to consider more broadly how solid data collection can bolster your benefit-risk arguments. In neurology trials, strong effectiveness data for potentially subjective endpoints can help balance safety concerns. Because of this, data on both safety and effectiveness must be precise, standardized, and auditable.
Minimizing missing data is another important aspect of robust data collection. There have been past cases where missing data has led to initial safety concerns, which were later shown to be incorrect after subsequent evaluations with more complete data. Prevention of missing data is the best defense.
Use Validated Assessment Tools
Neurological trials often rely on scales that measure motor, cognitive, and functional outcomes. Using validated tools reduces variability that can introduce uncertainties or invalidate safety conclusions.
Examples of standardized tools
- NIHSS for stroke severity
- mRS for functional independence
- MoCA for cognitive function
While neurological scales are often considered measures of effectiveness, they may also be part of an evaluation of safety. For example, an endpoint may be based on “major stroke” where the classification of “major” is determined based on a stroke severity score. This dual nature makes it doubly important to standardize data collection, site training, and monitoring to ensure complete and accurate data collection with these tools.
Standardize Training
Inter-rater variability is a major risk in neurology trials. Training isn’t just necessary for clinical site staff. Centralize review by a well-staffed and trained Clinical Events Committee, Data Safety Monitoring Committee, or Safety Monitor help ensure consistency of safety data across sites.
Best Practices:
- Conduct centralized training for all site staff on scoring methods
- Consider formal training and certification programs for NIHSS or mRS raters
- Implement refresher training during long trials
Step 5: Establish an Independent Adjudication Process
Adjudication committees ensure unbiased classification of safety events—a critical requirement for neurology trials where attribution is complex. A good adjudication committee starts with a good charter.
Develop a Clear Charter
The adjudication charter should define:
- Roles and responsibilities
- Event classification rules (device-related vs. procedure-related vs. disease-related)
- Temporal association rules for event attribution
- Format of adjudication (independent electronic adjudication vs. meetings)
- Decision-making criteria and quorum requirements
- Endpoint definitions, including composite endpoint interpretation and clearly stating the source of the endpoint (what endpoints are CEC adjudicated vs. site-reported vs. core laboratory determined)
Using experienced and properly qualified individuals is recommended, but even for well-seasoned committee members, Sponsors should provide training sessions for adjudicators before trial initiation to communicate the nuances of their device and clinical study.
Maintain Consistency
- For RCTs, use blinded adjudication to minimize bias.
- Document decisions in audit-ready systems.
Step 6: Perform Continuous Safety Oversight
Careful and continuous monitoring of safety is critical. This ensures patient protection through early detection of risks and helps satisfy regulatory requirements. Continuous monitoring should include automated processes (e.g., data entry screens, edits checks, statistically based sampling) and expert review (i.e. medical monitor review of all AEs).
Before study initiation, the FDA may have safety concerns that delay IDE approval. A staged analysis may help move things along. This is an approach where the FDA provides approval for a small initial number of treated patients, with expansion to the full study sample size based on the FDA’s review of initial safety data.
Carefully Consider Any Changes
Returning to the question of whether device modifications or changes to instructions can impact safety, there is occasionally a potential to make changes to improve safety while not altering effectiveness. Ideally this is done in the early stages of design or during a pilot or feasibility study. The FDA’s Early Feasibility Study program was designed with this sort of device change or iteration in mind.
While generally changes to ongoing studies are not recommended, there are times where late changes that lead to safety improvements may be acceptable. A sound justification and demonstration that changes do not raise questions about the validity of the overall study data, and especially data on effectiveness, is crucial. Close discussion with the FDA is also recommended and Sponsors are explicitly encouraged to engage the agency regarding possible fundamental device modifications during a study.
Do Interim Analyses Make Sense?
Interim analyses are often thought of in terms of early stopping rules for effectiveness, but they have a role to play for safety. This can be to speed initial IDE approval, or to help identify potential mitigations when it may be possible to implement mitigations without fundamentally changing the device or study, thereby preserving the scientific integrity of the data.
Update Criteria as Evidence Evolves
It may be important to revise adjudication rules if new device risks emerge, but care must be taken to ensure consistent adjudication over time. Another complexity may come from external factors – findings from other studies or regulatory communications or requirements from other studies may require re-evaluation.
Step 7: Tell The Story
Your trial is complete, and the topline results are back – your study failed to achieve success for a primary safety endpoint. Is that the end of the road? Not necessarily. Data does not speak for itself – it requires interpretation and elaboration.
The FDA always evaluates the totality of the data and considers this in a benefit-risk context. “Poor” safety results may appear different under scrutiny. This could be due to an overly stringent threshold for safety success that in hindsight is offset by a more favorable result for effectiveness. Perhaps you have a composite safety endpoint that includes components of varying severity or importance, and failure was driven by the less severe components with few or no cases of the more severe components.
Truly understanding safety data requires both a detailed and broad view: accounting for an individual patient’s history and background, the treatment process, specific details regarding each adverse event, the detailed endpoint definition, and the comparator. Not all these nuances can be anticipated, so a thorough dissection of the data after it’s in hand, and the formulation of a coherent scientifically sound explanation will be a key last step in supporting a successful regulatory submission.
Frequently Asked Questions
How can adaptive trial designs improve safety evaluation?
- In many cases, a formal adaptive approach may not be required for safety as there may be less concern about a false positive finding. For example, if there is no formal hypothesis test for a safety endpoint, adaptive approaches that facilitate type I error control would be irrelevant.
- When there are formal hypothesis tests, sample size re-estimation can be used to protect against event rates differing from assumptions.
- Early stopping rules can protect patients if unexpected safety signals emerge.
- Pilot/feasibility studies may help minimize risk. Utilizing a seamless design, where a pilot/feasibility and pivotal study are combined into one over-arching protocol, can speed pivotal study completion.
What are common neurological adverse events in device trials?
- Stroke devices: Hemorrhage, vessel perforation, thromboembolic events
- Neurostimulators: Infection, hardware migration, cognitive or mood changes
- Aneurysm implants: Ischemic events, rupture
Your protocol should pre-specify classification rules for these events and define thresholds for severity.
