Performance Evaluation as per the EU-IVDR

The following piece is the third in a series of blogs related to the recent updates to EU IVDR. Read the first post about the EU IVDR key changes and its impact on the IVD industry. You can also view the second post, which outlines the impact of the EU IVDR on QMS requirements.

The entire world came to a standstill due to the Coronavirus, which started off as just a few cases, but quickly spread across the globe like a wildfire. It affected the entire world, killing millions, impacting the lives of many, and leaving others economically shattered.

In the midst of these troubling times, we also saw how the IVD industry quickly reacted to the situation with diagnostic solutions, and how vaccine and drug manufacturers also did their part to counter the worst healthcare catastrophe in recent memory.

Consider a COVID-19 kit. In this situation, what would a kit that doesn’t perform up to the mark lead to? Wrong diagnosis? Wrong treatment? Late hospitalization? Ventilators and a fight for survival? This is exactly the consequence of any IVD kit or any other diagnostic instrument which does not perform as it is intended to. False positives or false negatives can lead to a misdiagnosis, a delayed diagnosis, and wrong or delayed treatments.

Indeed, the COVID-19 pandemic has brought a great deal of significance to the Performance Evaluation of IVD products. Global regulatory bodies have always focused on the performance characteristics, without them, registration of devices cannot happen. The EU CE marking regulations are no exception, and have included Performance Evaluation as part of the IVD Directive. However, the requirements were not well defined and focused mainly on Analytical Performance.

The new European Regulation IVDR 2017/746 is far stricter. While strengthening the requirements of the IVDD, it provides greater detail and clear guidelines for conducting performance evaluations of IVD products. The intention for these changes is to bring in better regulatory control to enhance and safeguard patient safety. Thus, compliance to the EU IVDR and the CE marking regulation involves proving compliance not only to the GSPRs (General Safety and Performance Requirements), but it also mandates conducting a premarket performance evaluation.

This article takes a look at the new requirements for performance evaluation under the European IVD regulation.

The IVDR defines Performance Evaluation in article 2 as:

“Performance evaluation’ means an assessment and analysis of data to establish or verify the scientific validity, the analytical and, where applicable, the clinical performance of a device;”

Performance Evaluation must demonstrate the following three aspects newly introduced by the IVDR (Article 56 and Annex XIII):

1. Scientific Validity of an analyte means the association of the analyte with a clinical condition or physiological state. For example, the spike protein of Corona virus is an analyte linked to a detection of infection that causes COVID-19. For established analytes like hemoglobin, enough data exists to correlate it to anemia. But for new biomarkers, the scientific validity needs to be established.
2. Analytical Performance means the ability of a device to correctly detect or measure a particular analyte. This is done by the manufacturer during the design and development phase.
3. Clinical Performance means the ability of a device to yield results that are correlated with a particular clinical condition, a physiological or pathological process, or in accordance with the target population and intended user.

The approach taken by the IVDR is a logical one. It requires manufacturers to establish the scientific correlation of the analyte, determine its analytical performance, and then validate it clinically in relation to the current state of the art. Sufficient and quality data has to be collected for demonstrating safety, performance, and notably the acceptability of the benefit-risk ratio. The data should be thorough and objective, and it should take into account any favourable and unfavourable situations which we could see in real life. The results are analyzed, interpreted, and concluded in the Performance Evaluation Report to make a decision on the benefit-risk ratio. This constitutes the Clinical Evidence for a device, supports the use of the device, and takes into consideration the current state of the art.

The performance evaluation becomes a critical part of the technical documentation. It is critically assessed during CE marking and it helps the Notified Bodies do a qualified assessment of the device’s effectiveness, safety, and performance characteristics. Manufacturers have the freedom to define the depth and extent to which performance evaluation needs to be done for their devices. They must assure that it is proportionate to the risk class of the device, and is appropriate for its characteristics and intended purpose.

Once the devices are launched in to the market and used as per the intended purpose, there will be several types of feedback from the users regarding the performance. Therefore, performance evaluation is intended to be a continuous process in which clinical evidence is gathered throughout the life span of the product with the help of the post market performance follow-up process.

Annex XIII part A describes performance evaluation as a three step process. As a first step, manufacturers are expected to establish, document, and update a Performance Evaluation Plan (PEP). They must also have an SOP that describes in detail, how the scientific validity, and the analytical and clinical performance will be established for all of the devices that they manufacture.

The Performance Evaluation Plan (PEP) is critical because it defines the effectiveness of the whole process, and it documents the scientific rationale of the methods used. It must include the acceptance criteria for the methods used, and how the acceptability of the benefit-risk ratio will be determined. The second step is conducting the evaluation to collect data, and the third step involves analyzing the data, the conclusions, and the reporting. The contents of the PEP and PER are clearly defined in Annex XIII.

Some critical points of PEP include –

• Intended purpose/intended use
• The analyte or marker
• Target populations
• Description of the state of the art

Establishing scientific validity:

This can be done by any of the following methods:

• For established products, gather information from:
  • Peer reviewed literature from databases like PubMed, Embase, Cochrane, or Medline
  • Relevant information from other devices
  • Consensus expert opinions
  • Historical data and experience

• For new devices/analytes:
  • Proof of concept studies
  • Clinical studies

Analytical performance plans and protocols need to be developed for each device or device group. Here, the manufacturer determines whether the device detects and measures the analyte accurately, typically by using various laboratory tests and certified reference materials in analytical performance studies. The IVDR expects a comparison to be made to a reference method to demonstrate the accuracy of the results.

The results of such studies help define the performance parameters like analytical sensitivity and specificity, limits of detection, quantification, range, linearity, repeatability and reproducibility, accuracy, and precision characteristics done under favourable and unfavourable conditions. Such a robust device is then further used for the clinical performance evaluation.

Clinical performance is based on the intended use defined by the manufacturer and needs to be done at the user’s end, in a clinical laboratory or in a hospital environment with use conditions, on the target population, and by the intended user. The clinical performance plan defines the methods involved.

For some devices the following methods may be sufficient:

• Published literature
• Experience gained from routine testing

But for novel devices/markers:

• Comparison with a clinical standard practice or method comparator is necessary by conducting a clinical performance study to establish the trueness of the value or the diagnostic accuracy

Clinical Performance studies define the parameters such as diagnostic sensitivity and specificity, positive and negative predictive values and likelihood ratios, expected values in normal and affected population, and also the sample collection and handling requirements. For new devices, such studies can confirm performance that cannot be tested by analytical performance studies. They are similar to clinical trials and therefore require ethics committee approval. The manufacturer needs approvals from the Competent Authority of the member states before conducting such studies. The studies are planned in such a manner as to protect the patients in every possible way. Depending on the type of product, different types of studies may be required, such as non-interventional, interventional, or those involving special cases like minors, pregnant women etc. (refer to Articles 57-58 and Annex XIII). If a manufacturer thinks that clinical studies are not required, proper justification needs to be provided.

The Scientific Validity Report, Analytical Performance Report, and Clinical Performance Report are collated in the Performance Evaluation Report. They are a very critical part of the technical documentation. The PER is updated throughout the life cycle of the device, with new and factual data gathered from the market with the help of a Post Market Performance Follow-up plan, or PMPF plan. It is the clinical evidence data that actually helps define how PMPF should be done. The results are documented in a PMPF report and updated in the PER.

An important point to note is that the evaluation of the device may help understand new and emerging risks. Therefore, a system is required to link the performance evaluation and PMPF to the Risk Management process.

For Classes A and B, the low-risk devices, the PER is updated as necessary, but at least once every three years is recommended. It also must be made available to the notified body as and when required. For Class C and Class D, the PER should be updated when necessary, but at least annually.

Notified Bodies will assess the procedures and documentation relating to the performance evaluation, ensuring they are adequate. They will review the planning (PEP), conduct, data collection, evaluation or data appraisal, reporting, and finally, how the report will be updated. Moreover, they would like to see how the data from performance evaluation feeds into the Risk File for the particular device.

Legacy devices are devices with well established technologies that have been on the European market for some time, and have a CE mark under the IVD Directive. Since they are on the market, it is necessary to have a performance evaluation done as per the current intended use, and taking into consideration today’s state of the art for that particular device. This may prove challenging to some manufacturers, since under the IVDD most devices were not scrutinized by a Notified Body. They may have some analytical performance data, but it may not be sufficient or relevant now, and not all may have scientific validity reports or any clinical performance data. The IVDR does not advocate a “grand fathering” approach!

• GHTF SG5/N6:2012– Clinical evidence for IVDs: Key definitions and concepts
• GHTF  SG5/N7:2012– Clinical evidence for IVDs: Scientific Validity Determination and Performance Evaluation
• GHTF SG5/N8:2012– Clinical evidence for IVDs – Clinical performance studies
• ISO 20916:2019– Clinical performance studies using specimens from human subjects
• EN 13612:2002– Performance evaluation of in vitro diagnostic medical devices
• MEDDEV 2.7/1 rev 4

The EU-IVDR has set stricter requirements for the Performance Evaluation of devices. A robust plan needs to be put in place for conducting performance evaluations and gathering clinical evidence for all devices, including legacy devices. Clinical Evidence is done as a continuous process, it supports the use of the device taking into consideration the current state of the art, and it clearly demonstrates its benefits.