Clinical Validation of an Artificial Intelligence System for the Interpretation of Laboratory Test Results Combined With a Structured Medical History

Condition(s) studied

Investigational drug(s) / intervention(s)

AI-based laboratory result interpretation software: A CE-marked class IIa medical device software for the automated interpretation of laboratory test results, operated within its intended purpose. The software administers a structured medical history questionnaire in which subsequent questions are selected dynamically on the basis of the laboratory results and of previous answers. For each laboratory result the software generates one interpretation, comprising a category of urgency of medical contact (immediate, urgent, routine, or no need for medical contact), a suggested medical specialty, a list of suggested further laboratory tests, and an explanatory text addressed to the patient. Interpretations are not aggregated by the software. The interpretation report is generated once per participant, after completion of the questionnaire, and is released to the participant after the assessment of the attending physician has been recorded and locked. The generated report is not used to direct the clinical management of the participant.

Study summary

This study evaluates a certified artificial intelligence (AI) based software system that automatically interprets laboratory test results in combination with a structured, dynamically generated medical history questionnaire.

The purpose of the study is to determine how accurately and how safely the system assigns a patient to a category of urgency of medical contact, and how closely the interpretations produced by the system correspond to the assessment of an expert physician.

Adults treated at two university hospitals in Katowice, Poland, who are referred for laboratory testing, complete an electronic medical history questionnaire after their laboratory results become available. The system then generates an interpretation for each laboratory result, including a category of urgency of medical contact, a suggested medical specialty and suggested further laboratory tests. The attending physician independently records a clinical assessment of the same laboratory results and medical history without access to the output of the system. An independent expert physician subsequently establishes the reference assessment, blinded to the authorship of the assessments being compared. The interpretation report is released to the participant only after the assessment of the attending physician has been recorded and locked.

The study is conducted in two phases. The first phase (609 participants) is exploratory and uses the initial version of the system. The second phase (290 participants) is confirmatory and uses the final, frozen version of the system. All outcome measures are assessed in both phases; the pre-specified confirmatory hypothesis test applies to the second phase.

The endpoints of this study are properties of the software system, namely the concordance of its output with an expert reference assessment. Health outcomes of participants are not measured.

Eligibility

Primary outcome measure(s)

• Percentage of Participants for Whom the Urgency Category Assigned by the Software Was Not Lower Than the Category Assigned by the Expert Physician (Patient Triage Safety Indicator) — Through completion of the expert reference assessment for all enrolled participants, up to 16 months after study start
  The urgency category assigned by the software is compared with the urgency category assigned by an expert physician serving as the reference standard. The urgency categories, ordered from highest to lowest, are: immediate medical contact, urgent medical contact, routine medical contact, and no need for medical contact. A participant is counted in the numerator when the category assigned by the software is the same as or higher than the category assigned by the expert physician. The measure is the percentage of participants counted in the numerator. Range: 0 to 100 percent; higher values indicate better performance. The measure is assessed separately in each study phase and data from the two phases are not combined. Each comparison is based on data locked at the time of enrollment; the expert assessment is performed retrospectively, in batches, on a locked dataset, and the calendar timing of the batch review does not affect the measured quantity.
• Percentage of Participants for Whom the Urgency Category Assigned by the Software Was Identical to the Category Assigned by the Expert Physician (Patient Triage Accuracy) — Through completion of the expert reference assessment for all enrolled participants, up to 16 months after study start
  The urgency category assigned by the software is compared with the urgency category assigned by an expert physician serving as the reference standard. The urgency categories, ordered from highest to lowest, are: immediate medical contact, urgent medical contact, routine medical contact, and no need for medical contact. A participant is counted in the numerator when the category assigned by the software is identical to the category assigned by the expert physician. The measure is the percentage of participants counted in the numerator. Range: 0 to 100 percent; higher values indicate better performance. The measure is assessed separately in each study phase and data from the two phases are not combined. Each comparison is based on data locked at the time of enrollment; the expert assessment is performed retrospectively, in batches, on a locked dataset, and the calendar timing of the batch review does not affect the measured quantity.

Trial sites (2)