Evaluating AI and Human Expert Decisions in Colorectal Cancer

Condition(s) studied

Investigational drug(s) / intervention(s)

LLM-MDT: Leveraging large language models (LLMs) to Generate Multidisciplinary Team (MDT) Treatment Recommendations

Study summary

The goal of this observational study is to evaluate the decision-making consistency between large language models (LLMs) and expert multidisciplinary teams (MDTs) in adult patients diagnosed with colorectal cancer who underwent MDT consultation between January 2023 and December 2024.

The main questions it aims to answer are:

How consistent are the treatment decisions generated by LLMs compared to actual MDT decisions? Do different LLMs (e.g., ChatGPT, DeepSeek) show varying levels of agreement with expert recommendations? What clinical factors contribute to differences between AI-generated and human expert decisions? Researchers will compare the AI-generated treatment recommendations with real-world MDT decisions using anonymized patient records to see if LLMs can reliably support clinical decision-making in oncology.

Participants will:

Have their de-identified clinical data (e.g., imaging, pathology, MDT notes) processed through several LLMs Not be contacted or receive any interventions, as this is a retrospective study using existing clinical records only.

Eligibility

Primary outcome measure(s)

• Agreement Between AI-Generated and MDT Treatment Decisions — January 1, 2023 to December 31, 2024 (based on MDT consultation date)
  Description: The primary outcome is the consistency between treatment recommendations generated by large language models (LLMs) and those made by expert multidisciplinary teams (MDTs) for colorectal cancer cases. Consistency will be quantified using Cohen's Kappa coefficient. Higher Kappa values indicate stronger agreement

Trial sites (1)

More Peking University Cancer Hospital & Institute trials in China

Other trials for the same condition