Using large language models to automate assessments of the integrity of randomised trials

Systematic reviews that include randomised trials are widely considered to provide the strongest evidence on the effects of health interventions. Yet research shows that systematic reviews frequently include studies that exhibit serious flaws (e.g., gross errors or signs of misconduct) which can distort their conclusions, leading to ineffective or harmful treatments being delivered to countless patients worldwide. Concerns about the high prevalence of research integrity issues have sparked the creation of tools to help systematic reviewers identify and address integrity issues. However, the use of such tools is critically low due to the time, expertise and resources required to use them. This project will evaluate the performance of large language models (LLMs) prompted to undertake assessments of the integrity of randomised trials and their underlying individual participant data (IPD). Key research questions include: Compared to human assessors, how accurately do LLMs detect integrity issues arising in trial reports and datasets? Does accuracy vary according to factors such as the LLM used, the source of information (trial report versus trial data), test subjectivity, or medical specialty? What prompt engineering approaches most effectively reduce LLM errors? How do developed models perform in real-world settings (e.g., IPD meta-analyses)? The project sits at the intersection of machine learning, research integrity and evidence-based medicine, and offers opportunities to explore the utility of machine learning techniques to detect research integrity issues in various fields, and other pressing scientific issues such as reporting and methodological deficiencies. The student will apply various methods to develop, refine and benchmark the performance of models, as well as work with partners such as journal editors and IPD meta-analysis groups to investigate the real-world impact of the developed model.