Evaluating the Development and Impact of AI-Assisted Integrity Assessment of Randomised Trials in Evidence Syntheses

BACKGROUND The integrity of research publications is crucial for future research, and guidance arising from that research. Retractions, which are removals of research publications, are rapidly increasing, mainly due to misconduct. Many fraudulent publications remain unidentified without quick and effective ways to assess them. Systematic reviews of randomised clinical trials (RCTs), used by organisations like The National Institute for Health and Care Excellence (NICE), depend on the integrity of RCT publications for evidence on clinical care. In the absence of such integrity assessment, we found that 27 belatedly retracted RCTs affected conclusions in more than half of 88 systematic reviews and clinical guidelines. In a recent small study for The Cochrane Library of systematic reviews, a quarter of RCTs did not pass integrity checks - a third of their guidelines might need to be changed. The unknowing inclusion of problematic RCTs in systematic reviews can potentially harm patients, misguide future research, and waste public funds. The growing awareness of problematic RCT publications (especially where data are fabricated, falsified or plagiarised) highlights the need for effective and rapid integrity assessment for these issues. Recent efforts focus on developing criteria and checklists to help this assessment. The National Institute for Health and Care Research (NIHR) funded ‘INSPECT-SR’ project, in collaboration with The Cochrane Library is creating a checklist, currently with 26 items. This is likely to form the basis for RCT integrity checks by other international organisations. Completing checklists and their data analysis is very labour-intensive and time-consuming. This will limit use and considerably slow the production of trustworthy systematic reviews and clinical guidelines. Artificial Intelligence (AI) has great potential to speed-up and ease use of a checklist. AIM To engage with Project Partners in evidence synthesis to co-design and roll out a user-friendly, automated prototype tool to aid large-scale integrity assessment of RCTs, evaluating their experiences in that process. METHODS To start to address this challenge, we initiated a University of Aberdeen pump-priming project to start using AI for the integrity assessment of published RCTs. Early results are promising. Our approach showed that Large Language Models (LLMs), like ChatGPT, can automate taking numbers from tables in PDFs of published RCTs, needed for 3/26 checks. We have partnered with Vorland, a researcher and computer programmer at Indiana University Bloomington, who developed an earlier prototype for RCT data assessment for 4/26 checks. With the requested funding, we will collaborate with Vorland and Project Partners from five evidence synthesis organisations to co-design a prototype tool for additional checks. We will engage with the Project Partners to evaluate their experiences co-designing the prototype, collecting their feedback on the design processes. We will work with them to design their evaluation of the prototype tool – in multiple steps initially focussing on technical aspects with later evaluations focusing on tasks. EXPECTED OUTPUTS The prototype tool will be freely available to Project Partners. Our evaluations of Project Partners’ experiences and use of the prototype tool will inform our future work with them, refinement of the prototype, and aid our thinking, engaging and training later users. Our work will be written up for publication and presented at international conferences. We will continue to collaborate with our Project Partners and later users to adapt the prototype. This work will benefit patients and the wider research and development community.