Background: Healthcare practitioners use Clinical Decision Support Systems (CDSS) as an aid in the crucial task of clinical reasoning and decision making. Traditional CDSS are Online Repositories (OR) and Clinical Practice Guidelines (CPG). Recently, Large Language Models (LLMs) like ChatGPT have emerged as potential alternatives. They have proven to be powerful innovative tools, yet they are not devoid of worrisome risks. Objective: This study aims to explore how medical students utilize ChatGPT as a CDSS in a teaching environment. Methods: The authors randomly divided medical students into three groups and assigned each group a different type of CDSS for guidance in answering prespecified questions, assessing how students’ ability at resolving the same clinical case varied accordingly. External reviewers evaluated all answers based on accuracy and completeness metrics (Score: 1-5). The authors analyzed and categorized group scores according to the skill investigated: Differential Diagnosis, Diagnostic Workup, and Clinical Decision Making. Results: Answering time showed a trend for the ChatGPT group to be the fastest. The mean scores for completeness were: 4.0 (CPG), 3.7 (OR), 3.8 (ChatGPT). The mean scores for accuracy were: 4.0 (CPG), 3.3 (OR), 3.7 (ChatGPT). Aggregating scores according to the three students’ skill domains, trends in differences among the groups emerge more clearly, with the CPG group that performed best in nearly all domains and maintained almost perfect alignment between its completeness and accuracy. Conclusions: This hands-on session provided valuable insights into the potential perks and associated pitfalls of LLMs in medical education and practice. It suggested the critical need to include teachings in Medical Degree Courses on how to properly take advantage of LLMs, as the potential for misuse is evident and real

Clinical decision support systems during teaching: a hands-on comparison (Preprint) / Montagna, Marco; Chiabrando, Filippo; De Lorenzo, Rebecca; Rovere Querini, Patrizia. - (2023 Dec 21). [10.2196/preprints.55709]

Clinical decision support systems during teaching: a hands-on comparison (Preprint)

Montagna, Marco
Co-primo
Writing – Original Draft Preparation
;
Chiabrando, Filippo
Co-primo
Writing – Original Draft Preparation
;
De Lorenzo, Rebecca
Methodology
;
Rovere Querini, Patrizia
Ultimo
Supervision
2023-12-21

Abstract

Background: Healthcare practitioners use Clinical Decision Support Systems (CDSS) as an aid in the crucial task of clinical reasoning and decision making. Traditional CDSS are Online Repositories (OR) and Clinical Practice Guidelines (CPG). Recently, Large Language Models (LLMs) like ChatGPT have emerged as potential alternatives. They have proven to be powerful innovative tools, yet they are not devoid of worrisome risks. Objective: This study aims to explore how medical students utilize ChatGPT as a CDSS in a teaching environment. Methods: The authors randomly divided medical students into three groups and assigned each group a different type of CDSS for guidance in answering prespecified questions, assessing how students’ ability at resolving the same clinical case varied accordingly. External reviewers evaluated all answers based on accuracy and completeness metrics (Score: 1-5). The authors analyzed and categorized group scores according to the skill investigated: Differential Diagnosis, Diagnostic Workup, and Clinical Decision Making. Results: Answering time showed a trend for the ChatGPT group to be the fastest. The mean scores for completeness were: 4.0 (CPG), 3.7 (OR), 3.8 (ChatGPT). The mean scores for accuracy were: 4.0 (CPG), 3.3 (OR), 3.7 (ChatGPT). Aggregating scores according to the three students’ skill domains, trends in differences among the groups emerge more clearly, with the CPG group that performed best in nearly all domains and maintained almost perfect alignment between its completeness and accuracy. Conclusions: This hands-on session provided valuable insights into the potential perks and associated pitfalls of LLMs in medical education and practice. It suggested the critical need to include teachings in Medical Degree Courses on how to properly take advantage of LLMs, as the potential for misuse is evident and real
21-dic-2023
Inglese
File in questo prodotto:
File Dimensione Formato  
preprint-55709-submitted.pdf

solo gestori archivio

Tipologia: Submitted manuscript (manoscritto inviato all’editore)
Licenza: Copyright dell'autore
Dimensione 1.06 MB
Formato Adobe PDF
1.06 MB Adobe PDF   Visualizza/Apri   Richiedi una copia

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/20.500.11768/160298
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
social impact