ChatGPT's quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questions.

Hoch, Cosima C; Wollenberg, Barbara; Lüers, Jan-Christoffer; Knoedler, Samuel; Knoedler, Leonard; Frank, Konstantin; Cotofana, Sebastian; Alfertshofer, Michael

doi:10.1007/s00405-023-08051-4

2023

Back
Back to start of result list
Permanent link for displayed object

Title:: ChatGPT's quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questions.
Document type:: Journal Article
Author(s):: Hoch, Cosima C; Wollenberg, Barbara; Lüers, Jan-Christoffer; Knoedler, Samuel; Knoedler, Leonard; Frank, Konstantin; Cotofana, Sebastian; Alfertshofer, Michael
Abstract:: PURPOSE: With the increasing adoption of artificial intelligence (AI) in various domains, including healthcare, there is growing acceptance and interest in consulting AI models to provide medical information and advice. This study aimed to evaluate the accuracy of ChatGPT's responses to practice quiz questions designed for otolaryngology board certification and decipher potential performance disparities across different otolaryngology subspecialties. METHODS: A dataset covering 15 otolaryngology subspecialties was collected from an online learning platform funded by the German Society of Oto-Rhino-Laryngology, Head and Neck Surgery, designed for board certification examination preparation. These questions were entered into ChatGPT, with its responses being analyzed for accuracy and variance in performance. RESULTS: The dataset included 2576 questions (479 multiple-choice and 2097 single-choice), of which 57% (n = 1475) were answered correctly by ChatGPT. An in-depth analysis of question style revealed that single-choice questions were associated with a significantly higher rate (p < 0.001) of correct responses (n = 1313; 63%) compared to multiple-choice questions (n = 162; 34%). Stratified by question categories, ChatGPT yielded the highest rate of correct responses (n = 151; 72%) in the field of allergology, whereas 7 out of 10 questions (n = 65; 71%) on legal otolaryngology aspects were answered incorrectly. CONCLUSION: The study reveals ChatGPT's potential as a supplementary tool for otolaryngology board certification preparation. However, its propensity for errors in certain otolaryngology areas calls for further refinement. Future research should address these limitations to improve ChatGPT's educational use. An approach, with expert collaboration, is recommended for the reliable and accurate integration of such AI models.
Journal title abbreviation:: Eur Arch Otorhinolaryngol
Year:: 2023
Journal volume:: 280
Journal issue:: 9
Pages contribution:: 4271-4278
Fulltext / DOI:: doi:10.1007/s00405-023-08051-4
Pubmed ID:: http://view.ncbi.nlm.nih.gov/pubmed/37285018
Print-ISSN:: 0937-4477
TUM Institution:: Klinik und Poliklinik für Hals-, Nasen- und Ohrenheilkunde (Prof. Wollenberg); Lehrstuhl für Plastische Chirurgie und Handchirurgie (Prof. Machens)
BibTeX

Occurrences:

mediaTUM Gesamtbestand Einrichtungen Schools TUM School of Medicine and Health Departments Clinical Medicine Klinik und Poliklinik für Plastische Chirurgie und Handchirurgie (Prof. Machens)Lehrstuhl für Plastische Chirurgie und Handchirurgie (Prof. Machens)2023

mediaTUM Gesamtbestand Einrichtungen Schools TUM School of Medicine and Health Departments Clinical Medicine Klinik und Poliklinik für Hals-, Nasen- und Ohrenheilkunde (Prof. Wollenberg)2023