Abstract
Objective
To comparatively evaluate the reliability, quality, and readability of responses generated by widely used large language model (LLM)-based chatbots to congenital hypothyroidism (CH)-related patient questions.
Methods
Forty frequently asked questions (FAQs) about CH, derived from clinician-reviewed patient education resources, were submitted under standardized conditions (December 2025) to Chat Generative Pre-Trained Transformer-4 (ChatGPT-4), ChatGPT-5.2, Gemini, and Copilot. The modified DISCERN (mDISCERN) instrument was used to assess reliability, whereas the Global Quality Score (GQS) was used to evaluate quality. Readability was evaluated using Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Index (GFI), Coleman-Liau Index (CLI), and Simple Measure of Gobbledygook (SMOG). Scores were compared using Friedman tests with Bonferroni-corrected post-hoc analyses.
Results
Median mDISCERN scores were 5.0 for ChatGPT-4, ChatGPT-5.2, and Gemini, and 4.0 for Copilot. Median GQS scores were 5.0 for ChatGPT-4, ChatGPT-5.2, and Gemini, and 4.0 for Copilot. Differences among models were significant for both mDISCERN and GQS (p<0.001), with ChatGPT-5.2 outperforming others in key pairwise comparisons. Readability differed significantly across all indices (all p<0.001). ChatGPT-5.2 demonstrated the highest FRE and lowest FKGL, whereas Gemini produced the most complex text. However, all models exceeded the recommended sixth-grade reading level.
Conclusion
LLM-based chatbots produced generally moderate-to-high quality CH information, but readability remains suboptimal for patient education. ChatGPT-5.2 showed the best overall performance. LLM outputs may support patient information needs but should complement, not replace, clinician-provided counseling.
Keywords: Artificial intelligence, ChatGPT, congenital hypothyroidism, copilot, Google Gemini, large language models
Read the full text
5 sections · 4 tables · PDF
Create your free account to read the full text.
- Full text with figures and tables
- PDF and supplementary files
- Access on every Galenos journal
One Galenos account works on every Galenos journal site.
References
Kurinczuk JJ, Bower C, Lewis B, Byrne G. Congenital hypothyroidism in Western Australia 1981-1998. J Paediatr Child Health. 2002;38:187-191.
Hinton CF, Harris KB, Borgfeld L, Drummond-Borg M, Eaton R, Lorey F, Therrell BL. Trends in incidence rates of congenital hypothyroidism related to select demographic factors: data from the United States, California, Massachusetts, New York, and Texas. Pediatrics. 2010;125(Suppl 2):S37-S47.
Waller DK, Anderson JL, Lorey F, Cunningham GC. Risk factors for congenital hypothyroidism: an investigation of infant’s birth weight, ethnicity, and gender in California, 1990-1998. Teratology. 2000;62:36-41.
Tuli G, Munarin J, Tessaris D, Matarazzo P, Einaudi S, de Sanctis L. Incidence of primary congenital hypothyroidism and relationship between diagnostic categories and associated malformations. Endocrine. 2021;71:122-129. Epub 2020 Jun 7
Deladoëy J, Ruel J, Giguère Y, Van Vliet G. Is the incidence of congenital hypothyroidism really increasing? A 20-year retrospective population-based study in Québec. J Clin Endocrinol Metab. 2011;96:2422-2429. Epub 2011 Jun 1
Danner E, Niuro L, Huopio H, Niinikoski H, Viikari L, Kero J, Jääskeläinen J. Incidence of primary congenital hypothyroidism over 24 years in Finland. Pediatr Res. 2022;93:649-653. Epub 2022 Jun 3
McGrath N, Hawkes C, McDonnell C, Cody D, O’Connell SM. Incidence of congenital hypothyroidism over 37 years in Ireland. Pediatrics. 2018;142:e20181199.
van Trotsenburg P, Stoupa A, Léger J, Rohrer T, Peters C, Fugazzola L, Cassio A, Heinrichs C, Beauloye V, Pohlenz J, Rodien P, Coutant R, Szinnai G, Murray P, Bartés B, Luton D, Salerno M, de Sanctis L, Vigone M, Krude H, Persani L, Polak M. Congenital hypothyroidism: a 2020-2021 consensus guidelines update-an ENDO-European Reference Network Initiative Endorsed by the European Society for Pediatric Endocrinology and the European Society for Endocrinology. Thyroid. 2021;31:387-419.
Selva KA, Harper A, Downs A, Blasco PA, Lafranchi SH. Neurodevelopmental outcomes in congenital hypothyroidism: comparison of initial T4 dose and time to reach target T4 and TSH. J Pediatr. 2005;147:775-780.
Rose SR, Wassner AJ, Wintergerst KA, Yayah-Jones NH, Hopkin RJ, Chuang J, Smith JR, Abell K, LaFranchi SH; Section on Endocrinology Executive Committee; Council on Genetics Executive Committee. Congenital hypothyroidism: screening and management. Pediatrics. 2023;151:e2022060420.
Aleksander PE, Brückner-Spieler M, Stoehr AM, Lankes E, Kühnen P, Schnabel D, Ernert A, Stäblein W, Craig ME, Blankenstein O, Grüters A, Krude H. Mean high-dose l-thyroxine treatment is efficient and safe to achieve a normal IQ in young adult patients with congenital hypothyroidism. J Clin Endocrinol Metab. 2018;103:1459-1469.
Gunduz ME, Matis GK, Ozduran E, Hanci V. Evaluating the readability, quality, and reliability of online patient education materials on spinal cord stimulation. Turk Neurosurg. 2024;34:588-599.
Bagcier F, Yurdakul O, Ozduran E. Top 100 cited articles on ankylosing spondylitis. Reumatismo. 2020;72:218-227.
Gül Ş, Erdemir İ, Hancı V, Aydoğmuş E, Erkoç YS. How artificial intelligence can provide information about subdural hematoma: Assessment of readability, reliability, and quality of ChatGPT and BARD. Medicine (Baltimore). 2024;103:e38009.
Hopkins JM, Logan J, Kichenadasse G, Sorich MJ. Artificial intelligence chatbots will revolutionize how cancer patients access information: ChatGPT represents a paradigm-shift. JNCI Cancer Spectr. 2023;7:pkad010.
Zhang S, Liau ZQG, Tan KLM, Chua WL. Evaluating the accuracy and relevance of ChatGPT responses to frequently asked questions regarding total knee replacement. Knee Surg Relat Res. 2024;36:15.
Ozduran E, Hanci V, Erkin Y. Evaluating the readability, quality and reliability of online patient education materials on chronic low back pain. Natl Med J India. 2024;37:124-130.
Erkin Y, Hanci V, Ozduran E. Evaluation of the reliability and quality of YouTube videos as a source of information for transcutaneous electrical nerve stimulation. PeerJ. 2023;11:e15412.
Ozduran E, Hanci V. Evaluating the readability, quality, and reliability of online information on Sjögren’s syndrome. Indian J Rheumatol. 2023;18:16-25.
Özduran E, Hanci V. Evaluating the readability, quality and reliability of online information on Behçet’s disease. Reumatismo. 2022;74:49-60.
Özduran E, Büyükçoban S. Evaluating the readability, quality and reliability of online patient education materials on post-COVID pain. PeerJ. 2022;10:e13686.
Alkaissi H, McFarlane SI. Artificial hallucinations in ChatGPT: implications in scientific writing. Cureus. 2023;15:e35179.
Özbek İ, Hancı V, of EÖTJ, 2025 undefined. Digital guidance: quality and readability analysis of Artificial Intelligence-generated spondyloarthropathy texts. Turk J Osteoporos. 2025;31:12-18.
Erkin Y, Hanci V, Ozduran E. Evaluating the readability, quality and reliability of online patient education materials on transcutaneous electrical nerve stimulation (TENS). Medicine (Baltimore). 2023;102.16:e33529.
Ladhar S, Koshman SL, Yang F, Turgeon R. Evaluation of online written medication educational resources for people living with heart failure. CJC Open. 2022;4:858-865.
Moult B, Franck LS, Brady H. Ensuring quality information for patients: development and preliminary validation of a new instrument to improve the quality of written health care information. Health Expect. 2004;7.2:165-175.
Avra TD, Le M, Hernandez S, Thure K, Ulloa JG. Readability assessment of online peripheral artery disease education materials. J Vasc Surg. 2022;76:1728-1732. Epub 2022 Aug 2
Simpson D. The readability test tool. Available from: https://www.readabilityformulas.com
Pohl NB, Derector E, Rivlin M, Bachoura A, Tosti R, Kachooei AR, Beredjiklian PK, Fletcher DJ. A quality and readability comparison of artificial intelligence and popular health website education materials for common hand surgery procedures. Hand Surg Rehabil. 2024;43:101723. Epub 2024 May 21
Lee B, Dixon E, Wales DP. Evaluation of reading level of result letters sent to patients from an academic primary care practice. Health Serv Res Manag Epidemiol. 2023;10:23333928231172142.
Onder E, Ensari E. ChatGPT-4o’s performance on pediatric vesicoureteral reflux. J Pediatr Urol. 2025;21:504-509. Epub 2024 Dec 7
Kara M, Ozduran E, Kara MM, Özbek İC, Hancı V. Evaluating the readability, quality, and reliability of responses generated by ChatGPT, Gemini, and Perplexity on the most commonly asked questions about Ankylosing spondylitis. PLoS One. 2025;20:e0326351.
Khosravi M, Zare Z, Mojtabaeian SM, Izadi R. Artificial intelligence and decision-making in healthcare: a thematic analysis of a systematic review of reviews. Health Serv Res Manag Epidemiol. 2024;11:23333928241234863.
Musheyev D, Pan A, Loeb S, Kabarriti AE. How well do artificial intelligence chatbots respond to the top search queries about urological malignancies? Eur Urol. 2024;85:13-16. Epub 2023 Aug 10
Ozduran E, Akkoc I, Büyükçoban S, Erkin Y, Hanci V. Readability, reliability and quality of responses generated by ChatGPT, Gemini, and Perplexity for the most frequently asked questions about pain. Medicine (Baltimore). 2025;104.11;e41780.
Sarangi P, Mondal H. Response generated by large language models depends on the structure of the prompt. Indian J Radiol Imaging. 2024;34:574-575.
Momenaei B, Wakabayashi T, Shahlaee A, Durrani AF, Pandit SA, Wang K, Mansour HA, Abishek RM, Xu D, Sridhar J, Yonekawa Y, Kuriyan AE. Appropriateness and readability of ChatGPT-4-generated responses for surgical treatment of retinal diseases. Ophthalmol Retina. 2023;7:862-868. Epub 2023 Jun 3
Kim MJ, Admane S, Chang YK, Shih KK, Reddy A, Tang M, La Cruz M, Taylor TP. Chatbot performance in defining and differentiating palliative care, supportive care, hospice care. J Pain Symptom Manage. 2024;67:e381-e391. Epub 2024 Jan 12
Foster BJ, Mitsnefes M, Dahhou X, Zhang X, Laskin BL. Changes in excess mortality from end-stage renal disease in the United States from 1995 to 2013. Clin J Am Soc Nephrol. 2018;13:91-99. Epub 2017 Dec 14
Garcia Valencia OA, Thongprayoon C, Miao J, Suppadungsuk S, Krisanapan P, Craici IM, Jadlowiec CC, Mao SA, Mao MA, Leeaphorn N, Budhiraja P, Cheungpasitporn W. Cheungpasitporn W. Empowering inclusivity: improving readability of living kidney donation information with ChatGPT. Front Digit Health. 2024;6:1366967.
Zaki HA, Mai M, Abdel-Megid H, Liew SQR, Kidanemariam S, Omar AS, Tiwari U, Hamze J, Ahn SH, Maxwell AWP. Using ChatGPT to improve readability of interventional radiology procedure descriptions. Cardiovasc Intervent Radiol. 2024;47:1134-1141. Epub 2024 Jul 9
Brito LNS, de Andrade CLO, de Aragão Dantas Alves C. Adhesion to treatment by children with congenital hypothyroidism: knowledge of caregivers in Bahia State, Brazil. Rev Paul Pediatr. 2021;39:e2020074.
Wolf MS, Gazmararian JA, Baker DW. Health literacy and functional health status among older adults. Arch Intern Med. 2005;165:1946-1952.
Grippaudo FR, Nigrelli S, Patrignani A, Ribuffo D, Tognini S, La Mastra C. Quality of the information provided by ChatGPT for patients in breast plastic surgery: are we already in the future? JPRAS Open. 2024;40:99-105.
Association of Women’s Health, Obstetric and Neonatal Nurses. Health literacy. Available from: http://lib.ncfh.org/pdfs/6617.pdf
Özduran E. “Bel ağrısı” ile ilgili Türkçe internet kaynaklı hasta eğitim materyallerinin okunabilirliklerinin değerlendirilmesi. DEU Tıp Derg. 2022;36:135-150.
