Comparative Study of GenAI (ChatGPT) vs. Human in Generating Multiple Choice Questions Based on the PIRLS Reading Assessment Framework

Yu Yan Lam; Samuel Kai Wah Chu; Elsie Li Chen Ong; Winnie Wing Lam Suen; Lingran Xu; Lavender Chin Lui Lam; Scarlett Man Yu Wong

doi:10.1002/pra2.1054

Comparative Study of GenAI (ChatGPT) vs. Human in Generating Multiple Choice Questions Based on the PIRLS Reading Assessment Framework

Yu Yan Lam^*
, Samuel Kai Wah Chu
, Elsie Li Chen Ong
, Winnie Wing Lam Suen
, Lingran Xu
, Lavender Chin Lui Lam
, Scarlett Man Yu Wong

^*Corresponding author for this work

Academy of Wellness and Human Development

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

1 Citation (Scopus)

Abstract

Human-generated multiple-choice questions (MCQs) are commonly used to ensure objective evaluation in education. However, generating high-quality questions is difficult and time-consuming. Generative artificial intelligence (GenAI) has emerged as an automated approach for question generation, but challenges remain in terms of biases and diversity in training data. This study aims to compare the quality of GenAI-generated MCQs with humans-created ones. In Part 1 of this study, 16 MCQs were created by humans and GenAI individually with alignment to the Progress in International Reading Literacy Study (PIRLS) assessment framework. In Part 2, the quality of MCQs generated was assessed based on the clarity, appropriateness, suitability, and alignment to PIRLS by four assessors. Wilcoxon rank sum tests were conducted to compare GenAI versus humans generated MCQs. The findings highlight GenAI's potential as it was difficult to differentiate from human created questions and offer recommendations for integrating AI technology for the future.

Original language	English
Title of host publication	Association for Information Science & Technology 2024 87th Annual Meeting, assis&t 2024
Editors	Heather O’Brien, June Abbas
Publisher	Wiley
Pages	537-540
Number of pages	4
DOIs	https://doi.org/10.1002/pra2.1054
Publication status	Published - Oct 2024
Event	87th Annual Meeting of the Association for Information Science and Technology - Calgary, Canada Duration: 25 Oct 2024 → 29 Oct 2024 https://www.asist.org/meetings-events/am/am24/ (Conference website) https://asistdl.onlinelibrary.wiley.com/toc/23739231/2024/61/1 (Conference proceeding)

Publication series

Name	Proceedings of the Association for Information Science and Technology
Publisher	John Wiley & Sons Ltd
Number	1
Volume	61
ISSN (Print)	2373-9231
ISSN (Electronic)	2373-9231

Conference

Conference	87th Annual Meeting of the Association for Information Science and Technology
Abbreviated title	ASIS&T 2024
Country/Territory	Canada
City	Calgary
Period	25/10/24 → 29/10/24
Internet address	https://www.asist.org/meetings-events/am/am24/ (Conference website) https://asistdl.onlinelibrary.wiley.com/toc/23739231/2024/61/1 (Conference proceeding)

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 4 Quality Education

User-Defined Keywords

GenAI
PIRLS
question assessment
question creation
Reading

Access to Document

10.1002/pra2.1054

Cite this

Lam, Y. Y., Chu, S. K. W., Ong, E. L. C., Suen, W. W. L., Xu, L., Lam, L. C. L., & Wong, S. M. Y. (2024). Comparative Study of GenAI (ChatGPT) vs. Human in Generating Multiple Choice Questions Based on the PIRLS Reading Assessment Framework. In H. O’Brien, & J. Abbas (Eds.), Association for Information Science & Technology 2024 87th Annual Meeting, assis&t 2024 (pp. 537-540). (Proceedings of the Association for Information Science and Technology; Vol. 61, No. 1). Wiley. https://doi.org/10.1002/pra2.1054

Lam, Yu Yan ; Chu, Samuel Kai Wah ; Ong, Elsie Li Chen et al. / Comparative Study of GenAI (ChatGPT) vs. Human in Generating Multiple Choice Questions Based on the PIRLS Reading Assessment Framework. Association for Information Science & Technology 2024 87th Annual Meeting, assis&t 2024. editor / Heather O’Brien ; June Abbas. Wiley, 2024. pp. 537-540 (Proceedings of the Association for Information Science and Technology; 1).

@inproceedings{73a8c93a04ed4664a44e404c2d0595f0,

title = "Comparative Study of GenAI (ChatGPT) vs. Human in Generating Multiple Choice Questions Based on the PIRLS Reading Assessment Framework",

abstract = "Human-generated multiple-choice questions (MCQs) are commonly used to ensure objective evaluation in education. However, generating high-quality questions is difficult and time-consuming. Generative artificial intelligence (GenAI) has emerged as an automated approach for question generation, but challenges remain in terms of biases and diversity in training data. This study aims to compare the quality of GenAI-generated MCQs with humans-created ones. In Part 1 of this study, 16 MCQs were created by humans and GenAI individually with alignment to the Progress in International Reading Literacy Study (PIRLS) assessment framework. In Part 2, the quality of MCQs generated was assessed based on the clarity, appropriateness, suitability, and alignment to PIRLS by four assessors. Wilcoxon rank sum tests were conducted to compare GenAI versus humans generated MCQs. The findings highlight GenAI's potential as it was difficult to differentiate from human created questions and offer recommendations for integrating AI technology for the future.",

keywords = "GenAI, PIRLS, question assessment, question creation, Reading",

author = "Lam, \{Yu Yan\} and Chu, \{Samuel Kai Wah\} and Ong, \{Elsie Li Chen\} and Suen, \{Winnie Wing Lam\} and Lingran Xu and Lam, \{Lavender Chin Lui\} and Wong, \{Scarlett Man Yu\}",

note = "Publisher Copyright: 87 Annual Meeting of the Association for Information Science \& Technology | Oct. 25 – 29, 2024 | Calgary, AB, Canada.; 87th Annual Meeting of the Association for Information Science and Technology, ASIS\&T 2024 ; Conference date: 25-10-2024 Through 29-10-2024",

year = "2024",

month = oct,

doi = "10.1002/pra2.1054",

language = "English",

series = "Proceedings of the Association for Information Science and Technology",

publisher = "Wiley",

number = "1",

pages = "537--540",

editor = "Heather O{\textquoteright}Brien and June Abbas",

booktitle = "Association for Information Science \& Technology 2024 87th Annual Meeting, assis\&t 2024",

url = "https://www.asist.org/meetings-events/am/am24/, https://asistdl.onlinelibrary.wiley.com/toc/23739231/2024/61/1",

}

Lam, YY, Chu, SKW, Ong, ELC, Suen, WWL, Xu, L, Lam, LCL & Wong, SMY 2024, Comparative Study of GenAI (ChatGPT) vs. Human in Generating Multiple Choice Questions Based on the PIRLS Reading Assessment Framework. in H O’Brien & J Abbas (eds), Association for Information Science & Technology 2024 87th Annual Meeting, assis&t 2024. Proceedings of the Association for Information Science and Technology, no. 1, vol. 61, Wiley, pp. 537-540, 87th Annual Meeting of the Association for Information Science and Technology, Calgary, Canada, 25/10/24. https://doi.org/10.1002/pra2.1054

Comparative Study of GenAI (ChatGPT) vs. Human in Generating Multiple Choice Questions Based on the PIRLS Reading Assessment Framework. / Lam, Yu Yan; Chu, Samuel Kai Wah; Ong, Elsie Li Chen et al.
Association for Information Science & Technology 2024 87th Annual Meeting, assis&t 2024. ed. / Heather O’Brien; June Abbas. Wiley, 2024. p. 537-540 (Proceedings of the Association for Information Science and Technology; Vol. 61, No. 1).

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

TY - GEN

T1 - Comparative Study of GenAI (ChatGPT) vs. Human in Generating Multiple Choice Questions Based on the PIRLS Reading Assessment Framework

AU - Lam, Yu Yan

AU - Chu, Samuel Kai Wah

AU - Ong, Elsie Li Chen

AU - Suen, Winnie Wing Lam

AU - Xu, Lingran

AU - Lam, Lavender Chin Lui

AU - Wong, Scarlett Man Yu

N1 - Publisher Copyright: 87 Annual Meeting of the Association for Information Science & Technology | Oct. 25 – 29, 2024 | Calgary, AB, Canada.

PY - 2024/10

Y1 - 2024/10

N2 - Human-generated multiple-choice questions (MCQs) are commonly used to ensure objective evaluation in education. However, generating high-quality questions is difficult and time-consuming. Generative artificial intelligence (GenAI) has emerged as an automated approach for question generation, but challenges remain in terms of biases and diversity in training data. This study aims to compare the quality of GenAI-generated MCQs with humans-created ones. In Part 1 of this study, 16 MCQs were created by humans and GenAI individually with alignment to the Progress in International Reading Literacy Study (PIRLS) assessment framework. In Part 2, the quality of MCQs generated was assessed based on the clarity, appropriateness, suitability, and alignment to PIRLS by four assessors. Wilcoxon rank sum tests were conducted to compare GenAI versus humans generated MCQs. The findings highlight GenAI's potential as it was difficult to differentiate from human created questions and offer recommendations for integrating AI technology for the future.

AB - Human-generated multiple-choice questions (MCQs) are commonly used to ensure objective evaluation in education. However, generating high-quality questions is difficult and time-consuming. Generative artificial intelligence (GenAI) has emerged as an automated approach for question generation, but challenges remain in terms of biases and diversity in training data. This study aims to compare the quality of GenAI-generated MCQs with humans-created ones. In Part 1 of this study, 16 MCQs were created by humans and GenAI individually with alignment to the Progress in International Reading Literacy Study (PIRLS) assessment framework. In Part 2, the quality of MCQs generated was assessed based on the clarity, appropriateness, suitability, and alignment to PIRLS by four assessors. Wilcoxon rank sum tests were conducted to compare GenAI versus humans generated MCQs. The findings highlight GenAI's potential as it was difficult to differentiate from human created questions and offer recommendations for integrating AI technology for the future.

KW - GenAI

KW - PIRLS

KW - question assessment

KW - question creation

KW - Reading

UR - https://www.scopus.com/pages/publications/85206810074

UR - https://asistdl.onlinelibrary.wiley.com/doi/10.1002/pra2.1054

U2 - 10.1002/pra2.1054

DO - 10.1002/pra2.1054

M3 - Conference proceeding

AN - SCOPUS:85206810074

T3 - Proceedings of the Association for Information Science and Technology

SP - 537

EP - 540

BT - Association for Information Science & Technology 2024 87th Annual Meeting, assis&t 2024

A2 - O’Brien, Heather

A2 - Abbas, June

PB - Wiley

T2 - 87th Annual Meeting of the Association for Information Science and Technology

Y2 - 25 October 2024 through 29 October 2024

ER -

Lam YY, Chu SKW, Ong ELC, Suen WWL, Xu L, Lam LCL et al. Comparative Study of GenAI (ChatGPT) vs. Human in Generating Multiple Choice Questions Based on the PIRLS Reading Assessment Framework. In O’Brien H, Abbas J, editors, Association for Information Science & Technology 2024 87th Annual Meeting, assis&t 2024. Wiley. 2024. p. 537-540. (Proceedings of the Association for Information Science and Technology; 1). doi: 10.1002/pra2.1054

Comparative Study of GenAI (ChatGPT) vs. Human in Generating Multiple Choice Questions Based on the PIRLS Reading Assessment Framework

Abstract

Publication series

Conference

UN SDGs

User-Defined Keywords

Access to Document

Other files and links

Fingerprint

Cite this