Artificial Intelligence-Generated and Human Expert-Designed Vocabulary Tests: A Comparative Study

Artificial intelligence (AI) technologies have the potential to reduce workload for second language (L2) teachers and test developers. We propose two AI distractor-generating methods creating Chinese vocabulary items: semantic similarity visual similarity. Semantic refers antonyms synonyms, while phenomenon that phrases share one or more characters in common. This study explores construct validity of types selected-response tests (AI-generated items human expert-designed items) compares their item difficulty discrimination. Both quantitative qualitative data were collected. Seventy-eight students from Beijing Language Culture University asked respond AI-generated respectively. Students’ scores analyzed using two-parameter response theory (2PL-IRT) model. Thirteen then invited report taking strategies think-aloud section. The findings students’ responses revealed easier but had discriminating power than items. results indicated might assess different constructs, which former elicited takers’ bottom-up test-taking latter seemed likely trigger rote memorization ability.