Vocabulary learning is a nearly universal feature of language education curricula. A common assessment approach involves multiple-choice cloze items (MCC; aka 'ana-ume' in Japanese). However, with the advent of the Coronavirus situation and assessment methods moving to online platforms, MCC items present difficulties: Once used online, they should be considered "public" and cannot be re-used; producing items manually is laborious; and there are very few available end-to-end tools to produce items automatically (though see Word Quiz Constructor discussed in Rose 2020).

This poster is a progress report on a project that aims to build a system to generate items automatically for vocabulary training and testing. We will identify and summarize some of the key issues involved in this process and the approaches we are taking to resolve these issues. In particular, the poster will focus on the preparatory steps for using machine learning methods to generate items: the construction and validation of a "gold-standard" set of items.

At present, the project has created gold-standard items using both the General Service List (GSL: West, 1953) and the Academic Word List (AWL: Coxhead, 2000), consisting of a total of 2786 items. This standard is currently being tested in a pilot experiment together with an original vocabulary learning app, with some gamification. We plan to increase the gold standard list over time to at least 4,000 items and evaluate them as well as training and testing methods with a large scale population (>1000 students) in spring, 2022.

Issues that have arisen during the process thus far include estimating item difficulty (cf., Kurdi et al 2019), suitability for various audiences, and list coverage of items. These issues will be discussed along with our resolutions in the present project. This poster should be of interest to vocabulary specialists and programmers working on vocabulary-related educational applications.