In-conference workshop
Automated Item Generation – A practical approach

Facilitated by
Associate Professor José Miguel Pêgo
School of Medicine – University of Minho, Portugal

José Miguel Pêgo is the Chairman of the Item Reviewing Committee and the Chairman of the Assessment Committee at School of Medicine – University of Minho. He is also a shareholder and product manager at iCognitus that specializes in the development of assessment software. He has run multiple Item Writing Workshops and has dedicated is recent research to AIG.

Workshop description

Multiple-choice items are the current standard for assessment in the cognitive domain of medical education. Despite being an efficient validated method, the development of each item is, still, a complex and time-consuming and, for that reason, a very expensive process.
With the increasing demand for large number of multiple choice items by medical schools, the current approach to item development struggles to meet this high demand. An innovative solution to this challenge is the use of automatic item generation (AIG) to develop new multiple-choice items.

AIG uses computer technology to generate new questions based in cognitive and item models, created by content specialists. While full automated item generation demands the access to IT resources (hardware and manpower) it is possible to develop simpler generators with household tools that are available in every personal computer. The critical step is the development of a cognitive model and the definition of its parameters that can be done on a simple piece of paper.

Attendees will be given the opportunity to work through development of cognitive models and program an excel worksheet to accomplish the algorithm of decision within a set of items.

AIG is a relatively new and lesser known technology, that requires a lot of development and further validation, but, with no doubt, it will be a technology that will shape the future of item development in medical education.

Workshop format

We will present templates and algorithms we use to design items and cognitive models. Attendees will be encouraged to work in groups, to think about what scenarios they could develop and implement AIG in their own exams. We will demonstrate the hardware/software and allow attendees to try these out.

After a didactic presentation of the case for AIG using this methodology we would invite participants to discuss possible scenarios and how these might be implemented. Participants will be organized in small workgroups that will develop a cognitive model. We would encourage hands-on experimentation with excel worksheets understanding the basic programming that will lead to the development of an automated item generator. No prior work would be needed.

Objectives

  • To introduce the concept of AIG and cognitive models
  • To provide a framework that participants can take to use in their own institutions

Workshop outcomes

We shall provide attendees with copies of slides, examples of cognitive models and data and excel files they can take to adapt for their own exams.
Presenting a workshop at an EBMA conference has reputational benefit for our University. Additionally we would hope that this workshop would benefit the aims of the EBMA Conference.

References

  • Gierl MJ, Lai H, Turner SR. Using automatic item generation to create multiple-choice test items. Med Educ. 2012;46(8):757-765. doi:10.1111/j.1365-2923.2012.04289.x.
  • Pugh D, De Champlain A, Gierl M, Lai H, Touchie C. Using cognitive models to develop quality multiple-choice questions. Med Teach. 2016;38(8):838-843. doi:10.3109/0142159X.2016.1150989.
  • Paniagua MA, Swygert KA. Constructing Written Test Questions For the Basic and Clinical Sciences. 2016. http://www.nbme.org/publications/item-writing-manual.html.

Additional Information – Participants are asked to bring their own laptop

email José Miguel Pêgo: jmpego@med.uminho.pt