If you teach at McGill and give students multiple choice question (MCQ) final exams, you’ll be receiving a “Test Item Statistics Report” from the Exam Office sometime after the end of term. This report, also known as an “item analysis report,” is sent to you along with the exam results. The report lets you know all kinds of interesting things about your exam, such as:
- how difficult your questions were
- how well your questions discriminated
- how well your distractors worked
The numbers in the report may seem daunting to interpret, but resources exist to help you make sense of them. McGill’s Exam Office links to an interpretation report that will help you understand the numbers.
If you’d like to be walked through the statistics step-by-step, with detailed explanations and concrete examples of what the numbers mean, we recommend a resource that even the most numbers-challenged among us can understand: Chapter 6, Improving Your Tests, in Learning and assessing with multiple-choice questions in college classrooms (Parkes & Zimmaro, 2016). The book is available online through the McGill library.
As the authors explain, the interpretation of item analysis reports is contextual. They pose the question: “How difficult should your MCQs be for students?” (p. 72). The answer to that question has to take into consideration the purpose of the exam: Is it to measure what students know/don’t know (as in gate-keeping)? Is it to assess what students have learned from your teaching? These might be competing goals. The authors address this matter on pages 71-73. Emphasis is on the importance of interpreting numbers in context.
Need help writing quality MCQs? Teaching and Learning Services (TLS) usually offers a workshop in the fall term that addresses best practices for writing MCQs, including assessment of higher order learning. If you’re a McGill instructor interested in attending this workshop, please check the TLS calendar in the fall.
Acknowledgements
This blog post was inspired by Prof. Argerie Tsimicalis, Ingram School of Nursing, who suggested that support for interpreting MCQ statistics be more widely advertised at McGill.
This blog post was reviewed prior to publication by Rhonda Amsel, professor of statistics in the Department of Psychology.
References
Parkes, J., & Zimmaro, D. (2016). Learning and assessing with multiple-choice questions in college classrooms. New York: Routledge.
Original publication date: April 24, 2018
Associate Director, Faculty and Teaching Development, and Senior Academic Associate, at McGill's Teaching and Learning Services; former Senior Faculty Lecturer at the McGill Writing Centre; area of specialization: Second Language Education; loves teaching and learning!
(Photo credit: Owen Egan)
You must log in to post a comment.