Getting students to focus on questions, not the answers

On May 5, 2017, McGill’s Assessment and Feedback Group held an event entitled Getting students to focus on the questions, not the answers as part of its Brown Bag Series. To an audience of peers, two instructors described assignments they use in their courses that call upon students to create questions as a means for engaging them with course content and getting them to think about how they learn.

Below, Penelope Kostopoulos, a Faculty Lecturer in the Department of Psychology, describes her assignment. Carolyn Samuel, formerly a Senior Faculty Lecturer at the McGill Writing Centre, describes her assignment in a post called What’s the prof gonna ask?

Turning the tables: Getting students to ask the questions

Let me introduce myself. My name is Penelope and I’m a member of the Assessment and Feedback Group @ McGill. I’m part of this group because I teach large introductory classes—200 to just over 650 students—and I love it!

What? You love teaching in ginormous auditoriums where students don’t know each other?

Yes! As daunting as it might sound, I love the challenge this represents. I want to harvest the great potential that lies in my students. I want students to actively engage with the course material and support each other’s learning.

But how?

Sure, I can ask students to pair with their neighbor to work on assignments during class time to encourage peer learning, but anyone who has attempted such learning activities in large classrooms will tell you that half the class will pretend not to understand where to turn to find their neighbor and will stare into space. In small classes, professors can go to students and facilitate interactions but, let’s face it, the vision of me jumping over seats to get the students that are sitting next to each other mute to start conversing is not one I would like to foster.

The idea for this assignment came to me while I was reading articles on teaching undergraduate psychology classes: I discovered PeerWise (Denny et al., 2008), a free web-based platform that allows students to create, share and evaluate multiple choice questions related to course material. I decided then to design an assignment where I turned the tables: instead of me giving students questions to answer, I assigned students the task of creating questions for each other to answer. 

Assignment description – The devil is in the details

Counting towards 10% of their final grade, the assignment asks students to create multiple choice questions on the course material on two separate dates and then answer each other’s questions prior to the midterm and final exam dates. Students receive grades both for creating and for answering and evaluating each other’s questions. Simple to setup and implement, and a piece of cake to grade.

Students create six multiple choice questions related to the course material. Each question has to include one correct answer and a minimum of three distractors. Students provide an explanation for the choice of correct answer and reasons why the distractors are incorrect.

Students then answer 20 questions created by their peers, rate them in terms of difficulty (easy, medium, hard) and quality (with a numeric rating), and provide constructive comments to the author. All of this is done anonymously, which promotes broader student participation in conversations. Note that, as the instructor, I can identify the students and intervene in conversations where students are clearly struggling with concepts or where questions are inappropriate. The platform allows students to filter questions: they can choose which ones to answer based on popularity among peers, level of difficulty, and number of comments received. Most importantly, students can filter questions by topic in order to hone in on areas where they feel they need more practice.

Students by and large create the required number of questions plus or minus a couple, but many students answer more and a handful of students have come close to—or even surpassed—answering 1000 questions! Overly eager? Maybe!

I think the gaming interface that awards badges to students for different actions helps. The gamification of something as dreaded as studying can be extremely motivating for students. As an instructor, I’m just happy to see students actively involved with the course material.

As an additional motivator, I let students know that I include at least one student-created question in each exam. How do they respond to that? I will let one of my students answer: “One of my questions got used on the final exam! It was amazing!!!”

 Student reaction to the assignment

Change in the classroom can be a challenge and is not always well received by students. I wanted to know how students felt having the tables turned with this assignment. So I surveyed them. To my delight, students’ responses have been overwhelmingly positive. They find the assignment:

• easy to complete on the online platform (92%)
• worth the marks allocated (86%)
• helpful in increasing their understanding of the course material (86%)
• helpful in encouraging them to think critically about the material (83%)

Fewer students, however, find the assignment helpful in preparing them for the course exams (66%) … But more on this later.

In addition to the positive survey responses, students indicated that they got a better appreciation for how difficult it is to create good multiple choice questions. They told me that they learned to value instructors’ efforts in designing their exams. Okay, okay … they might not have used those exact words. Turning the tables did, however, give them the opportunity to see things from the instructor’s perspective and they appreciated that.

Is the assignment effective?

Let’s recap: I designed the assignment to get students to actively engage with the study material and to encourage peer learning. I also wanted students to develop their skills in forming questions. Most importantly, I wanted to cultivate students’ metacognitive skills by having them  reflect on their learning.

So, while you’ll probably agree that, in theory, this assignment is great for achieving these outcomes, you might still wonder: Does it work? Does it really engage students in learning? To my delight, students do get involved in conversations about the choice of distractors or the applicability of a question to course material, and they do so in a constructive and respectful manner. It is not uncommon for a question to receive over thirty comments in a class of 200 students. These conversations definitely encourage peer learning. Students also indicate having a better grasp on their learning and how they are doing in their progress with this assignment. But you don’t have to take my word for it. Here is a comment from one of my students: “I think that it’s definitely a great study tool that is made by the students through a collaborative effort. It’s a great idea.”

And do students use the assignment to prepare for course exams? The graph below shows PeerWise activity for one of my classes. Students clearly used the assignment to prepare for exams. However, in their feedback, many students indicated that it did not actually help them obtain better grades on the exams. With proper ethics approval, it would be worthwhile to find out whether or not that was really the case.

Lessons learned and words of wisdom

 The assignment has gone through some revisions over the course of three years and five classes where I have used it. Live and learn. Here are some of the updates that I thought are worthwhile sharing:

• It’ s important to not only explain the assignment and show students how to write multiple choice questions, but to also provide students with detailed assignment instructions and written guidelines on what constitutes a good and a bad multiple choice question and how to go about creating good questions. Teaching assistants have always been helpful with this part.
• Simply stating assignment requirements is not enough to get students to complete the task according to the instructions, especially given the multiple deadlines. I now use many different ways to highlight information in the instructions document: use BOLD, underline, change fonts, increase fonts, and repeat, repeat, repeat anything related to grading.
• Following a discussion I had with the always helpful staff at Teaching and Learning Services (TLS), I have added a twist to the assignment: I ask students to create questions at different levels of learning. Levels of learning are explained to students using Bloom’s Taxonomy. Students have to create four questions that tap into lower level learning (i.e., questions that involve remembering and understanding) and two questions that tap into higher level learning (i.e., questions that involve applying, analyzing and evaluating).

And my final words of wisdom: If you decide to use an assignment like this in your course, encourage your students to be creative and I promise you will be rewarded. Usernames like one student’s choice of “afreudtolove” for my introductory psychology course make me smile. And I have a collection of Donald Trump questions from Fall 2016 that are extremely entertaining!

1. Harry asked his friend Sally who she is voting for in the Presidential election. Sally said all her friends say Donald Trump is the smartest man alive, which must mean he is the best, so she will vote for him. What would we call the thinking trap Sally is caught in based on her reasoning?
   a) Not me fallacy
   b) Bandwagon fallacy
   c) Bias blind spot
   d) Emotional reasoning fallacy 

2. Trump is now the President of the United States! All his life, he possessed a strong desire to lead and dominate over others; from being a class monitor as a child, to a business mogul, and finally as President. The personality theory which best fits with Trump’s personality is:
   a) Carl Jung’s collective unconsciousness
   b) Henry Thoreau’s civil disobedience
   c) Sigmund Freud’s Oedipus complex
   d) Alfred Adler’s inferiority complex

Hover over this text for the answers!

References

Denny, P., Luxton-Reilly, A., Hamer, J. (2008). The PeerWise system of student contributed assessment questions. Proceedings of the Tenth Conference on Australasian Computing Education, Wollongong, NSW, Australia(78), 69-74.