How to Write Effective Assessment Questions

The following article is based on the valuable work done by following professionals and researchers in universities and professional bodies in the USA and Canada. I acknowledge their original work with thanks.

To name a few:

A Quick Refresher

Before we start on the topic, I thought it would be useful even for experienced professionals to quicky revise the basics.

Mager’s Instructional Objectives

Robert Mager in his seminal book, Setting Instructional Objectives, posits that an effective learning objective should have the following three qualities.

1. Performance: The objective should be measurable overtly. It should contain verbs that can measured externally. If it is a covert performance (say adding, which happens inside a person’s head) then it should be qualified with an overt performance (like write down the answer). The attribute of ‘performance’ is fundamental to setting a ‘good’ objective that will make learning maximally effective. If you cannot measure it, it is not a good objective.
2. Conditions: This attribute includes information about the conditions or situations where the performance would take place. Does the learner demonstrate the product at a customer’s place, or the customer comes to the demo center? Or is it done virtually? Conditions are important because they affect the performance. If I am presenting to my colleagues as opposed to a group of strangers, my performance is bound to be affected. Mager advises us to take it into consideration.
3. Criteria: This is the level of acceptable performance. Again, your content, assessment and design strategy will be influenced by the passing score. Is it 100% or 80% or 50%? There are certain tasks that are risky (working with explosives), costly (operating a delicate machinery) or difficult to perform (demonstrating a complex software). The criterion of acceptable performance varies according to the type of task, so will your design and assessment.

Bloom’s Taxonomy

Benjamin’s Bloom’s taxonomy is boon to learning designers. It is a ready-to-use lexicon to not only set learning objectives but also to place your course/assessment in the appropriate level of learning. The pyramid/circle shows different levels of learning, starting with the basic level, knowledge recall to comprehension and goes all the way to synthesis. Later versions of his model add “judgement” as the highest level of learning. It is so easy to just pick and/or modify from the exhaustive list of performance verbs.

Learning Design

The sequence of designing a learning course, be in a classroom or eLearning, usually follows this path – Analysis, Design and Assessments (adapted from the ADDIE model). I dropped development and implementation as we are talking only about the design aspects.

1. Analysis: Identifying the performance goals, study of learners and context of their learning.
2. Design: Setting learning objectives, framing summative assessment and then the course structure.

Please note that the assessments are framed immediately after the objectives are set, not at the end. Doing so, will create a lot of other problems for the instructional designer.

Multiple Choice Questions (MCQs): A Primer

An MCQ contain two parts:

1. Stem (contains the question)
2. Options (contain answer and distracters)

MCQs are used for testing facts, concepts, principles. As per levels of Bloom’s Taxonomy, they are used to test knowledge, comprehension, application, analysis, and evaluation. MCQs are not used for synthesis as it involves creating something new.

The stem may be in the form of a question or an incomplete statement. The list of options/alternatives contains one correct or best alternative (answer) and several incorrect or inferior options/alternatives (distracters).

It is recommended that the distractors or the other options should be 2x the number of correct answers. It means if it is a single select (one right answer), the distractors should be 2. It is a multiple select with 2 correct answers, the distractors should be 4.

Options/Alternatives should plausible solutions to those who don’t know the answer, implausible solutions for those who know the answer. They should not be some random option which can be easily eliminated by the test taker. The option should make them think for a few seconds if it is the right answer.

It is again recommended that MCQs should be single and not negative, multiple select or combined Response.

Guidelines

1. Each question should assess a single learning objective
2. Several questions can assess a single objective if it warrants
3. The learning that is tested should be clearly stated
4. Add explanatory words if needed but remove irrelevant words
5. Try to keep the question in the positive form
6. Distractors should be clear and concise
7. Distractors should be mutually exclusive
8. Distractors should be homogeneous or similar
9. Do not use “all of the above” and “none of the above”
10. The right answer should be only one and it should be clearly the best answer
11. Place the correct answer in each position (a, b, or c) an equal number of times but in a random order
12. Take care not to have mutually exclusive options
13. Avoid overlapping content in the response options
14. Do not use ambiguous words like sometimes, frequently, and often
15. Try not to use negative terms in the lead-in question (like All of the following except)

Steps in Constructing MCQs

1. Identify the Learning that needs to be assessed: Use the Learning Objectives as the frame of reference. Depending on the importance of each of the objectives, decide the number of assessment questions that must be fra Important objectives warrant more assessment questions. Based on the learning objectives and the questions, allot score points to each question. More important the question and the objective, more the points. Based on the points for each question, decided the passing score. It is not always 80%!
2. Frame the Question Stem: If you have set your learning objectives properly, this is the easiest part. All you must do is to flip the objective statement into a question form. Bloom’s taxonomy is handy at this stage, especially when you need to frame multiple questions for a single objective. Make sure you are not using verbs from a different level of learning than where the objective belongs.
3. Create the correct answer: The correct answer is in the objective. Make sure that the answer is worded in a concise and clear manner. It should not give rise to any ambiguity or multiple interpretations.
4. Create the distracters: This is the toughest part. But the good news is that the experts recommend only 2 distractors for each correct answer. They also recommend single select MCQ. So, one right answer and 2 distractors per question. You need to come up with only 2. After all the hard work that you have done till now, don’t take this step lightly as it will spoil everything done till now. Here are some tips:
   1. Put yourself in the test taker’s shoes
   2. Go back to the course content
   3. Identify where there can be a wrong interpretation
   4. Frame the distractor around it
   5. Follow the guidelines given above