Practice: Evaluating Exam Questions

This section of the online workshop gives you an opportunity to practice evaluating some multiple choice questions. For each one, consider what attributes of the question might decrease its validity or reliability. You can click the "Show More" to see a revised (better) version of each question along with a rationale for the revision.

These examples are drawn from pharmacy, but the principles apply to any multiple choice questions, and in many cases, more broadly to various types of exam questions. The correct answer for each question is in bold font.

  1. All of the following are common adverse effects associated with drug overdoses of thyroid hormone replacement therapy except --
    1. Cardiac palpitation
    2. Arrhythmias
    3. Tachycardia
    4. Weight gain 
    5. Heat intolerance

Show more

Revised question:
  1. Which of the following adverse effects would be associated with a drug overdose of thyroid hormone replacement therapy?
    1. Cardiac palpitation 
    2. Dry skin
    3. Weight gain 
    4. Dehydration
    5. Unusual odor
Rationale: Questions should ask students to identify the correct answer, rather than an incorrect answer. Identifying what is wrong does not mean the student knows what is correct, so asking students to find an incorrect answer decreases reliability.
  1. A 55 year-old patient has been referred to you. She complains about a skin rash and a cough. In the course of history taking, she tells you that she takes high blood pressure medication but doesn’t remember the name. You suspect --
    1. Captopril 
    2. Nifedipine
    3. Prazosin
    4. Propanolol
    5. Clonidine

Show more

Revised Question:
  1. A 55 year-old patient has been referred to you. She complains about a skin rash and a cough. In the course of history taking, she tells you that she takes high blood pressure medication but doesn’t remember the name. What antihypertensive agent could be causing these drug toxicity symptoms?
    1. Captopril 
    2. Nifedipine
    3. Prazosin
    4. Propanolol
    5. Clonidine
Rationale: Base each item on a specific problem stated in the question. Students should not have to infer what problem is being presented. The question should test specific course-based knowledge or skills, not the ability of the student to make an inference. Reliability and validity decrease if the problems tests inference about the question rather than course material. 
  1. Your PharmD project involves plotting dose response curves for immunosuppressants. You get approval from your advisor to test three currently used products for your project. After an extensive literature review you decide on a methodology and choose rabbits as your test species. When applying for IRB approval you learn that you must provide adequate disposal of all waste including all animal feces and urine, any tissue samples, and all unused immunosuppressants.  You meet with your advisor to discuss your methodology and data collection procedures. She asks, “What would be the best parameter to measure to evaluate efficacy of rabbit antithymocyte globulin and basiliximab, respectively?”
    1. Total lymphocyte counts, B lymphocyte counts
    2. CD3+ cell counts, CD25 
    3. CD4+ cell counts, IL-2 concentration
    4. IL-2 concentration, CD4+ cell counts

Show more

Revised Question: 
  1. Your PharmD project involves plotting dose response curves for immunosuppressants. What would be the best parameter to measure to evaluate efficacy of rabbit antithymocyte globulin and basiliximab, respectively?
    1. Total lymphocyte counts, B lymphocyte counts
    2. CD3+ cell counts, CD25 
    3. CD4+ cell counts, IL-2 concentration
    4. IL-2 concentration, CD4+ cell counts
Rationale: Remove excess information in the question. Test learning objectives, not the student’s reading speed or ability to pinpoint the relevant information in a question (unless of course you're deliberately testing the student's ability to do such sorting and prioritizing -- in which case, the question should pertain directly to identifying such information).
 
  1. Gastric acid secretion is not stimulated by the presence of:
    1. Gastrin
    2. Histamine
    3. Acetylcholine
    4. Norepinephrine 

Show more

Revised Question:
  1. Gastric acid secretion is stimulated by the presence of:
    1. Cimetidine
    2. Histamine 
    3. Pepsin
    4. Norepinephrine 
Rationale: The question should be stated as positive instead of negative, unless testing an exception to a rule (e.g., a question about which drugs should not be administered in a given situation due to safety concerns). If using negatives, use clear wording, emphasize that the student is looking for the wrong answer, and avoid double negatives to prevent students from making mistakes due to the question format rather than lack of knowledge.
 
Alternative revised question:
  1. A 20 year-old woman goes to the emergency department stating that within the past hour she ingested “a handful of sleeping pills.” She is still awake. Which of the following drugs SHOULD NOT be given to induce vomiting?
    1. Metoclopramide 
    2. Ipecac
    3. Apomorphine
Rationale: Here, safety issues are involved. They key point is which drug should be avoided, so use of the negative is justified. Note the signaling of a negative response using all capital letters.
 
  1. Carbonic anhydrase inhibitors act on the enzyme responsible for the conversion of carbon dioxide to which of the following?
    1. Bicarbonate and calcium ions
    2. Bicarbonate and sodium ions
    3. Bicarbonate and hydrogen ions 
    4. All of the above.

Show more

Revised question:
  1. Carbonic anhydrase inhibitors act on the enzyme responsible for the conversion of carbon dioxide to produce bicarbonate and which of the following?
    1. calcium ions
    2. sodium ions
    3. hydrogen ions 
    4. lithium ions
Rationale: Alternative responses should not overlap. Each answer should be mutually exclusive to be able to discriminate what part of the answer the student understands. For these reasons, "all of the above" is generally a bad choice for multiple choice exams. 
 
  1. A 52 year-old woman with a history of eczema and heavy alcohol use begins taking ibuprofen to control hip and knee pain due to osteoarthritis. Over the course of six months, as the pain worsens, she increases her dosage to a high level (600 mg four times daily). What toxicity is most likely to occur?
    1. Abnormal heart rhythms
    2. Necrotizing fasciitis
    3. Gastric ulceration - heavy alcohol use increases the susceptibility of an individual to ibuprofen induced GI toxicity. 
    4. Confusion and ataxia
    5. Eosinophilia

Show more

Revised question: 
  1. A 52 year-old woman with a history of eczema and heavy alcohol use begins taking ibuprofen to control hip and knee pain due to osteoarthritis. Over the course of six months, as the pain worsens, she increases her dosage to a high level (600 mg four times daily). What toxicity is most likely to occur?
    1. Abnormal heart rhythms
    2. Necrotizing fasciitis
    3. Gastric ulceration 
    4. Confusion and ataxia
    5. Eosinophilia
Rationale: Keep answer options consistent in length and structure to avoid providing clues to the correct answer.
  1. The ability of chemotherapy to kill cancer cells depends on which of the following factors?
    1. Its ability to destroy specific tissues
    2. Its ability to stop cell division 
    3. Its ability to cause necrosis of tissues
    4. All of the above
    5. None of the above

Show more

Revised question:
  1. The ability of chemotherapy to kill cancer cells depends on which of the following factors? Its ability to
    1. destroy specific tissues
    2. stop cell division 
    3. cause necrosis of tissues
    4. initiate cell regeneration
Rationale: Avoid use of both “All of the Above” and “None of the Above.” The second decreases reliability because it allows the student to eliminate "all of the above" if one wrong answer is identified. Identifying none of the above does not indicate that the student knows the correct answer.
 

Next Section

Click to go to the next section: Building Exams

 

shadow

Comments

Wow

Definitely intriguing changes presented here. Most of these I would not have thought were wrong (e.g., asking for "all of the above" or what is not the correct answer). The class I GSI for at the moment has some of these question types, so it is neat that I am already going to be able to apply what I am learning.

negative questions

This was really eye opening for me in relation to positive and negative questions. I've GSIed for a few classes where negative questions have appeared in exams. I wonder if there isn't a value in the end for teaching students to think backwards as well by using negative questions although I don't see that aligned with any of the goals of testing.

reflection on phrasing multiple choice questions

This part of the lesson gave me a sense for how I should write up multiple choice questions in the future. A few things I'm going to keep in mind are the following: 1) construct questions in positive terms 2) write the questions in succinct manner 3) avoid using none of the above/all of the above if at all possible.

Writing Exams

This section was probably the most surprising so far - I definitely use "all of the above," "none of the above," and "all of the following EXCEPT" somewhat regularly when writing exam questions. While I see the points that are made, there are some times when it is very difficult to write a multiple choice question and come up with 3 or 4 incorrect answers - this is often when I will resort to using "all of the following except" or "all of the above/none of the above."

refining question

I like the way of how the question is refined. it is concise ,not complicated and easy to think about the answers.

Refining question

I agree. When writing quizzes and exams I always try to make the question I'm asking obvious, and I always tell students to make sure they are answering the question, not something else. The refined questions make it very plain exactly what that question is.

Negative questions

Is it really that bad to put negative questions (e.g. #4)? This course seems very averse to them. I have been taking tests with questions in this format my entire life, and haven't really ran into any problems with comprehension or efficacy of the questions. On the contrary, I think they encourage complete understanding of a subject *when written correctly*. So should I avoid these despite my long history with them? Or just ensure that I know how to write them before putting them in?

I think they are fine. I'm

I think they are fine. I'm used to them as well. I think as long as capitalization/bold/underline--some way of alerting the student that it's a different type of question--is used, it is okay. Being able to carefully read things seems to also be in important skill too.