Knowing What the Learner Knows Readings

Assessing Selection-Level Learning

How to Write Multiple-Choice Questions:

A multiple-choice question consists of three parts:

  1. The Question
  2. The correct (or most correct) answer, and
  3. A list of “distracter” (incorrect) answers.

Each of these has one or more guidelines associated with it.

1. Rules of thumb for writing the question.

  • Write the question clearly.
  • Do not try to “trick” the learner with misleading information or ambiguous prose.
  • The question can contain information that is not relevant to the correct answer, but make sure you do not use information that is obviously misleading.
  • Make the question the longest part of the multiple choice item.

2. Rules of thumb for writing the correct answer

  • Do not make the correct answer too obvious or too subtle.
  • Avoid “All of the above” or “None of the above” choices or any other that requires logical thinking in addition to the learning being examined.

3. Rules of thumb for writing the distracters

  • Do not make the distracters obviously wrong.
  • Make the distracters reasonable considerations.
  • Ideally, each distracter would be selected by a equal number of learners who do not know the correct answer.
  • It is a common mistake to make the correct answer the longest of available answers. Avoid this by varying the length of distracters in relation to the correct answer.

How to Write True/False Questions

  • Use true/false questions only when the information is clearly true or clearly false.
  • Use simply constructed sentences for true false statements.
  • Provide a clear, unambiguous way for learners to respond.

How to Write Fill-in-the-Blank/Short Answer Questions

  • Use only simple sentences.
  • Make sure that the missing word or phrase is a specific term (e.g. “pressure”) or qualifying word (e.g. “increase”).
  • Make it clear how many or what length response is required.
  • When used in addition to recognition items, confirm that the recognition items are not cuing performance on these items.

Assessing Organization Level Learning

How to Design Demonstration Assessments

  • Prior to assessment (during instruction), make it clear to students what performance is expected and what cues will be provided when they are evaluated.
  • Ensure that the students are not asked to demonstrate skills or mastery using methods unrelated to mastery, or in conflict with developmental readiness. For example, asking young children to draw diagrams to show mastery of verbal skills such as saying "please" or "thank you" would not measure the desired skills. What would be measured instead would be their ability to draw diagrams, which most young children are developmentally unable to demonstrate.
  • Prepare a scoring guide or rubric that will be used to evaluate responses. It is usually a good idea to share this with learners prior to actual assessment.

How to Write Short Essay Questions (Applies to Integration as well)

  • Describe the approximate length of an expected answer (e.g. 1/2 page, single spaced, 12 point Times font).
  • Provide examples of good answers prior to the assessment for lower levels of learning.
  • For higher levels of learning, do NOT provide examples of good answers. Students will likely attempt to memorize prior examples rather than think of unique responses.
  • For transfer questions, provide criteria of good answers for learners to use in self-evaluation practice.
  • Clearly specify what is required in an acceptable answer.

Creating Effective Rubrics (Applies to Integration as Well)

  • Criteria should be as unambiguous as possible.
  • Specify levels for distinguishing between minimal and higher level performance as needed.
  • Use checkmarks for easily identifiable criterion.
  • Allow room for rater’s comments when the reason for justifying meeting or not meeting a criterion is not obvious.

Assessing Integration Learning

How to Create High-Level Self-Evaluations (Transfer)

  • Require learners to respond concretely and completely prior to beginning the evaluation of the response.
  • Record responses for learner review (via text, recorded audio, recorder video).
  • After responding, display learner and expert performance in like media if possible.
  • Provide prompting questions or self-evaluation items after performance is recorded.
  • Require learners to suggest ways in which their performance can be improved before the next cycle of evaluation.
  • Allow repetition of performance/evaluation cycle until performance is proficient (individuals may vary greatly in time and effort to attain proficiency).
  • Do not allow unlimited access to evaluation. Student performance will often plateau in any specific area, causing additional practice to be increasingly counterproductive. Also, students tend to practice skills they do well more than those they don’t do well in order to achieve success.