| More Than You Ever Wanted to Know About “All That Apply” Questions
by Steven B. Just
Although our assessment system offers many different
question types we know that the vast majority of questions
actually created are multiple choice. The reason for this
is obvious: With the exception of True/False and Yes/No questions
multiple choice (M/C) questions are the easiest questions
to create – and even inexperienced question writers
know not to rely too heavily on T/F and Y/N questions since
there is a 50% chance of guessing the right answer.
All That Apply (ATA) questions (where there
are multiple correct choices) are not nearly as common as
M/C questions, and test takers often complain about them because
they are considered more difficult. With that in mind we pooled
the results data of several of our larger clients and analyzed
their use of ATA questions. (There is no reason to believe
that the same findings wouldn’t be true for our small
to mid-sized clients -- we just wanted to be sure we had a
large enough pool of questions to make our analysis valid.)
Here are some results from our analysis:
- Of the more than 35,000
questions in the item pools approximately 2,400 (or 6.8%)
were All That Apply.
- For the All That Apply
questions, the average percent correct (each question is
judged correct/incorrect; partial credit was not allowed)
was 66%. Since it is typical for our clients to set the
passing test score at 90% and the vast majority of learners
pass the test, that means that on average for any given
question (most of which are multiple choice) 90% or more
of test takers answer each question correctly. This confirms
what the test takers believe: ATA questions are quite a
bit more difficult than M/C questions.
- The most common number
of choices was four (57%), followed by three (29%) and five
(10%). A small number of questions (1%) had nine or more
choices!
- For three-choice questions,
there was a strong bias towards two choices being correct:
one choice was the correct answer 7% of the time, two choices
were correct 64% of the time and all three choices were
correct 29% of the time.
- For four-choice questions,
one choice was the correct answer 4% of the time, two choices
were correct 48% of the time, three choices were correct
36% of the time and all four choices were correct 12% of
the time.
- The pattern of having
either only one choice or all of the choices rarely being
correct continues for all the questions, no matter how many
total choices there are. This makes sense: If only one choice
is correct the question writer would likely use a multiple
choice question rather than an All That Apply, and most
question writers would probably be inclined to put at least
one incorrect choice in the list among the correct choices.
- Bonus fact
(if you are taking a test in which the choices are not randomized,
e.g. a paper and pencil test or a testing system that does
not have the ability to randomize choices): In 66% of the
questions the first choice was one of the correct answers.
This again makes sense if you think about how most people
write questions. Before putting in a choice that is a distractor
(incorrect choice) most (66%!) question writers are inclined
to list one of the correct choices first and then start
considering distractors intermixed with correct answers.
Are these results in line with what we
expected? Clearly we expected All That Apply questions to
be more difficult than Multiple Choice questions. I’m
not sure what we expected about numbers of choices and numbers
of correct answers, but you might use this data as a benchmark
to compare to your own item pool.
|
|
