Video Title: How ETS Develops Test Questions

People in this video

Barbara Wiener - Test Development Manager

Jerry Bicknell - Test Developer

Willisa Roland - Fairness Reviewer & Test Development Manager

Bernadette Gur - Editor

Rochelle Michael - Statistician


[music playing]

Barbara Wiener (Test Development Manager): It takes multiple jobs to produce a test. It's not one person that's putting the test together. It's multiple test developers. It's people trained in fairness. It's editors looking at the questions, the statisticians that are analyzing the test. So it's a huge process from beginning to end to make a successful test.

Create Framework
Question (“Item”) Formats:  multiple choice, fill in the blank, essay, other formats, performance assessment
Make Blueprint

Narrator: Every part of the test creation process is a team effort. The first step usually involves a committee of clients, content experts and other stakeholders that provide a framework, an outline of the purpose of the assessment and who and what should be tested. The framework also specifies the formats of the questions or items as they’re often called, as well as whether the test will be offered on paper, computer, or both.
Using the framework as a guide, the next step is a blueprint.A detailed specification for every question that will appear on the test. Different forms or versions of the test can be built from the same blueprint.

Barbara Wiener: It’s like a little jigsaw puzzle. It’s got all these little pieces that every form has to fit, every test question has a little spot on that blueprint and we want to be able to replicate that. So, it is pretty detailed and pretty descriptive.

Write Question

Narrator: The blueprint guides the question writers, most of whom have years of experience teaching the subject being tested.

Jerry Bicknell (Test Developer): Typically, I'll come up with the basic idea of what I want to ask so the question, we call it the stem part of the item, and the correct answer of course. You have to have one correct answer and only one correct answer. Then, the harder part is coming up with distracters, we call them, answers that the student who doesn't know the material finds attractive for the wrong reason.

Question: If there are exactly 5 times as many children as there are adults at a show, which of the following could be the number of people at the show?
Answer: (B) 72

Narrator: The goal is to get accurate information about the test-takers knowledge and skills.

Jerry Bicknell: I think a lot of people and some teachers even have this philosophy of, "Oh, I'm really going to get them with this question. I'm going to make a really tough question that nobody can answer, and that's not our point at all. The point is to write a question that tells us solid information about the students and what they do know.

Content Review

Narrator: The question then gets reviewed by other test developers to determine the accuracy of the correct answer and to ensure that none of the distractors could be correct under any plausible interpretation of the question. The reviewers also confirm that the item matches the test blueprint. 

Question: If there are exactly 5 times as many children as there are adults at a show, which of the following could be the number of people at the show?
Answer: (B)* 72
*correct number must be a multiple of 6
a= number of adults
5a=number of children
6a=total number of people

Fairness Review

Narrator: Every test developer is trained in ETS’s fairness guidelines. In addition, specialized fairness reviewers scrutinize each question to be sure that it is free from bias.

Willisa Roland (Fairness Reviewer & Test Development Manager): If a person has a particular skill, we want to make sure that the test question is testing that particular skill and nothing else. It’s not just a matter of not offending anybody. If a test question is unfair, it’s not really useful for the person who’s taking it or the person who is using that score to make a decision. So, it’s very important that we make sure that everyone has the same opportunity to show their true knowledge skills and abilities.

Editorial Review

Narrator: The next step is the editorial review.

Bernadette Gur (Editor): There's actually volumes of guidelines that we have to apply to every item. The item should be as clean and as clear as possible. We're looking for clarity. We're looking for editorial soundness. We're looking for usage. We also look for various things that are program specific, which means certain programs have phrases and nuances that are specific to that program that have to be applied. So, there's a lot of different things that go into editing an item. It sounds really simple, but it can be pretty complicated.

Question: Which of the following is most directly involved in controlling sugar?

Stakeholder Review

Narrator: Stakeholders have several opportunities to review the questions before they're included in a test in addition to all the reviews performed by ETS Experts.

Barbara Weiner: No one passes the item through a particular review until they're really happy how an item looks, and that's how we get really good multiple reviews.

Pilot Testing

Narrator: To see how a new question performs with actual test-takers, the next step is pilot testing followed by a statistical review.

Rochelle Michael (Statistician): During a pilot test, you have test items administered to small groups of individuals to see how items are performing in terms of their psychometric properties, their statistical properties, the difficulty, how well it's differentiating between high and low performers, getting a sense if it's performing equally well for different groups, and I might look at things and say, "It's not performing as we would like," and I'll make some notes on that, and feed that back to the test developers so they can put that content perspective on why the test item might be performing the way that it is.

Ready for Use

Narrator: A question must meet rigorous standards and pass every review before it can finally appear on an ETS test.

Barbara: We take every question seriously. We take every test seriously.

Bernadette: That's really our mission to provide excellent assessments to candidates all over the world.

Willisa: One of the important goals of ETS is really equity is to give everyone a fair chance.

Jerry: That's always my primary motivation is to make the best item I can so that it's fair for all the students.

Rochelle: It's very satisfying to know that when a student sits down to take an assessment that I have worked on, that they're getting a good assessment.

Barbara: We're affecting people's lives, so we have to make sure that what we do is the right thing.

Total length of video: 5:59