ETS offers writing assessment and scoring expertise to institutions that incorporate ETS scoring technology in their courses and online services.
To make use of the e-rater® technology, institutions and companies may either use standard writing prompts and grade-level scoring models available from ETS or develop their own. Standard writing prompts from ETS, and instructor-created prompts that follow ETS guidelines, are submitted to the ETS grade-level scoring models and can receive a holistic score without custom development.
Those who choose to develop their own customized writing prompts, utilizing custom scoring rubrics and requirements to generate a custom scoring model, must collect essay responses to these prompts, score them, and transmit them to ETS so that a scoring model can be built and certified.
ETS can help institutions develop the prompts and score the responses so that a custom e-rater engine scoring model can be built. The steps are outlined below.
Our experienced writing experts at ETS can assist with the development of customized essay topics that reflect national, state or district standards. ETS writing specialists provide expert advice as an institution develops its own customized topics, conducts scoring sessions and analyzes the writing scores. Or, our development specialists can do the development work and conduct the pretest scoring sessions for a client organization, according to its specifications.
Incorporating scoring methodology and quality assurance procedures used in high-stakes performance-assessment testing, ETS development specialists conduct essay scoring sessions and collect scoring data for customized model building.
In addition to handling reader recruitment, training and monitoring, we offer an online scoring system that instructors and trained readers can use to score text-processed essay responses. When scoring is complete, the essay files and accompanying reader scores can be transmitted to the scoring engine for automated model building.
Building the e-rater Engine Models
For the e-rater technology to build a scoring model for a topic, it must first be trained by analyzing essays written on that topic. These essays must have received two scores from independent readers who have applied consistent and fair scoring standards. This pool of scored essays must also represent the entire scoring scale (for example, a scale from 1 to 6), so that the e-rater engine can be trained to recognize essays at each of the score points.
To build a customized scoring model, you will need:
- scoring standards clearly defined in a scoring guide
- an essay topic that reflects the writing task represented in the scoring guide
- a student population identical or at least similar to the population that will be using the topic
- two independent reader scores for each essay response (readers must be trained to use the scoring guide)
- at least 350 student responses per topic in an electronic file, formatted according to ETS specifications
Contact us for more information and assistance in developing topics and guides, scoring essays and building customized writing prompts scored with your scoring guide.