Kentucky Educators Become "Test Readers for a Day" During Louisville Scoring Tutorial

Kentucky educators recently tested their skills during a “crash course” in scoring constructed student responses using Measured Progress’s iScore technology. Ninety teachers and administrators traveled from the Kentucky Teaching and Learning Conference to Measured Progress’s Louisville scoring facility on March 6 to participate in a “mini scoring session.” They were trained to score student responses to a fourth-grade released item from the Commonwealth Accountability Testing System (CATS).

Measured Progress has worked with the Kentucky Department of Education since 2006 to develop and administer the CATS, a standards-based assessment system for approximately 650,000 students each year in grades three through twelve. The CATS includes the Kentucky Core Content Test (KCCT), Writing Portfolios, and the Alternate Assessment Program. The KCCT assesses reading, mathematics, science, social studies, arts and humanities, practical living and vocational studies, and on-demand writing.

Participants in the scoring tutorial were divided into morning and afternoon sessions, both filled to capacity. According to Rhonda Sims, director of the Kentucky Department of Education’s Division of Assessment Support, the sessions were designed to “dispel the mystery of assessment scoring” by offering those who administer the CATS a true hands-on scoring experience. 

“When teachers get to experience how scoring is done, it changes the misconceptions and addresses the rumors. Then teachers are able to say, ‘Oh, that makes sense,’” Sims explained. “Every opportunity teachers have to experience the process, they come away with a new understanding of how student work is scored.” 

Marcia Tibbetts, Measured Progress program director for the Kentucky CATS, opened the scoring session with an explanation of Measured Progress’s unique iScore technology, which allows readers to score individual student responses electronically. Tibbetts explained that the process differs radically from the way student assessments were scored 20 years ago, when responses were hand-sorted and hand-scored.

Real-time Quality Assurance

For CATS and all Measured Progress programs that include constructed-response items and writing prompts, iScore captures electronic pictures of pages from student response booklets and routes one image at a time to one or more readers working at computer stations on the scoring floor. In addition to increasing reader productivity, iScore automates and measures in real time many of the supervisory and quality assurance tasks associated with scoring.  

In turn, iScore automatically produces statistics to enable scoring leaders to assess individual and collective scoring accuracy, reliability, and consistency. This information includes a variety of reports, such as agreement rates and correlations, which track the performance of readers. Data may be collected from statistics on pre-determined read-behinds; embedded, pre-scored responses; calibration sets; and/or double-blind scoring rates. 

David Price, Measured Progress’s Louisville scoring site manager, said the sessions demonstrated to Kentucky teachers the scoring process, how iScore upholds accuracy throughout the process, and how the system allows scoring leaders to quickly and easily assess each reader’s accuracy on each item and address any scoring problems along the way.   

Marc Snow, a Measured Progress chief reader, gave each teacher/reader a reading anchor set, which included the item’s passage, the constructed-response item to which the students had responded, the scoring rubric associated with that item, and multiple-choice questions associated with the passage. Two sample student responses for each possible student score (“4,” “3,”  “2,” “1,” and “0”) were also included in the training pack to provide concrete examples of student responses at each score point. 

After a quick review of the passage, item, and sample responses, teachers were then asked to apply their knowledge of the passage and scoring rubric to score a qualification set of ten pre-loaded student responses. This is similar to how candidates qualify to score CATS responses. Teachers then read each student response and used the iScore keypad to key in what they thought was the appropriate score for each. As they would in a live qualification round, scoring leaders monitored their tables of “readers” and were readily able to determine the progress of each individual reader and the table as a whole. 

As teachers qualified and began to score student responses, Tibbetts reminded them of some general rules for scoring in a content area. For example, more complex vocabulary would not automatically boost a “2” to a “3,” nor would mistakes in spelling and mechanics pull a “4” down to a “3,” if all of the concepts and supporting examples from the passage were included in that response. 

Affirming Processes 

LaTonya Meekins, a highly skilled educator who works with students not meeting proficiency in Franklin County, Kentucky, said it is essential for her to understand the CATS scoring process to better help prepare the elementary and middle school teachers for student assessments. 

Meekins has nine years of experience as a teacher and has been a curriculum coach for the past three years for grades six, seven, and eight. The different checks inherent in each layer of Measured Progress’s scoring technology, coupled with the various levels of scoring leadership—including readers, quality assurance coordinators, and senior readers—assured Meekins that controls are in place to ensure that CATS scoring is as accurate as possible. 

Jana Hammock, director of elementary instruction for the Simpson County, Kentucky Schools, said she has worked as a writing cluster leader in her region for several years, focused primarily on training teachers to develop and score their own items and rubrics to measure student work. The Measured Progress mini-scoring session affirmed her county’s training process and the process employed to score CATS. 

“I am glad to see this is the way it’s done,” Hammock said. “There have been a lot of misconceptions thrown around. iScore is very secure.There is more quality control using this system.”

Kentucky educators score a released item using Measured Progress’s iScore technology.