Lord's Binomial Method

Lords binomial Method is a classical test theory (CTT) based method for estimating conditional standard errors of observed test scores.

Why the Standard Errors look the way they do with Lord's binomial method

When error bars are estimated using Lord's binomial method, scores closer to the ends of the scale have smaller error estimates than the scores towards the center. This is due to the fact that scores that are very low or very high tell us something very specific about the examinees receiving these scores: the examinees are performing at a level above or below the level of the assessment, and assigning a scale score is relatively certain. This is especially true in a performance assessment such as a writing exam, where you cannot guess at the correct answers.

Take for example a student who gets a score very close to zero. It is clear that this student is either unable to perform the task being evaluated, or has decided to not participate in the testing program. When a score is this low there is little doubt what scale score should be assigned this examinee based on the work submitted. Similarly, a student who earns a perfect or near perfect score is not a student who did so by chance. This student is clearly showing mastery of the skills being assessed, and is assigned a very high scale score with little doubt about that student's true ability.

As the scores drift away from these extremes, it starts becoming less clear what the examinee's true ability might be, and there is a little more room for measurement error. A student could receive a middle score for various reasons, especially when the score is made up of sub-scores based on many different attributes, such as content, structure and/or conventions. Since all students taking the same form who receive the same raw score receive the same scale score, and these middle scoring students are less like each other than those scoring perfect/near-perfect, or receiving a zero/near-zero, the error bar around their score is just that much larger to account for this variance.

The differences between Lord's binomial and IRT information based methods

Lord's binomial method is a method for computing conditional standard errors. The method relies on the test length and the student scores to estimate the conditional standard error of measurement (SEM) for a given raw score. The nature of the estimates is such that the error is estimated to be greatest at the mid-point of the scale, and lowest at the end points. This is different from IRT based conditional errors, where the maximum error is usually found at the endpoints, and the minimum error found near the center.

This may appear to generate interpretations of error that are at odds with each other, but this is not entirely the case. With IRT, strong assumptions must be made about unidimensionality and local independence. These assumptions are not necessary within a CTT framework, and a lack of unidimensionality and local independence is exactly why classical methods are used for this type of assessment. Within IRT we get a very specific type of information about the examinees' ability estimates that relies on these strong assumptions. Without these assumptions, we are forced to model the error in a different manner, and as a result the nature of the SEM changes.