Rating performance assessments of students with disabilities: A study of reliability and bias

Inclusion of students with disabilities in district-wide and state assessments is mandated by federal regulations, and teachers sometimes play an important role in rating these students' work. In this study, trained teachers rated student proficiency in performance assessments in language arts and mathematics in third, fifth, and ninth grades. The scores assigned by teacher raters to students with and without disabilities in an initial blind rating were compared with the ratings assigned in a second occasion when raters were aware of each student's disability status. A series of generalizability studies was used to determine if there are differences in the patterns of variability across groups and whether rater bias may play a role in these differences. Although knowledge of a student's disability status did not increase or decrease the scores assigned by raters on average, the findings point to differences in the sources of variability across groups and specifically to greater inconsistency when rating papers from students with disabilities. The findings suggest that individual teachers may behave differently when scoring students with disabilities. A survey was also used to investigate rater perceptions of one's own and other teacher's bias when grading papers of students with disabilities. Implications for decision making in rating assessments are discussed.