A Comparison of Rubrics and Graded Category Rating Scales with Various Methods Regarding Raters' Reliability

Abstract

The aim of this study was to determine the extent at which graded-category rating scales and rubrics contribute to inter-rater reliability. The research was designed as a correlational study. Study group consisted of 82 students attending sixth grade and three writing course teachers in a private elementary school. A performance task was administered to students and student works were divided into two randomly. The teachers first conducted independent scoring on the works in group one by using the graded category rating scale, and then they scored the works in groups two with rubrics. Raters reliability was estimated by intraclass correlation coefficient, generalizability theory (G-theory) and many-facet Rasch model. The results indicated higher inter-rater reliability when graded category rating scale was used. Moreover qualitative data revealed that raters prefer using graded category rating scale in situations where they need to do quick scoring in short intervals. It is recommended that teachers use graded-category rating scale as a practical tool for quick scoring with the aim of determining student achievement with grades rather than giving detailed feedback.