A critical review of the arguments against the use of rubrics

Empirical research on the use of rubrics for formative purposes has grown rapidly during the last decade (Dawson, 2017). From this research, we know that the use of rubrics can have significant and positive effects on students’ learning, academic performance, and self-regulation – provided that the design and implementation are adequate (Brookhart & Chen, 2015; Panadero & Jonsson, 2013). In parallel to this research, there is also a number of publications criticizing the use of rubrics, ranging from the general idea of rubrics (e.g. Kohn, 2006), to rubrics being ill-designed and/or poorly implemented, and therefore being more harmful than beneficial (e.g. Wilson, 2007). In addition, there is literature arguing that sharing explicit criteria with students is inadequate and almost inevitably leads to instrumental learning and “criteria compliance” among students (e.g. Sadler, 2014; Torrance, 2007), or have other negative effects on teachers, teaching, or students. The latter view has obvious repercussions for the use of rubrics, since rubrics are probably the most common way of sharing explicit assessment criteria with students. As these concerns are widespread, despite the accumulated empirical support for mainly positive effects, the aim of this review is to review the claims made by the critics, in order to identify the main areas of critique and investigate to what extent these claims are supported by empirical research. This critique, and the empirical support, is then compared to empirical findings.

Publications for the review were found by keyword searches in research databases, searches in the personal libraries of the authors, and by consulting experts on research about rubrics. Furthermore, the “snowball method”, based on citations or quotes from already included publications, was used to find further references.  After reviewing more than 300 references, a total of 27 publications were included in the final review. A database with verbatim excerpts from the original articles was created, where critical remarks about rubrics were made. These excerpts (n=93) were used as data for the analysis.

The excerpts were organized around 5 main themes, called “Standardization and narrowing the curriculum”, “Instrumentalism and ‘criteria compliance’”, “Simple implementations don’t work”, “Limitations of criteria”, and “Context dependence”.

One of the main conclusions from the review is that the empirical evidence supporting the claims of the critiques is, with only a few exceptions, neither direct nor strong. First, some critics refer to anecdotal evidence and/or own personal experiences, which have limited value as empirical evidence. More importantly, only a few of these critics clearly state the nature of the data, while others even write as if their claims were backed up by strong empirical evidence. Second, there are a number of misreadings or/and misrepresentations of previous research. Several of the studies included in the review present previous studies as having strong empirical evidence against the use of rubrics, when in fact these studies are based on either anecdotal evidence or personal experience. There were also publications where reference is made to previous studies, which do not investigate, or even mention, rubrics.

The findings thus suggest that the relationship between theoretically grounded concerns and empirical data has not been “nurtured” in the critique of rubrics. Some of the critics present a significant number of concerns about rubrics and anticipate negative consequences based on theoretical considerations. Even if some of these considerations are legitimate, the critique is often based on an overly deterministic position. From these findings we therefore propose a more pragmatically oriented approach, where it is investigated what actually happens when using rubrics, and where decisions are based on considerations of empirical data.

Another important finding from the review is that a number of critics make assumptions about rubrics with a very narrow conceptualization of rubrics in mind. For example, one prevalent assumption is that rubrics are only used for large-scale testing and/or other summative assessment situations, such as grading. Other common assumptions are that rubrics are designed to be universal and transferable across contexts (i.e. based on a representational view of criteria), that rubrics specify exactly what to do and how, that rubrics are not valid or mirror the curriculum, and that rubrics cannot be supplemented with other means for communicating expectations. Since it is clearly not fair to judge the merits of a rubric designed for formative use in the classroom, based on the assumption that rubrics can only be used for large-scale assessments, this narrow conceptualization of rubrics needs to be replaced with a broader understanding, which may encompass rubrics designed and used for different purposes, both formative and summative.