Assessment and Measurement in Higher Education with multiple criteria, scales, and raters: A theoretical approach and a practical example

Assessment and measurement (A&M) in higher education can be holistic, relying on the teacher’s mental assessment model (covert), or analytic in the sense of computing the sum or average of scores from student’s responses on test items or teacher’s quality criteria applied to student’s course work (overt). Usually, scores are numbers lying on a reduced sort of scale called n-point scale. Although researchers have come up with statistical models to enable A&M with multiple criteria, scales, or raters, the acceptance of those advanced models has remained minimal: they may be too complex for educational practice as they are primarily aimed at educational research projects with enough resources (time, money, people) in appropriate organizational settings.
Since 2003 we have been developing, testing, and applying more appropriate approaches and frameworks for A&M that take the challenging work practices of teachers in higher education into account. They enable A&M with multiple quality criteria on scales of various types, and they allow several assessors to be responsible for distinct quality aspects. The approaches and frameworks were designed with flexibility and adaptability in mind. Often, it is no longer enough to test students for their factual knowledge or general problem-solving ability. Instead, the focus may be on skills that will be assessed through observation of students’ behaviour and/or examination of the results of assignments in the form of long-term projects with goals that may be achieved in different but equally valid ways.
Our most recent A&M framework called provides scoring models for instructional types not yet covered in a satisfying way by traditional A&M, such as group work. We have successfully used to offer customizable solutions for Group-Peer-Assessment (GPA), deriving individual student scores from a group score and mutual peer ratings, given just two parameters: the spread of student scores around the group score and the impact of peer ratings on student scores. GPA comes with a built-in test called the Split-Join-Invariance (SJI) to check if the final results (marks or grades) are correctly calculated. GPA imposes no artificial restrictions on group size, type or number of quality criteria, or other context-specific aspects of GPA.