CrowdGrader indicates which review grades were considered low-confidence. |
CrowdGrader classified the -2 feedback as possible tit-for-tat, and discarded it from the computation of the feedback grade. |
How it works
In CrowdGrader, the grade a student receives for an assignment is the sum of three components:- The submission grade, computed from the grades other students gave to the submission.
- The accuracy grade, which measures how precise the student was in giving grades.
- The feedback grade, which depends on the feedback the student received for the reviews she or he wrote.
Submission grade. When computing the submission grade, not all reviews are considered equal: CrowdGrader uses a reputation system to give more weight to the input of more credible reviewers. CrowdGrader now lets students know which reviews were considered low confidence, and given less weight in the computation of the submission grade.
Feedback grade. The feedback a review receives can range from -2 (very unhelpful or bogus) to +2 (very helpful). CrowdGrader uses a sophisticated algorithm to determine when a negative feedback might have been given as a tit-for-tat retribution for a low grade. Such negative feedbacks are not considered in the computation of the feedback grade.
Instructors should compute the grades once all feedback is in
When CrowdGrader computes the grades, it classifies each feedback as either legitimate or possible tit-for-tat. Once grades are computed, students can no longer modify their feedback: this ensures that the feedback they see is the same that was used to compute the grades. For this reason, it is important that instructors compute grades only once all the feedback is in.