Once the review phase is over in CrowdGrader, students can leave feedback on the reviews they receive, ranging from -2 (incorrect, bogus) to +2 (very helpful), with 0 being the neutral feedback (the default when comments on the reviews are left). As the chart above shows, helpful reviews outnumber unhelpful ones by over a 10:1 factor.
The chart is based on all review feedback collected so far on CrowdGrader. Overall, students have given feedback for approximately 40% of the reviews they received. There are multiple reasons why students do not leave feedback for all the reviews:
- Initially, we used a sub-optimal UI that was not very effective in eliciting feedback.
- Some instructors do not emphasize the feedback phase.
- Many students, who may feel happy with their reviews, felt no special need to give feedback. Again, we have recently improved the UI and the incentive scheme to increase the amount of feedback.
From our limited manual inspection of feedback, it seems that much of the negative feedback is left when students are disappointed at the low grades received, more than as a consequence of poor reviews -- in other words, students who do not like the message often "shoot" the messenger with negative feedback. Fortunately the algorithms in CrowdGrader prevent the "messengers" from suffering in terms of the grade they receive (and they also protect students from getting low grades due to unfair reviews).
Overall, we are very satisfied with these results, by a factor of 10!
No comments:
Post a Comment