Helpful reviews outnumber unhelpful ones by over 10:1!

In CrowdGrader, helpful reviews outnumber unhelpful ones by over 10:1, as shown in the table below.

Once the review phase is over in CrowdGrader, students can leave feedback on the reviews they receive, ranging from -2 (incorrect, bogus) to +2 (very helpful), with 0 being the neutral feedback (the default when comments on the reviews are left).  As the chart above shows, helpful reviews outnumber unhelpful ones by over a 10:1 factor.

The chart is based on all review feedback collected so far on CrowdGrader.   Overall, students have given feedback for approximately 40% of the reviews they received.  There are multiple reasons why students do not leave feedback for all the reviews:

• Initially, we used a sub-optimal UI that was not very effective in eliciting feedback.
• Some instructors do not emphasize the feedback phase. 
• Many students, who may feel happy with their reviews, felt no special need to give feedback.  Again, we have recently improved the UI and the incentive scheme to increase the amount of feedback. 

From our limited manual inspection of feedback, it seems that much of the negative feedback is left when students are disappointed at the low grades received, more than as a consequence of poor reviews -- in other words, students who do not like the message often "shoot" the messenger with negative feedback.  Fortunately the algorithms in CrowdGrader prevent the "messengers" from suffering in terms of the grade they receive (and they also protect students from getting low grades due to unfair reviews).

Overall, we are very satisfied with these results, by a factor of 10!

Students can now compare their reviews with those of others

Once the review phase of an assignment is concluded, students can now compare the reviews they wrote with the reviews other students wrote for the same assignments.

By comparing their reviews with those by others, students can discover any reviewing errors they might have made, or any aspects of the submissions they might have overlooked.  Students also learn by seeing which advice and feedback other students gave in the same circumstances.

Here are two screenshots (of a fake assignment we use for testing), showing how this feature looks like.

Click to enlarge

Click to enlarge

We are considering introducing an option that lets students view the reviews written by others on the same submissions also during the review process, so that students can compare their observations with those of others, and come up with more balanced reviews that reflect a broader consensus.  This would make the review phase a collaborative phase with many discussions (not unlike the workings of a conference program committee).  This would be an option, so that instructors would turn it on or off depending on their wishes.

Students would see reviews written by others only once they turn in a review themselves, and we plan to store the initial review and grade.  This should act as a disincentive for turning in initially a blank review, and using the comments written by others to come up with the final review.  Other disincentives towards such lazy behavior could be built into our statistical analysis tool that we use for assigning grades.

We would love to hear your thoughts on this.

CrowdGrader via HTTPS: Secure Your Connection

https://www.crowdgrader.org

You can now access CrowdGrader over HTTPS!

We encourage everybody to access CrowdGrader over HTTPS, rather than HTTP, to protect your data and your identity.  The benefits of HTTPS are:

• Your data is protected.  Homework solutions and reviews are transmitted encrypted to CrowdGrader, secure from prying eyes.
• Nobody can impersonate you and steal your access credentials. When you access CrowdGrader, you authenticate to the website via a cookie.  If you connect over an insecure network (such as an open WiFi access point, not protected by password), other people over the same network can steal your cookie and impersonate you.  By connecting to CrowdGrader over HTTP, your communication is encrypted, and nobody can steal your cookie.

For the time being, we are still allowing unencrypted connections over HTTP, but we are considering making the use of HTTPS compulsory.