CrowdGrader

Saturday, March 21, 2015

Getting reviews, not stress

Students can learn much from feedback from their peers -- but sometimes, the grades that come with such feedback can be a cause for undue stress. Often, what is desired is peer feedback -- without the judgement component.

For instance, a typical way in which CrowdGrader is used is to provide students feedback on their work -- be it an essay, an Android application, or a lab report. The students can then use the feedback to improve their work, and resubmit it. When CrowdGrader is used in this way, the grades that are associated with reviews can be a spurious source of stress: students are often better off focusing on improving their work than on the grades. Nevertheless, having some grades can be useful to the teacher to get an idea of class progress.

Teachers can now check the "Hide review grades" box in assignments:

This has the effect of showing students only the final instructor-assigned grade. Teachers and their assistants have access to the full information. Presto, feedback without judgement!

Towards supporting submit-feedback-resubmit

The above change is one in a series of planned changes that aim at supporting a submit-feedback-resubmit workflow. In this workflow, students will be able to submit a draft of their work, and receive feedback (but no grades) on their draft. Students would then be able to improve their work in light of the peer feedback, and re-submit it for evaluation.

Implementing the option to hide review grades is a first step. We are soon going to allow to clone an assignment together with the reviewer assignment. In this way, instructors can create two assignments for a given homework. In the first one, students get feedback (but no grades) on their preliminary work. Students would then be able to improve their work in view of the received feedback, and re-submit it to the second assignment. In this second assignment, their work would be assigned to the same reviewers who wrote the initial feedback. These reviewers would be able to judge how well the feedback has been taken into account, and they would assign an overall grade to the work. This is similar to what happens in scientific conferences with an author-response phase.

We plan to provide support for submit-feedback-resubmit in less than a month.

Sunday, March 8, 2015

Giving more information and control to instructors

Today we made two changes to CrowdGrader that we hope will give instructors better control of the grades, and students more assurance that their work is properly evaluated.

Average grade, and grade difference, for every submission

First, the list of all submissions now includes both the average grade received by a submission, and the delta between the highest and lowest grades. You can sort on either column, thus having quick access to the students who got the lowest average grade, and those who got the most widely differing grades.

This is how this looks on a sample assignment, and on a real assignment (with student identities hidden for privacy):

Average grade and grade difference
for a sample assignment.

Average grade and grade difference
for a real assignment.

Instructors can now also grade submissions

The other change, which goes hand in hand with the previous one, is that instructors can now grade submissions, if they wish. The option is offered when viewing the details of a submission, and here is how it looks:

If instructors select Review and grade this submission, they are led to a page where they enter a review, and a grade, for the submission, using an interface that is the same as the one used by regular reviewers. When the assignment grades are computed, CrowdGrader gives precedence to the teacher grade over the student-assigned grades.

The result: more information, and more control

These two changes together give more information and more control to instructors. Instructors can easily see which submissions received the most widely different (and thus, more unreliable) grades, and they can easily re-grade a subset of the submissions, ensuring that they receive proper grades.

Furthermore, in case where students give widely exaggerated grades, the instructors can easily intervene and assign proper grades to the submissions.

Since instructor-assigned grades take precedence over student-assigned grades, this also creates a powerful incentive for students to be fair in their grading: students whose grade differs markedly from the instructor-assigned grade will receive a low review grade.

Saturday, February 14, 2015

Authors, reviewers, and instructors can discuss submissions

Associated with each CrowdGrader submission is a forum where submission author, reviewers, and instructors can discuss the submission. This can help clarify any problems that might arise during the review process. For example, authors that make minor mistakes that might prevent the reviewers from properly evaluating their submissions can alert the reviewers. Reviewers who are unable to understand parts of the submission can ask the author for clarifications. The goal is to have a more transparent review process, where submission authors have confidence that their submission is properly evaluated, and reviewers have all the information they need.

Recently, we have given instructors the ability to actively participate in these discussion forums. Here is an example of how this works.

A discussion between submission
author and reviewers, as seen
by the instructor.

The instructor can add a message.

The discussion, with the message
added by the instructor.

Saturday, February 7, 2015

Reviewers can annotate submissions, and upload them with their reviews

A new feature of CrowdGrader enables reviewers to upload files with their review. In this way, students can collaborate with reviewers using any file format, from pdf, to doc, to file bundles for code. For example, students can download essays or papers in pdf/doc format, annotate them in place, then upload the annotated versions of the files together with their reviews.

This feature is already in successful use in some early-adopter classes. We hope it might be useful to you!

Friday, January 30, 2015

Visual Tour

It can be difficult for a user of CrowdGrader to get a full picture of the tool. Instructors and students each see only one side of it, and due to the various phases of assignments, from submission to evaluation and grading, it is difficult to form at once an idea of how it looks like.

To help you, we have created a visual tool, which gives you both the instructor and the student perspective along the various phases of the assignment.