Students can now see what contributes to their grade

CrowdGrader now lets students see which information is used in computing their grade, and which information is disregarded as unreliable.  Here are two examples, taken from an actual assignment:

CrowdGrader indicates
which review grades were
considered low-confidence.

CrowdGrader classified the -2 feedback as possible
tit-for-tat, and discarded it from the computation of
the feedback grade.    

How it works

In CrowdGrader, the grade a student receives for an assignment is the sum of three components:

1. The submission grade, computed from the grades other students gave to the submission.
2. The accuracy grade, which measures how precise the student was in giving grades.
3. The feedback grade, which depends on the feedback the student received for the reviews she or he wrote. 

Submission grade.  When computing the submission grade, not all reviews are considered equal: CrowdGrader uses a reputation system to give more weight to the input of more credible reviewers. CrowdGrader now lets students know which reviews were considered low confidence, and given less weight in the computation of the submission grade.

Feedback grade.  The feedback a review receives can range from -2 (very unhelpful or bogus) to +2 (very helpful).  CrowdGrader uses a sophisticated algorithm to determine when a negative feedback might have been given as a tit-for-tat retribution for a low grade.  Such negative feedbacks are not considered in the computation of the feedback grade. 

Instructors should compute the grades once all feedback is in

When CrowdGrader computes the grades, it classifies each feedback as either legitimate or possible tit-for-tat. Once grades are computed, students can no longer modify their feedback: this ensures that the feedback they see is the same that was used to compute the grades.  For this reason, it is important that instructors compute grades only once all the feedback is in. 

Simpler assignment creation and class list management

We just rolled out a few more features!

Simpler assignment creation

Only one type of assignment

There is no longer a distinction between standard assignments (where the instructor specifies a class list) and open assignments (where students can self-enroll by clicking on a join link provided by the instructor).

Rather, the instructor can always edit the class list, adding and removing students, and also has the option of handing out the join link if desired.

You can now clone assignments

We have implemented one of the top user requests: you can now clone assignments.  So, once you create the first assignment for your class, you can create the subsequent assignments by cloning the first assignment, changing just the title and the submission and review dates.

When you clone an assignment, you create an assignment that is identical to the cloned one, except for:

• The name ("Copy of " is added to the name)
• The dates (you have to provide new dates).

The class list is copied over. 

Note that, as explained below, each assignment has its own class list.  So, if you clone an assignment, then add a student to either the cloned or the original assignment, the student is not automatically added to the other assignment.  For this reason, if you expect the students enrolled in the class to change, you might want to wait and clone assignments only just before the clones are needed, so their class list will contain all changes since the start of the class.

Simpler class list management

We have simplified the management of class lists.  Each assignment has its own class list, not shared with any other assignment.  You can edit the list as one of the options in the assignment: 

Selecting Class List will take you to a page where you can interactively edit the class list, as well as downloading it.

If you need to manually copy the class list from one assignment to another, you can first download it, then cut and paste it into the class list of the other assignment:

If you need to do this frequently, let us know, and we may provide some better import options.

As always, if you had any feedback, we would love to hear from you.  You can contact us at info@crowdgrader.org

CrowdGrader now lets reviewers and authors discuss submissions

We have just added a new feature to CrowdGrader: authors and reviewers can discuss the submissions anonymously.  Attached to each submission is a discussion forum, where author and reviewers alike can post messages and discuss, in anonymous fashion.  The discussion forum is visible only to the submission author and reviewers, and to the instructor.

In our experience as teachers, perhaps the most common cause of errors in grading is misunderstandings between the student who submitted a solution, and the students who grade it.  Many of these misunderstandings could be clarified quite easily if reviewers and authors could communicate. Some typical examples: 

• I cannot find the file solution.txt in the .zip you uploaded, how did you name it?
• You have two versions of the solutions; which one am I supposed to grade? 
• Is your Android app designed for tablets or phones? 
• How do I install your app?
• I was not able to compile your program.  What command line did you use, precisely?

We hope that, by letting authors and reviewers communicate anonymously, we will help avoid misunderstandings, and we will give authors and reviewers a better educational experience.

Whenever a new message is added, the students involved in the review are notified via email; reminders also appear on the CrowdGrader site.  To prevent conversations from dragging on too long, the forums are active only during the review period: after the review period closes, the messages can be read, but no new messages can be posted. 

For the moment, this feature is available only if you select it.  To do so, edit an assignment, and check the option:

How does it look?

Suppose a reviewer has a question: 

The author can view the question, and reply:

The reviewer then thanks the author.  When another reviewer looks at the submission, the new reviewer is notified of the ongoing discussion:

And can contribute in turn:

This is how the same discussion appears to the author of the submission:

We hope that you find this useful.  In the future, we might make submission discussions enabled by default. 

Helpful reviews outnumber unhelpful ones by over 10:1!

In CrowdGrader, helpful reviews outnumber unhelpful ones by over 10:1, as shown in the table below.

Once the review phase is over in CrowdGrader, students can leave feedback on the reviews they receive, ranging from -2 (incorrect, bogus) to +2 (very helpful), with 0 being the neutral feedback (the default when comments on the reviews are left).  As the chart above shows, helpful reviews outnumber unhelpful ones by over a 10:1 factor.

The chart is based on all review feedback collected so far on CrowdGrader.   Overall, students have given feedback for approximately 40% of the reviews they received.  There are multiple reasons why students do not leave feedback for all the reviews:

• Initially, we used a sub-optimal UI that was not very effective in eliciting feedback.
• Some instructors do not emphasize the feedback phase. 
• Many students, who may feel happy with their reviews, felt no special need to give feedback.  Again, we have recently improved the UI and the incentive scheme to increase the amount of feedback. 

From our limited manual inspection of feedback, it seems that much of the negative feedback is left when students are disappointed at the low grades received, more than as a consequence of poor reviews -- in other words, students who do not like the message often "shoot" the messenger with negative feedback.  Fortunately the algorithms in CrowdGrader prevent the "messengers" from suffering in terms of the grade they receive (and they also protect students from getting low grades due to unfair reviews).

Overall, we are very satisfied with these results, by a factor of 10!

Students can now compare their reviews with those of others

Once the review phase of an assignment is concluded, students can now compare the reviews they wrote with the reviews other students wrote for the same assignments.

By comparing their reviews with those by others, students can discover any reviewing errors they might have made, or any aspects of the submissions they might have overlooked.  Students also learn by seeing which advice and feedback other students gave in the same circumstances.

Here are two screenshots (of a fake assignment we use for testing), showing how this feature looks like.

Click to enlarge

Click to enlarge

We are considering introducing an option that lets students view the reviews written by others on the same submissions also during the review process, so that students can compare their observations with those of others, and come up with more balanced reviews that reflect a broader consensus.  This would make the review phase a collaborative phase with many discussions (not unlike the workings of a conference program committee).  This would be an option, so that instructors would turn it on or off depending on their wishes.

Students would see reviews written by others only once they turn in a review themselves, and we plan to store the initial review and grade.  This should act as a disincentive for turning in initially a blank review, and using the comments written by others to come up with the final review.  Other disincentives towards such lazy behavior could be built into our statistical analysis tool that we use for assigning grades.

We would love to hear your thoughts on this.

CrowdGrader via HTTPS: Secure Your Connection

https://www.crowdgrader.org

You can now access CrowdGrader over HTTPS!

We encourage everybody to access CrowdGrader over HTTPS, rather than HTTP, to protect your data and your identity.  The benefits of HTTPS are:

• Your data is protected.  Homework solutions and reviews are transmitted encrypted to CrowdGrader, secure from prying eyes.
• Nobody can impersonate you and steal your access credentials. When you access CrowdGrader, you authenticate to the website via a cookie.  If you connect over an insecure network (such as an open WiFi access point, not protected by password), other people over the same network can steal your cookie and impersonate you.  By connecting to CrowdGrader over HTTP, your communication is encrypted, and nobody can steal your cookie.

For the time being, we are still allowing unencrypted connections over HTTP, but we are considering making the use of HTTPS compulsory.

Download all assignment data!

Assignment managers (instructors and TAs) can now download all assignment data!

There are two steps required.

First, you need to download a Json file that contains information on the assignment.  You can do so by going to the assignment, and selecting Download Assignment Data. 

Second, you need to get the Python CrowdGrader assignment downloader from GitHub.

You can then run the downloader via:

./download_assignment.py <assignment_file.json> <destination_directory>

This will produce a directory in <destination_directory>, and it will create one sub-directory for each submission.  In the sub-directory for each submission, you will find the submission metadata, the submission text, the files that were submitted as attachment, and the reviews.  The format should be self-explanatory.

The reason you have to go through this two-step process is that an assignment can consist of a very large amount of data (up to several GB for some assignments, depending on the files that were submitted as attachments).  Therefore, downloading an entire assignment can take a long time, and cannot be performed as a single download.

The downloader code is supplied with a MIT license, so please feel free to improve it and to share back your improvements.

Powerful new editor supports math, code, and more

We have integrated in CrowdGrader a powerful new editor, which supports math, code, and more.  In this way, students can write complex literature, math, computer science, and other assignments directly into the editor used by CrowdGrader, without requiring the use of external programs.  As a plus, writing math in CrowdGrader is much easier than in tools such as Word or Google Documents, since CrowdGrader now supports LaTeX/TeX syntax directly! Here is how the editor looks like:

The above text is rendered as follows: