Sunday, October 11, 2015

Stars for reviews

In CrowdGrader, students can give feedback on the quality of the reviews they receive.  The grades assigned by CrowdGrader take into account this feedback, creating an incentive for students to write helpful and insightful reviews.

Students can now leave feedback simply by assigning stars to the reviews, ranging from 1 star (very unhelpful, bogus) to 5 stars (very helpful).  A guide to the meaning of each star rating appears when students hover over the stars.  Here is how the new interface looks like.

A student's submission feedback.  Students can rate
the helpfulness of the reviews via a 1-to-5 star rating.

This change is part of an ongoing effort to streamline and simplify the CrowdGrader UI.  The new start-based feedback replaces an implementation based on forms and links.

Sunday, September 20, 2015

CrowdGrader can now check submission similarity

One of the problems of crowd-grading is that there is no-one who grades all submissions.  Thus, it is difficult to detect when students submit the same solution.

To help with the detection of similar submissions, we have implemented in CrowdGrader a similarity checker.  And not just any similarity checker: the most full-features similarity checker we could wish for.

The feature is accessible from an assignment page by selecting Submissions > Check Similarity.
Note that it is still in Beta - please report any problems.

Input formats

The CrowdGrader similarity checker can process:

  • Text typed directly in CrowdGrader
  • Attached Word (docx, not doc), PDF, HTML, RTF documents.
  • Attached source files in any programming language (it deals correctly with comments in C, C++, Java, Python).
  • zip, tar, tgz archives.  In the archives, you can specify which subset of files should be processed, so the similarity results are meaningful and not drowned out in a myriad of standard files. 
  • Compressed versions of above files via gzip.
CrowdGrader also accepts any nesting of the above, for instance, multiple Word files included in a single zip file are ok. 

Comparison output

CrowdGrader distinguishes between text that is: 
  • Unchanged: equal in the two submissions
  • Renamed: uniformly renamed (for instance, when a variable is renamed)
  • Different
CrowdGrader also clusters for you the similar submissions, according to a threshold of your choice.  
Perhaps the best is to look at a couple of screenshots.

Submissions are clustered according to their similarity.
You can dynamically vary the similarity threshold and explore the resulting clusters.

You can examine similar submissions side-by-side.
Identical content is highlighted in blue; content that has been renamed is highlighted in green.

Saturday, April 25, 2015

Calendar view, multiple assignment download, and more

This weekend brought several improvements to CrowdGrader.  This is a mix of what users wanted, and what we thought they would like.

Calendar View

We have introduced a calendar view for assignments.  Keeping track of the various deadlines should be much easier now!

Multiple Assignment Download

Instructors can now download the grades for multiple assignments at once, simplifying class management.

Clone Group Membership

When cloning assignments, it is now possible to clone also the group membership of students.  This should make it easier to run classes where student work in the same groups throughout the class. Students can still edit their membership, should something change from one assignment to the other.

We hope this changes will make your work as instructors easier.  Enjoy! - and give us feedback: we love implementing features that make your life easier. 

Saturday, March 21, 2015

Getting reviews, not stress

Students can learn much from feedback from their peers -- but sometimes, the grades that come with such feedback can be a cause for undue stress.  Often, what is desired is peer feedback -- without the judgement component.

For instance, a typical way in which CrowdGrader is used is to provide students feedback on their work -- be it an essay, an Android application, or a lab report.  The students can then use the feedback to improve their work, and resubmit it.  When CrowdGrader is used in this way, the grades that are associated with reviews can be a spurious source of stress: students are often better off focusing on improving their work than on the grades.  Nevertheless, having some grades can be useful to the teacher to get an idea of class progress.

Teachers can now check the "Hide review grades" box in assignments:

This has the effect of showing students only the final instructor-assigned grade.  Teachers and their assistants have access to the full information.  Presto, feedback without judgement!

Towards supporting submit-feedback-resubmit

The above change is one in a series of planned changes that aim at supporting a submit-feedback-resubmit workflow.  In this workflow, students will be able to submit a draft of their work, and receive feedback (but no grades) on their draft.  Students would then be able to improve their work in light of the peer feedback, and re-submit it for evaluation. 

Implementing the option to hide review grades is a first step.  We are soon going to allow to clone an assignment together with the reviewer assignment.  In this way, instructors can create two assignments for a given homework.  In the first one, students get feedback (but no grades) on their preliminary work.  Students would then be able to improve their work in view of the received feedback, and re-submit it to the second assignment.  In this second assignment, their work would be assigned to the same reviewers who wrote the initial feedback.  These reviewers would be able to judge how well the feedback has been taken into account, and they would assign an overall grade to the work.  This is similar to what happens in scientific conferences with an author-response phase. 

We plan to provide support for submit-feedback-resubmit in less than a month.

Sunday, March 8, 2015

Giving more information and control to instructors

Today we made two changes to CrowdGrader that we hope will give instructors better control of the grades, and students more assurance that their work is properly evaluated.

Average grade, and grade difference, for every submission

First, the list of all submissions now includes both the average grade received by a submission, and the delta between the highest and lowest grades.  You can sort on either column, thus having quick access to the students who got the lowest average grade, and those who got the most widely differing grades.

This is how this looks on a sample assignment, and on a real assignment (with student identities hidden for privacy):

Average grade and grade difference
for a sample assignment.
Average grade and grade difference
for a real assignment.

Instructors can now also grade submissions

The other change, which goes hand in hand with the previous one, is that instructors can now grade submissions, if they wish.  The option is offered when viewing the details of a submission, and here is how it looks: 

If instructors select Review and grade this submission, they are led to a page where they enter a review, and a grade, for the submission, using an interface that is the same as the one used by regular reviewers.  When the assignment grades are computed, CrowdGrader gives precedence to the teacher grade over the student-assigned grades. 

The result: more information, and more control

These two changes together give more information and more control to instructors.  Instructors can easily see which submissions received the most widely different (and thus, more unreliable) grades, and they can easily re-grade a subset of the submissions, ensuring that they receive proper grades. 

Furthermore, in case where students give widely exaggerated grades, the instructors can easily intervene and assign proper grades to the submissions. 

Since instructor-assigned grades take precedence over student-assigned grades, this also creates a powerful incentive for students to be fair in their grading: students whose grade differs markedly from the instructor-assigned grade will receive a low review grade

Saturday, February 14, 2015

Authors, reviewers, and instructors can discuss submissions

Associated with each CrowdGrader submission is a forum where submission author, reviewers, and instructors can discuss the submission.  This can help clarify any problems that might arise during the review process.  For example, authors that make minor mistakes that might prevent the reviewers from properly evaluating their submissions can alert the reviewers.  Reviewers who are unable to understand parts of the submission can ask the author for clarifications.  The goal is to have a more transparent review process, where submission authors have confidence that their submission is properly evaluated, and reviewers have all the information they need.

Recently, we have given instructors the ability to actively participate in these discussion forums.  Here is an example of how this works.

A discussion between submission
author and reviewers, as seen
by the instructor.
The instructor can add a message.
The discussion, with the message
added by the instructor.

Saturday, February 7, 2015

Reviewers can annotate submissions, and upload them with their reviews

A new feature of CrowdGrader enables reviewers to upload files with their review.  In this way, students can collaborate with reviewers using any file format, from pdf, to doc, to file bundles for code.  For example, students can download essays or papers in pdf/doc format, annotate them in place, then upload the annotated versions of the files together with their reviews.

This feature is already in successful use in some early-adopter classes.  We hope it might be useful to you!

Friday, January 30, 2015

Visual Tour

It can be difficult for a user of CrowdGrader to get a full picture of the tool.  Instructors and students each see only one side of it, and due to the various phases of assignments, from submission to evaluation and grading, it is difficult to form at once an idea of how it looks like.

To help you, we have created a visual tool, which gives you both the instructor and the student perspective along the various phases of the assignment. 

Visual Tour: A student's evaluation

Once the assignment is concluded, a student can see the final grade received, as well as all reviews received.

The full evaluation received
by a student.

A student can also view all the reviews she or he wrote, and compare them with those by others:

All the reviews written by a student.
A student can compare her/his
reviews with those by others.

Visual Tour: Computing the grades

When the grades are computed,
feedback on the reviews is
no longer accepted.
The crowd-grades have been computed.
The instructor can decide when grades
become visible by students.
The instructor can assign some final
grades by hand...

...and the remaining ones
are computed by interpolation.

Visual Tour: The feedback phase

Once reviewing is complete, students can see their reviews, and give feedback on the reviewers:

At the end of the review
phase, students can leave
feedback on their reviews.
Leaving feedback on a review.

Visual Tour: Reviewing, as seen by the instructor

Instructor's view of the review progress:

The instructor has a bird's eye view
 on the submissions,
the grades assigned, and any discussion.
A discussion as seen by the teacher.

The instructor has easy access to all the reviews, making it easy to form an idea of how the assignment went:

All the reviews.
All the reviews by a given student.
All the reviews
for a given submission.

Visual Tour: Reviewing, as seen by a student

The student has not done any
reviews yet.
A student in the process of entering a review.
Reviews are checked for
consistency with the grading rubric.
The first two reviews are done.

Students can also discuss the submissions while doing the reviews:

The initial message from
the submission author.
The reviewer adds a message.
The message is added
to the discussion.

Visual Tour: The submissions, as seen by the instructor

The submission list,
before any review.
A submission, as seen by the instructor
The discussion on the submission

Visual Tour: Submitting a solution to an assignment

Here is how submitting a solution to an assignment looks like, from the students point of view.

An image has just been uploaded.
The text of a submission can be
written using a powerful markup language.

A completed submission.

Students can also discuss their submissions with the people who will review them; in this case notifying them of the fact that the image has been resized.

An empty discussion list.
Writing the first message.
The first message.

Visual Tour: Creating an assignment

Here is how creating an assignment, entering a student list, and creating a grading rubric looks like.

A brand new assignment. 

You can either let students self-enroll,
or you can cut-and-paste their emails.
A student list with 5 students.
You can edit grading rubrics.
A grading rubric.