tag:blogger.com,1999:blog-45963982502581434202024-02-19T05:28:45.450-08:00CrowdGraderCrowdGrader lets students submit and collaboratively grade solutions to homework assignments. Students learn from the work and feedback of their peers.Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.comBlogger26125tag:blogger.com,1999:blog-4596398250258143420.post-60950700911775127532017-02-01T12:00:00.000-08:002017-02-01T16:10:42.304-08:00CrowdGrader Pricing<br />
CrowdGrader runs on the Google Cloud, which guarantees top-notch security, high availability, full scalability, and geographically distributed replicas of the data to ensure that the student work is not lost. In order to help pay for this infrastructure, we started charging for the service. The intention is to provide an affordable service, while covering the costs. <br />
<br />
Each assignment can be either <i>instructor-sponsored</i> or <i>student-supported</i>.<br />
<div>
<h2>
Instructor-Sponsored Assignments</h2>
Instructor-sponsored assignments come in three tiers:<br />
<ul>
<li>Free: up to 10 students</li>
<li>Small: $2.99, up to 35 students</li>
<li>Medium: $6.99, up to 100 students</li>
<li>Large: $13.98, unlimited students</li>
</ul>
The prices above are per-assignment.<br />
<h2>
Student-Supported Assignments.</h2>
<ul>
<li><b>Three-weeks free trial.</b> A three-week trial period gives students time to sort out any payment issues at the beginning of a class. </li>
<li>$<b>1.99/six months</b>: Students pay a non-refundable $1.99 subscription fee for six months of access to CrowdGrader. </li>
</ul>
Subscription can be renewed, but they do not auto-renew. The access period determines when students can submit solutions to CrowdGrader: students can perform reviews, or view their feedback, at any time.<br />
<br />
In a month or so, we will take a look at the costs vs. income, and we will assess whether we can lower prices. We hope you consider our service good value for the money, and we thank our users for their support.<br />
<h3>
Payment Terms and Conditions</h3>
Please refer to the <a href="https://sites.google.com/site/crowdgraderorg/pricing/payment-terms-and-conditions">Payment Terms and Conditions</a> governing payments and refunds.</div>
Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-64673658787363778562016-11-16T10:47:00.000-08:002017-02-01T16:32:39.724-08:00Consensus-based grading: more accuracy and less stressWe just introduced a new method for grading, called <b><span style="color: #cc0000;">Consensus-based grading</span></b>. In consensus-based grading, the grades students assign are compared on each rubric item separately, to obtain a consensus for each rubric item independently. Subsequently, the consensus grades for each entry are combined in an overall grade for the submission. <br />
<div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
Rubric-based consensus increases precision, as errors are eliminated on each rubric entry, before the totals for the entries are combined. </div>
<div>
<ul>
</ul>
<div>
You can <a href="http://doc.crowdgrader.org/crowdgrader-documentation/crowd-grades" target="_blank">read here the full details of how consensus-based grading works</a>, along with examples, and you can <a href="http://doc.crowdgrader.org/crowdgrader-documentation/rubric-based-consensus" target="_blank">read here a comparison with the older approach</a>. </div>
<h2>
A more streamlined, less stressful student experience</h2>
<div>
Corresponding to this change, we made a few other changes to CrowdGrader, based on our experience of running large classes: <br />
<br />
<ul>
<li><b>Students see only consensus grades.</b> If you choose rubric-based consensus, students will see only the conseusus grades for their submission, rather than the grades individually submitted by students. In our experience, this increases student satisfaction, as students no longer see anxiety-inducing, but harmless, outlier grades in their evaluation.</li>
</ul>
<ul>
<li><b>The review feedback phase is no longer present.</b> We found that the review feedback phase was only mildly useful, and often worked as a dis-incentive for students to submit honest evaluations, in spite of the fact that CrowdGrader does eliminate tit-for-tat behavior in feedback.</li>
</ul>
</div>
</div>
Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-50203463081120108952015-10-11T17:35:00.000-07:002015-10-11T17:35:00.103-07:00Stars for reviewsIn CrowdGrader, students can give feedback on the quality of the reviews they receive. <a href="http://doc.crowdgrader.org/crowdgrader-documentation/crowd-grades" target="_blank">The grades assigned by CrowdGrader take into account this feedback</a>, creating an incentive for students to write helpful and insightful reviews.<br />
<br />
Students can now leave feedback simply by assigning stars to the reviews, ranging from 1 star (very unhelpful, bogus) to 5 stars (very helpful). A guide to the meaning of each star rating appears when students hover over the stars. Here is how the new interface looks like.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3FWfl_WxfqJLzJ9TYFghdxqqZwqWqlEM9QSGBMRdnIyG65Q8IOtggTb-_CbsET8cBuKgoP7i7r2FP4NHBPG8kVzQgPFrqyyygyF5PI9yzEJRgj4ChklODaz8oW78uCL2wWYxPuHm1W1s/s1600/Screen+Shot+2015-09-21+at+4.48.22+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3FWfl_WxfqJLzJ9TYFghdxqqZwqWqlEM9QSGBMRdnIyG65Q8IOtggTb-_CbsET8cBuKgoP7i7r2FP4NHBPG8kVzQgPFrqyyygyF5PI9yzEJRgj4ChklODaz8oW78uCL2wWYxPuHm1W1s/s320/Screen+Shot+2015-09-21+at+4.48.22+PM.png" width="290" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A student's submission feedback. Students can rate<br />
the helpfulness of the reviews via a 1-to-5 star rating.</td></tr>
</tbody></table>
<br />
<br />
This change is part of an ongoing effort to streamline and simplify the CrowdGrader UI. The new start-based feedback replaces an implementation based on forms and links.<br />
<br />
<br />
<br />
<br />Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-15224789499236153732015-09-20T13:27:00.000-07:002015-09-22T11:27:58.740-07:00CrowdGrader can now check submission similarityOne of the problems of crowd-grading is that there is no-one who grades all submissions. Thus, it is difficult to detect when students submit the same solution.<br />
<br />
To help with the detection of similar submissions, we have implemented in CrowdGrader a similarity checker. And not just any similarity checker: the most full-features similarity checker we could wish for.<br />
<br />
The feature is accessible from an assignment page by selecting Submissions > Check Similarity.<br />
Note that it is still in Beta - please report any problems.<br />
<h2>
Input formats</h2>
The CrowdGrader similarity checker can process:<br />
<br />
<ul>
<li>Text typed directly in CrowdGrader</li>
<li>Attached Word (docx, not doc), PDF, HTML, RTF documents.</li>
<li>Attached source files in any programming language (it deals correctly with comments in C, C++, Java, Python).</li>
<li>zip, tar, tgz archives. In the archives, you can specify which subset of files should be processed, so the similarity results are meaningful and not drowned out in a myriad of standard files. </li>
<li>Compressed versions of above files via gzip.</li>
</ul>
<div>
CrowdGrader also accepts any nesting of the above, for instance, multiple Word files included in a single zip file are ok. </div>
<h2>
Comparison output</h2>
<div>
CrowdGrader distinguishes between text that is: </div>
<div>
<ul>
<li><b>Unchanged: </b>equal in the two submissions</li>
<li><b>Renamed: </b>uniformly renamed (for instance, when a variable is renamed)</li>
<li><b>Different</b></li>
</ul>
<div>
CrowdGrader also clusters for you the similar submissions, according to a threshold of your choice. </div>
</div>
<div>
Perhaps the best is to look at a couple of screenshots.</div>
<div>
<br /></div>
<div style="text-align: left;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<br />
<div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieyYYh8Ab5xmykjJ_Epvwv6G0mZELTTri4TOomX-m2ns7K3KZounRMeV0olGWc_5LGjqHRogWeZtpWSLp2qNN1w4sO6xiPsVQdD-EZni14UbxE6gWCalvarjKbh85x_toOltfrTSi_dFk/s1600/Screen+Shot+2015-09-22+at+11.17.06+AM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="267" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieyYYh8Ab5xmykjJ_Epvwv6G0mZELTTri4TOomX-m2ns7K3KZounRMeV0olGWc_5LGjqHRogWeZtpWSLp2qNN1w4sO6xiPsVQdD-EZni14UbxE6gWCalvarjKbh85x_toOltfrTSi_dFk/s400/Screen+Shot+2015-09-22+at+11.17.06+AM.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Submissions are clustered according to their similarity. <br />
You can dynamically vary the similarity threshold and explore the resulting clusters.</td></tr>
</tbody></table>
<br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdq-lqYtFCZL2cf4jwAk7ObsuLMch9J8tsQA7AahlKtmfBSYb9spJQNhAzxQJvBpa8LNzwFJLS36k-C0zBFYoBhJ5LO8rW-AE2KvXzbeFeyRaH7Nz7tc1-2P6Z1OPkhwNMWetAVTy6W8U/s1600/Screen+Shot+2015-09-22+at+11.17.53+AM.png" imageanchor="1" style="display: inline !important; margin-left: auto; margin-right: auto; text-align: center;"><img border="0" height="315" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdq-lqYtFCZL2cf4jwAk7ObsuLMch9J8tsQA7AahlKtmfBSYb9spJQNhAzxQJvBpa8LNzwFJLS36k-C0zBFYoBhJ5LO8rW-AE2KvXzbeFeyRaH7Nz7tc1-2P6Z1OPkhwNMWetAVTy6W8U/s400/Screen+Shot+2015-09-22+at+11.17.53+AM.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">You can examine similar submissions side-by-side.<br />
Identical content is highlighted in blue; content that has been renamed is highlighted in green.</td></tr>
</tbody></table>
<br />
<br /></div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-21389118240946650522015-04-25T23:00:00.003-07:002015-04-25T23:41:20.505-07:00Calendar view, multiple assignment download, and moreThis weekend brought several improvements to CrowdGrader. This is a mix of what users wanted, and what we thought they would like.<br />
<br />
<h2>
Calendar View</h2>
We have introduced a calendar view for assignments. Keeping track of the various deadlines should be much easier now!<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9grDB8JxT9-I8P-AX2KKAyzBuRzk-gryDjWW2q5h52Y59injUq7Py5XTOczxTZBvcWsPrU4314c145ssoWuWzFGQZ-l_msaqj3qSsfRhGm9PX3hEtG0JKhiYU5Jaxwcu0FbFiZkUo0Bs/s1600/Screen+Shot+2015-04-25+at+10.52.50+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9grDB8JxT9-I8P-AX2KKAyzBuRzk-gryDjWW2q5h52Y59injUq7Py5XTOczxTZBvcWsPrU4314c145ssoWuWzFGQZ-l_msaqj3qSsfRhGm9PX3hEtG0JKhiYU5Jaxwcu0FbFiZkUo0Bs/s1600/Screen+Shot+2015-04-25+at+10.52.50+PM.png" height="130" width="400" /></a></div>
<br />
<h2>
Multiple Assignment Download</h2>
Instructors can now download the grades for multiple assignments at once, simplifying class management.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQBetWwl3p7gGQV6m0oRSf2QsdlF787lJ6vIT9dNLGqQrr6U2QyN3n0vOnVyMHp8LYeKh0RYQgekHh5YizPqUbwteVShWfuXu_cidq8qEtGzl9vag-SRmYNW-c30sgIx5LhRu-WQWJCEM/s1600/Screen+Shot+2015-04-25+at+10.55.15+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQBetWwl3p7gGQV6m0oRSf2QsdlF787lJ6vIT9dNLGqQrr6U2QyN3n0vOnVyMHp8LYeKh0RYQgekHh5YizPqUbwteVShWfuXu_cidq8qEtGzl9vag-SRmYNW-c30sgIx5LhRu-WQWJCEM/s1600/Screen+Shot+2015-04-25+at+10.55.15+PM.png" height="197" width="400" /></a> </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<h2>
Clone Group Membership</h2>
When cloning assignments, it is now possible to clone also the group membership of students. This should make it easier to run classes where student work in the same groups throughout the class. Students can still edit their membership, should something change from one assignment to the other.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjR_SLHX2VDn-Aee2WNkW-mtlas2-ejYXHIExwR9o2KkICIocj16-KWZ4aH62Bv-X55y6OuVvMolCvtpNhyphenhyphenlBlPmG7Bk6ihqBHB073_TqMSD-nz2-ckcCEJ2Vz9dU-H9h2VG_Tt2HZuxos/s1600/Screen+Shot+2015-04-25+at+11.40.31+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjR_SLHX2VDn-Aee2WNkW-mtlas2-ejYXHIExwR9o2KkICIocj16-KWZ4aH62Bv-X55y6OuVvMolCvtpNhyphenhyphenlBlPmG7Bk6ihqBHB073_TqMSD-nz2-ckcCEJ2Vz9dU-H9h2VG_Tt2HZuxos/s1600/Screen+Shot+2015-04-25+at+11.40.31+PM.png" height="128" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
We hope this changes will make your work as instructors easier. Enjoy! - and give us feedback: we love implementing features that make your life easier. </div>
<br />Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-18754796201724761752015-03-21T11:25:00.000-07:002015-03-21T11:25:00.031-07:00Getting reviews, not stressStudents can learn much from feedback from their peers -- but sometimes, the grades that come with such feedback can be a cause for undue stress. Often, what is desired is peer feedback -- without the judgement component.<br />
<br />
For instance, a typical way in which CrowdGrader is used is to provide students feedback on their work -- be it an essay, an Android application, or a lab report. The students can then use the feedback to improve their work, and resubmit it. When CrowdGrader is used in this way, the grades that are associated with reviews can be a spurious source of stress: students are often better off focusing on improving their work than on the grades. Nevertheless, having some grades can be useful to the teacher to get an idea of class progress.<br />
<br />
Teachers can now check the "Hide review grades" box in assignments:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgS-fwPCZMk9rTVEzMPZA2URK3tY9iShoXgTe64Nn1GIQyRGcWQGjvUGIvrOi-q3KnoazQ34Oebvny1NRwwLVRhdL-VSqbv9s-OtqE-7mfEaUcgs2zS2uwO2jDcMvc3yfPDv10dROV6nJQ/s1600/Screen+Shot+2015-03-19+at+11.27.00+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgS-fwPCZMk9rTVEzMPZA2URK3tY9iShoXgTe64Nn1GIQyRGcWQGjvUGIvrOi-q3KnoazQ34Oebvny1NRwwLVRhdL-VSqbv9s-OtqE-7mfEaUcgs2zS2uwO2jDcMvc3yfPDv10dROV6nJQ/s1600/Screen+Shot+2015-03-19+at+11.27.00+AM.png" height="65" width="320" /></a></div>
<br />
This has the effect of showing students only the final instructor-assigned grade. Teachers and their assistants have access to the full information. Presto, feedback without judgement!<br />
<br />
<h2>
Towards supporting submit-feedback-resubmit</h2>
<div>
The above change is one in a series of planned changes that aim at supporting a submit-feedback-resubmit workflow. In this workflow, students will be able to submit a draft of their work, and receive feedback (but no grades) on their draft. Students would then be able to improve their work in light of the peer feedback, and re-submit it for evaluation. </div>
<div>
<br /></div>
<div>
Implementing the option to hide review grades is a first step. We are soon going to allow to clone an assignment <i>together with the reviewer assignment. </i>In this way, instructors can create two assignments for a given homework. In the first one, students get feedback (but no grades) on their preliminary work. Students would then be able to improve their work in view of the received feedback, and re-submit it to the second assignment. In this second assignment, their work would be assigned to the same reviewers who wrote the initial feedback. These reviewers would be able to judge how well the feedback has been taken into account, and they would assign an overall grade to the work. This is similar to what happens in scientific conferences with an author-response phase. </div>
<div>
<br /></div>
<div>
We plan to provide support for submit-feedback-resubmit in less than a month.</div>
<div>
<br /></div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-8357624962202816632015-03-08T17:09:00.001-07:002015-03-08T17:10:22.038-07:00Giving more information and control to instructorsToday we made two changes to CrowdGrader that we hope will give instructors better control of the grades, and students more assurance that their work is properly evaluated. <br />
<br />
<h2>
Average grade, and grade difference, for every submission</h2>
First, the list of all submissions now includes both the average grade received by a submission, and the delta between the highest and lowest grades. You can sort on either column, thus having quick access to the students who got the lowest average grade, and those who got the most widely differing grades.<br />
<br />
This is how this looks on a sample assignment, and on a real assignment (with student identities hidden for privacy):<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_nQoaLSnu1imY0j7ox5vnh40CrRBr7PW3Mum8wUCtnu9c_cUPHFLKzr1OKv2j8i7h67d0BGbVPQ8isQ_N1lC476US_nLTibHSiaRC-tasZxCVwrsoxLuVXaxZNIs8zIo_bleiKHzZuJI/s1600/Screen+Shot+2015-03-08+at+4.21.05+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_nQoaLSnu1imY0j7ox5vnh40CrRBr7PW3Mum8wUCtnu9c_cUPHFLKzr1OKv2j8i7h67d0BGbVPQ8isQ_N1lC476US_nLTibHSiaRC-tasZxCVwrsoxLuVXaxZNIs8zIo_bleiKHzZuJI/s1600/Screen+Shot+2015-03-08+at+4.21.05+PM.png" height="135" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Average grade and grade difference <br />
for a sample assignment.</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9dppVWVLSE4QxiiNkUVhuIrGTAMYAC_EuMACHHofjvXZl1Qekl_w4PDGcvM9HOp1HJN1sxScLIYs3jZpjLUgYt8CQhg6nSnUIH8yoHQajFKop7W9yTS1c6fXoo_kPMHNIVi4J_VCDzOk/s1600/Screen+Shot+2015-03-08+at+4.28.12+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9dppVWVLSE4QxiiNkUVhuIrGTAMYAC_EuMACHHofjvXZl1Qekl_w4PDGcvM9HOp1HJN1sxScLIYs3jZpjLUgYt8CQhg6nSnUIH8yoHQajFKop7W9yTS1c6fXoo_kPMHNIVi4J_VCDzOk/s1600/Screen+Shot+2015-03-08+at+4.28.12+PM.png" height="128" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Average grade and grade difference <br />
for a real assignment.</td></tr>
</tbody></table>
<h2 style="clear: both; text-align: left;">
Instructors can now also grade submissions</h2>
<div class="separator" style="clear: both; text-align: left;">
The other change, which goes hand in hand with the previous one, is that instructors can now grade submissions, if they wish. The option is offered when viewing the details of a submission, and here is how it looks: </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiP5oThrgbCRnTQpciv7f_e6lHHnj_17INCPJAyTsUcTlCM_KelZHJCeCy6hSAC_IlOGwijHX7peDh3Wgz8pvCQIZSgj15htcrTzU1LmbhG0Mngn1_JwpKjeussj7iLAncdMnd9Yi7n7oU/s1600/Screen+Shot+2015-03-08+at+4.21.36+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiP5oThrgbCRnTQpciv7f_e6lHHnj_17INCPJAyTsUcTlCM_KelZHJCeCy6hSAC_IlOGwijHX7peDh3Wgz8pvCQIZSgj15htcrTzU1LmbhG0Mngn1_JwpKjeussj7iLAncdMnd9Yi7n7oU/s1600/Screen+Shot+2015-03-08+at+4.21.36+PM.png" height="130" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
If instructors select <i>Review and grade this submission, </i>they are led to a page where they enter a review, and a grade, for the submission, using an interface that is the same as the one used by regular reviewers. When the assignment grades are computed, CrowdGrader gives precedence to the teacher grade over the student-assigned grades. </div>
<h2 style="clear: both; text-align: left;">
The result: more information, and more control</h2>
<div class="separator" style="clear: both; text-align: left;">
These two changes together give more information and more control to instructors. Instructors can easily see which submissions received the most widely different (and thus, more unreliable) grades, and they can easily re-grade a subset of the submissions, ensuring that they receive proper grades. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Furthermore, in case where students give widely exaggerated grades, the instructors can easily intervene and assign proper grades to the submissions. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Since instructor-assigned grades take precedence over student-assigned grades, this also creates a powerful incentive for students to be fair in their grading: students whose grade differs markedly from the instructor-assigned grade will receive a low <a href="http://doc.crowdgrader.org/crowdgrader-documentation/crowd-grades" target="_blank">review grade</a>. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<br />Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-28552448951618171412015-02-14T14:31:00.000-08:002015-02-14T14:31:00.407-08:00Authors, reviewers, and instructors can discuss submissionsAssociated with each <a href="http://www.crowdgrader.org/" target="_blank">CrowdGrader</a> submission is a forum where submission author, reviewers, and instructors can discuss the submission. This can help clarify any problems that might arise during the review process. For example, authors that make minor mistakes that might prevent the reviewers from properly evaluating their submissions can alert the reviewers. Reviewers who are unable to understand parts of the submission can ask the author for clarifications. The goal is to have a more transparent review process, where submission authors have confidence that their submission is properly evaluated, and reviewers have all the information they need. <br />
<br />
Recently, we have given instructors the ability to actively participate in these discussion forums. Here is an example of how this works.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEGKD7luPzVrqQoLgErsxytAX1-pAScoUoSpq7ifa-gMUW4X0XqKRJTi0MAJZvOdZQmP7nwFyorNuhf4UNJfxP5PRdS4jI921lw7F1BJcwVgj_HRzpucgQkD2_qJ7IIxzOWBsKIz6Et8g/s1600/Screen+Shot+2015-01-31+at+2.29.33+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEGKD7luPzVrqQoLgErsxytAX1-pAScoUoSpq7ifa-gMUW4X0XqKRJTi0MAJZvOdZQmP7nwFyorNuhf4UNJfxP5PRdS4jI921lw7F1BJcwVgj_HRzpucgQkD2_qJ7IIxzOWBsKIz6Et8g/s1600/Screen+Shot+2015-01-31+at+2.29.33+PM.png" height="144" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A discussion between submission<br />
author and reviewers, as seen<br />
by the instructor.</td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgklwiuiJGEN2vuiNqAaDMqrgVPBQkjQYQdr2u3o5kwqiVwXkWeoQrGhYQn3_lBhvE169p74iOoqpdeHRAQzLtG7Yr9wFKCjqQ9Z9Rwafr-KT7h7f92rGfd4iynRwrmdk9gy_TjzRKS_Lo/s1600/Screen+Shot+2015-01-31+at+2.30.14+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgklwiuiJGEN2vuiNqAaDMqrgVPBQkjQYQdr2u3o5kwqiVwXkWeoQrGhYQn3_lBhvE169p74iOoqpdeHRAQzLtG7Yr9wFKCjqQ9Z9Rwafr-KT7h7f92rGfd4iynRwrmdk9gy_TjzRKS_Lo/s1600/Screen+Shot+2015-01-31+at+2.30.14+PM.png" height="170" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The instructor can add a message.</td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiamy1JrWnJ0-8-I3TiLZAbw_cYm0nBVjF_HG_kyuIJRVAY-yP1f1HxC3CmAgZg6jxuLI6Jux6GSR5r6tr6fUFlusciriIt8jdBiLyYtryW-ymYxDk5RphTC4NTapbE-VZBR6wj-XlwepY/s1600/Screen+Shot+2015-01-31+at+2.30.32+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiamy1JrWnJ0-8-I3TiLZAbw_cYm0nBVjF_HG_kyuIJRVAY-yP1f1HxC3CmAgZg6jxuLI6Jux6GSR5r6tr6fUFlusciriIt8jdBiLyYtryW-ymYxDk5RphTC4NTapbE-VZBR6wj-XlwepY/s1600/Screen+Shot+2015-01-31+at+2.30.32+PM.png" height="158" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The discussion, with the message<br />
added by the instructor.</td></tr>
</tbody></table>
<br />Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-64844873826513622122015-02-07T14:19:00.000-08:002015-02-07T14:19:00.329-08:00Reviewers can annotate submissions, and upload them with their reviewsA new feature of <a href="http://www.crowdgrader.org/" target="_blank">CrowdGrader</a> enables reviewers to upload files with their review. In this way, students can collaborate with reviewers using any file format, from pdf, to doc, to file bundles for code. For example, students can download essays or papers in pdf/doc format, annotate them in place, then upload the annotated versions of the files together with their reviews.<br />
<br />
This feature is already in successful use in some early-adopter classes. We hope it might be useful to you!<br />
<br />Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-33969247549584919752015-01-30T19:18:00.002-08:002015-01-30T19:18:17.126-08:00Visual TourIt can be difficult for a user of CrowdGrader to get a full picture of the tool. Instructors and students each see only one side of it, and due to the various phases of assignments, from submission to evaluation and grading, it is difficult to form at once an idea of how it looks like. <br />
<div>
<br /></div>
<div>
To help you, we have created a visual tool, which gives you both the instructor and the student perspective along the various phases of the assignment. </div>
<div>
<br />
<ul>
<li><a href="http://crowdgrader.blogspot.com/2015/01/visual-tour-creating-assignment.html" target="_blank">Creating an assignment</a></li>
<li><a href="http://crowdgrader.blogspot.com/2015/01/visual-tour-submitting-solution-to.html" target="_blank">Submitting to an assignment</a></li>
<li><a href="http://crowdgrader.blogspot.com/2015/01/visual-tour-submissions-as-seen-by.html" target="_blank">The submission phase, seen by the teacher</a></li>
<li><a href="http://crowdgrader.blogspot.com/2015/01/visual-tour-reviewing-as-seen-by-student.html" target="_blank">Reviewing, as seen by a student</a></li>
<li><a href="http://crowdgrader.blogspot.com/2015/01/visual-tour-reviewing-as-seen-by-teacher.html" target="_blank">Reviewing, as seen by the teacher</a></li>
<li><a href="http://crowdgrader.blogspot.com/2015/01/visual-tour-feedback-phase.html" target="_blank">The feedback phase</a></li>
<li><a href="http://crowdgrader.blogspot.com/2015/01/visual-tour-computing-grades.html" target="_blank">Computing the grades</a></li>
<li><a href="http://crowdgrader.blogspot.com/2015/01/visual-tour-students-evaluation.html" target="_blank">A student's evaluation</a></li>
</ul>
</div>
<div>
Enjoy!</div>
Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-82439013902251884512015-01-30T19:13:00.003-08:002015-01-31T10:51:23.422-08:00Visual Tour: A student's evaluationOnce the assignment is concluded, a student can see the final grade received, as well as all reviews received.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUWBp0jDmNEJQzsGDbWCHxG0MtqOplhCQQvoRd4e7r8tYP-4oAsco1C0N9NjsR-yQwtRCQBVzQVZCXnAOAKPkAwAG9Upw_nGqcBvgZPwrfD8_7y7nywlgzLX6Hijv29xM8FE6vbx3p8Wg/s1600/Screen+Shot+2015-01-30+at+7.07.30+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUWBp0jDmNEJQzsGDbWCHxG0MtqOplhCQQvoRd4e7r8tYP-4oAsco1C0N9NjsR-yQwtRCQBVzQVZCXnAOAKPkAwAG9Upw_nGqcBvgZPwrfD8_7y7nywlgzLX6Hijv29xM8FE6vbx3p8Wg/s1600/Screen+Shot+2015-01-30+at+7.07.30+PM.png" height="320" width="225" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The full evaluation received<br />
by a student.</td></tr>
</tbody></table>
<br />
<br />
<div style="clear: both;">
</div>
<br />
A student can also view all the reviews she or he wrote, and compare them with those by others:<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbMAqqIvWbYI0Yk6-keyNbOa8wibDICX3U7kHU7H9cQxrH2sjowgnlqCs933YsNQnRwBoGY8kpdrNtHm-l_YEib_rGjsPJiWSAI5McMVtAgHuB9DVdej3ZM6uUufddQ2dj57CrmwW1z38/s1600/Screen+Shot+2015-01-30+at+7.07.54+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbMAqqIvWbYI0Yk6-keyNbOa8wibDICX3U7kHU7H9cQxrH2sjowgnlqCs933YsNQnRwBoGY8kpdrNtHm-l_YEib_rGjsPJiWSAI5McMVtAgHuB9DVdej3ZM6uUufddQ2dj57CrmwW1z38/s1600/Screen+Shot+2015-01-30+at+7.07.54+PM.png" height="193" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">All the reviews written by a student.</td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj31KODD275XyQusCtmBmbIZKdnD_dSWXlPWPv2UarvB-4SCs5rnWuAmSye-v0iditPw-s5WCns6C8XmkQ0OEN9XiN85uM1e6_lIrVh4jcg3JO9mC5C_Uwjc6aAGcXo5revTh7bJ064wqQ/s1600/Screen+Shot+2015-01-30+at+7.08.13+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj31KODD275XyQusCtmBmbIZKdnD_dSWXlPWPv2UarvB-4SCs5rnWuAmSye-v0iditPw-s5WCns6C8XmkQ0OEN9XiN85uM1e6_lIrVh4jcg3JO9mC5C_Uwjc6aAGcXo5revTh7bJ064wqQ/s1600/Screen+Shot+2015-01-30+at+7.08.13+PM.png" height="180" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A student can compare her/his<br />
reviews with those by others.</td></tr>
</tbody></table>
<br />Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-20509588719121809172015-01-30T18:57:00.001-08:002015-01-31T10:51:01.898-08:00Visual Tour: Computing the grades<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjACaf_UP1Mrj0NO7RytPULUr5ZpnsOPzxtGT-360ewnubEz_mmALJRJFVFQ0rOvb_u4fbkxDGeoR5X1iYIMlBKp8w-SG-JulfntRPmW8SAYpI7UrV_ueSnSYWrnT-OMWYqKkW4D5tnHgE/s1600/Screen+Shot+2015-01-30+at+12.15.12+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjACaf_UP1Mrj0NO7RytPULUr5ZpnsOPzxtGT-360ewnubEz_mmALJRJFVFQ0rOvb_u4fbkxDGeoR5X1iYIMlBKp8w-SG-JulfntRPmW8SAYpI7UrV_ueSnSYWrnT-OMWYqKkW4D5tnHgE/s1600/Screen+Shot+2015-01-30+at+12.15.12+PM.png" height="83" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">When the grades are computed,<br />
feedback on the reviews is<br />
no longer accepted.</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvtqx37Fov6fCtPxoB33yhUga76G36Qa490ObJkVmP-HdWjCjRY1YkrU3rgmFV7Ff7D2EeTA4ldW7-4UgB9Vs7HOLi0dMAtRkDhfWun_BplSEuJLepVpH925AnH612JNHcLCW4pQTgyRQ/s1600/Screen+Shot+2015-01-30+at+12.15.40+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvtqx37Fov6fCtPxoB33yhUga76G36Qa490ObJkVmP-HdWjCjRY1YkrU3rgmFV7Ff7D2EeTA4ldW7-4UgB9Vs7HOLi0dMAtRkDhfWun_BplSEuJLepVpH925AnH612JNHcLCW4pQTgyRQ/s1600/Screen+Shot+2015-01-30+at+12.15.40+PM.png" height="141" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The crowd-grades have been computed.</td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhrixl2IoZKLah-Nc-Xa5cChDyrCzZlzY_cIXQXYSEYsCKVB3zH7g9NHTJjw13epUnnlaFyykk91nnD1tXL_zikw7gsp21_i9Ir5vA85OZeAKHgAVIDJ3MychHdH9fcQSCt0aVSI3zHsE/s1600/Screen+Shot+2015-01-30+at+12.16.07+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhrixl2IoZKLah-Nc-Xa5cChDyrCzZlzY_cIXQXYSEYsCKVB3zH7g9NHTJjw13epUnnlaFyykk91nnD1tXL_zikw7gsp21_i9Ir5vA85OZeAKHgAVIDJ3MychHdH9fcQSCt0aVSI3zHsE/s1600/Screen+Shot+2015-01-30+at+12.16.07+PM.png" height="140" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The instructor can decide when grades<br />
become visible by students.</td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhP5WRNx0VHVvH943KxEjttf6eKDYOzOnl8skw1y6jSm2DTwbYUCpBt9j-S3i6k0zlsmON6SPK7FbSNyZD1g5MrA6LsGjr4Onvo6uLVMYb8rowkYnaJaJACQb0OC8xBlAtVQGnDy9aasok/s1600/Screen+Shot+2015-01-30+at+12.17.01+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhP5WRNx0VHVvH943KxEjttf6eKDYOzOnl8skw1y6jSm2DTwbYUCpBt9j-S3i6k0zlsmON6SPK7FbSNyZD1g5MrA6LsGjr4Onvo6uLVMYb8rowkYnaJaJACQb0OC8xBlAtVQGnDy9aasok/s1600/Screen+Shot+2015-01-30+at+12.17.01+PM.png" height="140" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The instructor can assign some final<br />
grades by hand...</td></tr>
</tbody></table>
<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYo-8RmCejCaAmtp3gkytMji-QCX_A8weExlGiO6PAiNnmK_rATOrhyphenhyphensPWPqbcbVGMabs05XPZjClKcy2GXEwHXgcc560f2ujqYXHfBu-sq4GNnfg_BGVSZeb4HYYhmzfvVgiHLEXi4iY/s1600/Screen+Shot+2015-01-30+at+12.17.35+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYo-8RmCejCaAmtp3gkytMji-QCX_A8weExlGiO6PAiNnmK_rATOrhyphenhyphensPWPqbcbVGMabs05XPZjClKcy2GXEwHXgcc560f2ujqYXHfBu-sq4GNnfg_BGVSZeb4HYYhmzfvVgiHLEXi4iY/s1600/Screen+Shot+2015-01-30+at+12.17.35+PM.png" height="140" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">...and the remaining ones<br />
are computed by interpolation.</td></tr>
</tbody></table>
<br />Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-53008650081805690782015-01-30T18:50:00.000-08:002015-01-31T19:18:15.317-08:00Visual Tour: The feedback phaseOnce reviewing is complete, students can see their reviews, and give feedback on the reviewers:<br />
<br />
<div style="text-align: left;">
</div>
<div style="text-align: left;">
</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrq9nCNyrlgt0-co0g9UU0rhnqO9tW6jsRuD2K-TlqcWrp_Dw4T9XVQY7yXVtL-lvn6ZhDIAUTfKVWDeAF9fgQ7LjJ8XwsqxuN0mfkTYGdY47AFdR71NoFbksegT4r0Q10fQCdVQNI9NY/s1600/Screen+Shot+2015-01-29+at+7.29.07+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrq9nCNyrlgt0-co0g9UU0rhnqO9tW6jsRuD2K-TlqcWrp_Dw4T9XVQY7yXVtL-lvn6ZhDIAUTfKVWDeAF9fgQ7LjJ8XwsqxuN0mfkTYGdY47AFdR71NoFbksegT4r0Q10fQCdVQNI9NY/s1600/Screen+Shot+2015-01-29+at+7.29.07+PM.png" height="200" width="159" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">At the end of the review<br />
phase, students can leave<br />
feedback on their reviews.</td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLEYzgFcwxtkp7iDMIjZygbnIi2yJAsgq7etax19mSaQiQgxWF3EXIgMJ2Cx1ueP9FPnEsvfl9eppeznZcFFpmeCM7z6Frn9ofQrg7Z4XtUN-TrolTIGI7ZVHEDB5Ng_ONlX-JKvfRDks/s1600/Screen+Shot+2015-01-29+at+7.30.02+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLEYzgFcwxtkp7iDMIjZygbnIi2yJAsgq7etax19mSaQiQgxWF3EXIgMJ2Cx1ueP9FPnEsvfl9eppeznZcFFpmeCM7z6Frn9ofQrg7Z4XtUN-TrolTIGI7ZVHEDB5Ng_ONlX-JKvfRDks/s1600/Screen+Shot+2015-01-29+at+7.30.02+PM.png" height="126" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Leaving feedback on a review.</td></tr>
</tbody></table>
<br />Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-33709164531938913532015-01-30T18:43:00.002-08:002015-01-31T10:49:59.482-08:00Visual Tour: Reviewing, as seen by the instructorInstructor's view of the review progress:
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgOCkmFRACKXyQw-LRnQv1UBBKgdkTzHLywSZYbU1w8NrY9h9-6mdUngOcaLrf4N1PIo9WZ-GlyCAVEEFzZRiKyyQJ5mXjx_M-1lvbDdiXrnQw1fJp_kyTztQGnIoFEiYujgNZIgiSMAVs/s1600/Screen+Shot+2015-01-29+at+6.03.35+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgOCkmFRACKXyQw-LRnQv1UBBKgdkTzHLywSZYbU1w8NrY9h9-6mdUngOcaLrf4N1PIo9WZ-GlyCAVEEFzZRiKyyQJ5mXjx_M-1lvbDdiXrnQw1fJp_kyTztQGnIoFEiYujgNZIgiSMAVs/s1600/Screen+Shot+2015-01-29+at+6.03.35+PM.png" height="114" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The instructor has a bird's eye view<br />
on the submissions,<br />
the grades assigned, and any discussion.</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_FaxR-lqKHiAWh1tdVuuyf9ILCAi8CckLk8WwsuNSYXLbfjE-5FaGv7VGFnISUtGf0X9Lil0Ms82UySrRkqaljx-6AtWSg8LsZEQemzZiNqkiviWOVypp30Pkfm3EhNT-1DNhlw199iA/s1600/Screen+Shot+2015-01-29+at+6.04.23+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_FaxR-lqKHiAWh1tdVuuyf9ILCAi8CckLk8WwsuNSYXLbfjE-5FaGv7VGFnISUtGf0X9Lil0Ms82UySrRkqaljx-6AtWSg8LsZEQemzZiNqkiviWOVypp30Pkfm3EhNT-1DNhlw199iA/s1600/Screen+Shot+2015-01-29+at+6.04.23+PM.png" height="106" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A discussion as seen by the teacher.</td></tr>
</tbody></table>
<br />
<div style="clear: both;">
</div>
<br />
The instructor has easy access to all the reviews, making it easy to form an idea of how the assignment went:
<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbZsP6K6fvluSYZAwEIAqQKHPaCh1zzxqvhTMIsMrO9uQUmtlCj6Whw_9lCphVBigSZDR9zLOqjD2194YvbibOxKizRKsH2qhw-BcepAcOOr_WRxzhHMXQ7NpTGiFkiAq2mrNBqEwf340/s1600/Screen+Shot+2015-01-29+at+7.26.22+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbZsP6K6fvluSYZAwEIAqQKHPaCh1zzxqvhTMIsMrO9uQUmtlCj6Whw_9lCphVBigSZDR9zLOqjD2194YvbibOxKizRKsH2qhw-BcepAcOOr_WRxzhHMXQ7NpTGiFkiAq2mrNBqEwf340/s1600/Screen+Shot+2015-01-29+at+7.26.22+PM.png" height="148" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">All the reviews.</td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvIiiRW1Z_iFghx4D3VSkZQHggQDx0UzRCAtg5V5zAgOGp7ENh95hqziY8_hbgwlxllerZV_H4exhoEu0XVWQu3WLi06mGGgGhq-gGPP5wjAQuYjlwB2f5MqQ9qISJ3zJYIZgDQ1BveY4/s1600/Screen+Shot+2015-01-29+at+7.26.37+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvIiiRW1Z_iFghx4D3VSkZQHggQDx0UzRCAtg5V5zAgOGp7ENh95hqziY8_hbgwlxllerZV_H4exhoEu0XVWQu3WLi06mGGgGhq-gGPP5wjAQuYjlwB2f5MqQ9qISJ3zJYIZgDQ1BveY4/s1600/Screen+Shot+2015-01-29+at+7.26.37+PM.png" height="99" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">All the reviews by a given student.</td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJ5Xas7IjfD4hucXODH41TxF-m8BS8LzR3IepUTzqo54REF1IJZBRcWQKxp3ksZ1X4EKnO0UEI-2Eo2ncqT8xjWNVtTjCrJK0XdqU6ez28QpeXQQlK66Cjkmtj18vAITcTNhkOe7JsAH8/s1600/Screen+Shot+2015-01-29+at+7.29.07+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJ5Xas7IjfD4hucXODH41TxF-m8BS8LzR3IepUTzqo54REF1IJZBRcWQKxp3ksZ1X4EKnO0UEI-2Eo2ncqT8xjWNVtTjCrJK0XdqU6ez28QpeXQQlK66Cjkmtj18vAITcTNhkOe7JsAH8/s1600/Screen+Shot+2015-01-29+at+7.29.07+PM.png" height="200" width="159" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">All the reviews<br />
for a given submission.</td></tr>
</tbody></table>
Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-87286294831836731362015-01-30T18:40:00.002-08:002015-01-31T10:48:28.316-08:00Visual Tour: Reviewing, as seen by a student<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvgXQEyBWx3aqFFeX-J2es4yDX-VEEaGebN-TPbkPvyKHbJSdjwW2hwf9Oq16hVXHkKxvG2dB2QbwHYmts-eOKA3epBvYaZJWe8_7pBFgW0zW93sfbCL3NOtqDLu0C4gy-bG3exYfjcVI/s1600/Screen+Shot+2015-01-28+at+4.01.21+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvgXQEyBWx3aqFFeX-J2es4yDX-VEEaGebN-TPbkPvyKHbJSdjwW2hwf9Oq16hVXHkKxvG2dB2QbwHYmts-eOKA3epBvYaZJWe8_7pBFgW0zW93sfbCL3NOtqDLu0C4gy-bG3exYfjcVI/s1600/Screen+Shot+2015-01-28+at+4.01.21+PM.png" height="128" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The student has not done any<br />
reviews yet.</td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyeG9AlX7tYD9odPK2PRaNUQtq24B25c-hY7DDNZwpu4YX-R7y_CpEYVp3rThM4mNH1TjCPgjbbBhsC6airdTcyaFovzFSUH6IES1UaYEGjl38zBwcf7ZPib_9O4vxuOSqDECc3BfkjSk/s1600/Screen+Shot+2015-01-29+at+3.05.13+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyeG9AlX7tYD9odPK2PRaNUQtq24B25c-hY7DDNZwpu4YX-R7y_CpEYVp3rThM4mNH1TjCPgjbbBhsC6airdTcyaFovzFSUH6IES1UaYEGjl38zBwcf7ZPib_9O4vxuOSqDECc3BfkjSk/s1600/Screen+Shot+2015-01-29+at+3.05.13+PM.png" height="200" width="170" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A student in the process of entering a review.</td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEineZ5x7MuSJh4E_be3syG2ghxMUQbh3SiMl8JeICsHPBp2fyhwMijECtuqGMwwMCrnSjWUbR7Ytt8kgFxvu69q0SSQAq6r7P8DwOtKjqiBjoiuTi3hTpqbabKdl93AiaDp0RRt4S5YXLM/s1600/Screen+Shot+2015-01-29+at+3.06.41+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEineZ5x7MuSJh4E_be3syG2ghxMUQbh3SiMl8JeICsHPBp2fyhwMijECtuqGMwwMCrnSjWUbR7Ytt8kgFxvu69q0SSQAq6r7P8DwOtKjqiBjoiuTi3hTpqbabKdl93AiaDp0RRt4S5YXLM/s1600/Screen+Shot+2015-01-29+at+3.06.41+PM.png" height="200" width="187" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Reviews are checked for<br />
consistency with the grading rubric.</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLE_EKwmIIRC05OyqohIPCp1KnYpeQhcPAPryDts9OHlQMbbjAAYKYkVVb_BCITJTaV9_QVJdWHJJEKrDFRfUhP2ayhFZsgt1RteePRYt6NgC2bhVlis7SMXUPjLAjUTfrk-vK5GyyFww/s1600/Screen+Shot+2015-01-29+at+3.07.10+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLE_EKwmIIRC05OyqohIPCp1KnYpeQhcPAPryDts9OHlQMbbjAAYKYkVVb_BCITJTaV9_QVJdWHJJEKrDFRfUhP2ayhFZsgt1RteePRYt6NgC2bhVlis7SMXUPjLAjUTfrk-vK5GyyFww/s1600/Screen+Shot+2015-01-29+at+3.07.10+PM.png" height="163" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The first two reviews are done.</td></tr>
</tbody></table>
<br />
<div style="clear: both;">
</div>
<br />
Students can also discuss the submissions while doing the reviews:
<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiY6gTvMeeU5Uyuq5gXp-CVegDkzYWEuVWje8ada2fIHi3a51OXtcG5xVryvJfVpnI_qUKq0l-UFKltCL7yo6eoArH1EIw1Nsy4509fyJsd5SHQqFJRonby33Et2P2IwSxp_op4EWSARUU/s1600/Screen+Shot+2015-01-29+at+3.13.59+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiY6gTvMeeU5Uyuq5gXp-CVegDkzYWEuVWje8ada2fIHi3a51OXtcG5xVryvJfVpnI_qUKq0l-UFKltCL7yo6eoArH1EIw1Nsy4509fyJsd5SHQqFJRonby33Et2P2IwSxp_op4EWSARUU/s1600/Screen+Shot+2015-01-29+at+3.13.59+PM.png" height="120" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The initial message from<br />
the submission author.</td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnC3456ZrWxz1ZKfBSUbJXMblbIvJI8MLRdJFkA7Ynj-ry04yyeygba8vmKAk2ba5TR_WOpuLQXNk-s7999U_wtmxSXZkIoLM7ImUxRXp-iWFJgb2eB6UngeSPHjT4DJ0Y6DPBRdVfwkw/s1600/Screen+Shot+2015-01-29+at+3.14.24+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnC3456ZrWxz1ZKfBSUbJXMblbIvJI8MLRdJFkA7Ynj-ry04yyeygba8vmKAk2ba5TR_WOpuLQXNk-s7999U_wtmxSXZkIoLM7ImUxRXp-iWFJgb2eB6UngeSPHjT4DJ0Y6DPBRdVfwkw/s1600/Screen+Shot+2015-01-29+at+3.14.24+PM.png" height="169" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The reviewer adds a message.</td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwLwAHmwNJ18OzSPUG3HTWEyNq2cuUoIcwDWYyogxNY5Tn2FaqD0GXkCXbQtHxxSaCj9tE9jszRAqX7AZipH9w-ksiod898XC8hcAxf4R_PbIJ8bprDnNnBQcNDm-N5fghEpSr-Fe7nu4/s1600/Screen+Shot+2015-01-29+at+3.14.35+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwLwAHmwNJ18OzSPUG3HTWEyNq2cuUoIcwDWYyogxNY5Tn2FaqD0GXkCXbQtHxxSaCj9tE9jszRAqX7AZipH9w-ksiod898XC8hcAxf4R_PbIJ8bprDnNnBQcNDm-N5fghEpSr-Fe7nu4/s1600/Screen+Shot+2015-01-29+at+3.14.35+PM.png" height="135" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The message is added<br />
to the discussion.</td></tr>
</tbody></table>
<br />Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-30614035351682302102015-01-30T18:28:00.002-08:002015-01-31T11:23:34.725-08:00Visual Tour: The submissions, as seen by the instructor<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAQrKcX9Dqu6rU_KqIJ9_xXfuC_hrYermgtpe9qaAsEsbCa_q68bCen_44z4Hc41X2j1YybnWqOPIrL-b1fEjg4jJBzEcUtQUzi3sedDFqz8U0dxNTnCLX3cXXkVYw04VbkPvgr_OYxWg/s1600/Screen+Shot+2015-01-28+at+3.57.01+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAQrKcX9Dqu6rU_KqIJ9_xXfuC_hrYermgtpe9qaAsEsbCa_q68bCen_44z4Hc41X2j1YybnWqOPIrL-b1fEjg4jJBzEcUtQUzi3sedDFqz8U0dxNTnCLX3cXXkVYw04VbkPvgr_OYxWg/s1600/Screen+Shot+2015-01-28+at+3.57.01+PM.png" height="131" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The submission list,<br />
before any review.</td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhL96LI4khRFYS36_ltzJfuBhDsiSmPBnDTq4hzUb-DY-5zLmkFlkm47yIPRS0wxMATszcZLfEAFYI4Aaqck1_Oml73SNGxKaCdrcVgWv7KRZjL3KzN_5pVlkDClsZR76BPy4WBwUizLTY/s1600/Screen+Shot+2015-01-28+at+3.57.42+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhL96LI4khRFYS36_ltzJfuBhDsiSmPBnDTq4hzUb-DY-5zLmkFlkm47yIPRS0wxMATszcZLfEAFYI4Aaqck1_Oml73SNGxKaCdrcVgWv7KRZjL3KzN_5pVlkDClsZR76BPy4WBwUizLTY/s1600/Screen+Shot+2015-01-28+at+3.57.42+PM.png" height="178" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A submission, as seen by the instructor</td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDpm9L0blD9FNICuThzYRCzpNpV_wRsQ4bPrREQhdpchUH__aoy650XBnajRRXCnHCU_47A8BqhQAnIa10__7GsdIrKYGf9IBZUg0Q2m2WUkCKEPRRwc_L8spWWtrzNKLAVrDqRe4zXlE/s1600/Screen+Shot+2015-01-28+at+3.57.19+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDpm9L0blD9FNICuThzYRCzpNpV_wRsQ4bPrREQhdpchUH__aoy650XBnajRRXCnHCU_47A8BqhQAnIa10__7GsdIrKYGf9IBZUg0Q2m2WUkCKEPRRwc_L8spWWtrzNKLAVrDqRe4zXlE/s1600/Screen+Shot+2015-01-28+at+3.57.19+PM.png" height="133" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The discussion on the submission</td></tr>
</tbody></table>
<br />Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-70234314490663547862015-01-30T18:01:00.001-08:002015-01-31T10:47:00.162-08:00Visual Tour: Submitting a solution to an assignmentHere is how submitting a solution to an assignment looks like, from the students point of view.<br />
<br />
<br />
<div style="clear: both;">
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTFzb40vC1vcBNk1vFgdAS_o7AXgjmmpzvCNFORTa6ejMwiq3oZwinS1h21DRVh50hWCyQmiq8Rxqi5M0qyImvQ8Ny1DkGstEIiDGc3zV1cVpWL8vTx1OqObQPob3dfxs40xZRMayveJI/s1600/Screen+Shot+2015-01-28+at+3.29.06+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTFzb40vC1vcBNk1vFgdAS_o7AXgjmmpzvCNFORTa6ejMwiq3oZwinS1h21DRVh50hWCyQmiq8Rxqi5M0qyImvQ8Ny1DkGstEIiDGc3zV1cVpWL8vTx1OqObQPob3dfxs40xZRMayveJI/s1600/Screen+Shot+2015-01-28+at+3.29.06+PM.png" height="125" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">An image has just been uploaded.</td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIYHQgWfVSxO0hlzU89IoEd6BCLOUit_0b3mCsi7cs29uO2kwXFea6bUuqpqujO9iF6KjZLqQbBYV6rutl5m7zdv8_y2I4C8p_aXMlB1kPSsTOnv9lJAJAsrPiD7EXYL_2TVHXoFlNoDk/s1600/Screen+Shot+2015-01-28+at+3.56.07+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIYHQgWfVSxO0hlzU89IoEd6BCLOUit_0b3mCsi7cs29uO2kwXFea6bUuqpqujO9iF6KjZLqQbBYV6rutl5m7zdv8_y2I4C8p_aXMlB1kPSsTOnv9lJAJAsrPiD7EXYL_2TVHXoFlNoDk/s1600/Screen+Shot+2015-01-28+at+3.56.07+PM.png" height="162" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The text of a submission can be<br />
written using a <a href="http://crowdgrader.blogspot.com/2014/01/powerful-new-editor-supports-math-code.html" target="_blank">powerful markup language</a>.</td></tr>
</tbody></table>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGSLMLDNN41Ejas0DbC_teupNjKpKPELJSkvb_IWLPI_y2bhqj7vc8zaFg6M3WDRFyksbDA2lAPdRXEmRWcNptELedUjfZqgOw3fC_0qSJW7cdtY7JAy0hOEsg3H5pElr-fSBSeHo1zIY/s1600/Screen+Shot+2015-01-28+at+3.30.40+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGSLMLDNN41Ejas0DbC_teupNjKpKPELJSkvb_IWLPI_y2bhqj7vc8zaFg6M3WDRFyksbDA2lAPdRXEmRWcNptELedUjfZqgOw3fC_0qSJW7cdtY7JAy0hOEsg3H5pElr-fSBSeHo1zIY/s1600/Screen+Shot+2015-01-28+at+3.30.40+PM.png" height="233" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A completed submission.</td></tr>
</tbody></table>
<br /></div>
<div style="clear: both;">
<br />
Students can also discuss their submissions with the people who will review them; in this case notifying them of the fact that the image
has been resized.
<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGGrsTXdj06AvYhB2UDzYaOGKOyk_HnaGBXtWiHZa5-wQSLuDuT4PCmZn3T7MKZHCeAEm0tjnRqJfUMIkIYTMNVzk959-fyh5JSjVJxi_AfnQLsFEwFXw47uAIJix0KWOOQt27Vj4woF8/s1600/Screen+Shot+2015-01-28+at+3.31.04+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGGrsTXdj06AvYhB2UDzYaOGKOyk_HnaGBXtWiHZa5-wQSLuDuT4PCmZn3T7MKZHCeAEm0tjnRqJfUMIkIYTMNVzk959-fyh5JSjVJxi_AfnQLsFEwFXw47uAIJix0KWOOQt27Vj4woF8/s1600/Screen+Shot+2015-01-28+at+3.31.04+PM.png" height="150" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">An empty discussion list.</td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-66o4QM1DmeLuTH_zJYxEwI__aXx_pHRE-jrmyQOuLb1GSBtaOiniQ18kHgHq1GOmISArwonJR1piYdqzjOLNYTYr6pL2wi6J2De1etIxY9r0PA77nPOe926xvqOUrunkd-jv-smU2XY/s1600/Screen+Shot+2015-01-28+at+3.31.35+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-66o4QM1DmeLuTH_zJYxEwI__aXx_pHRE-jrmyQOuLb1GSBtaOiniQ18kHgHq1GOmISArwonJR1piYdqzjOLNYTYr6pL2wi6J2De1etIxY9r0PA77nPOe926xvqOUrunkd-jv-smU2XY/s1600/Screen+Shot+2015-01-28+at+3.31.35+PM.png" height="138" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Writing the first message.</td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGQu4domm0iN1UyUGaTfX6qGVecTOENT2rTV9u99Rt4EYZq7Ppl6Oyyy3hpJNGCiqzKw5EVe146pN_j-RuwmCB6UZlTt0dH4UkcASTMEjNspubSgxO5FNhBKx6bDoE7T1KUaoB5kUVyCs/s1600/Screen+Shot+2015-01-28+at+3.31.48+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGQu4domm0iN1UyUGaTfX6qGVecTOENT2rTV9u99Rt4EYZq7Ppl6Oyyy3hpJNGCiqzKw5EVe146pN_j-RuwmCB6UZlTt0dH4UkcASTMEjNspubSgxO5FNhBKx6bDoE7T1KUaoB5kUVyCs/s1600/Screen+Shot+2015-01-28+at+3.31.48+PM.png" height="163" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The first message.</td></tr>
</tbody></table>
<br /></div>
Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-77222933111714228572015-01-30T17:47:00.000-08:002015-01-31T10:40:54.789-08:00Visual Tour: Creating an assignmentHere is how creating an assignment, entering a student list, and creating a grading rubric looks like.<br />
<br />
<table><tbody>
<tr>
<td><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><div style="text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7_CK9aXBDNVPZvRpLz2XnNc6Z61MjUHizRtbUWufVpeLOqeslXKnNNRewv-kVzP2DLJZOcvkgQpl4RrdaaZi_61h90N80sKxT8szR3mYYa8ycldmsAwJ2QiDiUnMVsFpcMAC0nW0e9ak/s1600/Screen+Shot+2015-01-28+at+3.05.36+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7_CK9aXBDNVPZvRpLz2XnNc6Z61MjUHizRtbUWufVpeLOqeslXKnNNRewv-kVzP2DLJZOcvkgQpl4RrdaaZi_61h90N80sKxT8szR3mYYa8ycldmsAwJ2QiDiUnMVsFpcMAC0nW0e9ak/s1600/Screen+Shot+2015-01-28+at+3.05.36+PM.png" height="161" width="200" /></a></div>
</td></tr>
<tr><td class="tr-caption" style="text-align: center;">A brand new assignment. </td></tr>
</tbody></table>
</td>
<td><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgeTaq6AybceidBrvBgtvNUx24wxDnirkTfjjvIBaJNxXtdS_Id7Re19O4qN1j5664cBn480HdV0GOC-y1bed8RBYrUezu6Q6KpUMY3iPSCEO0z5xqWRiKuSwTBYexHLw-x4SyjZ2ZgLtw/s1600/Screen+Shot+2015-01-28+at+3.07.26+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgeTaq6AybceidBrvBgtvNUx24wxDnirkTfjjvIBaJNxXtdS_Id7Re19O4qN1j5664cBn480HdV0GOC-y1bed8RBYrUezu6Q6KpUMY3iPSCEO0z5xqWRiKuSwTBYexHLw-x4SyjZ2ZgLtw/s1600/Screen+Shot+2015-01-28+at+3.07.26+PM.png" height="146" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">You can either let students self-enroll, <br />
or you can cut-and-paste their emails.</td></tr>
</tbody></table>
</td></tr>
<tr><td><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijxcvak1jvZuTEThHnvjWv3GN8UsRm_e8DSdIqpokXTwVvHhEe2mkXXBPGcqfyVzK-uC7swfhkSISdt_eCay4-Sqx_G-LZiJdTp4e5wDT5Gp5deLWcfaUZboG6WhtYDFtDWRd7grmMQ4c/s1600/Screen+Shot+2015-01-28+at+3.07.41+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijxcvak1jvZuTEThHnvjWv3GN8UsRm_e8DSdIqpokXTwVvHhEe2mkXXBPGcqfyVzK-uC7swfhkSISdt_eCay4-Sqx_G-LZiJdTp4e5wDT5Gp5deLWcfaUZboG6WhtYDFtDWRd7grmMQ4c/s1600/Screen+Shot+2015-01-28+at+3.07.41+PM.png" height="151" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A student list with 5 students.</td></tr>
</tbody></table>
</td>
<td><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhcQ4USxSLGyrkc-CK_vPblj3nyvTHFWaX6BsOV-J2VwqBf5hkwdo_9rEIX7sgzd8OIzfk0FIUME-yT3fhVJvLBP2sISJgpkDOc6HZQH5tB51Gxo5AO9JtuwOeVVi8nnezhMjBiAFdOAL4/s1600/Screen+Shot+2015-01-28+at+3.09.43+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhcQ4USxSLGyrkc-CK_vPblj3nyvTHFWaX6BsOV-J2VwqBf5hkwdo_9rEIX7sgzd8OIzfk0FIUME-yT3fhVJvLBP2sISJgpkDOc6HZQH5tB51Gxo5AO9JtuwOeVVi8nnezhMjBiAFdOAL4/s1600/Screen+Shot+2015-01-28+at+3.09.43+PM.png" height="133" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">You can edit grading rubrics.</td></tr>
</tbody></table>
</td></tr>
<tr><td><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEix_6AbzMECKR5V02F5Py0__onZZcl01YCUa8fxIL8wsm-udw0baJ7hhd2278gUz2J9cC2Kn-bxvn4MoWyCF62PjqxaQDF_1VJyobQk-iQE8RqM4nz8fBTZ4IOOBz2iNfYwlLB1XLbkggk/s1600/Screen+Shot+2015-01-28+at+3.10.42+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEix_6AbzMECKR5V02F5Py0__onZZcl01YCUa8fxIL8wsm-udw0baJ7hhd2278gUz2J9cC2Kn-bxvn4MoWyCF62PjqxaQDF_1VJyobQk-iQE8RqM4nz8fBTZ4IOOBz2iNfYwlLB1XLbkggk/s1600/Screen+Shot+2015-01-28+at+3.10.42+PM.png" height="98" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A grading rubric.</td></tr>
</tbody></table>
</td></tr>
</tbody></table>
Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-43914387579905911332014-10-19T19:30:00.000-07:002014-10-20T23:02:09.867-07:00Students can now see what contributes to their gradeCrowdGrader now lets students see which information is used in computing their grade, and which information is disregarded as unreliable. Here are two examples, taken from an actual assignment:<br />
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEioSX4Jzz-cr3ZePTOrT4z6fK9rdiYfsPYQI_8rByerNKC0aR99zFjKocXDNU6SwgYKZaux_0IPJW4EJW_KYqE4IPXY2Vd65Bkl_ErGb3UvzwbL3cea1XP11kmwR1qos39FpeFAkfM2XyI/s1600/Screen+Shot+2014-10-20+at+2.05.43+PM.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><br /></a></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_1oF43arkBqmhkLBJ-NlKuUmDXcCzGkKAEbRqOia4ZjEzSflJ2N759uXsSnocqxQrFT4b90fBleARSPWNLtLYjulhfE3zMbz8w15nHqlrEQ2wZG4SGl7fH6f2X-Y58yXgUX7KLEsz2fM/s1600/Screen+Shot+2014-10-20+at+2.11.12+PM.png" imageanchor="1" style="clear: left; display: inline !important; margin-bottom: 1em; margin-left: auto; margin-right: auto; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_1oF43arkBqmhkLBJ-NlKuUmDXcCzGkKAEbRqOia4ZjEzSflJ2N759uXsSnocqxQrFT4b90fBleARSPWNLtLYjulhfE3zMbz8w15nHqlrEQ2wZG4SGl7fH6f2X-Y58yXgUX7KLEsz2fM/s1600/Screen+Shot+2014-10-20+at+2.11.12+PM.png" height="400" width="55" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">CrowdGrader indicates <br />
which review grades were <br />
considered low-confidence.</td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEioSX4Jzz-cr3ZePTOrT4z6fK9rdiYfsPYQI_8rByerNKC0aR99zFjKocXDNU6SwgYKZaux_0IPJW4EJW_KYqE4IPXY2Vd65Bkl_ErGb3UvzwbL3cea1XP11kmwR1qos39FpeFAkfM2XyI/s1600/Screen+Shot+2014-10-20+at+2.05.43+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEioSX4Jzz-cr3ZePTOrT4z6fK9rdiYfsPYQI_8rByerNKC0aR99zFjKocXDNU6SwgYKZaux_0IPJW4EJW_KYqE4IPXY2Vd65Bkl_ErGb3UvzwbL3cea1XP11kmwR1qos39FpeFAkfM2XyI/s1600/Screen+Shot+2014-10-20+at+2.05.43+PM.png" height="400" width="271" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">CrowdGrader classified the -2 feedback as possible<br />
tit-for-tat, and discarded it from the computation of<br />
the feedback grade. </td></tr>
</tbody></table>
<div>
<div style="clear: both;">
<div>
<br /></div>
<h2>
How it works</h2>
In CrowdGrader, the grade a student receives for an assignment is the sum of three components:<br />
<ol>
<li>The <i>submission grade</i>, computed from the grades other students gave to the submission.</li>
<li>The <i>accuracy grade, </i>which measures how precise the student was in giving grades.</li>
<li>The <i>feedback grade, </i>which depends on the feedback the student received for the reviews she or he wrote. </li>
</ol>
<div>
<b>Submission grade. </b>When computing the submission grade, not all reviews are considered equal: CrowdGrader uses a reputation system to give more weight to the input of more credible reviewers. CrowdGrader now lets students know which reviews were considered low confidence, and given less weight in the computation of the submission grade.</div>
<div>
<br /></div>
<div>
<b>Feedback grade. </b>The feedback a review receives can range from -2 (very unhelpful or bogus) to +2 (very helpful). CrowdGrader uses a sophisticated algorithm to determine when a negative feedback might have been given as a tit-for-tat retribution for a low grade. Such negative feedbacks are not considered in the computation of the feedback grade. </div>
<h2>
Instructors should compute the grades once all feedback is in</h2>
<div>
When CrowdGrader computes the grades, it classifies each feedback as either legitimate or possible tit-for-tat. Once grades are computed, students can no longer modify their feedback: this ensures that the feedback they see is the same that was used to compute the grades. For this reason, it is important that instructors compute grades only once all the feedback is in. </div>
</div>
</div>
Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-22333484862430882912014-09-08T11:22:00.001-07:002014-09-08T11:22:07.926-07:00Simpler assignment creation and class list managementWe just rolled out a few more features!<br />
<h2>
Simpler assignment creation</h2>
<h3>
Only one type of assignment</h3>
There is no longer a distinction between <i>standard assignments </i>(where the instructor specifies a class list) and <i>open assignments </i>(where students can self-enroll by clicking on a <i>join link </i>provided by the instructor).<br />
<br />
Rather, the instructor can always edit the class list, adding and removing students, and also has the option of handing out the join link if desired.<br />
<br />
<h3>
You can now clone assignments</h3>
We have implemented one of the top user requests: you can now clone assignments. So, once you create the first assignment for your class, you can create the subsequent assignments by cloning the first assignment, changing just the title and the submission and review dates.<br />
<br />
When you clone an assignment, you create an assignment that is identical to the cloned one, except for:<br />
<br />
<ul>
<li>The name ("Copy of " is added to the name)</li>
<li>The dates (you have to provide new dates).</li>
</ul>
<div>
The class list is copied over. </div>
<div>
<br /></div>
<div>
Note that, as explained below, each assignment has its own class list. So, if you clone an assignment, then add a student to either the cloned or the original assignment, the student is not automatically added to the other assignment. For this reason, if you expect the students enrolled in the class to change, you might want to wait and clone assignments only just before the clones are needed, so their class list will contain all changes since the start of the class.</div>
<div>
<br /></div>
<h2>
Simpler class list management</h2>
<div>
We have simplified the management of class lists. Each assignment has its own class list, not shared with any other assignment. You can edit the list as one of the options in the assignment: </div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhci5JACeQufHi4kwL9mdc9vw9q0dFueDpiMFSsIYMaVm82PKa0zRbiI3zKl1qZOqW2K9cXeB_2TJ7Nk5vViwugrXx0bpDrnt5XtkV9ckfHiTqoiRRH7o2U3BmpIuJbkvOuZPsxcupb07c/s1600/Screen+Shot+2014-09-08+at+11.15.21+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhci5JACeQufHi4kwL9mdc9vw9q0dFueDpiMFSsIYMaVm82PKa0zRbiI3zKl1qZOqW2K9cXeB_2TJ7Nk5vViwugrXx0bpDrnt5XtkV9ckfHiTqoiRRH7o2U3BmpIuJbkvOuZPsxcupb07c/s1600/Screen+Shot+2014-09-08+at+11.15.21+AM.png" /></a></div>
<div>
Selecting <i>Class List </i>will take you to a page where you can interactively edit the class list, as well as downloading it.</div>
<div>
<br /></div>
<div>
If you need to manually copy the class list from one assignment to another, you can first download it, then cut and paste it into the class list of the other assignment:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisfMMP6Wq4obyAHoxCmJTvtauzy_3m0gMJf2V0pXMFdUZqmZTVUXydTnoFahpiasjWi9tUkYLrMMH18DvVUZTmgKS2NxsxGK5AFBVDcM0m3fLCVn2XPynGM59PVIFEz_8QEKxHMaMMOic/s1600/Screen+Shot+2014-09-08+at+11.18.21+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisfMMP6Wq4obyAHoxCmJTvtauzy_3m0gMJf2V0pXMFdUZqmZTVUXydTnoFahpiasjWi9tUkYLrMMH18DvVUZTmgKS2NxsxGK5AFBVDcM0m3fLCVn2XPynGM59PVIFEz_8QEKxHMaMMOic/s1600/Screen+Shot+2014-09-08+at+11.18.21+AM.png" height="118" width="320" /></a></div>
<div>
If you need to do this frequently, let us know, and we may provide some better import options.</div>
<div>
<br /></div>
<div>
As always, if you had any feedback, we would love to hear from you. You can contact us at info@crowdgrader.org</div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-54126979941528197772014-09-02T12:38:00.000-07:002014-09-02T13:20:32.116-07:00CrowdGrader now lets reviewers and authors discuss submissionsWe have just added a new feature to CrowdGrader: authors and reviewers can discuss the submissions anonymously. Attached to each submission is a discussion forum, where author and reviewers alike can post messages and discuss, in anonymous fashion. The discussion forum is visible only to the submission author and reviewers, and to the instructor.<br />
<br />
<div>
In our experience as teachers, perhaps the most common cause of errors in grading is misunderstandings between the student who submitted a solution, and the students who grade it. Many of these misunderstandings could be clarified quite easily if reviewers and authors could communicate. Some typical examples: </div>
<div>
<ul>
<li><i>I cannot find the file solution.txt in the .zip you uploaded, how did you name it?</i></li>
<li><i>You have two versions of the solutions; which one am I supposed to grade? </i></li>
<li><i>Is your Android app designed for tablets or phones? </i></li>
<li><i>How do I install your app?</i></li>
<li><i>I was not able to compile your program. What command line did you use, precisely?</i></li>
</ul>
<div>
We hope that, by letting authors and reviewers communicate anonymously, we will help avoid misunderstandings, and we will give authors and reviewers a better educational experience.<br />
<br />
Whenever a new message is added, the students involved in the review are notified via email; reminders also appear on the <a href="https://www.crowdgrader.org/" target="_blank">CrowdGrader</a> site. To prevent conversations from dragging on too long, the forums are active only during the review period: after the review period closes, the messages can be read, but no new messages can be posted. </div>
</div>
<div>
<br /></div>
<div>
For the moment, this feature is available only if you select it. To do so, edit an assignment, and check the option:</div>
<div>
<span id="goog_726577441"></span><span id="goog_726577442"></span><br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiIKmPcBcxNOJ2qk63suRoLJqWh7cGyZGc3sljZkyNO-wCYF5hMTspTt0TWENSY5ktCXyvD7tSZ9pe1_gSZLJk4Vc6lJAO5VGYSHKHbxLy8qb83ck438f1Efz0rAb6Xv_en1OzxOgB9JL4/s1600/Screen+Shot+2014-09-02+at+11.48.59+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiIKmPcBcxNOJ2qk63suRoLJqWh7cGyZGc3sljZkyNO-wCYF5hMTspTt0TWENSY5ktCXyvD7tSZ9pe1_gSZLJk4Vc6lJAO5VGYSHKHbxLy8qb83ck438f1Efz0rAb6Xv_en1OzxOgB9JL4/s1600/Screen+Shot+2014-09-02+at+11.48.59+AM.png" height="31" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<h2>
<span style="font-size: large; font-weight: normal;">How does it look?</span></h2>
</div>
<div>
Suppose a reviewer has a question: </div>
<div>
<br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIyUlgEDEFb7EFU7LCNBSGiIdwWUxqMt1kdJHhyphenhyphenOKbQjAYEwcocWO-xgLRfq-Fgg0mTD_WF7CaD3qPN49jK4kWzxKQZc4UeKvkTdCkXnxxbc_Nn16Dw37EkzGFtZqLBVVZ5DmKLAwR2e8/s1600/Screen+Shot+2014-09-02+at+12.04.04+PM.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIyUlgEDEFb7EFU7LCNBSGiIdwWUxqMt1kdJHhyphenhyphenOKbQjAYEwcocWO-xgLRfq-Fgg0mTD_WF7CaD3qPN49jK4kWzxKQZc4UeKvkTdCkXnxxbc_Nn16Dw37EkzGFtZqLBVVZ5DmKLAwR2e8/s1600/Screen+Shot+2014-09-02+at+12.04.04+PM.png" height="57" width="640" /></a></div>
<br />
The author can view the question, and reply:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4sMZDPp21sYmc_dk4C-78oWcYGDzJSksxyX7Mc4M5JC7UzbhPC2QhS_HOe_Rcw1x2zjYlAIqdSjH6wX6fDul7y2NepkPP9zXwt8rczjjeB7sCyE-qRF4_WKTw4NWEy6ZX0MW-YEprUl4/s1600/Screen+Shot+2014-09-02+at+12.06.02+PM.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4sMZDPp21sYmc_dk4C-78oWcYGDzJSksxyX7Mc4M5JC7UzbhPC2QhS_HOe_Rcw1x2zjYlAIqdSjH6wX6fDul7y2NepkPP9zXwt8rczjjeB7sCyE-qRF4_WKTw4NWEy6ZX0MW-YEprUl4/s1600/Screen+Shot+2014-09-02+at+12.06.02+PM.png" height="86" width="640" /></a></div>
The reviewer then thanks the author. When another reviewer looks at the submission, the new reviewer is notified of the ongoing discussion:<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWOrV2YFIh7CbR_93hu3DtIb3WUFanJHwHyCkvdsoWy9z4dA7OztM8sLe-5tgntIG4SY-ZrOTJSr6umrpPbIfBjhiEu5CQGEy1kbWdqA42aXUWhbZucOVxznEJZCsAL0_-_SW2ewuOllw/s1600/Screen+Shot+2014-09-02+at+12.10.16+PM.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWOrV2YFIh7CbR_93hu3DtIb3WUFanJHwHyCkvdsoWy9z4dA7OztM8sLe-5tgntIG4SY-ZrOTJSr6umrpPbIfBjhiEu5CQGEy1kbWdqA42aXUWhbZucOVxznEJZCsAL0_-_SW2ewuOllw/s1600/Screen+Shot+2014-09-02+at+12.10.16+PM.png" height="108" width="640" /></a></div>
And can contribute in turn:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYtgOEcImIHyGtqccuJD-OAlSPLcRO4laaT-GbTG7psgrCfPr4SSb7qhAb8AygghFra8EZ5NYtXjyOE8Lflr0tSrl80Sse1r-P6X7lEBtLja3-StrrD8gkb77wP6koIuHGRz_N4_-3OCA/s1600/Screen+Shot+2014-09-02+at+12.07.42+PM.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYtgOEcImIHyGtqccuJD-OAlSPLcRO4laaT-GbTG7psgrCfPr4SSb7qhAb8AygghFra8EZ5NYtXjyOE8Lflr0tSrl80Sse1r-P6X7lEBtLja3-StrrD8gkb77wP6koIuHGRz_N4_-3OCA/s1600/Screen+Shot+2014-09-02+at+12.07.42+PM.png" height="140" width="640" /></a></div>
<br />
This is how the same discussion appears to the author of the submission:<br />
<br />
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirK7Gkf8FeYpRoWzthflV3tuhzeytzq8qqhTy_6Jm23fXQO0umABt_7RSpPx60tkxQpSxXD6rYtEv104eG0colWWQoGarNg6tLjc-5AtXyN5p6RgxpOgnl5jeRxJP2_lavK-a1-ULpNXE/s1600/Screen+Shot+2014-09-02+at+12.08.13+PM.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirK7Gkf8FeYpRoWzthflV3tuhzeytzq8qqhTy_6Jm23fXQO0umABt_7RSpPx60tkxQpSxXD6rYtEv104eG0colWWQoGarNg6tLjc-5AtXyN5p6RgxpOgnl5jeRxJP2_lavK-a1-ULpNXE/s1600/Screen+Shot+2014-09-02+at+12.08.13+PM.png" height="143" width="640" /></a></div>
<div class="separator" style="clear: both;">
We hope that you find this useful. In the future, we might make submission discussions enabled by default. </div>
</div>
Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-17652236332705256952014-03-26T15:35:00.001-07:002014-03-31T09:24:37.135-07:00Helpful reviews outnumber unhelpful ones by over 10:1!In CrowdGrader, helpful reviews outnumber unhelpful ones by over 10:1, as shown in the table below.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi90nGFsS0EjEm3492V4RJj5hLQi17pbvzWcIOdA-ZxsBpgZsW8cd2OQCgMOiSr-XeLK4qkLaR1aF7JQW4-6i_E4LXM9c-tRz0WCRLq_t7LJMxNMRScZ-rNq7rHtBfSBjynjeW0pSUE52U/s1600/Screen+Shot+2014-03-26+at+3.20.57+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi90nGFsS0EjEm3492V4RJj5hLQi17pbvzWcIOdA-ZxsBpgZsW8cd2OQCgMOiSr-XeLK4qkLaR1aF7JQW4-6i_E4LXM9c-tRz0WCRLq_t7LJMxNMRScZ-rNq7rHtBfSBjynjeW0pSUE52U/s1600/Screen+Shot+2014-03-26+at+3.20.57+PM.png" /></a></div>
<br />
Once the review phase is over in CrowdGrader, students can leave feedback on the reviews they receive, ranging from -2 (incorrect, bogus) to +2 (very helpful), with 0 being the neutral feedback (the default when comments on the reviews are left). As the chart above shows, helpful reviews outnumber unhelpful ones by over a 10:1 factor.<br />
<br />
The chart is based on all review feedback collected so far on CrowdGrader. Overall, students have given feedback for approximately 40% of the reviews they received. There are multiple reasons why students do not leave feedback for all the reviews:<br />
<br />
<ul>
<li>Initially, we used a sub-optimal UI that was not very effective in eliciting feedback.</li>
<li>Some instructors do not emphasize the feedback phase. </li>
<li>Many students, who may feel happy with their reviews, felt no special need to give feedback. Again, we have recently improved the UI and the incentive scheme to increase the amount of feedback. </li>
</ul>
<div>
From our limited manual inspection of feedback, it seems that much of the negative feedback is left when students are disappointed at the low grades received, more than as a consequence of poor reviews -- in other words, students who do not like the message often "shoot" the messenger with negative feedback. Fortunately the algorithms in CrowdGrader prevent the "messengers" from suffering in terms of the grade they receive (and they also protect students from getting low grades due to unfair reviews).</div>
<div>
<br /></div>
<div>
Overall, we are very satisfied with these results, by a factor of 10!</div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-14516028131147126952014-03-23T15:48:00.002-07:002014-03-23T15:51:16.537-07:00Students can now compare their reviews with those of othersOnce the review phase of an assignment is concluded, students can now compare the reviews they wrote with the reviews other students wrote for the same assignments.<br />
<br />
By comparing their reviews with those by others, students can discover any reviewing errors they might have made, or any aspects of the submissions they might have overlooked. Students also learn by seeing which advice and feedback other students gave in the same circumstances.<br />
<br />
Here are two screenshots (of a fake assignment we use for testing), showing how this feature looks like.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgiVLuQTL1t44mvO15MV779jnH9LvuWL8gKUCRSgZGePyk9q6dbHfUYbYD0I3Ahd8U8CqU3xezv-sihwYIt-m3KDTEu1cRLqJUgTRjwGyWCfb4cmP-dqYf6jfy2EfMth2A-fe2WH-ba0vo/s1600/Screenshot+from+2014-03-23+12-17-16.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgiVLuQTL1t44mvO15MV779jnH9LvuWL8gKUCRSgZGePyk9q6dbHfUYbYD0I3Ahd8U8CqU3xezv-sihwYIt-m3KDTEu1cRLqJUgTRjwGyWCfb4cmP-dqYf6jfy2EfMth2A-fe2WH-ba0vo/s1600/Screenshot+from+2014-03-23+12-17-16.png" height="160" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Click to enlarge</td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAMSNu5lZ5Wegua7Qp2adYJuHUOUh0_tqawSoZ9pF9RV2oiXcoOHc7GrTToGhuk-x8SngCow5u7ublWlJbmdOK_ub7BRedqyRb7phgB9Dg_bSsM1Mddsb1Y3YI7Lt3WTctrtTk_sD1NLg/s1600/Screenshot+from+2014-03-23+12-17-50.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAMSNu5lZ5Wegua7Qp2adYJuHUOUh0_tqawSoZ9pF9RV2oiXcoOHc7GrTToGhuk-x8SngCow5u7ublWlJbmdOK_ub7BRedqyRb7phgB9Dg_bSsM1Mddsb1Y3YI7Lt3WTctrtTk_sD1NLg/s1600/Screenshot+from+2014-03-23+12-17-50.png" height="235" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Click to enlarge</td></tr>
</tbody></table>
We are considering introducing an option that lets students view the reviews written by others on the same submissions also during the review process, so that students can compare their observations with those of others, and come up with more balanced reviews that reflect a broader consensus. This would make the review phase a collaborative phase with many discussions (not unlike the workings of a conference program committee). This would be an option, so that instructors would turn it on or off depending on their wishes.<br />
<br />
Students would see reviews written by others only once they turn in a review themselves, and we plan to store the initial review and grade. This should act as a disincentive for turning in initially a blank review, and using the comments written by others to come up with the final review. Other disincentives towards such lazy behavior could be built into our statistical analysis tool that we use for assigning grades.<br />
<br />
We would love to hear your thoughts on this.Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-25657916607352151322014-03-16T20:00:00.000-07:002014-03-23T16:39:09.752-07:00CrowdGrader via HTTPS: Secure Your Connection<br />
<h2 style="text-align: center;">
<a href="https://www.crowdgrader.org/"><span style="font-family: Arial, Helvetica, sans-serif; font-size: large; font-weight: normal;">https://www.crowdgrader.org</span></a></h2>
<br />
<span style="font-family: inherit;">You can now access CrowdGrader over HTTPS!</span><br />
<span style="font-family: inherit;"><br />
We encourage everybody to access CrowdGrader over HTTPS, rather than HTTP, to protect your data and your identity. The benefits of HTTPS are:</span><br />
<ul>
<li><span style="font-family: inherit;">Your data is protected. Homework solutions and reviews are transmitted encrypted to CrowdGrader, secure from prying eyes.</span></li>
<li><span style="font-family: inherit;">Nobody can impersonate you and steal your access credentials. When you access CrowdGrader, you authenticate to the website via a cookie. If you connect over an insecure network (such as an open WiFi access point, not protected by password), other people over the same network can steal your cookie and impersonate you. By connecting to CrowdGrader over HTTP, your communication is encrypted, and nobody can steal your cookie.</span></li>
</ul>
<span style="font-family: inherit;">For the time being, we are still allowing unencrypted connections over HTTP, but we are considering making the use of HTTPS compulsory. </span><br />
<br />Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0tag:blogger.com,1999:blog-4596398250258143420.post-84311998731505059952014-01-14T12:00:00.000-08:002014-03-24T10:05:57.427-07:00Download all assignment data!Assignment managers (instructors and TAs) can now download all assignment data!<br />
<br />
There are two steps required.<br />
<br />
First, you need to download a Json file that contains information on the assignment. You can do so by going to the assignment, and selecting <i>Download Assignment Data. </i><br />
<i><br /></i>
Second, you need to get the <a href="https://github.com/lucadealfaro/crowdgrader-download-assignment">Python CrowdGrader assignment downloader</a> from GitHub.<br />
<br />
You can then run the downloader via:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">./download_assignment.py <assignment_file.json> <destination_directory></span><br />
<br />
This will produce a directory in <i><destination_directory></i>, and it will create one sub-directory for each submission. In the sub-directory for each submission, you will find the submission metadata, the submission text, the files that were submitted as attachment, and the reviews. The format should be self-explanatory.<br />
<br />
The reason you have to go through this two-step process is that an assignment can consist of a very large amount of data (up to several GB for some assignments, depending on the files that were submitted as attachments). Therefore, downloading an entire assignment can take a long time, and cannot be performed as a single download.<br />
<br />
The downloader code is supplied with a MIT license, so please feel free to improve it and to share back your improvements.<br />
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/04149574714536748924noreply@blogger.com0