During the review phase, an instructor can use the Detailed Stats Page to gather some basic information about how the assignment is proceeding. One of those stats is Conflicting Rating Documents.
You can find this page by clicking the Stats tab on the left side navigation bar of the main assignment dashboard. Click on the assignment name and then View Detailed Stats. From there you will see the Conflicting Rating Documents section.
The reason a document gets populated in the Conflicting Rating Documents section is because at least one of the rating prompts on that document has a standard deviation of greater than 2.5. When making decisions on a document in this section, the action you decide to take will be influenced by how far along the assignment is in the process.
If an assignment is still in the review phase...
The first thing to do in this situation is to take a look at the document and determine why there is such a high standard deviation. If you discover that one student's review is causing this because of extreme inaccuracy, you might consider reaching out to the inaccurate student and ask them to take another look at their review to see if they want to modify any ratings. They will not be able to see the other reviewers' ratings during the review phase, so they will not be influenced by that. In this case, maybe they take another look and decide they were overly generous/critical on a particular rating and make the change themselves.
Another alternative is to add a teacher review on this document. You can do this during the review phase or after the deadline has passed. See the description below for more details.
If the review deadline has passed...
The other alternative is to add a teacher review on this document. You can do this during the review phase or after the deadline has passed. When adding a teacher review, the system always interprets your review to be 100% accurate. It will adjust the Accuracy Weighting of the student reviewers on this document accordingly. This will result in adding value to the more accurate review while decreasing the value of the inaccurate review.
The other benefit of adding a teacher review is that once the assignment is complete, reviewers can go back and compare their ratings to the teacher ratings on the documents they reviewed. They can use this information to adjust their rating behavior on future assignments so that their accuracy score improves.
Keep in mind that the weight of the teacher review is a setting that you can adjust when creating an assignment. A teacher review can be one in the set of reviews, be equal to a certain percentage of the review, or override the other reviews entirely. The default "One in the set" makes an instructor review equal to student reviews.
Alternatively, you could decide not to take any action. The system is designed in a way to value accurate reviewers' ratings more highly. These accuracy ratings are determined by an individual's rating behavior on multiple documents compared to multiple other reviewers. If one of the reviewers was inaccurate, the system will adjust for this behavior. If all of the ratings were significantly different (no agreement whatsoever), there would be a more pressing need to take action.