Exploring Novice Raters’ Textual Considerations in Independent and Negotiated Ratings


  • Alireza Ahmadi Department of Foreign Languages and Linguistics, Shiraz University

Educators often employ various training techniques to reduce raters’ subjectivity. Negotiation is a technique which can assist novice raters to co-construct a shared understanding of the writing assessment when rating collaboratively. There is little research, however, on rating behaviors of novice raters while employing negotiation techniques and the effect of negotiation on their understanding of writing and rubric features. This study uses a qualitative method to keep track of 11 novice raters’ scoring behaviors and examine their textual foci during three phases of scoring through an analytic rubric: pre-negotiation, negotiation, and post-negotiation. To ensure triangulation, multiple sources of data including raters’ verbal protocols of independent scoring during the initial and final phases, audio-recorded interactions in the negotiation phase, and semi-structured interviews were gathered and analyzed. Results indicated that in their initial independent rating, raters mostly scored based on their understanding of the writing skill and the writing features that were important to them, but negotiation sessions aided them to refine their judgments and attend to a wider array of textual features more consistently and in line with the rubric, thereby expanding their understanding of the rubric categories. Post-negotiation ratings were also more similar to negotiation than prenegotiation ratings, meaning that the raters attended to more features of the rubric for scoring. The findings may have implications for rater training. In the absence of expert raters to train novice raters, negotiation can be considered a useful technique to improve raters’ understanding of the rubric features.

برای دانلود باید عضویت طلایی داشته باشید

برای دسترسی به متن کامل این مقاله و 23 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

User Ratings of Ontologies: Who Will Rate the Raters?

The number of ontologies and knowledge bases covering different domains and available on the World-Wide Web is steadily growing. As more ontologies are available, it is becoming harder, and not easier, for users to find ontologies they need. How do they evaluate if a particular ontology is appropriate for their task? How do they choose among many ontologies for the same domain? We argue that al...

متن کامل

Raters who pursue different goals give different ratings.

J. N. Cleveland and K. R. Murphy (1992) suggested that phenomena such as rater errors and interrater disagreements could be understood in terms of differences in the goals pursued by various raters. We measured 19 rating goals of students at the beginning of a semester, grouped them into scales, and correlated these with teacher evaluations collected at the end of the semester. We found signifi...

متن کامل

: metadiscourse in introduction sections of applied linguistics and physics research articles: exploring variation in frequency and type

abstract in written mode of language, metadiscourse markers are used commonly to help writers in general and academic writers in particular to produce coherent and professional texts. the purpose of the present study was to compare introduction sections of applied linguistics and physics articles regarding their use of interactive and interactional metadiscourse markers based on the model pro...

15 صفحه اول

Rating Communication in GP Consultations: The Association Between Ratings Made by Patients and Trained Clinical Raters

Patient evaluations of physician communication are widely used, but we know little about how these relate to professionally agreed norms of communication quality. We report an investigation into the association between patient assessments of communication quality and an observer-rated measure of communication competence. Consent was obtained to video record consultations with Family Practitione...

متن کامل

Exploring Novice User Considerations for a Mobilde Device Application Through Card Sorting

Mobile technologies applied through a variation of devices ranging from smart phones, netbooks and tablets, are developing quite fast and are expanding continuously. Users and media consumers interact increasingly nowadays which such devices, whose inbuilt functions and services are continuously developing and advancing. Based on this fact, this paper deals with the process, analysis, results a...

متن کامل

Inter-rater reliability of ultrasound imaging of the trunk musculature among novice raters.

OBJECTIVE The purpose of this study was to determine the inter-rater reliability of ultrasound imaging for assessing trunk muscle morphologic characteristics at rest and while contracted among different pairs of novice raters. The secondary purpose was to compare 3 different measurement techniques for assessing lateral abdominal muscle thickness. METHODS A single-group repeated measures relia...

متن کامل

ذخیره در منابع من

ذخیره در منابع من قبلا به منابع من ذحیره شده

{@ msg_add @}

  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

دانلود متن کامل

برای دسترسی به متن کامل این مقاله و 23 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

عنوان ژورنال:

دوره 39  شماره 2 (Summer 2020)

صفحات  43- 87

تاریخ انتشار 2020-08-01

با دنبال کردن یک ژورنال هنگامی که شماره جدید این ژورنال منتشر می شود به شما از طریق ایمیل اطلاع داده می شود.

میزبانی شده توسط پلتفرم ابری doprax.com

copyright © 2015-2022