in any case, any such adjustment should be based on an explicit model of how chance and error affect raters' decisions.

When the number of categories being used is small (e.g. 2 or 3), the likelihood for 2 raters to agree by pure chance increases dramatically. This is because both raters must confine themselves to the limited number of options available, which impacts the overall agreement rate, and not necessarily their propensity for "intrinsic" agreement (an agreement is considered "intrinsic" if it is not due to chance).

Therefore,

1.Previous
3.Next