Testing the Significance of the Agreement Among Observers Ratings by observers are often used in speech pathology to measure complex speech dimensions; this seems reasonable since a speech “disorder” represents the product of an observer’s evaluation and a speaker’s performance. An index of the validity of these evaluations may be estimated by the amount of agreement among the ... Research Article
Research Article  |   March 01, 1968
Testing the Significance of the Agreement Among Observers
 
Author Notes
Article Information
Speech, Voice & Prosodic Disorders / Research Issues, Methods & Evidence-Based Practice / Research Articles
Research Article   |   March 01, 1968
Testing the Significance of the Agreement Among Observers
Journal of Speech, Language, and Hearing Research, March 1968, Vol. 11, 5-17. doi:10.1044/jshr.1101.05
History: Received June 1, 1967
 
Journal of Speech, Language, and Hearing Research, March 1968, Vol. 11, 5-17. doi:10.1044/jshr.1101.05
History: Received June 1, 1967

Ratings by observers are often used in speech pathology to measure complex speech dimensions; this seems reasonable since a speech “disorder” represents the product of an observer’s evaluation and a speaker’s performance. An index of the validity of these evaluations may be estimated by the amount of agreement among the observers. In this paper, the semi-interquartile range and the intraclass correlation are discussed as possible indices of agreement, and another index is suggested, based on the range of observer ratings. Under the assumption that the distribution of ratings is uniform when ratings are randomly assigned, that is, the observers show no agreement, tables were constructed to indicate the probability of any range for selected numbers of observers and rating scale categories. Some applications for this index concern the training of observers, estimating the number of observers needed, and the construction of master scales.

Order a Subscription
Pay Per View
Entire Journal of Speech, Language, and Hearing Research content & archive
24-hour access
This Article
24-hour access