Observer Agreement: Cumulative Effects of Repeated Ratings of the Same Samples and of Knowledge of Group Results The validity of measurements derived from the rating by observers of nonmetric multidimensional stimuli—in this study ratings of stuttering severity—is determined by the amount of observer agreement. The effects on observer agreement of two experimental conditions thought to increase agreement were examined: repeated rating of the same samples, and feedback ... Research Article
Research Article  |   March 01, 1969
Observer Agreement: Cumulative Effects of Repeated Ratings of the Same Samples and of Knowledge of Group Results
 
Author Affiliations & Notes
  • Martin A. Young
    Illinois State University, Normal, Illinois
Article Information
Research Articles
Research Article   |   March 01, 1969
Observer Agreement: Cumulative Effects of Repeated Ratings of the Same Samples and of Knowledge of Group Results
Journal of Speech, Language, and Hearing Research, March 1969, Vol. 12, 144-155. doi:10.1044/jshr.1201.144
History: Received July 17, 1968
 
Journal of Speech, Language, and Hearing Research, March 1969, Vol. 12, 144-155. doi:10.1044/jshr.1201.144
History: Received July 17, 1968

The validity of measurements derived from the rating by observers of nonmetric multidimensional stimuli—in this study ratings of stuttering severity—is determined by the amount of observer agreement. The effects on observer agreement of two experimental conditions thought to increase agreement were examined: repeated rating of the same samples, and feedback to the observer of the group mean rating after each sample had been rated. Repeated rating of the same samples did not affect observer agreement. Feedback of the group mean rating did produce a shift in overall level of rating in those observers whose ratings were, on the average, far from the group mean. Another effect was a significant decrease in the range of ratings on individual samples. A replication with another group of observers yielded identical findings. Although within- and between-group scale value reliability was uniformly high, it is believed that observer agreement could be increased even more.

Order a Subscription
Pay Per View
Entire Journal of Speech, Language, and Hearing Research content & archive
24-hour access
This Article
24-hour access