Reliability Issues and Solutions for Coding Social Communication Performance in Classroom Settings Purpose To explore the utility of time-interval analysis for documenting the reliability of coding social communication performance of children in classroom settings. Of particular interest was finding a method for determining whether independent observers could reliably judge both occurrence and duration of ongoing behavioral dimensions for describing social communication performance. ... Research Article
Research Article  |   November 14, 2016
Reliability Issues and Solutions for Coding Social Communication Performance in Classroom Settings
 
Author Affiliations & Notes
  • Lesley B. Olswang
    University of Washington, Seattle
  • Liselotte Svensson
    University of Washington, Seattle
  • Truman E. Coggins
    University of Washington, Seattle
  • Jill S. Beilinson
    University of Washington, Seattle
  • Amy L. Donaldson
    University of Washington, Seattle
  • Contact author: Lesley B. Olswang, Department of Speech and Hearing Sciences, University of Washington, 1417 Northeast 42nd Street, Seattle, WA 98105-6246l. E-mail: lolswang@u.washington.edu
Article Information
School-Based Settings / Research Issues, Methods & Evidence-Based Practice / Language Disorders / Social Communication & Pragmatics Disorders / Language / Research Articles
Research Article   |   November 14, 2016
Reliability Issues and Solutions for Coding Social Communication Performance in Classroom Settings
Journal of Speech, Language, and Hearing Research, November 2016, Vol. 49, 1058-1071. doi:10.1044/1092-4388(2006/075)
History: Received August 10, 2005 , Revised December 14, 2005 , Accepted February 25, 2006
 
Journal of Speech, Language, and Hearing Research, November 2016, Vol. 49, 1058-1071. doi:10.1044/1092-4388(2006/075)
History: Received August 10, 2005; Revised December 14, 2005; Accepted February 25, 2006

Purpose To explore the utility of time-interval analysis for documenting the reliability of coding social communication performance of children in classroom settings. Of particular interest was finding a method for determining whether independent observers could reliably judge both occurrence and duration of ongoing behavioral dimensions for describing social communication performance.

Method Four coders participated in this study. They observed and independently coded 6 social communication behavioral dimensions using handheld computers. The dimensions were mutually exclusive and accounted for all verbal and nonverbal productions during a specified time frame. The technology allowed for coding frequency and duration for each entered code. Data were collected from 20 different 2-min video segments of children in kindergarten through 3rd-grade classrooms. Data were analyzed for interobserver and intraobserver agreements using time-interval sorting and Cohen’s kappa. Further, interval size and total observation length were manipulated to determine their influence on reliability.

Results The data revealed interval sorting and kappa to be a suitable method for examining reliability of occurrence and duration of ongoing social communication behavioral dimensions. Nearly all comparisons yielded medium to large kappa values; interval size and length of observation minimally affected results.

Implications The analysis procedure described in this research solves a challenge in reliability: comparing coding by independent observers of both occurrence and duration of behaviors. Results indicate the utility of a new coding taxonomy and technology for application in online observations of social communication in a classroom setting.

Acknowledgments
This research was supported in part by the University of Washington Tools for Transformation Award; Centers for Disease Control Grant No. UA/CCU010163, “Intervening With Children/Adolescents With FAS/ARND”; and the University of Washington Royalty Research Fund and Department of Speech and Hearing Sciences. We also wish to acknowledge Roger Ingham and Anne Bothe for suggestions regarding time-interval sorting and Barbara Bain, Janet Norris, and Ann Mastergeorge for videotapes of children in classrooms.
Order a Subscription
Pay Per View
Entire Journal of Speech, Language, and Hearing Research content & archive
24-hour access
This Article
24-hour access