The reliability of listeners’ ratings of voice quality is a central issue in voice research because of the clinical primacy of such ratings and because they are the standard against which other measures are evaluated. However, an extensive literature review indicates that both intrarater and interrater reliability fluctuate greatly from study to study. Further, our own data indicate that ratings of vocal roughness vary widely across individual clinicians, with a single voice often receiving nearly the full range of possible ratings. No model or theoretical framework currently exists to explain these variations, although such a model might guide development of efficient, valid, and standardized clinical protocols for voice evaluation. We propose a theoretical framework that attributes variability in ratings to several sources (including listeners’ backgrounds and biases, the task used to gather ratings, interactions between listeners and tasks, and random error). This framework may guide development of new clinical voice and speech evaluation protocols, ultimately leading to more reliable perceptual ratings and a better understanding of the perceptual qualities of pathological voices.