Advertisement

The Reliability of Scores from Affective Instruments

  • D. Betsy McCoachEmail author
  • Robert K. Gable
  • John P. Madura
Chapter

Abstract

In psychometrics, reliability is used to describe the consistency of a measure. Reliability can be understood in the terms when describing its meaning in the instrument development process. The goal of reliability becomes the determination of how much variability in the instrument score is due to “measurement error” and how much is due to variability in “true score” of the respondent. This chapter approaches reliability from the classical test theory (CTT) model. Most importantly in this chapter, however, the focus is on the concept of correlation and how to index it for purpose of understanding the “reliability” or consistency of the instrument under the CTT framework. This chapter discusses a variety of ways that reliability can be understood and how each represent distinct quantifications of consistency. Through several examples, different types of reliability are illustrated along with “acceptable” levels of consistency that should be expected from their measurement of characteristics in the affective domain. One of the central limitations of true score theory is that only one type of error can be addressed at a time. This chapter provides brief introduction to the conceptual framework of Generalizability theory (G-theory), which provides one potential solution to this issue. The chapter concludes with a discussion that links validity and reliability, the two central concepts that frame all the work behind the development of instruments for measuring affective characteristics.

Keywords

Reliability Classical true score theory True score variance Error variance Reliability index Reliability coefficient Average inter-item correlation Standard deviation of the inter-item correlation Unidimensionality Congeneric Generalizability theory Items Occasions Internal consistency reliability Split-half reliability Cronbach’s alpha Reliability statistics Correlations Stability reliability Item-total (scale) correlation 

References

  1. American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (1999). The standards for educational and psychological testing. Washington, DC: American Educational Research Association.Google Scholar
  2. Brennan, R. L. (1992). Generalizability theory. Educational Measurement: Issues and Practice, 11(4), 27–34.CrossRefGoogle Scholar
  3. Brennan, R. L. (2001). Generalizability theory. New York, NY: Springer.Google Scholar
  4. Cardinet, J., Johnson, S., & Pini, G. (2010). Applying generalizability theory using EduG. New York, NY: Taylor and Francis.Google Scholar
  5. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334.CrossRefGoogle Scholar
  6. DeVellis, R. F. (1991). Scale development: Theory and applications. Newbury Park, CA: Sage.Google Scholar
  7. Isaac, S., & Michael, W. B. (1981). Handbook in research and evaluation (2nd ed.). San Diego, CA: Edits publishers.Google Scholar
  8. Kerlinger, F. N. (1973). Foundations of Behavioral Research (2nd ed.). New York: Holt, Rinehart and Winston.Google Scholar
  9. Loehlin, J. C. (2004). Latent variable models: An introduction to factor, path and structural analysis (4th ed.). Hillsdale: Erlbaum.Google Scholar
  10. McCoach, D. B. (2002). A validity study of the School Attitude Assessment Survey (SAAS). Measurement and Evaluation in Counseling and Development, 35, 66–77.Google Scholar
  11. Meyer, P. (2010). Reliability: Understanding statistics measurement. New York: Oxford University Press.CrossRefGoogle Scholar
  12. Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill.Google Scholar
  13. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.Google Scholar
  14. Pett, M. A., Lackey, N. R., & Sullivan, J. J. (2003). Making sense of factor analysis: The use of factor analysis for instrument development in health care research. Thousand Oaks, CA: Sage.Google Scholar
  15. Raykov, T., & Marcoulides, G. A. (2011). Introduction to psychometric theory. New York: Taylor and Francis Group.Google Scholar
  16. Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory. Newbury Park, CA: Sage.Google Scholar
  17. Shavelson, R. J., Webb, N. M., & Rowley, G. L. (1989). Generalizability theory. American Psychologist, 44(6), 922–932.CrossRefGoogle Scholar
  18. Stanley, J. C. (1971). Reliability. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 356–442). Washington, DC: American Council on Education.Google Scholar
  19. Thompson, B. (2002). Score reliability: Contemporary thinking on reliability issues. Thousand Oaks, California: Sage Publications.Google Scholar
  20. Webb, N. M., Rowley, G. L., & Shavelson, R. J. (1988). Using generalizability theory in counseling and development. Measurement and Evaluation in Counseling and Development, 21, 81–90.Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • D. Betsy McCoach
    • 1
    Email author
  • Robert K. Gable
    • 2
  • John P. Madura
    • 3
  1. 1.Educational Psychology DepartmentUniversity of ConnecticutStorrsUSA
  2. 2.Alan Shawn Feinstein Graduate SchoolJohnson and Wales UniversityStorrsUSA
  3. 3.Department of Educational PsychologyUniversity of ConnecticutStorrsUSA

Personalised recommendations