Evidence Based on Relations to Other Variables: Bolstering the Empirical Validity Arguments for Constructs

  • D. Betsy McCoachEmail author
  • Robert K. Gable
  • John P. Madura


In this chapter, the concept of validity is examined using evidence based on the relation of constructs within the instrument to constructs that are external to the instrument. This chapter addresses two major categories of validity evidence based on these external relationships. The first is what has historically been referred to as construct validity, which includes analyses of convergent and divergent validity. This first part of the chapter is dedicated to discussing the methodological framework (correlations and multitrait-multimethod matrices) and statistical techniques [structural equation modeling (SEM)] needed to quantify these relationships. The second half of the chapter discusses what has commonly been referred to as criterion validity and includes evidence with external variables that is often predictive in nature. The final section discusses the complex tasks of gathering incremental validity evidence and gathering evidence for use of the instrument with other populations.


Correlation Convergent Discriminant Structural equation modeling Path diagrams Causation Concurrent validity Known group analysis Discriminant function analysis Incremental vailidity Multitrait-multimethod matrix (mtmm) Monotrait monomethod Heterotrait monomethod Monotrait-heteromethod Heterotrait-heteromethod nomological net Disturbance Identification Exogenous Endogenous Knowns/unknowns Overidentified Underidentified Degrees of freedom Parameters Inadmissible solution Heywood case Lack of convergence Normality Linearity Sampling Criterion relationships Criterion related validity 


  1. Allport, G. W., Vernon, P. E., & Gardner, L. (1960). Study of values. Oxford, England: Houghton Mifflin.Google Scholar
  2. American Educational Research Association (AERA), American Psychological Association (APA) & National Council on Measurement in Education (NCME). (1999). The standards for educational and psychological testing. Washington: American Educational Research Association.Google Scholar
  3. Anderson, R. E., Barnes, G. E., & Murray, R. P. (2011). Psychometric properties and long-term predictive validity of the Addiction-Prone Personality (APP) scale. Personality and Individual Differences, 50(5), 651–656.CrossRefGoogle Scholar
  4. Baron, R. M., & Kenny, D. A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51(6), 1173–1182.PubMedCrossRefGoogle Scholar
  5. Beck, C. T., & Gable, R. K. (2000). Postpartum depression screening scale: Development and psychometric testing. Nursing Research, 49, 272–282.PubMedCrossRefGoogle Scholar
  6. Beck, C. T., Gable, R. K. (2002). Postpartum depression screening scale. Los Angeles: Western Psychological Services.Google Scholar
  7. Beck, C. T. (1992). The lived experience of postpartum depression: A phenomenological study. Nursing Research, 41, 166–170.PubMedCrossRefGoogle Scholar
  8. Beck, C. T. (1993). Teetering on the edge: A substantive theory of postpartum depression. Nursing Research, 42, 42–48.PubMedCrossRefGoogle Scholar
  9. Beck, C. T. (1995). The effects of postpartum depression on maternal-infant interaction: A meta-analysis. Nursing Research, 44(5), 298–304.PubMedCrossRefGoogle Scholar
  10. Beck, C. T. (1996). Postpartum depressed mothers’ experiences interacting with their children. Nursing Research, 45, 98–104.PubMedCrossRefGoogle Scholar
  11. Beck, C. T., & Gable, R. K. (2001). Further validation of the postpartum depression screening scale. Nursing Research, 50, 155–164.Google Scholar
  12. Beck, A. T., Steer, R. A., & Brown, G. K. (1996). BDI-II manual. San Antonio: The Psychological Corporation.Google Scholar
  13. Bennett, G. K., Seashore, H. G., & Westman, A. G. (1997). The differential aptitude test. San Antonio, Texas: Psychological Corporation.Google Scholar
  14. Bollen, K. A. (1989). Structural equations with latent variables. New York: Wiley.Google Scholar
  15. Borsboom, D. (2005). Measuring the mind: Conceptual issues in contemporary psychometrics. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  16. Boutlton, M. J., & Smith, P. K. (1994). Bully/victim problems in middle school children: Stability, self-perceived competence, peer acceptance. British Journal of Developmental Psychology, 12, 315–325.CrossRefGoogle Scholar
  17. Bovaird, J. A., & Koziol, N. A. (2012). Measurement models for ordered-categorical indicators. In R. Hoyle (Ed.), Handbook of structural equation modeling (pp. 495–511). New York: The Guilford Press.Google Scholar
  18. Campbell, D. P. (1973). The strong vocational interest blank for men. In D. G. Zytowski (Ed.), Contemporary approaches to interest measurement. Minneapolis: University of Minnesota Press.Google Scholar
  19. Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105.PubMedCrossRefGoogle Scholar
  20. Campbell, D. T., & O’Connell, E. J. (1967). Method factors in multitrait-multimethod matrices: Multiplicative rather than additive? Multivariate Behavioral Research, 2, 409–426.CrossRefGoogle Scholar
  21. Carmines, E. G., & Zeller, R. A. (1979). Reliability and validity assessment. Beverly Hills: Sage.Google Scholar
  22. Cox, J. L., Holden, J. M., & Sagovsky, R. (1987). Detection of postnatal depression: Development of the 10-item Edinburgh Postnatal Depression Scale. British Journal of Psychiatry, 150, 782–786.PubMedCrossRefGoogle Scholar
  23. Cronbach, L. J. (1971). Test validation. In R. L. Thorndike (Ed.), Educational measurement (2nd ed.). Washington, DC: American Council on Education.Google Scholar
  24. Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302.PubMedCrossRefGoogle Scholar
  25. Duncan, T. E., Duncan, S. C., Strycker, L. A., Li, F., & Alpert, A. (1999). An introduction to latent variable growth curve modeling: Concepts, issues, and applications. Mahwah: Erlbaum.Google Scholar
  26. Edwards, A. L. (1959). Edwards personal preference schedule manual. New York: Psychological Corp.Google Scholar
  27. Edwards, M. C., Wirth, R. J., Houts, C. R., & Xi, N. (2012). Categorical data in the structural equation modeling framework. In R. Hoyle (Ed.), Handbook of structural equation modeling (pp. 195–208). New York: Guilford Press.Google Scholar
  28. Eid, M., & Nussbeck, F. W. (2009). The multitrait-multimethod matrix at 50! Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 5(3), 71.CrossRefGoogle Scholar
  29. Gable, R. K. (1970). A multivariate study of work value orientations. Unpublished doctoral dissertation, State University of New York at Albany.Google Scholar
  30. Gordon, L. V. (1960). Survey of interpersonal values. Chicago: Science Research Associates.Google Scholar
  31. Grimm, K. J., & Widaman, K. F. (2012). Construct validity. In H. Cooper (Ed.), APA handbook of research methods in psychology. Washington, DC: APA.Google Scholar
  32. Hawker, D. S., & Boulton, M. J. (2000). Twenty years’ research on peer victimization and psychosocial maladjustment: A meta-analytic review of cross-sectional studies. Journal of Child Psychology and Psychiatry, 41, 441–455.PubMedCrossRefGoogle Scholar
  33. Haynes, S. N., & Lench, H. C. (2003). Incremental validity of new clinical assessment measures. Psychological Assessment, 15(4), 456–466.PubMedCrossRefGoogle Scholar
  34. Hovling, V., Schermelleh-Engel, K., & Moosbrugger, H. (2009). Analyzing multitrait-multimethod data: A comparison of three approaches. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 5(3), 99–111.CrossRefGoogle Scholar
  35. Hoyle, R. H. (Ed.). (1995). Structural equation modeling: Concepts, issues, and applications. Thousand Oaks: Sage Publications.Google Scholar
  36. Kaplan, D. (2000). Structural Equation Modeling: Foundations and Extensions. Newbury Park, CA: Sage.Google Scholar
  37. Kaplan, D. (2009). Structural equation modeling: Foundations and extensions (2nd ed.). New York: Sage Publications.Google Scholar
  38. Kenny, D. A. (1976). An empirical application of confirmatory factor analysis to the multitrait-multimethod matrix. Journal of Experimental Social Psychology, 65, 507–516.Google Scholar
  39. Kenny, D. A., & Kashy, D. A. (1992). Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin, 112, 165–172.CrossRefGoogle Scholar
  40. Kenny, D. A., Kashy, D. A., & Bolger, N. (1998). Data analysis in social psychology. In D. T. Gilbert, S. T. Fiske, & G. Lindzey (Eds.), Handbook of social psychology (4th ed., pp. 233–265). New York: McGraw Hill.Google Scholar
  41. Kline, R. B. (2010). Principles and practice of structural equation modeling (3rd ed.). New York: The Guilford Press.Google Scholar
  42. Kuder, G. F. (1949). Manual for the Kuder preference record (personal). Chicago: Science Research Associates.Google Scholar
  43. Lance, C. E., Noble, C. L., & Scullen, S. E. (2002). A critique of the correlated trait-correlated method and correlated uniqueness models for multitrait-multimethod data. Psychological Methods, 7(2), 228–244.PubMedCrossRefGoogle Scholar
  44. Maas, C. J. M., Lensvelt-Mulders, G. J. L. M., & Hox, J. J. (2009). A multilevel multitrait-multimethod analysis. Methodology, 5, 72–77.Google Scholar
  45. Marsh, H. W. (1989). Confirmatory factor analysis of multitrait-multimethod data: Many problems and a few solutions. Applied Psychological Measurement, 12, 335–361.CrossRefGoogle Scholar
  46. Marsh, H. W., & Bailey, M. (1991). Confirmatory factor analysis of multitrait-multimethod data: A comparison of the behavior of alternative models. Applied Psychological Measurement, 15, 47–70.CrossRefGoogle Scholar
  47. Marsh, H. W., Byrne, B. M., & Craven, R. (1992). Overcoming problems in confirmatory factor analysis of MTMM data: The correlated uniqueness model and factorial invariance. Multivariate Behavioral Research, 27, 489–507.CrossRefGoogle Scholar
  48. Marsh, H. W., & Grayson, D. (1995). Latent variable models of multitrait-multimethod data. In R. Hoyle (Ed.), Structural equation modeling (pp. 177–198). Thousand Oaks, CA: Sage.Google Scholar
  49. Marsh, H. W., Nagengast, B., Morin, A. J. S., Parada, R. H., Craven, R. G., & Hamilton, L. R. (2011). Construct validity of the multidimensional structure of bullying and victimization: An application of exploratory structural equation modeling. Journal of Educational Psychology, 103(3), 701–732.CrossRefGoogle Scholar
  50. Marsh, H. W., Wen, Z., Nagengast, B., & Hau, K. (2012). Handbook of structural equation modeling (pp. 436–455). New York: The Guilford Press.Google Scholar
  51. Matarazzo, J. D., Guze, S. B., & Matarazzo, R. G. (1955). An approach to the validity of the Taylor Anxiety Scale: Scores of medical and psychiatric patients. The Journal of Abnormal and Social Psychiatry, 51(2), 276–280.CrossRefGoogle Scholar
  52. McCoach, D. B. (2003). SEM isn’t just the school wide enrichment model anymore: structural equation modeling (SEM) in gifted education. Journal for the Education of the Gifted, 27, 36–61.Google Scholar
  53. McCoach, D. B., & Siegle, D. (2003a). The school attitude assessment survey-revised: A new instrument to identify academically able students who underachieve. Educational and Psychological Measurement, 63(3), 414–429.CrossRefGoogle Scholar
  54. McCoach, D. B., & Siegle, D. (2003b). Factors that differentiate underachieving gifted students from high-achieving gifted students. Gifted Child Quarterly, 47(2), 144–154.CrossRefGoogle Scholar
  55. Muthen, L. K., & Muthen, B. O. (1998–2007). Mplus Users Guide (4th Ed.). Los Angeles: Muthen & Muthen.Google Scholar
  56. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.Google Scholar
  57. Nussbeck, F. W., Eid, M., Geiser, C., Courvoisier, D. S., & Lischetzke, T. (2009). A CTC(M-1) model for different types of raters. Methodology, 5, 88–98.Google Scholar
  58. Oort, F. J. (2009). Three-mode models for multitrait-multimethod data. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 5(3), 78–87.CrossRefGoogle Scholar
  59. Parada, R. (2000). Adolescent Peer Relations Instrument: A theoretical and empirical basis for the measurement of participant roles in bullying and victimization of adolescence: An interim test manual and a research monograph: A test manual. Publication Unit, Self-concept Enhancement and Learning Facilitation (SELF) Research Centre, University of Western Sydney.Google Scholar
  60. Peterson, J. S., & Colangelo, N. (1996). Gifted achievers and underachievers: A comparison of patterns found in school files. Journal of Counseling and Development, 74, 399–406.CrossRefGoogle Scholar
  61. Rabe-Hesketh, S., & Skrondal, A. (2012). Multilevel and longitudinal modeling using stata (3rd edn ). College Station, TX: Stata Press.Google Scholar
  62. Raykov, T., & Marcoulides, G. A. (2011). Introduction to psychometric theory. New York: Routledge.Google Scholar
  63. Raykov, T., & Marcoulides, G. A. (2006). A first course in structural equation modeling (2nd ed.). Mahway: Lawrence Erlbaum Associates, Inc.Google Scholar
  64. Silver, H. A., & Barnette, W. L. (1970). Predictive and concurrent validity of the Minnesota vocational interest inventory. Journal of Applied Psychology, 54(5), 436–440.CrossRefGoogle Scholar
  65. Sireci, S. G. (2006). Content validity. In N. J. Salkind (Ed.) Encyclopedia of measurement and statistics. Thousand Oaks, CA: Sage.Google Scholar
  66. Sireci, S. G., & Parker, P. (2006). Validity on trial: Psychometric and legal conceptualizations of validity. Educational Measurement: Issues and Practice, 25(3), 27–34.CrossRefGoogle Scholar
  67. Schmidt, F. L. (1988). The problem of group differences in ability scores in employment selection. Journal of Vocational Behavior, 33, 272–292.CrossRefGoogle Scholar
  68. Schumacker, R. E., & Lomax, R. G. (1996). A beginner’s guide to structural equation modeling. Mahwah: Lawrence Erlbaum Associates.Google Scholar
  69. Spitzer, R. L., Endicott, J., & Robins, E. (1978). Research diagnostic criteria: Rationale and reliability. Archives of General Psychiatry, 35(6), 773–782.PubMedCrossRefGoogle Scholar
  70. Suldo, S. M., Shaffer, E. J., & Shaunessy, E. (2008). An independent investigation of the validity of the School Attitude Assessment SurveyRevised. Journal of Psychoeducational Assessment, 26(1), 69–82.CrossRefGoogle Scholar
  71. Süss, H.-M., Oberauer, K., Wittmann, W. W., Wilhelm, O., & Schulze, R. (2002). Working-memory capacity explains reasoning ability—And a little bit more. Intelligence, 30, 261–288.CrossRefGoogle Scholar
  72. Super, D. E. (1970). Work values inventory manual. Boston, MA: Houghton Mifflin Company.Google Scholar
  73. Tabachnick, B. G., & Fidell, L. S. (2001). Using multivariate statistics. New York: Harper Collins.Google Scholar
  74. Taylor, J. (1953). A personality scale of manifest anxiety. The Journal of Abnormal and Social Psychology, 48(2), 285–290.CrossRefGoogle Scholar
  75. Warner, W. L., Meeker, M., & Eells, K. (1949). Social class in America; a manual of procedure for the measurement of social status. Oxford, England: Science Research Associates.Google Scholar
  76. Widaman, K. F. (1985). Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement, 9, 1–26.CrossRefGoogle Scholar
  77. Widaman, K. F. (1992). Multitrait-multimethod models in aging research. Experimental Aging Research, 18, 185–201.PubMedCrossRefGoogle Scholar
  78. Wilhelm, O., & Schulze, R. (2002). The relation of speeded and unspeeded reasoning with mental speed. Intelligence, 30, 537–554.CrossRefGoogle Scholar
  79. Wolfle, J. A. (1991). Underachieving gifted males: Are we missing the boat? Roeper Review, 13, 181–184.CrossRefGoogle Scholar
  80. Wood, A. M., Joseph, S., & Maltby, J. (2008). Gratitude uniquely predicts satisfaction with life: Incremental validity above the domains and facets of the five factor model. Personality and Individual Differences, 45(1), 49–54.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • D. Betsy McCoach
    • 1
    Email author
  • Robert K. Gable
    • 2
  • John P. Madura
    • 3
  1. 1.Educational Psychology DepartmentUniversity of ConnecticutStorrsUSA
  2. 2.Alan Shawn Feinstein Graduate SchoolJohnson and Wales UniversityStorrsUSA
  3. 3.Department of Educational PsychologyUniversity of ConnecticutStorrsUSA

Personalised recommendations