Advertisement

Additional Evidence Based on the Internal Structure of the Instrument

  • D. Betsy McCoachEmail author
  • Robert K. Gable
  • John P. Madura
Chapter
  • 1.7k Downloads

Abstract

In many ways, Chap. 5 is an extension of the methods and techniques used to gather evidence based on the internal structure of the instrument and continues to describe the latent variable approach to instrument development. In this chapter, we introduce two common instrument development situations where discrete mathematical structures are encountered: latent class analysis (LCA) and item response theory (IRT). In LCA, both the latent variable and the indicator variable are best represented by a categorical structure. In IRT, the latent trait is continuous, but the items are the items are categorical. IRT has historically been used in educational achievement applications, however, it is gaining popularity in some affective characteristic measurement scenarios. The last section of the chapter is devoted to the topic of measurement invariance and the analytic techniques that allow instrument developers to explore whether the scale functions in the same manner (exhibits the same internal structure) across subgroups. If the items that reflect their latent constructs and the connections between the constructs operate in a fundamentally different way depending on group membership, cross group comparisons become difficult, if not impossible. This chapter discusses the most common statistical techniques (along with examples) for establishing invariance for both scenarios where the indicators are either continuous or categorical.

Keywords

Latent class analysis Confirmatory latent class analysis Exploratory LCA Entropy Information criterion measures Classification Average posterior probabilities Invariance property Rasch Model Item response theory Response styles Invariance Vuong-Lo-Mendell-Rubin likelihood ratio test Item difficulty Item discrimination Item Fit Person fit Multiple Group Confirmatory Factor Analysis (MG-CFA) Configural model Unconstrained model Model constraints Measurement weights Measurement intercepts Structural paths 

References

  1. Akaike, H. (1987). Factor analysis and AIC. Psychometrika, 52(3), 317–332.CrossRefGoogle Scholar
  2. American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (1999). The standards for educational and psychological testing. Washington, DC: American Educational Research Association.Google Scholar
  3. Andrich, D. (1978a). Application of a psychometric rating model to ordered categories which are scored with successive integers. Applied Psychological Measurement, 2, 581–594.CrossRefGoogle Scholar
  4. Andrich, D. (1978b). Rating formulation for ordered response categories. Psychometrika, 43, 561–573.CrossRefGoogle Scholar
  5. Andrich, D. (1978c). Scaling attitude items constructed and scored in the Likert tradition. Educational and Psychological Measurement, 38, 665–680.CrossRefGoogle Scholar
  6. Beck, C. T., & Gable, R. K. (2001). Further validation of the postpartum depression screening scale. Nursing Research, 50, 155–164.PubMedCrossRefGoogle Scholar
  7. Boscardin, C. (2012). Profiling students for remediation using latent class analysis. Advances in Health Sciences, 17, 56–63.Google Scholar
  8. Boscardin, C. K., Muthen, B., Francis, D. J., & Baker, E. L. (2008). Early identification of reading difficulties using heterogeneous developmental trajectories. Journal of Educational Psychology, 100(1), 192–208.CrossRefGoogle Scholar
  9. Bozdogan, H. (1987). Model selection and Akaike’s information criteria (AIC): The general theory and its analytical extensions. Psychometrika, 52(3), 345–370.CrossRefGoogle Scholar
  10. Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York: The Guilford Press.Google Scholar
  11. Collins, L. M., & Lanza, S. T. (2010). Latent class and latent transition analysis with applications in the social, behavioral, and health sciences. Hoboken, NJ: Wiley.Google Scholar
  12. Dias, J. G., & Vermunt, J. K. (2006). Bootstrap methods for measuring classification uncertainty in latent class analysis. In A. Rizzi & M. Vichi (Eds.), Proceedings in computational statistics (pp. 31–41). Heidelberg: Springer.Google Scholar
  13. Dimitrov, D. M. (2010). Testing factorial invariance in the context of construct validation. Measurement and Evaluation in Counseling and Development, 43(2), 121–149.CrossRefGoogle Scholar
  14. Embretson, S. E. (1999). Issues in the measurement of cognitive abilities. In S. Embretson & S. Hershberger (Eds.), The new rules of measurement: What every psychologist and educator should know (pp. 1–15). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  15. French, B. F., & Finch, W. H. (2008). Multigroup confirmatory factor analysis: Locating the invariant reference sets. Structural Equation Modeling: A Multidisciplinary Journal, 15, 96–113.CrossRefGoogle Scholar
  16. Finch, W. H., & Bronk, K. C. (2011). Conducting confirmatory latent class analysis using Mplus. Structural Equation Modeling: A Multidisciplinary Journal, 18(1), 132–151.CrossRefGoogle Scholar
  17. Franek, M. (2005/2006). Foiling cyberbullies in the new Wild West. Educational Leadership, 63, 39–43.Google Scholar
  18. Gable, R. K., Ludlow, L. H., Kite, S. L., McCoach, D. B., & Filippelli, L. P. (2009, April). Development and validation of the survey of knowledge of internet risk and internet behavior. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.Google Scholar
  19. Gable, R. K., Ludlow, L. H., McCoach, D. B., & Kite, S. L. (2010, October). Construct invariance of the survey of knowledge of internet risk and internet behavior. Paper presented at the Annual Conference of the Northeastern Educational Research Association, Rocky Hill, CT.Google Scholar
  20. Gable, R. K., Ludlow, L. H., McCoach, D. B., & Kite, S. L. (2011). Validation of the survey of knowledge of internet risk and internet behavior. Educational and Psychological Measurement, 71(1), 217–230.CrossRefGoogle Scholar
  21. Gable, R. K., Ludlow, L. H., & Wolf, M. B. (1990). The use of classical and Rasch latent trait models to enhance the validity of affective measures. Educational and Psychological Measurement, 50(4), 869–878.CrossRefGoogle Scholar
  22. Gable, R. K., & Wolf, M. B. (1993). Instrument development in the affective domain: Measuring attitudes and values in corporate and school settings (2nd ed.). Boston: Kluwer-Nijhoff.CrossRefGoogle Scholar
  23. Hadzi-Pavlovic, D. (2009). Finding patterns and groupings: I. Introduction to latent class analysis. Acta Neuropsychiatrica, 21(6), 312–313.CrossRefGoogle Scholar
  24. Hagenaars, J. A., & McCutcheon, A. L. (2002). Applied latent class analysis. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  25. Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage.Google Scholar
  26. Hattie, J., Jaeger, R. M., & Bond, L. (1999). Persistent methodological questions in educational testing. Review of Research in Education, 24, 393–446.Google Scholar
  27. Helms, B. J., & Gable, R. K. (1989). School situation survey manual. Palo Alto: Consulting Psychologists Press/The Mind Garden.Google Scholar
  28. Holland, W., & Wainer, H. (1993). Differential item functioning. Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  29. Henson, J. M., Reise, S. P., & Kim, K. H. (2007). Detecting mixtures from structural model differences using latent variable mixture modeling: A comparison of relative model fit statistics. Structural Equation Modeling: A Multidisciplinary Journal, 14(2), 202–226.CrossRefGoogle Scholar
  30. Johnson, W. J., Dixon, P. N., & Ryan, J. M. (1991, April). Factorial and Rasch analysis of the Charles F. Kettering Ltd. school climate profile. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL.Google Scholar
  31. Kite, S. L., Gable, R. K., & Filippelli, L. P. (2010a). Assessment of students’ knowledge of Internet risk and Internet behaviors: Potential threat to bullying and contact by InternetpPredators. Paper presented at the Annual Meeting of the Northeastern Educational Research Association, Rocky Hill, CT.Google Scholar
  32. Kite, S. L., Gable, R. K., & Filippelli, L. (2010b). Assessing middle school students’ knowledge of conduct/consequences and their behaviors regarding the use of social networking sites. The Clearing House, 1939-912X, 83, 158–163.Google Scholar
  33. Lin, T. H., & Dayton, C. M. (1997). Model selection information criteria for non-nested latent class models. Journal of Educational and Behavioral Statistics, 22(3), 249–264.Google Scholar
  34. Lo, Y., Mendell, N. R., & Rubin, D. B. (2001). Testing the number of components in a normal mixture. Biometrika, 88(3), 767–778.CrossRefGoogle Scholar
  35. Ludlow, L. H., Enterline, S., & Cochran-Smith, M. (2008). Learning to teach for social justice-beliefs scale: An application of Rasch measurement principles. Measurement and Evaluation in Counseling and Development, 20, 194–214.Google Scholar
  36. Ludlow, L. H., & Haley, S. M. (1995). Rasch model logits: Interpretation, use, and transformation. Educational and Psychological Measurement, 55, 967–975.CrossRefGoogle Scholar
  37. Magidson, J., & Vermunt, J. K. (2004). Latent class models. In D. Kaplan (Ed.), The Sage handbook of quantitative methodology for the social sciences (pp. 175–198). Thousand Oaks, CA: Sage.Google Scholar
  38. Masters, G. N. (1980). A Rasch model for rating scales (Unpublished doctoral dissertation). Chicago, IL: University of Chicago.Google Scholar
  39. Masters, G. N., & Hyde, N. H. (1984). Measuring attitude to school with a latent trait model. Applied Psychological Measurement, 8(1), 39–48.CrossRefGoogle Scholar
  40. McCoach, D. B. (2002). A validity study of the school attitude assessment survey (SAAS). Measurement and Evaluation in Counseling and Development, 35, 66–77.Google Scholar
  41. McCoach, D. B., & Siegle, D. (2003a). The SAAS-R: A new instrument to identify academically able students who underachieve. Educational and Psychological Measurement, 63, 414–429.CrossRefGoogle Scholar
  42. McCoach, D. B., & Siegle, D. (2003b). The structure and function of academic self-concept in gifted and general education samples. Roeper Review, 25, 61–65.CrossRefGoogle Scholar
  43. McCutcheon, A. L. (1987). Latent class analysis. Newbury Park, CA: Sage.Google Scholar
  44. McKenna, P. (2007). The rise of cyberbullying. New Scientist, 195(2613), 60.CrossRefGoogle Scholar
  45. Muthén, B., & Muthén, L. K. (2000). Integrating person-centered and variable-centered analyses: Growth mixture modeling with latent trajectory classes. Alcoholism: Clinical and Experimental Research, 24(6), 882–891.Google Scholar
  46. Muthen, L. K., & Muthen, B. O. (2010). MPLUS user’s guide (6th ed.). Los Angeles, CA: Muthen & Muthen.Google Scholar
  47. Muthén, B. O., & Muthén, L. K. (2000). The development of heavy drinking and alcohol-related problems from ages 18 to 37 in a U.S. national sample. Journal of Studies on Alcohol, 61, 290–300.PubMedGoogle Scholar
  48. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.Google Scholar
  49. Nylund, K. L., Asparouhov, T., & Muthen, B. O. (2007). Deciding on the number of classes in latent class analysis and growth mixture modeling: A Monte Carlo simulation study. Structural Equation Modeling: A Multidisciplinary Journal, 14(4), 535–569.CrossRefGoogle Scholar
  50. Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen: Danmarks Paedogogiske Institute.Google Scholar
  51. Rasch, G. (1966). An item analysis which takes individual differences into account. British Journal of Mathematical and Statistical Psychology, 19, 49–57.PubMedCrossRefGoogle Scholar
  52. Rensvold, R. B., & Cheung, G. W. (2001). Testing for metric invariance using structural equation models: Solving the standardization problem. In C. A. Schriesheim & L. L. Neider (Eds.), Research in management: Equivalence in measurement (pp. 25–50). Greenwich, CT: Information Age Publishing.Google Scholar
  53. Sass, D. A. (2011). Testing measurement invariance and comparing latent factor means within a confirmatory factor analysis framework. Journal of Psychoeducational Assessment, 29(4), 347–363.CrossRefGoogle Scholar
  54. Schwartz, S. A. (1978). A comprehensive system for item analysis in psychological scale construction. Journal of Educational Measurement, 15, 117–123.CrossRefGoogle Scholar
  55. Sclove, S. L. (1987). Application of model-selection criteria to some problems in multivariate analysis [Special section]. Psychometrika, 52(3), 333–343.CrossRefGoogle Scholar
  56. Tormakangas, K. (2011). Advantages of the Rasch measurement model in analyzing educational tests: An applicator’s reflection. Educational Research and Evaluation: An International Journal on Theory and Practice, 17(5), 307–320.CrossRefGoogle Scholar
  57. Tovar, E., & Simon, M. A. (2010). Factorial structure and invariance analysis in sense of belonging scales. Measurement and Evaluation in Counseling and Development, 43(3), 199–217.CrossRefGoogle Scholar
  58. Vermunt, J. K., & Magidson, J. (2002). Latent class cluster analysis. In J. A. Hagenaars & A. L. McCutcheon (Eds.), Applied latent class analysis (pp. 89–106). Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  59. Vermunt, J. K., & Magidson, J. (2004). Latent class analysis. In M. S. Lewis-Beck, A. Bryman, & T. F. Liao (Eds.), The sage encyclopedia of social sciences research methods (pp. 549–553). Thousand Oakes, CA: Sage Publications.Google Scholar
  60. Vuong, Q. H. (1989). Likelihood ratio tests for model selection and non-nested hypotheses. Econometrica, 57(2), 307–333.CrossRefGoogle Scholar
  61. Weiss, D. J., & Yoes, M. E. (1991). Item response theory. In R. Hambleton & J. Zaal (Eds.), Advances in educational and psychological testing: Theory and applications (pp. 69–95). New York, NY: Kluwer Academic/Plenum Publishers.Google Scholar
  62. Wilson, M. (2005). Constructing measures: An item response modeling approach. New York, NY: Taylor & Francis.Google Scholar
  63. Wirth, R. J., & Edwards, M. C. (2007). Item factor analysis: Current approaches and future directions. Psychological Methods, 12(1), 58–79.PubMedCrossRefGoogle Scholar
  64. Wright, B. D., & Linacre, M. (1998). Winsteps. Chicago, IL: Mesa Press.Google Scholar
  65. Wright, B. D., & Masters, G. N. (1982). Rating scale analysis. Chicago, IL: Mesa Press.Google Scholar
  66. Wright, B. D., & Stone, M. H. (1979). Best test design. Chicago, IL: Mesa Press.Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • D. Betsy McCoach
    • 1
    Email author
  • Robert K. Gable
    • 2
  • John P. Madura
    • 3
  1. 1.Educational Psychology DepartmentUniversity of ConnecticutStorrsUSA
  2. 2.Alan Shawn Feinstein Graduate SchoolJohnson and Wales UniversityStorrsUSA
  3. 3.Department of Educational PsychologyUniversity of ConnecticutStorrsUSA

Personalised recommendations