Assessing response shifts and true changes in longitudinal evaluation research: the impact of the first COVID-19 lockdown on subjective well-being in Italy

Journal title RIV Rassegna Italiana di Valutazione
Author/s Tiziano Gerosa
Publishing Year 2023 Issue 2022/82 Language English
Pages 18 P. 75-92 File size 861 KB
DOI 10.3280/RIV2022-082005
DOI is like a bar code for intellectual property: to have more infomation click here

Below, you can see the article first page

If you want to buy this article in PDF format, you can do it, following the instructions to buy download credits

Article preview

FrancoAngeli is member of Publishers International Linking Association, Inc (PILA), a not-for-profit association which run the CrossRef service enabling links to and from online scholarly content.

Latent constructs measured at multiple points in time are frequently used in social and evaluation research to assess the impact of interven-tions or relevant phenomena on individuals and groups (e.g. training programs, policies, exogenous events, etc.). However, researchers of-ten do not consider the accuracy of such models depends on the as-sumption of longitudinal measurement invariance (LMI), which states that repeatedly measured latent variables should effectively represent the same construct in the same metric over time. Non-invariance can incur when individuals recalibrate their standards of measurement, reprioritize their values or reconceptualize the target construct as a di-rect consequence of the intervention or the phenomena under study. Such response shifts may obfuscate true changes in the latent means of the construct, thus leading to inaccurate inferences. The present study aims to explore this issue by first offering a detailed description of the potential sources of response shifts and the general procedures used for LMI testing in a structural equation modelling framework. An em-pirical application is then carried out on a sample of Italian respond-ents interviewed before and immediately after the COVID-19 lock-down and, based on the results of the analysis, practical implications for evaluation research using latent outcome variables in a longitudinal setting are discussed.

  1. Bowen, N. K., & Masa, R. D. (2015). Conducting Measurement Invariance Tests with Ordinal Data: A Guide for Social Work Researchers. Journal of the Society for Social Work and Research, 6(2), 229–249.
  2. Byrne, B. M., Shavelson, R. J., & Muthén, B. (1989). Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin, 105(3), 456–466. DOI: 10.1037/0033-2909.105.3.45
  3. Campbell, D. T., & Stanley, J.C. (1963). Experimental and quasi-experimental designs for research on teaching. In N.L. Gage (Ed.), Handbook of research on teaching. Chicago, Rand McNally.
  4. Case-Smith, J., Holland, T., Lane, A., & White, S. (2012). Effect of a coteaching handwriting program for first graders: One-group pretest–posttest design. The American Journal of Occupational Therapy, 66(4), 396-405.
  5. Diener, E. (1984). Subjective well-being. Psychological Bulletin, 95(3), 542-575.
  6. Dimitrov, D. M., & Rumrill Jr, P. D. (2003). Pretest-posttest designs and measurement of change. Work, 20(2), 159-165.
  7. Fayers, P. M., & Hand, D. J. (2002). Causal variables, indicator variables and measurement scales: an example from quality of life. Journal of the Royal Statistical Society: Series A (Statistics in Society), 165(2), 233-253.
  8. Gerosa, T. (2021). Measurement invariance with ordered categorical variables: applications in longitudinal survey research. In A. Cernat & J.W. Sakshaug (eds). Measurement error in longitudinal data, Oxford Academic, 265-278.
  9. Gerosa, T., & Respi, C. (2022). Ad Hoc Module 1: ITA.LI COVID-19. Brief Report. Institute for Advanced Study of Social Change. Retrieved from:
  10. Giorgi, G., Lecca, L. I., Alessio, F., Finstad, G. L., Bondanini, G., Lulli, L. G., ... & Mucci, N. (2020). COVID-19-related mental health effects in the workplace: a narrative review. International journal of environmental research and public health, 17(21), 7857.
  11. Hesterberg, T. (2011). Bootstrap. Wiley Interdisciplinary Reviews: Computational Statistics, 3(6), 497-526.
  12. Hobart, J. C., Cano, S. J., Zajicek, J. P., & Thompson, A. J. (2007). Rating scales as outcome measures for clinical trials in neurology: problems, solutions, and recommendations. The Lancet Neurology, 6(12), 1094-1105.
  13. Hong, S., Malik, M. L., & Lee, M. K. (2003). Testing configural, metric, scalar, and latent mean invariance across genders in sociotropy and autonomy using a non-Western sample. Educational and psychological measurement, 63(4), 636-654.
  14. Hox, J. J., De Leeuw, E. D., & Zijlmans, E. A. (2015). Measurement equivalence in mixed mode surveys. Frontiers in psychology, 6, 87.
  15. Khoo, S. T., West, S. G., Wu, W., Kwok, O. M (2006). Longitudinal methods. In Eid, M., Diener, E. (eds). Handbook of multimethod measurement in psychology. Washington, D.C: American Psychological Association, 301-317.
  16. Kline, R. B. (2015). Principles and practice of structural equation modeling (4th ed.). Guilford Publications.
  17. Knapp, T. R. (2016). Why is the one-group pretest–posttest design still used?. Clinical Nursing Research, 25(5), 467-472.
  18. Little, T. D. (2013). Longitudinal structural equation modeling. Guilford press. Liu, Y., & West, S. G. (2018). Longitudinal Measurement Non-Invariance with Ordered-Categorical Indicators: How are the Parameters in Second-Order Latent Linear Growth Models Affected? Structural Equation Modeling: A Multidisciplinary Journal, 25(5), 762–777.
  19. Lucchini, M., De Santis, G., Gerosa, T., & Respi, C. (2022). Italian Lives – wave 1: Report Metodologico. Institute for Advanced Study of Social Change. Retrieved from:
  20. Lucchini, M., Gerosa, T., Pancheva, M., Pisati, M., Respi, C., & Riva, E. (2021). Differential effects of COVID-19 and containment measures on mental health: Evidence from ITA. LI—Italian Lives, the Italian household panel. Plos one, 16(11), e0259989.
  21. Lyubomirsky, S., King, L., & Diener, E. (2005). The benefits of frequent positive affect: Does happiness lead to success? Psychological bulletin, 131(6), 803.
  22. Marsh, H. W., Liem, G. A. D., Martin, A. J., Morin, A. J. S., & Nagengast, B. (2011). Methodological Measurement Fruitfulness of Exploratory Structural Equation Modeling (ESEM): New Approaches to Key Substantive Issues in Motivation and Engagement. Journal of Psychoeducational Assessment, 29(4), 322–346.
  23. Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58(4), 525–543.
  24. Muthén, B., & Asparouhov, T. (2002). Latent variable analysis with categorical outcomes: Multiple-group and growth modeling in Mplus. Mplus web notes, 4(5), 1-22.
  25. Muthén, L. K., & Muthén, B. O. (2012). Mplus: Statistical Analysis with Latent Variables – User’s Guide.
  26. Olino, T. M., Case, J. A., Hawes, M. T., Szenczy, A., Nelson, B., & Klein, D. N. (2021). Testing invariance of measures of internalizing symptoms before and after a major life stressor: the impact of COVID-19 in an adolescent and young adult sample. Assessment, 10731911211015315.
  27. Oort, F. J. (2005). Using structural equation modeling to detect response shifts and true change. Quality of Life Research, 14(3), 587-598.
  28. Pierce, M., Hope, H., Ford, T., Hatch, S., Hotopf, M., John, A., ... & Abel, K. M. (2020). Mental health before and during the COVID-19 pandemic: a longitudinal probability sample survey of the UK population. The Lancet Psychiatry, 7(10), 883-892.
  29. Prati, G., & Mancini, A. D. (2021). The psychological impact of COVID-19 pandemic lockdowns: a review and meta-analysis of longitudinal studies and natural experiments. Psychological medicine, 51(2), 201-211.
  30. Putnick, D. L., & Bornstein, M. H. (2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review, 41, 71–90.
  31. Respi, C., & Gerosa, T. (2021). Survey participation and non-response error in a pandemic scenario. Results from the ITA. LI Covid-19 study. Rassegna Italiana di Sociologia, 62(1), 39-65. DOI: 10.1423/10062
  32. Rhemtulla, M., Brosseau-Liard, P. É., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354–373.
  33. Riva, E., Terraneo, M., Lucchini, M., & Gerosa, T. (2022). The prevalence of insomnia in different COVID-19 policy phases: Longitudinal evidence from ITA. LI–Italian Lives. BMC Public Health, 22(1), 1-10.
  34. Sass, D. A. (2011). Testing Measurement Invariance and Comparing Latent Factor Means Within a Confirmatory Factor Analysis Framework. Journal of Psychoeducational Assessment, 29(4), 347–363.
  35. Sass, D. A., Schmitt, T. A., & Marsh, H. W. (2014). Evaluating Model Fit With Ordered Categorical Data Within a Measurement Invariance Framework: A Comparison of Estimators. Structural Equation Modeling: A Multidisciplinary Journal, 21(2), 167–180.
  36. Sprangers, M. A., & Schwartz, C. E. (1999). Integrating response shift into health-related quality of life research: a theoretical model. Social science & medicine, 48(11), 1507-1515.
  37. Steenkamp, J.-B. E. M., & Baumgartner, H. (1998). Assessing Measurement Invariance in Cross-National Consumer Research. Journal of Consumer Research, 25(1), 78–90.
  38. Topp, C. W., Østergaard, S. D., Søndergaard, S., & Bech, P. (2015). The WHO-5 Well-Being Index: a systematic review of the literature. Psychotherapy and psychosomatics, 84(3), 167-176.
  39. Van De Schoot, R., Schmidt, P., De Beuckelaer, A., Lek, K., & Zondervan-Zwijnenburg, M. (2015). Measurement invariance. Frontiers in psychology, 6, 1064.
  40. Vandenberg, R. J., & Lance, C. E. (2000). A Review and Synthesis of the Measurement Invariance Literature: Suggestions, Practices, and Recommendations for Organizational Research. Organizational Research Methods, 3(1), 4–70. DOI: 10.1177/10944281003100
  41. Vindegaard, N., & Benros, M. E. (2020). COVID-19 pandemic and mental health consequences: Systematic review of the current evidence. Brain, behavior, and immunity, 89, 531-542.
  42. Wang, J., & Wang, X. (2012). Structural equation modeling: Methods and applications. CERN Document Server.
  43. Ware Jr, J. E. (2000). SF-36 health survey update. Spine, 25(24), 3130-3139.
  44. Widaman, K. F. & Reise, S. P. (1997). Exploring the measurement invariance of psychological instruments: applications in the substance use domain. In K. J. Bryant & M. Windle (Eds.), The science of prevention: Methodological advances from alcohol and substance abuse research. Washington, DC: American Psychological Association. 281-324.

Tiziano Gerosa, Assessing response shifts and true changes in longitudinal evaluation research: the impact of the first COVID-19 lockdown on subjective well-being in Italy in "RIV Rassegna Italiana di Valutazione" 82/2022, pp 75-92, DOI: 10.3280/RIV2022-082005