Optimizing the efficiency and precision of the Delaware School Climate Scale: an application of multivariate generalizability theory
Date
2019
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
University of Delaware
Abstract
This dissertation examined issues about the efficiency and precision of Delaware School Climate Scale (DSCS) scores in elementary, middle, and high schools, and among samples of students, teachers, and parents across three years. It also contributed to the nascent research on measuring school climate as a multilevel, multi-informant (i.e., student, teacher, parent), and multi-dimensional construct using survey questionnaires. To illustrate how multivariate generalizability theory can address issues in measuring school climate, I rely on data from the DSSC to demonstrate the application of the proposed approach. In total, students in elementary (n = 41,754), middle (n = 32,919), and high schools (n = 19,374) responded to the DSCS-Student survey composed of 30 items in seven subscales. Teachers in elementary (n = 5,542), middle (n = 2,885), and high schools (n = 2,577) responded to the DSCS-Teacher/Staff survey composed of 38 items in nine subscales. Parents in elementary (n = 27,310), middle (n = 8,641), and high schools (n = 2768) responded to the DSCS-Home survey composed of 25 items in six subscales. A multivariate generalizability theory model with the class-means design was used to investigate the sources of variation in the DSCS scale and subscale scores: person within school, items, and interactions among those facets and the object of measurement, schools. Separate G and D studies were conducted, and school profile scores were produced for student, teacher, and parent groups by grade levels and years. Then, a second-order model was used to examine the sources of variation when measuring school climate longitudinally as a multi-informant index: occasions and interaction between occasions and the object of measurement, schools. The results of these two models indicated that the DSCS can provide reliable and precise estimates of school-level school climate in most groups, except for the parent respondent group for elementary schools across years. The results of D studies also showed that fewer respondents per school are needed to reach the recommended threshold of .80 of G coefficients in most groups. ☐ Specific recommendations for future DSCS measurement procedures and implications for the use of the DSCS scale and subscale scores are provided. Implications for measuring school climate and sampling respondents are also discussed. If the future measurement procedure focuses on students, fewer students (40% of the original number of students per school) and one less item per subscale is necessary to reach the recommended threshold .80 across grade levels. For the teachers’ sample, fewer teachers per school (60% of the original number of teachers per school) are needed to achieve the threshold of reliability in middle and high schools. For the parents’ sample, fewer parents per school (80% of the original number of parents per school) are required to achieve the recommended threshold in high schools, but more parents per school are required in elementary and middle school. In addition, the results suggest that the use of multiple survey forms, with matrix sampling of items, can simultaneously increase reliability while also reducing response burden. If resources are limited, a survey of students or teachers can be sufficient for evaluating school-level school climate.