Psychological Development and Education ›› 2020, Vol. 36 ›› Issue (3): 378-384.doi: 10.16187/j.cnki.issn1001-4918.2020.03.15

Previous Articles    

Estimating the Best Sample Size of Teaching Level Evaluation for College Teachers under Budget Constraints in Generalizability Theory

LI Guangming, CHEN Zihao, ZHANG Minqiang   

  1. School of Psychology, Center for Studies of Psychological Application, South China Normal University, Guangzhou 510631
  • Published:2020-06-02

Abstract: Using the evaluation questionnaire of Teachers' Teaching Level Evaluation Scale for College Students(TTLES-CS), 566 students were asked to evaluate 19 teachers, and the data were generalized to four kinds of designs such as t×i,(s:ti,(s:t)×(i:v) and(s:t)×(i:vo. Based on generalizability theory, we combined the budget constraints and derived a unification algorithm adapting with the optimal sample size estimation for different designs of generalizability theory, i.e. LaGrange multiplication. On this basis, we applied LaGrange multiplication to estimate the optimal observation number in different generalizability designs. The results show that: (1) LaGrange multiplication that we derived has a stronger robustness, which can apply to different study designs effectively under budget constraints. (2) Occasion is an important factor that has a great effect on teachers' teaching level evaluation. So, we need attach importance to it. (3) According to the actual circumstances under the budget constraints, the design of(s:t)×(i:vo is the optimal generalized design. (4) Under budget constraints of teachers' teaching level evaluation for college teachers in generalizability theory, the best sample size of students is 20 for each teacher and the best sample size of items is 3 for each dimension.

Key words: generalizability theory, budget constraints, estimating the best sample size, college teachers, teaching level evaluation

CLC Number: 

  • B844
Brennan, R. L. (2001a). Manual for urGENOVA. Iowa City, IA:Iowa Testing Programs, University of Iowa.
Brennan, R. L. (2001b). Generalizability theory. New York:Springer-Verlag.
Casabianca, J. M., Lockwood, J. R., & McCaffrey, D. F. (2015). Trends in classroom observation scores. Educational and Psychological Measurement, 75(2), 311-337.
Chang, L., & Hocevar, D. (2000). Models of generalizability theory in analyzing existing faculty evaluation data. Applied Measurement in Education, 13(3), 255-275.
Gitomer, D. H., Bell, C. A., Qi, Y., McCaffrey, D. F., Hamre, B. K., & Pianta, R. C. (2014). The instructional challenge in improving teaching quality:Lessons from a classroom observation protocol. Teachers College Record, 116(6), 1-32.
Goldstein, Z., & Marcoulides, G. A. (1991). Maximizing the coefficient of generalizability in decision studies. Educational and Psychological Measurement, 51(1), 79-88.
Hill, H. C., Charalambous, C. Y., & Kraft, M. A. (2012). When rater reliability is not enough:Teaching observation systems and a case for the generalizability study. Educational Researcher, 41(2), 56-64.
Iqbal, I., Lee, J. D., Pearson, M. L., & Albon, S. P. (2016). Student and faculty perceptions of student evaluations of teaching in a Canadian pharmacy school. Currents in Pharmacy Teaching & Learning, 8(2), 191-199.
Lakes, K. D. (2013). Restricted sample variance reduces generalizability. Psychological Assessment, 25(2), 643-650.
Marcoulides. G. A. (1993). Maximizing power in generalizability studies under budget constraints. Journal of Educational Statistics, 18(2), 197-206.
Marcoulides, G. A. (1997). Optimizing measurement designs with budget constraints:The variable cost case. Educational and Psychological Measurement, 57(5), 808-812.
Meyer, P. J., Liu, X., & Mashburn, A. J. (2014). A practical solution to optimizing the reliability of teaching observation measures under budget constraints. Educational and Psychological Measurement, 74(2), 280-291.
Wolbing, T., & Riordan, P. (2016). How beauty works. Theoretical mechanisms and two empirical applications on students' evaluation of teaching. Social Science Research, 57(5), 253-272.
乐美玲, 辛涛. (2015). 高等教育评估的几个关键问题. 清华大学教育研究, 36(1), 48-52.
王政民, 吴阔华. (1999). 高等数学(第三版). 南昌:江西高校出版社.
杨志明, 张雷. (2003). 测评的概化理论及其应用.北京:教育科学出版社.
[1] LI Guangming, ZHANG Minqiang. Weight Effect Analysis of Multivariate Generalizability Theory for Teaching Level Evaluation of College Teachers [J]. Psychological Development and Education, 2017, 33(1): 122-128.
[2] LI Guang-ming, ZHANG Min-qiang. Analysis of Cross-distribution for Estimating Variance Components in Generalizability Theory [J]. Psychological Development and Education, 2012, 28(6): 665-672.
[3] HAN Ming, DONG Xue-an, FAN Dan, HE Xian-you. Developing Questionnaire of Psychological Contract for College Teachers [J]. Psychological Development and Education, 2010, 26(3): 315-321.
[4] HAN Ming, DONG Xue-an, FAN Dan, HE Xian-you. Developing Questionnaire of Psychological Contract for College Teachers [J]. Psychological Development and Education, 2010, 26(3): 315-321.
[5] HAN Ming, DONG Xue-an, FAN Dan, HE Xian-you. Developing Questionnaire of Psychological Contract for College Teachers [J]. Psychological Development and Education, 2010, 26(3): 315-321.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!