Psychological Development and Education ›› 2011, Vol. 27 ›› Issue (2): 210-215.

Previous Articles     Next Articles

A Review of Decision Consistency Indices of Criteria-Reference Test

CHEN Ping, LI Zhen, XIN Tao, GAO Hui-jian   

  1. Institute of Developmental Psychology, Beijing Normal University, Beijing 100875
  • Online:2011-03-15 Published:2011-03-15

PDF (PC)

Abstract: This paper presented an overview of various procedures for estimating single-administration decision consistency index which is an important quality standard of criterion-referenced test.Researchers have proposed dozens of estimation methods based on classical test theory or item response theory,and have made some comparisons among them.Future studies should focus on validating these methods and exploring its application in educational measurement,providing psychometricians with a basis for choosing the appropriate estimation method for decision consistency in particular situation.

Key words: decision consistency, reliability, index p, index Kappa

CLC Number: 

  • G449
[1] AERA,APA,& NCME(1999).Standards for educational and psychological testing.Washington,DC:Author.35-36.
[2] Brennan,R.L.(2003).Coefficients and indices in generalizability theory(CASMA Research Report No.1).Iowa City,IA:Center for Advanced Studies in Measurement and Assessment,The University of Iowa.(Available on http://www.education.uiowa.edu/casma).
[3] Brennan,R.L.,& Wan,L.(2004).A bootstrap procedure for estimating decision consistency for single-administration complex assessments(CASMA Research Report No.17).Iowa City,IA:Center for Ad-vanced Studies in Measurement and Assessment,The University of Iowa.(Available on http://www.education.uiowa.edu/casma).
[4] Crocker,L.M.,& Algina,J.(1986).Introduction to classical and modern test theory.Belmont in USA:Thomson Learning Academic Resource Center,192-211.
[5] Hanson,B.A.,& Brennan,R.L.(1990).An investigation of classification consistency indexes estimated under alternative strong true score models.Journal of Educational Measurement,27(4),345-359.
[6] Lee,W.C.,et al.(2002).Estimating consistency and accuracy indices for multiple classifications.Applied Psychological Measurement,26(4),412-432.
[7] Lee,W.C.(2005).Classification consistency under the compound multinomial model(CASMA Research Report No.13).Iowa City,IA:Cen-ter for Advanced Studies in Measurement and Assessment,The University of Iowa.(Available on http://www.education.uiowa.edu/cas-ma).
[8] Lee,W.C.(2008a).Classification consistency and accuracy for complex assessments using item response theory(CASMA Research Report No.27).Iowa City,IA:Center for Advanced Studies in Measurement and Assessment,The University of Iowa.(Available on http://www.education.uiowa.edu/casma).
[9] Lee,W.,& Kolen,M.J.(2008b).IRT CLASS:A computer program for item response theory classification consistency and accuracy(Version 2.0) ［Computer software］.Iowa City,IA:University of Iowa,Center for Advanced Studies in Measurement and Assessment.(Available on http://www.education.uiowa.edu/casma).
[10] Li,S.H.(2006).Evaluating the consistency and accuracy of proficiency classifications using Item Response Theory.Unpublished doctoral dissertation,University of Massachusetts Amherst.
[11] Livingston,S.A.,& Lewis,C.(1995).Estimating the consistency and accuracy of classifications based on test scores.Journal of Educational Measurement,32(2),179-197.
[12] Rudner,L.M.(2005).Expected classification accuracy.Practical Assessment Research & Evaluation,10(13),1-4.
[13] Wan,L.,Brennan R.L.,& Lee,W.C.(2007).Estimating classification consistency for complex assessments(CASMA Research Report No.22).Iowa City,IA:Center for Advanced Studies in Measurement and Assessment,The University of Iowa.(Available on http://www.education.uiowa.edu/casma).
[14] Yoo,H.,Sukin,M.T.,& Hambleton,R.K.(2009).Evaluating consistency and accuracy of proficiency classifications using a single administration IRT method(Final Report).Amherst,MA:University of Massachusetts,Center for Educational Assessment.
[15] Yoo,H.,& Bishop,N.S.(2010,April).Evaluating proficiency classification using testlet response theory.Paper presented at the annual meeting of the National Council on Measurement in Education,Denver,CO.
[16] 韩宁.(2008).评价考试质量的新指标:决策一致性和决策准确性.中国考试,(6),3-6.
[17] 赵世明.(2006).资格认证测验的分类一致性信度估计.考试研究,(10),30-34.
[1] LI Guangming, ZHANG Minqiang. Weight Effect Analysis of Multivariate Generalizability Theory for Teaching Level Evaluation of College Teachers [J]. Psychological Development and Education, 2017, 33(1): 122-128.
[2] LUO Jie, ZHOU Yuan, CHEN Wei, PAN Yun, ZHAO Shouying. A Reliability Generalization of the Big-Five Factor Personality Tests in China [J]. Psychological Development and Education, 2016, 32(1): 121-128.
[3] JIANG Jiang, LU Zheng-rong, JIANG Bi-jing, XU Yan. Revision of the Short-form Egna Minnenav Barndoms Uppfostran for Chinese [J]. Psychological Development and Education, 2010, 26(1): 94-99.
[4] LIU Junsheng, ZHOU Ying, SANG Biao. Psychometric Analysis of Children Sense of Coherence Scale in Chinese Cultural Context [J]. Psychological Development and Education, 2010, 26(1): 87-93.
[5] MENG Hong, CHENG Hui-jun, CAO Zhong-ping, HU Kun. The Development and Validation of Parents’ Evaluation on Social Skills of Pupils [J]. Psychological Development and Education, 2008, 24(4): 88-92.
[6] ZHANG Jing-huan, CHU Yu-xia, LIN Chong-de. Construct of Test on Creative Teaching Behavior in Teachers [J]. Psychological Development and Education, 2008, 24(3): 107-112.
[7] Zhang Ling-ling, Zhang Wen-xin, Ji Lin-qin, Jari-Erik Nurmi. Psychometric Analysis of Adolescent Future Orientation Questionnaire in Chinese Cultural Context [J]. Psychological Development and Education, 2006, 22(1): 103-108.
[8] LI Hong. Development of College Working Stress Scale [J]. Psychological Development and Education, 2005, 21(4): 105-109.
[9] HUANG Xi-Shan. Reliability and Validity of the Chinese Version of Teacher Efficacy Scale [J]. Psychological Development and Education, 2005, 21(1): 115-118.
[10] WANG Yi-wen, LIN Chong-de, ZHANG Wen-xin. Using multiple methods to measure children’s aggressive behavior [J]. Psychological Development and Education, 2004, 20(2): 69-74.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
Copyright 2018 © Editorial office of Psychological Development and Education
Supported by Beijing Magtech Co.Ltd