유럽 언어 공통기준 연계를 통한 영어 독해 시험의 수준설정: 수정된 앙고프 방법과 북마크 방법 비교 = Standard Setting to Relate an English Reading Comprehension Test to the CEFR: A Comparison of Modified Angoff and Bookmark Methods

다국어 초록 (Multilingual Abstract)

The purpose of this study is to link a reading comprehension subtest in a large-scale English proficiency test to the Common European Framework of Reference for Languages(CEFR). A panel consisting of 13 members made judgments using the modified Angoff and bookmark methods. Based on these judgments, cut scores that divide test takers into the six levels of the CEFR were derived, and the validity of the cut scores was evaluated using internal, external, and procedural criteria. Analysis showed that the standard deviation of the judgments decreased across rounds; the cut scores derived from the two methods were similar; and almost all of the panel members expressed confidence in the final cut scores on the post-standard-setting survey. Furthermore, most panel members responded that they found the preparation materials, the standard setting process, and the facilitators’ explanations to be clear and helpful when making judgments.

국문 초록 (Abstract)

본 연구는 국내에서 개발된 대규모 영어 능력 시험에 포함된 독해 시험을 유럽 언어 공통기준(Common European Framework of Reference for Languages)의 읽기 척도와 연계하기 위해 전문가 패널을 구성하고 수정된 앙고프 방법과 북마크 방법을 이용하여 분할점수를 산출하였다. 유럽 언어 공통기준의 6개 능력 수준을 나누는 독해 시험의 분할점수를 산출하였고, 그 결과에 대해 내적, 외적, 절차적 준거에 따라 평가하여 타당성을 검토하였다. 수집된 점수의 분석 결과, 전문가 패널 구성원이 판단한 분할점수의 표준편차가 라운드를 거칠수록 작아지는 경향을 보였고, 수정된 앙고프 방법과 북마크 방법을 통해 얻은 분할점수 결과는 매우 유사했으며, 수준설정 이후 패널 구성원의 대다수가 최종 분할점수에 확신을 갖는다고 응답했다. 또한 대부분의 패널 구성원들이 수준설정 준비 및 진행과정, 설명이 명확하였고 판단을 내릴 때 도움이 되었다고 평가하였다.

참고문헌 (Reference)

• 1 Cizek, G. J., "준거설정" 학지사 2011
• 2 이용상, "영어 쓰기 검사의 준거 설정 방안" 한국중등영어교육학회 11 (11): 147-163, 2018
• 3 반재천, "북마크 기준설정 방법에서 반응 확률 .67에 대응하는 능력추정치를 구하기 위한 Bifactor IRT와 Testlet IRT모형의 활용" 한국교육평가학회 30 (30): 549-568, 2017
• 4 송미영, "대규모 준거참조평가에서 성취수준의 분류 일치도와 정확도 분석" 한국교육평가학회 26 (26): 391-413, 2013
• 5 김성숙, "다변량 일반화가능도 이론을 적용한 성취수준 설정에서의 오차분석과 최적 조건 탐색" 한국교육평가학회 25 (25): 679-700, 2012
• 6 정현철, "논술형 수행평가의 고정 점수, Angoff, BoW 기준설정 결과 타당도 연구" 한국교육평가학회 31 (31): 463-482, 2018
• 7 박연복, "군집분석을 이용한 수준설정 방법과 타당성 연구" 한국교육평가학회 24 (24): 645-664, 2011
• 8 임해미, "국가수준 학업성취도 평가와 국제 학업성취도 평가의 연계를 통한 우리나라 학생들의 수학 성취 특성 분석" 대한수학교육학회 22 (22): 1-22, 2012
• 9 김선, "검사동등화를 이용한 앵고프 방법과 북마크 방법의 비교" 한국교육평가학회 22 (22): 1035-1055, 2009
• 10 Kane, M. T., "Validating the interpretations and uses of test scores" 50 : 1-73, 2013
• 11 Ilc, G., "Validating the Slovenian alignment to CEFR: The case of the B2 reading comprehension examination in English" 32 : 443-462, 2015
• 12 Wu, R. Y. F., "Validating second language reading examinations: Establishing the validity of the GEPT through alignment with the Common European Framework of Reference, Studies in Language Testing 41" Cambridge University Press 2014
• 13 Karantonis, A., "The bookmark standard‐setting method: A literature review" 25 : 4-12, 2006
• 14 Papageorgiou, S., "The association between TOEFL iBT test scores and the Common European Framework of Reference (CEFR) levels" Educational Testing Service 2015
• 15 Lim, G. S., "Standard setting to an international reference framework: Implications for theory and practice" 13 : 32-49, 2013
• 16 Bechger, T. M., "Standard setting in relation to the Common European Framework of Reference for Languages: The case of the State examination of Dutch as a second language" 6 : 126-150, 2009
• 17 Eckes, T., "Setting cut scores on an ESL placement test using the prototype group method: A receiver operating characteristic (ROC) analysis" 34 : 383-411, 2017
• 18 Peterson, C. H., "Reliability and validity of bookmark‐based methods for standard setting: comparisons to Angoff‐based methods in the National Assessment of Educational Progress" 30 : 3-14, 2011
• 19 Council of Europe, "Relating language examinations to the Common European Framework of Reference for Languages: Learning, teaching, assessment. A Manual" Council of Europe 2009
• 20 Lee, W. -C., "Program IRT-CLASS: For item response theory classification consistency and accuracy. Program Manual" University of Iowa 2008
• 21 Dunlea, J., "Linking to the CEFR levels: Research perspectives" CITO and EALTA 2009
• 22 Knoch, U., "Linking the GEPT Writing sub-test to the Common European Framework of Reference (CEFR)" LTTC 2016
• 23 Harding, L., "Linking the GEPT Listening Test to the Common European Framework of Reference" LTTC 2014
• 24 Tannenbaum, R. J., "Linking English-language test scores onto the Common European Framework of Reference: An application of standard-setting methodology" Educational Testing Service 2008
• 25 Green, A., "GEPT Speaking – CEFR benchmarking" LTTC 2017
• 26 Shaw, S. D., "Examining writing: Research and practice in assessing second language speaking, Studies in Language Testing 26" UCLES/Cambridge University Press 2007
• 27 Taylor, L., "Examining speaking: Research and practice in assessing second language speaking, Studies in Language Testing 30" UCLES/Cambridge University Press 2011
• 28 Khalifa, H., "Examining reading: Research and practice in assessing second language speaking, Studies in Language Testing 29" UCLES/Cambridge University Press 2009
• 29 Geranpayeh, A., "Examining listening: Research and practice in assessing second language speaking, Studies in Language Testing 35" UCLES/Cambridge University Press 2013
• 30 Kane, M., "Educational measurement" Greenwood Publishing 17-64, 2006
• 31 Angoff, W. H., "Educational measurement" American Council on Education 1971
• 32 Messick, S. J., "Educational measurement" Macmillan 1989
• 33 Harsch, C., "Designing and scaling level-specific writing tasks in alignment with the CEFR: A test-centered approach" 8 : 1-33, 2011
• 34 Tannenbaum, R. J., "Critical factors to consider in evaluating standard-setting studies to map language test scores to frameworks of language proficiency" 11 : 233-249, 2014
• 35 Council of Europe, "Common European Framework of Reference for Languages: Learning, teaching, assessment. Companion volume with new descriptors" Council of Europe 2018
• 36 Council of Europe, "Common European Framework of Reference for Languages: Learning, teaching, assessment" Cambridge University Press 2001
• 37 Zhao, W., "Calibrating the CEFR against the China Standards of English for College English vocabulary education in China" 7 : 1-18, 2017
• 38 Chapelle, C. A., "Building a validity argument for the Test of English as a Foreign LanguageTM" Routledge, Taylor & Francis Group 2008
• 39 Hanson, B. A., "An investigation of classification consistency indexes estimated under alternative strong true score models" 27 : 345-359, 1990
• 40 Kane, M. T., "An argument-based approach to validity" 112 : 527-535, 1992
• 41 Wu, J. R. W., "Aligning tests with the CEFR, Studies in Language Testing 33" Cambridge University Press 204-224, 2010
• 42 National Educational Examination Authority, "About CSE"
• 43 Hurtz, G. M., "A meta-analysis of the effects of modifications to the Angoff method on cutoff scores and judgment consensus" 63 : 584-601, 2003
• 44 Berk, R. A., "A consumer’s guide to setting performance standards on criterion-referenced tests" 56 : 137-172, 1986
• 45 Kaufman, D. M., "A comparison of standard-setting procedures for an OSCE in undergraduate medical education" 75 : 267-271, 2000
• 46 Buckendahl, C. W., "A comparison of Angoff and Bookmark standard setting methods" 39 : 253-263, 2002
• 47 이규민, "A Comparison Study on the Applications of the Angoff and Bookmark Standard Setting Methods" 27 (27): 1-17, 2014