대단위 쓰기 평가 설계를 위한 평가 체제 검토 -SAT essay와 GRE Analytical Writing을 중심으로- = Examination of Assessment Systems for Large-Scale Writing Assessment Design -focused on the SAT Essay and GRE Analytical Writing-

국문 초록 (Abstract)

대단위 쓰기 평가는 교실 쓰기 평가와 대별되어 특정 규모 이상의 학습자를 대상으로 하는 직접 평가 방식의 쓰기 평가를 말한다. 서논술형 평가에 대한 사회적 요구가 강해지는 가운데, 타당도와 신뢰도, 공정성을 확보한 대단위 쓰기 평가를 개발하기 위한 체계적 연구가 선행되어야 한다. 이 연구의 목적은 평가틀의 관점에서 대학 이상 기관의 선발 시험에서 시행한 대단위 쓰기 평가 체제를 검토하여 대단위 쓰기 평가의 체제 개발의 시사점을 도출하는 것이다. 이를 위해 미국의 SAT와 GRE의 쓰기 평가 체제와 개발 과정을 검토하여 시사점에 대해 논의하였다. 대단위 쓰기 평가의 설계를 위해 증거 중심 설계와 평가 공학적 접근의 활용 방안, 다양한 심리측정학적 접근 방법을 활용한 대단위 쓰기 평가 설계 연구 방안을 제시하였고, 공정성 확보를 위한 실질적 대안을 논의하였다.

다국어 초록 (Multilingual Abstract)

Large-scale writing assessments differentiate from classroom writing assessments as they are a direct assessment method targeting a significant number of learners. With the increasing societal demand for essay type assessments, systematic research must precede the development of large-scale writing assessments, ensuring validity, reliability, and fairness. This study aims to review the assessment systems of large-scale writing assessments administered in college-level selection exams, extracting implications for the development of a large-scale writing assessment framework. In pursuit of this objective, the study examined and discussed the implications of the writing assessment systems and development processes of the United States' SAT and GRE. For the design of the large-scale writing assessments, this study proposed research methods utilizing evidence-centered design, assessment engineering approaches, and various psychometric approaches. It also discussed practical alternatives to secure fairness.

참고문헌 (Reference)

• 1 한국교육과정평가원, "대학수학능력시험의 성과와 발전 방향" 한국교육과정평가원 2020
• 2 장성민 ; 민병곤, "대학수학능력시험 “국어” 작문 영역 평가의 타당성 검토 - 선다형 점수와 수행형 점수의 상관관계를 중심으로 -" 한국작문학회 (29) : 163-209, 2016
• 3 이영진, "대단위 작문 평가를 위한 문항 개발과 채점 방법 연구" 한국교원대학교 대학원 2012
• 4 정혜진 ; 지은림, "국어 교과 ‘비판적·창의적 사고 역량’ 측정도구 개발 및 타당화에 대한 탐색적 연구" 한국교육평가학회 33 (33): 511-532, 2020
• 5 채정관, "국가영어능력평가시험(NEAT) 정책의 실패 원인에 대한 시차적 접근" 한국교육정치학회 22 (22): 1-32, 2015
• 6 McNamara, T. F., "‘Interaction’ in second language performance assessment:whose performance?" 18 : 446-466, 1997
• 7 Newton, S., "Utilizing evidence-centered design to develop assessments: A high school introductory computer science course" Frontiers Media SA 6 : 695376-, 2021
• 8 Brentani, E., "Unidimensionality in the Rasch model : how to detect and interpret" 67 (67): 253-261, 2007
• 9 Zhao, J. C., "Timing configuration and psychometric properties of the redesigned Analytical Writing measure of the GRE General Test" 2006
• 10 Wendler, Cathy, "The Research Foundation for the GRE revised General Test: A Compendium of Studies" Educational Testing Service 2014
• 11 College Board, "Test specifications for the redesigned SAT" College Board 24-38, 2014
• 12 College Board, "Test Specifications for the Redesigned SAT"
• 13 American Educational Research Association, "Standards for educational and psychological testing"
• 14 ETS News & Insights, "Shorter GRE® Test Coming September 2023"
• 15 College Board, "SAT® suite of assessments technical manual: Characteristics of the SAT"
• 16 College Board, "SAT® suite of assessments technical manual Appendixes"
• 17 SAT Scores, "SAT Essay Scoring"
• 18 Schaeffer, G. A., "Psychometric Evaluation of the New GRE® Writing Assessment" ETS i-53, 2001
• 19 Lane, S., "Performance assessment: The state of the art (SCOPE Student Performance Assessment Series)" Stanford University, Stanford Center for Opportunity Policy in Education 2010
• 20 ETS, "Overview of the Analytical Writing Section"
• 21 The GRE® General Test, "Overview of the Analytical Writing Measure"
• 22 Dana Gioia, "Meanwhile: When reading wanes, it's time to worry"
• 23 Linacre, J. M., "Many-faceted Rasch measurement" The University of Chicago 1989
• 24 Zieky, M. J., "Looking Back" Springer 115-127, 2011
• 25 McNamara, T. F., "Language assessment as social practice : Challenges for research" 18 (18): 333-349, 2001
• 26 Bejar, I. I., "Item Generation for Test Development" Erlbaum 199-217, 2002
• 27 Kim, S., "Investigating the effectiveness of equating designs for constructed response tests in large scale assessment" 47 : 186-201, 2010
• 28 ETS, "Introduction to the Analytical Writing measure"
• 29 Mislevy, R. J., "Implications of evidence-centered design for educational testing" 25 (25): 6-20, 2006
• 30 Rosenfeld, M., "Identifying the writing tasks important for academic success at the undergraduate and graduate levels" ETS i-91, 2004
• 31 College board, "Helping Higher Education Understand the SAT Essay"
• 32 Embretson, S., "Handb. Statist. Psychometr. 26" 747-768, 2006
• 33 ETS, "GRE® Guide to the Use of Scores"
• 34 ETS, "GRE® Guide to the Use of Scores"
• 35 Mislevy, R. J., "Focus article : On the structure of educational assessments" 1 (1): 3-62, 2003
• 36 College Board, "Fairness on the SAT Suite of Assessments" College Board 2021
• 37 Almond, R. G., "Enhancing the design and delivery of assessment systems: A four-process architecture" 1 (1): 2002
• 38 Messick, S., "Educational measurement" Macmillan 13-103, 1989
• 39 Sehmitt, A. P., "Differential item functioning for minority examinees on the SAT" 27 (27): 67-81, 1990
• 40 Zenisky, A. L., "Detection of differential item functioning in large-scale state assessments : A study evaluating a two-stage approach" 63 (63): 51-64, 2003
• 41 Powers, D. E., "Correlates of satisfaction with graduate school applicants’ performance on the GRE Writing measure" Educational Testing Service 1997
• 42 Bridgeman, B., "Comparability of essay question variants" 16 (16): 237-255, 2011
• 43 Luecht, R. M., "Automatic item generation: Theory and practice" 59-76, 2013
• 44 Gierl, M. J., "Automatic Item Generation : Theory and Practice" Routledge 2013
• 45 Yu, C. H., "Assessing unidimensionality: A comparison of Rasch modeling, parallel analysis, and TETRAD" 12 (12): 14-, 2007
• 46 "Analytical Writing―Analyze an Issue, Pool of Issue Topics"
• 47 "Analytical Writing―Analyze an Argument, Pool of Argument Topics"
• 48 Uto, M., "A multidimensional generalized many-facet Rasch model for rubric-based performance assessment" 48 (48): 425-457, 2021
• 49 Uto, M., "A generalized many-facet Rasch model and its Bayesian estimation using Hamiltonian Monte Carlo" 47 : 469-496, 2020
• 50 Mislevy, R. J., "A brief introduction to evidence-centered design" ETS i-29, 2003
• 51 교육부, "2028 수능 국·수·탐 선택과목 없이 통합 평가 학업포기 내모는 내신 9등급제, 2025부터 5등급 체제로"
• 52 Robin, F., "2.3 Evaluating and Maintaining the Psychometric and Scoring Characteristics of the Revised Analytical Writing Measure. A Compendium of Studies"