Validation of the criterion-referenced English assessment in the college scholastic ability test (CSAT) in Korea
저자
발행사항
Seoul : Graduate School, Korea University, 2018
학위논문사항
학위논문(박사)-- 고려대학교 대학원: 영어교육학과 2018. 8
발행연도
2018
작성언어
영어
주제어
발행국(도시)
서울
형태사항
xiv, 285 p. : 삽화, 도표 ; 26 cm
일반주기명
지도교수: 최인철
부록수록
참고문헌: p. 219-229
UCI식별코드
I804:11009-000000081915
DOI식별코드
소장기관
ABSTRACT
Validation of the Criterion-referenced English Assessment in the College Scholastic Ability Test (CSAT) in Korea
Wonhwa Seo
This study is aimed at validating the English domain of the newly implemented Criterion-referenced College Scholastic Ability Test (CR-CSAT). As the English section of CSAT greatly influences on test-takers, English classrooms, and general English education in Korea, validation of the CSAT is required for successful implementation of the CR-CSAT.
This study investigated the new criterion-referenced English test in the CSAT (CR-CSAT) in terms of the framework of test usefulness by Bachman and Palmer (1996). Usefulness consisted of six qualities: reliability, construct validity, authenticity, interactiveness, impact, and practicality. The study first explained the specifications of the test, including test environment, format, and language input, and evaluated its usefulness regarding five tests qualities of the CR-CSAT: reliability, construct validity, authenticity, interactiveness, and the impact on test-takers, teachers, and education.
To evaluate construct validity of the CSAT, CSAT samples from 2014 to 2018 were selected and examined. Regarding lexical input over time, there were fluctuations observed in tokens, types, and type token ratio of all sections examined. Such fluctuations and inconsistencies have an adverse effect on the exam fairness and reliability as they fail to consider that the same test-taker who did well in one test could face failure in the next. The task types employed throughout the exams are very limited: both tests include multiple-choice questions which may enhance objectivity and reliability but fail to measure ‘true communicative skills.’ The language input of listening subtest was composed of mainly low level vocabulary while the vocabulary of reading subtest was high.
To examine reliability of the CR-CSAT, the researcher conducted a number of analyses. Grade level and test score data from 1,226 students who participated in the simulated CSAT in June, 1,218 students who participated in the simulated CSAT administered in September, and 1,195 students who took the 2018 CSAT were analyzed. Analyses included qualitative analyses of test content and quality, a correlation analysis of the test scores, and an item analysis based on CTT and IRT collected from the two mock CSATs and the 2018 CSAT. Results of this study based on CTT and IRT analyses revealed that the difficulty and discrimination of the listening subtest items were very low whereas some reading test items were extremely difficult but their discrimination power was very low. These items need revision. Items with the low degree of difficulty and discrimination were allocated three points while some difficult items were assigned two points. As a result of this problematic score scaling, test-takers within a higher ability parameter obtained a lower scaled score than those within a lower ability parameter. It is desirable to revise test items which are extremely easy or difficult and inappropriately discriminate in order to design more reliable test items. Three native English-speaking instructors were invited to examine the test items of the CSATs from 2014 to 2018. The results revealed that although the CSAT had relatively high internal reliability, the consistency between the tests of different years and the test discriminability did not reach a satisfactory level. Some test items that require revision were found through both quantitative and qualitative item analyses.
For the evaluation of the authenticity and interactiveness analysis, six raters were asked to score the degree of authenticity and interactiveness of test item types of the 2018 CSAT and indirect speaking type was found to have the lowest degree of authenticity and interactiveness. On the other hand, the item types of understanding details and understanding context in both listening and reading comprehension displayed high degrees of authenticity and interactiveness. Overall the reading subtests’ authenticity and interactiveness were higher than those of the listening subtest. Raters’ qualitative review supported the results of quantitative analysis. Concerning the representation of national curriculum, the subject matters of the listening subtest were confined to general and familiar topics whereas subject matters of the reading subtest were concentrated on specified academic topics such as social studies or natural sciences. A diversification of topics needs to be considered when designing the CSAT.
Finally, a survey of teachers’ and students’ perceptions toward the 2018 CR-CSAT was conducted using a questionnaire survey collected from 372 high school students and 102 secondary school teachers. A descriptive analysis of the questionnaire results was completed. It indicated that the CSAT has substantial impacts on English teaching and learning in high schools. The survey results revealed that high school students and English teachers’ perception of the CSAT in terms of test usefulness: reliability, construct validity, authenticity, interactiveness, and impact. One noticeable result was that the two groups both did not think that the 2018 CSAT contributed the equality of English education, decreased private education cost, or changed language teaching and learning. From the qualitative analysis of the high school English teachers’ responses, the number of students who applied to English in after-school program reduced significantly after the implementation of the 2018 CSAT. Also, the number of English class hours decreased. English teachers who participated in the written interview provided alternatives for successful implementation of the CR-CSAT and normalization of the public English education.
The findings from this study provide opportunities to explore pedagogical implications for entrance test developers and education policy makers in Korea in regards to exam quality and methods. The CSAT should be developed by including test items more related to the current national English curriculum and by fulfilling the need for construct validity, authenticity, and positive impacts on both candidates and society. It is important that the various voices of the stakeholders are known to the public in order to promote further discussion on how to improve test quality and to normalize public English education. This study will contribute to validate the first implemented 2018 CR-CSAT based on the Bachman & Palmer’s test usefulness framework. Finally, it is hoped that the study’s comprehensive investigation of the test qualities of construct validity, reliability, authenticity, interactiveness, and impact by analyzing quantitative and quantitative data will raise the awareness of different stakeholders of the CSAT.
분석정보
서지정보 내보내기(Export)
닫기소장기관 정보
닫기권호소장정보
닫기오류접수
닫기오류 접수 확인
닫기음성서비스 신청
닫기음성서비스 신청 확인
닫기이용약관
닫기학술연구정보서비스 이용약관 (2017년 1월 1일 ~ 현재 적용)
학술연구정보서비스(이하 RISS)는 정보주체의 자유와 권리 보호를 위해 「개인정보 보호법」 및 관계 법령이 정한 바를 준수하여, 적법하게 개인정보를 처리하고 안전하게 관리하고 있습니다. 이에 「개인정보 보호법」 제30조에 따라 정보주체에게 개인정보 처리에 관한 절차 및 기준을 안내하고, 이와 관련한 고충을 신속하고 원활하게 처리할 수 있도록 하기 위하여 다음과 같이 개인정보 처리방침을 수립·공개합니다.
주요 개인정보 처리 표시(라벨링)
목 차
3년
또는 회원탈퇴시까지5년
(「전자상거래 등에서의 소비자보호에 관한3년
(「전자상거래 등에서의 소비자보호에 관한2년
이상(개인정보보호위원회 : 개인정보의 안전성 확보조치 기준)개인정보파일의 명칭 | 운영근거 / 처리목적 | 개인정보파일에 기록되는 개인정보의 항목 | 보유기간 | |
---|---|---|---|---|
학술연구정보서비스 이용자 가입정보 파일 | 한국교육학술정보원법 | 필수 | ID, 비밀번호, 성명, 생년월일, 신분(직업구분), 이메일, 소속분야, 웹진메일 수신동의 여부 | 3년 또는 탈퇴시 |
선택 | 소속기관명, 소속도서관명, 학과/부서명, 학번/직원번호, 휴대전화, 주소 |
구분 | 담당자 | 연락처 |
---|---|---|
KERIS 개인정보 보호책임자 | 정보보호본부 김태우 | - 이메일 : lsy@keris.or.kr - 전화번호 : 053-714-0439 - 팩스번호 : 053-714-0195 |
KERIS 개인정보 보호담당자 | 개인정보보호부 이상엽 | |
RISS 개인정보 보호책임자 | 대학학술본부 장금연 | - 이메일 : giltizen@keris.or.kr - 전화번호 : 053-714-0149 - 팩스번호 : 053-714-0194 |
RISS 개인정보 보호담당자 | 학술진흥부 길원진 |
자동로그아웃 안내
닫기인증오류 안내
닫기귀하께서는 휴면계정 전환 후 1년동안 회원정보 수집 및 이용에 대한
재동의를 하지 않으신 관계로 개인정보가 삭제되었습니다.
(참조 : RISS 이용약관 및 개인정보처리방침)
신규회원으로 가입하여 이용 부탁 드리며, 추가 문의는 고객센터로 연락 바랍니다.
- 기존 아이디 재사용 불가
휴면계정 안내
RISS는 [표준개인정보 보호지침]에 따라 2년을 주기로 개인정보 수집·이용에 관하여 (재)동의를 받고 있으며, (재)동의를 하지 않을 경우, 휴면계정으로 전환됩니다.
(※ 휴면계정은 원문이용 및 복사/대출 서비스를 이용할 수 없습니다.)
휴면계정으로 전환된 후 1년간 회원정보 수집·이용에 대한 재동의를 하지 않을 경우, RISS에서 자동탈퇴 및 개인정보가 삭제처리 됩니다.
고객센터 1599-3122
ARS번호+1번(회원가입 및 정보수정)