개방형 설문으로 조사된 직업정보를 이용한 직업분류 코딩 결과의 일치도 평가 = Concordance for job codes which three raters chose based on the information collected by open-ended questionnaire

다국어 초록 (Multilingual Abstract)

Background: Information on occupation is known as one of the important information sources for socioeconomic and epidemiological research. It can be used to assume socioeconomic status and level of physical activity, to forecast the likelihood of a work-related accident or disease, and to assess the existence and degree of exposure to various risk factors in workplace.
Objective: Individual information on occupation is often collected by open-ended questions and then coded into job categories by researchers according to the standard job classification systems. This study aims to assess the reliability among the job codes that three raters independently classified using the information on occupation collected by open-ended questionnaire.
Methods: Information on occupation was collected from two surveys with different populations (145 health examinees and 154 college students of open university) with open-ended questionnaire. Two surveys were basically self-administered, but the health examinee survey questionnaire was checked by a trained interviewer. Two occupational classification systems, i.e., the Korean Standard Classification of Occupational (KSCO, 14 categories) and the Korean Standard Industrial Classification (KSIC, 21 categories) were used for job-coding by three junior researchers in the field of public health, The degree of agreement/concordance was assessed by Cohen’s Kappa or Fleiss Kappa and 95% confidence interval (CI).
Results: Concordance of job codes among three raters for KSCO was moderate for health examinees (k=0.56, 95% CI=0.52-0.58) and fair for open university students (k=0.34, 95% CI=0.31-0.38). That of KSIC was moderate for both health examinees (k=0.60, 95% CI=0.57-0.63) and open university students (k=0.46, 95% CI=0.43-0.49). There was observed a lot of variation in category-specific kappa values for both KSCO and KSIC classification.
Conclusion: Increased concordance of job coding is expected when interview survey is used to collect open-ended questions for the information on occupation and then definite protocol for classification is adopted or to conduct prior education before coding.

국문 초록 (Abstract)

연구목적: 본 연구는 연구자 간 직업분류 결과가 얼마나 일치하는지 확인함으로써 직업분류 과정의 신뢰도 평가를 목적으로 한다.
연구방법: 설문 응답자에게 자신이 경험한 직업 중 가장 오래 종사한 직업을 개방형으로 답하도록 하였고, 세 명의 보건학 전공자에게 독립적으로 설문 답변을 직업분류표의 직업 코드로 변환하도록 하였다. 이후 이들의 분류 결과가 얼마나 일치하는지 평가하기 위해 카파통계량을 구하였다.
주요결과: 면접 설문 요원의 개입이 있었던 병원 내원자들을 설문 대상으로 한 경우(n=145), ‘중간 정도’의 일치도를 보였다(한국표준직업분류 14범주: k=0.56, 95% CI=0.52-0.58, 한국표준산업분류 21범주: k=0.60, 95% CI=0.57-0.63). 면접 설문 요원의 개입이 없었던 원격대학 재학생들을 대상으로 한 경우(n=154), ‘중간 미만’ 이나 ‘중간 정도’의 일치도를 보였다(각각, k=0.34, 95% CI=0.31-0.38, k=0.46, 95% CI=0.43-0.49).
결론: 일치도를 높이기 위해서는 직업 종류에 대한 상세한 이해, 코딩을 위한 세부적인 지침 마련, 사전 교육 등이 필요하며, 직업분류 일치도를 해석할 때는 설문 방법 확인, 설문 집단의 연령 분포, 직업 분포, 전문직의 비율 등 다양한 원인을 고려하여 해석할 필요가 있다.

참고문헌 (Reference)

• 1 주온주 ; 김인자, "한국 표준직업분류에 따른 근로자의 구강기능문제" 한국융합학회 11 (11): 341-347, 2020
• 2 김현정 ; 박창언, "체계적 문헌고찰에서 평가자 간의 신뢰도 측정" 한양대학교 의과대학 35 (35): 44-49, 2015
• 3 권명진 ; 정선경, "직업에 따른 기혼여성의 건강관련 삶의 질 영향요인" 한국산학기술학회 22 (22): 650-659, 2021
• 4 International Labour Organization, "What kind of work do you do? Data collection and processing strategies when measuring" occupation" for statistical surveys and administrative records"
• 5 Institute for Social and Economic Research, "The review of ISCO88: a European perspective"
• 6 Winch RF, "The reliability of respondent-coded occupational prestige" 245-251, 1969
• 7 Landis JR, "The measurement of observer agreement for categorical data" 33 : 159-174, 1977
• 8 Sturgis P, "The Effect of Coding Error on Time Use Survey Estimates" 20 : 467-480, 2004
• 9 Kish L, "Studies of Interviewer Variance for Attitudinal Variables" 57 : 92-115, 1962
• 10 Kong KA, "Statistical methods: reliability assessment and method comparison" 40 (40): 9-16, 2017
• 11 Fleiss JL, "Statistical methods for rates and proportions" John Wiley &Sons 2003
• 12 Layne LA, "Occupational Injury Mortality Surveillance in the United States : An Examination of Census Counts from Two Different Surveillance Systems, 1992–1997" 45 : 1-13, 2004
• 13 Reichard AA, "Occupational Injuries among Emergency Responders" 53 : 1-11, 2010
• 14 Kauppinen T, "Occupational Exposure to Carcinogens in the European Union" 57 : 10-18, 2000
• 15 Cawley JC, "Occupational Electrical Injuries in the United States, 1992–1998, and Recommendations for Safety Research" 34 (34): 241-248, 2003
• 16 Christoph B, "Occupation-Based Measures : An Overview and Discussion" 72 (72): 41-78, 2020
• 17 Free University Amsterdam, "Occupation Coding:Do’s And Dont’s"
• 18 Petrie A, "Medical statistics at a glance" John Wiley & Sons 2009
• 19 University Ca'Foscari of Venice, "Measurement error in occupational coding: an analysis on SHARE data"
• 20 Blalock H, "Making Causal Inferences for Unmeasured Variables from Correlations Among Indicators" 69 : 53-62, 1963
• 21 Biemer PP, "Introduction to Survey Quality" John Wiley & Sons 234-241, 2003
• 22 Vries JD, "Hoe meet ik beroep? Open en gesloten vragen naar beroep toegepast in een statusverwervingsmodel" 83 (83): 71-96, 2008
• 23 Tijdens K, "Dropout rates and response times of an occupation search tree in a web survey" 30 (30): 23-43, 2014
• 24 Statistics Canada, "Automatic Coding of Occupations Using Machine Learning Algorithms for Occupation Coding in Several German Panel Surveys"
• 25 김민선 ; 송기준 ; 남정모 ; 정인경, "A Study on Comparison of Generalized Kappa Statistics in Agreement Analysis" 한국통계학회 25 (25): 719-731, 2012
• 26 Kromhout H, "A Comprehensive Evaluation of Within-and Between-Worker Components of Occupational Exposure to Chemical Agents" 37 : 253-270, 1993
• 27 Cohen J, "A Coefficient of Agreement for Nominal Scales" 20 (20): 37-46, 1960