생성 AI와 음성 인식을 활용한 메타버스 가상 면접 플랫폼 = Metaverse Virtual Interview Platform Leveraging Generative AI and Speech Recognition

국문 초록 (Abstract)

본 논문에서 제안하는 새로운 가상 면접 플랫폼은 메타버스와 인공지능의 결합을 통해 구현된 시스템이다. 음성 생성 및 합성, GPT 기반의 질문 생성 기술을 활용하여, 사용자와 인터뷰 모델 간의 상호작용을 강화하고 감정이 담긴 음성 데이터를 통한 면접을 제공한다. GPT-4를 기반으로 하는 면접 질문 생성은 역량과 잠재 능력을 효과적으로 평가하기 위해 프롬프트 엔지니어링을 적용한다. 가상 면접 환경은 유니티 엔진을 통해 구축되며, 디지털 휴먼을 활용한 가상 면접관과 립싱크 기술의 적용으로 실제 면접과 유사한 상황을 보여준다. 제안 방법은 답변을 분석해 상세한 면접 결과를 제공하는 기능이 포함되기 때문에, 참여자들이 실제 면접 전에 자신의 능력을 점검하고 면접 기술을 향상시킬 수 있는 기회를 제공할 수 있다.

다국어 초록 (Multilingual Abstract)

This paper proposes a new virtual interviewing platform, implemented through the combination of the metaverse and artificial intelligence. Utilizing voice synthesis and generation, along with GPT-based question generation techniques, it enhances the interaction between users and the interview model and provides an interview experience enriched with emotion-laden voice data. The GPT-4 based interview question generation employs prompt engineering to effectively assess competencies and potential abilities. The virtual interviewing environment is built using the Unity engine and features digital human interviewers with lipsync technology, creating a scenario akin to real-life interviews. The proposed method includes a feature for analyzing responses to provide detailed interview results, thereby offering participants the opportunity to assess their capabilities and enhance their interview skills before the actual interview.

참고문헌 (Reference)

• 1 A. Baevski, "wav2vec 2. 0 : A Framework for Self-Supervised Learning of Speech Representations" 12449-12460, 2020
• 2 "uLipSync"
• 3 J. Park, "TEA-VITS :Emotion Voice Synthesis" 2023
• 4 Y. Wang, "Speech Emotion Diarization: Which Emotion Appears When?" 1-7, 2023
• 5 "SALSA Lipsync suite"
• 6 "Radar Chart"
• 7 J. Shen, "Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions" 4799-4783, 2018
• 8 J. Si, "Metaverse Interview Room Creation with Virtual Interviewer Generation Using Diffusion Model" 1-4, 2023
• 9 "Lipsync pro"
• 10 V. Popov, "Grad-TTS : A Diffusion Probabilistic Model for Text-to-Speech" 8599-8608, 2021
• 11 J. Kim, "Glow-TTS : A Generative Flow for Text-to-Speech via Monotonic Alignment Search" 33 : 8067-8077, 2020
• 12 J. Si, "Generative AI models for virtual interviewers : applicability and performance comparison" 27-28, 2023
• 13 J. Achiam, "GPT-4 Technical Report"
• 14 M. M. Robles, "Executive Perception of the Top 10 Soft Skills Needed in Today’s Workplace" 75 (75): 453-465, 2012
• 15 S. Park, "Dimensional Emotion Detection from Categorical Emotion" 4367-4380, 2021
• 16 B. C. Lee, "Development of an AI-based interview system for remote hiring" 12 (12): 654-663, 2021
• 17 S. Yang, "Development of a Unity-Based Interview room Prototype for Virtual Interviews in a Metaverse Environment" 31 (31): 225-226, 2023
• 18 C. Yoon, "Constructing a Semi-automatic Instruction Set for Effective AI interview Models" 408-410, 2023
• 19 F. Chopra, "Conducting Qualitative Interviews with AI, CESifo Working Paper, No. 10666"
• 20 J. Kim, "Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech" 139 : 5530-5540, 2021
• 21 "Character creator"
• 22 J. Kum, "Can Gestural Filler Reduce User-Perceived Latency in Conversation with Digital Humans" 12 (12): 10972-10988, 2022
• 23 "Business Office"
• 24 유기중 ; 박기범 ; 조성원 ; 김동호, "A Study on the Application Direction of Financial Industry Metaverse Platform to secure MZ Generation Contact Points" 23 (23): 127-137, 2023
• 25 이재성 ; 김주연, "A Study on Spatial Characteristics of Immersion and Reality in Cases of VR and AR Technology and Contents" 28 (28): 23-, 2019
• 26 양진 ; 곽경민 ; 노영주, "A Study on Metaverse Framework Design for Education and Training of Hydrogen Fuel Cell Engineers" 24 (24): 207-212, 2024
• 27 X. Huang, "A Process for the Semi-Automated Generation of Life-Sized, Interactive 3D Character Models for Holographic Projection" 1-8, 2019
• 28 N. G. Narin, "A Content Analysis of the Metaverse Articles" 1 (1): 17-24, 2021