Sequence-to-text 기법을 활용한 언어 모형의 소개 및 분류 성능 비교 = Comparison and application of sequence-to-text based language models

국문 초록 (Abstract)

최근에 활발한 언어 모형의 개발과 함께 많은 분야의 연구자들이 이에 관심을 갖게 되었다. 언어 모형의 사용처는 매우 다양하여 많은 분야에 사용될 수 있다. 전통적인 통계학이나 머신 러닝의 분야에서는 이진 분류에 대한 연구가 활발하게 진행되어 왔고, 언어 모형들 역시 동일한 과제를 수행할 수 있다. 언어 모형의 특징으로 인해 입력 데이터는 테이블 형식의 숫자로 이루어진 데이터가 아니라 문자 형식의 변수를 입력해야 한다. 이 논문에서는 열 가지의 최신 언어 모형들의 목적과 특징들을 소개한다. 한편, 언어 모형의 데이터 입력을 위해 시퀀스 데이터를 텍스트로 변환하는 sequence-to-text 방법의 여러가지 예시를 구체적으로 제시하고, 예제 코드가 담긴 링크도 제공한다. 이를 통해 언어 모형 구현에 경험이 없는 독자들에게는 도움이 될 것으로 기대한다. 마지막으로, 제시한 여러가지 sequence-to-text 방법을 실제 데이터를 이용한 모의 실험을 통하여 이 논문에서 소개하는 최신 언어 모형들의 성능을 비교하고 머신 러닝 모형들의 결과도 함께 제공하여 언어 모형들의 우수한 분류 성능을 확인한다.

다국어 초록 (Multilingual Abstract)

Recently, the active development of language models has garnered significant interest from researchers across various fields. The applications of language models are exceedingly diverse, making them suitable for a multitude of domains. In traditional statistics and machine learning, extensive research has been conducted on binary classification, a task that language models can also perform effectively. Due to the inherent characteristics of language models, the input data required is not merely numerical in tabular form, but rather comprises textual variables. This paper presents an overview of the objectives and features of ten state-of-the-art language models. Additionally, various examples of sequence-to-text methods for transforming sequence data into textual formats are detailed to assist readers who may lack experience in implementing language models. Finally, through a series of simulations utilizing actual data, this study compares the performance of the aforementioned modern language models across several sequence-to-text methods, while also providing results from machine learning models to demonstrate the superior classification performance of language models.

참고문헌 (Reference)

• 1 Wang, T., "player2vec : A language modeling approach to understand player behavior in games"
• 2 Tang, G., "Why self-attention? A targeted evaluation of neural machine translation architectures"
• 3 Xu, Z., "Using large pre-trained language models to assist FDA in premarket medical device"
• 4 이승언 ; 김현세 ; 김남호 ; 엄정윤 ; 우지환, "Study on the real owner distinction in financial transactions using deep learning" 32 : 781-797, 2021
• 5 Joshi, M., "Spanbert : Improving pre-training by representing and predicting spans" 8 : 64-77, 2020
• 6 Bhowmik, S., "Sentiment analysis with hotel customer reviews using FNet" 13 : 1298-1306, 2024
• 7 Liu, Y., "Roberta : A robustly optimized bert pretraining approach"
• 8 Kitaev, N., "Reformer: The efficient transformer" 2020
• 9 Mikolov, T., "Recurrent neural network based language model. Interspeech 2010" 1045-1048, 2010
• 10 권순보 ; 유진은, "Online career counseling text classification using BERT and FastText" 33 : 991-1006, 2022
• 11 Song, K., "MPNet: Masked and permuted pre-training for language understanding" 33 : 16857-16867, 2020
• 12 Loh, N. K. N., "MPNet-GRUs : Sentiment analysis with masked and permuted pre-training for language understanding and gated recurrent units" 12 : 74069-74080, 2024
• 13 Beltagy, I., "Longformer : The long-document transformer"
• 14 Tay, Y., "Long range arena : A benchmark for efficient transformers"
• 15 Wang, Y., "LoBERTa : A composition named entity recognition method based on longformer and DeBERTa model" 266-270, 2022
• 16 Wu, Y., "Google’s neural machine translation system : Bridging the gap between human and machine translation"
• 17 Dai, Z., "Funnel-transformer: Filtering out sequential redundancy for efficient language processing" 33 : 4271-4282, 2020
• 18 Riaz, N., "From transformers to reformers" 1-6, 2021
• 19 Lee-Thorp, J., "FNet : Mixing tokens with Fourier transforms"
• 20 Cortiz, D., "Exploring transformer models for emotion recognition: A comparison of BERT, Distil-BERT, RoBERTa, XLNET and ELECTRA" 230-234, 2022
• 21 Bojanowski, P., "Enriching word vectors with subword information" 5 : 135-146, 2017
• 22 Wu, S., "Enriching pre-trained language model with entity information for relation classification" 2361-2364, 2019
• 23 Chen, Q., "End-to-end user behavior retrieval in clickthrough rate prediction model"
• 24 Clark, K., "ELECTRA: Pre-training text encoders as discriminators rather than generators" 2020
• 25 Sanh, V., "DistilBERT, a distilled version of BERT : Smaller, faster, cheaper, and lighter"
• 26 Samadi, M., "Deep contextualized text representation and learning for fake news detection" 58 : 102723-, 2021
• 27 He, P., "DeBERTaV3 : Improving DeBERTa using ELECTRA-style pretraining with gradient-disentangled embedding sharing"
• 28 Brown, P. F., "Class-based n-gram models of natural language" 18 : 467-479, 1992
• 29 Zaheer, M., "Big bird: Transformers for longer sequences" 33 : 17283-17297, 2020
• 30 Lewis, M., "Bart : Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension"
• 31 Devlin, J., "BERT : Pre-training of deep bidirectional transformers for language understanding"
• 32 Vaswani, A., "Attention is all you need" 5998-6008, 2017
• 33 Liu, Z., "Anomaly detection of command shell sessions based on DistilBERT : Unsupervised and supervised approaches"
• 34 Zhang, S., "An emotional classification method of Chinese short comment text based on ELECTRA" 34 : 254-273, 2022
• 35 Gasparetto, A., "A survey on text classification algorithms : From text to predictions" 13 : 83-, 2023
• 36 Bengio, Y., "A neural probabilistic language model" 3 : 1137-1155, 2003