목적지향 대화시스템에서 LSTM 언어모델 기반의 한국어 자연어 생성 = Korean Natural Language Generation Using LSTM-based Language Model forTask-Oriented Spoken Dialogue System

다국어 초록 (Multilingual Abstract)

Natural language generation in the dialogue system is a task that transforms the semantic frame of the system utterance determined in the dialogue management phase into a natural language that can be understood by humans. Existing studies have still faced some obstacles in that only very limited types of utterances or grammatically incomplete ones are generated from the semantic frames. In order to address these issues simultaneously, we propose a Korean natural language generation model using a long short term memory based language model. In particular, we exploit the beam search decoding method to obtain system utterances with diverse structures and grammatical correctness. The experiments were conducted individually with respect to the word, morpheme, and syllable units, and the generated utterances were evaluated in both quantitative and qualitative ways. As a result, the morpheme-based model with the beam search decoding has achieved the most robust result of all. In fact, in the quantitative evaluation result of the generated sentence, the BLEU-4 score was 0.86 and the SER was 0.03, and the qualitative evaluation was also confirmed to be grammatically correct and contextually natural.

국문 초록 (Abstract)

대화시스템에서 자연어 생성은 대화관리 단계에서 결정한 시스템 발화의 의미표현을 사람이 이해할 수 있는 자연어로 생성하는 것이다. 기존의 자연어 생성 연구는 의미표현에 대하여 매우 제한된 종류의 발화만을 생성하거나 문법적으로 불완전한 발화를 생성한다는 문제점이 있다. 그래서 본 논문에서는 문제점들을 동시에 처리하기 위하여 Long Short Term Memory 기반의 언어모델을 이용한 한국어 자연어 생성 모델을 제안한다. 특히 우리는 시스템 발화의 다양성과 문법적 정확성을 높이기 위하여 빔서치 디코딩을 적용한다. 실험은 어절, 형태소, 음절단위에 따라 개별적으로 진행하였으며, 생성한 문장들은 정량적, 정성적 평가를 모두 진행하였다. 그 결과 형태소 단위로 학습한 제안모델에 빔서치 디코딩을 적용한 방법은 가장 좋은 성능을 보였다. 실제로 해당 생성 문장은 정량평가 결과에서 BLEU 지표는 0.86, Slot Error Rate 지표는 0.03을 기록하였으며 정성평가 역시 문법적으로 정확하고 문맥적으로 충분히 자연스러운 결과임을 확인하였다.

참고문헌 (Reference)

• 1 이형직, "효과적인 정답 추출을 위한 언어 분석 및 질문 패턴 기반의 질문 분할" 한국차세대컴퓨팅학회 10 (10): 31-38, 2014
• 2 강상우, "대화 시스템을 위한 사용자 발화 문장의 감정 분류" 한국인지과학회 21 (21): 459-480, 2010
• 3 이선정, "대용량 고립단어 언어처리를 위한 통계적 방안 연구" 한국차세대컴퓨팅학회 9 (9): 17-24, 2013
• 4 Ye Jia, "Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis" 31 : 4480-4490, 2018
• 5 Tianyu Liu, "Towards Comprehensive Description Generation from Factual Attribute-value Tables" 5985-5996, 2019
• 6 Jason Williams, "The dialog state tracking challenge" 404-413, 2013
• 7 Yuxuan Wang, "Tactron : Towards End-to-End Speech Synthesis" 4006-4010, 2017
• 8 Tsung-Hsien Wen, "Stochastic language generation in dialogue using recurrent neural networks with convolutional sentence reranking" 275-284, 2015
• 9 Francois Mairesse, "Stochastic language generation in dialogue using factored language models" 40 (40): 763-799, 2014
• 10 Alice H. Oh, "Stochastic language generation for spoken dialogue systems" Association for Computational Linguistics 3 : 27-32, 2000
• 11 정민우, "Statistical Spoken Language Understanding for Human-Computer Dialog System" 한국차세대컴퓨팅학회 4 (4): 40-48, 2008
• 12 Dan Jurafsky, "Speech and language Processing. Vol 3" Pearson 2014
• 13 Alex Graves, "Sequence transduction with recurrent neural networks" 2012
• 14 Tsung-Hsien Wen, "Semantically conditioned lstm-based natural language generation for spoken dialogue systems" 1711-1721, 2015
• 15 Tomas Mikolov, "Recurrent neural network based language model" 2 : 1045-1048, 2010
• 16 Van-Khanh Tran, "Natural Language Generation for Spoken Dialogue System using RNN Encoder-Decoder Networks" 442-451, 2017
• 17 Tsubasa Ochiai, "Multichannel End-to-end Speech Recognition" 70 : 2632-2641, 2017
• 18 Adam Cheyer, "Method and apparatus for building an intelligent automated assistant"
• 19 Sepp Hochreiter, "Long short-term memory" 9 (9): 1735-1780, 1997
• 20 Martin Sundermeyer, "LSTM neural networks for language modeling" 194-197, 2012
• 21 Xiang Gao, "Jointly Optimizing Diversity and Relevance in Neural Response Generation" 1 : 1229-1238, 2019
• 22 Sepp Hochreiter, "Gradient flow in recurrent nets: the difficulty of learning long-term dependencies" IEEE Press 237-244, 2000
• 23 Mrkšić, Nikola, "Fully statistical neural belief tracking" 2 : 108-113, 2018
• 24 Yi Ren, "FastSpeech : Fast, Robust and Controllable Text to Speech" 32 : 3171-3180, 2019
• 25 Amanda Stent, "Evaluating evaluation methods for generation in the presence of variation" 341-351, 2005
• 26 Dongling Xiao, "ERNIE-GEN : An Enhanced Multi-Flow Pretraining and Fine-tuning Framework for Natural Language Generation"
• 27 Danilo Mirkovic, "Dialogue management using scripts"
• 28 Kishore Papineni, "BLEU : a method for automatic evaluation of machine translation" Association for Computational Linguistics 311-318, 2002
• 29 Altinok Duygu, "An Ontology-Based Dialogue Management System for Banking and Finance Dialogue Systems"
• 30 Michael C. Mozer, "A focused backpropagation algorithm for temporal pattern recognition" 3 : 137-169, 1995