피드백 강화학습을 통한 검색 모델 개선 = Improving Retrieval Models through Reinforcement Learning with Feedback

국문 초록 (Abstract)

오픈 도메인 질의응답 작업은 검색을 통해 단서를 얻고 문제를 해결하는 과정이다. 이러한 작업에서 검색 모델이 적절한 단서를 제공하는 것은 매우 중요하며, 이는 최종 성능에 직접적인 영향을 미친다. 또한, 정보 검색은 일상 생활에서도 빈번히 사용되는 중요한 기능이다. 본 논문에서는 이러한 문제의 중요성을 인식하고, 검색 모델의 성능 향상을 목표로 한다. 최근 디코더 모델에서 Reinforcement learning from human feedback(RLHF)을 통해 출력을 조정하는 방식이 자주 사용되고 있는 것처럼, 본 연구에서는 강화학습을 활용하여 검색 모델을 강화하고자 한다. 구체적으로, 답변 모델의 손실과 검색 문서와 정답 문서 간의 유사도라는 두 가지 보상을 정의하고, 이를 바탕으로 강화학습을 적용하여 검색 모델의 문서 확률 분포에서 1위 문서의 확률 점수를 조정한다. 이러한 방법을 통해 강화학습 방법의 일반성과 이를 통한 추가적인 성능 향상을 확인한다.

다국어 초록 (Multilingual Abstract)

Open-domain question answering involves the process of retrieving clues through search to solve problems. In such tasks, it is crucial that the search model provides appropriate clues, as this directly impacts the final performance. Moreover, information retrieval is an important function frequently used in everyday life. This paper recognizes the significance of these challenges and aims to improve performances of search models. Just as the recent trend involves adjusting outputs in decoder models using Reinforcement Learning from Human Feedback (RLHF), this study seeks to enhance search models through the use of reinforcement learning. Specifically, we defined two rewards: the loss of the answer model and the similarity between the retrieved documents and the correct document. Based on these, we applied reinforcement learning to adjust the probability score of the top-ranked document in the search model's document probability distribution. Through this approach, we confirmed the generality of the reinforcement learning method and its potential for further performance improvements.

참고문헌 (Reference)

• 1 criteo, "autofaiss: Automatically create Faiss knn indices with the most optimal similarity search parameters. — github.com"
• 2 C. Wu, "Visual chatgpt: Talking, drawing and editing with visual foundation models" 2023
• 3 G. Izacard, "Unsupervised dense information retrieval with contrastive learning" 2022
• 4 S. Robertson, "Understanding inverse document frequency : on theoretical arguments for idf" 60 (60): 503-520, 2004
• 5 L. Ouyang, "Training language models to follow instructions with human feedback" 35 : 27 730-27 744, 2022
• 6 A. Deshpande, "Toxicity in chatgpt : Analyzing persona-assigned language models"
• 7 S. Robertson, "The probabilistic relevance framework : Bm25 and beyond" 3 (3): 333-389, 2009
• 8 N. Reimers, "Sentence-bert: Sentence embeddings using siamese bert-networks" 2019
• 9 S. Li, "Semireward : A general reward model for semi supervised learning"
• 10 P. Lewis, "Retrieval-augmented generation for knowledge-intensive nlp tasks" 33 : 9459-9474, 2020
• 11 D. Chen, "Reading Wikipedia to answer open-domain questions" 1 : 2017
• 12 L. Wang, "Query2doc: Query expansion with large language models" 9414-9423, 2023
• 13 T. Dettmers, "Qlora : Efficient finetuning of quantized llms"
• 14 J. Schulman, "Proximal policy optimization algorithms"
• 15 R. Nogueira, "Passage re-ranking with bert"
• 16 H. Touvron, "Llama 2 : Open foundation and finetuned chat models"
• 17 G. Izacard, "Leveraging passage retrieval with generative models for open domain question answering" 874-880, 2021
• 18 K. Lee, "Latent retrieval for weakly supervised open domain question answering" 6086-6096, 2019
• 19 T. Brown, "Language models are few-shot learners" 33 : 1877-1901, 2020
• 20 A. Radford, "Improving language understanding by generative pre-training" 2018
• 21 A. Wang, "GLUE : A multi-task benchmark and analysis platform for natural language understanding" 353-355, 2018
• 22 G. Izacard, "Few-shot Learning with Retrieval Augmented Language Models"
• 23 C. Raffel, "Exploring the limits of transfer learning with a unified text-to-text transformer" 21 : 140:1-140:67, 2020
• 24 R. Rafailov, "Direct preference optimization : Your language model is secretly a reward model" 2023
• 25 V. Karpukhin, "Dense passage retrieval for open-domain question answering" 6769-6781, 2020
• 26 Y. LeCun, "Deep learning" 521 (521): 436-444, 2015
• 27 O. Khattab, "Colbert : Efficient and effective passage search via contextualized late interaction over BERT" 39-48, 2020
• 28 "Chatgpt,openai blog"
• 29 J. Johnson, "Billion-scale similarity search with gpus" 7 (7): 535-547, 2019
• 30 X. Wu, "Better aligning text-to-image models with human preference" 2023
• 31 J. D. M. -W. C. Kenton, "Bert : Pretraining of deep bidirectional transformers for language understanding" 4171-4186, 2019
• 32 A. Vaswani, "Attention is all you need" 30 : 2017
• 33 A. Aizawa, "An information-theoretic perspective of tf-idf measures" 39 (39): 45-65, 2003