생성형 인공지능의 수학 문제 풀이에 대한 성능 분석: ChatGPT 4, Claude 3 Opus, Gemini Advanced를 중심으로 = Analysis of generative AI’s mathematical problemsolving performance: Focusing on ChatGPT 4, Claude 3 Opus, and Gemini Advanced

국문 초록 (Abstract)

디지털・AI 기반 교수・학습이 강조됨에 따라 생성형 AI의 교육적 활용에 대한 논의가 활발해지고 있다. 본 연구는 고등학교 1학년 수학 교과서 5종의 예제와 문제 풀이에 대한 ChatGPT 4, Claude 3 Opus, Gemini Advanced의 수학적 성능을 분석하였다. 총 1,317개 문항에 대해 전체 정답률과 기능별 특징을 살펴본 결과, ChatGPT 4의 전체 정답률이 0.85로가장 높았고, Claude 3 Opus가 0.67, Gemini Advanced가 0.42 순으로 나타났다. 기능별로는 함수 구하기와 증명하기에서 세 모델 모두 높은 정답률을 보였으나, 설명하기와 그래프 그리기에서는 상대적으로 낮은 정답률을 보였다. 특히 경우의 수 세기에서 ChatGPT 4와 Claude 3 Opus가 1.00의 정답률을 보인 반면, Gemini Advanced는 0.56으로 낮았다. 또한 모든 모델이 벤 다이어그램을 이용한 설명하기와 이미지 생성이 필요한 문제에서 어려움을 겪었다. 연구 결과를 바탕으로 교사들은 각 AI 모델의 강점과 한계를 파악하고 이를 수업에 적절히 활용할 수 있을 것이다. 본 연구는 생성형 AI의 수학적 성능을 분석함으로써, 실제 수학 수업에서의 생성형 AI의 활용 가능성을 제시했다는 점에서 의의가 있다. 또한 인공지능시대의 수학 교육에서 교사의 역할을 재정립하는 데 중요한 시사점을 제공하였다. 향후 생성형 AI와 교사의 협력적 교육 모델 개발, AI를 활용한 개별화 학습 방안 연구 등이 필요할 것이다.

다국어 초록 (Multilingual Abstract)

As digital·AI-based teaching and learning is emphasized, discussions on the educational use of generative AI are becoming more active. This study analyzed the mathematical performance of ChatGPT 4, Claude 3 Opus, and Gemini Advanced on solving examples and problems from five first-year high school math textbooks. As a result of examining the overall correct answer rate and characteristics of each skill for a total of 1,317 questions, ChatGPT 4 had the highest overall correct answer rate of 0.85, followed by Claude 3 Opus at 0.67, and Gemini Advanced at 0.42.
By skills, all three models showed high correct answer rates in ‘Find functions’ and ‘Prove’, while relatively low correct answer rates in ‘Explain’ and ‘Draw graphs’. In particular, in ‘Count’, ChatGPT 4 and Claude 3 Opus had a correct answer rate of 1.00, while Gemini Advanced was low at 0.56. Additionally, all models had difficulty in explaining using Venn diagrams and creating images. Based on the research results, teachers should identify the strengths and limitations of each AI model and use them appropriately in class. This study is significant in that it suggested the possibility of use in actual classes by analyzing the mathematical performance of generative AI. It also provided important implications for redefining the role of teachers in mathematics education in the era of artificial intelligence. Further research is needed to develop a cooperative educational model between generative AI and teachers and to study individualized learning plans using AI.

참고문헌 (Reference)

• 1 Galatolo, F. A., "Zero-shot mathematical problem solving via generative pre-trained transformers" 1 : 479-483, 2022
• 2 이상구 ; 박도영 ; 이재윤 ; 임동선 ; 이재화, "Use of ChatGPT in college mathematics education" 63 (63): 123-138, 2024
• 3 Govender, R., "The impact of artificial intelligence and the future of ChatGPT for mathematics teaching and learning in schools and higher education" 44 (44): 1-2, 2023
• 4 Wong, T. K. Y., "Teacher support in learning : Instrumental and appraisal support in relation to math achievement" 28 (28): 202-219, 2018
• 5 Ministry of Education, "Support plan for strengthening education innovation capacity based on digital technology" Ministry of Education
• 6 Mhlanga, D., "Open AI in education, the responsible and ethical use of ChatGPT towards lifelong learning. In FinTech and artificial intelligence for sustainable development: The role of smart technologies in achieving development goals" Springer 387-409, 2023
• 7 Vuorikari, R, "Measuring digital skills across the EU: Digital Skills Indicator 2.0"
• 8 Ministry of Education, "Mathematics curriculum. Bulletin of MOE No. 2022-33 [Separate Volume #8]"
• 9 Ministry of Education, "Mathematics Curriculum . Bulletin of MOE No. 2015-74 [Separate Volume #8]"
• 10 Frieder, S., "Mathematical capabilities of ChatGPT"
• 11 Kim, J., "Leading teachers’ perspective on teacher-AI collaboration in education" 1-32, 2023
• 12 CTOL Editors, "Latest LLM market share Mar 2024: ChatGPT leads, Gemini surges, and Claude triples"
• 13 Zhang, Z., "Large language models for spatial trajectory patterns mining"
• 14 Brown, T., "Language models are few-shot learners" 33 : 1877-1901, 2020
• 15 Lv, H. Z., "Innovative music education : Using an AI-based flipped classroom" 28 (28): 15301-15316, 2023
• 16 Lee, J., "High school mathematics" Chunjae 2018
• 17 Kim, W., "High school mathematics" Visang 2018
• 18 Ko, S., "High school mathematics" Sinsago 2018
• 19 Hong, S., "High school mathematics" Jihaksa 2018
• 20 Ryu, H., "High school mathematics" Chunjae 2018
• 21 손태권, "Exploring the possibility of using ChatGPT in mathematics education : Focusing on student product and pre-service teachers’ discourse related to fraction problems" 26 (26): 99-113, 2023
• 22 Supriyadi, E., "Exploring the future of mathematics teaching : Insight with ChatGPT" 11 (11): 305-316, 2023
• 23 오세준, "Effective ChatGPT prompts in mathematical problem solving : Focusing on quadratic equations and quadratic functions" 37 (37): 545-467, 2023
• 24 고보경 ; 임철일 ; 신병철, "Development of a math-AI convergence instructional model using a generative AI chatbot" 40 (40): 1-40, 2024
• 25 Zhu, Y., "ChatGPT/aigc and educational innovation : Opportunities, challenges, and the future" 41 (41): 1-, 2023
• 26 윤정은 ; 박수민 ; 권오남, "ChatGPT-flipped mathematics class case study : Focused on learners’ engagement" 39 (39): 1011-1047, 2023
• 27 Zafrullah, Z., "ChatGPT open AI : Analysis of mathematics education students learning interest" 1 (1): 1-10, 2023
• 28 안도연 ; 손태권 ; 이광호, "ChatGPT as a scaffolding tool: Evaluating the impact on elementary students’ mathematical logic problem-solving skills" 13 (13): 183-196, 2023
• 29 Zhu, J. J., "ChatGPT and environmental research" 57 (57): 17667-17670, 2023
• 30 Wardat, Y., "ChatGPT : A revolutionary tool for teaching and learning mathematics" 19 (19): em2286-, 2023
• 31 Parra, V., "Can generative AI solve geometry problems? Strengths and weaknesses of LLMs for geometric reasoning in Spanish" 8 (8): 65-74, 2024
• 32 Vaswani, A., "Attention is all you need" 30 : 2017
• 33 박지현 ; 송미영 ; 남민우 ; 최길찬, "Applying an integrated approach to analyse item features for curriculumbased assessments" 24 (24): 101-122, 2021
• 34 권오남 ; 오세준 ; 윤정은 ; 이경원 ; 신병철 ; 정원, "Analyzing mathematical performances of ChatGPT : Focusing on the solution of national assessment of educational achievement and the college scholastic ability test" 37 (37): 233-256, 2023
• 35 Christudas, B. C. L., "An evolutionary approach for personalization of content delivery in e-learning systems based on learner behavior forcing compatibility of learning materials" 35 (35): 520-533, 2018
• 36 오세준, "An analysis of the use of technology tools in high school mathematics textbooks based" 38 (38): 263-286, 2024
• 37 이유진, "An analysis of pre-service teachers’ mathematics lesson design using ChatGPT" 37 (37): 497-516, 2023
• 38 강윤지, "A study on the didactical application of ChatGPT for mathematical word problem solving" 38 (38): 49-67, 2024
• 39 Drori, I., "A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level" 119 (119): e2123433119-, 2022