하둡 및 Spark 기반 공간 통계 핫스팟 분석의 분산처리 방안 연구 = Distributed Processing Method of Hotspot Spatial Analysis Based on Hadoop and Spark

다국어 초록 (Multilingual Abstract)

One of the spatial statistical analysis, hotspot analysis is one of easy method of see spatial patterns. It is based on the concept that "Adjacent ones are more relevant than those that are far away". However, in hotspot analysis is spatial adjacency must be considered, Therefore, distributed processing is not easy. In this paper, we proposed a distributed algorithm design for hotspot spatial analysis. Its performance was compared to standalone system and Hadoop, Spark based processing.
As a result, it is compare to standalone system, Performance improvement rate of Hadoop at 625.89% and Spark at 870.14%. Furthermore, performance improvement rate is high at Spark processing than Hadoop at as more large data set.

국문 초록 (Abstract)

공간통계 분석중 하나인 핫스팟 분석은 “인접해 있는 것은 멀리 있는 것 보다 더 연관성이 있다”는 법칙에 따라 공간속성이나 사건의 공간 패턴을 쉽게 파악할 수 있는 기법 중 하나 이지만, 공간의 인접성이 고려되어야 하므로 분산 처리하기 용이하지 않다. 본 논문에서는 핫스팟 분석의 분산처리 방안을 기술하고 성능을 하둡 및 인메모리 기반인 Spark으로 평가한 결과 단일 시스템 대비 하둡기반 처리는 625.89%, Spark기반 처리는 870.14%의 성능향상을 확인하였으며, 하둡 기반과 Spark기반의 비교에서는 대용량 데이터 셋을 처리 할수록 Spark기반의 성능향상율이 높아짐을 확인하였다.

참고문헌 (Reference)

• 1 "http://spark.apache.org/"
• 2 "http://hadoop.apache.org/"
• 3 Anthony Fox, "Spatio-temporal Indexing in Non-relational Distributed Databases" 2013
• 4 Ahmed Eldawy, "Spatial Hadoop: A MapReduce Framework for Spatial Data" 2015
• 5 "Spatial Autocorrelation"
• 6 "Incremental Spatial Autocorrelation"
• 7 "How Hot Spot Analysis (Getis-Ord Gi*) works"
• 8 H. Kang, "Hotspot Analysis: Basic of Spatial Analysis, Understanding and utilization to Closet Grouping Analysis and Local Moran I" 116-121, 2008
• 9 "Hotspot Analysis"
• 10 Kim Changsoo, "Hadoop based Spatial Bigdata Index Creation and Processing" 87-89, 2016
• 11 Ablimit Aji, "Hadoop GIS: a high performance spatial data warehousing system over mapreduce" 6 (6): 2013
• 12 "Geovision"
• 13 "Geomesa"
• 14 "Geohash"
• 15 Jason Long, "GIS Tools for Hadoop: Big Data Spatial Analytics for the Hadoop Framework"
• 16 "Cluster and outlier analysis"
• 17 Cisco, "Cisco Mobile Visual Networking Index"
• 18 "Biz-gis XsDB"
• 19 Franklin, Carl, "An introduction to GIS: linking maps to databases" 15 (15): 17-22, 1992
• 20 W.R. Tobler, "A Computer Movie Simulating Urban Growth in the Detroit Region" 46 : 234-240, 1970