Simplification and Refinement for Speedy Spatio-temporal Hot Spot Detection Using Spark

This paper describes a spatio-temporal hot spot identification program submitted to the ACM SIGSPATIAL Cup 2016. The advent of large-scale spatio-temporal data (e.g., vehicle tracking data), together with the availability of inmemory distributed computing framework (i.e., Spark), provides an opportunity to quickly identify unusual patterns in a statistically manner, also called hot spots. We propose an efficient approach on Spark to select top-k hot spots using the Getis-Ord statistic. Our method consists of three phases: simplification, detection, and refinement. The simplification phase picks a reasonable number, e.g., K, of most important cells; then the detection phase computes the Getis-Ord statistic for the K cells; finally, the refinement phase puts the top M candidates and their neighbors back in the original set of cells to refine the results.