Calculating Large All - Pair - Shortest - Path Matrices under Google MapReduce A project report of CS 736 Fall 2012

The all-pair-shortest-path (APSP) problem has a general application in many research fields. Different types of applications have different performance requirements. A traffic safety project in identifying spatially close crashes favors a pre-calculated APSP between all reference-sites in the roadway network. However, with the actual size of a practical statewide roadway network, the targeted APSP can be so large that a single commodity computer cannot store it all in the memory, not even finishing the calculation without disk I/O’s. Once disk I/O is involved, the calculation can be extremely time-consuming. To make things worse, every time the topology of the roadway network changes, the long journey to calculate a new APSP will start again. To alleviate the time expense of calculating large APSP matrices that must involve disk I/O’s, the project team aims to implement two APSP algorithms in a distributed way. The two algorithms are the Dijkstra’s shortest path algorithm and the Floyd-Warshall algorithm. Google App Engine (GAE) servers are chosen as the underlying distributed environment. Upon GAE, the Google MapReduce library is used as the major API to convert the two algorithms into MapReduce tasks, which are distributedready.