We propose to predict the future trajectories of observed agents (e.g., pedestrians or vehicles) by estimating and using their goals at multiple time scales. argue that goal a moving agent may change over time, modeling continuously provides more accurate detailed information for trajectory estimation. To this end, we present recurrent network prediction, called Stepwise Goal-Driven Network (SG...