We derive a stochastic hierarchy of matrix product states (HOMPS) for non-Markovian dynamics in open quantum system at finite temperature, which is numerically exact and efficient. HOMPS obtained from the recently developed pure (HOPS) by expressing HOPS terms formal creation annihilation operators. The resulting first order differential equation then formulated In this way exponential complexi...