We introduce a novel spectral element method based on the ultraspherical and hierarchical Poincar\'{e}-Steklov scheme for solving second-order linear partial differential equations polygonal domains with unstructured quadrilateral or triangular meshes. Properties of lead to almost banded systems, allowing be competitive in high-polynomial regime ($p > 5$). The enables precomputed solution opera...