To integrate the hierarchy structure of discrimination between all HMM states for Chinese Initials and Finals, we constructed in this paper Hierarchical Neural Networks (HNN), which differ from Jordan's HME in such extensions as more complex parameterization for gate and/or expert and dimension-reduced expert network. With these extensions, we can reuse those pre-trained simple node networks in...