In this report, we proposes a neural network structure that combines a recurrent neural network (RNN) and a deep highway network. Compared with the highway RNN structures proposed in other studies, the one proposed in this study is simpler since it only concatenates a highway network after a pre-trained RNN. The main idea is to use the ‘iterative unrolled estimation’ of a highway network to fin...

As neural network algorithms show high performance in many applications, their efficient inference on mobile and embedded systems are of great interests. When a single stream recurrent neural network (RNN) is executed for a personal user in embedded systems, it demands a large amount of DRAM accesses because the network size is usually much bigger than the cache size and the weights of an RNN a...

A new approach for the adaptive algorithm of a fully connected recurrent neural network (RNN) based upon the digital filter theory is proposed. Each recurrent neuron is modeled by using an infinite impulse response (IIR) filter. The weights of each layer in the RNN are updated adaptively so that the error between the desired output and the RNN output can converge to zero asymptotically. The pro...

Neural machine translation is a recently proposed approach which has shown competitive results to traditional MT approaches. Standard neural MT is an end-to-end neural network where the source sentence is encoded by a recurrent neural network (RNN) called encoder and the target words are predicted using another RNN known as decoder. Recently, various models have been proposed which replace the ...

Neurogenesis is that new neurons are generated in the human brain. The new neurons create new network. The neurogenesis causes the improvement of memory, learning, thinking ability, and so on. We consider that the neurogenesis can be applied to an artificial neural network. In this study, we propose the Recurrent Neural Network (RNN) with neurogenesis and apply to pattern learning. In the RNN w...

Speech Emotion Recognition (SER) is an important part of speech-based Human-Computer Interface (HCI) applications. Previous SER methods rely on the extraction of features and training an appropriate classifier. However, most of those features can be affected by emotionally irrelevant factors such as gender, speaking styles and environment. Here, an SER method has been proposed based on a concat...

We studied the semi-supervised training in a fully connected deep neural network (DNN), unfolded recurrent neural network (RNN), and long short-term memory recurrent neural network (LSTM-RNN) with respect to the transcription quality, the importance data sampling, and the training data amount. We found that DNN, unfolded RNN, and LSTM-RNN are increasingly more sensitive to labeling errors. For ...

We demonstrate that inference-based goal-directed behavior can be done by utilizing the temporal gradients in recurrent neural network (RNN). The RNN learns a dynamic sensorimotor forward model. Once the RNN is trained, it can be used to execute active-inference-based, goal-directed policy optimization. The internal neural activities of the trained RNN essentially model the predictive state of ...

This paper presents a new recurrent neural network (RNN) structure called ENEM for dynamic system identification. ENEM structure is based on Elman network and NARX neural network. In order to show the performance of ENEM for system identification, the results were also compared to the results of Elman network, Jordan network and their modified models. The identification results of linear and no...

This paper presents a new stochastic learning approach to construct a latent variable model for recurrent neural network (RNN) based speech recognition. A hybrid generative and discriminative stochastic network is implemented to build a deep classification model. In the implementation, we conduct stochastic modeling for hidden states of recurrent neural network based on the variational auto-enc...