In the framework of Markov Decision Processes, we consider the problem of learning a linear approximation of the value function of some fixed policy from one trajectory possibly generated by some other policy. We describe a systematic approach for adapting on-policy learning least squares algorithms of the literature (LSTD [5], LSPE [15], FPKF [7] and GPTD [8]/KTD [10]) to off-policy learning w...

Recently developed recursive least squares schemes, where the square root of both the covariance and the information matrix are stored and updated, are known to be particularly suited for parallel implementation. However, when finite precision arithmetic is used, round-off errors apparently accumulate unboundedly, so that after a number of updates the computed least squares solutions turn out t...

Recursive Least Squares (RLS) algorithms have wide-spread applications in many areas, such as real-time signal processing, control and communications. This paper shows that the unique solutions to linear-equality constrained and the unconstrained LS problems, respectively, always have exactly the same recursive form. Their only difference lies in the initial values. Based on this, a recursive a...

In this paper, an asymptotic bias of the recursive least squares (RLS) estimate in the closed loop environment is analyzed and its compensation method is proposed under the assumption that the noise is white. Namely, a bias compensated RLS method in the closed loop environment based on output error (OE) model is proposed. A posteriori error is also analyzed for the estimation of the noise varia...

In this paper application of recursive least squares with multiple forgetting factors is explained for online estimation of Heavy Duty Vehicle mass and road grade. The test data is obtained from highway experiments with a Freightliner truck. The experimental setup and particular concerns in the experiments are explained in detail. This data is used to validate the longitudinal dynamics model of...

This paper explores the potential of several popular equalization techniques and proposes new approaches to overcome their disadvantages. Such as the conventional least-mean-square (LMS ) algorithm, the recursive least-squares ( RLS ) algorithm, the filtered-X LMS algorithm and their development. An H 2 optimal initialization has been proposed to overcome the slow convergence problem while keep...

A new frequency domain system identification method based on a multi-frequency input signal is proposed. Frequency contents of the oscillating signal are estimated using a modified Kaczmarz algorithm proposed in this paper. Lyapunov stability analysis is performed for this new Kaczmarz algorithm and transient bounds for estimation error are established. Moreover, a new method for estimation of ...

After many Internet-scale worm incidents in recent years, it is clear that a simple self-propagation worm can quickly spread across the Internet. And every worm incidents can cause severe damage to our society. So it is necessary to build a system that can detect the presence of worm as quickly as possible. This paper first analyzes the worm’s framework and its propagation model. Then, we descr...

This paper presents the current progress on a spinning flying object robotic catching system, using general purpose and readily available hardware. The authors propose a recursive least squares (RLS) algorithm to extract and predict the position of a flying object in a 3D environment with the information gathered from only one camera.