Downloads & Free Reading Options - Results

This volume was digitized and made accessible online due to deterioration of the original print copy.

Recent developments in the area of reinforcement learning have yielded a number of new algorithms for the prediction and control of Markovian environments. These algorithms, including the TD lambda) algorithm of Sutton (1988) and the Q-learning algorithm of Watkins (1989), can be motivated heuristically as approximations to dynamic programming (DP). In this paper we provide a rigorous proof of convergence of these DP-based learning algorithms by relating them to the powerful techniques of stochastic approximation theory via a new convergence theorem. The theorem establishes a general class of convergent algorithms to which both TD(lambda) and Q-learning belong. reinforcement learning, Stochastic approximation, Convergence, Dynamic programming.

The energetic optimization problem, e.g., searching for the optimal switch- ing protocol of certain system parameters to minimize the input work, has been extensively studied by stochastic thermodynamics. In current work, we study this problem numerically with iterative dynamic programming. The model systems under investigation are toy actuators consisting of spring-linked beads with loading force imposed on both ending beads. For the simplest case, i.e., a one-spring actuator driven by tuning the stiffness of the spring, we compare the optimal control protocol of the stiffness for both the overdamped and the underdamped situations, and discuss how inertial effects alter the irreversibility of the driven process and thus modify the optimal protocol. Then, we study the systems with multiple degrees of freedom by constructing oligomer actuators, in which the harmonic interaction between the two ending beads is tuned externally. With the same rated output work, actuators of different constructions demand different minimal input work, reflecting the influence of the internal degrees of freedom on the performance of the actuators.