Reinforcement learning is a computational approach to learning from direct interaction with the environment. In order to ensure survival, an agent must adapt to new situations and challenges. Using trial and error the agent learns which actions lead to higher returns in the long run and exploits the accumulated knowledge to behave optimally. The trade-off between exploration and exploitation is the principal subject of this document. A simple yet powerful novel technique named past success directed exploration is introduced and investigated, together with its implications on the implementation of reinforcement algorithms. In particular, the Sarsa algorithm is implemented in both a traditional connectionist way, based on a backpropagation neural network, and a novel way, based on the fuzzy ARTMAP neural architecture. A number of experiments demonstrating advantages and better performance of the fuzzy ARTMAP implementation of Sarsa and its augmentation with the new exploration method are described.