Learning is its Own Reward: Exploring Worlds with Curiosity-driven Spiking Neural Networks

While curiosity is an important part of human learning, it is rarely explicitly a part of machine learning: given some objective, machine learning algorithms optimize directly to get the best possible score. This has led to numerous impressive results but needs many examples and fails in situations where performance feedback is limited. Approaches attempting to incorporate curiosity generally still use a top-down optimization paradigm.

However, spiking neural networks (SNNs), which aim to better mimic biological neurons, present an alternative. Research has shown that local, spike-timing-dependent learning rules allow for the objective-free and bottom-up learning of complex patterns in data. But, limited work has been done applying spiking neural networks in contexts where they must interact with a simulated world making a series of actions.

We show that by combining bottom-up and top-down learning mechanisms in an SNN, namely spike-timing dependent plasticity (STDP) and backpropagation, and rewarding the SNN agent for the amount that it learns, as measured by its STDP update before modulation, our agent quickly learns to explore its environment in increasingly complex ways and seek increasingly novel situations. We demonstrate that, in terms of extrinsic rewards, this approach often results in better performance much more quickly (Up to 1,510 times faster with better performance for some environments and state-of-the-art models) than other curiosity-driven and general reinforcement-learning methods.