Main content

Differentiable Hebbian Consolidation for Continual Lifelong Learning

Show full item record

Title: Differentiable Hebbian Consolidation for Continual Lifelong Learning
Author: Thangarasa, Vithursan
Department: School of Engineering
Program: Engineering
Advisor: Taylor, Graham W.
Abstract: Catastrophic forgetting poses a grand challenge for continual learning systems. It prevents neural network models from protecting previously learned knowledge while learning new tasks in a sequential manner. As a result, neural network models that are deployed in the real world often struggle in scenarios where the data distribution is non-stationary (concept drift), imbalanced, or not always fully available, i.e., rare or novel edge cases. In this thesis, we propose a Differentiable Hebbian Consolidation model which replaces the traditional softmax layer with a Differentiable Hebbian Plasticity (DHP) Softmax that adds a fast learning plastic component to the fixed (slowly changing) parameters of the softmax output layer. Similar to the hippocampal system in Complementary Learning Systems (CLS) theory, the DHP Softmax behaves as a compressed episodic memory that reactivates existing long-term memory traces, while simultaneously creating new short-term memories. We demonstrate the flexibility of our approach by combining our model with existing well-known task-specific synaptic consolidation methods to penalize changes in the slow weights that are important for each target task. We evaluate our approach on the Permuted MNIST, Split MNIST and Vision Datasets Mixture benchmark problems, and introduce an imbalanced variant of Permuted MNIST --- a dataset that combines the challenges of class imbalance and concept drift. Our proposed model requires no additional hyperparameters and outperforms comparable baselines by reducing forgetting.
URI: http://hdl.handle.net/10214/17440
Date: 2019-09
Terms of Use: All items in the Atrium are protected by copyright with all rights reserved unless otherwise indicated.


Files in this item

Files Size Format View
Thangarasa_Vithursan_201909_MASc.pdf 1.307Mb PDF View/Open

This item appears in the following Collection(s)

Show full item record