Gierhos et al 2024 – Towards flexible perception with Visual Memory

Setting:

• We can build a classifier, but what happens when we want to change the output labels, maybe because of distribution shift?

Basic idea:

• Let's separate the representation learning and the labeling parts of a traditional classifier
• Let's store a "Visual Memory" which consists of feature vectors and the corresponding label.
• Now, how do we classify a new image? Comb over the visual memory, find the feature vectors that are closest to the features for the new image, and take the majority label