Prof. Anima Anandkumar

Bongard analogy challenge uses 6 positive and negative examples to convey a concept (e.g. convex). Original one from 1960s ~ few hundred hand-engineered problem instances. Instead, we use LOGO to programmatically generate concepts at scale => no data scarcity for #DeepLearning

Human evaluation accuracy on our benchmark was as high as 99% for careful evaluators. Yet, all current meta/few-shot/self-supervised methods < 70%. Even the best #DeepLearning methods are not able to learn underlying simple concepts at human level.

We carefully designed benchmark to not reward overfitting by #DeepLearning methods. This is a problem with previous few-shot learning benchmarks. E.g. birds dataset uses very similar birds that differ only on simplistic attributes like color of beak.

We designed our benchmark to reward qualities seen in human vision: (1)Context dependence (2)Analogy Making (3)Few-shot learning with infinite vocabulary. Simply overfitting and memorization by #DeepLearning will not be enough to do well on our benchmark.