Yes I think so, humans are fairly good at CLEVR. Bigger point is the fact that CLEVR went from <70% to >95% with "one weird trick" means it is a poor benchmark for progress. ILSVRC took 5 years to go 72%-93%, and most of that was incremental (2012 was "only" ~10%).
-
A good benchmark needs to find a sweet spot of hard but tractable but won't fall to one good idea, and I don't think anyone knows what that looks like right now.
More importantly no one knows what the next major benchmark should be about or what it should incentivize & measure 
-
-
How about a systems centric benchmark? http://dawn.cs.stanford.edu/benchmark/
-
Systems have proved essential in catalyzing recent technological progress. Typically create value by increasing the speed and coordination of AI development, reducing its cost, and enabling AI to address large-scale, real world problems
- Show replies
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.