I like your previous idea of assembling furniture but I'd modify it to strictly IKEA instructions/build.
Want to help build a new benchmark for AI, that aims to push machines toward a deeper understanding?
If yes, please contribute a few questions at Google Form below, and please pass this along 
Suggestions for improvements welcome!
https://docs.google.com/forms/d/e/1FAIpQLSekOYnhHejR_aeXfB1yqVrYtWAQlDNiMOy7zoXYpDCnJTIc3Q/viewform …
-
-
-
Charles Ortiz has an article about this (it's his idea) in the special issue of AI Magazine that Manuela Veloso,
@frossi_t and I edited.
End of conversation
New conversation -
-
-
As
@fchollet wrote here https://arxiv.org/abs/1911.01547 , machine learning always finds shortcuts. We cannot compare levels of understanding by machines and humans via one simple test. Need to evaluate abilities via the capacity to adapt to new (unseen) skills, like answering these tests. -
i certainly don’t think one test will suffice and have been arguing that for years, hence my proposals for a Turing Olympics, but i think we can at least start to focus on facets that are important and understudied.
End of conversation
New conversation -
-
-
I’m concerned about this method of crowdsourcing the questions. Will you test for how much is contributed from different socioeconomic backgrounds, developing parts of the world, or other strata?
-
not at this stage - we are keeping things anonymous, and can’t solve all problems at once - but it’s a good point, and I might seek a collaborator for this on a later iteration. i think it would be possible to filter at a later stage.
- 1 more reply
New conversation -
-
-
Filled it out! I made some other suggestions, but would also love to see some questions that required reasoning over multiple modalities at once (images/video/audio)!
-
@JhendersonIMB may be helping with that :) and would love more help for sure. - 1 more reply
New conversation -
-
-
My kind of
#AI :) Joking apart however wouldn’t this be like the TREC questions?pic.twitter.com/RdTtmTFtIS
-
no expert on TREC, which seems to have fractionated into many tracks, but none seem as focused as what i am trying to do, but there is commonality in goals
- 1 more reply
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.