AI doesn’t replicate. Having worked in the field, I can usually see why a paper’s result is nonsense, but the public can’t, and many researchers can’t.https://twitter.com/stephaniemlee/status/964612382650646529 …
-
Show this thread
-
I’ve been skeptical about DL results because 25 years ago I reran the key experiments that were hyped as showing backprop (the underlying tech) was incredible. In each case I found that the researchers were fooling themselves. Not deliberate fraud, but sloppy work.
3 replies 2 retweets 18 likesShow this thread -
-
Replying to @xuenay
The only one I wrote up was this one. The then-most-hyped version of RL+backprop turned out to work less well than RL+perceptron. https://www.ijcai.org/Proceedings/91-2/Papers/018.pdf …
1 reply 0 retweets 2 likes -
Replying to @Meaningness @xuenay
The one that got me really annoyed was XOR. The narrative was that Minsky&Papert unfairly killed perceptrons with that, and you could learn XOR if you added hidden layers. 1/
1 reply 0 retweets 1 like -
Replying to @Meaningness @xuenay
This turned out not to be true in any interesting sense. You can compute XOR with a feedforward network, but backprop won’t learn it reliably, nor in a reasonable length of time. It has to get lucky. 2/
2 replies 0 retweets 2 likes -
Replying to @Meaningness
OH. And I thought that I was doing something wrong back when I was trying to get an intuition for backprop and was simulating it by hand in a spreadsheet, but couldn't get it to converge for XOR.
1 reply 0 retweets 0 likes -
Replying to @xuenay
I did this 25 years ago, but my recollection is that backprop can only find the solution by accident. There’s no guiding gradient. You have to set the hyperparameters to force a random walk over the whole space, and hope it falls into the golf hole eventually.
1 reply 0 retweets 1 like -
Replying to @Meaningness @xuenay
This means that it takes O(2^n) steps to learn n-bit XOR, and the probability of getting permanently stuck in a local minimum (wrong answer) goes up rapidly with n too. (I forget exactly how rapidly.)
1 reply 0 retweets 1 like -
Replying to @Meaningness @xuenay
That said, parity is an unusually hard problem to solve feedforward (see circuit complexity), although trivial if you process it as a sequence. The same is true for us: easy to compute a 20 bit parity step by step, impossible in one glance.
2 replies 0 retweets 1 like
Yes; my complaint is not that XOR was a good test of backprop, but that the narrator was “see, artificial brains CAN TOO learn the thing you said was impossible, because they are magic!”
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.