https://overcast.fm/+Ic2hwsH2U/1:10:49 … #AI
“You could build a mind that thought that 51 was a prime number but otherwise had no defect of its intelligence – if you knew what you were doing” —@ESYudkowsky
Is it possible build a mind able to learn but incapable of correcting this error? (Why?)
-
-
Replying to @reasonisfun @ESYudkowsky
It isn't possible. Because from '51 not prime' you could lead it into a contradiction, such as 1=0. Then it would display another defect e.g. denying that that was a contradiction.
4 replies 1 retweet 24 likes -
Replying to @DavidDeutschOxf @reasonisfun
See my reply downthread. You'd need to prevent further propagation of the inconsistent consequences while still allowing enough propagation to make associated behaviors real. The *hard* part would be consistency under iterated reflection.
1 reply 0 retweets 2 likes -
It would *not* be simple the way a superintelligent paperclip maximizer is simple and coherent. I don't know exactly how to do it. But I'm confident it could be done by someone with a completed understanding of AI, because thought steps are physical and not metaphysical.
1 reply 0 retweets 1 like -
It would look to us like a mind with a lot of weird fiddly bits attached to maintain the delusion plus the weird fiddly bits, but the things you'd need to fiddle would be finite. The meta-meta-meta delusion would look a lot like the meta-meta delusion; there'd be a fixed point.
4 replies 0 retweets 2 likes -
To respect the power, depth, entanglement, interrelation, and consequences of intelligence is to think that the number of fiddly bits would be large and that lots of naive approaches wouldn't work--not to believe that it could never ever be done.
1 reply 0 retweets 1 like -
But I suppose it should be made precise that when I said (out loud on a podcast) that there wouldn't be further defects, I meant defects of external behavior and capability not related to the number 51. If you regard the internal fiddling as a defect then it's a futher defect.
2 replies 0 retweets 1 like
Also to be super clear, no human should ever try to pull this kind of shenanigan while doing AGI alignment. Find simple, compact, coherent, consistent ways to do stuff or don't do it.
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.