I hadn’t seen this critique of superintelligence before. Interesting. It lands roughly where I did but via a different route (his term is much cleverer, “AI cosplay”). Ht
Deleted previous version of the tweet where I mistakenly attributed it to Bret Victor rather than Maciej Cegłowski. That makes much more sense. I was surprised to find myself agreeing with what I thought was Victor. In my head “idlewords” somehow sounds close to “worrydream”
My diagnosis was always a kind of anti-projection.
a) You think in a totalizing (INTJish) way and are impressed by its power
b) You see a machine that thinks in analogous ways and looks like it lacks your limits
c) You extrapolate its future as you minus biological limits
Note this is specifically a critique of the Bostrom-LW vision of the future of AI, based on an IQ++ model of what intelligence is. Not of all possible futures for the tech. It’s one that commits to a a sequential evolutionary model where the prefix “super” makes sense.
The reason I don’t bother engaging with this conversation is that my starting point is ontologically at the opposite pole from IQ++. I don’t find “entrance tests for bureaucratic industrial orgs to test aptitude for their legible functions” to be an interesting place to start.
Mine is: “the brain is a 100 billion neuron system that from the inside (“mind”) doesn’t *feel* like it has 100 billion elements, but more like dozens to 100s of high level salient emergent phenomena operating on a rich narrative and verbal memory... what else looks like that?”
The answers are things like markets, ecosystems, weather systems. Billions of atomic moving parts, but quasi-stable macro-phenomenology. There may be nothing it is “like” to be a “market” but setting aside the hard problem of consciousness it is in the brain-class of things.
The most interesting and salient thing about these systems is that they are coherent and stable in a thermodynamic sense, maintaining boundary integrity and internal structural identity continuity for periods of time ranging from tens to thousands of years.
You could but that term never really made sense to me and I’m not sure what it gets you. Prigognine &co took solid chemistry work and went slightly crackpot with it, the way Weiner did with controls —> “cybernetics”