Ordinary Banana

does anyone really think the class "humans" can have a coherent anything, much less coherent extrapolated volition

Tacit assumption of rationalism is that humans are rational in their essence and only contingently irrational due to

accident of running on lousy hardware. We are shadows of our Platonic Ideal selves, who have tidy mathematical definitions.

Benevolent AI could easily see which Ideal we each correspond to, and replace the corrupt wetware with that.

not multiple incompatible ones for each person, much less different people?

And they think Aumann’s Agreement Theorem proves that we’d all be identical if only it weren’t for wetware limitations.