I’ve been enjoying ’s deeper dive on the orthogonality thesis recently, and wanted to try to lay out in my words a quick argument against strong orthogonality, feel free to comment or rebut etc
Conversation
If we are talking about super intelligent AGI such that they’re deriving all the laws of physics from watching a video feed or whatever, it seems quite highly likely that they’ll independently derive game theory…
1
8
Once you have a notion of game theory, you’ve already developed the foundations for understanding cooperation, including some very complex forms of cooperation.
2
8
Do we have a strong reason to believe such an AGI would in fact defect at the first / a pivotal opportunity, regardless of its other arbitrary goals? Is cooperation a type of emergent behavior, a la survival drives etc?
2
7
The AGI will presumably also understand evolution and all convergent drives that produces, including specifically what led to humans, and be able to model humans as being cooperative agents including various forms of reciprocal altruism, “superrationality”, etc.
1
8
As social mammals the AGI models us as having both general cooperative features, as well as specific notions of family loyalty, raising the young, respecting elders, etc. This seems like an easy slot for an AGI to fill/utilize in human interactions.
3
7
The combination of understanding game theory, convergent evolutionary drives, and specific human cooperation modes seems like it could result in any sufficiently powerful and general AGI deciding to cooperate with humanity broadly (especially if not specifically threatened by us)
5
6
What if the AGI considers us like we consider ants?
1) as humans expand their circle of concern, in fact insect welfare has entered the discussion!
2) there probably will be some tradeoffs and it’s appropriate humans don’t have *infinite* weight vs AGI concerns either
3
1
8
A process of uploading and uplifting other entities it comes across seems in line with a generally cooperative game theoretic agent, engaging in various forms of trade (including acausally)
3
4
If this argument is correct, an extremely powerful but *narrow* AI seems far more dangerous than a genuine AGI. (Which is certainly something people have argued, maybe not always for good reasons.)
2
12
Judging by the responses here, people are assuming the AGI has the game theory understanding of an econ undergraduate. You think an AGI that deeply groks UDT doesn’t realize that it *exists more often* in worlds where it *doesn’t* annihilate its creators?
9
1
12
