Member Of Species

@Meaningness @A_P_Mason @asilentsky Rather, it's supposed to be U×P with U a huge number and P maybe smallish but not super small.

@MemberOfSpecies Hmm, not sure it was clear I was using “∞” as shorthand for “very large, unspecified, finite number.”

@MemberOfSpecies If that *was* clear, I’m not sure how what I attributed to singularitarianism differs from the actual argument.

@Meaningness The difference is in the ε, which I take to refer only to probabilities so small they make max-EU heuristics problematic.

@MemberOfSpecies The problem is more that U and P are both so uncertain that UxP could be anything; was subject of y aborted LW sequence.

@Meaningness The P is very hard to give precisely justified estimates for, but it can be constrained somewhat with informal arguments.

@Meaningness Maybe action in advance of AI progress is only 10% as effective, and, say, $100M buys you 0.05% instead of 0.5% risk reduction.

@Meaningness Saying, "yes, this argument decreases how much success we should anticipate, but the payoff is huge" seems reasonable here.