Opens profile photo
Follow
Click to Follow ArbResearch
Arb
@ArbResearch
We can research it for you wholesale. A consultancy by &
Science & Technologyarbresearch.comJoined February 2022

Arb’s Tweets

David returns to his roots and sharpens our ignorance about some important concepts people have not been treating well: problem-solving ability vs goal-orientedness vs phenomenal consciousness vs self-consciousness vs moral patiency
3
Couple important caveats: 1. Asimov stands out but the other two don't. He still got lots of big things very wrong (mainly supply chains and overpopulation) 2. We looked at their nonfiction rather than their fiction because novelists have a complicated objective with truth last
Quote Tweet
Please stop using "science fiction" to mean "nonsense"! 1. Sci-fi is one of the few ways we have to debate the future we are building 2. Despite not being primarily about prediction, sci-fi is actually about as good as anything in predicting the future cold-takes.com/the-track-reco
Show this thread
One aspect of expertise we've been thinking about since writing this is Heisenberg's line "an expert is someone who knows exactly which mistakes you must not make". Clipping the left tail often matters more than the mean.
Quote Tweet
With thanks to Sam Enright, Sam Harsimony, Sam Glover, Irish Sam Bowman, Alejandro Ortega, Patrick Atwater, Max Langenkamp, Christian Robshaw, Ozzie Gooen, Kristi Uustalu, Michael Story, Nick Whittaker, Philip Stephens, Adam Russell, Alec Stapp, Santi Ruiz, Nuño Sempere for input twitter.com/IFP/status/163…
2
(Note the separate analysis for the community predictions and the weighted "Metaculus predictions") + as always, this stuff is only possible because Metaculus are really open with sharing data.
5
Show this thread
With thanks to Sam Enright, Sam Harsimony, Sam Glover, Irish Sam Bowman, Alejandro Ortega, Patrick Atwater, Max Langenkamp, Christian Robshaw, Ozzie Gooen, Kristi Uustalu, Michael Story, Nick Whittaker, Philip Stephens, Adam Russell, Alec Stapp, Santi Ruiz, Nuño Sempere for input
Quote Tweet
How should policymakers think about integrating forecasters, prediction markets, and expert judgment into decision-making? @g_leech_ and @mishayagudin lay out a framework: progress.institute/can-policymake
1
This is a nice overview of why use softmax (do good options often, worse options less often) rather than argmax (do the best option 100%) in many situations. Big issue for me is how to find optimal temperature schedule - but we can have individual ones!
2
51
Dan Luu has released a characteristically superhuman red teaming of futurism which includes Arb's study of the Big 3. He covers more ground than did our team of 6 people.
Quote Tweet
Evaluating futurist prediction methods and accuracy: danluu.com/futurist-predi
Show this thread
Ray Kurzweil: 7% accuracy
Relies on: exponential or super exponential progress that is happening must continue; predicting the future based on past trends continuing; optimistic "rounding up" of facts and interpretations of data; panacea thinking about technologies and computers; cocktail party ideas on topics being predicted *Jacque Fresco: predictions mostly too far into the future to judge, but seems very low for judgeable predictions
Relies on: panacea thinking about human nature, the scientific method, and computers; certainty that human values match Fresco's values
Buckminster Fuller: too few predictions to rate, but seems very low for judgeable predictions
Relies on: cocktail party ideas on topics being predicted to an extent that's extreme even for a futurist
Michio Kaku: 3% accuracy
Relies on: panacea thinking about "quantum", computers, and biotech; exponential progress of those
John Naisbitt: predictions too vague to score; mixed results in terms of big-picture accuracy, ...
Ray Kurzweil has claimed to have an 86% accuracy rate on his predictions, a claim which is often repeated, such as by Peter Diamandis where he says:

Of the 147 predictions that Kurzweil has made since the 1990's, fully 115 of them have turned out to be correct, and another 12 have turned out to be "essentially correct" (off by a year or two), giving his predictions a stunning 86% accuracy rate.

The article is titled "A Google Exec Just Claimed The Singularity Will Happen by 2029" opens with "Ray Kurzweil, Google's Director of Engineering, is a well-known futurist with a high-hitting track record for accurate predictions." and it cites this list of predictions on wikipedia. 86% is an astoundingly good track record for non-obvious, major, predictions about the future. This claim seems to be the source of other people claiming that Kurzweil has a high accuracy rate, such as here and here. I checked the accuracy rate of the wikipedia list Diamandis cited myself (using archive.org to g...
1
33
This has implications for the laudable drive to forecast important things like machine learning progress: it might be better to train ML experts in forecasting rather than vice versa.
2
21
Show this thread
Tracking down the original studies, we find mixed results and weak methods. The best study* found similar performance between forecasters and experts. Another decent one found a small (3%) forecaster advantage. Compare the original 30% claim! *
1
14
Show this thread
Superforecasting - that certain teams of people are reliably better than other prediction mechanisms like crowds and statistical rules - seems sound. But serious interest in superforecasting stems from the reported triumph of forecaster generalists over non-forecaster experts.
1
6
Show this thread