Here's a quick summary of our paper (with wonderful coauthors @jhaushofer and @cp_roth). Title: "Measuring and Bounding Experimenter Demand," just accepted at the AER (we're thrilled!). (1/12)pic.twitter.com/5eJeobtTuW
Show this thread
Basic worry: how can you be sure your experiment or survey findings are not driven or distorted by “demand effects”: participants acting out what they think you want to find? (2/12)
-
-
Our simple idea: amplify those effects by explicitly *telling them* what you want or expect them to do. Use those choices to construct bounds on behavior or treatment effects. Is my result robust to a range of possible demand effects? (3/12)
Show this thread -
Simple theory: it works if you can send messages that are more persuasive than whatever they infer from the experiment/survey question itself. (4/12)
Show this thread -
Headline finding (across 11 classic tasks): saying you "expect" participants to give more/less, work more/less, etc, shifts behavior only about 0.13 standard deviations. This is already a pretty strong manipulation. Demand effects seem pretty small. Good news! (5/12)
Show this thread -
But that doesn't mean participants don't care what we want. Telling them they "would do us a favor" (a very strong manipulation) elicits larger responses. So it's wise to be attentive at the design stage to possible demand effects. (6/12)pic.twitter.com/jta8QG3LEa
Show this thread -
We do a few other things. One is explore when these treatments can be used to “control for” demand effects. The idea is to use them to harmonize beliefs across participants, to de-bias your treatment effect estimate. (7/12)
Show this thread -
Another is to use a simple structural model (based on work by
@sdellavi and@Devin_G_Pope) to estimate the "value of pleasing the experimenter" in a real effort experiment. It's worth about a 20% increase in the performance pay. (8/12)Show this thread -
We also look at who “complies” (does what we ask) and who “defies” (does the opposite). We find minimal defiance, which is important for these methods to work. (9/12)pic.twitter.com/J9UEC54aaL
Show this thread -
Bottom line: we give a simple, portable set of tools for measuring and bounding demand effects. They’re applicable across settings, and give reasonable, informative bounds. We hope people find them and our estimates useful in their own work! (10/12)
Show this thread -
I’ll post a separate thread with a bit more detail on how to use them (11/12)
Show this thread -
You would do us a favor if you retweet this message (12/12)
Show this thread -
And here's the linkhttps://www.dropbox.com/s/4bgq445p8bgwvsl/deQuidt_Haushofer_Roth_Demand.pdf?dl=0 …
Show this thread
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.