Quid Pro Ho

I think the "we don't really know how bad this is yet" team won over the "look, it's close enough" team, and they decided to pull it. And I'm okay with that. I think that's the right response.

From my perspective as someone who's worked on this stuff as a developer, I think it's incredibly likely that something--as yet unknown--went wrong in the data collection process, and people have been trying to figure out what the right response is.

These are all hard questions I've worked on and dealt with in my career. And the answers are not obvious. If I were working on this problem, I'd say it would take a team of 3 people working non-stop for about 4-5 days to get a clear answer about.

The additional problem is that if you are rim weighting on hard-to-reach demographics, a single interview *can* legit change an outcome. If one person happens to be a high-income, likely voter, African-American, living in rural IA, that response can get a lot of weight.

Is the whole response broken? Can you keep some aspects of it and throw out the problem point? Can you impute a value? Does this really change things? The whole point of a poll/survey is that an individual interview shouldn't change things. What about a small class of them?

If you don't find an obvious programming error, then you are stuck listening to all of this interviewers calls to find how how many were wrong. That takes time. Once you *do* find the source of the problem, you have to decide how to deal with the data for that interview.

If you find a survey programming error, great. You can get your data team to look at all paths to that node in the decision tree and find out how often Mayor Pete shows up in the answer set. But that means modeling expected values and that takes time.

What you'd have to do to troubleshoot this to figure out the scope of the problem is recreate the problematic interview by reviewing recordings and following the specific path of that respondent. If you're lucky, you find a programming error you can deal with.

These are all recorded, so it's possible to figure out what was asked and answered over enough time. But that's an incredibly slow process. And you still don't know what the interviewer saw on the screen. Operator vs. technical error is still quite murky.

With a phone interview, you have the interviewer acting on the survey software in response to the conversation that's happening. So not only do you not know what options the interviewer saw, you also don't know what options were read to the respondent.

But that's the best you're going to get unless someone has added some special logging to the platform default to show all possible answers. And that's not common. In a phone interview situation, you have additional error conditions that are possible.

If it's a really good platform, it will log the order the selected answer was positioned in (because you might want to de-rotate or de-randomize an answer to account for whether an answer is at the beginning or end of a list).

When this does happen, it's way harder to figure out than you might think. Survey platforms don't log which answers are displayed for any given question. All you see in the results is what the selected answer is.

Depending on how complex your screener logic is, it can be very easy to drop an answer from a specific path that doesn't get hit very often. I've made this mistake programming surveys and had to deal with it. Not fun. There are no good choices when that happens.

Testing the logic is difficult. The number of logical paths through a survey increases at a very high rate. Automated testing can only guarantee that you don't have a dead-end. It can't guarantee correctness at any given node in a tree. So most testing is actually manual.

Your choices in the screener determine if you qualify to continue and what questions you will see/hear later in the survey and what the answer options will be. All this has to be programmed by a person.

At the beginning of an interview (whether online or phone) there's a section called a screener. To see if you qualify as a good respondent. In this case, it's probably a segmentation that decides if you fit Seltzer's model of a likely voter.

I don't have any inside info about the DMR/Seltzer poll. But I spent 6 years working in the market research industry on the back end of survey platform systems. Here's a thread about how these things work, what can go wrong, and how hard it is to figure out the damage.

Here's a retweet so that an internet stranger doesn't have to start drinking again. https://twitter.com/ginnyhogan_/status/1220446045416587264 …