Conversation

Aug 12, 2021

1/ Let's talk a little about how people learn in the real world. No, I'm not going to talk about classroom instruction, or pedagogical development, or enrolling in a cohort based course. None of that. Just a simple question: how do people ACTUALLY learn from doing?

Show this thread

9:36 AM · Aug 13, 2021threadcreator.com

2/ In 1993, Clark Chinn and William Brewer published a famous paper on how science students react to anomalous data — data that clashed with their mental models of the world. They then drew on the history of science to show how common these reactions are amongst scientists.

Replying to

3/ It turns out there are basically only 7 ways you can respond to inconvenient data. 6 of them allow you to preserve your existing mental models. See if any of these are familiar to you, before we go through them in order:

4/ First: you ignore the data. Really simple. You stuff your fingers in your ears and pretend you didn't see the data. (Or you just gloss over it and pretend it wasn't in the article/report/paper).

GIF

5/ Second: you reject the data. This is saying something like: "oh that is shitty experimental design", or "this is a joke, right?"

6/ Third: you find a way to exclude the data from consideration. This is something like "Oh, this data is interesting, but it has nothing to do with our department. Go tell sales, this is their problem." Also Brownian motion:

7/ Fourth: you hold the data in abeyance — meaning you suspend it temporarily from consideration. e.g. "Ahh, this is probably noise, though it's strangely persistent. Hmm, we'll worry about it later." Also: quantum mechanics is weird, I'm sure the physicists will solve it soon!

8/ Fifth: you reinterpret the data while retaining your mental model. "Ok, I accept we're seeing a dip in sales in June, but this is simply part of a yearly pattern. See, sales was down last June too!" Also, 'iridium, meh':

9/ Sixth: you reinterpret the data while making peripheral changes to your mental model. e.g. "Ok, our sales starting June is lower than normal, but that's also because top 3 salespeople were sick. Things will get better next week, you'll see!"

10/ And finally, seventh: you accept the data and update your mental model. Which is good! Note that this is the only response where you accept the data and update your mental model. This becomes harder the more expertise you have.

11/ The authors point out that all 7 responses may be reduced to just 3 questions: 1. Do you accept the data as valid? 2. Can you provide an explanation for why the data is accepted or not accepted? 3. Have you changed your prior mental model?

12/ I'll tell you why this is interesting to me. Cognitive Transformation Theory tells us that the way we learn from the real world is that we run trial & error cycles, and then we reflect on those cycles and break our old, flawed mental models — transforming them into new ones.

13/ But as our mental models become progressively more sophisticated, growth begins to slow. We begin to use our models to ignore or reject anomalous data. Our sensemaking slows. CTT calls this 'knowledge shields'. It happens to all of us.

14/ The best experts — the best businesspeople, or investors, or what-have-you — have simply figured out ways to fight this tendency. And they continually invest in better methods to question their mental models.

15/ Chinn & Brewer's paper is useful because it tells us what forms our knowledge shields would take. It really comes in the form of three rejections: 1. You don't accept the data as valid 2. Or you explain away the data. 3. And therefore you don't have to update anything.

16/ We should guard against this. And I think we should stay on the lookout for better methods, as well as prepare ourselves to increase the amount of energy we spend breaking our own knowledge shields, the better we get. Chinn and Brewer's paper is merely a start. The end.