Conversation

I’ve talked a lot about cognitive task analysis, which is this fairly new (30 yo) technique to extract expertise from the heads of experts. You know how pros can’t tell you how or what they’re doing? e.g. “It just feels right” Yeah, CTA solves for that. Some podcast clips. 👇

5

13

128

“What are some of the biggest challenges in CTA” “One major problem is figuring out who, exactly, has expertise (…) often we get recommended people who are well liked (but just not very good)”

Embedded video

0:56

779 views

1

1

11

Sometimes experts tend to leave out certain decisions that they make. Because it’s so intuitive to them. But this has problems when you’re training students. Clark tells the story of surgeons leaving out a major decision in training, resulting in bad outcomes for patients:

Embedded video

1:18

518 views

1

1

6

“We have found similar things in the energy industry, with people making decisions on where to drill wells” In this case, experts who have been doing this for awhile leave out specific decision steps when explicating their knowledge. So: training was bad!

Embedded video

1:00

465 views

1

6

“It’s resulted in the discovery of a lot more successful wells, let’s put it that way.” “And this has happened in almost every case where we’ve done a cognitive task analysis.”

4:54 AM · Sep 6, 2022Twitter Web App

Replying to

“We’ve actually done a series of cognitive task analysis on how large orgs train managers (…) and how good managers motivate.” “The findings from those are going to be published in the Harvard Business Review” That article: hbr.org/2019/03/4-reas

Embedded video

0:51

395 views

1

6

Why does this happen? Well, experts may see what they’re doing with their actions, but they’re not as good with what goes on in their heads. Clark says that chunking is the heart of expertise, but chunking also means intuition and expert decision making becomes invisible.

2

2

4

As a result, traditional task analysis misses out on 80% of the instruction necessary for effective training, which CTA solves for. Clark also notes that the 1 cost benefit study he’s seen: 7x up front time costs, but 1/2 training time and save 2.5 person years per trainee.

Embedded video

1:25

316 views

1

1

6

So what *is* CTA? And how do you go about researching or doing it? “There are over 100 CTA methods, but there are really only 4-6 with results, and only 3 with major results” I’m not really sure which meta analysis this is, but probably: journals.sagepub.com/doi/10.1177/15

Embedded video

1:23

248 views

1

7

Unfortunately it takes 5-6 months of training before someone can be really proficient in performing CTA. The reason is that experts tend to tell you about the decisions they typically make, but tend to forget what they do when unexpected things crop up. Good CTA captures that.

Embedded video

1:20

252 views

1

4

The cool thing about CTA is that it usually results in everything you need to create a new training program. The only thing is that you have to explicitly ask for is more examples, because you can do (some forms of) CTA without. But then you have instant training at the end!

Embedded video

1:08

208 views

1

4

Good question: “does it result in more training or job aids?” (Note: job aids could be decision tools, or it could be redesigned user interfaces like in the military or in NASDAQ’s fraud detection dept) Clark guesses in most CTA engagements, 80% training, 20% job aids.

Embedded video

0:59

171 views

1

1

Finally, I suspect CTA is going to be big soon. “The Obama administration heard about us and asked us to make a presentation (…) then they asked us to do engagements with federal agencies and those said no. This angered the White House. But now these folk are in foundations…”

Embedded video

1:18

219 views

1

1

2

“Eric Schmidt’s foundation looked into CTA, and they realised that the huge up front cost of training someone to do CTA … made it ideal for replacement with an AI system … so they funded one. And we’re now a year into this … and then we want to offer it to people to use.”

Embedded video

1:29

339 views

2

2

6

🤯 Apparently they’re a year away from a first release (at the point of the podcast), so I’ve reached out to ask for more details. But, holy hell, AI-powered CTA, coming to a knowledge system near you.

1

2

17

End thread. I’ll add more details after I get a reply. For updates, follow me, or read more about my experiments with CTA here:

John Cutler’s Product Org Expertise

A few weeks ago, I helped Amplitude head of product education John Cutler extract tacit expertise around diagnosing and improving product organisations. Here's how that went.

1

12

Finally, if you want to learn CTA for yourself, I recommend the courses available at cta.institute !

Visit the post for more.

2

1

11