Mason  🏃‍♂️ ✂️

Interesting - what were these tests like/what did they test?

The strongest one was a play on the (now) somewhat well-known cognitive reflection test. I was actually involved in a hiring process that utilized it: applicants completed the test online, with unlimited time. Out of hundreds, a tiny handful of 10/10s. Most scored well under.

(By tiny handful, I *think* we had 2-3 perfect scores out of ~250-300 applicants, many with incredible credentials. Managing to hire someone who could deliver a perfect score would be super, super impressive.)

Looked it up - I have never heard of it! Though were the applicants told that what was being tested was their "tendency to override an incorrect 'gut' response"? Because that piece of information alone would seem to give you a huge advantage.

No — but they were explicitly reminded that the test would be untimed and that they should think carefully about their answers! (Which is true of life, where our gut responses are also often useful but not sufficient to solve tricky problems.)

The reason I ask is because I feel one huge problem with tests is that all kinds of meta information can often help you score much better. Driving exams are one of the few tests I can think of that don't have this property.

Does this matter, if that meta information is of the sort that (a) isn't inorganically available to some test-takers but not others, and (b) also isn't available in the real-world scenarios the test is emulating, albeit in a simplified form?

Well no but I would guess (a) is rarely true. In this case someone knowing about cognitive reflection tests might have instantly recognised they were facing one and adjust accordingly (as opposed to someone who doesn't know this is a thing).