Will Crichton

I'm consistently surprised by the lack of depth in programming psychology work. Search Google Scholar for "program readability", see all top-cited work. Consistently, the only methodology: literally ask people whether they think code is readable, and check for agreement.

Studies have consistently shown no "significant relationships between perceived readability and... reading time and performance on a simple cloze test." https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7404062 …

"Our results demonstrate that none of the (existing and new) metrics we considered is able to capture code understandability, not even the ones assumed to assess quality attributes strongly related with it, such as code readability and complexity." https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8115654 …

You would get laughed out of the field presenting this stuff at a psychology conference instead of ICSE. Imagine, "To analyze racism in job hiring, we surveyed a group of managers to ask 'do you think you're racist?'" You have to validate with external measures...

Love this thread! Question: what's the external measure that Harry Potter is readable? It seems to me like the way to test code comprehensibility is simply to give it to new people and see how quickly they can comprehend it

The challenge is measuring comprehension. This ranges from standard assessment (multiple choice) to memory tests (cloze tests, free recall). Your results are as believable as your test design.