One of my working theories is that all the horror stories about data and incentives are due to lack of exposure to W. Edwards Deming and his ilk.
That, to be clear, includes me.
Conversation
I mean, look at this! From Understanding Variation, by Donald Wheeler:
“When people are pressured to meet a target value there are three ways they can proceed
1. They can work to improve the system
2. They can distort the system
3. Or they can distort the data”
2
3
31
This is way more useful than pithy Goodhart’s Law, which goes “when a measure becomes a target, it ceases to be a good measure” — which is great, but what do you DO about it?
Breaking it down to 3 things allows you to target each behaviour separately.
3
9
I got to Understanding Variation by digging into Amazon’s Weekly Business Review, which in turn got its start during the crazy 2000 holiday season.
The WBR was influenced by a whole bunch of folks, many of whom came from operations research/business backgrounds.
Replying to
As a result, Amazon’s WBR process is most accurately seen as a process control tool; it implements many of the recommendations in Understanding Variation! Compare, on the left, Wheeler, on the right, a typical WBR chart.
1
2
More importantly, it's VERY clear that Amazon thought through how to deal with the 3 possible outcomes of managing to a target.
For starters, they keep an independent department (Finance) that administers targets and audits metrics. From Working Backwards:
1
3
The weekly cadence of the WBR itself acts as a feedback mechanism.
Say you incentivise the wrong behaviour by picking the wrong target.
The only way you know it's going wrong is to watch your output metrics, and see what happens over a period of time.
1
2
But the problem with doing that is that most orgs aren't disciplined enough to look at input or output metrics on a regular basis.
So: the WBR becomes a forcing function to have all execs sync on ALL of the company's metrics. On a weekly cadence. Together.
1
2
My favourite anecdote from Working Backwards involves exactly this feedback process on a controllable input metric:
1
7
Anyway, more notes on putting the WBR to practice in a few weeks/months. I'm not sure how deep this rabbit hole goes, but I think the literature around process control is a lot richer and more useful (especially outside of manufacturing) than most people think.
5