“When a measure becomes a target, it ceases to be a good measure” — [Goodhart’s Law]
“When quantifying things, people naturally focus on things that can easily be measured. Measuring the final result doesn’t provide enough quantitative data, so it’s tempting to include the data from intermediate steps. This is an attempt to shorten the feedback loop, and trying to shorten feedback loops is very dangerous in complex systems.”
See also: paperclip optimizer