When I suggest that people will game whatever metrics we put in place, I'm often met with shocked indignation. We would never game the numbers! And yet we do.

I took my car in for service this morning and I was asked if it was ok that they split the bill across two transactions. "You're being measured on number of cars through?" I asked. The answer was obviously yes, and this way I counted as two cars.

It's not just a matter that the numbers are now wrong, we have now introduced waste into the system. There were two credit card transactions rather than one. Two receipts instead of one. There was additional time for the workers to explain why they wanted to do it this way. Overall, this was complete waste, but because they felt they were being judged on the count of cars through, it was justified.

If people think that they'll be judged based on measurements then they'll game those. The more judgement, the more the numbers will be inaccurate, and the more waste will be introduced into the overall system.

You might think that I'm opposed to measuring anything then but that's not at all true. I'm a big proponent of measuring those things we want to improve. I'm just a realist and recognize that we have to design our measurements very carefully. If we measure the wrong things, or in the wrong way, we'll drive the wrong behaviours and that's our problem to solve.

#metrics

When I suggest that people will game whatever metrics we put in place, I'm often met with shocked indignation. We would never game the numbers! And yet we do.

I took my car in for service this morning and I was asked if it was ok that they split the bill across two transactions. "You're being measured on number of cars through?" I asked. The answer was obviously yes, and this way I counted as two cars.

It's not just a matter that the numbers are now wrong, we have now introduced waste into the system. There were two credit card transactions rather than one. Two receipts instead of one. There was additional time for the workers to explain why they wanted to do it this way. Overall, this was complete waste, but because they felt they were being judged on the count of cars through, it was justified.

If people think that they'll be judged based on measurements then they'll game those. The more judgement, the more the numbers will be inaccurate, and the more waste will be introduced into the overall system.

You might think that I'm opposed to measuring anything then but that's not at all true. I'm a big proponent of measuring those things we want to improve. I'm just a realist and recognize that we have to design our measurements very carefully. If we measure the wrong things, or in the wrong way, we'll drive the wrong behaviours and that's our problem to solve.

#metrics

When I suggest that people will game whatever metrics we put in place, I'm often met with shocked indignation. We would never game the numbers! And yet we do.

I took my car in for service this morning and I was asked if it was ok that they split the bill across two transactions. "You're being measured on number of cars through?" I asked. The answer was obviously yes, and this way I counted as two cars.

It's not just a matter that the numbers are now wrong, we have now introduced waste into the system. There were two credit card transactions rather than one. Two receipts instead of one. There was additional time for the workers to explain why they wanted to do it this way. Overall, this was complete waste, but because they felt they were being judged on the count of cars through, it was justified.

If people think that they'll be judged based on measurements then they'll game those. The more judgement, the more the numbers will be inaccurate, and the more waste will be introduced into the overall system.

You might think that I'm opposed to measuring anything then but that's not at all true. I'm a big proponent of measuring those things we want to improve. I'm just a realist and recognize that we have to design our measurements very carefully. If we measure the wrong things, or in the wrong way, we'll drive the wrong behaviours and that's our problem to solve.

#metrics

What tools do you use for your on-host #FreeBSD| #BSD system metrics collection and monitoring? #Sysadmin #DevOps #SRE #Monitoring #Systems #Observability #Metrics

What tools do you use for your on-host #FreeBSD| #BSD system metrics collection and monitoring? #Sysadmin #DevOps #SRE #Monitoring #Systems #Observability #Metrics

What's going on when some #universities jump more than 950% in one year on #metrics used in university #rankings? Are they gaming the metrics?
https://www.biorxiv.org/content/10.1101/2025.05.09.653229v1

"Key findings include publication growth of up to 965%, concentrated in STEM fields; surges in hyper-prolific authors and highly cited articles; and dense internal co-authorship and citation clusters. The group [of studied institutions] also exhibited elevated shares of publications in delisted journals and high retraction rates. These patterns illustrate vulnerabilities in global ranking systems, as metrics lose meaning when treated as targets (Goodhart’s Law) and institutions emulate high-performing peers under competitive pressure (institutional isomorphism). Without reform, rankings may continue incentivizing behaviors that distort scholarly contribution and compromise research integrity."

#Academia
@academicchatter

Excellent: "More than 100 institutions and funders worldwide have confirmed that research published in #eLife continues to be considered in hiring, promotion, and funding decisions, following the journal’s bold move to forgo its Journal Impact Factor."
https://elifesciences.org/for-the-press/bf74b86e/more-than-100-institutions-and-funders-confirm-recognition-of-elife-papers-signalling-support-for-open-science

PS: This is not just a step to support eLife, but a step to break the stranglehold of bad metrics in research assessment. For the same reason, it's a step toward more honest and less simplistic assessment.

#Academia#Assessment#JIF#Metrics#Universities
@academicchatter