Some individuals are biased by having good taste. That poses a challenge for schemes of aggregating opinions. More in my latest newsletter:
5
Replies
0
Boosts
@tomstafford for grant application review the idea of coming up with a score has always seemed like nonsense to me for exactly this reason. I wonder would a series of yes/no questions aggregated into a score by a common formula be better? Like "the proposed research has no fundamental errors that couldn't be easily corrected". Perhaps not.
@neuralreckoning i think this would be more consistent across judges (and maybe even easier to do), but I bet some researchers would resent be "tied" to the formula, such they couldn't give a great proposal a great score, but had to enumerate the value point by point
@tomstafford we could let reviewers give a score and then not use it. 😉
Of the professional journal editor approach:
1. Desk-reject a paper;
2. Send it for review after an appeal;
3. Receive back 3 positive reviews;
4. Reject the paper stating that the reviews aren't positive.
5. Receive an appeal stating the reviews were positive;
6. Send out again to 2 more reviewers, get two positive reviews again (making it 5 positive reviews), reject again.
The editor didn't like the paper to begin with and didn't want it, but still wasted everybody's time.
Grant agency programme officers are known to do have done the same.
@neuralreckoning ah. The "programme manager" approach!
