Whoah, CHI is introducing a new reviewing phase where ACs read AI-generated summaries of papers in order to desk-reject 40-50% of submissions: https://chi2027.acm.org/2026/08/17/how-the-chi-2027-papers-review-process-will-work/
6
Replies
1
Boost
@tonofcrates
I don't envy the CHI Papers chairs - this is a really challenging situation.
That said, the conference is moving ever more into a direction I don't feel comfortable with. I'm happy that I'm not involved in any CHI submission this year. Looking on from the sidelines.
@RaphaelWimmer @tonofcrates Agreed - I understand the motivation, but the process is completely opaque.
From TFA: "To help ACs manage the volume, an AI-assisted tool will assess the paper against a rubric based on ACM criteria (Originality, Correctness, Novelty, Importance, and Clarity of Exposition) to produce a preliminary report for each paper."
At the very least, I would want to know which exact LLM they are using and what the prompt is, otherwise, they might just as well flip a coin. 😑
@floe @RaphaelWimmer @tonofcrates I wouldn’t trust that process because I’ve personally found LLMs to be very inconsistent at any kind of “evaluation” task like that. Third is also bound to introduce a heavy bias against any unconventional paper, regardless if it differs in language, structure or topic from the rest.
@jaseg @RaphaelWimmer @tonofcrates Absolutely! What I would really like to do, given the model and prompt, is to run some papers from previous years through the pipeline (maybe 3 times each) and collect some data on how they would fare ...
@floe @jaseg @RaphaelWimmer @tonofcrates I suspect that's what they've been trained on (maybe even mine), but they should include the rejections too.
Of course, that just bakes in the opinions of previous iterations of reviewers.
Maybe they're just overwhelmed by AI-assisted papers just as OSS projects are with pull requests and CVEs, and this is their way to try and cope
@stevel @jaseg @RaphaelWimmer @tonofcrates I don't think they actually trained their own LLM - they have their "rubric", which I guess is basically an extended prompt, and bolted that onto one of the current frontier models?
