@RonjaBiernat

If the references are made up, I have no way of knowing whether the experiments were run at all, or whether the results are made up.

This is why I have stopped reviewing. A large enough fraction of paper authors are willing to submit complete fabrications that I no longer trust a process of reviewing a paper as an adequate safeguard. Unless a publication process has some mechanism to ensure that the experiments conducted in the paper actually took place, I see no value in doing the review. Slop merchants have made academic misconduct cheap and easy, and academic promotions and hiring processes have incentivised it.

I think, at least for computer science, we need to move away from papers as the principal output from research. They serve two purposes and are bad at both:

• Communicating science.
• Being a record of specific experiments.

The fixed page limits and rigid style (which largely enforces in-group bias because it’s rarely a documented style, but if you don’t know how to write a paper that sounds like a good {venue} paper, it will be rejected) limit the first. No interactive visualisations, no videos, and often a requirement to minimise colour mean that we’re leaving a bunch of tools for effective communication on the table. And the long delay for publication means that the research is often superseded as state of the art by the time anyone actually sees the publication. There are far better ways of communicating science.

The record is also poor. There’s some work now on artefact evaluation, but that doesn’t happen until after the paper is accepted and it’s rarely mandatory. If you do science well, make everything public and solicit early and wide feedback, you’re actively penalised because you can’t deanonymise yourself and so you find yourself in the ludicrous position of getting negative reviews because the reviewer is familiar with your work and doesn’t see much delta between your work and your paper about the work.

I would love to see a focus on public and transparent work, with papers being invited submissions based on already-evaluated work. Organisations such as IEEE and ACM could then fund at least copyeditors, if not writers, to make sure that the papers are readable and clearly communicate the work.