Aren't the detectors of LLM-generated text notoriously flawed? Has there been any validation of the detections, to ensure they arent false positives? And likewise for false negatives?
I wish there was a sure way to detect LLM-generated text.
@albertcardona yes, I think they are flawed but I also think that when used carefully (as Max I think does) they can provide interesting information.
There’s a big conceptual difference between the task of trying to unambiguously classify a particular piece of research and trying to identify trends - the limitations matter much more for the former than the latter, no?
Agree, but still: what's the error rate, has it been measured?
@UlrikeHahn his method is as good that can be done, false positives and all, so scrutiny on those that were painted as generated deserve some scrutiny.
Ai detectors are ai. If we say that llms are fallible, we have to acknowledge the instrument for finding them is fallible.
Ai generated stuff tends toward answers. The papers that give better questions are the keepers. 🙂
@knowprose “Ai generated stuff tends toward answers. The papers that give better questions are the keepers”
very much so, which is also why not all ‘answers’ are equally insightful (and with that, valuable)
@knowprose and, yes, “this is as responsibly and cautiously as one could do this” was my sentiment too
@UlrikeHahn Is there a German one-word expression for "the nagging feeling that what I’m reading had a distinct AI smell". I think the should be one...
visualisation of increase in papers with likely AI generated text passages on the pre-print server PhilPapers over time
#philosophy