RE: https://mas.to/@zzt/117217691089439622
btw we literally did this one, an awful.systems post going through how to fool a chatbot at logic puzzles, and the vendors promptly special-cased our examples
3
Replies
1
Boost
from the estimable dizekat
he fed the chatbots river crossing puzzles, they failed hard, the specific puzzles showed up specifically trained a bit later but slight variants still broke
https://awful.systems/post/1769506
https://awful.systems/post/3875809
https://awful.systems/post/4738230
https://awful.systems/post/5053309
https://awful.systems/post/4027490
@davidgerard LLMs can’t do anything that involves keeping track of several entities in spatial, temporal and/or causal relation to each other, unless you train stories about specific cases into them.
This is not surprising, of course, when you don’t expect them to have reasoning capability in the first place but must be very frustrating for AI bros who believe in these things as a religion.
@davidgerard Transposing from game to set theory, I remember a couple of time when I suggested to an AI user that they ask it if the catalog of catalogs which mention themselves should mention itself. Of couse the AI immediately started babbling happily about the Russell paradox...
