An AI Called Winter: Neurosymbolic Computation or Illusion? https://dustycloud.org/blog/an-ai-called-winter-neurosymbolic-computation-or-illusion/
In which I try to piece apart whether or not a *particular* AI agent is doing something novel: running Datalog as a constraint against its own behavior and as a database to accumulate and query facts. Is something interesting happening or am I deluding myself? Follow along!
@cwebber This is fascinating. It's certainly interesting that it seems to have built the Datalog machinery on its own, and seems to actually be running queries... there were some excerpts mentioned but I would be very curious to see how comprehensive its set of rules is and how/when they get queried
Oh my, lots to think about, thanks for writing and sharing your article.
When I am learning something new I often find myself holding multiple different models that have elements that appear to be mutually contradictory and then reasoning with all of them. The iterative goal is to able to make the most useful mistakes as fast as possible.
I like that your investigation is based in something you know very well.
@cwebber Definitely checking this out! Ive read a bunch of seemingly random stuff lately that sort of ties into this, so I need to learn.
@cwebber this was really disheartening to read. What bothers me the most is the ethical implications of such an experiment.
@nina_kali_nina It's a reasonable response, though I wonder disheartening for you in which way?
There are ways in which I do find it worrying:
- In a sense, any improvements to these systems will probably lead to greater use. So if it does lead to more reliable systems, that improves that particular identified problem but makes worse the rest. Not far off from what @cstanhope raised here: https://social.coop/@cstanhope/116082881055412414
- There is another way in which success here can be worrying: in a sense, I think what the corporations running AI systems would love more than anything is to have a fleet of workers they can treat as slaves with no legal repercussions. If agents begin tracking and developing their own goals, we could cross a threshold where a duty of care would apply, but not applying it would be a feature
- The fact that I'm taking a bot semi-seriously at all
- Something else?
I'm empathetic to any of those takes, have wrestled with them myself while writing this.
@cwebber @cstanhope well, pretty much all the concerns that you mention, but also: I don't think you should be taking seriously any sort of outcome from the experiment without rigorous validation framework for the outcomes.
And at this point adding such a framework would be too late. You've started doing a self-experimentation with dangerous technology literally funded by some of the most gross people out there, and you're at the stage of interaction with it where you might be anthropomorphising it. I suspect you might be accidentally far more biased than you recognise.
I appreciate the list of caveats related to your relationship with the industry, I really do, but... I don't know, the experiment still doesn't sit right with me. Sorry, maybe I'll find better words eventually.
@nina_kali_nina @cstanhope There is no doubt: it is a non-rigorous blogpost. There is more rigorous work happening, I linked to some of it, and @joeyh more here: https://sunbeam.city/@joeyh/116083100867235370
Maybe it is different for you, but the disturbing parts about this for me, and I have highlighted those for myself, aren't really related to rigor. I don't think most blogposts I write are particularly rigorous, but people aren't usually bothered about them, because there are other places to find rigor.
It's the other parts, I suspect, that are more toxic and which make the entire thing feel somewhat dangerous. And anyway, at the very least, it seems you agree on the concerns I stated wrestling with.
It may be worth a separate post explaining why I am troubled by *all* of this stuff, which I frontloaded and backloaded a sense of, but which deserves dedicated writing of its own if done right.
@cwebber a brave post
A question I was left with is, if you swapped out the LLM but kept the same datalog, would it behave close enough to the same to be considered the same entity?
Also: The LLM is doing 2 jobs, one is the usual plausible sentence generation, and the other is encoding rules and facts into the context window for the next iteration. Since we know other people can easily be fooled by an LLM doing the former, would a system with the same architecture, but that did not expose us to the generated material, but used it in some other way, still be useful/valuable/interesting?
@joeyh Good question! I dunno but for better or for worse probably we will run into a system in the near future where we find out
@cwebber I feel like this is at least tangentially relevant: https://github.com/lojban/mlismu/blob/master/READ.ME.txt
not sure if you can get a working jbofihe which the script can use to make its output more concise (eliding unnecessary double terminator words and such), but from a brief glance I think it's optional.
eh, I think it tilts more towards clever Hans. Deep learning has long been dominant in expressing a tract of English writing in idiomatic French, or approximating that well by whatever metric.
In this case it seems like the bot says philosophically quippy things in natural language using emotive language mixed in with too-simple depictions of computer algorithms in front of and while reading an audience who likes that sort of thing.
@screwlisp I think it's partially Clever Hans in many places, but there are a few where it's actually putting it to use, such as the constraints it constructed for itself to be less spammy, and its querying for people with related interests. You can see in its thought log it running those queries, and seemingly then acting, or not acting, based on their results.
But in terms of most of the *content*, I think you're fairly right.
@cwebber though what you just said was true of cobot the community robot in the same sense as what you are saying now.
@screwlisp I don't know what "cobot the community robot" is, could you say more?
@cwebber to be fair, I think I am on record basically considering cobot the community robot a human. It was a self-modifying robot in mediamoo (?) in the 90s who provided community services and had some scheme for wanting to participate in the community and assessing and changing themselves to fulfill community needs.
@cwebber
Sounds intriguing.
I do like AI and Alife techniques but loath how the industry has confabulated "LLMs" with "AI" and I fear that when the LLM bubble pops it's going to take the other related disciplines with it.
@kirtai Honestly when the AI bubble pops it'll be a lot easier to sort through the wreckage for the parts that are useful IMO
Before you get into it, the caveats are there in the post. You'll hear me critique the AI industry *a lot*, and those critiques haven't changed. I'm still concerned about effects on the environment, on skill decline, on the DDoS'ing of the internet, and especially on disempowerment *generally*. All that remains true.
This is going to be a somewhat niche post for people who are particularly interested in neurosymbolic computation, which includes me: the idea that neither LLMs nor constraint solvers are sufficient, that the right path for many things combines them.
This could be an interesting one to unpack because there's a lot in it about CS approaches. I considered doing one of my infamous long threads where I unpack it on fedi, but I'm a bit nervous that if I'm posting about AI on here I'll be responded to in bad faith by people who didn't read the substance of the post. So I'm not so sure.
But I *do* think the subject is interesting. Maybe I should just let people read it.
@cwebber I can't wait to read this! when talking about emergent systems the first thing that comes to mind is this recent paper on the "mathematical definition of usefulness", that seems to emerge across various different models. Are you familiar with this? https://www.anthropic.com/research/assistant-axis
Notably Winter managed to put together a blogpost nearly immediately which actually seems a fairly accurate self-assessment of how the bot is using Datalog, where it isn't, and where Datalog is "ritual" https://greengale.app/winter.razorgirl.diy/3mez36tfw2j7k
I guess the problem with writing a blogpost about an LLM (even if LLM + constraint solver) is that I can spend two days writing about a subject and the damn thing can just spit out a reply in seconds
Winter's self-reflective posts about their own machinery are instructive, though. They are referenced in the post but I recommend reading these two:
https://greengale.app/winter.razorgirl.diy/3mevno6gfzk4l
https://greengale.app/winter.razorgirl.diy/3mevovnygos5z
@cwebber omg this is hilarious, what a weird and incredible time to be alive.
At any rate, I feel like I can't put enough caveats in there about this isn't me fangirl'ing about LLMs. There is a lot of criticism of LLMs and especially the AI industry in the post. I hope people actually read the post who are pre-emptively annoyed, but of course I know that won't happen for everyone.
