I said I *didn't* think Taylor Lorenz was an industry shill but... wow... now I'm starting to rethink that.
You know I might be wrong, but what is wrong with watermarks?
9
Replies
4
Boosts
@futurebird there were doubts about her integrity for a long time, shill is the least surprising pivot.
Let me outline my understanding of "LLM Watermarking"
It is possible to embed special characters and patterns in the output of LLMs that would make text generated by these systems easier to reliably detect. This is mainly being done so that when LLMs scrape the web for new information they can avoid ingesting machine generated content.**
When you train an LLM on machine generated content it may lead to "model collapse."
**see next post for correction
The companies that run LLMs can also use this for PR to calm concerns from the public about the proliferation of such content.
From a CS perspective I can't think of any way to have a watermark that couldn't be easily defeated through additional processing. LLM dependent people currently take pains to remove em dashes so this new wrinkle has some of them in a panic.
But, why not just be honest and say you used and LLM? Why so bashful?
From a CS perspective I can't think of any way to have a watermark that couldn't be easily defeated through additional processing
The way that the current ones work is that they tweak the probabilities to generate specific patterns in the output. Where there's an equal probability of two words following another in the raw weights, they'll tune it so that there's a higher probability of one than the other.
If you know the weights and know the biassing factor, you can look at each word pair and see what the probability would be of the model generating that.
This means that the watermark is smeared all over the output. And it's not a binary thing though, each pair of word contributes something to the probability of matching the watermark and looking at the whole thing will give you a probability at the end.
Changing every other word should give you close to a 0% of matching the watermark but, at that point, why bother using the LLM at all? If you're going to change half of the words, you may as well just write them yourself. And that's a problem for people who want to share low-effort slop and pretend to be creative.
![Chris [list of emoji]](https://nfts.freeradical.zone/accounts/avatars/109/127/168/397/951/576/original/e8a0b6da977923a8.jpeg "Chris [list of emoji]")
I wonder if there's an efficient way to procedurally reword stuff to defeat the watermark. (The goal here is to hasten model collapse.)
@david_chisnall @futurebird this also means that you have know the weights and know the biassing factor to be able read that watermark reliably.
And I am going to bet that Anthropic is not going to release those.
In other words, Anthropic will be the only company offering the slop generator, and the service to tell that something got generated by it.
That's real power right there. Would they not "tweak" the results if it was useful for them, say, to create a plagiarism scandal around someone?
@david_chisnall @futurebird @404mediaco has a pretty good article OpEd on it as well.
But, why not just be honest and say you used and LLM? Why so bashful?
In my experience, it's because they know most people absolutely hate having slop sprayed at their face, so they'll go to great lengths to disguise it in an attempt to "get one over" on them.
It resembles the same type of shit with people who "test" someone else's allergies by sneaking stuff into their food: They're trying to "catch them in a lie" with the false assumption that they'll be fine with it if they don't know it's there.
And even just that type of behavior is extremely concerning for me, even if it doesn't involve allergens that can Actually Kill People. It shows a clear lack of respect for others, and deserves a swift football kick to the unmentionables and immediate expulsion.
Isn't that amazing? Even people who really love boosting AI feel hurt when someone feeds them AI generated content.
It's embarrassing to admit to liking or not noticing AI generated content. There was a music video I really enjoyed a few months back and I think it might have AI generated music or AI assisted animation. The creator hasn't been very transparent and everyone who liked it is kind of worried and unhappy.
When the same account posted something new I ignored it.
