@david_chisnall A lawyer signs their briefs, and that signature is an explicit affirmation that they believe everything in it to be true. This is an important aspect of how courts work, and I have had ringside seats as lawyers hold each others feet to the fire over specific words.

I have sympathy for the fellow who messed up, but also: in limited amounts.

@david_chisnall the judge: "My 13-year-old nephew knows about hallucinations. My 75-year-old stepmother knows about hallucinations. So, either you made a decision not to be aware of what's happening in the world in using this tool appropriately, which is a willful choice. Or you decided to roll the dice." Lmaooo

@david_chisnall This paper has great examples refuting the vibes-based marketing that NotebookLM is a data-isolated platform producing summaries wholly grounded in the input. Of course it's largely system prompt and training corpus. And fabricates heavily on stereotypes, like falsely claiming the Norwegian faculty are talking about tuition, or producing entire podcasts from a blank PDF without acknowledging that it's blank, all while regurgitating fragments of the system prompt.

Rettberg (2026) AI-generated podcasts:
Synthetic intimacy and
cultural mistranslation in
audio overviews from Google’s
NotebookLM

@david_chisnall Not sure if one can derive that legal tech can't work with LLMs from the use of some incompetent users alone.

That is imho not a problem since at least a year.
https://claude.com/blog/introducing-citations-api

@david_chisnall I would have had sympathy for them maybe a year and a half ago, but at this point it's well known that judges are getting angry with lawyers using them (wasn't one even cited for contempt?) At this point a lawyer using one has to be very much not paying attention to the legal scene, which is really really not a good sign...

@david_chisnall You're replying to people as though this post isn't in that increased sarcasm range, so..

The lawyer didn't even read the summary, because they surely would have noticed that the LLM had hallucinated witnesses. What should a client do with such an inept lawyer? Contempt is getting off easy, they should be disbarred.

Also, the FTC probably isn't going to do a lot besides limit free speech for the next couple of years. I wouldn't hold my breath waiting for them to solve anything.

@david_chisnall
At this moment we really do need to laser focus on the biggest single risk being the gap between what the tech does and what non-technical C-suite folks and other powerful decisionmakers are being told it can do by the world's highest paid sales teams.

@david_chisnall

sympathy for a pathetic, lazy, rich asshole?

he should have his license to practice law revoked.

also, he committed perjury. he should be in prison.

@david_chisnall This is why I always refuse consent (while I can) to AI transcription and summarization. While transcription may be kinda-sorta OK and it probably uses older forms of speech-to-text ML, summarization is a big no-no because the probabilistic word extruder is liable to put words in my mouth that I never said.

https://sfba.social/@drahardja/117262408172666801

@david_chisnall I remain intensely frustrated every time someone says it’s me than just likely token generation, because clearly that’s still the enduring core of what happens. Everything can get better trained and given more APIs, but it will always be likely token generation

@david_chisnall i am not a lawyer but even i am aware of several cases like this where ChatGPT hallucinated facts, laws or legal precedents during court proceedings that got high-profile press coverage, so i have very little sympathy for a lawyer who isn't aware of those things.

@david_chisnall

i have no sympathy for this lawyer at all, he was playing with people's lives because he responded to AI advertising? he should go to jail, i wonder how many of his clients have because of his incompetence

@david_chisnall On one hand, it's the responsibility of a licensed attorney to be more careful than that; and anyone who has actually been keeping up with the profession should also be aware by now that AI has been a problem in courts.

On the other hand, if your "AI summary" product fundamentally doesn't do what you market it as, but sometimes basically succeeds anyway, while failing horribly other times... shouldn't there be consequences for that?

"You'd have to be stupid to believe our product actually works" isn't actually a defense anyway.