"Senate Republicans blocked an attempt by Democrats on Thursday to
stop a Trump administration pilot program for Medicare that uses artificial intelligence to approve or deny physician-ordered care."
You can feed the same person into the LLM in three completely separate sessions and you will get three different recommendations for some of these people.
This is why this is a problem.
The other problem is that, and I feel like I should explain this in bold text:
NO HUMAN IS THINKING
Decisions concerning humans should be made by humans, and not a string of probabilistic math plausibility generator that spits out things that convincingly sound like something a human would say.
And even if it was, the only way it could arguably be fair at all is if it was deterministic.
It's like if it was the perfect fair judge, you'd go to the LLM judge and after the prosecution and defense have concluded their case and the judge has to "deliberate", you'd spit out 10 different judgements, and if they aren't all the same, you fire the LLM judge and get a real person to handle the judgment.
And then there's the issue of accountability. Who is liable when a claim is denied and someone dies of a preventable illness?
Well, with an LLM, no one fuckin' knows, do they? Suddenly it's not an employee it's just institutional, these things happen, heh. We'll have to go to trial to clear this up, or you can have a settlement, which serves business just fine, really.
In fact, having a human in the loop and real accountablity? How tedious. Better to say 'it was instance 9 billion of the autocomplete machine, and we have no idea why it came up with that determination, but it sure did the best it could with the data it was provided.....that time. That particular time we ran it and decided to trust those results.
Like at a bare minimum, if it's non-deterministic, then you can at least treat it like you're polling an unreliable population.
You run it hundreds or thousands of times and investigate and weed out outliers until you come up with a somewhat reliable consensus position, with the general caveat of your methodology and how you came up with it, and how many times you ran the simulation, and repeating the full data in detail somewhere so it can be checked by someone, providing a smaller summary of what happened via general consumption....
That at least approaches science and computer engineering. Still not deterministic, but if you're going to make life or death decisions based on it, you'd better run that motherfucker a hundred or a thousand times before you can be sure of its answer.
People won't do that, though, and that's one issue. The other being that this on balance a terrible idea, anyway.
Polling a non-thinking non-deterministic probabilistic answer generator is inherently not a search for truth, anyway. Consensus? Sure. But "I asked the answer machine 100 times and go back these answers" is....what is that? Useful? How is that useful?
We've built deterministic systems for years, people are trained to ask the computer for a reasonably static answer. Why would they ask multiple times or subject it to a 'polling' methodology..... much less spend the expensive tokens it would take to do that?
And avoiding accountability is the American business MBA superpower and I can see why it's used in cases like this.
