How do you improve on this?
- Find more data sources
- Filter out bad data, both via humans and regexes
- RAG!
Now we're seeing a demo of https://fighthealthinsurance.com/, what this talk has been all about.
How do you improve on this?
- Find more data sources
- Filter out bad data, both via humans and regexes
- RAG!
How do we do the fine-tuning?
- Correct data structure
- Pile o' shell scripts
- Axolotl! this was clutch. https://axolotl.ai/
Once we have the data, how do we pick a model to fine tune?
- Make sure the model will fit in memory
- Check the license. Then check it again.
- Then consider good base performance, that's got good scores on best-related tasks
How do we do the fine-tuning?
- Correct data structure
- Pile o' shell scripts
- Axolotl! this was clutch. https://axolotl.ai/
First problem: training data!
- Insurance companies won't give the data
- Doctor's offices have the data, but it's hard for them to give it out
- Reddit has some data! But there's sample bias
- Insurance commissioners!
Once we have the data, how do we pick a model to fine tune?
- Make sure the model will fit in memory
- Check the license. Then check it again.
- Then consider good base performance, that's got good scores on best-related tasks
This includes:
- An ML model to take denials and produce appeals
- Frontend interface for people to access the model
- Library to remove and put the personal data back in
First problem: training data!
- Insurance companies won't give the data
- Doctor's offices have the data, but it's hard for them to give it out
- Reddit has some data! But there's sample bias
- Insurance commissioners!
We have a social problem, and we're going to try and duct tape it with computers!
"Making healthcare leak a little less fast"
How? AI to take denials and produce appeals.
This includes:
- An ML model to take denials and produce appeals
- Frontend interface for people to access the model
- Library to remove and put the personal data back in
What's the problem? Health Insurance in America frequently denies medical care, Insurance _might_ be using AI to deny claims, and appealing claim denials is hard
And they have a lot more money than us!
We have a social problem, and we're going to try and duct tape it with computers!
"Making healthcare leak a little less fast"
How? AI to take denials and produce appeals.
What's the problem? Health Insurance in America frequently denies medical care, Insurance _might_ be using AI to deny claims, and appealing claim denials is hard
And they have a lot more money than us!
Net talk at AI #bythebay: "AI For Good: Fighting Health Insurance with AI" by Holden Karau.
Starting with CWs around surgery, hospitals, trans references, and broken bones.
Net talk at AI #bythebay: "AI For Good: Fighting Health Insurance with AI" by Holden Karau.
Starting with CWs around surgery, hospitals, trans references, and broken bones.
Uber has a combined interface for doing debugging, visualizing, and evals on an agent.
(This would be amazing to have at work)
This tooling supports local servers, online servers, and versioning for rollbacks and testing.
Evals can even be run offline, or in CI/CD
Why use the Agent Protocol? Standardized:
- Chat history management
- Low Level APIs
- Observability (would love to know more about this)
Uber has a combined interface for doing debugging, visualizing, and evals on an agent.
(This would be amazing to have at work)
AgentFX is a standard interface to agents at Uber.
the runtime uses the OSS Agent Protocol (with langchain.ai)
Provides Threads, Runs, Messages, and Agents under one protocol.
Why use the Agent Protocol? Standardized:
- Chat history management
- Low Level APIs
- Observability (would love to know more about this)
Team is trying to provide the best tools for working in AI.
Their verticals:
- Agent Builder for rapid prototyping
- Write production-ready AI applications with AgentFX
- Monitor and deploy with Agent Studio
AgentFX is a standard interface to agents at Uber.
the runtime uses the OSS Agent Protocol (with langchain.ai)
Provides Threads, Runs, Messages, and Agents under one protocol.
Uber has a sense of levels for AI agents.
L1 - Responders, simple chat bot
L2 - Assistants, ChatGPT without reasoning
L3 - Collaborators, ChatGPT with deep research, Claude Code
L4 - Experts, AI agents SWE teams that build new features
L5 - Autonomous Teams - AI manager builds team of agents to get things done
Team is trying to provide the best tools for working in AI.
Their verticals:
- Agent Builder for rapid prototyping
- Write production-ready AI applications with AgentFX
- Monitor and deploy with Agent Studio
Uber has a sense of levels for AI agents.
L1 - Responders, simple chat bot
L2 - Assistants, ChatGPT without reasoning
L3 - Collaborators, ChatGPT with deep research, Claude Code
L4 - Experts, AI agents SWE teams that build new features
L5 - Autonomous Teams - AI manager builds team of agents to get things done
Next talk: "Uber's Multi-Agent Platform" by Jamieson Leibovitch
This is one of the talks I've been most excited for!
Jamieson has been at Uber for 4 years, starting inside embedded hardware and moving into the Agent Platform team.
Next talk: "Uber's Multi-Agent Platform" by Jamieson Leibovitch
This is one of the talks I've been most excited for!
Example 3: Voice transcription project
Lesson 4: Voice is faster than typing (not sure I agree with this, but fascinating take!)
using superwhisper, MacWhisper
Don't be afraid of starting over if the context is getting unwieldy!