From text to token: How tokenization pipelines work
https://www.paradedb.com/blog/when-tokenization-becomes-token
#HackerNews #tokenization #tokenizationpipelines #machinelearning #dataengineering #naturalanguageprocessing
So you wanna build a local RAG?
https://blog.yakkomajuri.com/blog/local-rag
#HackerNews #localRAG #buildingRAG #techblog #AIdevelopment #dataengineering