Open sourcing Dicer: Databricks's auto-sharder
https://www.databricks.com/blog/open-sourcing-dicer-databricks-auto-sharder
#HackerNews #OpenSourcing #Dicer #Databricks #AutoSharder #DataEngineering #BigData
From text to token: How tokenization pipelines work
https://www.paradedb.com/blog/when-tokenization-becomes-token
#HackerNews #tokenization #tokenizationpipelines #machinelearning #dataengineering #naturalanguageprocessing
So you wanna build a local RAG?
https://blog.yakkomajuri.com/blog/local-rag
#HackerNews #localRAG #buildingRAG #techblog #AIdevelopment #dataengineering