tomcp.org – Turn any URL into an MCP server
https://github.com/Ami3466/tomcp
#HackerNews #tomcp #org #MCP #server #URL #shortener #GitHub #project #web #development
Curlie - The Collector of URLs
Some search engines for the web are also based on @Curlie, among other things. You can also enter your websites in this so that they can be found more easily when people search for keywords that relate to your pages.
#websearch #searchengine #collection #url #curlie #internet #webfind #find #webpage #web #followerfriday #website #keywords
deutrino
and 1 other
boosted
added #Shig (https://shig.de), #Encyclia ( @encyclia), #Photon ( @photon), #url.life (https://code.uri.life) and Ibis (https://ibis.wiki) icons to #FediverseIconography at https://iconography.fediverse.info
added #Letterbook (https://letterbook.com) and #Oeee (https://oeee.cafe) and updated the #Loops icon to the newer version by @nclm
added #Shig (https://shig.de), #Encyclia ( @encyclia), #Photon ( @photon), #url.life (https://code.uri.life) and Ibis (https://ibis.wiki) icons to #FediverseIconography at https://iconography.fediverse.info
Okay, Back of the napkin math:
- There are probably 100 million sites and 1.5 billion pages worth indexing in a #search engine
- It takes about 1TB to #index 30 million pages.
- We only care about text on a page.
I define a page as worth indexing if:
- It is not a FAANG site
- It has at least one referrer (no DD Web)
- It's active
So, this means we need 40TB of fast data to make a good index for the internet. That's not "runs locally" sized, but it is nonprofit sized.
My size assumptions are basically as follows:
- #URL
- #TFIDF information
- Text #Embeddings
- Snippet
We can store an index for 30kb. So, for 40TB we can store an full internet index. That's about $500 in storage.
Access time becomes a problem. TFIDF for the whole internet can easily fit in ram. Even with #quantized embeddings, you can only fit 2 million per GB in ram.
Assuming you had enough RAM it could be fast: TF-IDF to get 100 million candidated, #FAISS to sort those, load snippets dynamically, potentially modify rank by referers etc.
6 128 MG #Framework #desktops each with 5tb HDs (plus one raspberry pi to sort the final condidates from the six machines) is enough to replace #Google. That's about $15k.
In two to three years this will be doable on a single machine for around $3k.
By the end of the decade it should be able to be run as an app on a powerful desktop
Three years after that it can run on a #laptop.
Three years after that it can run on a #cellphone.
By #2040 it's a background process on your cellphone.