Running Deepseek R1 671b fully locally on a $2000 EPYC server. Idle wattage is just 60w while with 260w under load.
This setup runs a 671B model in Q4 quantization at 3-4 TPS, running a Q8 would need something beefier. To run a 671B model in the original Q8 at 6-8 TPS you'd need a dual socket EPYC server motherboard with 768GB of RAM.
The idea that LLMs use an inordinate amount of power to run is very much outdated at this point.
https://digitalspaceport.com/how-to-run-deepseek-r1-671b-fully-locally-on-2000-epyc-rig/
