I need to rant about local models

Anthropic liked this video so much they extended Fable availability until Sunday. Keep em going Theo

Didn't realize sending death threats over criticizing local AI was on the table, I gotta step my game up.

My favorite AI addicted web dev

I use my local spark model for handling data that cannot be permitted to be run through a cloud provider - because once you put data into a prompt for a cloud model, you have no control over what happens to it. None of these labs can be trusted with private data, they already proved that when they made the models in the first place. So, mostly I use it to read my email, and organize my private data, setting up the process using my frontier AI, but executing it with local inference. Slow ingestion of Air Xiv papers into a database meant for my frontier AI to query. Automations that don't need a strong intelligence.

To correct an incomplete point: with MoE models, you can offload experts to RAM, though prefill performance can take a hit—but that's acceptable. I run Qwen3.6-35B-MTP-IMAT-IQ4-XS (which is Q8nextn, sitting at ~19GB) on an RTX 3080 with 8GB VRAM + 32GB system RAM, on a 5-year-old Asus laptop, alongside a small model plus TTS and STT models. Basically, I built my own voice assistant because I like ChatGPT's voice mode—it's just a PWA running over a WireGuard virtual network. I get the audio response in less than 2 seconds most of the time, depending on the request. And yes, it's not flawless, but I use it as a glorified Siri. Besides that, there are breakthroughs in clustering, and I have hope that specialized local AIs will do a better job than SOTA models in their respective fields. Finally, about Zhipu: $40 for 3 months with only a 5-hour limit is insane value without any equivalent. My max usage was 3 billion tokens in a month. So, I'm not a Cassandra. The thing is, PCs weren't built for AI—PCIe speed, RAM, VRAM... There is room for improvement in hardware, architecture, and the software stack. There is room for improvement in the fundamentals of quantization and LLM architecture. There's also the fine-tuning process, which is becoming accessible to everyone to fit their specific usage. BUT local ai is needed in many sectors who for different reason and this could help us making local LLM feasible for coding.

Theo out there burning 25k a month in tokens he's getting comped by cloud providers, and then complains a local LLM would cost $2k a year in electricity

you are thinking of model with a specific use for coding. If you just want a vision caption model, a chatbot, an image generation model, etc. Local models are just fine. You want agentic coding, you need a larger model.

I tried to comment first but my local model just managed to post this reply now

"I am not using one agent, I'm going between zero and forty" You have to realize how insane that cost is to an average person or company. Someone who is tinkering or a company that is not yet profitable can’t afford that long term. Especially if you are running frontier models at their current cost. Are the other options slower? Yeah. Is the hardware insanely priced for open source frontier models? Yes. But a few months of running Opus or GPT5.6 (nevermind Fable or Sol, pending availability) is going to eclipse the hardware cost of a medium-tier GPU setup in a few months of use.

COO of an AI company here. Not everyone needs opus 4.6 Small models have reached beyond GPT 4.1 and for customer service, Q&A and context translations have proven a safe option for companies that want to control their data, $ 10k is a small price for a pc that's can run Qwen 3.6 27b at 15 answers per minute at 90% precision.