Jim shares his adventure into running LLMs on his own hardware. For him it's less about saving money and more about privacy — working in healthcare, he can't send patient data to the cloud.
- App vs. model: Claude Code and Codex are applications, not models. Features like plan mode come from the app. (Wolf's "Opus Plan" is a Claude Code mode that uses Sonnet 4.6 for most work and Opus 4.8 for planning.)
- Ollama makes local models easy — ~15-min install, runs on macOS/Linux/Windows, and exposes a REST API. Not to be confused with Meta's Llama models. Example: `ollama run llama3`.
- Parameters & training: Think of an 8B model as "8 billion knobs." Training randomly initializes them, then refines predictions over billions of iterations. Wolf ties this to Markov models (parameters ≈ weighted edges) and the Bayes episode (random init = priors).
- Fitting big models in memory: Quantization shrinks 32-bit parameters down to ~4 bits. Mixture of Experts (MoE) keeps only part of a model active (e.g., Llama 4 is ~108B params but ~17B active).
- Jim's tests (M1 Mac Studio, 64 GB), asking why H₂O is liquid: Llama 4 Scout took ~10–15 min and maxed out RAM/swap; Llama 3 (8B) answered in ~31s; Qwen (36B) gave the best answer in just 34s.
- The open question: Is local "good enough"? Wolf's real test isn't trivia — can a local model write and iterate on an 8-page implementation plan? (Homework for Wolf's 128 GB MacBook Pro.)
- Build your own: Fine-tune an existing model or train from scratch. Jim's dream: a local model fine-tuned on his DB schema + 2,000 SQL queries so users could ask in plain English and get runnable Postgres — no cloud required. Browse Hugging Face for specialized models
Hosts:
Jim McQuillan can be reached at jam@RuntimeArguments.fm
Wolf can be reached at wolf@RuntimeArguments.fm
Follow us on Mastodon: @RuntimeArguments@hachyderm.io
If you have feedback for us, please send it to feedback@RuntimeArguments.fm
Checkout our webpage at http://RuntimeArguments.fm
Theme music:
Dawn by nuer self, from the album Digital Sky
