Local AI: How to Run Generative AI on Your Own Hardware

Local AI means running a model on hardware you control instead of sending every request to a third-party API. That can be a laptop, a workstation, a home server or a rented GPU box you administer yourself.

The question people actually arrive with is not whether local AI works — it does — but what will run on the machine I already own. That question has an arithmetic answer, and it is worth getting to before anything else.

Start with memory, because memory decides everything

Almost every “can I run this?” question reduces to whether the model weights plus the context cache fit in memory. For quantized language models the estimate is close to linear in parameter count:

• Q4_K_M (~4.5 bits per weight) — roughly 0.56 GB per billion parameters
• Q5_K_M (~5.5 bits) — roughly 0.69 GB per billion parameters
• Q8_0 (8-bit) — roughly 1.0 GB per billion parameters
• FP16 (unquantized) — roughly 2.0 GB per billion parameters

So an 8B model at Q4 lands near 4.5 GB of weights, a 32B model near 18 GB, and a 70B model near 40 GB. Add context on top: a long conversation or a large document window can add several gigabytes of KV cache, which is why a model that loads fine at 4K context can fail at 32K.

Translated into hardware tiers, the practical ceilings look like this:

The gap between tiers matters more than the gap between models inside a tier. Moving from 12 GB to 24 GB changes what class of model you can run; moving between two 14B models does not.

Why run AI locally

The honest case for local inference is narrower than the enthusiasm around it, and stronger where it applies:

• Privacy. Prompts, documents and generated output never leave infrastructure you control. For client work under NDA, or anything touching personal data, this is often the only argument that matters.
• Version pinning. A hosted model can change under you without notice. A local checkpoint does not. If you have built prompts, evaluations or a product around specific behaviour, that stability is worth real money.
• Offline operation. Useful when connectivity is unreliable or the machine is deliberately air-gapped.
• Customisation. LoRAs, custom quantizations, private fine-tunes, ControlNet-style guidance and custom nodes are all things hosted APIs either restrict or do not expose.
• Cost, eventually. Local inference replaces a per-token bill with a fixed hardware cost. Whether that is cheaper depends entirely on volume — see below.

Local LLMs: which runtime

The runtime matters as much as the model, and the four common choices are genuinely different tools rather than variations on one idea.

Two practical notes. Ollama and LM Studio both expose /v1/chat/completions without extra work; llama.cpp needs its separate server binary to do the same. And the update cadence differs sharply — llama.cpp ships daily, Ollama every week or two, LM Studio monthly — which decides how quickly a brand-new model architecture becomes runnable.

Two tools worth knowing at the edges: llamafile collapses model and runtime into one portable executable, and Rapid-MLX targets Apple Silicon specifically with an OpenAI-compatible API.

Throughput: what independent benchmarks actually measure

Vendor specifications describe what a card can do in theory. What matters is what it does with a quantized model on an ordinary desk — and there the most useful number is not the fastest result, but the point at which a card stops working at all.

The table below comes from an independent benchmark run on llama.cpp b3520 at Q4_K_M, 2,048 tokens of context, single-batch (one person at a keyboard rather than a server), measured warm so cold-start is excluded. The 70B column uses Q2_K, because nothing larger fits on a 24 GB card.

Read the OOM cells first, because they carry the real information. A 12 GB RTX 3060 runs a 13B model at a perfectly usable 22 tok/s and then simply cannot load a 34B at all. No amount of patience fixes that: it is the memory wall from earlier in this article showing up as a hard failure rather than a slowdown.

The Apple row is the other one worth studying. An M3 Max with 64 GB is slower than a 4090 at every size both can run — 40 tok/s against 135 on a 7B — yet it is the only consumer entry that completes the 70B column at all, because unified memory holds a model no 24 GB card can fit. Slower, but it finishes.

One user is not the same as four

Single-user figures flatter local setups. A published benchmark of Qwen3-30B-A3B at Q6_K_XL on an RTX 5090 recorded 192 tokens per second with a 0.17-second time-to-first-token for a single request — and 47 tokens per second with a 0.75-second TTFT once four requests ran concurrently. Roughly a quarter of the throughput, for four times the work.

That is the honest shape of self-hosting. It is excellent for one person, and it becomes a capacity-planning problem the moment a colleague, a second app or an agent loop shows up.

The numeric format matters more than the next GPU

A study of consumer Blackwell cards measured the same Qwen3-8B under three numeric formats on one RTX 5090, at 8k context and concurrency 8, running vLLM 0.12 with power sampled through NVIDIA DCGM:

NVFP4 delivered about 1.58× the throughput of the BF16 baseline and cut energy per million tokens by roughly 41%, at a measured quality cost of 2–4% across MMLU, GSM8K and HellaSwag. Whether that trade is acceptable depends on the work, but it is a better return than most hardware upgrades deliver.

The same study found that doubling context roughly halves throughput: 411 tok/s at 8k became 232 tok/s at 16k on the same card. Long-context work is expensive in a way that is easy to forget when benchmarks quote short prompts.

Across the card range, on the same 8k RAG workload at concurrency 8:

The time-to-first-token column is the one to watch: 9.7 seconds on the 5060 Ti against 0.45 seconds on the 5090, for identical work. For overnight batch jobs that gap barely matters. For anything a person waits on, it is the difference between a tool and a frustration.

One more caution the tables do not show: once a model spills out of VRAM into system RAM, throughput collapses toward the CPU row regardless of how fast the GPU is. Fitting the model matters more than the GPU’s headline speed.

Local image generation

Image models are the most forgiving category — many run comfortably in 8–12 GB. They are usually driven through a node graph such as ComfyUI, where a workflow chains a checkpoint, a VAE, text encoders, LoRAs, guidance models, upscalers and editing nodes.

The node-graph approach is more setup than a single prompt box, but it is what makes the output reproducible: the graph is the recipe, and it can be versioned and shared. See Open Source AI Image Generators for the model landscape.

Local video generation

Video is where hardware stops being a detail. Memory scales with frame count as well as resolution and model size, and the gap between running a model unquantized and running it quantized is the difference between a datacentre card and a gaming GPU:

Read that 14B row carefully, because it is the single most useful fact about local video: the same model needs 54 GB unquantized and roughly 6 GB as a GGUF at 480p. Quantization and offloading are not optimisations here — they are the entire reason local video generation is possible on consumer hardware at all.

The cost is quality and speed, and the honest way to choose is to generate the same prompt at two quantization levels and decide whether you can see the difference in the output you actually ship. For the model landscape see Open Source AI Video Generators, and ComfyUI Video Workflows for how the pipeline fits together.

What a video generation actually costs in time

Memory tells you whether a model loads. It says nothing about whether you will wait long enough to stop using it. One benchmark of Wan2.1 text-to-video 14B — 33 frames at 16fps, so a two-second clip, at 30 steps — recorded these times:

Two things stand out. A two-second clip takes 281 seconds on an RTX 4090 at 480p — roughly 140× real time. And the 4090, the strongest consumer card in that test, is marked not supported at 720p on the 14B model: the 24 GB ceiling again, this time as a wall rather than a wait. Reports elsewhere put the practical cut-off in the same region, with cards at 12 GB and below failing outright on heavy 720p workflows.

Plan around the clock, not the VRAM table. On consumer hardware, local video generation is an overnight-batch medium rather than an interactive one — which changes how you work with it more than any quality difference does.

Hardware: what to buy

Beyond the GPU, two things get underestimated. Disk: a working collection of checkpoints, LoRAs and video models runs to hundreds of gigabytes, and quantized variants multiply that. System RAM: it does not make inference fast, but it is what allows offloading to work at all rather than failing outright.

Apple Silicon

Apple’s unified memory is genuinely different rather than marketing: CPU and GPU draw from one pool, so a 64 GB Mac can hold a model that would need a 64 GB discrete card. For local LLMs this makes memory capacity, not GPU core count, the specification that decides what you can run — a slower Mac with more memory beats a faster one with less.

MLX is the Apple-native path; other runtimes reach the GPU through Metal or PyTorch MPS. The trade-off is ecosystem: CUDA-first projects reach Apple Silicon later, and some never do.

NVIDIA and AMD

CUDA remains the default assumption of most generative-AI projects, which makes NVIDIA the path of least resistance. AMD support through ROCm has improved substantially, but “improved” is not “equivalent” — before buying AMD hardware for a specific workflow, check that the actual custom nodes and model loaders you intend to use list ROCm support, not just the base framework.

Local APIs turn a model into infrastructure

A model running in a terminal is a demo. A model behind an HTTP endpoint is infrastructure. The reason OpenAI-compatible endpoints matter so much is leverage: existing clients, agent frameworks and RAG systems can usually be pointed at http://localhost:11434/v1 by changing a base URL and passing a dummy key, with no other integration work.

That one property is what lets a local model slot into a stack that was written against a hosted API.

Privacy is a property of the workflow, not the install

Installing a local interface does not make a pipeline local. ComfyUI can mix local models with remote API nodes in the same graph; an LLM stack can call out to a hosted embedding model or a web-search tool without announcing it. If privacy is the reason for the setup, the workflow has to be audited node by node, and the machine’s outbound traffic is the only real proof.

Local versus cloud: where the line actually falls

The useful comparison is not philosophical, it is volume. Hardware is a fixed cost paid once; API calls are a variable cost paid forever. Below a certain monthly usage the API is cheaper and always will be; above it, the hardware pays for itself and then keeps paying. Work out your own crossover before buying anything — take your actual monthly token or image spend and divide the cost of the GPU by it.

One measurement worth having before that calculation: the Blackwell study above put the electricity cost of self-hosted inference at roughly $0.001–0.04 per million tokens at a US rate of $0.12/kWh, which it estimated to be 40–200× cheaper per token than budget-tier cloud APIs. That is the running cost only. It excludes the hardware, which is the whole question.

Most working setups end up hybrid for reasons that have nothing to do with cost: local for sensitive material, routine generation and development; hosted for the largest frontier models, demand spikes and capabilities nobody has open-sourced yet.

A practical local-AI stack

1. A model runtime — Ollama or llama.cpp for language, ComfyUI for media.
2. An OpenAI-compatible API layer, so applications are not coupled to the runtime.
3. Model storage with enough headroom, and a naming convention you will still understand in six months.
4. Monitoring for memory pressure and latency — the first sign of trouble is throughput quietly collapsing as something spills to system RAM.
5. Your application or automation layer on top.

Related The Signal coverage

• Open Source LLMs
• Open Source AI Image Generators
• Open Source AI Video Generators
• Rapid-MLX
• llamafile

Sources

The Signal did not run these benchmarks. Every figure above is taken from a published third-party measurement with a stated method — the benchmark suite, quantization, context length and concurrency are named alongside each table so you can judge how closely they match your own case. Results vary with driver version, cooling, CPU and background load, sometimes substantially. Treat them as well-documented planning estimates, not guarantees.

• LLM Tokens/Sec Benchmarks 2026 (llama.cpp, Q4_K_M)
• Viability and Performance of a Private LLM Server for SMBs (arXiv)
• Private LLM Inference on Consumer Blackwell GPUs (arXiv)
• Wan2.1 GPU performance testing across GPUs (InstaSD)
• 2026 Local LLM Hardware Guide: VRAM Tiers + GPUs
• Ollama vs. LM Studio vs. llama.cpp (Machine Learning Mastery)
• Video Generation VRAM Requirements 2026
• ComfyUI official repository
• MLX official repository
• llamafile official repository