Open Source LLMs: How to Choose, Run, and Deploy Local Language Models

Open-source LLMs and open-weight language models let developers run language models outside a closed API, inspect inference code, choose their own runtime and often fine-tune or quantize the model for a specific workload.

The most important distinction is that a downloadable checkpoint is not automatically fully open source. The model-weight license, code license, training information and redistribution terms all matter — and, as the licensing section below shows, some of the biggest names in open weights have recently moved away from fully permissive terms.

What is an open-source LLM?

Under the Open Source Initiative’s Open Source AI Definition, a genuinely open AI system should support the freedoms to use, study, modify and share it, with the materials needed for modification. Many popular “open LLMs” are more accurately described as open-weight models because the final parameters are available but the full training stack is not.

How to choose an LLM

Do not choose from a single benchmark score. Evaluate:

• task fit: chat, coding, extraction, agents or reasoning;
• model size and memory requirements;
• context length;
• language support;
• tool/function-calling support;
• quantization availability;
• license and commercial terms;
• runtime support and community maintenance.

Model size and quantization

Quantization stores model weights at lower precision so an LLM can use less memory and often run faster. Common local runtimes support formats such as GGUF and multiple 8-bit, 6-bit, 5-bit or 4-bit quantization schemes, alongside GPU-serving formats like GPTQ and AWQ.

The trade-off is quality versus memory and speed, but the size of that trade-off depends heavily on which specific quantization method and kernel you use — “4-bit” is not one number, it’s a family of implementations with meaningfully different results.

Quantization in practice: what the numbers actually show

A systematic benchmark (methodology: Qwen2.5-32B-Instruct, NVIDIA H200, all methods quantized to 4-bit, Wikitext-2 for perplexity, HumanEval for code accuracy, ShareGPT traffic for latency) found that the quantization method and the inference kernel matter independently — a “good” algorithm with a slow kernel can be worse in practice than a simpler one with an optimized kernel:

The benchmark’s own framing is worth repeating directly: “kernels matter more than algorithms” — Marlin kernels delivered a 2.6x speedup for GPTQ and a 10.9x speedup for AWQ over their standard-kernel counterparts, at identical quantization quality (same perplexity and HumanEval score in each pair). GGUF, which is the format most local desktop tools default to, actually performed relatively poorly by this benchmark’s numbers when run inside vLLM specifically — GGUF’s usual home is llama.cpp-based tools, where its performance profile is different; this figure is about running a GGUF file in an environment it wasn’t primarily built for, not a verdict on GGUF as a format for its native runtime.

Local inference runtimes

The runtime matters almost as much as the model. Different tools optimize for different platforms and workflows.

llamafile packages an LLM runtime into a portable executable, reducing installation friction across operating systems. Rapid-MLX targets Apple Silicon and exposes local models behind familiar OpenAI- and Anthropic-compatible API patterns. For GPU-serving at scale, vLLM and Ollama are the two most widely deployed options, and they are optimized for different things.

Serving framework throughput: vLLM vs Ollama

Independent benchmarking found that vLLM and Ollama perform almost identically for a single user, but diverge sharply once multiple requests arrive at once:

The gap comes from vLLM’s continuous-batching scheduler and PagedAttention memory management, which are built specifically for serving many concurrent requests efficiently — the source describes it as letting “a single H100 serve roughly eight to nine times the aggregate tokens of an Ollama deployment” under load. That doesn’t make Ollama a bad choice: for a single developer running one chat session at a time, the two are within a few percent of each other, and Ollama’s simpler setup is a real advantage. The choice is about your concurrency profile, not raw model quality.

Apple Silicon

Apple Silicon is particularly interesting for local LLMs because unified memory can be shared between CPU and GPU workloads. MLX-based runtimes are designed specifically for this architecture and can make local inference practical on M-series Macs with sufficient memory.

The main limit is still memory capacity. A model plus its KV cache and runtime overhead must fit within available unified memory, so model size, quantization and context length all interact.

CPU vs GPU

CPU inference is widely supported and can be useful for smaller or heavily quantized models, but GPU or accelerator-backed inference is usually much faster for interactive workloads. The “best” deployment therefore depends on latency targets, concurrency and hardware budget — the vLLM/Ollama gap above is a useful illustration of how much the serving stack itself can matter, independent of the hardware underneath it.

OpenAI-compatible local APIs

One of the most useful patterns in local AI is exposing a local model through an OpenAI-compatible API. Applications can then switch from a hosted provider to a local endpoint with limited code changes.

This is particularly useful for coding agents, internal tools, RAG systems and privacy-sensitive workflows because the application layer can remain stable while the underlying model changes.

Licensing

Always inspect the exact model license. Vendors can publish multiple models under different terms, and “open model” can cover anything from Apache-licensed checkpoints to custom licenses with commercial restrictions — and the terms attached to any given release can change between generations, so don’t assume a family’s older license still applies to its newest flagship:

The pattern worth noting: several vendors now ship permissive licenses on their smaller, faster models while reserving custom, more restrictive terms for their largest flagship releases — a shift from the fully-Apache-2.0-across-the-board approach that characterized earlier open-weight waves. Check the license of the exact checkpoint you plan to deploy, not the license the family used a generation ago.

When local LLMs make sense

• private or sensitive data should stay on your infrastructure;
• you need predictable per-request cost;
• you want offline capability;
• you need to pin a model version;
• you want to fine-tune or deeply customize the runtime;
• latency to a local service is more important than access to the largest hosted model.

When a hosted API is better

Hosted APIs remain attractive when you need very large frontier models, elastic scale, minimal infrastructure work or access to proprietary multimodal capabilities. Many production systems use both: local models for routine/private workloads and hosted models for the hardest tasks.

About the figures in this article

The Signal did not run the quantization or serving-framework benchmarks cited above. Both come from third-party testing with stated methodology (model, hardware, dataset and evaluation task), which we’ve kept attached to each figure rather than presenting the numbers as universal. Quantization and serving-framework performance both change quickly as kernels, drivers and framework versions update — treat the numbers here as illustrating the shape of the trade-offs (kernel choice matters more than the underlying algorithm; concurrency, not single-user speed, is where serving frameworks diverge) rather than as a guarantee you’ll see the identical figures on your own stack.

Related guides

• Local AI: How to Run Generative AI on Your Own Hardware
• Open Source AI Models
• llamafile
• Rapid-MLX
• Open Source AI Image Generators

Sources

• Open Source Initiative — Open Source AI Definition
• llamafile official repository
• MLX official repository
• Jarvislabs — vLLM quantization benchmark guide
• Tech Insider — vLLM vs Ollama throughput benchmark
• ComputingForGeeks — open-source LLM license comparison