What Hardware Do You Need to Run Large Language Models (LLMs)? A Practical Guide
Running large language models like Llama 3, Mistral, DeepSeek, Gemma, or Falcon on your own hardware is increasingly accessible — but knowing what specs you need is essential. This guide covers hardware requirements for running open-source LLMs locally, from lightweight 7B models to massive 70B+ parameter models.
1. Model Size and Memory Requirements
Model size is the single biggest factor determining hardware needs. The table below shows VRAM (GPU memory) and RAM (system memory) estimates for common parameter sizes when running at full precision. Models from different families (Llama, Mistral, DeepSeek, Qwen, etc.) follow similar memory scaling.
| Model Size |
Typical VRAM (GPU) |
Typical RAM (CPU-only) |
Example Models |
| 1B–3B parameters |
4–8 GB VRAM |
8–16 GB RAM |
Gemma 2B, Phi-3 Mini, Qwen2.5-1.5B |
| 7B–8B parameters |
8–16 GB VRAM |
16–32 GB RAM |
Llama 3 8B, Mistral 7B, DeepSeek 7B |
| 13B–14B parameters |
16–28 GB VRAM |
32–64 GB RAM |
Llama 2 13B, Code Llama 13B |
| 30B–34B parameters |
24–48 GB VRAM |
64–96 GB RAM |
Yi 34B, Code Llama 34B |
| 65B–70B parameters |
40–80 GB VRAM |
128+ GB RAM |
Llama 3 70B, DeepSeek 67B |
| 100B+ parameters |
80+ GB VRAM (multi-GPU) |
192+ GB RAM |
Falcon 180B, Llama 3 405B* |
* Models above ~100B typically require multiple GPUs or high-end server hardware. Running on CPU alone is not recommended.
GPU VRAM Rules of Thumb:
- NVIDIA RTX 3060 (12 GB) — comfortable for 7B models at full precision
- NVIDIA RTX 3090/4090 (24 GB) — handles 7B–13B models with ease, up to 30B with quantization
- NVIDIA A100 (40–80 GB) or H100 (80 GB) — handles 30B–70B+ models
- Apple Silicon (M1/M2/M3 Max/Ultra with 64+ GB unified memory) — can run 30B–70B models using Metal acceleration
CPU-only Considerations:
Running LLMs on CPU-only is possible but significantly slower (expect 1–5 tokens/second on large models). Modern CPUs with high core counts and fast DDR5 RAM help, but GPU inference is always preferred for responsiveness.
2. GPU Recommendations by Tier
NVIDIA remains the standard for LLM inference thanks to CUDA and widespread framework support. AMD GPUs work with ROCm, and Apple Silicon is supported via Metal/MPS, but compatibility can vary between tools.
- Entry-level: RTX 3060 12 GB, RTX 4060 Ti 16 GB — fine for 7B models and quantized 13B
- Mid-range: RTX 3090 24 GB, RTX 4090 24 GB — great for 7B–13B at full precision, up to 30B quantized
- Enthusiast: Used RTX A6000 (48 GB), dual RTX 3090s — opens up 30B–70B models
- Professional/server: A100 80 GB, H100 80 GB — for 70B+ models and production inference
3. CPU and RAM Recommendations
- Modern multi-core CPU (AMD Ryzen 7/9 5000 series or newer, Intel Core i7/i9 12th-gen or newer).
- At least 32 GB RAM for comfortable 7B–13B model usage; 64+ GB recommended if partially offloading to system RAM.
- DDR5 RAM offers better bandwidth, which directly improves CPU inference speed.
- Expect 5–10× slower performance on CPU vs. GPU for comparably sized models.
4. Storage Requirements
Model files range widely in size depending on parameters and precision:
- 7B model (full FP16): ~15 GB
- 7B model (4-bit GGUF/Q4_K_M): ~4–5 GB
- 70B model (full FP16): ~140 GB
- 70B model (4-bit GGUF): ~40 GB
Recommendation: Use an NVMe SSD (1–2 TB minimum) for fast model loading. If you plan to run multiple models, allocate 500 GB–1 TB for model storage.
5. Quantization: Make Big Models Fit Small Hardware
Quantization is the single most effective way to lower hardware requirements. Modern formats like GGUF (used in llama.cpp) and GPTQ/AWQ (used in ExLlama) can reduce memory usage by 50–75% with minimal quality loss.
- 8-bit (INT8): ~50% memory reduction. A 7B model drops from ~14 GB to ~7 GB VRAM.
- 4-bit (INT4): ~75% reduction. A 7B model fits in ~4–6 GB VRAM — perfect for laptops and mid-range GPUs.
- Popular tools: llama.cpp, Ollama, LM Studio, Text Generation WebUI (oobabooga), ExLlamaV2
- Format note: GGUF/GGML for CPU and Apple Silicon; GPTQ/AWQ for NVIDIA GPU-only; MLX for Apple Silicon.
6. Example Hardware Setups and What They Can Run
- Budget/Laptop Setup:
- GPU: RTX 3060 12 GB (or Apple M2 Pro with 16 GB unified memory)
- RAM: 32 GB
- Storage: 1 TB NVMe SSD
- Can run: 7B models at full precision, 13B quantized (Q4_K_M)
- Mid-Range/Enthusiast Setup:
- GPU: RTX 4090 24 GB (or dual 3090s)
- RAM: 64 GB DDR5
- Storage: 2 TB NVMe SSD
- Can run: 7B–13B at full precision, up to 34B quantized, or 70B quantized (GGUF offloaded)
- High-End/Pro Setup:
- GPU: A100 80 GB or H100 (or Apple M2 Ultra 76-core GPU, 192 GB unified memory)
- RAM: 128+ GB
- Storage: 4 TB NVMe SSD
- Can run: 70B+ models at full precision or with light quantization
7. Cloud Alternatives: Skip the Hardware
If purchasing expensive hardware isn't an option, cloud GPU services provide flexible access:
- RunPod, Vast.ai, Lambda Labs — affordable GPU rentals (from ~$0.30/hr for RTX 3090 to ~$2/hr for A100)
- AWS (p4d/p5), Google Cloud (A100/H100), Azure — enterprise cloud with on-demand instances
- Together.ai, Fireworks, Groq — serverless inference APIs for open models (no infrastructure management)
- Hugging Face Inference Endpoints — one-click deployment of supported models
8. Software Stack: Tools to Get Started
- One-click local inference: Ollama, LM Studio, GPT4All (beginner-friendly, supports multiple models)
- Advanced local inference: Text Generation WebUI (oobabooga), llama.cpp, ExLlamaV2
- Python libraries: Hugging Face transformers, accelerate, bitsandbytes, peft
- Operating system: Linux (Ubuntu 22.04+ recommended) offers best compatibility; Windows and macOS also well-supported via WSL2 or native tools
- Model hubs: Hugging Face, Ollama model library, LM Studio's built-in downloader
Note: Requirements vary by framework, quantization method, and context length. Always check the model card on Hugging Face or the tool documentation for specific recommendations. These estimates assume a standard context window of 4K–8K tokens; larger contexts require additional memory.
