How to Run LLMs Locally for Free with Ollama

Running AI models locally means zero API costs, complete privacy, and offline access. With Ollama, it takes under 5 minutes to have a powerful model running on your laptop.

---

Why Run LLMs Locally?

• Free forever — no monthly subscription or per-token costs
• Privacy — your code and data never leave your machine
• Speed — no API latency for long context tasks
• Offline — works without internet

Best for: Code completion in private codebases, processing sensitive documents, unlimited experimentation.

---

Hardware Requirements

Model Size	Minimum RAM	Recommended
7B params	8GB RAM	16GB RAM
13B params	16GB RAM	32GB RAM
34B params	32GB RAM	64GB RAM
70B+ params	64GB+ RAM	M3 Ultra or 2×H100

GPU: Ollama automatically uses Apple Silicon (M-series), NVIDIA CUDA, or AMD ROCm if available. Without a GPU, models run on CPU (slower but functional).

---

Step 1: Install Ollama

macOS

# Download from the official website
curl -fsSL https://ollama.com/install.sh | sh
# Or install via Homebrew
brew install ollama

Windows

Download the installer from ollama.com and run it. Ollama installs as a system service.

Linux

curl -fsSL https://ollama.com/install.sh | sh

Verify installation

ollama --version
# → ollama version 0.x.x

---

Step 2: Pull and Run a Model

# Pull Llama 3 (8B) — the best general-purpose local model
ollama pull llama3

# Pull DeepSeek Coder V2 — best for code
ollama pull deepseek-coder-v2

# Pull Mistral 7B — fast and efficient
ollama pull mistral

# Pull Microsoft Phi-4 — best quality in small size
ollama pull phi4

# Pull Qwen 2.5 Coder — excellent for code completion
ollama pull qwen2.5-coder

Run a model interactively:

ollama run llama3
>>> Tell me a short explanation of transformers in AI

Press Ctrl+D to exit.

---

Step 3: Use the REST API

Ollama exposes a local API on port 11434 — compatible with the OpenAI API format:

# Chat completion
curl http://localhost:11434/api/chat -d '{
  "model": "llama3",
  "messages": [
    {"role": "user", "content": "Write a Python function to reverse a linked list"}
  ]
}'

# Generate (simpler)
curl http://localhost:11434/api/generate -d '{
  "model": "deepseek-coder-v2",
  "prompt": "# Fibonacci sequence in Python\ndef"
}'

---

Step 4: Integrate with Your IDE

VS Code + Continue Extension

Continue is a free, open-source Copilot alternative that connects to Ollama:

1. Install the Continue extension in VS Code
2. Open ~/.continue/config.json
3. Add your Ollama models:

{
  "models": [
    {
      "title": "DeepSeek Coder V2 (Local)",
      "provider": "ollama",
      "model": "deepseek-coder-v2",
      "contextLength": 16384
    },
    {
      "title": "Llama 3 (Local)",
      "provider": "ollama",
      "model": "llama3",
      "contextLength": 8192
    }
  ]
}

Now press Ctrl+I in VS Code to use your local model for code editing.

Cursor + Ollama

Cursor supports local models via the OpenAI-compatible API:

1. In Cursor Settings → Models → Add Custom Model
2. Set Base URL: http://localhost:11434/v1
3. Set Model: deepseek-coder-v2

---

Step 5: Useful Commands

# List downloaded models
ollama list

# Show model info
ollama show llama3

# Remove a model (free up disk space)
ollama rm mistral

# Run model with system prompt
ollama run llama3 --system "You are an expert Python developer. Be concise."

# Check running models
ollama ps

---

Recommended Models by Use Case

Use Case	Best Model	Size
Code completion	qwen2.5-coder:7b	4.7GB
General coding	deepseek-coder-v2	8.9GB
Chat & reasoning	llama3:8b	4.7GB
Complex tasks	llama3:70b	40GB
Fast & lightweight	phi4	9.1GB
Privacy-critical	mistral	4.1GB

---

Performance Tips

1. Use GPU acceleration — Ollama auto-detects GPU. Verify with ollama ps (shows GPU utilization)
2. Increase context — Set OLLAMA_MAX_LOADED_MODELS=2 to keep multiple models in memory
3. Use quantized models — llama3:8b-q4_0 is 50% smaller with ~5% quality loss
4. Allocate more VRAM — macOS: increase shared memory in GPU settings

---

Next Steps

• Try LM Studio for a GUI interface to Ollama
• Explore Open WebUI for a ChatGPT-like interface to your local models
• Check our AI Models Directory for more LLMs you can run locally