Running Local LLMs (Ollama, vLLM, llama.cpp)

The Case for Local LLMs

Running LLMs locally means the model runs on your own hardware - your laptop, server, or GPU cluster. No data leaves your machine, no per-token charges, and you have complete control over which model, version, and configuration you use.

Think of it like cooking at home vs ordering from Swiggy. Swiggy (cloud APIs) is convenient but expensive and you have no control over ingredients. Cooking at home (local LLMs) takes more setup but gives you full control over quality, cost, and privacy.

When to Go Local:

• Data Privacy - Sensitive data (medical records, financial data, legal docs) that cannot leave your infrastructure
• Cost Optimization - High-volume use cases where API costs become prohibitive (100K+ requests/day)
• Latency Requirements - Edge deployments where network latency to cloud APIs is unacceptable
• Offline Access - Systems that must work without internet (defense, remote locations, embedded devices)
• Customization - Need to fine-tune or modify the model for specific tasks

Trade-offs to Consider:

• Hardware Cost - You need GPUs (NVIDIA RTX 4090 ~ Rs 1.5L) or powerful CPUs with lots of RAM
• Model Quality - Local models (7B-70B params) are improving but still behind GPT-4/Claude for complex reasoning
• Maintenance - You manage updates, scaling, monitoring, and hardware failures yourself
• Setup Complexity - CUDA drivers, model downloads, quantization - initial setup can be tricky

What is Ollama?

Ollama is the easiest way to run LLMs locally. It packages models with their runtime into a single binary, similar to how Docker packages applications. One command and you have a model running with an OpenAI-compatible API.

It handles model downloading, quantization selection, GPU detection, memory management, and serving - all automatically. Think of it as the "Homebrew for LLMs."

How Ollama Works:

• Model Library - Curated collection of 100+ models (Llama 3, Mistral, Gemma, Phi, CodeLlama, DeepSeek)
• Modelfile - Like a Dockerfile but for LLMs. Defines base model, system prompt, parameters, and template format
• Automatic GPU Detection - Finds your NVIDIA/AMD/Apple Silicon GPU and uses it automatically
• OpenAI-Compatible API - Serves on localhost:11434 with the same API format as OpenAI, so existing code works
• Model Layers - Only downloads the diff when switching between related models, saving bandwidth

Popular Models on Ollama:

Ollama Limitations:

• Single Request - Default: processes one request at a time (no batching)
• No Distributed Inference - Cannot split a model across multiple GPUs on different machines
• Production Readiness - Great for dev/prototyping, but lacks production features like auth, rate limiting, load balancing

What is vLLM?

vLLM (Virtual LLM) is a high-throughput, memory-efficient LLM serving engine developed at UC Berkeley. It is designed for production deployments where you need to serve thousands of concurrent users with minimum latency.

If Ollama is like a home kitchen, vLLM is like a professional restaurant kitchen - designed for high volume, efficiency, and consistency.

PagedAttention - The Secret Sauce:

vLLM introduced PagedAttention, a revolutionary memory management technique inspired by OS virtual memory:

• Problem - Traditional inference wastes GPU memory by pre-allocating fixed-size KV cache for each request
• Solution - PagedAttention allocates memory in small blocks (pages), only as needed, like how OS manages RAM
• Result - 2-4x higher throughput compared to naive serving. Can serve 2-4x more concurrent requests on same hardware

vLLM Key Features:

• Continuous Batching - New requests join the batch as old ones finish, maximizing GPU utilization
• Tensor Parallelism - Split large models across multiple GPUs seamlessly
• Streaming - Token-by-token streaming for responsive UIs
• OpenAI-Compatible API - Drop-in replacement for OpenAI API endpoints
• Speculative Decoding - Use a small draft model to speed up generation from large models

When to Choose vLLM:

• Serving 50+ concurrent users
• Need maximum throughput per GPU dollar
• Running on dedicated GPU servers (cloud or on-prem)
• Production deployment with SLA requirements

What is llama.cpp?

llama.cpp is a pure C/C++ implementation of LLM inference created by Georgi Gerganov. It can run models on CPU-only machines, Apple Silicon, and even phones. No Python, no PyTorch, no CUDA required (though GPU acceleration is supported).

This is the engine that powers Ollama under the hood. llama.cpp gives you direct low-level control, while Ollama wraps it with a user-friendly interface.

GGUF Format - The Universal Model Format:

• GGUF (GPT-Generated Unified Format) - Created by the llama.cpp project, now the standard for local model distribution
• Self-Contained - One file has the model weights, tokenizer, and metadata
• Quantization Built-In - GGUF files come pre-quantized (Q4_K_M, Q5_K_S, Q8_0 etc.)
• HuggingFace - TheBloke and other quantizers upload thousands of GGUF models to HuggingFace

Quantization Levels Explained:

llama.cpp Use Cases:

• Edge Devices - Run on Raspberry Pi, phones, IoT devices
• CPU-Only Servers - When GPU budget is not available
• Maximum Efficiency - Best tokens-per-watt ratio
• Embedding Under the Hood - Powers Ollama, LM Studio, GPT4All, and many other tools

Ollama vs vLLM vs llama.cpp:

Hardware Requirements Guide:

• Laptop (16GB RAM, no GPU) - Use Ollama with 7B Q4 models. Good for experimentation and simple tasks
• Gaming PC (RTX 4070, 16GB VRAM) - Run 13B models at full speed. Great for coding assistants and local RAG
• Workstation (RTX 4090, 24GB VRAM) - Run 33B models or multiple 7B models. Production-quality for most tasks
• Server (2x A100, 80GB each) - Run 70B models with vLLM. Serve hundreds of concurrent users
• Apple Silicon (M2/M3 Pro/Max) - Unified memory is a game-changer. M3 Max with 96GB can run 70B models on a laptop!

Cost Comparison - Local vs Cloud (Monthly, 100K requests):

Pitfall 1: Wrong Model Size for Your Hardware

The most common mistake. A 70B model on 16GB RAM will either crash or run at 1 token/second. Rule of thumb: model size in GB should be less than your available VRAM/RAM. For CPU inference, you need roughly 1.5x the model file size in RAM.

Pitfall 2: Over-Quantizing

Going below Q4 quantization (like Q2 or Q3) saves memory but destroys model quality. The model will give wrong answers confidently. Q4_K_M is the sweet spot for most use cases - good balance of size and quality.

Best Practices:

• Always Benchmark - Test your model on your specific task before deploying. Generic benchmarks do not tell the full story
• Monitor GPU Memory - Use nvidia-smi or Activity Monitor to track VRAM usage
• Set Context Length Wisely - Longer context = more memory. Do not set 128K context if you only need 4K
• Use Streaming - Always enable streaming for better user experience. First token appears faster
• Keep Models Updated - New quantizations and model versions drop regularly. Stay current

Security Considerations:

• Never expose Ollama to internet - By default it binds to localhost. Keep it that way or add authentication
• Model Provenance - Only download models from trusted sources (Ollama library, HuggingFace official repos)
• Prompt Injection - Local models are equally vulnerable to prompt injection attacks. Sanitize inputs

Q: When would you choose local LLM deployment over cloud APIs?

Q: What is PagedAttention and why does it matter for LLM serving?

Q: What is model quantization and what are the trade-offs?

Q: Compare Ollama, vLLM, and llama.cpp for different use cases.

Q: How do you estimate hardware requirements for a local LLM deployment?