Get Started with Local LLMs Using Ollama on Ubuntu

Running an LLM on your own machine keeps your data local and costs nothing per query. Ollama is the easiest way to do it — install once, pull a model, and chat. Here's the practical path.

1Install Ollama

The official installer:

curl -fsSL https://ollama.com/install.sh | sh

Confirm it's running:

ollama --version
ollama serve      # starts the server (or it may already be running as a service)

2Pull a model

Ollama downloads models as "tags." Pick a size that fits your RAM — a small model runs comfortably on 8 GB; bigger ones need more.

ollama pull llama3.2   # a ~3B lightweight model, runs on modest hardware

Pull a few and see what's installed:

ollama list

3Chat from the terminal

ollama run llama3.2    # opens a chat REPL

Type a question, get an answer, /bye to exit. This is the zero-API-cost path: no key, no account, nothing leaves your machine.

4Use it as an API (OpenAI-compatible)

Ollama exposes a local HTTP server on port 11434, with an OpenAI-compatible endpoint at /v1. So anything that speaks OpenAI's API can point at it — just change the base URL. For example, an OpenAI-style client configured to http://localhost:11434/v1 with model llama3.2 will work with no external account.

You can test the endpoint directly with curl:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3.2","messages":[{"role":"user","content":"Say hello"}]}'

This is why local models plug into existing tooling with minimal fuss.

5The trade-offs to know

Hardware. Local models are limited by your RAM and (ideally) GPU. A 70B-parameter model needs 48 GB+ of RAM; a 3B or 8B runs on a laptop. Pick the model that fits your machine.
Quality. Small open models are capable but not frontier. They're great for summarising, classification, drafting, and privacy-sensitive work — not the best choice for the hardest reasoning tasks.
Speed. Without a GPU, generation is slower than a cloud API. Fine for interactive use, less good for high-volume batch work.
Quantisation. Models are often served in quantised forms (e.g. Q4) that trade a little quality for a fraction of the memory. Ollama handles this automatically per tag.

6When local is the right call

Choose local when you care about privacy (your data never leaves the machine), cost (no per-token billing), or offline use. Choose a hosted API when you need the biggest model, the fastest throughput, or you're scaling production traffic.

The fun part is that "local or hosted" isn't a one-time decision — Ollama and cloud providers are both OpenAI-compatible, so you can switch where a workload points in one config line.