Running LLMs Locally With Ollama: A Practical Field Guide

A few years ago, "run a large language model on your laptop" meant either renting an expensive GPU or accepting a toy model that could barely hold a conversation. That changed. Open-weight models have gotten small enough and quantization has gotten good enough that a capable local LLM is now a realistic tool for a working engineer, not a novelty.

The tool most people reach for is Ollama, which has become the default CLI and server for running local LLMs. This guide walks through what it actually takes: the file formats, the memory math, the commands, and the honest caveats.

What you're actually running

When you pull a model with Ollama, you are downloading an open-weight model, usually in the GGUF format. GGUF (short for GPT-Generated Unified Format) is the standard format used by llama.cpp and Ollama, and it is optimized for efficient inference. Crucially, a GGUF file can bundle a model that has already been quantized: its weights stored at lower precision to shrink the file and the memory it needs to run.

This matters because memory, not raw compute, is usually the binding constraint. A model has to fit in memory to run at acceptable speeds, so smaller quantized versions are how people squeeze reasonably capable models onto consumer hardware. The rough guidance you will see repeated: a 7B model at 4-bit quantization needs on the order of 4 to 6 GB of VRAM, while a 32B model under the same quantization asks for something more like 20 to 24 GB. These are approximations that shift with context length and quantization method, so treat them as a starting point, not a spec sheet.

The commands

Getting a model running is deliberately unceremonious. Install Ollama, then pull and run a model:

ollama pull llama3.1:8b
ollama run llama3.1:8b

ollama pull downloads the model (with the version tag it needs), and ollama run drops you into an interactive prompt. To see what's already on your machine, ollama list.

The Llama 3.1 line, released by Meta on July 23, 2024, is a good reference point for what "open-weight" gets you: the 8B, 70B, and 405B variants all support a 128K-token context, and the weights are downloadable for you to run locally, fine-tune, or put behind your own service. The 8B instruct model in particular runs comfortably on a mid-range consumer GPU.

It's a server, not just a CLI

Under the hood, every Ollama install runs a local HTTP server on port 11434, and ollama run is a thin client on top of it. That means you can call the same models from your own code over plain HTTP:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Why is the sky blue?"
}'

Ollama also exposes an OpenAI-compatible API at /v1 (with /v1/chat/completions and /v1/embeddings), so you can point an existing OpenAI SDK client at it by changing the base URL to http://localhost:11434/v1 and passing any non-empty API key. That single compatibility layer is a large part of why local models became drop-in replacements in so many projects.

If you want a chat-room experience rather than a terminal, Open WebUI is a popular self-hosted browser front end that talks to Ollama and adds user accounts, persistent chat history, and document upload. It runs in Docker and speaks to the Ollama API on port 11434.

The security caveat that bites people

Open-weight models running locally keep your data on your machine, which is the whole appeal. But the convenience comes with a trap: the Ollama API has no built-in authentication. Anyone who can reach port 11434 can pull models, use them, or delete them. By default Ollama binds to 127.0.0.1, which is fine for a single user. The moment you set OLLAMA_HOST=0.0.0.0:11434 to expose it to other machines, you are handing an open door to your network. If you need remote access, put a reverse proxy with auth and TLS in front of it, and keep the raw port internal.

Where local models fit

Local models are not a free frontier model. They trail the biggest closed models on capability, and a 128K context on paper is not the same as a 128K context that stays coherent in practice. Where they shine is privacy-sensitive work, offline systems, cost control, and the simple joy of running something you own. For many tasks, an 8B model on your own hardware is genuinely useful. For tasks that need the ceiling, you still reach for the cloud.

The honest takeaway: local LLMs are now a practical engineering tool, and Ollama is the lowest-friction way to start using one. Pull a small model, wire it into a script, and learn what it is actually good at before you decide whether it replaces anything.

Key takeaways

• Ollama runs open-weight models locally as a CLI and an HTTP server on port 11434, with an OpenAI-compatible /v1 API.
• GGUF files bundle quantized weights, which is how capable models fit on consumer hardware; a 7B at 4-bit roughly wants 4 to 6 GB of VRAM.
• Open-weight lines like Llama 3.1 (8B/70B/405B, 128K context, released July 2024) are runnable on machines you own.
• The API has no built-in authentication, so keep port 11434 on loopback or put authenticated TLS in front of it.