Run Your Own AI: A Local Model You Control

Every mainstream AI assistant runs on someone else's computer. You send your words to a company, it decides what you may ask, it logs the exchange, it bills you and it can cut you off or change the terms whenever it likes. When the connection drops, so does the tool. For a country that keeps discovering it has agreed to dependencies nobody put to it, an assistant that answers only to you is worth the afternoon it takes to set up. The weights sit on your disk, the work happens on your own GPU and nothing you type leaves the room. This is the practical stack: one engine, with a chat page, your code editor and your own scripts all feeding off a model that is entirely yours.

The one idea that makes a stack

The seam is the API. When llama-server loads a model it exposes a small web API in the exact shape the big providers use, the format everyone calls OpenAI-compatible. Anything built to talk to a hosted model can talk to yours by changing one setting: the base URL.

That is the whole trick. You run one engine on one port and every tool points at that port. The chat page is not special, it is just the first of many clients. Swap in a code editor, a shell script, your own app, they are all equal customers of the same engine.

Get the engine up (the short version)

Grab the Vulkan build of llama.cpp from its releases page, unpack it to ~/ai, then serve a model:

export LLAMA_CACHE=~/ai/models

~/ai/llama.cpp/llama-server \
  -hf unsloth/Qwen3.6-27B-GGUF:UD-Q4_K_XL \
  -ngl 999 -c 32768 \
  --host 127.0.0.1 --port 8089

Confirm it is live:

curl http://localhost:8089/v1/models

A short block of JSON naming the model means the engine is up on 127.0.0.1:8089. That address, http://localhost:8089/v1, is the one thing every client below needs.

Models by role, one at a time

A single card holds one model at a time, so you keep two or three on disk and load the one that fits the job. Swap by stopping the engine with pkill -x llama-server, then starting it again with a different -hf line. These fit a 24GB card at a 4-bit quant:

Smaller cards scale the idea down: a 14B generalist for 16GB, a 7B to 9B for 12GB. The point is the same. Right tool for the task, one loaded at once.

Client 1: the chat page

The browser interface (Open WebUI in a container) is client number one. It points at http://host.containers.internal:8089/v1 and gives you a commercial-feeling chat page over your local model. Nothing to add here, it is already wired in. Everything below joins it on the same engine.

Client 2: your code editor

Most editor AI extensions let you name a custom OpenAI-compatible endpoint instead of a cloud provider. Point one at your engine and you get chat, explain and refactor inside the editor, running on the coder model, entirely local. Load the coder model first.

The shape is the same whichever extension you use: a base URL, a model name, a placeholder key.

{
  "models": [
    {
      "title": "Local coder",
      "provider": "openai",
      "model": "qwen3-coder",
      "apiBase": "http://localhost:8089/v1",
      "apiKey": "sk-local-unused"
    }
  ]
}

The key is required by the format but never checked locally, so any string does. The model name is a label, the engine serves whatever is loaded.

Client 3: your own scripts

Because the endpoint is plain HTTP in a standard shape, anything can call it. A one-liner proves it:

curl http://localhost:8089/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"local","messages":[{"role":"user","content":"Summarise this in one line: ..."}]}'

From there it is a few lines in any language. The official client libraries take a base URL, so you point them at localhost and the rest of your code is unchanged:

from openai import OpenAI

ai = OpenAI(base_url="http://localhost:8089/v1", api_key="sk-local-unused")

r = ai.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Classify this note as urgent or not: ..."}],
)
print(r.choices[0].message.content)

Now you can batch-summarise a folder, tag notes, draft replies, sort a backlog, all in a loop, all offline, all free. This is where a local stack stops being a novelty and starts doing chores.

One quiet benefit falls out of the standard format. Code you write against your own engine runs unchanged against a hosted model later, only the base URL and key change. That cuts lock-in both ways. The same code runs locally or in the cloud, so you move a workflow in whichever direction you need with no rewrite.

One launcher, pick a model

Two commands every time gets old. Worse, launching the engine twice makes both fight over the port. A small launcher that takes the role as an argument fixes both:

#!/bin/sh
# start-ai.sh [general|coder]: bring the stack up with one model

MODEL="${1:-general}"
case "$MODEL" in
  general) HF="unsloth/Qwen3.6-27B-GGUF:UD-Q4_K_XL" ;;
  coder)   HF="unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL" ;;
  *) echo "usage: start-ai.sh [general|coder]"; exit 1 ;;
esac

podman start open-webui 2>/dev/null

if ss -ltn | grep -q ':8089'; then
  echo "Engine already on :8089. Stop it first to switch model: pkill -x llama-server"
else
  setsid ~/ai/llama.cpp/llama-server -hf "$HF" \
    -ngl 999 -c 32768 --host 127.0.0.1 --port 8089 \
    >/tmp/llama-server.log 2>&1 &
  echo "Engine starting on :8089 ($MODEL)"
fi

echo "Chat: http://localhost:3000"

chmod +x start-ai.sh, then ./start-ai.sh coder when you want the code model or ./start-ai.sh for the generalist. Stop with pkill -x llama-server (exact name, not -f) when you are done.

Reaching it from your other machines

By default the engine listens on 127.0.0.1 only, which is correct. If you want the model from a laptop elsewhere in the house, two clean ways:

• Keep it loopback, tunnel in over SSH. From the laptop: ssh -N -L 8089:localhost:8089 you@desktop. Now the laptop's own localhost:8089 is the desktop's engine. Nothing new is exposed to the network.
• Bind it to the LAN. Start the engine with --host 0.0.0.0 and firewall the port to your own subnet.

One hard line, worth its own paragraph. Do not put an inference endpoint on the public internet the way you would a website. An open llama-server is unauthenticated compute: anyone who finds it gets free tokens on your electricity and your GPU. Loopback plus SSH is the safe default. If you genuinely must expose a model, put real authentication in front of it (a reverse proxy that checks a key) and rate-limit it. This is the opposite posture to a self-hosted website or forge, which are built to face the world behind a login. A raw model is not.

Making it yours

Out of the box the model does not know you. Two ways to fix that. One pastes a system prompt and default sampling values into the chat interface's settings. The other bakes your operating notes into a named model so they are always on. Write the notes once, they apply to every chat and every client above.

Field notes

Quick start:

1. Engine up on :8089.
2. Point each client at http://localhost:8089/v1: the chat page, your editor, your scripts.
3. Keep two or three models on disk, load one per task, swap with pkill -x llama-server then restart.
4. start-ai.sh [general|coder] brings it up in one command.
5. Off-machine access is an SSH tunnel, never a naked public port.

Footguns:

• One card, one model. Switching models means stopping the engine first.
• The API key is required by the format but unused locally. Any placeholder works. Do not paste a real one.
• The standard format cuts both ways. A client config that points at your engine also works against a hosted model by changing the base URL, so never hardcode a real secret into a config you might share.
• An open inference endpoint is unauthenticated compute. Loopback plus SSH, not a public port.
• Point editors and scripts at the coder model for code, the generalist for everything else. A coder writing prose is worse, a generalist writing code is slower.
• The model owns your VRAM while it runs. Stop it before a game or a sim.

Sources

• llama.cpp (the engine)
• Open WebUI (the chat interface)
• unsloth GGUF model releases on Hugging Face
• Qwen3-Coder-30B GGUF