Hermes Agent + llama.cpp: Local Inference Without a GPU

Why llama.cpp?

llama.cpp is the most portable way to run an open-weight model. It targets CPU first, uses Metal on Apple Silicon, and optionally CUDA, ROCm or Vulkan — so it runs on hardware that vLLM will not touch. Quantized GGUF weights mean a 7–8B model fits comfortably in 8 GB of RAM.

Compared with the alternatives in the Hermes ecosystem: Ollama is friendlier but wraps llama.cpp and hides its flags; vLLM is far faster under concurrency but is built around GPU serving (it has a CPU path, though that is not where its throughput advantage lives). llama.cpp is the right pick when you want direct control, minimal dependencies, or inference on a laptop.

Step 1: Build or Install llama.cpp

# macOS (Metal acceleration included)
brew install llama.cpp

# Or build from source — the binary lands in ./build/bin,
# so either add it to PATH or call ./build/bin/llama-server below
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && cmake -B build && cmake --build build --config Release

# Or run the prebuilt server container
docker run -p 8080:8080 ghcr.io/ggml-org/llama.cpp:server \
  -hf bartowski/Qwen2.5-7B-Instruct-GGUF

Step 2: Start the Server

llama-server exposes an OpenAI-compatible API on port 8080 by default. It can pull a GGUF straight from Hugging Face with -hf, so you do not have to download weights by hand.

# Pull and serve in one step
llama-server -hf bartowski/Qwen2.5-7B-Instruct-GGUF -c 8192

# Or serve a local GGUF file
llama-server -m ./models/qwen2.5-7b-instruct-q4_k_m.gguf \
  -c 8192 \
  --port 8080

# Offload layers to GPU if you have one (-ngl 999 = all)
llama-server -m ./model.gguf -ngl 999 -c 16384

-c sets the context window, and this is the setting most likely to bite you. Agent workloads carry long system prompts and tool definitions before the conversation even starts, so the default is far too small: treat 16384 as a working floor and go higher if RAM allows. If you cannot afford the context, trim the toolset rather than hoping a small window is enough.

Check it answers before wiring Hermes to it:

curl -s http://localhost:8080/v1/models

Step 3: Point Hermes at llama.cpp

Hermes treats llama.cpp as a custom OpenAI-compatible endpoint. You do not have to remember that, though — llamacpp, llama.cpp and llama-cpp are all declared aliases of that provider, so any of the three resolves correctly.

# Hermes accepts OPENAI_API_BASE and OPENAI_API_KEY
export OPENAI_API_BASE=http://localhost:8080/v1
export OPENAI_API_KEY=local   # llama-server ignores the value by default

# Pick the provider and default model (interactive wizard)
hermes model

# config.yaml equivalent:
# model:
#   provider: llamacpp        # alias of the built-in custom provider
#   default: qwen2.5-7b-instruct
#   base_url: http://localhost:8080/v1

A model change applies to new sessions without a restart. If you changed anything else in the config, restart the gateway with hermes gateway restart.

Hermes Knows llama.cpp Is Not Ollama

This detail saves a confusing failure. Ollama accepts a non-standard think field, and Hermes sends it — but only to endpoints it positively identifies as Ollama, meaning port 11434 or a hostname with ollama in it. A llama.cpp server on localhost:8080 is deliberately not matched, so it never receives a field it would reject.

The practical consequence: do not run llama.cpp on port 11434 to “look like Ollama”. That is the one configuration where Hermes would guess wrong about your server.

Context and Token Limits

• Set -c generously. A prompt larger than the context you launched with is rejected with a context-size error rather than quietly trimmed — that is a failed turn, not a degraded one.
• model.max_tokens is yours to override. The custom provider supplies a deliberately generous default so local servers do not truncate replies; set your own value per model if you want a tighter cap.
• ollama_num_ctx does nothing here. That knob maps to an Ollama-specific option; on llama.cpp the context is fixed at launch by -c.

Picking a Model

Tool calling is the thing to check before committing to a model. Agent work depends on it, and small quantized models vary a lot in how reliably they emit well-formed tool calls. Test your actual workflow rather than trusting the model card.

On OpenClaw

OpenClaw connects to llama.cpp through its own custom-endpoint configuration rather than a provider alias — see OpenClaw + llama.cpp. The llama.cpp server is identical either way; only the client config differs, and one server can back both frameworks at once.

What's Next?

• Hermes Agent + Ollama — the friendlier local wrapper
• Hermes Agent + vLLM — when you have GPUs and concurrency
• Hermes Agent + LM Studio — a desktop GUI alternative
• Hermes Custom Endpoints — any OpenAI-compatible API