Models and engines

An engine is a place where a model runs. Settings → Engines lists your engines in two groups, one row each, with its status, a Primary button and an on/off switch. Open a row to configure it and to select which of its models to offer.

The status bar sums the engines up ("2/3 engines online"). Click it for the details of each engine.

Primary engine#

One llama.cpp engine is primary. New chats start on it, the download list belongs to it and the memory embeddings run on it. Models of the other engines show the engine name in the picker.

Built-in llama.cpp#

The built-in runner loads a model when a chat asks for it and keeps one model in memory. The first request after a model change takes a moment.

Its editor shows:

• Models folder and the models in it. The default folder is inside the app data directory. Use Browse to select another one.
• Context size, the number of tokens the model can see at once. A larger value needs more memory.
• Extra arguments for the runner, for advanced use.
• GPU acceleration (below), the Download list and the Catalogue.
• The last lines of the runner log, with Restart and Stop buttons.

The runner listens on your computer only. To offer it to other programs on your network, see Remote access.

GPU acceleration#

The app ships a CPU build on Linux and Windows (on macOS the GPU is used through Metal). On a computer with an NVIDIA graphics card:

1. Open the built-in engine. The GPU acceleration section shows the card found.
2. Press Install CUDA build. Assistant downloads the matching build (about 170 MB on Linux, 250 MB on Windows) and, when your computer needs them, the CUDA libraries (400 to 600 MB more).
3. Assistant checks that the build sees the card, then restarts the runner on it.

The section then lets you switch between the CUDA and the CPU build and remove the download. An AMD card (ROCm) needs an external server.

Download models#

The Download section of the primary engine lists the models of the catalogue that you do not have. Press the download button of a model. The row shows the progress; you can cancel. Assistant restarts the runner when the file is complete so the model appears.

The catalogue is a list that Assistant reads from the website every hour. Each entry names the model, its vendor, its quantisation and whether it can think or see images. Catalogue opens the list as text; a saved copy replaces the website list until you restore it.

You can also copy a GGUF file into the models folder yourself. The runner lists it after a restart.

Adding an engine#

Every engine you add is an OpenAI-compatible API: press Add engine, enter the address and, when the server asks for one, your API key, and press Test. The models of the engine appear in the pickers with the name of the engine. The key stays in the local database and is only sent to that engine.

llama.cpp server on another computer#

Run llama.cpp on a computer with a GPU and add it with its address (for example http://gpu-box.local:9100). Assistant recognises a llama.cpp server when you press Test and shows it as such. Make it Primary when the built-in runner is not needed.

Cloud providers#

Enter the provider's address (for example https://api.example.com/v1) and your API key.

A cloud engine has its own queue. A request to it never waits behind the built-in runner.

Anthropic (Claude subscription)#

Press Add Anthropic. This engine uses your Claude plan through the Claude Code command-line tool, so nothing is billed to an API key. The engine editor checks two things on the computer that runs the assistant and tells you what to do:

1. Install the tool. Copy the command shown for your system into a terminal and run it. Restart the assistant if the tool is still not found.
2. Log in. Run claude in a terminal and follow the login; it opens claude.ai in your browser. Choose your Claude account, not an API console account. Type /exit once you are logged in.

Press Check again after each step. When both are done the editor shows the account you are logged in with, and the models Claude Opus, Claude Sonnet and Claude Haiku appear in the pickers. Each is the newest model of that tier your plan includes. Images work. The assistant's tools (web search, knowledge, weather) are not available on these models. Your plan's usage limits apply.

TypeSafe AI (Jev)#

Under Decision engines, press Add decision engine and enter a key from console.typesafe.ai. Jev answers a question with probabilities in a fraction of a second. It never writes chat replies. Select it as the coordinator of a recipe or as the coordinator model under Settings → Engines.

Model switches#

Every engine offers all the models it lists. Switch a model off in the engine editor to hide it from every picker. Switch it on again at any time.