Building a Local Agent Stack with Llama.cpp and Pi.dev

2026-08-08 (updated 2026-08-12)
Posts #agentic #llama.cpp #llm #pi.dev #self-hosted

I am sharing some workflow tips and resources I gathered while working with local agents.

I spent a long time ignoring the agentic craze. Most of the early frameworks felt like over-engineered wrappers that either relied on intrusive cloud APIs or were too fragile for real development work. I only started paying attention when local models seemed to reach a tipping point in reliability, specifically around the release of Qwen 3.5 35B.

Even then, my approach was conservative. I had experimented with local AI and agents early on, but I never integrated them into my actual development workflow. My usage was limited to occasional one-shot requests via a locally hosted llama.cpp instance. For longer sessions, I used aichat, a Rust-based CLI app that stayed out of the way.

In mid-2025, I decided to give local agentic workflows another try with OpenCode and QwenChat. I maintained a strict constraint: everything must be locally hosted. The experience was frustrating: more than half the tool calls failed and I spent more time debugging the environment and fiddling with the setup than actually writing code. It felt like the tools were fighting me, so I went back to Neovim and manually selected completion tasks.

I discovered pi.dev by chance through a Reddit comment and was immediately delighted by its minimalist, build-your-own-harness philosophy. It provides exactly what I was looking for: a thin layer that handles the basics and lets me build the rest.

This post showcases a set of tools and skills I have built to solve specific tasks in my workflow. I will continue to update this post as I add more tools to the stash.

The Stack

My setup is centered around a local workstation with multiple GPUs, with all components containerized via Docker.

Hardware

The machine is powered by an AMD Ryzen Threadripper 2950X (16 cores / 32 threads) on an X399 chipset with 128GB of DDR4. For inference, I use two NVIDIA RTX 3090s, acquired before the prices went haywire, providing a generous amount of VRAM to keep several models ready or to run larger ones with higher context.

Model Orchestration

I deployed llama-swap which acts as a proxy that manages llama.cpp instances. Instead of keeping every model loaded in VRAM, llama-swap loads models on demand and unloads them after a period of inactivity. It also handles routing, allowing me to switch between different model tiers—such as a heavy reasoner for complex tasks and a smaller model for fast iterations—without manual intervention.

I developed an automated build system for llama.cpp, or its forks, that builds a new version from any given upstream release tag. The build system will be covered in a dedicated blog article.

Model Hierarchy

These a are my go to models as of late:

Harness

The entire workflow is glued together by pi.dev. It provides a minimalist harness with a lightweight system prompt and a basic toolset. Its “build-your-own” nature allows me to extend the agent’s capabilities by adding custom tools and skills as needed.

The stash

These are some of tools and skills I have gathered and built with the help of LLMs to solve common issues and improve stability for long-running sessions. I will try to keep updating the repository with new stuff that feels relevant.

Internet search is an essential tool for any serious agentic workflow. The search engine is a self-hosted SearxNG instance running inside a container on a remote server accessible from my machines. When deploying the engine you need to make sure to enable json output mode in the engine’s settings file.

The agent’s interface with the engine is a thin CLI client made in Go that compiles to a single binary exported to the agent’s $PATH and symlinked as net-search.

Agents don’t know about SearxNG. They only see an opaque, meta search engine through a simple cli.

net-search-go abstracts any details about SearxNG specifics and only exposes a generic meta-search engine. The command help is self-documenting and presents the bare minimum for quick usage. It encourages the agent to progressively explore advanced search features.

By default it exposes the search interface with the most common search filters. It features a discovery mode that can be invoked with -mode discover. The discovery mode fetches all available engines and bang shortcuts by parsing the source of the self-hosted SearxNG homepage.

Agent discovers engines and searx shortcodes

Hiding SearxNG from the agent

Notice that nowhere within the tool or the skill do I expose the fact that the agent is interacting with SearxNG. Originally, the tool was named searxng-cli and the host URL was set in the variable name $SEARXNG_HOST. However, I noticed that no matter the instructions, agents often hallucinated the SearxNG host URL to http://localhost:8888, which is the default URL found in SearxNG documentation.

This meant that the training data had a strong bias on the LLM’s output. The fix was simply to completely hide that it was interacting with SearxNG. I stripped any reference to SearxNG and replaced it with a generic meta-search engine.

---
name: net-search
description: Search the internet via a self-hosted meta-search engine aggregating dozens of sources. Use for general queries, research, fact-checking, news, code, images, and more.
---

I encourage you to read and modify the skill to your needs. Don’t blindly copy it.

Scraping Content

For static pages, I use a tiered approach to extraction. I start with get-page-content --md, which uses a readability algorithm to strip noise and return clean markdown. This is the most efficient way to feed an agent an article or documentation page.

When I need to traverse a site, I use w3m -dump. It provides a console-optimized text version of the page where links are indexed by numbers, making it easy for the agent to identify and follow specific URLs.

For highly structured data, I use pup. It allows targeted extraction of elements or attributes using CSS selectors. For example, extracting all links from a page is a simple curl -s URL | pup 'a attr{href}'.

Browser Tools

Some websites are JS-heavy or SPAs that static scraping cannot handle. For these, I use a set of minimal CDP tools that connect to a remote headless Chrome instance. These tools are a modified version of the ones created by Mario Zechner (creator of pi.dev), which he detailed in his blog post.

The tools provide basic navigation (brw-goto), JavaScript execution (brw-eval), and content extraction (brw-content). One of the most useful additions is brw-pick, which allows me to interactively pick an element on the page to get its selector, which I then feed back to the agent.

Since Chrome’s CDP server binds exclusively to localhost, I use a socat proxy inside the container to bridge external network requests to the internal CDP port. Combined with tool-side handling of the CDP Host header restrictions, this allows the tools to connect over the network without requiring SSH port forwarding.

Session Dissection

Pi sessions are stored as JSONL files. These can become quite large, and reading them entirely is inefficient. I built dissect to analyze these sessions without loading the whole file.

I frequently use dissect think to review the agent’s reasoning process and dissect bashcmds to extract successful command patterns for future use. The dissect search command allows for fast keyword lookups across messages, tool calls, and thinking blocks.

Bash Stderr Recovery

Agents often use patterns like 2>&1 >/dev/null to silence verbose output, but this inadvertently discards critical error messages. This extension intercepts bash calls and detects when stderr is being hidden.

It automatically rewrites these commands to preserve stderr in a temporary file. This allows the agent to use the last_bash_error tool to recover the error stream from recent calls without having to re-run the command, even if multiple tool calls are made after the error.

Llama.cpp Extension

When using reasoning models via llama.cpp, managing the thinking budget is crucial. This extension maps Pi’s thinking levels (minimal, low, medium, high, xhigh) to reasoning_budget_tokens. It also handles DeepSeek reasoning modes high and max by automatically setting the chat template kwargs for the corresponding Pi thinking level.

To use extension you need to define a provider in your models.json with a name that starts with llama*, it will automatically inject the right parameters before the request is sent.

The most practical feature is in-flight control. I implemented this by analyzing how the llama.cpp source webui handles it. By pressing Ctrl+E during the reasoning phase, I can force the model to stop thinking and jump straight to the final answer. This is a huge time saver when the agent is over-thinking a trivial task.

Github repository: https://github.com/blob42/agent-stash