LIM: Stateful Local Inference Manager

LIM (Local Inference Manager) is a C++ terminal-based LLM controller built on llama.cpp. It provides a persistent, stateful session where the KV-cache is never discarded: each turn's tokens are simply appended, so new input costs only O(input tokens), not O(total history). LIM has native filesystem tools for reading, searching, editing, and writing files, plus web searching and PDF reading.

Benchmark Results

Cumulative decode time vs context length for Qwen3.6-35B-A3B-UD-Q4_K_XL on an NVIDIA RTX 5090 / Intel i9-12900K. Mode 0 (LIM) adds only O(input tokens) per turn. At 222041 context tokens, Mode 2 (CACHED), which emulates llama-server, is only 1.8% slower than Mode 0 since it uses prefix matching to avoid re-decoding the entire history: every turn pays the O(history) re-tokenize + prefix-compare cost, and when the re-tokenization drifts from the sampled tokens it re-decodes from the last prompt checkpoint. Mode 1 (CHATBOT) re-tokenizes the full conversation text and re-decodes the entire history at each turn, growing quadratically: at the same context it has used 73% more time than Mode 0. That is, LIM is 1.73x faster than a standard chatbot.

How It Works

Every mainstream chatbot (and most local LLM frontends using server APIs) follows the reprocess-every-turn model: each time you send a message, the entire conversation history is re-transmitted in full and re-tokenized from scratch. The per-turn decode cost grows linearly with context length, making long conversations progressively slower.

LIM avoids this by design: it runs locally as a single persistent process where the KV-cache is never discarded. Each turn simply continues from where the last one left off. This is the same approach used by llama-cli in interactive mode. LIM includes an integrated tool system, session save/restore, interactive undo, and LaTeX-aware browser output.

---

Architecture at a Glance

+-------------+     +---------------+     +---------------+
|  Your TTY   |<--->|   lim (C++)   |<--->|  llama.cpp    |
| (readline)  |     |  controller   |     |  inference    |
+-------------+     +-------+-------+     +---------------+
                            |
              +-------------+-------------+
              |             |             |
        +-----v-----+ +----v-----+ +-----v------+
        | Filesystem| | Web      | |  Browser   |
        |  Tools    | | Search   | | (FIFO)     |
        |           | | & PDFs   | |            |
        +-----------+ +----------+ +------------+

• lim: The C++ binary. Handles the REPL, KV-cache management, tool dispatch, and signal handling.
• limServer.py: An optional Python WebSocket server that streams output to a browser via /tmp/lim.fifo.

---

Prerequisites

1. A GPU with CUDA support (NVIDIA recommended) and the CUDA toolkit installed. For CPU-only builds, run make GGML_CUDA=off.
2. Python 3 with aiohttp for the browser server: pip3 install aiohttp.
3. A GGUF model file (e.g., Qwen, Llama, Mistral).
4. Optional: SearXNG for web search and Docling for PDF reading. LIM auto-starts them on demand; override the commands with LIM_SEARXNG_CMD / LIM_DOCLING_CMD (see Web Search & PDF Setup below).

Note: llama.cpp is bundled as a git subrepo with three patches applied in-tree (the patched code is committed and built directly by the Makefile; the matching *.patch files are kept as records so a future git subrepo pull can re-derive them):

• llama-checkpoint.patch (LIM-authored) -- recurrent state checkpointing, required for instant /undo on hybrid models. Pending PR upstream.
• llama-utf8.patch (LIM-authored) -- rejects out-of-range 4-byte UTF-8 sequences so malformed output can't terminate the process. Pending PR upstream.
• llama-pr28243.patch (upstream PR #28243, not authored by LIM) -- qwen4exp MTP draft graph, required for LIM_MTP on the qwen4exp architecture (Qwen3.8-Flash-Next). Two hunks were hand-merged against the current upstream base (the TENSOR_ALLOW_RESHAPE hyper-connection refactor).

---

Building

git clone https://github.com/statefullm/lim.git
cd lim
make
./lim

The build will automatically detect your GPU (if any) and compile llama with the appropriate architecture flags. No manual setup is required.

make also builds the VS Code extension (requires Node.js and npm; it is installed by make install). To build only the binary at any time, use make lim.

Custom Build Options

All auto-detected values can be overridden via environment variables:

Variable	Purpose	Example
CUDA_ARCH_FLAGS	Override GPU architecture detection	make CUDA_ARCH_FLAGS=120a
LIM_LLAMA_BUILD_DIR	Use pre-built llama from external directory	export LIM_LLAMA_BUILD_DIR=/path/to/llama/build
LIM_LLAMA_CMAKE_FLAGS	Extra cmake flags passed to the llama build	make LIM_LLAMA_CMAKE_FLAGS="-DGGML_AVX512=off"
GGML_CUDA	Force CUDA on/off	make GGML_CUDA=off for CPU-only
GGML_HIPBLAS	Enable ROCm/HIP build	make GGML_HIPBLAS=on

To clean only the llama build artifacts without touching lim:

make llama-clean

Installing LIM

After building, install the binary, config files, and VS Code extension. This must be run as $LIM_AI_USER (e.g., su - $LIM_AI_USER) so that files are installed into that user's home directory, which is where LIM will look for them at runtime:

su - $LIM_AI_USER
make install

This installs:

• lim binary to ~/bin/lim (ensure ~/bin is in your PATH via .bashrc)
• System prompt to ~/.config/lim/prompt
• Reincarnate instructions to ~/.config/lim/reincarnate
• Search cache directory at ~/.config/lim/searchCache
• Browser server files (limServer.py, viewer.html, libs/) to ~/.config/lim/
• VS Code extension (built, then installed into this user's VS Code via code)

All LIM configuration files live under ~/.config/lim/ by default. Override with the LIM_CONFIG_DIR environment variable (set in $LIM_AI_USER's .bashrc) to place them elsewhere:

# As $LIM_AI_USER
export LIM_CONFIG_DIR=/opt/lim/config
make install

The LIM_CONFIG_DIR is read at runtime, so no recompilation is needed.

---

User Setup

The dedicated LLM user is controlled by the LIM_AI_USER environment variable (defaults to ai). All references below use $LIM_AI_USER. The group name always matches the user name.

1. Creating the AI User

LIM must run as a dedicated user. The binary enforces this at startup: it checks getuid() and refuses to run otherwise. This is a security measure: the LLM has filesystem write access and shell execution capabilities, so isolating it behind a dedicated user limits blast radius.

Create the user and group if they don't exist:

export LIM_AI_USER=ai
sudo groupadd -f $LIM_AI_USER
sudo useradd -m -g $LIM_AI_USER -s /bin/bash $LIM_AI_USER

Then add your personal user to the $LIM_AI_USER group so you can read and write files the LLM creates without needing sudo chown:

sudo usermod -aG $LIM_AI_USER $USER

Log out and back in (or run newgrp $LIM_AI_USER) for the group change to take effect.

2. Git Safe Directories

By default, Git refuses to operate in repositories owned by a different user. Since LIM runs as $LIM_AI_USER but your project directories are owned by you, add the [safe] section to your personal ~/.gitconfig:

[safe]
    directory = /home/$LIM_AI_USER

This prevents "fatal: detected dubious ownership in repository" errors when the LLM runs git commands.

3. Directory Permissions

Your project directories should be group-writable by the $LIM_AI_USER group so the LLM can read and write files. Because you added your personal user to the $LIM_AI_USER group (step 1), access is symmetric: you can read files the LLM creates, and the LLM can read your files. The recommended layout for /home/$LIM_AI_USER:

$ ls -ld /home/$LIM_AI_USER
drwxrwsr-x. 42 $USER $LIM_AI_USER 4096 Jan  1 00:00 /home/$LIM_AI_USER/

The setgid bit (s) ensures new files inherit the $LIM_AI_USER group.

To set permissions in any project sandbox, copy aishare from this repository to your personal /home/$USER/bin/:

cp aishare /home/$USER/bin/

Then run it on any directory under /home/$LIM_AI_USER that you want the AI to access:

aishare /home/$LIM_AI_USER/project

It sets ownership to $USER:$LIM_AI_USER (via sudo), grants group read/write and setgid, and clears the sticky bit. It refuses targets outside /home/$LIM_AI_USER. When run on /home/$LIM_AI_USER, it fixes permissions on the home directory itself plus all top-level entries, skipping .ssh.

4. Connecting and Running LIM

LIM must be run as user $LIM_AI_USER. The simplest approach is to SSH into the host as that user, then launch lim directly. This keeps the workflow consistent whether you're using VS Code or a standalone terminal.

Add this to the ~/.bashrc of $LIM_AI_USER:

alias coder='lim ~/models/Qwen3.8-27B-Q6_K.gguf'

Replace the model path with whichever GGUF you want to use. Then, before running coder, connect to the LIM server $LIM_HOST:

ssh -a $LIM_AI_USER@$LIM_HOST
coder

If running locally (no SSH needed), you have two options:

Option A: Switch users fully

su - $LIM_AI_USER
coder

Option B: Launch from your personal shell via sudo

Add this function to your ~/.bashrc:

coder() { sudo -iu $LIM_AI_USER sh -c "cd \"$PWD\"; lim ~/models/Qwen3.8-27B-Q6_K.gguf $*"; }

You can then start lim as user $LIM_AI_USER in your current directory:

coder

Core Pinning (taskset)

LIM automatically detects P-cores and E-cores on hybrid CPUs (Intel Alder Lake, Raptor Lake, etc.) by comparing thread_siblings_list from sysfs. It pins background services accordingly:

Workload	Default Cores	Description
SearxNG search server	E-cores	Light background Python process
Browser WebSocket server	E-cores	Light background Python process
Docling PDF converter	P-cores	Heavy ML inference workload

On non-hybrid CPUs (all cores identical), no taskset pinning is applied. If the pinning command isn't found on $PATH (e.g., macOS has no taskset), pinning is silently skipped.

Override the auto-detected layout with LIM_TASKSET:

# Format: LIM_TASKSET="P_CORES:E_CORES"
export LIM_TASKSET="0-15:16-23"   # Classic i9-12900K layout
export LIM_TASKSET="0-7:"         # P-cores 0-7, no E-core pinning
export LIM_TASKSET=":8-15"        # No P-core pinning, E-cores 8-15
export LIM_TASKSET="::"           # Disable all taskset pinning

macOS / non-Linux systems: macOS has no taskset binary. To enable manual core pinning, install an alternative and point LIM_TASKSET_CMD at it:

# Install numactl via Homebrew, then use it as the pinning backend
brew install numactl
export LIM_TASKSET="0-3:4-7"
export LIM_TASKSET_CMD="numactl --physcpubind"

Or write a custom wrapper script and set LIM_TASKSET_CMD to its path. If the command isn't on $PATH, pinning is silently skipped -- services still start normally, just unpinned.

The detected topology and pinning status are logged to stderr at startup when LIM_DEBUG=1.

5. Setting Up $HOME/.bashrc for $LIM_AI_USER

Add these lines to /home/$LIM_AI_USER/.bashrc:

# --- /home/$LIM_AI_USER/.bashrc ---

umask 0002                    # Group-writable files by default

# Track current working directory so the LLM knows where you are.
cd() {
    builtin cd "$@" && pwd > $HOME/.cwd
}

# Block git add -A / --all and git add . to prevent the LLM from staging untracked files
git() {
    if [ "$1" = "add" ]; then
        for arg in "$@"; do
            if [ "$arg" = "-A" ] || [ "$arg" = "--all" ] || [[ "$arg" =~ ^-[a-zA-Z]*A[a-zA-Z]*$ ]] || [ "$arg" = "." ]; then
                echo "ERROR: git add -A / --all / . is disabled: use git commit -a or specify files explicitly" >&2
                return 1
            fi
        done
    fi
    command git "$@"
}

# Optional: add custom paths
export PATH="$HOME/bin:$PATH"

The cd override writes the current directory to $HOME/.cwd. LIM writes it at startup from its own working directory and reads it to resolve relative paths in the file tools and to set the working directory for exec_shell commands. The umask 0002 ensures files created by $LIM_AI_USER are group-readable/writable.

6. The System Prompt

The system prompt lives at ~/.config/lim/prompt. A default prompt file ships with this repository and is installed to that location by make install. This file is read at startup and baked into the KV-cache (and re-read from disk on /clear and /reincarnate). It defines the LLM's behavior: available tools, editing workflow, formatting rules, etc. You can customize it for different use cases (coding assistant, writer, researcher, etc.). An optional directory-specific file localprompt (with a fallback to $LIM_CONFIG_DIR/localprompt) will be prepended to the system prompt.

Reasoning effort: For models that support reasoning-effort steering, you can add a reasoning effort instructions to localprompt, matching how Qwen's own template places its reasoning instructions. The suggested Qwen 3.8 instructions are:

• Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.
• Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.

Omitting any such instruction leaves the model at its baseline behavior. For a full setup -- mode files, per-mode sampling env, and a login hook that keeps the two consistent -- see REASONING.md.

7. Message Shortcuts

Create /home/$LIM_AI_USER/.lim_aliases to define shorthand expansions at the >>> prompt:

# ~/.lim_aliases: one key=value per line; # comments are ignored
# Keys must start with / to distinguish them from regular messages

/test=Test the filesystem tools and clean up after.
/message=Show me a 1-sentence concise git commit message relative to HEAD.
/commit=run git commit -a -m "<message>"
/make=run make
/amend=Amend the previous commit.
/remind=Read the system prompt from ~/.config/lim/prompt and follow it strictly!
/prompt=Follow the system prompt strictly!
/tools=Use our robust filesystem tools.

At the >>> prompt, typing /commit expands to run git commit -a -m "<message>" before being sent to the LLM. Lines starting with # are treated as comments. Blank lines are skipped. Built-in command names (e.g., /quit, /clear) cannot be used as alias keys. This is loaded fresh each session.

---

Output Modes

Set via LIM_OUTPUT:

Value	stdout	Browser	Description
3	Y	Y	Both stdout and browser
2 (default)	N	Y	Browser only
1	Y	N	Stdout only
0	N	N	No LLM output

---

Environment Variables

Variable	Default	Description
LIM_AI_USER	ai	Username that LIM must run as. Used by the binary for the user check, and by the VS Code extension for SSH.
LIM_CACHE_DIR	.cache	Directory (relative to CWD) for fast restore KV-cache files.
LIM_CONFIG_DIR	~/.config/lim	Directory for LIM config and server files (prompt, reincarnate, userprompt, searchCache, limServer.py, viewer.html, libs/). The C++ binary launches python3 $LIM_CONFIG_DIR/limServer.py when browser output is enabled. Override to place everything elsewhere.
LIM_LOG_DIR	log	Directory (relative to CWD) for session logs, token traces, and autosave files.
LIM_SAVE_DIR	.	Directory prepended to relative paths in /save, /load, /delete, and the CLI restore argument. Absolute paths (starting with /) are unaffected.
LIM_HOST	unset	Hostname or IP of your LIM server. Used for SSH connection and browser viewer URL.
LIM_PORT	8765	Port for the browser WebSocket server
LIM_OUTPUT	2	Output mode: 0 = none, 1 = stdout only, 2 = browser only (default), 3 = both
LIM_VIEWER_URL	(auto)	Override the auto-generated viewer URL
LIM_WEB_CONTEXT_FRACTION	0.75	Fraction of LIM_CTX reserved for fetched web content budget. The per-file limit (LIM_WEB_FILE_MAX) defaults to 25% of this budget. Set between 0.0 and 1.0 (0.0 means the default value of 0.75).
LIM_WEB_FILE_MAX	(auto)	Max characters per fetched file before middle-drop truncation. Defaults to 25% of the session web budget (LIM_CTX * LIM_WEB_CONTEXT_FRACTION * 4). Set explicitly to override.
LIM_WEB_HTML_MAX	500000	Max bytes to buffer when downloading HTML/text pages via curl
LIM_WEB_PDF_MAX	50000000	Max bytes to buffer when downloading PDFs via curl (50 MB)
LIM_WEB_TIMEOUT	600	HTTP request timeout in seconds for fetches and searches
LIM_BRAVE_API_KEY	(empty)	Brave Search API key. When set, LIM falls back to the Brave Search API if SearXNG fails or returns no results.
GH_TOKEN	(empty)	Optional GitHub personal access token for higher API rate limits when cloning repos or fetching from GitHub.
LIM_SEARCH_COOLDOWN	3	Minimum seconds between web searches to avoid rate-limiting SearxNG
LIM_DOCLING_CMD	~/venv/bin/docling-serve run --enable-ui	Command to start the Docling PDF service. Override if installed elsewhere (e.g., via Docker or a different venv).
LIM_SEARXNG_CMD	cd ~/searxng && exec python -m searx.webapp	Command to start the SearxNG search service. Override if installed elsewhere.
LIM_DEBUG	0	Set to 1 for verbose token-level logging in $LIM_LOG_DIR/<N>.tokens
LIM_LLAMA_DEBUG_LOG	0	Only with LIM_DEBUG=1: set to 1 to also print DEBUG-level llama.cpp/ggml log lines (e.g. "CUDA Graph id N reused" on every decode). Default keeps INFO and above.
LIM_EOG_RESAMPLE_MAX	256	Maximum resampling attempts when a spurious EOG is detected. When the model emits an EOG token but hasn't finished its response, LIM resamples up to this many times trying to recover a non-EOG token. Increase if you see premature turn endings.
LIM_TOOL_IGNORE	100	Maximum tokens allowed outside parameters while inside a tool call before attempting a tool correction.
LIM_GPU_LAYERS	-1	Number of layers offloaded to GPU (-1 = auto-fit all layers). When set explicitly, bypasses auto-fitting. For MoE models that exceed VRAM, auto-fit uses partial layer offloading (dense weights on GPU, sparse expert weights on CPU) for optimal throughput.
LIM_HONEST_SPEED	0	Set to 1 for "honest" wall-clock speed diagnostic -- tokens / total generate() call time, including all CPU-side overhead (sampling, output rendering, tool-call detection). Default 0 reports generation time matching llama-cli's "Generation: X t/s" -- measured from first-token sample+sync to last-token sample+sync, covering N sampling operations and (N-1) decode cycles.
LIM_SPEED_INTERVAL	100	Number of tokens between in-loop speed diagnostic updates.
LIM_USE_MLOCK	1	Lock model in RAM to prevent swapping
LIM_USE_MMAP	0	Use memory-mapped model loading (faster startup, more RAM pressure)
LIM_BATCH	2048	Batch size for token feeding
LIM_CTX	262144	Context window size (KV-cache token capacity)
LIM_MTP	0	Set to 1 to enable MTP (multi-token-prediction) speculative decoding. A second context mirroring the main KV cache proposes up to LIM_MTP_DRAFT tokens per round, and the main model verifies all of them in a single batch decode; each committed token is a fresh sample from the full target sampler chain at its own verify position, so the output is distribution-exact under any sampling configuration. It is not token-identical to a non-MTP run, but two MTP runs at LIM_TEMPERATURE=0 are token-identical to each other. Speculation pauses only when the mirror can't be healed in place, due to a main decode truncated by KV exhaustion, a fast-restore from a cache saved without MTP, or sustained low acceptance. It auto-disables itself when committed tokens/round stays below 1.25 for two consecutive 32-round windows; /clear re-enables it. MTP is not implemented for benchmark chatbot modes 1 and 2.
LIM_MTP_BATCH	128	Cap on the MTP draft context's batch size (1-512, only with LIM_MTP=1). The draft context's scheduler work buffers scale with this; nothing ever decodes more rows than this at once (the draft chain is single-row, the mirror prefill simply runs in more chunks), so lowering it saves VRAM at the cost of a slightly slower one-time prefill mirror. The actual draft-context memory is printed at startup with LIM_DEBUG=1.
LIM_MTP_DRAFT	4	Number of tokens proposed per MTP round (1-32, only with LIM_MTP=1); must not exceed the model's recurrent rollback window. The default was found to be the speed sweet spot for the Qwen3.8-27B class -- longer drafts cost more memory for diminishing acceptance gains.
LIM_MTP_SIDECAR	(empty)	Path to a separate MTP-only GGUF file (only with LIM_MTP=1). When set, the MTP draft head is loaded from this file instead of the main model. Use this for models whose GGUF quantization omits the nextn tensors (e.g. some Qwen3.8-Flash-Next builds). The sidecar is a small GGUF containing only the MTP layer weights (~2 GB) with nextn_predict_layers in its metadata.
LIM_THREADS	(auto)	Threads for inference: physical P-core count on hybrid CPUs (E-cores excluded), physical core count on non-hybrid CPUs, logical core count if the CPU topology can't be read
LIM_THREADS_BATCH	(auto)	Threads for batch processing (same default as LIM_THREADS)
LIM_UBATCH	512	Unbatched size
LIM_MIN_P	0.0	Minimum probability threshold: keep tokens where P >= min_p * P(top)
LIM_FREQUENCY_PENALTY	0.0	Frequency penalty: discourages overused tokens proportional to frequency. Legacy name LIM_PENALTY_FREQ still accepted (new name wins if both set)
LIM_PRESENCE_PENALTY	1.5	Presence penalty: discourages repeating previously used tokens. Legacy name LIM_PENALTY_PRESENT still accepted (new name wins if both set)
LIM_REPETITION_PENALTY	1.0	Repetition penalty multiplier (1.0 = no penalty). Legacy name LIM_PENALTY_REPEAT still accepted (new name wins if both set)
LIM_SEED	(auto)	Random seed for reproducibility (default is time-based)
LIM_TEMPERATURE	0.7	Sampling temperature (set to 0 for deterministic/greedy decoding). At 0 LIM also omits the wall-clock timestamp from the system prompt (the only session-varying prompt content), making runs byte-reproducible; the model does not know the current date/time in this mode -- pass it explicitly in the prompt if needed. Legacy name LIM_TEMP still accepted (new name wins if both set)
LIM_THINKING	1	Set to 0 to suppress thinking blocks via a pre-filled stub for faster throughput. Not recommended for math or complex reasoning tasks, as it can cause incorrect answers by skipping intermediate steps.
LIM_DUMMY_THOUGHT	(built-in)	Pre-filled stub used when LIM_THINKING=0. Leave unset to use the built-in default, set to an empty string to emit an empty thinking block (Qwen 3.8's "no thinking" signal), or set to any other string.
LIM_ESCAPE_CONTRACT	0	Set to 1 to include the reserved-token escape contract in the system prompt. The escape mechanism itself is always active; this only controls whether the LLM sees the explicit rules.
LIM_INLINE_LATEX	1	Set to 0 to disable inline KaTeX rendering in the browser viewer. Inline $...$ candidates are validated before rendering (LaTeX whitespace rule, env-var identifier pattern, KaTeX parse check) and code spans / fenced blocks are never treated as math, so env vars like $HOME or $PATH are left as literal text. Block-level $$...$$ math always renders via katex.renderToString.
LIM_TOP_K	20	Keep only the top_k most likely tokens (applied after the penalty sampler, before top_p / min_p / temperature)
LIM_TOP_P	0.8	Nucleus sampling: consider tokens with cumulative probability <= top_p
LIM_CACHE_TYPE_K	Q8_0	KV-cache key storage type (F16, Q4_0, Q5_0, Q5_1, Q8_0, Q8_1)
LIM_CACHE_TYPE_V	Q8_0	KV-cache value storage type (F16, Q4_0, Q5_0, Q5_1, Q8_0, Q8_1)
LIM_CHATBOT_MODE	0	Benchmarking mode: 0 = LIM normal (persistent KV-cache, same as llama-cli interactive), 1 = standard chatbot (re-tokenize + re-decode full history each turn), 2 = cache-aware prefix match (emulates llama-server). Modes 1 and 2 force honest speed measurement.
LIM_EXEC_TRUNCATION	32768	Maximum bytes of exec_shell output before truncation
LIM_MAX_AUTO_CONTINUE	500	Maximum depth of automatic tool-call chaining
LIM_TURN_TIMEOUT	3600	Maximum seconds per generation turn before auto-abort
LIM_TASKSET	(auto)	Format: "P_CORES:E_CORES" (e.g., "0-15:16-23"). Auto-detected on hybrid CPUs. Set to "::" to disable all pinning.
LIM_TASKSET_CMD	taskset -c	Override the core-pinning command. On macOS (no taskset), install numactl via Homebrew and set to numactl --physcpubind. If the command isn't on $PATH, pinning is silently skipped.

Sampling Behavior

Between user turns, LIM resets the sampler chain (penalties ring buffer and RNG seed). This means repetition penalties apply only to tokens generated during the current turn, not to stale tokens from previous responses, and not to the user's input. llama-cli also penalizes only generated tokens, but it never resets the sampler chain between turns, so its penalty ring (64 tokens by default) can still carry tokens from the previous response into the next one. During auto-continue (tool-call chains, /continue), the sampler state is preserved so generation continues seamlessly.

---

The AI Sandbox

The $LIM_AI_USER operates in a sandboxed environment:

• File access is limited to directories where the $LIM_AI_USER group has write permission. Use aishare to grant access to new project directories. Files created by the LLM are owned by $LIM_AI_USER:$LIM_AI_USER with group read/write permissions (via umask 0002), so your personal user can access them directly since it is a member of the $LIM_AI_USER group.
• Shell commands executed via the exec_shell tool run as user $LIM_AI_USER. They inherit that user's PATH and environment.
• Git integration: The AI can clone public repos and commit locally, but should not be given the authentication needed to push changes back to remote repositories.

---

Usage

Starting a Session

SSH in as $LIM_AI_USER, then run your coder alias. You'll see the >>> prompt. Type your request and press Enter.

Available Tools

The LLM has access to six tools. You don't call them directly; just describe what you want and the LLM handles the rest:

• read_files: Read text files, PDFs, or URLs
• search_file: Search for text within a file, with optional line range
• edit_file: Replace exact text within a file (surgical editing)
• write_file: Write or overwrite a file
• exec_shell: Run a shell command as $LIM_AI_USER
• web_search: Search the web via SearXNG

Tools are invoked via XML-structured tags in the LLM's output. Results are fed back as continuation tokens into the KV-cache, avoiding the overhead of re-tokenizing the full conversation history.

Multi-line Input with Ctrl+J

The readline interface binds Ctrl+J to insert a literal newline into your input, while Enter (Return) submits the line. This lets you compose multi-line messages:

>>> Here's a function I need fixed:
(ctrl-j)
(ctrl-j)
def foo(x):
(ctrl-j)
    return x + 1
(ctrl-j)
(ctrl-j)
Please fix the type hints.      <-- Enter submits

The binding is restored after each input cycle so it doesn't leak into your shell session.

Readline Keybindings

The prompt uses GNU readline in callback mode with select() polling instead of blocking I/O. This allows Ctrl+C interrupts to be handled immediately during user input, not just during generation. All standard Emacs-style bindings are available:

Keys	Action
Ctrl+A / Ctrl+E	Beginning / end of line
Ctrl+B / Ctrl+F	Backward / forward character
Alt+B / Alt+F	Backward / forward word
Ctrl+K / Ctrl+U	Kill to end / start of line
Ctrl+W	Kill word backward
Ctrl+Y	Yank (paste)
Ctrl+_	Undo
Tab	Completion (if configured)
Up / Down	History navigation
Ctrl+R	Reverse search history

In-Session Commands

Command	Effect
/clear	Auto-save the current state to $LIM_LOG_DIR/<N>-clear.save, then clear the KV-cache (reset to system prompt only). The auto-saved file lets you restore if you change your mind. Use /save <name> to create a permanent restore point before clearing.
/undo	Interactive undo: auto-saves first to $LIM_LOG_DIR/<N>-clear.save, then presents an Undo> prompt listing all checkpoints (most recent first). Use up/down arrows to navigate, Enter to confirm. Ctrl+C, Ctrl+D, /quit, or /exit cancel the undo and return to the user prompt without losing your session. Selecting a checkpoint restores the session to the end of the turn associated with that prompt. On hybrid models (Qwen3.5/3.6), instant undo works for checkpoints generated in the current session; pre-restore checkpoints from a fast cache restore require re-decode fallback unless restored via --checkpoints. Readline history is updated to reflect the restored session state.
/continue	Resume generation after an interruption. If interrupted mid-tool-call, resumes from the exact point of interruption
/reset	Reset terminal and web search. Useful for recovering from a corrupted terminal or disabled web search after an interrupt or connection failure
/reincarnate	Ask the LLM to compose a new prompt in ~/.config/lim/userprompt, then clear and restart with it
/save	Save the full session state to $LIM_LOG_DIR/<N>.save, overwriting any previous save for this session
/save <path>	Save the full session state to <path>.save. The path can be relative or absolute. If it already ends in .save, no extra extension is added. Use this to create named restore points at meaningful moments in your session.
/quit or /exit	Auto-save the current state to $LIM_LOG_DIR/<N>.save, then exit the session
/quit <path> or /exit <path>	Named save with fast cache to <path>.save, then exit the session. Accepts the same path format as /save.
/load <path> [--checkpoints]	Load a saved session from within the current session. Must be used immediately after /clear: fails if any conversation tokens have been added since the clear. Accepts the same path format as /save: .save is appended automatically if not already present. Uses the fast cache when available, falling back to a re-decode that first offers checkpoint selection at a Restore> prompt. --checkpoints skips the fast cache (forcing the prompt) and skips the automatic fast-cache write.
/delete <path>	Delete a save file and its associated fast restore cache entry (if any). Accepts the same path format as /save: .save is appended automatically if not already present.
/help	Display a summary of all available commands

Save and Restore

You can save a running session and restore it later with zero context loss:

Save: Type /save at the >>> prompt to save to $LIM_LOG_DIR/<N>.save (overwrites any previous save for this session). Use /save <path> to create named checkpoints: e.g., /save cats saves to cats.save, and /save /tmp/checkpoint saves to /tmp/checkpoint.save. If the path already ends in .save, no extra extension is added. The save file contains only the conversation token sequence, keeping it small. Named saves also cache the full KV-cache for fast future restores.

Restore: Pass a save file as the last argument to coder. The .save extension is added automatically if not already present, matching /save behavior. The CLI restore runs the exact same code path as in-session /load (it injects /load <path> as the first command after startup), so behavior is identical in both, including --checkpoints:

coder $LIM_LOG_DIR/5.save    # explicit extension
coder $LIM_LOG_DIR/5         # .save appended automatically
coder cats                   # restores from cats.save

This restores the session exactly as it was: the full conversation, KV-cache position, and generation state. The LLM continues generating from where it left off. Typing /clear after a restore resets to a fresh system prompt with the current date and working directory (but first auto-saves the restored state).

In-session load: You can also restore from within a running session using /load <path>. This must be used immediately after /clear; if any conversation tokens have been added since the clear, the command will fail. It uses the same path conventions as /save (.save appended automatically) and takes advantage of the fast cache when available. Typical workflow:

>>> /clear
Context Cleared Successfully
>>> /load cats
Restoring session from cats.save... (75432 tokens)
Session #5 restored: 75432 tokens loaded (28%)

Partial restore via checkpoints: Save files record a checkpoint at the end of each conversation turn, storing your prompt text and the token position. Whenever a restore can't use the fast cache (or with --checkpoints, CLI or /load), LIM offers a choice of checkpoints before decoding so you can select a good restore point. Use up/down arrow keys to navigate through your prompts (most recent first). Press Enter to confirm. If no checkpoint matches your input, all tokens are restored by default. Press Ctrl+C, Ctrl+D, or type /quit at the Restore> prompt to cancel; the session continues fresh from the system prompt. Restoring to a checkpoint replays tokens only up to the end of that turn -- as if you had just typed that prompt and received the response, and the session is ready for your next message. The available checkpoints accumulate across restore/save cycles: restoring from a save file carries over its checkpoints, and new turns add more.

Checkpoint restore: Add --checkpoints to skip the fast-format cache, triggering the checkpoint selection prompt even when a fast cache exists. This also rebuilds recurrent-state checkpoints at each prompt boundary during decode, enabling instant /undo for all historical turns on hybrid models (Qwen3.5/3.6). The flag works on the command line (after the save file) and with the in-session /load command, so there is no need to exit and restart the model:

coder cats --checkpoints

# Or, within a running session:
>>> /clear
>>> /load cats --checkpoints

Fast restore cache: On first restore, tokens are decoded through the model to rebuild the KV-cache. During this decode, recurrent-state checkpoints are regenerated at each prompt boundary so that /undo works instantly for hybrid models right after restore. The rebuilt cache is then automatically written to $LIM_CACHE_DIR so all subsequent restores from the same save file are fast; when MTP is enabled, the draft mirror KV is also stored in the cache file, so a fast restore keeps MTP active. Named saves (e.g., /save cats) also write the fast-format cache immediately for faster future restores. The name prefix in the cache filename is purely informational; LIM identifies cache files by their content hash, not their name. If you run /save cats followed by /save dogs with identical session content, only one cache file is written since both resolve to the same hash. Unnamed /save and auto-saves from /quit, /clear, and /reincarnate skip the fast cache to save disk space, relying on the automatic cache built on first restore. For fast cache restores, recurrent checkpoints build up naturally as new conversation turns complete, supporting instant /undo for each new turn. The $LIM_CACHE_DIR/ directory is safe to delete at any time to reclaim space; it will be regenerated on the next restore.

Auto-save on clear and quit: Before clearing the context, LIM automatically saves the current state to $LIM_LOG_DIR/<N>-clear.save. Before exiting, it saves to $LIM_LOG_DIR/<N>.save. These use different filenames so neither clobbers the other: if you clear and then exit, both the pre-clear and post-clear states are preserved. To keep a permanent checkpoint at any point, use /save <name>.

Interrupting Generation

Press Ctrl+C during generation to interrupt. The partial output is preserved in the KV-cache. Type /continue to resume seamlessly: the LLM doesn't even know it was interrupted. This works because the KV-cache still holds all generated tokens up to the interruption point.

Browser Output

By default (LIM_OUTPUT=2), LIM streams output to a browser via WebSocket. Start your LIM session first, then load:

http://<hostname>:8765/viewer.html

This ensures the server is already running when the browser connects, avoiding any need to reload the page.

The Python server (limServer.py) is auto-started by the C++ binary when browser output is enabled. It runs on efficiency cores when a hybrid CPU is detected (unpinned on non-hybrid CPUs). The server reads from a named FIFO at /tmp/lim.fifo and broadcasts to all connected WebSocket clients.

VS Code Extension

The LIM Workspace extension provides a convenient way to set up your workspace: it creates an integrated terminal and opens a browser viewer once the server is ready.

The extension does two things:

• Opens a browser pointing to $LIM_HOST (or localhost) on the configured port, after waiting for the server to respond.
• Creates a terminal at $HOME. If $LIM_HOST is set and remote, it SSHs into that host as user $LIM_AI_USER; otherwise it opens a local shell.

You then run your coder alias in that terminal as usual.

Install:

make install (see Installing LIM) builds and installs the extension. To build or install it separately:

make vscode             # builds and packages the extension
make install-vscode     # installs the extension into VS Code

make install installs everything (binary, config files, and VS Code extension). To uninstall:

make uninstall          # removes the binary and config files
make uninstall-all      # removes installed files and uninstalls the extension

Then in VS Code, click the LIM rocket icon in the status bar (or Ctrl+Shift+P -- LIM: Open Workspace). This opens a terminal panel and waits for the browser server. Run your coder alias in that terminal, and the viewer will connect automatically once the server starts.

To rebuild both the C++ binary and the extension:

make

---

Benchmarking

LIM supports benchmarking modes controlled by LIM_CHATBOT_MODE to compare its persistent KV-cache approach against standard chatbot and cache-aware decoding. Each mode writes a TPS log (log/<N>.tps) recording context position and tokens-per-second at each speed diagnostic update (see Logging).

Value	Mode	Description
0 (default)	LIM normal	KV-cache persists across turns. Each token is decoded once and never re-decoded. Same approach as llama-cli interactive mode.
1	Standard chatbot	Emulates a plain text-based chatbot: the conversation is held as text, so each turn re-tokenizes the full conversation text and re-decodes everything from scratch (no KV reuse). TPS includes re-tokenize + re-decode + generation.
2	Cache-aware prefix match	Emulates llama-server behavior: KV-cache stays in memory, but each turn re-tokenizes the full conversation text and compares against the cached prefix to find where to resume decoding.

Chatbot modes (1 and 2) automatically enforce LIM_HONEST_SPEED=1. The TPS reported in logs includes the re-decode overhead. This ensures the benchmark numbers reflect the true wall-clock cost of each approach.

(See the cumulative-time benchmark figure at the top of this README.)

Note: For fair comparisons, run benchmarks with an empty system prompt (remove ~/.config/lim/prompt) to inhibit tool calls. In mode 1, previous tool calls would be both re-decoded and re-executed on every turn, causing massive slowdowns plus unwanted side effects.

---

Web Search & PDF Setup

LIM can start SearXNG and Docling automatically when needed, but you must install them first:

SearXNG (Web Search)

cd ~
git clone https://github.com/searxng/searxng.git
cd searxng
pip install --no-build-isolation -e .

Then edit searx/settings.yml: set a value for server.secret_key (required). The defaults for bind_address (127.0.0.1) and port (8888) are already correct for LIM. You may need to disable some search engines that block automated queries -- see the SearXNG docs for details. Override the start command with LIM_SEARXNG_CMD if installed elsewhere.

Docling (PDF Reading)

cd ~
python3 -m venv venv
source venv/bin/activate
pip3 install docling-serve
deactivate

LIM expects the Docling binary at ~/venv/bin/docling-serve and starts it on port 5001. Override the start command with LIM_DOCLING_CMD if installed elsewhere.

---

Troubleshooting

"FIFO not found" or browser output fails

The FIFO at /tmp/lim.fifo is created automatically when LIM starts. If it was removed manually, restart your LIM session. Ensure aiohttp is installed: pip3 install aiohttp.

"SearxNG connection failed"

LIM tries to connect to SearXNG on 127.0.0.1:8888. Make sure SearXNG is installed at ~/searxng/ and configured to listen on that port in settings.yml. You can also run SearXNG externally and point LIM to it -- just ensure the server is already listening before your first web search.

"Docling Error" when reading PDFs

Ensure Docling is installed. If it's not at ~/venv/bin/docling-serve, set LIM_DOCLING_CMD to the correct command, e.g.:

export LIM_DOCLING_CMD="/opt/docling/bin/docling-serve run --enable-ui"

"No GPU detected and CUDA_ARCH_FLAGS not set"

If you don't have an NVIDIA GPU, build for CPU only:

make GGML_CUDA=off

Browser viewer shows nothing / won't connect

1. Start LIM first, then open the viewer URL. The server must be running before the browser connects.
2. Check that nothing else is using port 8765 (or your custom LIM_PORT).
3. If behind a firewall or SSH tunnel, set LIM_VIEWER_URL to the correct address.

"detected dubious ownership in repository"

See Git Safe Directories in the User Setup section. Add [safe] directory = /home/$LIM_AI_USER to your personal ~/.gitconfig.

---

Session History

Input history is persisted to .lim_history in the current working directory and survives across sessions.

---

Logging

Each session writes a numbered log file in $LIM_LOG_DIR/<N>. With LIM_DEBUG=1, a companion token-level trace is written to $LIM_LOG_DIR/<N>.tokens showing every token fed into and generated by the model, with labels like FEED SYSTEM_PROMPT_INIT, FEED USER_INPUT, etc. A TPS log is also written to $LIM_LOG_DIR/<N>.tps recording context token count and tokens-per-second at each speed diagnostic update.

---

License

See LICENSE for details.