Skip to content
SaulDoesCodePublic

About

a means to an-end

Topics

Resources

Security policy

Stars

3 stars

Watchers

1 watching

Forks

Latest commit

 

History

453 Commits

Folders and files

Repository files navigation

anend

anend is a memory palace for short texts. You type a word or phrase and it is stored as an entry with its gematria sums. You find it again by its numbers, its words, its tags or the lists it's in, and you can fly through your memories laid out in space. It continues weum, whose numbers it reproduces exactly.

It comes as:

  • the app (app/, Dioxus): a desktop app, and the same app built to WebAssembly for the browser;
  • the hub (hub/, Salvo): a self-hosted server that runs the app for its members and serves it at /app/, and a place to share saves, like a game's workshop;
  • the core (core/): what both share. That includes the entry graph store, gematria, the app's API, and a deterministic transformer engine for classifiers and generators.
memory space
the memory view, with its menu the space
buckets notes
sums as buckets notes
reader at night a list in the browser
the reader, night colours a list's page (web)
stats filters
stats (web) filters (web)
a chain a bent block
a chain of texts under its title (web) a block bent by a ripple and a fold (web)
textures lights
a texture recipe on a block, a colour on one face, a sky (web) blocks and spheres under baked lights (web)
the spec builder a staging zone
a spec model in the builder texts spreading inside a staging zone (web)
switches, a gate and a lamp the prefabs window
two switches through an AND gate light a lamp, while the room plays (web) a copy of a prefab selected, with its menu (desktop)
superflight placing by hand
superflight: the texts flown through, ready to become an entry (web) placing a prefab by hand: red where it would overlap (web)
an s program asking a classifier the step debugger
an s program asking a classifier, its output beside each line (web) an s program in the editor (web)

The views

Ctrl+1 … Ctrl+9 switch views, Ctrl+, opens settings, and Esc closes things. Every view has the command bar at the bottom, and each view keeps its own text there.

memory (Ctrl+1) weum's memory view. Enter on text gsearches its sum and saves it when it's new. Enter on a number gsearches that bucket. Ctrl+Enter saves, Shift+Enter shows a key, Delete forgets. Enlist List Delist Unlist work on lists. r / o / f pick the bucket (reversed, ordinal, frequency), and ← → gsearch the term's sums. Hover shows the gsum line, and texts that share a sum are underlined in its colour. The circles at the left edge are the buckets. Shift+click (or Alt+click, or a box dragged on the field) selects several texts to put in a list or tag; Ctrl+Shift+click picks the list. Right-click opens the menu: edit, tag, select, list, candidate, key, forget.
space (Ctrl+2) memory texts floating in a room, laid out by their sums. On the desktop the room is Rust, drawn with wgpu with no JavaScript (render/; under the page on Linux, shown in it on macOS and Windows); on the web it is three.js for now. Clicking a text puts its triangle in the top bar: sides s (ordinal), r (reversed) and ⌊√(s·r)⌋. The arrow keys glide to the nearest text in that direction, and its card opens on the right. It's also where you build: copies that inherit, have their own text or are detached; each placement's size; chains of texts that hang under their title; saved scenes, combined with overlaps settled; and blocks showing texts, pictures, colours or texture recipes (cropped, tiled, tinted, with variants; on a whole block or one face), which move, resize, turn and bend or become spheres; a sky; lights baked onto the blocks; and staging zones whose texts spread among themselves. Blocks act on each other: each can listen for clicks, other blocks' rays, texts entering its zone, timers or signals, run an s program, cast a ray, play keyframe animations and sounds; blocks group into prefabs placed as copies, and starter prefabs (a switch, logic gates, a lamp, a door) show it working. Shift+click or a box selects many things at once to move, turn or delete together. Besides flying you can walk (V: gravity, collisions, run, sneak, crouch, lean, mantle), swing on a grapple (G), place prefabs by hand where you look, and superflight (B) through texts to collect them, in order, into an entry, a chain or a list.
lists (Ctrl+3) every list as a tile. A list's page has an editable name and members, and figures for the whole list: its sums, means, gsum and pfR chain.
reader (Ctrl+4) everything that matches, in pages. Short texts appear as a tiny grid, and notes appear as markdown. Besides its own three views (everything, notes, texts), it shows views written as s programs: a pipe or a search, an order, headings, a layout and pipes' effects.
notes (Ctrl+5) markdown notes, in a list by date, with tags and a read mode. A note can be attached to any entry. Typing stays quick in a note of any length. A note tagged s is a program in s (below), and the notes view is its code editor.
tags (Ctrl+6) tags on any entry: pick tags (all or any) and see the entries that have them.
stats (Ctrl+7) figures for the whole save
models (Ctrl+8) classifiers (learned from lists, trained and run here) and spec models: models described as data (what they read, their heads, what they learn from, critics for a writer), made in a builder with a control for every part or as JSON, then trained, run and, for a writer, asked to write. Tag sets are word types of your own (a language's, a conlang checker's right and wrong), learned from words you mark. Taxonomies put classifiers in a tree: each answer picks the next classifier, and every step can be marked right or wrong, which each node counts and its classifier learns from.
filters (Ctrl+9) words, a sum range, written between two dates, a tag, a kind. Each filter is kept by name in the save. It's also the pipe editor: a filter with effects (a colour, a classifier's colours, an underline, a label, a size, a glow, gathering in the space) shown in the memory view and the space while it's on, stacked with other pipes, and actions (a list, a tag, a program on each) done when asked.

The top bar holds the view icons, the search line with its triangle, undo (Ctrl+Z) and settings. Ctrl+= / Ctrl+− zoom and 0 resets it.

Settings are a dialog with folding sections:

  • colours (palettes, or your own) and the text size;
  • the memory view's sizes and spacing, and the space's room colour;
  • zen: the bars fade while you rest, and the top bar can hide until the pointer reaches the top edge;
  • keys, and the hub (a hub's address, to browse and share);
  • saves: open, new, copy, delete, backup, and import from weum.

Programs

A save can be scripted in s, the project's own language (docs/s.md). A program is a note tagged s: its title names it, its body is the code. The notes view is its editor: the text stands right-aligned (each line's acting word at the right), coloured by kind of word, with an outline, Ctrl+click to a definition, completion, ▶ to run it with each line's output beside it, and a step debugger showing the stack and the values.

A program reaches the rest of the save through two library modules of anend's own. With std/anend.s., op calls any of the app's ops (docs/app-api.md) and json, #json and obj read and build their JSON. With std/learn.s., classifiers and models are words whose answers are s values:

std/learn.s.
{c is the top class, c/p its probability} //
{things} {a ripe mango} class
{~c~ (~c/p~)} >
{found lists what the search finds} //
{mango} search
found all {, } l/
{found: ~all~} >

With the "things" starter classifier made under the name things, that prints fruit, stones and animals: fruit (0.888…) and the texts with "mango" in them.

Programs run from the notes view, from the bar anywhere (Run <title> [arguments], the output as a toast), from blocks in the space (a block's wiring runs a program when it's clicked, hit by a ray or signalled), and from a shell with anend-app run (below). On a shared hub a run takes at most 2 million steps, and the hub's limits apply to what it asks.

The taxonomy engine

The classifiers are meant to work together rather than one at a time (docs/architecture.md):

  • Taxonomies put classifiers in a tree: each answer decides which classifier reads the text next ("English or not?" before "which kind of thing?"). Every step of a routing can be marked right or wrong; each node counts its marks beside its classifier's held-out accuracy, and every mark teaches that classifier.
  • Pipes take the texts a filter lets through (words, sums, lists, tags, what classifiers say) and give them effects: a colour, a classifier's class colours, an underline, a label, a size, a glow, gathering in the space. Turned on, a pipe's effects show in the memory view and the space, stacked with other pipes'; nothing is written onto the texts. Its actions (a list, a tag, a program on each) run when asked.
  • Views are programs: an s program that loads std/view.s. says where a view's texts come from (a pipe, or a search), their order, headings and layout, and the reader shows it beside its own views, which are presets of the same thing.
  • Tag sets are word types of your own, learned from words you mark: a language's (a set can claim the texts a language classifier picks out), or a checker for a language of your own that flags wrong words with the nearest right ones.
a taxonomy a pipe's effects in the space
a taxonomy: English or not, then which kind of thing; marks per node (desktop) a pipe's colour, label, size and gathering in the space (desktop)
a view program a tag set
a view of one's own in the reader: a pipe's texts as a list (web) an Afrikaans tag set, learned from marked words (web)

Saves

A save is one file, an entry graph in redb. Every text, note, tag, list and setting is an entry with timestamps. Lists and tags are edges between entries. Identical texts are stored once, and an entry can borrow another's text. docs/format.md describes the format. The desktop keeps saves in <data dir>/anend/saves/ (~/.local/share/anend/saves on Linux), and starts with main.

Backup writes texts.json (every text) and other.json (everything else) into the save's backups/ folder. Import from weum reads weum's all-of-it.json (made by memory.backup in weum), and the texts keep their keys. Saves from older anend versions aren't read.

There's no limit on how long a text can be: one over 64 KiB is stored in pieces, and the views show long texts piece by piece. The limits a save does set (how much a call reads, the spelling check's size, and so on) are its own settings, under "limits of this save".

Gematria modes are ordinal (a=1 … z=26), reversed (a=26 … z=1) and frequency (the letters sorted, so anagrams share a bucket). Byte sums (bytes valued 1–256) are a separate measure, in core/src/measure.rs.

Building and running

# the desktop app (Linux needs libwebkit2gtk-4.1-dev, libxdo-dev and libasound2-dev)
cargo run --release -p anend-app

# the app for the browser: needs the wasm32-unknown-unknown target and wasm-bindgen-cli 0.2.129
app/web/build.sh            # writes app/web/dist

# a hub on your own machine: no logins, the app at http://127.0.0.1:8080/app/
cargo run --release -p anend-hub -- --local

# an s program on a save, headless: a file, or the save's program (a note tagged s) by
# title; stdin, when it isn't a terminal, is read to its end as the program's stdin
# (give it </dev/null when there's none); its output comes out, and its exit code
cargo run --release -p anend-app -- run tally --save ~/.local/share/anend/saves/main.redb

On Linux the app turns off WebKitGTK's DMA-BUF renderer, which leaves a white window ("Failed to create GBM buffer") on some drivers. If the window is still blank, try WEBKIT_DISABLE_COMPOSITING_MODE=1 anend-app. To keep DMA-BUF on, set WEBKIT_DISABLE_DMABUF_RENDERER=0.

Build the web app before the hub, so the hub can serve it at /app/. Without it, /app/ shows a page saying to run app/web/build.sh.

Before a change goes in, run the checks: cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace, and the wasm build (cargo build -p anend-app --target wasm32-unknown-unknown --no-default-features --features web). .github/workflows/ci.yml runs the same checks, and release.yml builds the desktop app and the hub (with the web app inside) for Linux and Windows when a v* tag is pushed.

The hub

the hub

cargo run --release -p anend-hub -- --admin yourname

This prints a one-time login link. The hub keeps everything in ./hub-data, and listens on 127.0.0.1:8080.

  • --mode picks what it serves:
    • hub: the sharing site;
    • app: the web app for members, each with their own saves;
    • both: both, the default.
  • --join sets how people get in: invite, apply or open.
  • Visibility: each save is private, unlisted or public.

Members share saves, star and download each other's, and earn points, levels and badges when others use what they made. The admin panel reviews saves and people. docs/deploy.md covers running one for real: modes, joining, TLS, backups, Docker, and running it as a paid service. docs/security.md covers the security model.

The web app and the desktop make the same calls (the app API), over a WebSocket at /app/ws or POST /app/api/{op}. Anything that speaks JSON can drive a save. docs/hub-api.md covers the sharing API.

Models

core/src/classify is a transformer engine written from scratch: RMSNorm, SwiGLU, RoPE, grouped key/value heads. The same inputs give the same output bits on any machine and thread count. Models are described by a declarative spec (docs/models.md):

  • inputs: bytes, numbers, pictures, and optionally gematria channels (off unless you choose them);
  • heads: classify, regress, and generate;
  • data queries, and critics: classifiers that steer generation, by reranking, guidance or reinforcement.

Pretrained models: Llama-family checkpoints (Llama, Mistral, Qwen2/3, SmolLM) load from safetensors and a tokenizer.json. Use "from": {"pretrained": "<dir>"} in a spec, or see core/examples/pretrained.rs.

The GPU: a backend (wgpu, the cargo feature gpu, on in the desktop app and the hub) trains and reads texts on the GPU. Classifiers read on it (trying, search, indexing, buckets, colouring, filters) with every text's windows packed into a few submissions; the settings' GPU section picks auto, the CPU, the GPU or the GPU fast. In exact precision it gives the CPU's bits on a device whose kernels the probe finds exact (this was checked on a software GPU, llvmpipe, only). write a GPU report in the settings (gpu.report) writes a JSON file about your GPU: the adapter, driver, limits, the probe of every kernel, and training and reading timings next to the CPU. core/examples/gpu_bench.rs measures larger models.

Starter classifiers (core/src/starters.rs) have texts whose classes are known for certain (made by rules, or public-domain books), so each is scored on held-out texts: cargo run --release -p anend-core --example starters prints the table.

core/src/classify/STATUS.md has the details and what's measured.

Layout

core/src/graph/ the entry graph store: records, the 5-bit codec with references and calls, posting lists, edges, queries
core/src/ops/ the app API over a save and a folder of saves, merge and forge, shared by the desktop and the hub
core/src/classify/ the transformer engine, the spec, generation, pretrained import, the GPU backend
core/src/measure.rs byte sums
app/src/ the Dioxus app: shell.rs (bars, keys, menu), views/, backend/ (local.rs the graph in-process, hub.rs the WebSocket), gem.rs (weum's numbers), native.rs (the room's picture under the page on Linux), picture.rs (in the page on macOS and Windows), space.js (the three.js room on the web)
render/ the room in Rust (anend-render): the wgpu renderer, the room itself (room.rs, room/: the spread, building, wires, sounds, walking), blocks and their tiles
app/web/ the web build: build.sh, the page, its loader
hub/src/ the hub: modes, accounts, saves, sharing, the game, the app's API (app.rs, appweb.rs)
docs/ the formats, the APIs, deployment, security, models, and next.md (the plan and status)
docs/weum-reference/ weum's own code, the reference for its numbers

docs/guide.md is the guide for people using the app: every view, the command bar, lists, classifiers and spec models, settings and the hub.

About

a means to an-end

Topics

Resources

Security policy

Stars

3 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages