diff --git a/README.md b/README.md index 9dea199..1e64d82 100644 --- a/README.md +++ b/README.md @@ -224,7 +224,7 @@ Common tasks are driven by `make` (see the [Makefile](Makefile) for the complete To build the node from source use: ```bash -make build-kdist +make kdist-build make build pip install dist/*.whl ``` diff --git a/docs/architecture.md b/docs/architecture.md index 88739ca..f8f1997 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -45,13 +45,13 @@ flowchart TB ### `server.py` — `StellarRpcServer` -`StellarRpcServer` is the long-running process around the semantics: a plain `http.server.HTTPServer` that keeps the HTTP socket open and the state files on disk across requests — the networking and persistence the one-shot K interpreter has no notion of. It receives JSON-RPC requests, uses `TransactionEncoder` to turn a transaction into a request envelope, hands the envelope to `NodeInterpreter`, and returns the `response.json` the semantics produced. It holds **no** ledger counter, receipt store, or response-formatting logic — those live in K. +`StellarRpcServer` is the long-running process around the semantics: a plain `http.server.HTTPServer` that keeps the HTTP socket open across requests — the networking the one-shot K interpreter has no notion of. It receives JSON-RPC requests, uses an `Encoder` to turn a transaction into a request envelope, hands the envelope to an `Interpreter`, and returns the `response.json` the semantics produced. It holds **no** ledger counter, receipt store, or response-formatting logic — those live in K — and it reaches the on-disk artifacts through a `Store` rather than touching paths itself. -`handle_rpc(method, params, id)` is the dispatch entry point and is usable without the HTTP layer (scripts, tests). +The three collaborators (`Encoder`, `Interpreter`, `Store`) are injected as the protocols declared in `interfaces.py`, not as their concrete classes, so a test can substitute fakes; `build_server` in `__main__.py` wires up the concretes. `handle_rpc(method, params, id)` is the dispatch entry point and is usable without the HTTP layer (scripts, tests). → **[Detailed documentation](server.md)** -The server implements eleven RPC methods — `getHealth`, `getNetwork`, `getLatestLedger`, `getVersionInfo`, `getFeeStats`, `sendTransaction`, `getTransaction`, `getTransactions`, `getLedgers`, `getLedgerEntries`, and the komet-specific `traceTransaction` extension — and the K semantics answer all of them. JSON-RPC framing (single calls, batch arrays, notifications) is handled in Python; the semantics see one request envelope per invocation. For `getLedgerEntries` the semantics' answer is an intermediate shape: K performs the state lookups, and `ledger_entries.py` translates between the base64 XDR wire format (`LedgerKey` in, `LedgerEntryData` out) and the JSON the semantics exchange, since K cannot parse or produce XDR. +The server implements thirteen RPC methods — `getHealth`, `getNetwork`, `getLatestLedger`, `getVersionInfo`, `getFeeStats`, `sendTransaction`, `simulateTransaction`, `getTransaction`, `getTransactions`, `getLedgers`, `getLedgerEntries`, `getEvents`, and the komet-specific `traceTransaction` extension — and the K semantics answer all of them. JSON-RPC framing (single calls, batch arrays, notifications) is handled in Python; the semantics see one request envelope per invocation. For `getLedgerEntries` the semantics' answer is an intermediate shape: K performs the state lookups, and `ledger_entries.py` translates between the base64 XDR wire format (`LedgerKey` in, `LedgerEntryData` out) and the JSON the semantics exchange, since K cannot parse or produce XDR. `sendTransaction` always returns `PENDING` and clients poll `getTransaction` for the result — matching the Stellar RPC async pattern even though the transaction executes synchronously. See [server.md](server.md) for details. @@ -73,6 +73,12 @@ The server implements eleven RPC methods — `getHealth`, `getNetwork`, `getLate --- +### `store.py` — `ChainStore` + +`ChainStore` owns the io-dir layout: it knows the paths, the file-naming conventions, and the JSON shapes, and it is the only component that reads or writes them. The server holds one and asks it for receipts, ledgers, events, and the ledger counter. It is the concrete `Store` the server depends on through the [io-dir table below](#the-io-dir). + +--- + ### `kdist/node.md` — K Semantics `node.md` is the K module compiled into the LLVM binary. It implements the whole RPC layer on the K side: it reads `request.json`, dispatches on the `method` field, reads and updates `metadata.json` and the per-transaction `receipts/` files, executes transaction steps via KASMER, and writes the JSON-RPC `response.json`. KASMER is the Komet execution harness whose `Step`s — `setAccount`, `deployContract`, `callTx`, `uploadWasm` — carry out the Soroban operations a transaction decodes into. @@ -89,7 +95,7 @@ All of the server's input and output artifacts live in one directory, the *io di |---|---|---|---| | `state.kore` | persistent | `NodeInterpreter` | the full K world-state configuration — accounts, contract code (including uploaded wasm `ModuleDecl`s), contract storage, ledger metadata — serialized in KORE. Read before each run and rewritten after a successful one. | | `metadata.json` | persistent | the K semantics | `{"latest_ledger": N}` — the server ledger counter, bumped by 1 per committed transaction. | -| `receipts/receipt_.json` | persistent | the semantics and the server (on success — the server rewrites the semantics' internal `returnValue` field into `resultXdr`/`resultMetaXdr`), or the server alone (on failure) | one stored receipt per transaction, keyed by tx hash, answering `getTransaction`. Each is `{status, ledger, createdAt, envelopeXdr, resultXdr, resultMetaXdr?}`. | +| `receipts/receipt_.json` | persistent | the semantics and the server (on success — the server rewrites the semantics' internal `returnValue` field into `resultXdr`/`resultMetaXdr`), or the server alone (on failure) | one stored receipt per transaction, keyed by tx hash, answering `getTransaction`. Each is `{status, applicationOrder, feeBump, envelopeXdr, resultXdr, ledger, createdAt, resultMetaXdr?}`. | | `traces/trace_.jsonl` | persistent | the semantics | one execution trace per transaction, keyed by tx hash — the instruction-level records, one JSON object per line. `traceTransaction` returns these records as a JSON array. | | `ledgers/ledger_.json` | persistent | the server (on success) | one record per closed ledger — `{sequence, txHash, closedAt, hash, headerXdr, metadataXdr}` — the ledger→transaction index behind `getTransactions` and `getLedgers`. Written in Python because the header artifacts are XDR, which K cannot construct. | | `requests/request_.json` | persistent | the server | an archive of each incoming JSON-RPC request, numbered by a monotonic counter, kept for debugging. | @@ -106,7 +112,7 @@ The world state stays in KORE (rather than a JSON snapshot) because an uploaded ```mermaid flowchart TB boot(["server start"]) --> exists{"state.kore exists?"} - exists -->|"no"| init["empty_config() builds the idle K config in KORE
write state.kore · seed metadata.json {latest_ledger: 0}
create receipts/ traces/ requests/ wasms/"] + exists -->|"no"| init["empty_config() builds the idle K config in KORE
write state.kore · seed metadata.json {latest_ledger: 0}
create receipts/ traces/ ledgers/ requests/ wasms/ events/"] exists -->|"yes"| reuse["use existing state.kore
seed metadata.json if missing · ensure artifact dirs exist"] init --> ready(["ready for requests"]) reuse --> ready diff --git a/docs/interpreter.md b/docs/interpreter.md index de9051d..a410b87 100644 --- a/docs/interpreter.md +++ b/docs/interpreter.md @@ -36,7 +36,7 @@ The result is the empty idle K configuration — no accounts, no contracts, no s The run happens in a throwaway empty directory on purpose. The idle configuration ends with empty ``/`` cells, which is exactly the precondition that makes the request-handling rule fire if a `request.json` is present. Running in an empty directory guarantees no stray `request.json` is picked up and dispatched into the configuration that is about to be saved as `state.kore`. -### `run(state_file, io_dir, request, program_steps=None)` +### `run(state_file, io_dir, request, program_steps=None, *, commit=True)` `run` is the main entry point. It runs a single RPC request envelope through the following steps: @@ -46,6 +46,8 @@ The run happens in a throwaway empty directory on purpose. The idle configuratio 4. Run the interpreter with its subprocess working directory set to `io_dir` (so the K file-system hooks resolve the relative paths `request.json`, `response.json`, `metadata.json`, and the `receipts/` and `traces/` files). The directory is set on the subprocess only — the server's own process never `chdir`s, so concurrent requests in other threads are unaffected. 5. If the semantics wrote `response.json`, persist the new configuration to `state.kore` and return the response text. If not, the transaction got stuck (failed) — leave `state.kore` unchanged and return `None`, so the caller can synthesise a failure response. +`commit` defaults to `True`. Pass `commit=False` to run a request without persisting its result: the response text is still returned, but the new configuration is discarded and `state.kore` is left unchanged even on success. `simulateTransaction` uses this to execute a dry run against the current world state without advancing the chain. + ### `_inject_program(pattern, steps)` — the wasm-upload path A wasm upload cannot be expressed as a JSON step, because the resulting `ModuleDecl` (the parsed Wasm AST from `wasm2kast`) has no JSON form. Instead the steps are injected directly into the `` cell of the already-parsed configuration, so KASMER runs them before the request is dispatched. diff --git a/docs/node-semantics.md b/docs/node-semantics.md index 41397f4..6a154cc 100644 --- a/docs/node-semantics.md +++ b/docs/node-semantics.md @@ -188,7 +188,7 @@ Tracing is always on. Before running the steps, `#enableTrace` clears the transa **Trace format** (one JSON record per line): ```json -{"pos": 597, "instr": ["local.get", 0], "stack": [["i64", 4]], "locals": {"0": ["i64", 4]}} +{"pos": 597, "instr": ["local.get", 0], "stack": [["i64", 4]], "locals": {"0": ["i64", 4]}, "mem": null} ``` | Field | Description | @@ -197,6 +197,9 @@ Tracing is always on. Before running the steps, `#enableTrace` clears the transa | `instr` | Instruction name and operands as a JSON array | | `stack` | Value stack at instruction entry, as `[type, value]` pairs | | `locals` | Local variable bindings, keyed by index, as `[type, value]` pairs | +| `mem` | Linear memory as a list of `{addr, bytes}` runs, emitted only when memory changed since the previous record and `null` otherwise (reuse the most recent snapshot) | + +Instruction records are one of several trace record kinds (`callContract`, `hostCall`, `contractData`, and `endWasm` are the others); see the [Trace a transaction](../README.md#trace-a-transaction) section of the README for all five. --- diff --git a/docs/notes.md b/docs/notes.md index 823a23b..8365b7e 100644 --- a/docs/notes.md +++ b/docs/notes.md @@ -43,7 +43,7 @@ The tests do not yet cover `bytes` / `address` SCVal arguments or `SCVec` / `SCM - `sendTransaction`'s `errorResultXdr` is always a generic `txMALFORMED` result (no per-cause codes such as `txBAD_SEQ` — sequence numbers, fees, and signatures are not modelled), and `diagnosticEventsXdr` is never populated (both fields are optional in the spec). `TRY_AGAIN_LATER` is never returned by design: it signals mempool backpressure, which cannot arise in a synchronous node without a mempool. - `SCVec` / `SCMap` contract arguments are not yet encoded. - The `xdrFormat` parameter is accepted on `getTransaction`, `sendTransaction`, `getTransactions`, `getLedgers`, `getLedgerEntries`, and `getEvents`, but only the default `'base64'` is supported; `'json'` is rejected with `-32602`. -- `simulateTransaction` only simulates invoke-contract host functions (not uploads/deploys); `auth` is always empty, `minResourceFee`/`transactionData` are synthetic constants, and `events`/`restorePreamble`/`stateChanges` and the `resourceConfig`/`authMode`/`xdrFormat` parameters are not supported. `ScMap` return values are not yet encoded. +- `simulateTransaction` only simulates invoke-contract host functions (not uploads/deploys); `auth` is always empty, `minResourceFee`/`transactionData` are synthetic constants, and `events`/`restorePreamble`/`stateChanges` and the `resourceConfig`/`authMode`/`xdrFormat` parameters are not supported. - `getEvents` serves contract events only: `system` events are never emitted (the semantics have no source of them), and events whose topics/data have no staging representation (maps, errors, 256-bit ints) are dropped with a warning. - TTL/footprint operations are not implemented. - `getFeeStats` reports constant distributions (there is no fee market); only its `latestLedger` is live. diff --git a/docs/server.md b/docs/server.md index 2938653..f325b1d 100644 --- a/docs/server.md +++ b/docs/server.md @@ -1,6 +1,8 @@ # `server.py` — `StellarRpcServer` -`StellarRpcServer` exposes the [Stellar RPC API](https://developers.stellar.org/docs/data/apis/rpc) over HTTP/JSON-RPC. Its job is to make the K semantics usable as a server. The compiled semantics are a one-shot interpreter — one process invocation per request, with no networking and no memory between runs — and `StellarRpcServer` is the long-running process wrapped around them: it keeps the HTTP socket open and the state files on disk, decodes the XDR envelope (via `TransactionEncoder`), runs each request through the semantics (via `NodeInterpreter`), and returns whatever `response.json` the semantics produced. All RPC dispatch, receipt bookkeeping, ledger accounting, and response formatting happen in K — the server holds none of that state itself. +`StellarRpcServer` exposes the [Stellar RPC API](https://developers.stellar.org/docs/data/apis/rpc) over HTTP/JSON-RPC. Its job is to make the K semantics usable as a server. The compiled semantics are a one-shot interpreter — one process invocation per request, with no networking and no memory between runs — and `StellarRpcServer` is the long-running process wrapped around them: it keeps the HTTP socket open, decodes the XDR envelope, runs each request through the semantics, and returns whatever `response.json` the semantics produced. All RPC dispatch, receipt bookkeeping, ledger accounting, and response formatting happen in K — the server holds none of that state itself. + +The server depends on three collaborators through the protocols in `interfaces.py` rather than on their concrete classes: an `Encoder` (the XDR → request-envelope decoder, `TransactionEncoder`), an `Interpreter` (the K runner, `NodeInterpreter`), and a `Store` (the io-dir reader/writer, `ChainStore`). A composition root wires the concrete classes together — see `build_server` in `__main__.py` — so a test can drive the server with fakes. --- @@ -8,17 +10,15 @@ ```python class StellarRpcServer: - interpreter: NodeInterpreter # the K runner - encoder: TransactionEncoder # the XDR → request-envelope decoder - io_dir: Path # directory holding every artifact - state_file: Path # io_dir / 'state.kore' - receipts_dir: Path # io_dir / 'receipts' — receipt_.json per transaction - traces_dir: Path # io_dir / 'traces' — trace_.jsonl per transaction - ledgers_dir: Path # io_dir / 'ledgers' — ledger_.json per closed ledger - requests_dir: Path # io_dir / 'requests' — request_.json archive - events_dir: Path # io_dir / 'events' — events_.json per ledger + interpreter: Interpreter # the K runner (NodeInterpreter) + encoder: Encoder # the XDR → request-envelope decoder (TransactionEncoder) + store: Store # the io-dir reader/writer (ChainStore) + io_dir: Path # the io-dir root, taken from the store + state_file: Path # io_dir / 'state.kore', taken from the store ``` +The per-item artifact directories (`receipts/`, `traces/`, `ledgers/`, `requests/`, `wasms/`, `events/`) belong to the `Store`, which owns the io-dir layout; the server reaches them through the store's methods rather than holding their paths. See [`ChainStore`](../src/komet_node/store.py) for the full layout. + The server is a plain `http.server.HTTPServer` (not pyk's `JsonRpcServer`). A `BaseHTTPRequestHandler` reads each POST body and calls `_handle`, which parses the JSON-RPC frame and delegates to `handle_rpc`. ### JSON-RPC framing @@ -40,7 +40,7 @@ Framing happens entirely in Python (`_handle` / `_handle_batch` / `_handle_singl `handle_rpc` is the dispatch entry point; it returns the JSON-RPC response envelope as a string. You can call it **without** the HTTP layer, which is convenient for scripts and tests: ```python -server = StellarRpcServer(io_dir=Path('out')) +server = build_server(io_dir=Path('out')) server.handle_rpc('sendTransaction', {'transaction': xdr}) ``` @@ -51,23 +51,23 @@ For `sendTransaction` it builds the request envelope with `encoder.build_tx_requ ## Startup ```python -server = StellarRpcServer( - host='localhost', - port=8000, +server = build_server( io_dir=Path('out'), # omit for a fresh temporary directory network_passphrase=Network.TESTNET_NETWORK_PASSPHRASE, + host='localhost', + port=8000, ) server.serve() ``` -`io_dir` defaults to `None`, in which case the server creates a fresh temporary directory (`tempfile.mkdtemp`) and runs against that — a throwaway chain that starts empty on every launch and leaves the working directory untouched. Pass an explicit `io_dir` to keep the state in a known place. +`build_server` (in `__main__.py`) is the composition root: it resolves the io-dir, constructs the `NodeInterpreter`, `TransactionEncoder`, and `ChainStore`, and injects them into the server. `io_dir` defaults to `None`, in which case `build_server` creates a fresh temporary directory (`tempfile.mkdtemp`) and runs against that — a throwaway chain that starts empty on every launch and leaves the working directory untouched. Pass an explicit `io_dir` to keep the state in a known place. -At construction the server prepares the *io dir*, where `state.kore` lives at `io_dir / 'state.kore'`: +At construction the `ChainStore` prepares the *io dir*, where `state.kore` lives at `io_dir / 'state.kore'`: -- **`state.kore` absent** — `interpreter.empty_config()` produces the initial idle K configuration (a blank-slate state with no accounts, contracts, or storage) and writes it; `metadata.json` is seeded with `{"latest_ledger": 0}`. +- **`state.kore` absent** — `interpreter.empty_config()` produces the initial idle K configuration (a blank-slate state with no accounts, contracts, or storage) and the store writes it; `metadata.json` is seeded with `{"latest_ledger": 0}`. - **`state.kore` present** — it is used as-is, and `metadata.json` is seeded only if missing. This lets you resume a previous session (ledger counter and stored receipts included) or start against a pre-built state. -In both cases the server creates the `receipts/`, `traces/`, `ledgers/`, `requests/`, `wasms/`, and `events/` directories if they do not already exist, because the K file-system hooks write into them but cannot create them. +In both cases the store creates the `receipts/`, `traces/`, `ledgers/`, `requests/`, `wasms/`, and `events/` directories if they do not already exist, because the K file-system hooks write into them but cannot create them. Once the socket is bound, `serve` logs three lines to stderr: whether it is starting from a fresh state (an empty io-dir) or resuming an existing one (with the latest ledger), the io-dir path, and the listening address. Instruction tracing is always on, so every transaction the semantics run produces a trace. (Tracing only produces records for contract invocations.) @@ -80,7 +80,7 @@ Once the socket is bound, `serve` logs three lines to stderr: whether it is star ``` startup (state.kore absent): → empty_config() → state.kore ; metadata.json {latest_ledger:0} - → create receipts/ traces/ requests/ wasms/ + → create receipts/ traces/ ledgers/ requests/ wasms/ events/ per successful transaction: → the semantics run the steps (trace → traces/trace_.jsonl), @@ -132,7 +132,7 @@ The node retains every ledger since genesis, so `oldestLedger` is always 0 and t `getVersionInfo` reports the komet-node package version as the RPC server version and the komet package (the K semantics of Soroban executing the transactions) as the "Captive Core". komet-node is a Python package, so no commit hash or build timestamp is baked in at build time — those fields are all-zeros / epoch placeholders with the spec-correct types. `protocolVersion` is a JSON number. ```json -{ "version": "0.1.0", "commitHash": "0000...0000", "buildTimestamp": "1970-01-01T00:00:00", "captiveCoreVersion": "komet 0.1.79 (K semantics of Soroban)", "protocolVersion": 22 } +{ "version": "0.1.0", "commitHash": "0000...0000", "buildTimestamp": "1970-01-01T00:00:00", "captiveCoreVersion": "komet 0.1.86 (K semantics of Soroban)", "protocolVersion": 22 } ``` ### `getFeeStats` @@ -189,7 +189,7 @@ Failures are reported in the result body, matching real stellar-rpc; only an und ```json [ - {"pos": 3, "instr": ["const", "i32", 1048576], "stack": [], "locals": {}} + {"pos": 3, "instr": ["const", "i32", 1048576], "stack": [], "locals": {}, "mem": null} ] ``` diff --git a/docs/transaction.md b/docs/transaction.md index 83fdf5b..e875e50 100644 --- a/docs/transaction.md +++ b/docs/transaction.md @@ -17,7 +17,7 @@ The encoder is stateless apart from this one configuration value; it holds no le ## `build_tx_request(method, rpc_id, transaction_xdr, now)` -`build_tx_request` is the entry point. It decodes the XDR envelope and returns a `(request, program_steps)` pair: +`build_tx_request` is the `sendTransaction` entry point. It decodes the XDR envelope and returns a `(request, program_steps, uploaded_wasms)` triple: ```python request = { @@ -30,11 +30,13 @@ request = { } ``` -- For the common case, every operation is encoded as a JSON step and `program_steps` is `None`. -- For a wasm-upload transaction, the operations cannot be expressed as JSON (the `ModuleDecl` has no JSON form), so `steps` is `[]` and `program_steps` carries the kasmer `uploadWasm` step for the interpreter to splice into the `` cell. Soroban allows only one host-function operation per transaction, so such a transaction is exactly one upload op. +- For the common case, every operation is encoded as a JSON step, and both `program_steps` and `uploaded_wasms` are empty (`None` and `{}`). +- For a wasm-upload transaction, the operations cannot be expressed as JSON (the `ModuleDecl` has no JSON form), so `steps` is `[]` and `program_steps` carries the kasmer `uploadWasm` step for the interpreter to splice into the `` cell. Soroban allows only one host-function operation per transaction, so such a transaction is exactly one upload op. `uploaded_wasms` then maps the module's hex sha256 to its raw bytes, which the server stores under `wasms/` so `getLedgerEntries` can return the original bytes for `CONTRACT_CODE` entries (the K state keeps modules parsed). `now` is read from the wall clock here, in Python, and passed through the envelope because K has no clock; the semantics use it to fill `createdAt` / `latestLedgerCloseTime`. +`build_simulate_request(rpc_id, transaction_xdr, now)` is the `simulateTransaction` counterpart. It decodes the single `InvokeHostFunction` operation into a `steps` array the same way, but returns only the request envelope (no `txHash` or `envelopeXdr`, since a dry run is never stored) and raises `SimulationRejected` for an operation that cannot be simulated — a wasm upload or contract deploy. + --- ## JSON step encoding