diff --git a/changes/unreleased/relationship-tables-uncharged.fixed.md b/changes/unreleased/relationship-tables-uncharged.fixed.md new file mode 100644 index 000000000..bbaa23aaa --- /dev/null +++ b/changes/unreleased/relationship-tables-uncharged.fixed.md @@ -0,0 +1 @@ +- **Relationship queries no longer spend their visit budget building edge tables.** `RelatedElements`, `WhereRelated` and relationship-derived columns build the edge table of a relationship kind by scanning every declaration in the workspace, and each declaration scanned was charged to the query's visit budget — so on a model past ~100,000 declarations every relationship query failed with `visit-budget` before traversing anything (a migrated TMT requirements-mapping document, 101,014 declarations, needed about 200 visits for its rows). The scan is a fixed cost of the model, not of the query: it is now memoized per model in `queryexec.Context.Related`, shared by every query a document (or a linked set of documents) evaluates, and left uncharged; the budget still bounds the traversal itself, paying one visit per element reached, and `visit-budget` is still the typed failure when that is exceeded. diff --git a/docs/manual/query-cookbook.md b/docs/manual/query-cookbook.md index ae773107b..8f645d75f 100644 --- a/docs/manual/query-cookbook.md +++ b/docs/manual/query-cookbook.md @@ -783,7 +783,10 @@ Direction is from the relationship's own point of view — `outgoing` follows it as declared, `incoming` follows it backwards. Traversal is breadth-first to `maxDepth` (unbounded when omitted or `null`), deduplicated, in declaration order, and bounded by a visit budget so a pathological model -terminates with a typed error rather than hanging. +terminates with a typed error rather than hanging. The budget pays only for +the elements reached: the edge table a relationship kind reads is built once +per model (again after an edit), from every declaration in the workspace, and is not charged to it — +so a matrix over a large model costs what its rows relate to, not the model's size. ### Connections diff --git a/docs/project/spec-compliance.md b/docs/project/spec-compliance.md index d9e12653f..af7cd68ab 100644 --- a/docs/project/spec-compliance.md +++ b/docs/project/spec-compliance.md @@ -2995,7 +2995,7 @@ It does not mutate the workspace or re-derive a parallel semantic representation | Projection produces ordered typed cells, and stable ordering applies explicit ascending/descending, missing-value and multiple-value policies without losing row/cell alignment | `queryexec/value.go`; `queryexec/operations.go` `executor.evaluateProject`, `executor.evaluateOrderBy` | `queryexec/execute_test.go:TestExecuteTraversalFilteringOrderingAndProjection`, `:TestExecuteOrderPoliciesAndProjectedCellAlignment` | ✅ Implemented | | Computed columns evaluate their expression once per result row against the row element's constant feature values, appended after declared properties in declaration order; integer arithmetic stays integer, mixed integer/real widens to real, `+` concatenates strings, `??` supplies defaults for absent values, and a failing expression — a multi-valued operand, an operand type mismatch, division by zero — fails the query with a typed error naming the query, column and row rather than yielding an empty cell, so `??` is the one deliberate absent-value mechanism; computed names are visible to `OrderBy` (ordering by the projected cells) and downstream grouping | `queryexec/computed.go` `executor.computedColumns`, `executor.evaluateColumnCell`, `executor.applyColumnOperator`; `queryexec/operations.go` `executor.evaluateProject`, `executor.evaluateOrderBy`; `queryexec/errors.go` `ErrorColumnOperand`, `ErrorColumnOperandType`, `ErrorColumnDivisionByZero` | `queryexec/computed_test.go:TestExecuteComputedArithmeticAndConcatenation`, `:TestExecuteOrdersAndGroupsByComputedColumns`, `:TestExecuteComputedColumnsAreDeterministic`, `:TestExecuteComputedResultsAreImmutableToCallers`, `:TestExecuteComputedColumnFailuresAreTyped` | ✅ Implemented | | Traversal consumes an explicit visit budget, and result rows, cells, columns and execution failures retain query or model provenance | `queryexec/execute.go` `Options`; `queryexec/operations.go` `executor.consumeVisit`; `queryexec/value.go`; `queryexec/errors.go` | `queryexec/execute_test.go:TestExecuteDescendantsBreadthFirstAndBounded`, `:TestExecuteTraversalFilteringOrderingAndProjection`, `:TestExecuteReportsUnknownAndUnevaluableFeatures` | ✅ Implemented | -| Named-relationship traversal (`RelatedElements`) follows specialization, subsetting, redefinition, typing, connection (connection/connector/interface usages), allocation, satisfaction and verification edges, outgoing or incoming, breadth-first to a bounded depth; edges are resolved through the semantic model and resolver, results keep declaration order, deduplicate by semantic identity, consume the shared visit budget and retain provenance, and an unknown relationship kind or direction is a typed failure | `queryexec/related.go` `executor.evaluateRelated`, `executor.relatedNeighbors`, `executor.relationshipEdges`; `symbols/index.go` `Index.WorkspaceDocuments` | `queryexec/related_test.go:TestExecuteRelatedLineageBothDirectionsAndDepth`, `:TestExecuteRelatedConnectionsAllocationsAndAssertions`, `:TestExecuteRelatedSeedsAreDeduplicatedBySemanticIdentity`, `:TestExecuteRelatedConsumesTheVisitBudget`, `:TestExecuteRelatedComposesWithFiltersProjectionAndInvocation`, `queryexec/execute_test.go:TestExecuteRejectsInvalidBindingsAndRelationshipTraversal` | ✅ Implemented | +| Named-relationship traversal (`RelatedElements`) follows specialization, subsetting, redefinition, typing, connection (connection/connector/interface usages), allocation, satisfaction and verification edges, outgoing or incoming, breadth-first to a bounded depth; edges are resolved through the semantic model and resolver into per-kind edge tables built once per model (memoized in `Context.Related` across a document's queries, rebuilt after the index takes an edit, not charged to the visit budget), results keep declaration order, deduplicate by semantic identity, charge the shared visit budget per element reached and retain provenance, and an unknown relationship kind or direction is a typed failure | `queryexec/related.go` `executor.evaluateRelated`, `executor.relatedNeighbors`, `executor.relationshipEdges`; `symbols/index.go` `Index.WorkspaceDocuments` | `queryexec/related_test.go:TestExecuteRelatedLineageBothDirectionsAndDepth`, `:TestExecuteRelatedConnectionsAllocationsAndAssertions`, `:TestExecuteRelatedSeedsAreDeduplicatedBySemanticIdentity`, `:TestExecuteRelatedConsumesTheVisitBudget`, `:TestExecuteRelatedLeavesTableConstructionUncharged`, `:TestExecuteSharesRelationshipTablesThroughTheContext`, `:TestExecuteRebuildsRelationshipTablesAfterAnIndexEdit`, `:TestExecuteRelatedComposesWithFiltersProjectionAndInvocation`, `queryexec/execute_test.go:TestExecuteRejectsInvalidBindingsAndRelationshipTraversal` | ✅ Implemented | | A named query invokes another compiled definition through the plan's dependency-ordered definitions; invoked bindings and declared results are re-validated at the invoked query, and rows, projected columns/cells, declaration order, semantic identity and provenance survive the invocation boundary, including nested and empty results | `queryexec/execute.go` `executor.evaluateInvoke`, `executor.bind`, `executor.validateResult` | `queryexec/execute_test.go:TestExecuteInvokesNamedQueriesPreservingOrderAndIdentity`, `:TestExecuteNestedInvocationPreservesProjectedColumnsAndCells`, `:TestExecuteInvocationPropagatesEmptyResults`, `:TestExecuteInvocationBindsProjectedArgumentRowElements`, `:TestExecuteValidatesInvokedResultsAtTheirDeclaration`, `:TestExecuteInvocationBindingMismatchFailsAtPlanning` | ✅ Implemented | | Invocation is bounded even for malformed or externally constructed plans: a target missing from the plan, a re-entered query, an exhausted invocation depth, and an exhausted total invocation count are distinct typed failures, and invoked traversal shares the caller's visit budget | `queryexec/execute.go` `executor.evaluateInvoke`, `Options.InvocationDepth`, `Options.InvocationBudget`; `queryexec/errors.go` `ErrorUnknownInvocation`, `ErrorInvocationCycle`, `ErrorInvocationDepth`, `ErrorInvocationBudget` | `queryexec/execute_test.go:TestExecuteInvocationDepthIsBounded`, `:TestExecuteInvocationCountIsBounded`, `:TestExecuteInvocationSharesTheVisitBudget` | ✅ Implemented | | `%run-query [

=...]` compiles the named query, binds its entry parameters from prompt expressions or element names, executes it and prints its ordered rows and projected cells; typed execution failures are reported as errors | `repl/docquery.go` `Session.RunDocumentQuery`, `Session.runDocumentQuery`, `queryValues`, `renderRowSet`; `repl/meta.go` (command table, dispatch) | `repl/docquery_test.go:TestRunQueryProjectsOrderedRows`, `:TestRunQueryBindingExpressions`, `:TestRunQuerySurfacesTypedExecutionFailures`, `:TestRunQueryRejectsNonQueryDefinition` | ✅ Implemented | @@ -3015,7 +3015,7 @@ It does not mutate the workspace or re-derive a parallel semantic representation | A terminated state machine holds no active configuration, so `States` answers no row for its object and `InState` matches nothing, while a machine that reached `done` reports it as the final state; a `States` or `Events` `source` bound to an object the run destroyed is refused with `ErrorObjectDestroyed` naming the object and the activation mark of its destruction, rather than answering a stale row or surfacing an unevaluable-feature failure; a destroyed object leaves the population — `Objects`, `InState` and element-derived sources skip it and its label still resolves for `Events` rows, while a source naming only destroyed objects is the same refusal | `runtime/lifetimes.go` `Context.Destroyed`; `queryexec/objects.go` `executor.eachSessionObject`, `executor.evaluateObjects`; `queryexec/states.go` `executor.evaluateInState`, `executor.objectArgument`, `executor.objectsDeclaredBy`, `executor.objectDestroyedError`; `queryexec/errors.go` `ErrorObjectDestroyed` | `queryexec/robustness_state_event_queries_test.go:TestQueryRobustnessStateEventQueries`; `runtime/robustness_state_event_queries_test.go:TestRuntimeRobustnessStateEventQueries`; `grpc/robustness_docquery_states_events_test.go:TestGRPCRobustnessDocumentQueryStatesEvents` | ✅ Faithful | | The trace is a typed relation the printer writes from: `TraceRecorder` keeps a `TraceRecord` per accept, send, transition, state entry, exit and do step, `choice` draw (the alternatives and the one taken, due order and region order included) and unevaluable guard, each with its `TraceOrigin` — the clock's instant, the object and the behavior — beside the free-text lines the other tracers write, and `Entries()` prints every record through `TraceRecord.Line`, so `-trace`/`%trace` output is byte-for-byte what it was and cannot drift from the record; `%trace off` and `Clear` discard it. `Events(source = null, kind = "all", since = null, before = null)` reads the records in the order the run made them as **event rows**: `kind`, `time` (the origin's instant as a quantity in the clock's unit), `object`/`path`/`machine`, `state`/`from`/`to`, `target` (a send's addressee), `event`, `payload` (`name = value` per field), `alternatives`/`taken` (a choice) and `text` (the printed line); `source` keeps the records of the objects behind its rows (every object's when left out), `kind` one or several comma-separated kinds, and `[since, before)` — inclusive start, exclusive end — a bare number in the clock's unit or a duration converted through `Context.ClockMagnitude`. Refused: a session recording no trace (`ErrorNoTrace`), a kind the record has not (`ErrorInvalidArgument`), a bound that is no instant — a non-duration quantity, a clock carrying no unit — or a `before` at or before `since` (`ErrorInvalidInterval`), and `OwnedElements`, `Descendants`, `Ancestors`, `RelatedElements`, `States`, `Verdicts` and `Events` themselves over a state or event row (`ErrorStateRow`, `ErrorEventRow`); `WhereFeature`, `OrderBy`, `Project` and `Column` read the state and event properties before the element's own, and `WhereName`/`WhereType` the state declaration or the object's type | `runtime/trace.go` `TraceKind`, `TraceOrigin`, `TraceRecord`, `TraceRecord.Line`, `TraceRecorder.Records`, `TraceRecorder.Entries`, `RecordAccept`, `RecordSend`, `RecordStateTransition`, `RecordStateEntry`, `RecordStateExit`, `RecordDoStep`, `RecordNote`; `runtime/context.go` `Context.ClockMagnitude`; `queryplan/plan.go` `OperationEvents`; `queryexec/value.go` `ValueEvent`; `queryexec/event.go` `Event`, `EventValue`, `Value.Event`; `queryexec/events.go` `executor.evaluateEvents`, `executor.eventKindArgument`, `executor.instantArgument`, `eventKinds`; `queryexec/errors.go` `ErrorNoTrace`, `ErrorInvalidInterval`, `ErrorEventRow`; `repl/trace.go` (the session's recorder); `cmd/sysml/check.go` `checks.runQueries` (queries run after `-state`/`-action` and `-advance`); `libs/stdlib/OpenSysML Libraries/DocumentQueries.sysml` `Events`, `Event` | `runtime/trace_test.go:TestExecutionTrace` (goldens unchanged); `queryexec/events_test.go:TestExecuteEventsReadsTheTraceInOrder`, `:TestExecuteEventsByKind`, `:TestExecuteEventsIntervalIsClosedOpen`, `:TestExecuteEventRowsThroughRowOperations`, `:TestExecuteEventsRefusals`; `repl/docquery_states_test.go:TestRunQueryEventsOverSession`; `cmd/sysml/run_query_test.go:TestRunQueryOverStatesAndTrace` | ✅ Implemented | | A state or event row renders everywhere a query result does: `%run-query`/`-run-query` print a state row as `. in ` and an event row as `t= .: `; a document cell is that text in Markdown and PDF, and in HTML a `span.sysml-state` with `data-machine`, `data-state`, `data-region` or a `span.sysml-event` with `data-event-kind`, `data-time`, each with the object's `data-object`; `RunDocumentQuery` answers the `state` and `event` arms of `DocumentValue` (`DocumentState`: `object`, `machine`, `name`, `path`, `region`, `enclosing`, `state`, `text`; `DocumentEvent`: `kind`, `time`, `object`, `machine`, `state`, `from`, `to`, `target`, `event`, `payload`, `alternatives`, `taken`, `text`) over the held population, whose run is traced from the first `Instantiate` into a recorder keeping the most recent `OPENSYSML_GRPC_MAX_HELD_EVENTS` records (default 100,000; an `Events` interval reaching a dropped record is a `trace-truncated` error, `FAILED_PRECONDITION`), and refuses either bound as a parameter with `INVALID_ARGUMENT`; the Go and Python clients decode them as `DocumentState`/`DocumentEvent` on the row and in cells and refuse to bind one, Node, Java and Rust carry the regenerated stubs | `repl/docquery.go` `formatQueryValue`; `docir/evaluate.go`; `docrender/markdown.go`, `docrender/html.go` (the PDF backend hands the HTML-input engines the HTML backend's page); `grpc/docquery.go` `documentState`, `documentEvent`, `boundValue`; `grpc/objects.go` `Service.objects` (the traced population); `api/proto/sysml.proto` `DocumentState`, `DocumentEvent`; `client/opensysml/documents.go` `DocumentState`, `DocumentEvent`, `Row.State`, `Row.Event`; `client/python/opensysml/document.py` `DocumentState`, `DocumentEvent`, `DocumentRow.state`, `DocumentRow.event`; `tools/cmd/conformance/pkgclient.go` `cellToProto`/`cellFromProto` | `docrender/states_test.go:TestMarkdownStateReportGolden` (`testdata/state_report.golden.md`), `:TestHTMLStateReport`; `docpdf/docpdf_test.go:TestRenderStateReportPage`; `docpdf/integration_test.go:TestRenderStateReportWithInstalledEngines` (real engines, skipped when absent); `repl/docquery_states_test.go`; `cmd/sysml/run_query_test.go:TestRunQueryOverStatesAndTrace`; gRPC conformance `document_query_states`, `document_query_in_state`, `document_query_events`; `client/opensysml/states_test.go:TestRunDocumentQueryAnswersStateAndEventRows`, `:TestStateAndEventRowsAreNotBound`; `client/python/tests/test_document.py::test_a_state_row_decodes_to_the_object_and_its_state`, `::test_an_event_row_decodes_to_the_trace_record`, `::test_a_state_or_event_binding_is_refused` | ✅ Implemented | -| A relationship-derived column traverses from each row's element — an object row's declaration, the assertion of a verdict row, the state or behavior a state or event row is of — with the breadth-first traversal `RelatedElements` uses (every relationship kind and direction it accepts, bounded by `maxDepth`, deduplicated by semantic identity, ordered as reached, charged to the shared visit budget); `list` yields the related elements as one multi-valued cell (an empty cell when none), `count` an integer and `any` a boolean that stops at the first element reached, and the cells are read downstream by name — `WhereFeature` and `OrderBy` over a projected column read its cells (an element compares and orders as its qualified name, under the text operators), a document table groups by it, and `%run-query`, Markdown, HTML and `RunDocumentQuery` carry every value as they carry a multi-valued `documentation` cell. An unknown relationship kind, an invalid direction and an exhausted budget are the typed failures `RelatedElements` raises, naming the column; a row no element declares (an event posted from outside the run) is a typed `undeclared-row` failure naming the column and the row | `queryexec/related_column.go` `relatedColumn`, `relatedColumnOf`, `executor.evaluateRelatedCell`, `columnScoped`; `queryexec/related.go` `executor.validateRelationship`, `executor.traverseRelated`; `queryexec/where_related.go` `executor.hasRelated`; `queryexec/computed.go` `computedColumns`, `executor.evaluateColumnCell`; `queryexec/operations.go` `projectedColumn`, `executor.featureValues`, `compareValue`, `executor.compareOrdered`; `queryexec/errors.go`; `docplan/compiler.go` `columnNames` | `queryexec/related_column_test.go:TestExecuteRelatedColumnsBuildATraceabilityMatrix`, `:TestExecuteRelatedColumnFollowsDepthAndDirection`, `:TestExecuteRelatedColumnReportsItsColumnInErrors`, `:TestExecuteRelatedColumnChargesTableConstructionToTheVisitBudget`, `:TestExecuteRelatedColumnAnyStopsAtTheFirstElement`, `:TestExecuteRelatedColumnsFilterAndOrderDownstream`, `:TestExecuteRelatedColumnElementsCompareByQualifiedName`, `:TestExecuteRelatedColumnTraversesFromAnObjectsDeclaration`, `:TestExecuteRelatedColumnRefusesARowNoElementDeclares`; `grpc/related_column_test.go:TestRunDocumentQueryCarriesRelatedColumns`; `docplan/runs_test.go:TestCompileGroupedTableSeesRelatedColumns` | ✅ Implemented | +| A relationship-derived column traverses from each row's element — an object row's declaration, the assertion of a verdict row, the state or behavior a state or event row is of — with the breadth-first traversal `RelatedElements` uses (every relationship kind and direction it accepts, bounded by `maxDepth`, deduplicated by semantic identity, ordered as reached, charged to the shared visit budget); `list` yields the related elements as one multi-valued cell (an empty cell when none), `count` an integer and `any` a boolean that stops at the first element reached, and the cells are read downstream by name — `WhereFeature` and `OrderBy` over a projected column read its cells (an element compares and orders as its qualified name, under the text operators), a document table groups by it, and `%run-query`, Markdown, HTML and `RunDocumentQuery` carry every value as they carry a multi-valued `documentation` cell. An unknown relationship kind, an invalid direction and an exhausted budget are the typed failures `RelatedElements` raises, naming the column; a row no element declares (an event posted from outside the run) is a typed `undeclared-row` failure naming the column and the row | `queryexec/related_column.go` `relatedColumn`, `relatedColumnOf`, `executor.evaluateRelatedCell`, `columnScoped`; `queryexec/related.go` `executor.validateRelationship`, `executor.traverseRelated`; `queryexec/where_related.go` `executor.hasRelated`; `queryexec/computed.go` `computedColumns`, `executor.evaluateColumnCell`; `queryexec/operations.go` `projectedColumn`, `executor.featureValues`, `compareValue`, `executor.compareOrdered`; `queryexec/errors.go`; `docplan/compiler.go` `columnNames` | `queryexec/related_column_test.go:TestExecuteRelatedColumnsBuildATraceabilityMatrix`, `:TestExecuteRelatedColumnFollowsDepthAndDirection`, `:TestExecuteRelatedColumnReportsItsColumnInErrors`, `:TestExecuteRelatedColumnLeavesTableConstructionUncharged`, `:TestExecuteRelatedColumnAnyStopsAtTheFirstElement`, `:TestExecuteRelatedColumnsFilterAndOrderDownstream`, `:TestExecuteRelatedColumnElementsCompareByQualifiedName`, `:TestExecuteRelatedColumnTraversesFromAnObjectsDeclaration`, `:TestExecuteRelatedColumnRefusesARowNoElementDeclares`; `grpc/related_column_test.go:TestRunDocumentQueryCarriesRelatedColumns`; `docplan/runs_test.go:TestCompileGroupedTableSeesRelatedColumns` | ✅ Implemented | **Known limitations:** relationship traversal covers the kinds above only; refinement and derivation links have no dedicated semantic representation yet and are unknown kinds. diff --git a/internal/doc/docir/evaluate.go b/internal/doc/docir/evaluate.go index d251d7391..5644988f6 100644 --- a/internal/doc/docir/evaluate.go +++ b/internal/doc/docir/evaluate.go @@ -59,6 +59,7 @@ func EvaluateSet( } collectCrossAnchors(plan.Content(), external) } + context = sharingRelationshipTables(context) documents := make([]*Document, 0, len(plans)) for _, plan := range plans { document, err := evaluate(plan, context, options, text, external[plan.Name()]) @@ -83,6 +84,7 @@ func evaluate( if context.Index == nil || context.Resolver == nil || context.Model == nil { return nil, &Error{Kind: ErrorInvalidContext, Document: plan.Name()} } + context = sharingRelationshipTables(context) referenced := referencedAnchors(plan.Content()) for anchor := range external { referenced[anchor] = true @@ -106,6 +108,16 @@ func evaluate( }, nil } +// sharingRelationshipTables gives a context without relationship tables its +// own, so every query of the documents evaluated under it builds each kind's +// edges once. +func sharingRelationshipTables(context queryexec.Context) queryexec.Context { + if context.Related == nil { + context.Related = queryexec.NewRelationshipTables() + } + return context +} + // collectCrossAnchors records, per target document, the anchors that other // documents' reference runs require it to emit. func collectCrossAnchors(planned []docplan.Content, external map[string]map[string]bool) { diff --git a/internal/doc/queryexec/execute.go b/internal/doc/queryexec/execute.go index f2198109d..d0b5f9e5e 100644 --- a/internal/doc/queryexec/execute.go +++ b/internal/doc/queryexec/execute.go @@ -27,6 +27,9 @@ type Context struct { // Roots are the objects the session holds directly, each under its label, // in the order Objects enumerates them. Roots []Root + // Related memoizes relationship edge tables across the executions sharing + // this context; nil builds them once per execution. + Related *RelationshipTables } // Root is one object a session holds directly, under its label (`Demo::car`, `#7`). @@ -80,7 +83,7 @@ type executor struct { program map[string]queryplan.Definition budget *visitBudget calls *visitBudget - related *relationshipTables + related *RelationshipTables derived *derivedValues depthLeft int stack []string @@ -118,6 +121,10 @@ func Execute(program *queryplan.Program, context Context, bindings Bindings, opt for _, compiledDefinition := range definitions { compiled[compiledDefinition.Name()] = compiledDefinition } + related := context.Related + if related == nil { + related = NewRelationshipTables() + } execution := &executor{ definition: definition, context: context, @@ -126,7 +133,7 @@ func Execute(program *queryplan.Program, context Context, bindings Bindings, opt program: compiled, budget: &visitBudget{remaining: budget}, calls: &visitBudget{remaining: calls}, - related: newRelationshipTables(), + related: related, derived: &derivedValues{}, depthLeft: depth, stack: []string{definition.Name()}, diff --git a/internal/doc/queryexec/related.go b/internal/doc/queryexec/related.go index 6eacdc7a9..d8316be53 100644 --- a/internal/doc/queryexec/related.go +++ b/internal/doc/queryexec/related.go @@ -47,14 +47,30 @@ type relationshipEdges struct { incoming map[symbols.ElementKey][]*symbols.Symbol } -// relationshipTables caches the per-kind edge tables of one execution, shared -// across invoked queries the way the visit budget is. -type relationshipTables struct { - entries map[string]*relationshipEdges +// RelationshipTables memoizes one model's per-kind edge tables across the +// executions whose Context shares them. Not safe for concurrent use. +type RelationshipTables struct { + index *symbols.Index + generation uint64 + model *semantics.Model + entries map[string]*relationshipEdges } -func newRelationshipTables() *relationshipTables { - return &relationshipTables{entries: make(map[string]*relationshipEdges)} +// NewRelationshipTables returns empty tables for a Context to carry. +func NewRelationshipTables() *RelationshipTables { + return &RelationshipTables{entries: make(map[string]*relationshipEdges)} +} + +// lookup returns the cached tables of one kind, first discarding every entry +// built against another index or model, or the index before an edit. +func (t *RelationshipTables) lookup(kind string, context Context) (*relationshipEdges, bool) { + generation := context.Index.Generation() + if t.index != context.Index || t.generation != generation || t.model != context.Model { + t.index, t.generation, t.model = context.Index, generation, context.Model + t.entries = make(map[string]*relationshipEdges) + } + edges, ok := t.entries[kind] + return edges, ok } // relationshipWalk is the validated relationshipKind, direction and maxDepth @@ -143,11 +159,7 @@ func (e *executor) traverseRelated( if walk.maxDepth.reached(next.depth) { continue } - neighbors, err := e.relatedNeighbors(expression, walk.kind, walk.direction, next.sym) - if err != nil { - return err - } - for _, neighbor := range neighbors { + for _, neighbor := range e.relatedNeighbors(walk.kind, walk.direction, next.sym) { key := symbols.KeyOf(neighbor) if _, duplicate := seen[key]; duplicate { continue @@ -182,19 +194,16 @@ func supportedRelationship(kind string) bool { // sym in the given direction, in declaration order. Outgoing lineage reads // sym's own declared relationships; every other combination reads the edge // tables built from the workspace's declarations. -func (e *executor) relatedNeighbors(expression queryplan.Expression, kind, direction string, sym *symbols.Symbol) ([]*symbols.Symbol, error) { +func (e *executor) relatedNeighbors(kind, direction string, sym *symbols.Symbol) []*symbols.Symbol { if relKind, lineage := lineageKinds[kind]; lineage && direction == directionOutgoing { - return e.lineageTargets(sym, relKind), nil - } - edges, err := e.relationshipEdges(expression, kind) - if err != nil { - return nil, err + return e.lineageTargets(sym, relKind) } + edges := e.relationshipEdges(kind) table := edges.outgoing if direction == directionIncoming { table = edges.incoming } - return table[symbols.KeyOf(sym)], nil + return table[symbols.KeyOf(sym)] } // lineageTargets resolves the targets of sym's declared relationships of the @@ -212,47 +221,36 @@ func (e *executor) lineageTargets(sym *symbols.Symbol, kind ast.RelationshipKind return out } -// relationshipEdges returns the edge tables for one relationship kind, -// building them on first use by scanning the workspace's documents in sorted -// name order and each document's symbols in declaration order. Every -// declaration examined charges the shared visit budget; a built table is -// cached, so later traversals of the same kind read it for free. -func (e *executor) relationshipEdges(expression queryplan.Expression, kind string) (*relationshipEdges, error) { - if cached, ok := e.related.entries[kind]; ok { - return cached, nil +// relationshipEdges returns one kind's edge tables, built on first use by scanning +// the workspace's documents in name order. The scan is memoized and uncharged; +// only the elements a traversal reaches pay the visit budget. +func (e *executor) relationshipEdges(kind string) *relationshipEdges { + if cached, ok := e.related.lookup(kind, e.context); ok { + return cached } edges := &relationshipEdges{ outgoing: make(map[symbols.ElementKey][]*symbols.Symbol), incoming: make(map[symbols.ElementKey][]*symbols.Symbol), } for _, document := range e.context.Index.WorkspaceDocuments() { - if err := e.scanScope(expression, edges, kind, e.context.Index.DocumentRoot(document)); err != nil { - return nil, err - } + e.scanScope(edges, kind, e.context.Index.DocumentRoot(document)) } e.related.entries[kind] = edges - return edges, nil + return edges } // scanScope records the edges of one relationship kind that the declarations -// in scope and its nested scopes state, charging the visit budget per -// declaration examined. -func (e *executor) scanScope(expression queryplan.Expression, edges *relationshipEdges, kind string, scope *symbols.Scope) error { +// in scope and its nested scopes state. +func (e *executor) scanScope(edges *relationshipEdges, kind string, scope *symbols.Scope) { if scope == nil { - return nil + return } for _, member := range scope.AllMembers() { - if !e.consumeVisit() { - return e.budgetError(expression) - } e.scanSymbol(edges, kind, member) } for _, child := range scope.Children() { - if err := e.scanScope(expression, edges, kind, child); err != nil { - return err - } + e.scanScope(edges, kind, child) } - return nil } // scanSymbol records the edges the given symbol's declaration states: the diff --git a/internal/doc/queryexec/related_column_test.go b/internal/doc/queryexec/related_column_test.go index ab837b2c7..e74f5521a 100644 --- a/internal/doc/queryexec/related_column_test.go +++ b/internal/doc/queryexec/related_column_test.go @@ -288,12 +288,24 @@ func TestExecuteRelatedColumnReportsItsColumnInErrors(t *testing.T) { } } -func TestExecuteRelatedColumnChargesTableConstructionToTheVisitBudget(t *testing.T) { +func TestExecuteRelatedColumnLeavesTableConstructionUncharged(t *testing.T) { fixture := loadExecutionFixtureFile(t, traceMatrixFixture) - // Building the satisfaction table scans the workspace, so a tiny budget - // fails even for a requirement nothing satisfies. - _, err := fixture.traced(t, ElementValue(fixture.symbol(t, "dataRequirement")), - "satisfaction", "incoming", 1, "count", Options{VisitBudget: 3}) + // Building the satisfaction table scans the workspace uncharged, so a + // requirement nothing satisfies is counted within a budget of one. + none, err := fixture.traced(t, ElementValue(fixture.symbol(t, "dataRequirement")), + "satisfaction", "incoming", 1, "count", Options{VisitBudget: 1}) + if err != nil { + t.Fatalf("unsatisfied: %v", err) + } + assertColumn(t, cellsByColumn(t, none), "related", [][]string{{"0"}}) + // The two satisfiers reached are what the budget pays for. + mass := ElementValue(fixture.symbol(t, "massRequirement")) + both, err := fixture.traced(t, mass, "satisfaction", "incoming", 1, "count", Options{VisitBudget: 2}) + if err != nil { + t.Fatalf("exact budget: %v", err) + } + assertColumn(t, cellsByColumn(t, both), "related", [][]string{{"2"}}) + _, err = fixture.traced(t, mass, "satisfaction", "incoming", 1, "count", Options{VisitBudget: 1}) execution := executionError(t, err, ErrorVisitBudget) if execution.Property != "related" { t.Fatalf("error column = %q, want related", execution.Property) diff --git a/internal/doc/queryexec/related_test.go b/internal/doc/queryexec/related_test.go index c94b31105..ae7f3e6c2 100644 --- a/internal/doc/queryexec/related_test.go +++ b/internal/doc/queryexec/related_test.go @@ -5,6 +5,8 @@ import ( "testing" "github.com/Open-MBEE/OpenSysML/internal/semantic/symbols" + "github.com/Open-MBEE/OpenSysML/internal/syntax/parser" + "github.com/Open-MBEE/OpenSysML/internal/syntax/source" ) func (f executionFixture) related( @@ -200,15 +202,111 @@ func TestExecuteRelatedConsumesTheVisitBudget(t *testing.T) { } } -func TestExecuteRelatedChargesTableConstructionToTheVisitBudget(t *testing.T) { +func TestExecuteRelatedLeavesTableConstructionUncharged(t *testing.T) { fixture := loadExecutionFixtureFile(t, "testdata/tmt_relationships.sysml") - // Building the edge table scans the workspace, so a tiny budget fails even - // when the source has no matching edges. - _, err := fixture.related(t, fixture.symbol(t, "Subsystem"), "connection", "incoming", 1, - Options{VisitBudget: 3}) - var executionError *Error - if !errors.As(err, &executionError) || executionError.Kind != ErrorVisitBudget { - t.Fatalf("visit budget error = %v", err) + // Building the edge table scans every declaration in the workspace, yet a + // source with no matching edges reaches nothing and so pays nothing. + none, err := fixture.related(t, fixture.symbol(t, "Subsystem"), "connection", "incoming", 1, + Options{VisitBudget: 1}) + if err != nil { + t.Fatalf("no edges: %v", err) + } + if len(none.Rows()) != 0 { + t.Fatalf("rows = %v, want none", rowNames(none)) + } + // Only the elements reached are charged: the two subsetters fit a budget of + // exactly two, however many declarations the scan examined. + both, err := fixture.related(t, fixture.symbol(t, "instruments"), "subsetting", "incoming", 1, + Options{VisitBudget: 2}) + if err != nil { + t.Fatalf("exact budget: %v", err) + } + if names := rowNames(both); len(names) != 2 || + names[0] != "Observatory::iris" || names[1] != "Observatory::modhis" { + t.Fatalf("rows = %v", names) + } +} + +func TestExecuteSharesRelationshipTablesThroughTheContext(t *testing.T) { + fixture := loadExecutionFixtureFile(t, "testdata/tmt_relationships.sysml") + tables := NewRelationshipTables() + context := Context{Index: fixture.index, Resolver: fixture.resolver, Model: fixture.model, Related: tables} + bindings := func(source string) Bindings { + return Bindings{ + "source": {ElementValue(fixture.symbol(t, source))}, + "kind": {StringValue("subsetting")}, + "direction": {StringValue("incoming")}, + "maxDepth": {IntegerValue(1)}, + } + } + program := fixture.program(t, "Related") + if _, err := Execute(program, context, bindings("instruments"), Options{}); err != nil { + t.Fatalf("first execution: %v", err) + } + built, ok := tables.entries["subsetting"] + if !ok { + t.Fatal("the first execution must leave its subsetting table in the context") + } + if _, err := Execute(program, context, bindings("Subsystem"), Options{}); err != nil { + t.Fatalf("second execution: %v", err) + } + if tables.entries["subsetting"] != built { + t.Fatal("a second execution under the same context must reuse the built table") + } + // Tables built against another model are discarded rather than trusted. + other := loadExecutionFixtureFile(t, "testdata/tmt_relationships.sysml") + _, err := Execute(other.program(t, "Related"), + Context{Index: other.index, Resolver: other.resolver, Model: other.model, Related: tables}, + Bindings{ + "source": {ElementValue(other.symbol(t, "instruments"))}, + "kind": {StringValue("subsetting")}, + "direction": {StringValue("incoming")}, + "maxDepth": {IntegerValue(1)}, + }, Options{}) + if err != nil { + t.Fatalf("other model: %v", err) + } + if tables.entries["subsetting"] == built || tables.index != other.index { + t.Fatal("tables built against another index must be rebuilt") + } +} + +func TestExecuteRebuildsRelationshipTablesAfterAnIndexEdit(t *testing.T) { + fixture := loadExecutionFixtureFile(t, "testdata/tmt_relationships.sysml") + context := Context{Index: fixture.index, Resolver: fixture.resolver, Model: fixture.model, Related: NewRelationshipTables()} + subsetters := func(step string) []string { + t.Helper() + rows, err := Execute(fixture.program(t, "Related"), context, Bindings{ + "source": {ElementValue(fixture.symbol(t, "instruments"))}, + "kind": {StringValue("subsetting")}, + "direction": {StringValue("incoming")}, + "maxDepth": {IntegerValue(1)}, + }, Options{}) + if err != nil { + t.Fatalf("%s: %v", step, err) + } + return rowNames(rows) + } + if names := subsetters("before the edit"); len(names) != 2 { + t.Fatalf("rows before the edit = %v", names) + } + + // The same index, edited in place: a document declaring one more subsetter + // is added, then removed again. The tables follow both edits. + edit := "edit.sysml" + p := parser.New(source.New(edit, []byte("package Edit { part nfiraos :> Observatory::instruments; }"))) + root := p.ParseFile() + if len(p.Diagnostics) > 0 { + t.Fatalf("parse edit: %v", p.Diagnostics) + } + fixture.index.AddDocument(edit, root) + fixture.index.ExpandWildcardImports() + if names := subsetters("after adding a subsetter"); len(names) != 3 || names[0] != "Edit::nfiraos" { + t.Fatalf("rows after adding a subsetter = %v", names) + } + fixture.index.RemoveDocument(edit) + if names := subsetters("after removing it again"); len(names) != 2 { + t.Fatalf("rows after removing the subsetter = %v", names) } } diff --git a/internal/doc/queryexec/where_related_test.go b/internal/doc/queryexec/where_related_test.go index 005de1c8f..fd8b9b628 100644 --- a/internal/doc/queryexec/where_related_test.go +++ b/internal/doc/queryexec/where_related_test.go @@ -285,7 +285,13 @@ func TestExecuteWhereRelatedChargesTheVisitBudget(t *testing.T) { pair := append(leaf, ElementValue(fixture.symbol(t, "OpticalSubsystem"))) _, err = fixture.filtered(t, pair, "specialization", "outgoing", 3, true, Options{VisitBudget: 1}) executionError(t, err, ErrorVisitBudget) - // Building an edge table charges each declaration scanned, as RelatedElements does. - _, err = fixture.filtered(t, leaf, "satisfaction", "incoming", 1, false, Options{VisitBudget: 3}) - executionError(t, err, ErrorVisitBudget) + // Building an edge table is not charged, as for RelatedElements: a row + // nothing satisfies is kept without spending a visit. + kept, err = fixture.filtered(t, leaf, "satisfaction", "incoming", 1, false, Options{VisitBudget: 1}) + if err != nil { + t.Fatalf("uncharged table: %v", err) + } + if got := rowNames(kept); len(got) != 1 || got[0] != "Observatory::MirrorAssembly" { + t.Fatalf("kept = %v", got) + } } diff --git a/internal/semantic/symbols/index.go b/internal/semantic/symbols/index.go index bb15efa71..ff162ecd6 100644 --- a/internal/semantic/symbols/index.go +++ b/internal/semantic/symbols/index.go @@ -293,6 +293,10 @@ func UsageAnnotatesOthers(u *ast.Usage) bool { // Frozen reports whether the index has been frozen. func (idx *Index) Frozen() bool { return idx.frozen } +// Generation counts the writes the index has taken; a value read from it is +// current while Generation is unchanged. +func (idx *Index) Generation() uint64 { return idx.generation.get() } + // Base is the frozen index an overlay reads through to, nil for an index that // stands alone. Two overlays over one base share its documents and symbols. func (idx *Index) Base() *Index { return idx.base }