fix(output): render JsonNode and JsonElement results as JSON data in human output

docs/output-system.md

@@ -23,6 +23,37 @@ The active output format is resolved in this order:
 | `yaml`     | YAML serialization.                  |
 | `markdown` | Markdown table/document rendering.   |
 
+### JSON results
+
+A handler can return `System.Text.Json.Nodes.JsonNode` (`JsonObject`, `JsonArray`, `JsonValue`) or a
+`JsonElement`, including as rows of an `IReplPageSource<T>`. `human` and `spectre` then render the JSON
+data rather than the CLR members of those types:
+
+- An object becomes one `key: value` line per field.
+- In `spectre` as in `human`, a table's header labels stay on one line: a label wider than its column is
+  truncated rather than wrapped, and a line break in a label reads as a space.
+- Rows that are all objects with at least one key become a table. Its columns are the union of the rows'
+  keys in first-seen order, matched ordinally, and a key missing from a row leaves that cell empty. The rows'
+  keys, not a type, define the columns, so each page the pager fetches can name other columns than the page
+  before it. See [Result Flow And Paging](result-flow.md) for how the pager shows their headers.
+- An array of scalars becomes one value per line. So does a page of rows among which one is a JSON null or
+  an empty object: each row reads as its literal, since a blank table row would read as no row at all.
+- A page declared with a JSON item type, such as `IReplPageSource<JsonNode?>`, renders as JSON even when
+  every item on it is a JSON null.
+- Values are compact JSON literals. So a string shows as `"x"`, an explicit JSON null shows as `null`, and
+  a nested object or array shows as itself. A `JsonElement` object that repeats a property name shows the
+  last value given for it, as JavaScript reads it.
+- A JSON value held by a property of an ordinary result object shows as a compact literal too, and a
+  property declared as JSON (`JsonNode?`, `JsonElement?`) that holds `null` reads `null`, unless its
+  `DisplayFormat` sets a `NullDisplayText`. A JSON value passed as a result's details renders like a JSON
+  result.
+
+In JSON strings and keys, control characters and Unicode format characters (such as bidirectional
+overrides) are escaped. So a payload can neither drive the terminal nor make it display something other
+than the data. Non-ASCII text stays readable. This covers JSON values only: a plain CLR `string` in a
+result is still written as-is. The `json` format and MCP output are unchanged, and `markdown` does not
+special-case JSON yet.
+
 ### Format aliases
 
 The built-in aliases are:

docs/result-flow.md

@@ -523,6 +523,28 @@ The full viewport is inspired by `less`: it does not depend on terminal
 scrollback. It renders from an internal buffer and fetches additional
 `IReplPageSource<T>` payloads as the user pages past the buffered end.
 
+The pager pins the first page's table header. `human` and `spectre` render a
+table's header on every page and tell the pager exactly which lines hold the
+header and which columns it names, whether the pager shows a result that is
+simply long or pages an `IReplPageSource<T>`. A long result keeps its footer
+asking to rerun for the next page, since that pager cannot fetch it. A fetched page whose header names
+the columns the page before it showed adds only its rows. A page naming other
+columns keeps its own header, since its rows would otherwise sit under headings
+that are not theirs. That can happen with JSON rows, whose columns come from
+their keys, or with pages of rows of different types. In `inline` and `full`,
+the header at the top of the viewport stays the first page's, and a later
+header scrolls with its rows.
+
+A custom `IReplPagerRenderer` receives each fetched page as text: a header
+naming the previous page's columns is stripped from it first, and one naming
+other columns is kept.
+
+For any other transformer, the pager recognizes a header from the text: a
+separator line under the first line, a first line starting with `#` and a
+space, or bold styling. It drops that header, and any line that reads like it,
+from every fetched page, and drops lines that read like the rerun footer from
+the first payload.
+
 Applications that need a different terminal experience can register a custom
 `IReplPagerRenderer` with
 `options.Output.ResultFlow.UsePagerRenderer(renderer)`. A custom renderer is

src/Repl.Core/CoreReplApp.Execution.cs

@@ -1286,11 +1286,11 @@ internal async ValueTask<bool> RenderOutputAsync(
 				.ConfigureAwait(false);
 		}
 
-		var payload = await transformer.TransformAsync(result, cancellationToken).ConfigureAwait(false);
-		payload = TryColorizeStructuredPayload(payload, format, isInteractive);
+		var (rendered, layout) = await RenderPayloadAsync(transformer, result, cancellationToken).ConfigureAwait(false);
+		var payload = TryColorizeStructuredPayload(rendered, format, isInteractive);
 		if (!string.IsNullOrEmpty(payload))
 		{
-			await WritePayloadAsync(payload, transformer, resultFlow, cancellationToken).ConfigureAwait(false);
+			await WritePayloadAsync(payload, layout, transformer, resultFlow, cancellationToken).ConfigureAwait(false);
 		}
 
 		return true;
@@ -1305,11 +1305,12 @@ private async ValueTask<bool> RenderPageSourceAsync(
 	{
 		var request = CreatePageSourceRequest(resultFlow);
 		var page = await FetchPageSourceAsync(source, request, cancellationToken).ConfigureAwait(false);
-		var payload = await transformer.TransformAsync(page, cancellationToken).ConfigureAwait(false);
-		payload = TryColorizeStructuredPayload(payload, transformer.Name, isInteractive);
+		var (rendered, layout) = await RenderPayloadAsync(transformer, page, cancellationToken).ConfigureAwait(false);
+		var payload = TryColorizeStructuredPayload(rendered, transformer.Name, isInteractive);
 
 		if (!TryCreatePager(
 				payload,
+				layout,
 				transformer,
 				resultFlow,
 				page.PageInfo.HasMore,
@@ -1358,9 +1359,13 @@ private async ValueTask<bool> RenderPageSourcePagerAsync(
 		CancellationToken cancellationToken)
 	{
 		var nextCursor = page.PageInfo.NextCursor;
-		var pagerPayload = await TransformPagerPageAsync(transformer, page, ResultFlowPageRenderMode.Initial, cancellationToken)
+		var (initialPayload, initialLayout) = await RenderPagerPageAsync(
+				transformer,
+				page,
+				ResultFlowPageRenderMode.Initial,
+				cancellationToken)
 			.ConfigureAwait(false);
-		pagerPayload = TryColorizeStructuredPayload(pagerPayload, transformer.Name, isInteractive);
+		var pagerPayload = TryColorizeStructuredPayload(initialPayload, transformer.Name, isInteractive);
 		await ResultFlowPager.WriteAsync(
 				pagerPayload,
 				ReplSessionIO.Output,
@@ -1372,6 +1377,7 @@ await ResultFlowPager.WriteAsync(
 					PagerMode = pagerMode,
 					AnsiEnabled = ansiEnabled,
 					HasMorePayload = page.PageInfo.HasMore,
+					PayloadLayout = initialLayout,
 					FetchNextPayload = FetchNextPayloadAsync,
 					PagerRenderers = _options.Output.ResultFlow.PagerRenderers,
 					MaxBufferedLines = _options.Output.ResultFlow.MaxBufferedLines,
@@ -1390,13 +1396,17 @@ await ResultFlowPager.WriteAsync(
 			var nextRequest = request with { Cursor = nextCursor };
 			var nextPage = await FetchPageSourceAsync(source, nextRequest, token).ConfigureAwait(false);
 			nextCursor = nextPage.PageInfo.NextCursor;
-			var nextPayload = await TransformPagerPageAsync(transformer, nextPage, ResultFlowPageRenderMode.Continuation, token)
+			var (nextPayload, nextLayout) = await RenderPagerPageAsync(
+					transformer,
+					nextPage,
+					ResultFlowPageRenderMode.Continuation,
+					token)
 				.ConfigureAwait(false);
-			nextPayload = TryColorizeStructuredPayload(nextPayload, transformer.Name, isInteractive);
 			return new ResultFlowPagerPage(
-				nextPayload,
+				TryColorizeStructuredPayload(nextPayload, transformer.Name, isInteractive),
 				nextPage.PageInfo.HasMore,
-				ContainsPresentationChrome: false);
+				ContainsPresentationChrome: false,
+				nextLayout);
 		}
 	}
 
@@ -1412,26 +1422,53 @@ await ResultFlowPager.WriteAsync(
 		return await RefuseGlobalOptionErrorsAsync(globalOptions, cancellationToken).ConfigureAwait(false);
 	}
 
-	private static ValueTask<string> TransformPagerPageAsync(
+	// A transformer that declares its layout says where its header and footer are; for any other, the pager
+	// detects them from the text.
+	private static async ValueTask<(string Payload, RenderedLayout? Layout)> RenderPayloadAsync(
+		IOutputTransformer transformer,
+		object? value,
+		CancellationToken cancellationToken)
+	{
+		if (transformer is ILayoutDeclaringOutputTransformer declaring)
+		{
+			var rendered = await declaring.RenderAsync(value, cancellationToken).ConfigureAwait(false);
+			return (rendered.Text, rendered.Layout);
+		}
+
+		return (await transformer.TransformAsync(value, cancellationToken).ConfigureAwait(false), null);
+	}
+
+	// A transformer built against 0.11 still gets the render mode that asks it to leave a continuation's header
+	// out, which the pager then detects on the first page only.
+	private static async ValueTask<(string Payload, RenderedLayout? Layout)> RenderPagerPageAsync(
 		IOutputTransformer transformer,
 		IReplPage page,
 		ResultFlowPageRenderMode mode,
 		CancellationToken cancellationToken)
 	{
 		var displayPage = CreatePagerDisplayPage(page);
-		return transformer is IResultFlowOutputTransformer resultFlowTransformer
-			? resultFlowTransformer.TransformPageAsync(displayPage, mode, cancellationToken)
-			: transformer.TransformAsync(displayPage, cancellationToken);
+		if (transformer is ILayoutDeclaringOutputTransformer declaring)
+		{
+			var rendered = await declaring.RenderPageAsync(displayPage, cancellationToken).ConfigureAwait(false);
+			return (rendered.Text, rendered.Layout);
+		}
+
+		var text = transformer is IResultFlowOutputTransformer resultFlowTransformer
+			? await resultFlowTransformer.TransformPageAsync(displayPage, mode, cancellationToken).ConfigureAwait(false)
+			: await transformer.TransformAsync(displayPage, cancellationToken).ConfigureAwait(false);
+		return (text, null);
 	}
 
 	private async ValueTask WritePayloadAsync(
 		string payload,
+		RenderedLayout? layout,
 		IOutputTransformer transformer,
 		ResultFlowInvocationOptions? resultFlow,
 		CancellationToken cancellationToken)
 	{
 		if (TryCreatePager(
 				payload,
+				layout?.WithFooter(0),
 				transformer,
 				resultFlow,
 				out var keyReader,
@@ -1448,6 +1485,9 @@ await ResultFlowPager.WriteAsync(
 						VisibleRows = visibleRows,
 						PagerMode = pagerMode,
 						AnsiEnabled = ansiEnabled,
+						// This pager cannot fetch more, so a footer asking to rerun for the next page is the only way
+						// to continue, and stays in view instead of being stripped.
+						PayloadLayout = layout?.WithFooter(0),
 						PagerRenderers = _options.Output.ResultFlow.PagerRenderers,
 						MaxBufferedLines = _options.Output.ResultFlow.MaxBufferedLines,
 					},
@@ -1461,6 +1501,7 @@ await ResultFlowPager.WriteAsync(
 
 	private bool TryCreatePager(
 		string payload,
+		RenderedLayout? layout,
 		IOutputTransformer transformer,
 		ResultFlowInvocationOptions? resultFlow,
 		[NotNullWhen(true)] out IReplKeyReader? keyReader,
@@ -1469,6 +1510,7 @@ private bool TryCreatePager(
 		out bool ansiEnabled)
 		=> TryCreatePager(
 			payload,
+			layout,
 			transformer,
 			resultFlow,
 			hasMorePayload: false,
@@ -1479,6 +1521,7 @@ private bool TryCreatePager(
 
 	private bool TryCreatePager(
 		string payload,
+		RenderedLayout? layout,
 		IOutputTransformer transformer,
 		ResultFlowInvocationOptions? resultFlow,
 		bool hasMorePayload,
@@ -1501,7 +1544,7 @@ private bool TryCreatePager(
 		}
 
 		if (!TryResolvePagerVisibleRows(out visibleRows)
-			|| (!hasMorePayload && ResultFlowPager.CountLines(payload) <= visibleRows)
+			|| (!hasMorePayload && ResultFlowPager.CountLines(payload, layout) <= visibleRows)
 			|| !TryResolvePagerKeyReader(out keyReader))
 		{
 			return false;

src/Repl.Core/ILayoutDeclaringOutputTransformer.cs

@@ -0,0 +1,21 @@
+namespace Repl;
+
+/// <summary>
+/// A human output transformer that declares the layout of what it renders, so the pager can pin its header, drop
+/// that header's repeats and strip its footer without inferring any of them from the text.
+/// </summary>
+/// <remarks>
+/// A table always renders its header, on every page: a page of JSON rows takes its columns from its own keys, so it
+/// can name other columns than the page before it. The pager drops a header that repeats the previous page's
+/// columns and keeps one that names others. This is a separate interface rather than new members of
+/// <see cref="IResultFlowOutputTransformer"/>: a friend assembly built against an earlier Repl.Core implements that
+/// one as it shipped, and still has to load when only Repl.Core is upgraded.
+/// </remarks>
+internal interface ILayoutDeclaringOutputTransformer : IOutputTransformer
+{
+	/// <summary>Renders <paramref name="value"/> as <see cref="IOutputTransformer.TransformAsync"/> does, with its layout.</summary>
+	ValueTask<RenderedPayload> RenderAsync(object? value, CancellationToken cancellationToken = default);
+
+	/// <summary>Renders a page fetched for the pager: its items, without the footer that asks to rerun for more.</summary>
+	ValueTask<RenderedPayload> RenderPageAsync(IReplPage page, CancellationToken cancellationToken = default);
+}

src/Repl.Core/IResultFlowOutputTransformer.cs

@@ -1,5 +1,8 @@
 namespace Repl;
 
+// Kept as it shipped in 0.11, with ResultFlowPageRenderMode: a friend assembly built against it, such as an older
+// Repl.Spectre, implements this exact signature and must still load when only Repl.Core is upgraded. Transformers
+// built with this version declare their layout through ILayoutDeclaringOutputTransformer instead.
 internal interface IResultFlowOutputTransformer : IOutputTransformer
 {
 	ValueTask<string> TransformPageAsync(

src/Repl.Core/InternalsVisibleTo.cs

@@ -8,3 +8,4 @@
 [assembly: InternalsVisibleTo("Repl.Spectre")]
 [assembly: InternalsVisibleTo("Repl.Mcp")]
 [assembly: InternalsVisibleTo("Repl.McpTests")]
+[assembly: InternalsVisibleTo("Repl.SpectreTests")]