diff --git a/docs/KNOWN-ISSUES.md b/docs/KNOWN-ISSUES.md index b7b679a..e730cd2 100644 --- a/docs/KNOWN-ISSUES.md +++ b/docs/KNOWN-ISSUES.md @@ -198,3 +198,5 @@ Read back through this library, xlsx and ods follow the same rules and add no ex - LibreOffice (and Excel) keep 15 significant digits when they re-save a number: a 16-digit integer such as 9007199254740992 comes back as 9007199254740990 after opening and saving the file there. Write identifiers longer than 15 digits as text. Reading ods back with this library, an all-empty row is passed over rather than handed out, so the import does not count it as skipped; the rows after it keep their numbers. + +TabularExport picks a column type by overload resolution, and some lambdas do not resolve: a char property becomes an integer column (its code point); ulong, `p => default` and a lambda returning null are ambiguous; an int or long with a DecimalImportField is ambiguous between decimal and double; TimeSpan, DateTimeOffset, Guid and enums have no overload and fail with a misleading "cannot convert to string" (CS0029). Convert explicitly (`.ToString()`, `.UtcDateTime`, …); write enums and identifiers as text. diff --git a/docs/superpowers/plans/2026-10-04-writing-part-4-export.md b/docs/superpowers/plans/2026-10-04-writing-part-4-export.md new file mode 100644 index 0000000..e92253b --- /dev/null +++ b/docs/superpowers/plans/2026-10-04-writing-part-4-export.md @@ -0,0 +1,1192 @@ +# Writing — part 4: `TabularExport` — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** A caller describes an export of objects once — columns as typed lambdas, optionally named by the import's own fields — and writes any source of those objects (chunks from gRPC, an async stream, a list) to csv, xlsx or ods in one call, with memory flat in the row count and the file importing back as the same values. + +**Architecture:** `TabularExport.For()` returns a mutable `TabularExportBuilder`; each `Column(...)` overload stores an internal `ExportColumn` — a typed value delegate plus a static typed write delegate, so a cell costs two delegate calls and no boxing. `Build()` checks the columns with the writer's own rules and returns an immutable, thread-safe `TabularExport`, whose `WriteAsync` (a whole file) and `WriteSheetAsync` (one sheet into a caller's `TabularWriter`) drive the existing `TabularWriter`, flushing after every chunk and whenever the writer recommends it. + +**Tech Stack:** .NET 8 + 10, base class library only, xunit v4 on Microsoft.Testing.Platform. + +**Spec:** `docs/superpowers/specs/2026-10-03-writing-design.md`, section "Object layer" — delivery step 4 of 5 (issue #73). Parts 1–3 (#77, #81, #83) are merged. + +## Global Constraints + +- No package references in the library; only the base class library (`TabularIndependenceTests`). +- No allocation per row or per cell on the write path; per chunk, per sheet and per file are fine. +- The target stream is never written, flushed or disposed synchronously (the writer guarantees it; this layer only drives the writer). +- A stream handed over is closed on every path, failures included, unless `TabularWriterOptions.LeaveOpen` is set. +- Programmer errors are `ArgumentException` / `ArgumentNullException` / `InvalidOperationException`; value problems stay the writer's `TabularWriteException` codes. +- Cancellation: an async source is enumerated `WithCancellation(cancellationToken)`; the token is the last parameter of every async method. +- Public API changes go in `src/TriasDev.Tabular/PublicAPI.Unshipped.txt` (RS0016 messages are authoritative). +- `dotnet format TriasDev.Tabular.slnx --verify-no-changes` must pass before every commit. +- Nothing product-specific in code, docs or commits (generic names: `Portfolio`, `Item`, `Batch`). + +## Rulings made while planning (deviations from or additions to the spec) + +- **No untyped `ImportField` overload.** The spec allowed one, checked at `Build()`. It would also catch a typed field passed with a lambda of the wrong type — `Column(DecimalImportField, p => p.Name)` would compile and fail at run time — which defeats the spec's own rule that a mismatch is a compile error. A caller with an untyped field writes `Column(field.Name, …)`. Cost: one more word at the call site. +- **A translated field's variant is refused at `Column(...)`, not at `Build()`** (`ArgumentException`): the earliest point, with the field in hand. `TranslatedImportField` itself is not an `ImportField` and cannot be passed at all. +- **A chunk-selector overload** (`IAsyncEnumerable` plus `Func>`) for both write methods. The driving case is a gRPC server stream of messages each carrying a repeated field (`RepeatedField` is an `IReadOnlyList`); without it, a net8 caller needs `System.Linq.Async` or a hand-written iterator just to call the library. +- **`Build()` validates the columns with the writer's own rules** — the check moves from `TabularWriter.CheckColumns` into an internal `TabularWriter.ColumnsProblem(ReadOnlySpan) → Exception?` both use — and throws its exception, so a duplicate header fails where the export is declared, not mid-request. +- **Default widths:** a `DateTime` column 19 characters (`yyyy-mm-dd hh:mm:ss`), a `DateOnly` column 10; other types none (the program's default). An explicit width wins. +- **Cancellation of a synchronous `IEnumerable` source** is observed at each flush (about every megabyte written), not per item. + +## Review Focus + +1. **A gRPC-shaped source** — messages each holding a list — written without any adapter. → Task 3 `WritesChunksTakenFromMessages`. +2. **A million rows from reused chunks allocate (almost) nothing per row**, in every format. → Task 3 `AllocatesNothingPerRow`. +3. **A value the format refuses mid-stream** (e.g. a 16-digit decimal at row 40,000): `TabularWriteException` with its row, the stream closed, nothing that opens as valid. → Task 2 `AValueTheFormatRefusesClosesTheStreamAndLeavesNoValidFile`. +4. **Cancellation between chunks**: `OperationCanceledException`, stream closed. → Task 2 `CancellationStopsBetweenChunksAndClosesTheStream`. +5. **The same export used concurrently** by two requests: both files correct (the export is immutable). → Task 3 `OneExportServesConcurrentWrites`. + +--- + +### Task 1: Columns — the builder and the export's declaration + +**Files:** +- Create: `src/TriasDev.Tabular/Writing/ExportColumn.cs` +- Create: `src/TriasDev.Tabular/Writing/TabularExportBuilder.cs` +- Create: `src/TriasDev.Tabular/Writing/TabularExport.cs` (the static `TabularExport.For()` and the class `TabularExport` with `Columns` only; writing comes in Task 2) +- Modify: `src/TriasDev.Tabular/Writing/TabularWriter.cs` (`CheckColumns` → `ColumnsProblem`) +- Modify: `src/TriasDev.Tabular/PublicAPI.Unshipped.txt` +- Test: `tests/TriasDev.Tabular.Tests/Writing/TabularExportBuilderTests.cs` + +**Interfaces:** +- Consumes: `TabularWriter` (`Write(string?/long/decimal/double/DateTime/DateOnly/bool)`, `WriteEmpty()`), `WriteColumn(string Header, double? Width = null)`, the typed import fields `TextImportField`, `IntegerImportField`, `DecimalImportField`, `DateImportField`, `BooleanImportField` (`Name`, `Variant`). +- Produces: + - `public static class TabularExport { public static TabularExportBuilder For(); }` + - `public sealed class TabularExportBuilder` with `Column(string header, Func value, double? width = null)` for X in `string?`, `long`, `long?`, `decimal`, `decimal?`, `double`, `double?`, `DateTime`, `DateTime?`, `DateOnly`, `DateOnly?`, `bool`, `bool?`; `Column(TextImportField, Func, double? width = null)`, `Column(IntegerImportField, Func, …)`, `Column(DecimalImportField, Func, …)`, `Column(DecimalImportField, Func, …)`, `Column(DateImportField, Func, …)`, `Column(DateImportField, Func, …)`, `Column(BooleanImportField, Func, …)` — each returns the builder; `TabularExport Build()`. + - `public sealed class TabularExport { public IReadOnlyList Columns { get; } }` (Task 2 adds the write methods). + - internal `abstract class ExportColumn` (`WriteColumn Column`, `abstract void Write(TabularWriter writer, T item)`), `sealed class ExportColumn`, `static class CellWriters`. + - internal `static Exception? TabularWriter.ColumnsProblem(ReadOnlySpan columns)`. + +- [ ] **Step 1: Write the failing tests** + +Create `tests/TriasDev.Tabular.Tests/Writing/TabularExportBuilderTests.cs`: + +```csharp +using Xunit; + +namespace TriasDev.Tabular.Tests.Writing; + +/// Declaring an export: a column's type from its lambda, its header from a field, the writer's rules at Build. +public sealed class TabularExportBuilderTests +{ + private sealed record Portfolio(long Id, int Count, string Name, decimal Amount, double Rate, DateTime Start, DateOnly Settled, bool Active, long? Parent); + + private static readonly IntegerImportField IdField = ImportField.Integer("Id"); + private static readonly TextImportField NameField = ImportField.Text("Name"); + private static readonly DecimalImportField AmountField = ImportField.Decimal("Amount"); + private static readonly DecimalImportField RateField = ImportField.Decimal("Rate"); + private static readonly DateImportField StartField = ImportField.Date("Start"); + private static readonly DateImportField SettledField = ImportField.Date("Settled"); + private static readonly BooleanImportField ActiveField = ImportField.Boolean("Active"); + + [Fact] + public void DeclaresColumnsInOrderWithTheirHeaders() + { + TabularExport export = TabularExport.For() + .Column("Id", p => p.Id) + .Column("Count", p => p.Count) + .Column("Name", p => p.Name) + .Column("Amount", p => p.Amount) + .Column("Rate", p => p.Rate) + .Column("Start", p => p.Start) + .Column("Settled", p => p.Settled) + .Column("Active", p => p.Active) + .Column("Parent", p => p.Parent) + .Build(); + + Assert.Equal(["Id", "Count", "Name", "Amount", "Rate", "Start", "Settled", "Active", "Parent"], export.Columns.Select(c => c.Header)); + } + + [Fact] + public void TakesItsHeadersFromTheImportsFields() + { + TabularExport export = TabularExport.For() + .Column(IdField, p => p.Id) + .Column(NameField, p => p.Name) + .Column(AmountField, p => p.Amount) + .Column(RateField, p => p.Rate) + .Column(StartField, p => p.Start) + .Column(SettledField, p => p.Settled) + .Column(ActiveField, p => p.Active) + .Build(); + + Assert.Equal(["Id", "Name", "Amount", "Rate", "Start", "Settled", "Active"], export.Columns.Select(c => c.Header)); + } + + [Fact] + public void WidensDateColumnsUnlessToldOtherwise() + { + TabularExport export = TabularExport.For() + .Column("Start", p => p.Start) + .Column("Settled", p => p.Settled) + .Column("Name", p => p.Name) + .Column("Narrow", p => p.Start, width: 12) + .Build(); + + Assert.Equal(new double?[] { 19, 10, null, 12 }, export.Columns.Select(c => c.Width)); + } + + [Fact] + public void RefusesATranslatedFieldsVariant() + { + ImportField variant = Assert.Single(ImportField.Translated("Title", ["en"])); + + Assert.Throws(() => TabularExport.For().Column((TextImportField)variant, p => p.Name)); + } + + [Fact] + public void RefusesAnExportWithoutColumns() + { + Assert.Throws(() => TabularExport.For().Build()); + } + + [Fact] + public void RefusesColumnsTheWriterWouldRefuseWhereTheyAreDeclared() + { + Assert.ThrowsAny(() => TabularExport.For().Column("Name", p => p.Name).Column(" name", p => p.Name).Build()); + Assert.ThrowsAny(() => TabularExport.For().Column("Name", p => p.Name).Column("NAME", p => p.Name).Build()); + Assert.ThrowsAny(() => TabularExport.For().Column("", p => p.Name).Build()); + Assert.ThrowsAny(() => TabularExport.For().Column("Name", p => p.Name, width: 0).Build()); + } + + [Fact] + public void RefusesANullHeaderOrLambda() + { + Assert.Throws(() => TabularExport.For().Column((string)null!, p => p.Name)); + Assert.Throws(() => TabularExport.For().Column("Name", (Func)null!)); + Assert.Throws(() => TabularExport.For().Column((TextImportField)null!, p => p.Name)); + } + + [Fact] + public void BuildsAnExportTheBuilderNoLongerChanges() + { + TabularExportBuilder builder = TabularExport.For().Column("Id", p => p.Id); + TabularExport export = builder.Build(); + + builder.Column("Name", p => p.Name); + + Assert.Single(export.Columns); + } +} +``` + +(`ImportField.Translated(...)` returns a `TranslatedImportField`, which enumerates its variant fields; check its element type in `src/TriasDev.Tabular/Import/TranslatedImportField.cs` and adapt the cast in `RefusesATranslatedFieldsVariant` if the variants are another typed field — the point is a field whose `Variant` is set.) + +- [ ] **Step 2: Run to verify they fail** + +Run: `dotnet test --project tests/TriasDev.Tabular.Tests --filter "FullyQualifiedName~TabularExportBuilderTests"` +Expected: build FAIL — `TabularExport` does not exist. + +- [ ] **Step 3: Move the column check into a reusable method** + +In `src/TriasDev.Tabular/Writing/TabularWriter.cs`, replace `CheckColumns` with: + +```csharp + /// + /// What is wrong with a sheet's columns, as the exception to throw, or null: 1 to 16,384 columns, + /// headers not empty, not padded, unique ignoring case and writable, widths more than 0 and at + /// most 255 characters. + /// + /// Shared with , so an export's columns fail where they are declared. + internal static Exception? ColumnsProblem(ReadOnlySpan columns) + { + if (columns.IsEmpty || columns.Length > MaxColumns) + { + return new ArgumentOutOfRangeException(nameof(columns), columns.Length, $"A sheet has 1 to {MaxColumns} columns."); + } + + HashSet seen = new(StringComparer.OrdinalIgnoreCase); + + foreach (WriteColumn column in columns) + { + if (HeaderProblem(column.Header, seen) is { } problem) + { + return new ArgumentException(problem, nameof(columns)); + } + + if (column.Width is { } width && (!double.IsFinite(width) || width <= 0 || width > MaxWidth)) + { + return new ArgumentOutOfRangeException(nameof(columns), width, $"A column is more than 0 and at most {MaxWidth} characters wide."); + } + } + + return null; + } + + private void CheckColumns(ReadOnlySpan columns) + { + if (ColumnsProblem(columns) is { } problem) + { + throw Faulting(problem); + } + } +``` + +(`Faulting` takes `T : Exception`; `throw Faulting(problem)` with `problem` typed `Exception` compiles.) + +- [ ] **Step 4: Implement the columns** + +Create `src/TriasDev.Tabular/Writing/ExportColumn.cs`: + +```csharp +namespace TriasDev.Tabular; + +/// One column of an export: its header and width, and how to write an item's value. +internal abstract class ExportColumn(WriteColumn column) +{ + public WriteColumn Column { get; } = column; + + /// Writes the item's value for this column as the writer's next cell. + public abstract void Write(TabularWriter writer, T item); +} + +/// +/// A column of a known type: the caller's lambda for the value, and a static delegate that writes a +/// value of that type — two delegate calls per cell, no boxing. +/// +internal sealed class ExportColumn(WriteColumn column, Func value, Action write) + : ExportColumn(column) +{ + public override void Write(TabularWriter writer, T item) => write(writer, value(item)); +} + +/// The typed writes, one per value type a column can have; a missing value is an empty cell. +internal static class CellWriters +{ + public static readonly Action Text = static (writer, value) => writer.Write(value); + + public static readonly Action Long = static (writer, value) => writer.Write(value); + + public static readonly Action NullableLong = static (writer, value) => + { + if (value is { } present) + { + writer.Write(present); + } + else + { + writer.WriteEmpty(); + } + }; + + public static readonly Action Decimal = static (writer, value) => writer.Write(value); + + public static readonly Action NullableDecimal = static (writer, value) => + { + if (value is { } present) + { + writer.Write(present); + } + else + { + writer.WriteEmpty(); + } + }; + + public static readonly Action Double = static (writer, value) => writer.Write(value); + + public static readonly Action NullableDouble = static (writer, value) => + { + if (value is { } present) + { + writer.Write(present); + } + else + { + writer.WriteEmpty(); + } + }; + + public static readonly Action DateTime = static (writer, value) => writer.Write(value); + + public static readonly Action NullableDateTime = static (writer, value) => + { + if (value is { } present) + { + writer.Write(present); + } + else + { + writer.WriteEmpty(); + } + }; + + public static readonly Action DateOnly = static (writer, value) => writer.Write(value); + + public static readonly Action NullableDateOnly = static (writer, value) => + { + if (value is { } present) + { + writer.Write(present); + } + else + { + writer.WriteEmpty(); + } + }; + + public static readonly Action Boolean = static (writer, value) => writer.Write(value); + + public static readonly Action NullableBoolean = static (writer, value) => + { + if (value is { } present) + { + writer.Write(present); + } + else + { + writer.WriteEmpty(); + } + }; +} +``` + +If the analyzer flags the six identical nullable bodies as duplication (S4144), replace them with one generic helper `private static void WriteOrEmpty(TabularWriter writer, TValue? value, Action write) where TValue : struct` and define each nullable delegate as `static (writer, value) => WriteOrEmpty(writer, value, Long)` and so on — same behaviour, no per-cell allocation (the delegates are static fields). + +Create `src/TriasDev.Tabular/Writing/TabularExportBuilder.cs`: + +```csharp +namespace TriasDev.Tabular; + +/// +/// Declares an export's columns, in order: a header and a lambda each, the column's type taken from +/// the lambda's. +/// +/// +/// +/// Overload resolution picks the column type: an property resolves to +/// , int? to long?, to . +/// Convert explicitly where it does not: a would resolve to and +/// write its code point; a is ambiguous; a method group returning +/// does not convert; p => null is ambiguous; an passed +/// with a is ambiguous between decimal and double. Write +/// enumerations, identifiers and other types as text: p => p.Status.ToString(). +/// +/// +/// A column named by an import field takes the field's name as its header, so a file written with +/// the export maps back onto the import's schema by header; the lambda's type must suit the field's, +/// or the call does not compile. +/// +/// +public sealed class TabularExportBuilder +{ + /// The width a date-time column gets unless told otherwise: yyyy-mm-dd hh:mm:ss. + private const double DateTimeWidth = 19; + + /// The width a date column gets unless told otherwise: yyyy-mm-dd. + private const double DateWidth = 10; + + private readonly List> _columns = []; + + internal TabularExportBuilder() + { + } + + /// A text column. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width, value, CellWriters.Text); + + /// An integer column. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width, value, CellWriters.Long); + + /// An integer column; null writes an empty cell. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width, value, CellWriters.NullableLong); + + /// A decimal column. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width, value, CellWriters.Decimal); + + /// A decimal column; null writes an empty cell. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width, value, CellWriters.NullableDecimal); + + /// A number column. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width, value, CellWriters.Double); + + /// A number column; null writes an empty cell. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width, value, CellWriters.NullableDouble); + + /// A date-time column, 19 characters wide unless told otherwise. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width ?? DateTimeWidth, value, CellWriters.DateTime); + + /// A date-time column, 19 characters wide unless told otherwise; null writes an empty cell. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width ?? DateTimeWidth, value, CellWriters.NullableDateTime); + + /// A date column, 10 characters wide unless told otherwise. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width ?? DateWidth, value, CellWriters.DateOnly); + + /// A date column, 10 characters wide unless told otherwise; null writes an empty cell. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width ?? DateWidth, value, CellWriters.NullableDateOnly); + + /// A boolean column. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width, value, CellWriters.Boolean); + + /// A boolean column; null writes an empty cell. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width, value, CellWriters.NullableBoolean); + + /// A text column named by an import field. + public TabularExportBuilder Column(TextImportField field, Func value, double? width = null) => Column(HeaderOf(field), value, width); + + /// An integer column named by an import field; null writes an empty cell. + public TabularExportBuilder Column(IntegerImportField field, Func value, double? width = null) => Column(HeaderOf(field), value, width); + + /// A decimal column named by an import field; null writes an empty cell. + public TabularExportBuilder Column(DecimalImportField field, Func value, double? width = null) => Column(HeaderOf(field), value, width); + + /// A number column named by a decimal import field; null writes an empty cell. + public TabularExportBuilder Column(DecimalImportField field, Func value, double? width = null) => Column(HeaderOf(field), value, width); + + /// A date-time column named by an import field; null writes an empty cell. + public TabularExportBuilder Column(DateImportField field, Func value, double? width = null) => Column(HeaderOf(field), value, width); + + /// A date column named by an import field; null writes an empty cell. + public TabularExportBuilder Column(DateImportField field, Func value, double? width = null) => Column(HeaderOf(field), value, width); + + /// A boolean column named by an import field; null writes an empty cell. + public TabularExportBuilder Column(BooleanImportField field, Func value, double? width = null) => Column(HeaderOf(field), value, width); + + /// + /// The export as declared so far, checked by the writer's rules for columns. The builder may go on + /// being changed; the export it returned does not change with it. + /// + /// No column was declared. + /// A header is empty, padded, repeated ignoring case or not writable, or a width is out of range. + public TabularExport Build() + { + if (_columns.Count == 0) + { + throw new InvalidOperationException("An export has at least one column."); + } + + ExportColumn[] columns = [.. _columns]; + WriteColumn[] declared = [.. columns.Select(column => column.Column)]; + + if (TabularWriter.ColumnsProblem(declared) is { } problem) + { + throw problem; + } + + return new TabularExport(columns, declared); + } + + private static string HeaderOf(ImportField field) + { + ArgumentNullException.ThrowIfNull(field); + + if (field.Variant is not null) + { + throw new ArgumentException( + $"\"{field.Name}\" is one language of a translated field; name the column with a header of its own.", + nameof(field)); + } + + return field.Name; + } + + private TabularExportBuilder Add(string header, double? width, Func value, Action write) + { + ArgumentNullException.ThrowIfNull(header); + ArgumentNullException.ThrowIfNull(value); + _columns.Add(new ExportColumn(new WriteColumn(header, width), value, write)); + return this; + } +} +``` + +Create `src/TriasDev.Tabular/Writing/TabularExport.cs`: + +```csharp +namespace TriasDev.Tabular; + +/// Where an export of objects is declared: TabularExport.For<Portfolio>().Column(…).Build(). +public static class TabularExport +{ + /// Begins declaring an export of . + public static TabularExportBuilder For() => new(); +} + +/// +/// An export of objects to a table: built once, kept in a static field, used by any number of +/// writes at once. +/// +/// Immutable and thread-safe: it holds the declared columns and nothing a write changes. +public sealed class TabularExport +{ + private readonly ExportColumn[] _columns; + private readonly WriteColumn[] _declared; + + internal TabularExport(ExportColumn[] columns, WriteColumn[] declared) + { + _columns = columns; + _declared = declared; + } + + /// The columns, in order, as a sheet's header row writes them. + public IReadOnlyList Columns => _declared; +} +``` + +`_columns` is first read by the write methods of Task 2. So the field is not unused in this commit (an analyzer error with warnings as errors), add Task 2's private `WriteRow` here already: + +```csharp + private void WriteRow(TabularWriter writer, T item) + { + writer.BeginRow(); + + foreach (ExportColumn column in _columns) + { + column.Write(writer, item); + } + + writer.EndRow(); + } +``` + +If the analyzer then flags `WriteRow` itself as an unused private method, keep it and add the justified suppression `[SuppressMessage("CodeQuality", "IDE0051", Justification = "Used by the write methods added next.")]`, which Task 2 removes. + +- [ ] **Step 5: Record the public API, run the tests** + +Append the RS0016 lines for `TabularExport`, `TabularExportBuilder` and `TabularExport` to `PublicAPI.Unshipped.txt`. + +Run: `dotnet test --project tests/TriasDev.Tabular.Tests --filter "FullyQualifiedName~TabularExportBuilderTests|FullyQualifiedName~TabularWriterTests"` +Expected: PASS — the new tests and the writer's existing column tests (`RefusesColumnsThatCouldNotBeReadBack`) unchanged. + +- [ ] **Step 6: Commit** + +```bash +dotnet format TriasDev.Tabular.slnx --verify-no-changes +git add src/TriasDev.Tabular tests/TriasDev.Tabular.Tests +git commit -m "feat(write): declare an export of objects — typed columns, headers from the import's fields" +``` + +--- + +### Task 2: Writing an export — a sheet, a file, every kind of source + +**Files:** +- Modify: `src/TriasDev.Tabular/Writing/TabularExport.cs` (the write methods; `WriteRow` already exists from Task 1 — do not add it twice, and drop any IDE0051 suppression Task 1 put on it) +- Modify: `tests/TriasDev.Tabular.Tests/Fixtures/WriteTarget.cs` (count asynchronous writes) +- Modify: `src/TriasDev.Tabular/PublicAPI.Unshipped.txt` +- Test: `tests/TriasDev.Tabular.Tests/Writing/TabularExportTests.cs` + +**Interfaces:** +- Consumes: Task 1's `TabularExport` (`_columns`, `_declared`), `TabularWriter` (`Create`, `BeginSheet`, `BeginRow`, `EndRow`, `FlushRecommended`, `FlushAsync`, `CompleteAsync`, `DisposeAsync`). +- Produces, on `TabularExport`: + - `ValueTask WriteSheetAsync(TabularWriter writer, string sheetName, IAsyncEnumerable> chunks, CancellationToken cancellationToken = default)` + - `ValueTask WriteSheetAsync(TabularWriter writer, string sheetName, IAsyncEnumerable chunks, Func> rows, CancellationToken cancellationToken = default)` + - `ValueTask WriteSheetAsync(TabularWriter writer, string sheetName, IAsyncEnumerable items, CancellationToken cancellationToken = default)` + - `ValueTask WriteSheetAsync(TabularWriter writer, string sheetName, IEnumerable items, CancellationToken cancellationToken = default)` + - the same four as `WriteAsync(Stream stream, TabularFormat format, string sheetName, , TabularWriterOptions? options = null, CancellationToken cancellationToken = default)` (the selector overload: `…, IAsyncEnumerable chunks, Func> rows, TabularWriterOptions? options = null, …`) — each creates the writer, writes one sheet, completes the file, and returns the number of data rows. + - `WriteTarget.AsyncWrites` (test fixture): how many times `WriteAsync` was called. + +- [ ] **Step 1: Count the target's writes** + +In `tests/TriasDev.Tabular.Tests/Fixtures/WriteTarget.cs`, add `public int AsyncWrites { get; private set; }` with a doc comment ("How many times an asynchronous write reached the target."), and increment it at the top of `WriteAsync(ReadOnlyMemory, CancellationToken)`, after the cancellation check. + +- [ ] **Step 2: Write the failing tests** + +Create `tests/TriasDev.Tabular.Tests/Writing/TabularExportTests.cs`: + +```csharp +using System.Runtime.CompilerServices; +using System.Text; + +using TriasDev.Tabular.Csv; +using TriasDev.Tabular.Tests.Fixtures; + +using Xunit; + +namespace TriasDev.Tabular.Tests.Writing; + +/// Writing an export: every kind of source, one sheet or a whole file, flushed as it goes. +public sealed class TabularExportTests +{ + private static CancellationToken Token => TestContext.Current.CancellationToken; + + private sealed record Item(long Id, string Name, decimal Amount); + + private static readonly TabularExport Export = TabularExport.For() + .Column("Id", i => i.Id) + .Column("Name", i => i.Name) + .Column("Amount", i => i.Amount) + .Build(); + + private static readonly TabularWriterOptions NoBom = new() { Csv = new CsvWriterOptions { ByteOrderMark = false } }; + + private static Item[] Items(int count) => [.. Enumerable.Range(1, count).Select(i => new Item(i, $"item {i}", i / 4m))]; + + private static async IAsyncEnumerable> Chunks(Item[] items, int size, [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + for (int at = 0; at < items.Length; at += size) + { + cancellationToken.ThrowIfCancellationRequested(); + await Task.Yield(); + yield return items[at..Math.Min(items.Length, at + size)]; + } + } + + private static async IAsyncEnumerable OneByOne(Item[] items) + { + foreach (Item item in items) + { + await Task.Yield(); + yield return item; + } + } + + private static string Csv(WriteTarget target) => Encoding.UTF8.GetString(target.ToArray()); + + [Fact] + public async Task WritesAHeaderAndARowPerItem() + { + WriteTarget target = new(); + + long rows = await Export.WriteAsync(target, TabularFormat.Csv, "data", Items(2), NoBom, Token); + + Assert.Equal(2, rows); + Assert.Equal("Id,Name,Amount\r\n1,item 1,0.25\r\n2,item 2,0.5\r\n", Csv(target)); + Assert.True(target.IsDisposed); + Assert.False(target.DisposedSynchronously); + } + + [Fact] + public async Task EveryKindOfSourceWritesTheSameFile() + { + Item[] items = Items(5_000); + + WriteTarget chunked = new(); + WriteTarget streamed = new(); + WriteTarget listed = new(); + WriteTarget selected = new(); + + Assert.Equal(5_000, await Export.WriteAsync(chunked, TabularFormat.Csv, "data", Chunks(items, 700), NoBom, Token)); + Assert.Equal(5_000, await Export.WriteAsync(streamed, TabularFormat.Csv, "data", OneByOne(items), NoBom, Token)); + Assert.Equal(5_000, await Export.WriteAsync(listed, TabularFormat.Csv, "data", items, NoBom, Token)); + Assert.Equal(5_000, await Export.WriteAsync(selected, TabularFormat.Csv, "data", Chunks(items, 700), chunk => chunk, NoBom, Token)); + + string expected = Csv(listed); + Assert.Equal(expected, Csv(chunked)); + Assert.Equal(expected, Csv(streamed)); + Assert.Equal(expected, Csv(selected)); + } + + [Fact] + public async Task FlushesAfterEveryChunk() + { + WriteTarget target = new(); + + await Export.WriteAsync(target, TabularFormat.Csv, "data", Chunks(Items(1_000), 100), NoBom, Token); + + // Ten chunks, each flushed as it is written, and the file's completion. + Assert.True(target.AsyncWrites >= 10, $"{target.AsyncWrites} writes reached the target"); + } + + [Fact] + public async Task FlushesInsideALargeChunkWhenTheWriterRecommendsIt() + { + WriteTarget target = new(); + Item[] items = [.. Enumerable.Range(1, 60_000).Select(i => new Item(i, new string('x', 40), i))]; + + await Export.WriteAsync(target, TabularFormat.Csv, "data", Chunks(items, items.Length), NoBom, Token); + + Assert.True(target.AsyncWrites >= 3, $"{target.AsyncWrites} writes reached the target for one 3 MB chunk"); + } + + [Fact] + public async Task WritesSeveralSheetsIntoOneWorkbook() + { + WriteTarget target = new(); + + await using (TabularWriter writer = TabularWriter.Create(target, TabularFormat.Xlsx)) + { + Assert.Equal(3, await Export.WriteSheetAsync(writer, "First", Items(3), Token)); + Assert.Equal(2, await Export.WriteSheetAsync(writer, "Second", Chunks(Items(2), 1), Token)); + await writer.CompleteAsync(Token); + } + + using ITabularCursor cursor = TabularFile.Open(new MemoryStream(target.ToArray(), writable: false), "t.xlsx", cancellationToken: Token); + Assert.Equal(["First", "Second"], cursor.Sheets.Select(s => s.Name)); + } + + [Fact] + public async Task AValueTheFormatRefusesClosesTheStreamAndLeavesNoValidFile() + { + TabularExport export = TabularExport.For().Column("Id", i => i.Id).Column("Amount", i => i.Amount).Build(); + Item[] items = [.. Items(50_000)]; + items[39_999] = items[39_999] with { Amount = 1_234_567_890_123.456m }; + WriteTarget target = new(); + + TabularWriteException refused = await Assert.ThrowsAsync(async () => + await export.WriteAsync(target, TabularFormat.Xlsx, "data", Chunks(items, 10_000), cancellationToken: Token)); + + Assert.Equal(ErrorCodes.Write.PrecisionLoss, refused.Code); + Assert.Equal(40_001, refused.RowNumber); + Assert.Equal("Amount", refused.Header); + Assert.True(target.IsDisposed); + Assert.False(target.DisposedSynchronously); + Assert.ThrowsAny(() => new TriasDev.Tabular.Xlsx.XlsxCursor(new MemoryStream(target.ToArray(), writable: false), cancellationToken: Token)); + } + + [Fact] + public async Task CancellationStopsBetweenChunksAndClosesTheStream() + { + using CancellationTokenSource cancel = CancellationTokenSource.CreateLinkedTokenSource(Token); + WriteTarget target = new(); + int chunksSeen = 0; + + async IAsyncEnumerable> Source([EnumeratorCancellation] CancellationToken cancellationToken = default) + { + await foreach (IReadOnlyList chunk in Chunks(Items(1_000), 100, cancellationToken)) + { + if (++chunksSeen == 3) + { + await cancel.CancelAsync(); + } + + yield return chunk; + } + } + + await Assert.ThrowsAnyAsync(async () => + await Export.WriteAsync(target, TabularFormat.Csv, "data", Source(), NoBom, cancel.Token)); + + Assert.True(chunksSeen < 10); + Assert.True(target.IsDisposed); + } + + [Fact] + public async Task LeavesTheStreamOpenWhenAsked() + { + WriteTarget target = new(); + + await Export.WriteAsync(target, TabularFormat.Csv, "data", Items(1), new TabularWriterOptions { LeaveOpen = true }, Token); + + Assert.False(target.IsDisposed); + } + + [Fact] + public async Task RefusesANullSourceAndStillClosesTheStream() + { + WriteTarget target = new(); + + await Assert.ThrowsAsync(async () => + await Export.WriteAsync(target, TabularFormat.Csv, "data", (IEnumerable)null!, cancellationToken: Token)); + + Assert.True(target.IsDisposed); + } + + [Fact] + public async Task RefusesANullChunk() + { + static async IAsyncEnumerable> WithANullChunk() + { + await Task.Yield(); + yield return null!; + } + + await Assert.ThrowsAsync(async () => + await Export.WriteAsync(new WriteTarget(), TabularFormat.Csv, "data", WithANullChunk(), cancellationToken: Token)); + } +} +``` + +- [ ] **Step 3: Run to verify they fail** + +Run: `dotnet test --project tests/TriasDev.Tabular.Tests --filter "FullyQualifiedName~TabularExportTests"` +Expected: build FAIL — no `WriteAsync`. + +- [ ] **Step 4: Implement** + +In `src/TriasDev.Tabular/Writing/TabularExport.cs`, add `using System.Runtime.CompilerServices;` only if needed, and to `TabularExport`: + +```csharp + /// Writes a file of one sheet from chunks as they arrive — the fast source for millions of rows. + /// The number of data rows written. + /// + /// The writer is flushed after every chunk and inside a chunk whenever it recommends it, so memory + /// holds a chunk and about a megabyte, however many rows the file has. On any failure the stream is + /// closed (unless ) and the file is incomplete. + /// + public ValueTask WriteAsync(Stream stream, TabularFormat format, string sheetName, IAsyncEnumerable> chunks, TabularWriterOptions? options = null, CancellationToken cancellationToken = default) => + WriteFileAsync(stream, format, options, writer => WriteSheetAsync(writer, sheetName, chunks, cancellationToken), cancellationToken); + + /// + /// Writes a file of one sheet from a stream of messages that each carry a chunk — a gRPC server + /// stream whose messages hold a repeated field, passed as it comes. + /// + /// The number of data rows written. + public ValueTask WriteAsync(Stream stream, TabularFormat format, string sheetName, IAsyncEnumerable chunks, Func> rows, TabularWriterOptions? options = null, CancellationToken cancellationToken = default) => + WriteFileAsync(stream, format, options, writer => WriteSheetAsync(writer, sheetName, chunks, rows, cancellationToken), cancellationToken); + + /// Writes a file of one sheet from items arriving one at a time; awaits per item, so prefer chunks for millions. + /// The number of data rows written. + public ValueTask WriteAsync(Stream stream, TabularFormat format, string sheetName, IAsyncEnumerable items, TabularWriterOptions? options = null, CancellationToken cancellationToken = default) => + WriteFileAsync(stream, format, options, writer => WriteSheetAsync(writer, sheetName, items, cancellationToken), cancellationToken); + + /// Writes a file of one sheet from items in memory or produced synchronously. + /// The number of data rows written. + /// Cancellation is observed at each flush, about every megabyte written. + public ValueTask WriteAsync(Stream stream, TabularFormat format, string sheetName, IEnumerable items, TabularWriterOptions? options = null, CancellationToken cancellationToken = default) => + WriteFileAsync(stream, format, options, writer => WriteSheetAsync(writer, sheetName, items, cancellationToken), cancellationToken); + + /// Writes one sheet into a writer the caller owns, from chunks as they arrive. + /// The number of data rows written. + /// The caller completes the writer; several exports may write their sheets into one workbook. + public async ValueTask WriteSheetAsync(TabularWriter writer, string sheetName, IAsyncEnumerable> chunks, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(writer); + ArgumentNullException.ThrowIfNull(chunks); + writer.BeginSheet(sheetName, _declared); + long rows = 0; + + await foreach (IReadOnlyList chunk in chunks.WithCancellation(cancellationToken).ConfigureAwait(false)) + { + rows += await WriteChunkAsync(writer, chunk, cancellationToken).ConfigureAwait(false); + } + + return rows; + } + + /// Writes one sheet into a writer the caller owns, from messages that each carry a chunk. + /// The number of data rows written. + public async ValueTask WriteSheetAsync(TabularWriter writer, string sheetName, IAsyncEnumerable chunks, Func> rows, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(writer); + ArgumentNullException.ThrowIfNull(chunks); + ArgumentNullException.ThrowIfNull(rows); + writer.BeginSheet(sheetName, _declared); + long written = 0; + + await foreach (TChunk message in chunks.WithCancellation(cancellationToken).ConfigureAwait(false)) + { + written += await WriteChunkAsync(writer, rows(message), cancellationToken).ConfigureAwait(false); + } + + return written; + } + + /// Writes one sheet into a writer the caller owns, from items arriving one at a time. + /// The number of data rows written. + public async ValueTask WriteSheetAsync(TabularWriter writer, string sheetName, IAsyncEnumerable items, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(writer); + ArgumentNullException.ThrowIfNull(items); + writer.BeginSheet(sheetName, _declared); + long rows = 0; + + await foreach (T item in items.WithCancellation(cancellationToken).ConfigureAwait(false)) + { + WriteRow(writer, item); + rows++; + + if (writer.FlushRecommended) + { + await writer.FlushAsync(cancellationToken).ConfigureAwait(false); + } + } + + return rows; + } + + /// Writes one sheet into a writer the caller owns, from items in memory or produced synchronously. + /// The number of data rows written. + public async ValueTask WriteSheetAsync(TabularWriter writer, string sheetName, IEnumerable items, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(writer); + ArgumentNullException.ThrowIfNull(items); + writer.BeginSheet(sheetName, _declared); + long rows = 0; + + foreach (T item in items) + { + WriteRow(writer, item); + rows++; + + if (writer.FlushRecommended) + { + await writer.FlushAsync(cancellationToken).ConfigureAwait(false); + } + } + + return rows; + } + + /// + /// Creates the writer, lets the sheet be written, completes the file. Arguments are checked inside + /// the writer's lifetime, so a refused one still closes the stream. + /// + private static async ValueTask WriteFileAsync(Stream stream, TabularFormat format, TabularWriterOptions? options, Func> write, CancellationToken cancellationToken) + { + TabularWriter writer = TabularWriter.Create(stream, format, options); + + await using (writer.ConfigureAwait(false)) + { + long rows = await write(writer).ConfigureAwait(false); + await writer.CompleteAsync(cancellationToken).ConfigureAwait(false); + return rows; + } + } + + /// Writes a chunk's rows, flushing inside it when recommended and once after it. + private async ValueTask WriteChunkAsync(TabularWriter writer, IReadOnlyList chunk, CancellationToken cancellationToken) + { + if (chunk is null) + { + throw new ArgumentException("A chunk is null.", nameof(chunk)); + } + + for (int i = 0; i < chunk.Count; i++) + { + WriteRow(writer, chunk[i]); + + if (writer.FlushRecommended) + { + await writer.FlushAsync(cancellationToken).ConfigureAwait(false); + } + } + + await writer.FlushAsync(cancellationToken).ConfigureAwait(false); + return chunk.Count; + } + + private void WriteRow(TabularWriter writer, T item) + { + writer.BeginRow(); + + foreach (ExportColumn column in _columns) + { + column.Write(writer, item); + } + + writer.EndRow(); + } +``` + +If the analyzer asks for the `ArgumentException` parameter name to match a parameter (S3928 / CA2208), use the same justified pragma the options classes use, or throw `InvalidOperationException("A chunk is null.")` — and update `RefusesANullChunk` to `ThrowsAnyAsync`'s narrower match accordingly (state which in the report). + +- [ ] **Step 5: Record the public API, run the tests** + +Append the RS0016 lines for the eight write methods. + +Run: `dotnet test --project tests/TriasDev.Tabular.Tests --filter "FullyQualifiedName~TabularExport"` +Expected: PASS on both targets. + +- [ ] **Step 6: Commit** + +```bash +dotnet format TriasDev.Tabular.slnx --verify-no-changes +git add src/TriasDev.Tabular tests/TriasDev.Tabular.Tests +git commit -m "feat(write): write an export from chunks, a message stream, an async or a plain sequence — flushed as it goes" +``` + +--- + +### Task 3: The export round-trips through the import — every format, flat memory, concurrent use + +**Files:** +- Test: `tests/TriasDev.Tabular.Tests/Writing/TabularExportRoundTripTests.cs` +- Modify: `docs/KNOWN-ISSUES.md` (one line: the export's overload pitfalls) + +**Interfaces:** +- Consumes: Tasks 1–2; `TabularImporter.Import(Stream, string, MappingPlan, ImportSchema, TabularRowMapper, ImportOptions?, CancellationToken)`, `ImportRun.ReadRows`, typed `ImportRow` indexers. +- Produces: the proof that an export declared with the import's own fields writes files the import maps back by header, in csv, xlsx and ods; that writing allocates nothing per row; that one export serves concurrent writes. + +- [ ] **Step 1: Write the tests** + +Create `tests/TriasDev.Tabular.Tests/Writing/TabularExportRoundTripTests.cs`: + +```csharp +using TriasDev.Tabular.Tests.Fixtures; + +using Xunit; + +namespace TriasDev.Tabular.Tests.Writing; + +/// An export declared with the import's fields writes files the import maps back by header. +public sealed class TabularExportRoundTripTests +{ + private static readonly IntegerImportField IdField = ImportField.Integer("Id"); + private static readonly TextImportField NameField = ImportField.Text("Name"); + private static readonly DecimalImportField AmountField = ImportField.Decimal("Amount"); + private static readonly DecimalImportField RateField = ImportField.Decimal("Rate"); + private static readonly DateImportField StartField = ImportField.Date("Start"); + private static readonly DateImportField SettledField = ImportField.Date("Settled"); + private static readonly BooleanImportField ActiveField = ImportField.Boolean("Active"); + + private static readonly ImportSchema Schema = new() { Fields = [IdField, NameField, AmountField, RateField, StartField, SettledField, ActiveField] }; + + private sealed record Portfolio(long Id, string? Name, decimal? Amount, double? Rate, DateTime Start, DateOnly? Settled, bool Active); + + private sealed record Batch(List Items); + + private static readonly TabularExport Export = TabularExport.For() + .Column(IdField, p => p.Id) + .Column(NameField, p => p.Name) + .Column(AmountField, p => p.Amount) + .Column(RateField, p => p.Rate) + .Column(StartField, p => p.Start) + .Column(SettledField, p => p.Settled) + .Column(ActiveField, p => p.Active) + .Build(); + + private static CancellationToken Token => TestContext.Current.CancellationToken; + + public static TheoryData Formats => new() + { + { TabularFormat.Csv, "t.csv" }, + { TabularFormat.Xlsx, "t.xlsx" }, + { TabularFormat.Ods, "t.ods" }, + }; + + private static DateTime At(int year, int month, int day, int hour = 0, int minute = 0, int second = 0, int millisecond = 0) => + new(year, month, day, hour, minute, second, millisecond, DateTimeKind.Unspecified); + + private static Portfolio[] Portfolios(int count) => + [ + .. Enumerable.Range(1, count).Select(i => new Portfolio( + i, + i % 7 == 0 ? null : $"Portfolio {i}, \"quoted\"", + i % 5 == 0 ? null : i * 1.25m, + i % 3 == 0 ? null : i / 8d, + At(2026, 1 + (i % 12), 1 + (i % 28), i % 24, i % 60, i % 60, i % 1000), + i % 4 == 0 ? null : new DateOnly(2026, 1 + (i % 12), 1 + (i % 28)), + i % 2 == 0)), + ]; + + private static async IAsyncEnumerable Batches(Portfolio[] items, int size) + { + for (int at = 0; at < items.Length; at += size) + { + await Task.Yield(); + yield return new Batch([.. items[at..Math.Min(items.Length, at + size)]]); + } + } + + private static List Import(byte[] file, string name) + { + using ITabularCursor cursor = TabularFile.Open(new MemoryStream(file, writable: false), name, cancellationToken: Token); + FileProfile profile = TabularAnalyzer.Analyze(cursor, cancellationToken: Token); + MappingPlan plan = MappingPlan.ByHeader(profile.Sheets[0], Schema); + + using ImportRun run = TabularImporter.Import( + new MemoryStream(file, writable: false), + name, + plan, + Schema, + row => new Portfolio(row[IdField]!.Value, row[NameField], row[AmountField], row[RateField] is { } rate ? (double)rate : null, row[StartField]!.Value, row[SettledField] is { } settled ? DateOnly.FromDateTime(settled) : null, row[ActiveField]!.Value), + cancellationToken: Token); + + List> outcomes = [.. run.ReadRows(Token)]; + Assert.All(outcomes, outcome => Assert.False(outcome.HasErrors, string.Join(", ", outcome.Errors.Select(e => e.Code)))); + return [.. outcomes.Select(outcome => outcome.Value!)]; + } + + [Theory] + [MemberData(nameof(Formats))] + public async Task MapsBackByHeaderAsWritten(TabularFormat format, string name) + { + Portfolio[] written = Portfolios(500); + WriteTarget target = new(); + + await Export.WriteAsync(target, format, "Portfolios", written, cancellationToken: Token); + + Assert.Equal(written, Import(target.ToArray(), name)); + } + + [Theory] + [MemberData(nameof(Formats))] + public async Task WritesChunksTakenFromMessages(TabularFormat format, string name) + { + Portfolio[] written = Portfolios(2_345); + WriteTarget target = new(); + + long rows = await Export.WriteAsync(target, format, "Portfolios", Batches(written, 1_000), batch => batch.Items, cancellationToken: Token); + + Assert.Equal(written.Length, rows); + Assert.Equal(written, Import(target.ToArray(), name)); + } + + [Theory] + [MemberData(nameof(Formats))] + public async Task OneExportServesConcurrentWrites(TabularFormat format, string name) + { + Portfolio[] first = Portfolios(3_000); + Portfolio[] second = [.. Portfolios(3_000).Select(p => p with { Name = "other " + p.Id })]; + WriteTarget one = new(); + WriteTarget two = new(); + + await Task.WhenAll( + Task.Run(async () => await Export.WriteAsync(one, format, "a", Batches(first, 100), batch => batch.Items, cancellationToken: Token), Token), + Task.Run(async () => await Export.WriteAsync(two, format, "b", Batches(second, 100), batch => batch.Items, cancellationToken: Token), Token)); + + Assert.Equal(first, Import(one.ToArray(), name)); + Assert.Equal(second, Import(two.ToArray(), name)); + } + + [Theory] + [InlineData(TabularFormat.Csv)] + [InlineData(TabularFormat.Xlsx)] + [InlineData(TabularFormat.Ods)] + public async Task AllocatesNothingPerRow(TabularFormat format) + { + // One reused chunk, a source that completes synchronously, and Stream.Null: every await stays + // on this thread, so the thread's allocation counter sees the whole write and nothing else. + Portfolio[] chunk = Portfolios(10_000); + + async ValueTask Allocated(int chunks) + { + long before = GC.GetAllocatedBytesForCurrentThread(); + await Export.WriteAsync(Stream.Null, format, "data", Repeat(chunk, chunks), new TabularWriterOptions { LeaveOpen = true }, Token); + return GC.GetAllocatedBytesForCurrentThread() - before; + } + + await Allocated(2); // warm-up: static state, pools, JIT + long tenThousandRows = await Allocated(1); + long millionRows = await Allocated(100); + + // A million rows against ten thousand: what grows with the row count is under a byte a row. + Assert.True(millionRows - tenThousandRows < 1_000_000, $"{format}: {tenThousandRows:N0} bytes for 10k rows, {millionRows:N0} for 1M"); + } + + private static async IAsyncEnumerable> Repeat(Portfolio[] chunk, int times) + { + for (int i = 0; i < times; i++) + { + yield return chunk; + } + + await Task.CompletedTask; + } +} +``` + +The `Import` helper maps by header through analysis (`MappingPlan.ByHeader`), proving the headers the export took from the fields are the ones the schema expects. A `Rate` written as a double is read back by the decimal field and converted here, so `Rate` values must be exact in both (they are eighths). If `MappingPlan.ByHeader`'s signature differs (check `src/TriasDev.Tabular/Mapping/MappingPlan.cs`), adapt only the call. + +If `AllocatesNothingPerRow` fails, it is a finding: report the measured bytes per format and do not loosen the bound. If the async source does not complete synchronously on some runtime (the counter then misses work on other threads, which would make the test pass vacuously — check that `tenThousandRows` is well above zero, e.g. at least the bytes of the writer's buffers, and report the numbers). + +- [ ] **Step 2: Run them** + +Run: `dotnet test --project tests/TriasDev.Tabular.Tests --filter "FullyQualifiedName~TabularExportRoundTripTests"` +Expected: PASS. A failing round trip is a finding: report the format, the row, the value written and read; do not change the writer, the reader or an expectation. + +- [ ] **Step 3: Note the overload pitfalls** + +In `docs/KNOWN-ISSUES.md`, writing section, append: `TabularExport: a char property resolves to an integer column (its code point), ulong and p => null are ambiguous, an int with a DecimalImportField is ambiguous between decimal and double — convert explicitly; write enums and identifiers as text.` + +- [ ] **Step 4: Run the whole suite and the Release build** + +Run: `dotnet test --solution TriasDev.Tabular.slnx`, `dotnet build TriasDev.Tabular.slnx -c Release`, `dotnet format TriasDev.Tabular.slnx --verify-no-changes`. +Expected: all green, 0 warnings, no format changes. + +- [ ] **Step 5: Commit** + +```bash +git add tests docs/KNOWN-ISSUES.md +git commit -m "test(write): an export maps back by header in every format, allocates nothing per row, serves concurrent writes" +``` + +--- + +## After the last task + +One pull request for part 4 (`feat(write): TabularExport — export objects by declaring their columns once`), body referencing #73 and #67; merge when CI is green. Then part 5 (#74): benchmarks, comparison, `docs/exporting.md`, ADR-0002, and the wide-sheet decision noted in the project memory. diff --git a/docs/superpowers/specs/2026-10-03-writing-design.md b/docs/superpowers/specs/2026-10-03-writing-design.md index e658ec5..a73a48b 100644 --- a/docs/superpowers/specs/2026-10-03-writing-design.md +++ b/docs/superpowers/specs/2026-10-03-writing-design.md @@ -206,10 +206,16 @@ static readonly TabularExport Export = TabularExport.For() `.Column(DecimalImportField, Func)` and `Func`, `.Column(DateImportField, Func)` and `Func`, `.Column(BooleanImportField, Func)`. A mismatch is a compile error. The header is - `field.Name`. An untyped `ImportField` is accepted too and checked against its `Type` at - `Build()`, with `ArgumentException`; a translated field throws there. -- Each column is an `ExportColumn` holding a typed delegate: one delegate call and one - typed `Write` per cell. + `field.Name`. An untyped `ImportField` is not accepted: an overload for it would also catch a + typed field passed with a lambda of the wrong type, turning the compile error into a run-time one; + a caller with an untyped field writes `.Column(field.Name, …)`. A translated field's variant is + refused at `Column(...)` with `ArgumentException`. +- Sources may also be a stream of messages that each carry a chunk, with a selector + (`IAsyncEnumerable` and `Func>`) — a gRPC server stream whose + messages hold a repeated field, passed as it comes. +- Each column is an `ExportColumn` holding a typed delegate: per cell, one virtual call + and two delegate calls (the accessor and the typed `Write`), no allocation; measured at 1–6% over + a hand-written loop. - Width: set per column, or defaulted by type (wider for date-time). - The built export is immutable and thread-safe. diff --git a/src/TriasDev.Tabular/PublicAPI.Unshipped.txt b/src/TriasDev.Tabular/PublicAPI.Unshipped.txt index 55d5731..feaafe3 100644 --- a/src/TriasDev.Tabular/PublicAPI.Unshipped.txt +++ b/src/TriasDev.Tabular/PublicAPI.Unshipped.txt @@ -121,3 +121,37 @@ static TriasDev.Tabular.Ods.OdsWriterOptions.Default.get -> TriasDev.Tabular.Ods static TriasDev.Tabular.Ods.OdsWriterOptions.operator !=(TriasDev.Tabular.Ods.OdsWriterOptions? left, TriasDev.Tabular.Ods.OdsWriterOptions? right) -> bool static TriasDev.Tabular.Ods.OdsWriterOptions.operator ==(TriasDev.Tabular.Ods.OdsWriterOptions? left, TriasDev.Tabular.Ods.OdsWriterOptions? right) -> bool TriasDev.Tabular.TabularFormat.Tar = 5 -> TriasDev.Tabular.TabularFormat +static TriasDev.Tabular.TabularExport.For() -> TriasDev.Tabular.TabularExportBuilder! +TriasDev.Tabular.TabularExport +TriasDev.Tabular.TabularExport +TriasDev.Tabular.TabularExport.Columns.get -> System.Collections.Generic.IReadOnlyList! +TriasDev.Tabular.TabularExportBuilder +TriasDev.Tabular.TabularExportBuilder.Build() -> TriasDev.Tabular.TabularExport! +TriasDev.Tabular.TabularExportBuilder.Column(string! header, System.Func! value, double? width = null) -> TriasDev.Tabular.TabularExportBuilder! +TriasDev.Tabular.TabularExportBuilder.Column(string! header, System.Func! value, double? width = null) -> TriasDev.Tabular.TabularExportBuilder! +TriasDev.Tabular.TabularExportBuilder.Column(string! header, System.Func! value, double? width = null) -> TriasDev.Tabular.TabularExportBuilder! +TriasDev.Tabular.TabularExportBuilder.Column(string! header, System.Func! value, double? width = null) -> TriasDev.Tabular.TabularExportBuilder! +TriasDev.Tabular.TabularExportBuilder.Column(string! header, System.Func! value, double? width = null) -> TriasDev.Tabular.TabularExportBuilder! +TriasDev.Tabular.TabularExportBuilder.Column(string! header, System.Func! value, double? width = null) -> TriasDev.Tabular.TabularExportBuilder! +TriasDev.Tabular.TabularExportBuilder.Column(string! header, System.Func! value, double? width = null) -> TriasDev.Tabular.TabularExportBuilder! +TriasDev.Tabular.TabularExportBuilder.Column(string! header, System.Func! value, double? width = null) -> TriasDev.Tabular.TabularExportBuilder! +TriasDev.Tabular.TabularExportBuilder.Column(string! header, System.Func! value, double? width = null) -> TriasDev.Tabular.TabularExportBuilder! +TriasDev.Tabular.TabularExportBuilder.Column(string! header, System.Func! value, double? width = null) -> TriasDev.Tabular.TabularExportBuilder! +TriasDev.Tabular.TabularExportBuilder.Column(string! header, System.Func! value, double? width = null) -> TriasDev.Tabular.TabularExportBuilder! +TriasDev.Tabular.TabularExportBuilder.Column(string! header, System.Func! value, double? width = null) -> TriasDev.Tabular.TabularExportBuilder! +TriasDev.Tabular.TabularExportBuilder.Column(string! header, System.Func! value, double? width = null) -> TriasDev.Tabular.TabularExportBuilder! +TriasDev.Tabular.TabularExportBuilder.Column(TriasDev.Tabular.BooleanImportField! field, System.Func! value, double? width = null) -> TriasDev.Tabular.TabularExportBuilder! +TriasDev.Tabular.TabularExportBuilder.Column(TriasDev.Tabular.DateImportField! field, System.Func! value, double? width = null) -> TriasDev.Tabular.TabularExportBuilder! +TriasDev.Tabular.TabularExportBuilder.Column(TriasDev.Tabular.DateImportField! field, System.Func! value, double? width = null) -> TriasDev.Tabular.TabularExportBuilder! +TriasDev.Tabular.TabularExportBuilder.Column(TriasDev.Tabular.DecimalImportField! field, System.Func! value, double? width = null) -> TriasDev.Tabular.TabularExportBuilder! +TriasDev.Tabular.TabularExportBuilder.Column(TriasDev.Tabular.DecimalImportField! field, System.Func! value, double? width = null) -> TriasDev.Tabular.TabularExportBuilder! +TriasDev.Tabular.TabularExportBuilder.Column(TriasDev.Tabular.IntegerImportField! field, System.Func! value, double? width = null) -> TriasDev.Tabular.TabularExportBuilder! +TriasDev.Tabular.TabularExportBuilder.Column(TriasDev.Tabular.TextImportField! field, System.Func! value, double? width = null) -> TriasDev.Tabular.TabularExportBuilder! +TriasDev.Tabular.TabularExport.WriteAsync(System.IO.Stream! stream, TriasDev.Tabular.TabularFormat format, string! sheetName, System.Collections.Generic.IAsyncEnumerable!>! chunks, TriasDev.Tabular.TabularWriterOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.ValueTask +TriasDev.Tabular.TabularExport.WriteAsync(System.IO.Stream! stream, TriasDev.Tabular.TabularFormat format, string! sheetName, System.Collections.Generic.IAsyncEnumerable! items, TriasDev.Tabular.TabularWriterOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.ValueTask +TriasDev.Tabular.TabularExport.WriteAsync(System.IO.Stream! stream, TriasDev.Tabular.TabularFormat format, string! sheetName, System.Collections.Generic.IEnumerable! items, TriasDev.Tabular.TabularWriterOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.ValueTask +TriasDev.Tabular.TabularExport.WriteAsync(System.IO.Stream! stream, TriasDev.Tabular.TabularFormat format, string! sheetName, System.Collections.Generic.IAsyncEnumerable! chunks, System.Func!>! itemsOf, TriasDev.Tabular.TabularWriterOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.ValueTask +TriasDev.Tabular.TabularExport.WriteSheetAsync(TriasDev.Tabular.TabularWriter! writer, string! sheetName, System.Collections.Generic.IAsyncEnumerable!>! chunks, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.ValueTask +TriasDev.Tabular.TabularExport.WriteSheetAsync(TriasDev.Tabular.TabularWriter! writer, string! sheetName, System.Collections.Generic.IAsyncEnumerable! items, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.ValueTask +TriasDev.Tabular.TabularExport.WriteSheetAsync(TriasDev.Tabular.TabularWriter! writer, string! sheetName, System.Collections.Generic.IEnumerable! items, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.ValueTask +TriasDev.Tabular.TabularExport.WriteSheetAsync(TriasDev.Tabular.TabularWriter! writer, string! sheetName, System.Collections.Generic.IAsyncEnumerable! chunks, System.Func!>! itemsOf, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.ValueTask diff --git a/src/TriasDev.Tabular/Writing/ExportColumn.cs b/src/TriasDev.Tabular/Writing/ExportColumn.cs new file mode 100644 index 0000000..359f3c5 --- /dev/null +++ b/src/TriasDev.Tabular/Writing/ExportColumn.cs @@ -0,0 +1,110 @@ +namespace TriasDev.Tabular; + +/// One column of an export: its header and width, and how to write an item's value. +internal abstract class ExportColumn(WriteColumn column) +{ + public WriteColumn Column { get; } = column; + + /// Writes the item's value for this column as the writer's next cell. + public abstract void Write(TabularWriter writer, T item); +} + +/// +/// A column of a known type: the caller's lambda for the value, and a static delegate that writes a +/// value of that type — two delegate calls per cell, no boxing. +/// +internal sealed class ExportColumn(WriteColumn column, Func value, Action write) + : ExportColumn(column) +{ + public override void Write(TabularWriter writer, T item) => write(writer, value(item)); +} + +/// The typed writes, one per value type a column can have; a missing value is an empty cell. +internal static class CellWriters +{ + public static readonly Action Text = static (writer, value) => writer.Write(value); + + public static readonly Action Long = static (writer, value) => writer.Write(value); + + public static readonly Action NullableLong = static (writer, value) => + { + if (value is { } present) + { + writer.Write(present); + } + else + { + writer.WriteEmpty(); + } + }; + + public static readonly Action Decimal = static (writer, value) => writer.Write(value); + + public static readonly Action NullableDecimal = static (writer, value) => + { + if (value is { } present) + { + writer.Write(present); + } + else + { + writer.WriteEmpty(); + } + }; + + public static readonly Action Double = static (writer, value) => writer.Write(value); + + public static readonly Action NullableDouble = static (writer, value) => + { + if (value is { } present) + { + writer.Write(present); + } + else + { + writer.WriteEmpty(); + } + }; + + public static readonly Action DateTime = static (writer, value) => writer.Write(value); + + public static readonly Action NullableDateTime = static (writer, value) => + { + if (value is { } present) + { + writer.Write(present); + } + else + { + writer.WriteEmpty(); + } + }; + + public static readonly Action DateOnly = static (writer, value) => writer.Write(value); + + public static readonly Action NullableDateOnly = static (writer, value) => + { + if (value is { } present) + { + writer.Write(present); + } + else + { + writer.WriteEmpty(); + } + }; + + public static readonly Action Boolean = static (writer, value) => writer.Write(value); + + public static readonly Action NullableBoolean = static (writer, value) => + { + if (value is { } present) + { + writer.Write(present); + } + else + { + writer.WriteEmpty(); + } + }; +} diff --git a/src/TriasDev.Tabular/Writing/TabularExport.cs b/src/TriasDev.Tabular/Writing/TabularExport.cs new file mode 100644 index 0000000..4caf2c0 --- /dev/null +++ b/src/TriasDev.Tabular/Writing/TabularExport.cs @@ -0,0 +1,361 @@ +namespace TriasDev.Tabular; + +/// Where an export of objects is declared: TabularExport.For<Portfolio>().Column(…).Build(). +public static class TabularExport +{ + /// Begins declaring an export of . + public static TabularExportBuilder For() => new(); +} + +/// +/// An export of objects to a table: built once, kept in a static field, used by any number of +/// writes at once. +/// +/// +/// Immutable and thread-safe: it holds the declared columns and nothing a write changes. +/// A source that is both and (an EF Core +/// DbSet, for one) is ambiguous: pass .AsAsyncEnumerable(). A literal null for the +/// options with a chunk source is ambiguous with the selector overload: write options: null or +/// omit it. Each chunk is flushed, so very small chunks mean one write to the target each. +/// +public sealed class TabularExport +{ + private readonly ExportColumn[] _columns; + private readonly WriteColumn[] _declared; + private readonly IReadOnlyList _columnsView; + + internal TabularExport(ExportColumn[] columns, WriteColumn[] declared) + { + _columns = columns; + _declared = declared; + _columnsView = Array.AsReadOnly(declared); + } + + /// The columns, in order, as a sheet's header row writes them. + public IReadOnlyList Columns => _columnsView; + + /// Writes a file of one sheet from chunks as they arrive — the fast source for millions of rows. + /// The stream the file is written to; closed at the end, unless . + /// The file's format. + /// The sheet's name; a csv file's single sheet takes no name in the file, but the name must still be valid. + /// The rows, a batch at a time. + /// Options of the writer, or for the defaults. + /// Stops the write; also flows into the enumeration of . + /// The number of data rows written. + /// + /// The writer is flushed after every chunk and inside a chunk whenever it recommends it, so memory + /// holds a chunk and about a megabyte, however many rows the file has. On any failure the stream is + /// closed (unless ) and the file is incomplete. + /// An xlsx or ods sheet holds at most 1,048,576 rows, the header included; the row past it throws + /// after the rows before it were written — choose csv when the + /// count may exceed it. + /// + /// A value the format cannot hold. + /// An xlsx or ods sheet outgrew its row limit. + /// was canceled. + /// An argument is . + /// A chunk is . + /// The writer is not in a state to begin a sheet. + public ValueTask WriteAsync(Stream stream, TabularFormat format, string sheetName, IAsyncEnumerable> chunks, TabularWriterOptions? options = null, CancellationToken cancellationToken = default) => + WriteFileAsync(stream, format, options, writer => WriteSheetAsync(writer, sheetName, chunks, cancellationToken), cancellationToken); + + /// + /// Writes a file of one sheet from a stream of messages that each carry a chunk — a gRPC server + /// stream whose messages hold a repeated field, passed as it comes. + /// + /// The message type. + /// The stream the file is written to; closed at the end, unless . + /// The file's format. + /// The sheet's name; a csv file's single sheet takes no name in the file, but the name must still be valid. + /// The messages. + /// Returns the items a message carries, as an — a protobuf RepeatedField is one. It must not return . + /// Options of the writer, or for the defaults. + /// Stops the write; also flows into the enumeration of . + /// The number of data rows written. + /// + /// Flushed like the chunk overload, so memory holds a message and about a megabyte. On any failure + /// the stream is closed (unless ) and the file is incomplete. + /// An xlsx or ods sheet holds at most 1,048,576 rows, the header included; the row past it throws + /// after the rows before it were written — choose csv when the + /// count may exceed it. + /// + /// A value the format cannot hold. + /// An xlsx or ods sheet outgrew its row limit. + /// was canceled. + /// An argument is . + /// returned for a message. + /// The writer is not in a state to begin a sheet. + public ValueTask WriteAsync(Stream stream, TabularFormat format, string sheetName, IAsyncEnumerable chunks, Func> itemsOf, TabularWriterOptions? options = null, CancellationToken cancellationToken = default) => + WriteFileAsync(stream, format, options, writer => WriteSheetAsync(writer, sheetName, chunks, itemsOf, cancellationToken), cancellationToken); + + /// Writes a file of one sheet from items arriving one at a time; awaits per item, so prefer chunks for millions. + /// The stream the file is written to; closed at the end, unless . + /// The file's format. + /// The sheet's name; a csv file's single sheet takes no name in the file, but the name must still be valid. + /// The rows. + /// Options of the writer, or for the defaults. + /// Stops the write; also flows into the enumeration of . + /// The number of data rows written. + /// + /// On any failure the stream is closed (unless ) and the + /// file is incomplete. An xlsx or ods sheet holds at most 1,048,576 rows, the header included; the + /// row past it throws after the rows before it were written — + /// choose csv when the count may exceed it. + /// + /// A value the format cannot hold. + /// An xlsx or ods sheet outgrew its row limit. + /// was canceled. + /// An argument is . + /// The writer is not in a state to begin a sheet. + public ValueTask WriteAsync(Stream stream, TabularFormat format, string sheetName, IAsyncEnumerable items, TabularWriterOptions? options = null, CancellationToken cancellationToken = default) => + WriteFileAsync(stream, format, options, writer => WriteSheetAsync(writer, sheetName, items, cancellationToken), cancellationToken); + + /// Writes a file of one sheet from items in memory or produced synchronously. + /// The stream the file is written to; closed at the end, unless . + /// The file's format. + /// The sheet's name; a csv file's single sheet takes no name in the file, but the name must still be valid. + /// The rows. + /// Options of the writer, or for the defaults. + /// Stops the write, observed at each flush, about every megabyte written. + /// The number of data rows written. + /// + /// On any failure the stream is closed (unless ) and the + /// file is incomplete. An xlsx or ods sheet holds at most 1,048,576 rows, the header included; the + /// row past it throws after the rows before it were written — + /// choose csv when the count may exceed it. + /// + /// A value the format cannot hold. + /// An xlsx or ods sheet outgrew its row limit. + /// was canceled. + /// An argument is . + /// The writer is not in a state to begin a sheet. + public ValueTask WriteAsync(Stream stream, TabularFormat format, string sheetName, IEnumerable items, TabularWriterOptions? options = null, CancellationToken cancellationToken = default) => + WriteFileAsync(stream, format, options, writer => WriteSheetAsync(writer, sheetName, items, cancellationToken), cancellationToken); + + /// Writes one sheet into a writer the caller owns, from chunks as they arrive. + /// The caller's writer; the caller completes it. + /// The sheet's name; a csv file's single sheet takes no name in the file, but the name must still be valid. + /// The rows, a batch at a time. + /// Stops the write; also flows into the enumeration of . + /// The number of data rows written. + /// + /// The caller completes the writer; several exports may write their sheets into one workbook. On any + /// failure the writer is faulted. An xlsx or ods sheet holds at most 1,048,576 rows, the header + /// included; the row past it throws after the rows before it were + /// written — choose csv when the count may exceed it. + /// + /// A value the format cannot hold. + /// An xlsx or ods sheet outgrew its row limit. + /// was canceled. + /// An argument is . + /// A chunk is . + /// The writer is not in a state to begin a sheet. + public async ValueTask WriteSheetAsync(TabularWriter writer, string sheetName, IAsyncEnumerable> chunks, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(writer); + ArgumentNullException.ThrowIfNull(chunks); + + try + { + writer.BeginSheet(sheetName, _declared); + long rows = 0; + + await foreach (IReadOnlyList chunk in chunks.WithCancellation(cancellationToken).ConfigureAwait(false)) + { + rows += await WriteChunkAsync(writer, chunk ?? throw new ArgumentException("A chunk is null.", nameof(chunks)), cancellationToken).ConfigureAwait(false); + } + + return rows; + } + catch + { + writer.Fault(); + throw; + } + } + + /// Writes one sheet into a writer the caller owns, from messages that each carry a chunk. + /// The message type. + /// The caller's writer; the caller completes it. + /// The sheet's name; a csv file's single sheet takes no name in the file, but the name must still be valid. + /// The messages. + /// Returns the items a message carries, as an — a protobuf RepeatedField is one. It must not return . + /// Stops the write; also flows into the enumeration of . + /// The number of data rows written. + /// + /// The caller completes the writer. On any failure the writer is faulted. An xlsx or ods sheet holds + /// at most 1,048,576 rows, the header included; the row past it throws + /// after the rows before it were written — choose csv when the + /// count may exceed it. + /// + /// A value the format cannot hold. + /// An xlsx or ods sheet outgrew its row limit. + /// was canceled. + /// An argument is . + /// returned for a message. + /// The writer is not in a state to begin a sheet. + public async ValueTask WriteSheetAsync(TabularWriter writer, string sheetName, IAsyncEnumerable chunks, Func> itemsOf, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(writer); + ArgumentNullException.ThrowIfNull(chunks); + ArgumentNullException.ThrowIfNull(itemsOf); + + try + { + writer.BeginSheet(sheetName, _declared); + long written = 0; + + await foreach (TChunk message in chunks.WithCancellation(cancellationToken).ConfigureAwait(false)) + { + IReadOnlyList chunk = itemsOf(message) ?? throw new ArgumentException("The selector returned null for a message.", nameof(itemsOf)); + written += await WriteChunkAsync(writer, chunk, cancellationToken).ConfigureAwait(false); + } + + return written; + } + catch + { + writer.Fault(); + throw; + } + } + + /// Writes one sheet into a writer the caller owns, from items arriving one at a time. + /// The caller's writer; the caller completes it. + /// The sheet's name; a csv file's single sheet takes no name in the file, but the name must still be valid. + /// The rows. + /// Stops the write; also flows into the enumeration of . + /// The number of data rows written. + /// + /// The caller completes the writer. On any failure the writer is faulted. An xlsx or ods sheet holds + /// at most 1,048,576 rows, the header included; the row past it throws + /// after the rows before it were written — choose csv when the + /// count may exceed it. + /// + /// A value the format cannot hold. + /// An xlsx or ods sheet outgrew its row limit. + /// was canceled. + /// An argument is . + /// The writer is not in a state to begin a sheet. + public async ValueTask WriteSheetAsync(TabularWriter writer, string sheetName, IAsyncEnumerable items, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(writer); + ArgumentNullException.ThrowIfNull(items); + + try + { + writer.BeginSheet(sheetName, _declared); + long rows = 0; + + await foreach (T item in items.WithCancellation(cancellationToken).ConfigureAwait(false)) + { + WriteRow(writer, item); + rows++; + + if (writer.FlushRecommended) + { + await writer.FlushAsync(cancellationToken).ConfigureAwait(false); + } + } + + return rows; + } + catch + { + writer.Fault(); + throw; + } + } + + /// Writes one sheet into a writer the caller owns, from items in memory or produced synchronously. + /// The caller's writer; the caller completes it. + /// The sheet's name; a csv file's single sheet takes no name in the file, but the name must still be valid. + /// The rows. + /// Stops the write, observed at each flush, about every megabyte written. + /// The number of data rows written. + /// + /// The caller completes the writer. On any failure the writer is faulted. An xlsx or ods sheet holds + /// at most 1,048,576 rows, the header included; the row past it throws + /// after the rows before it were written — choose csv when the + /// count may exceed it. + /// + /// A value the format cannot hold. + /// An xlsx or ods sheet outgrew its row limit. + /// was canceled. + /// An argument is . + /// The writer is not in a state to begin a sheet. + public async ValueTask WriteSheetAsync(TabularWriter writer, string sheetName, IEnumerable items, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(writer); + ArgumentNullException.ThrowIfNull(items); + + try + { + writer.BeginSheet(sheetName, _declared); + long rows = 0; + + foreach (T item in items) + { + WriteRow(writer, item); + rows++; + + if (writer.FlushRecommended) + { + await writer.FlushAsync(cancellationToken).ConfigureAwait(false); + } + } + + return rows; + } + catch + { + writer.Fault(); + throw; + } + } + + /// + /// Creates the writer, lets the sheet be written, completes the file. Arguments are checked inside + /// the writer's lifetime, so a refused one still closes the stream. + /// + private static async ValueTask WriteFileAsync(Stream stream, TabularFormat format, TabularWriterOptions? options, Func> write, CancellationToken cancellationToken) + { + TabularWriter writer = TabularWriter.Create(stream, format, options); + + await using (writer.ConfigureAwait(false)) + { + long rows = await write(writer).ConfigureAwait(false); + await writer.CompleteAsync(cancellationToken).ConfigureAwait(false); + return rows; + } + } + + /// Writes a chunk's rows, flushing inside it when recommended and once after it. + private async ValueTask WriteChunkAsync(TabularWriter writer, IReadOnlyList chunk, CancellationToken cancellationToken) + { + for (int i = 0; i < chunk.Count; i++) + { + WriteRow(writer, chunk[i]); + + if (writer.FlushRecommended) + { + await writer.FlushAsync(cancellationToken).ConfigureAwait(false); + } + } + + await writer.FlushAsync(cancellationToken).ConfigureAwait(false); + return chunk.Count; + } + + private void WriteRow(TabularWriter writer, T item) + { + writer.BeginRow(); + + foreach (ExportColumn column in _columns) + { + column.Write(writer, item); + } + + writer.EndRow(); + } +} diff --git a/src/TriasDev.Tabular/Writing/TabularExportBuilder.cs b/src/TriasDev.Tabular/Writing/TabularExportBuilder.cs new file mode 100644 index 0000000..ca4972e --- /dev/null +++ b/src/TriasDev.Tabular/Writing/TabularExportBuilder.cs @@ -0,0 +1,144 @@ +namespace TriasDev.Tabular; + +/// +/// Declares an export's columns, in order: a header and a lambda each, the column's type taken from +/// the lambda's. +/// +/// +/// +/// Overload resolution picks the column type: an property resolves to +/// , int? to long?, to . +/// Convert explicitly where it does not: a would resolve to and +/// write its code point; a is ambiguous; a method group returning +/// does not convert; p => null and p => default are ambiguous; an +/// or passed with a is ambiguous +/// between decimal and double. , , +/// and enumerations have no overload and fail with a misleading "cannot convert to string" (CS0029): +/// convert explicitly — p => p.Status.ToString(), p => p.At.UtcDateTime. +/// +/// +/// A column named by an import field takes the field's name as its header, so a file written with +/// the export maps back onto the import's schema by header; the lambda's type must suit the field's, +/// or the call does not compile. +/// +/// +public sealed class TabularExportBuilder +{ + /// The width a date-time column gets unless told otherwise: yyyy-mm-dd hh:mm:ss. + private const double DateTimeWidth = 19; + + /// The width a date column gets unless told otherwise: yyyy-mm-dd. + private const double DateWidth = 10; + + private readonly List> _columns = []; + + internal TabularExportBuilder() + { + } + + /// A text column. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width, value, CellWriters.Text); + + /// An integer column. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width, value, CellWriters.Long); + + /// An integer column; null writes an empty cell. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width, value, CellWriters.NullableLong); + + /// A decimal column. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width, value, CellWriters.Decimal); + + /// A decimal column; null writes an empty cell. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width, value, CellWriters.NullableDecimal); + + /// A number column. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width, value, CellWriters.Double); + + /// A number column; null writes an empty cell. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width, value, CellWriters.NullableDouble); + + /// A date-time column, 19 characters wide unless told otherwise. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width ?? DateTimeWidth, value, CellWriters.DateTime); + + /// A date-time column, 19 characters wide unless told otherwise; null writes an empty cell. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width ?? DateTimeWidth, value, CellWriters.NullableDateTime); + + /// A date column, 10 characters wide unless told otherwise. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width ?? DateWidth, value, CellWriters.DateOnly); + + /// A date column, 10 characters wide unless told otherwise; null writes an empty cell. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width ?? DateWidth, value, CellWriters.NullableDateOnly); + + /// A boolean column. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width, value, CellWriters.Boolean); + + /// A boolean column; null writes an empty cell. + public TabularExportBuilder Column(string header, Func value, double? width = null) => Add(header, width, value, CellWriters.NullableBoolean); + + /// A text column named by an import field. + public TabularExportBuilder Column(TextImportField field, Func value, double? width = null) => Column(HeaderOf(field), value, width); + + /// An integer column named by an import field; null writes an empty cell. + public TabularExportBuilder Column(IntegerImportField field, Func value, double? width = null) => Column(HeaderOf(field), value, width); + + /// A decimal column named by an import field; null writes an empty cell. + public TabularExportBuilder Column(DecimalImportField field, Func value, double? width = null) => Column(HeaderOf(field), value, width); + + /// A number column named by a decimal import field; null writes an empty cell. + public TabularExportBuilder Column(DecimalImportField field, Func value, double? width = null) => Column(HeaderOf(field), value, width); + + /// A date-time column named by an import field; null writes an empty cell. + public TabularExportBuilder Column(DateImportField field, Func value, double? width = null) => Column(HeaderOf(field), value, width); + + /// A date column named by an import field; null writes an empty cell. + public TabularExportBuilder Column(DateImportField field, Func value, double? width = null) => Column(HeaderOf(field), value, width); + + /// A boolean column named by an import field; null writes an empty cell. + public TabularExportBuilder Column(BooleanImportField field, Func value, double? width = null) => Column(HeaderOf(field), value, width); + + /// + /// The export as declared so far, checked by the writer's rules for columns. The builder may go on + /// being changed; the export it returned does not change with it. + /// + /// No column was declared. + /// A header is empty, padded, repeated ignoring case or not writable, or a width is out of range. + public TabularExport Build() + { + if (_columns.Count == 0) + { + throw new InvalidOperationException("An export has at least one column."); + } + + ExportColumn[] columns = [.. _columns]; + WriteColumn[] declared = [.. columns.Select(column => column.Column)]; + + if (TabularWriter.ColumnsProblem(declared) is { } problem) + { + throw problem; + } + + return new TabularExport(columns, declared); + } + + private static string HeaderOf(ImportField field) + { + ArgumentNullException.ThrowIfNull(field); + + if (field.Variant is not null) + { + throw new ArgumentException( + $"\"{field.Name}\" is one language of a translated field; name the column with a header of its own.", + nameof(field)); + } + + return field.Name; + } + + private TabularExportBuilder Add(string header, double? width, Func value, Action write) + { + ArgumentNullException.ThrowIfNull(header); + ArgumentNullException.ThrowIfNull(value); + _columns.Add(new ExportColumn(new WriteColumn(header, width), value, write)); + return this; + } +} diff --git a/src/TriasDev.Tabular/Writing/TabularWriter.cs b/src/TriasDev.Tabular/Writing/TabularWriter.cs index f022e3e..23d7dc1 100644 --- a/src/TriasDev.Tabular/Writing/TabularWriter.cs +++ b/src/TriasDev.Tabular/Writing/TabularWriter.cs @@ -393,11 +393,17 @@ public async ValueTask DisposeAsync() } } - private void CheckColumns(ReadOnlySpan columns) + /// + /// What is wrong with a sheet's columns, as the exception to throw, or null: 1 to 16,384 columns, + /// headers not empty, not padded, unique ignoring case and writable, widths more than 0 and at + /// most 255 characters. + /// + /// Shared with , so an export's columns fail where they are declared. + internal static Exception? ColumnsProblem(ReadOnlySpan columns) { if (columns.IsEmpty || columns.Length > MaxColumns) { - throw Faulting(new ArgumentOutOfRangeException(nameof(columns), columns.Length, $"A sheet has 1 to {MaxColumns} columns.")); + return new ArgumentOutOfRangeException(nameof(columns), columns.Length, $"A sheet has 1 to {MaxColumns} columns."); } HashSet seen = new(StringComparer.OrdinalIgnoreCase); @@ -406,14 +412,24 @@ private void CheckColumns(ReadOnlySpan columns) { if (HeaderProblem(column.Header, seen) is { } problem) { - throw Faulting(new ArgumentException(problem, nameof(columns))); + return new ArgumentException(problem, nameof(columns)); } if (column.Width is { } width && (!double.IsFinite(width) || width <= 0 || width > MaxWidth)) { - throw Faulting(new ArgumentOutOfRangeException(nameof(columns), width, $"A column is more than 0 and at most {MaxWidth} characters wide.")); + return new ArgumentOutOfRangeException(nameof(columns), width, $"A column is more than 0 and at most {MaxWidth} characters wide."); } } + + return null; + } + + private void CheckColumns(ReadOnlySpan columns) + { + if (ColumnsProblem(columns) is { } problem) + { + throw Faulting(problem); + } } /// Why a header could not be read back as written, or null; adds it to . @@ -521,6 +537,16 @@ private T Faulting(T exception) return exception; } + /// Marks the writer unusable after a failure it was driven through, so its file is not completed. + internal void Fault() + { + // A completed file is whole: a failure after it must not turn it into a failed one. + if (_state != State.Completed) + { + MarkFaulted(); + } + } + private void MarkFaulted() { if (_state != State.Disposed) diff --git a/src/TriasDev.Tabular/Writing/TextRules.cs b/src/TriasDev.Tabular/Writing/TextRules.cs index f66ea2b..2d22d9f 100644 --- a/src/TriasDev.Tabular/Writing/TextRules.cs +++ b/src/TriasDev.Tabular/Writing/TextRules.cs @@ -12,6 +12,16 @@ internal static class TextRules private static readonly SearchValues Forbidden = SearchValues.Create( [.. Enumerable.Range(0, 0x20).Where(c => c is not (0x09 or 0x0A or 0x0D)).Select(c => (char)c), '￾', '￿']); + /// U+D800 to U+DFFF, every high and low surrogate. + /// + /// A search value, not IndexOfAnyInRange('\uD800', '\uDFFF'): that generic call allocated + /// (96 bytes a text cell) while the method ran unoptimised (tier-0), and short writes and cold + /// processes never leave tier-0. SearchValues recognises the range (a range-based + /// implementation), is as fast once optimised, and allocates nothing throughout. + /// + private static readonly SearchValues Surrogates = SearchValues.Create( + [.. Enumerable.Range(0xD800, 0x800).Select(c => (char)c)]); + /// Null when every character may be written, otherwise . public static string? Check(string text) { @@ -22,7 +32,7 @@ internal static class TextRules return ErrorCodes.Write.InvalidCharacter; } - int surrogate = span.IndexOfAnyInRange('\uD800', '\uDFFF'); + int surrogate = span.IndexOfAny(Surrogates); return surrogate < 0 || PairsAreWhole(span[surrogate..]) ? null : ErrorCodes.Write.InvalidCharacter; } diff --git a/tests/TriasDev.Tabular.Tests/Fixtures/WriteTarget.cs b/tests/TriasDev.Tabular.Tests/Fixtures/WriteTarget.cs index 54c6a1b..3f4cfb4 100644 --- a/tests/TriasDev.Tabular.Tests/Fixtures/WriteTarget.cs +++ b/tests/TriasDev.Tabular.Tests/Fixtures/WriteTarget.cs @@ -18,6 +18,9 @@ public sealed class WriteTarget : MemoryStream /// How many times FlushAsync was called. public int AsyncFlushes { get; private set; } + /// How many times an asynchronous write reached the target. + public int AsyncWrites { get; private set; } + /// When set, every asynchronous write throws it — a client that went away. public Exception? FailWith { get; set; } @@ -46,6 +49,7 @@ public override Task WriteAsync(byte[] buffer, int offset, int count, Cancellati public override ValueTask WriteAsync(ReadOnlyMemory buffer, CancellationToken cancellationToken = default) { cancellationToken.ThrowIfCancellationRequested(); + AsyncWrites++; if (FailWith is { } failure) { diff --git a/tests/TriasDev.Tabular.Tests/Writing/TabularExportBuilderTests.cs b/tests/TriasDev.Tabular.Tests/Writing/TabularExportBuilderTests.cs new file mode 100644 index 0000000..2e19b06 --- /dev/null +++ b/tests/TriasDev.Tabular.Tests/Writing/TabularExportBuilderTests.cs @@ -0,0 +1,115 @@ +using Xunit; + +namespace TriasDev.Tabular.Tests.Writing; + +/// Declaring an export: a column's type from its lambda, its header from a field, the writer's rules at Build. +public sealed class TabularExportBuilderTests +{ + private sealed record Portfolio(long Id, int Count, string Name, decimal Amount, double Rate, DateTime Start, DateOnly Settled, bool Active, long? Parent); + + private static readonly IntegerImportField IdField = ImportField.Integer("Id"); + private static readonly TextImportField NameField = ImportField.Text("Name"); + private static readonly DecimalImportField AmountField = ImportField.Decimal("Amount"); + private static readonly DecimalImportField RateField = ImportField.Decimal("Rate"); + private static readonly DateImportField StartField = ImportField.Date("Start"); + private static readonly DateImportField SettledField = ImportField.Date("Settled"); + private static readonly BooleanImportField ActiveField = ImportField.Boolean("Active"); + + [Fact] + public void ColumnsCannotBeChangedThroughTheirCollection() + { + TabularExport export = TabularExport.For().Column(IdField, p => p.Id).Build(); + + Assert.IsNotType(export.Columns); + Assert.Throws(() => ((IList)export.Columns)[0] = new WriteColumn("x")); + } + + [Fact] + public void DeclaresColumnsInOrderWithTheirHeaders() + { + TabularExport export = TabularExport.For() + .Column("Id", p => p.Id) + .Column("Count", p => p.Count) + .Column("Name", p => p.Name) + .Column("Amount", p => p.Amount) + .Column("Rate", p => p.Rate) + .Column("Start", p => p.Start) + .Column("Settled", p => p.Settled) + .Column("Active", p => p.Active) + .Column("Parent", p => p.Parent) + .Build(); + + Assert.Equal(["Id", "Count", "Name", "Amount", "Rate", "Start", "Settled", "Active", "Parent"], export.Columns.Select(c => c.Header)); + } + + [Fact] + public void TakesItsHeadersFromTheImportsFields() + { + TabularExport export = TabularExport.For() + .Column(IdField, p => p.Id) + .Column(NameField, p => p.Name) + .Column(AmountField, p => p.Amount) + .Column(RateField, p => p.Rate) + .Column(StartField, p => p.Start) + .Column(SettledField, p => p.Settled) + .Column(ActiveField, p => p.Active) + .Build(); + + Assert.Equal(["Id", "Name", "Amount", "Rate", "Start", "Settled", "Active"], export.Columns.Select(c => c.Header)); + } + + [Fact] + public void WidensDateColumnsUnlessToldOtherwise() + { + TabularExport export = TabularExport.For() + .Column("Start", p => p.Start) + .Column("Settled", p => p.Settled) + .Column("Name", p => p.Name) + .Column("Narrow", p => p.Start, width: 12) + .Build(); + + Assert.Equal(new double?[] { 19, 10, null, 12 }, export.Columns.Select(c => c.Width)); + } + + [Fact] + public void RefusesATranslatedFieldsVariant() + { + ImportField variant = Assert.Single(ImportField.Translated("Title", ["en"])); + + Assert.Throws(() => TabularExport.For().Column((TextImportField)variant, p => p.Name)); + } + + [Fact] + public void RefusesAnExportWithoutColumns() + { + Assert.Throws(() => TabularExport.For().Build()); + } + + [Fact] + public void RefusesColumnsTheWriterWouldRefuseWhereTheyAreDeclared() + { + Assert.ThrowsAny(() => TabularExport.For().Column("Name", p => p.Name).Column(" name", p => p.Name).Build()); + Assert.ThrowsAny(() => TabularExport.For().Column("Name", p => p.Name).Column("NAME", p => p.Name).Build()); + Assert.ThrowsAny(() => TabularExport.For().Column("", p => p.Name).Build()); + Assert.ThrowsAny(() => TabularExport.For().Column("Name", p => p.Name, width: 0).Build()); + } + + [Fact] + public void RefusesANullHeaderOrLambda() + { + Assert.Throws(() => TabularExport.For().Column((string)null!, p => p.Name)); + Assert.Throws(() => TabularExport.For().Column("Name", (Func)null!)); + Assert.Throws(() => TabularExport.For().Column((TextImportField)null!, p => p.Name)); + } + + [Fact] + public void BuildsAnExportTheBuilderNoLongerChanges() + { + TabularExportBuilder builder = TabularExport.For().Column("Id", p => p.Id); + TabularExport export = builder.Build(); + + builder.Column("Name", p => p.Name); + + Assert.Single(export.Columns); + } +} diff --git a/tests/TriasDev.Tabular.Tests/Writing/TabularExportRoundTripTests.cs b/tests/TriasDev.Tabular.Tests/Writing/TabularExportRoundTripTests.cs new file mode 100644 index 0000000..b16580b --- /dev/null +++ b/tests/TriasDev.Tabular.Tests/Writing/TabularExportRoundTripTests.cs @@ -0,0 +1,162 @@ +using TriasDev.Tabular.Tests.Fixtures; + +using Xunit; + +namespace TriasDev.Tabular.Tests.Writing; + +/// An export declared with the import's fields writes files the import maps back by header. +public sealed class TabularExportRoundTripTests +{ + private static readonly IntegerImportField IdField = ImportField.Integer("Id"); + private static readonly TextImportField NameField = ImportField.Text("Name"); + private static readonly DecimalImportField AmountField = ImportField.Decimal("Amount"); + private static readonly DecimalImportField RateField = ImportField.Decimal("Rate"); + private static readonly DateImportField StartField = ImportField.Date("Start"); + private static readonly DateImportField SettledField = ImportField.Date("Settled"); + private static readonly BooleanImportField ActiveField = ImportField.Boolean("Active"); + + private static readonly ImportSchema Schema = new() { Fields = [IdField, NameField, AmountField, RateField, StartField, SettledField, ActiveField] }; + + private sealed record Portfolio(long Id, string? Name, decimal? Amount, double? Rate, DateTime Start, DateOnly? Settled, bool Active); + + private sealed record Batch(List Items); + + private static readonly TabularExport Export = TabularExport.For() + .Column(IdField, p => p.Id) + .Column(NameField, p => p.Name) + .Column(AmountField, p => p.Amount) + .Column(RateField, p => p.Rate) + .Column(StartField, p => p.Start) + .Column(SettledField, p => p.Settled) + .Column(ActiveField, p => p.Active) + .Build(); + + private static CancellationToken Token => TestContext.Current.CancellationToken; + + public static TheoryData Formats => new() + { + { TabularFormat.Csv, "t.csv" }, + { TabularFormat.Xlsx, "t.xlsx" }, + { TabularFormat.Ods, "t.ods" }, + }; + + private static DateTime At(int year, int month, int day, int hour = 0, int minute = 0, int second = 0, int millisecond = 0) => + new(year, month, day, hour, minute, second, millisecond, DateTimeKind.Unspecified); + + private static Portfolio[] Portfolios(int count) => + [ + .. Enumerable.Range(1, count).Select(i => new Portfolio( + i, + i % 7 == 0 ? null : $"Portfolio {i}, \"quoted\"", + i % 5 == 0 ? null : i * 1.25m, + i % 3 == 0 ? null : i / 8d, + At(2026, 1 + (i % 12), 1 + (i % 28), i % 24, i % 60, i % 60, i % 1000), + i % 4 == 0 ? null : new DateOnly(2026, 1 + (i % 12), 1 + (i % 28)), + i % 2 == 0)), + ]; + + private static async IAsyncEnumerable Batches(Portfolio[] items, int size) + { + for (int at = 0; at < items.Length; at += size) + { + await Task.Yield(); + yield return new Batch([.. items[at..Math.Min(items.Length, at + size)]]); + } + } + + private static List Import(byte[] file, string name) + { + using ITabularCursor cursor = TabularFile.Open(new MemoryStream(file, writable: false), name, cancellationToken: Token); + FileProfile profile = TabularAnalyzer.Analyze(cursor, cancellationToken: Token); + MappingPlan plan = MappingPlan.ByHeader(profile.Sheets[0], Schema); + + using ImportRun run = TabularImporter.Import( + new MemoryStream(file, writable: false), + name, + plan, + Schema, + row => new Portfolio(row[IdField]!.Value, row[NameField], row[AmountField], row[RateField] is { } rate ? (double)rate : null, row[StartField]!.Value, row[SettledField] is { } settled ? DateOnly.FromDateTime(settled) : null, row[ActiveField]!.Value), + cancellationToken: Token); + + List> outcomes = [.. run.ReadRows(Token)]; + Assert.All(outcomes, outcome => Assert.False(outcome.HasErrors, string.Join(", ", outcome.Errors.Select(e => e.Code)))); + return [.. outcomes.Select(outcome => outcome.Value!)]; + } + + [Theory] + [MemberData(nameof(Formats))] + public async Task MapsBackByHeaderAsWritten(TabularFormat format, string name) + { + Portfolio[] written = Portfolios(500); + WriteTarget target = new(); + + await Export.WriteAsync(target, format, "Portfolios", written, cancellationToken: Token); + + Assert.Equal(written, Import(target.ToArray(), name)); + } + + [Theory] + [MemberData(nameof(Formats))] + public async Task WritesChunksTakenFromMessages(TabularFormat format, string name) + { + Portfolio[] written = Portfolios(2_345); + WriteTarget target = new(); + + long rows = await Export.WriteAsync(target, format, "Portfolios", Batches(written, 1_000), batch => batch.Items, cancellationToken: Token); + + Assert.Equal(written.Length, rows); + Assert.Equal(written, Import(target.ToArray(), name)); + } + + [Theory] + [MemberData(nameof(Formats))] + public async Task OneExportServesConcurrentWrites(TabularFormat format, string name) + { + Portfolio[] first = Portfolios(3_000); + Portfolio[] second = [.. Portfolios(3_000).Select(p => p with { Name = "other " + p.Id })]; + WriteTarget one = new(); + WriteTarget two = new(); + + await Task.WhenAll( + Task.Run(async () => await Export.WriteAsync(one, format, "a", Batches(first, 100), batch => batch.Items, cancellationToken: Token), Token), + Task.Run(async () => await Export.WriteAsync(two, format, "b", Batches(second, 100), batch => batch.Items, cancellationToken: Token), Token)); + + Assert.Equal(first, Import(one.ToArray(), name)); + Assert.Equal(second, Import(two.ToArray(), name)); + } + + [Theory] + [InlineData(TabularFormat.Csv)] + [InlineData(TabularFormat.Xlsx)] + [InlineData(TabularFormat.Ods)] + public async Task AllocatesNothingPerRow(TabularFormat format) + { + // One reused chunk, a source that completes synchronously, and Stream.Null: every await stays + // on this thread, so the thread's allocation counter sees the whole write and nothing else. + Portfolio[] chunk = Portfolios(10_000); + + async ValueTask Allocated(int chunks) + { + long before = GC.GetAllocatedBytesForCurrentThread(); + await Export.WriteAsync(Stream.Null, format, "data", Repeat(chunk, chunks), new TabularWriterOptions { LeaveOpen = true }, Token); + return GC.GetAllocatedBytesForCurrentThread() - before; + } + + await Allocated(2); // warm-up: static state, pools, JIT + long tenThousandRows = await Allocated(1); + long millionRows = await Allocated(100); + + // A million rows against ten thousand: what grows with the row count is under a byte a row. + Assert.True(millionRows - tenThousandRows < 1_000_000, $"{format}: {tenThousandRows:N0} bytes for 10k rows, {millionRows:N0} for 1M"); + } + + private static async IAsyncEnumerable> Repeat(Portfolio[] chunk, int times) + { + for (int i = 0; i < times; i++) + { + yield return chunk; + } + + await Task.CompletedTask; + } +} diff --git a/tests/TriasDev.Tabular.Tests/Writing/TabularExportTests.cs b/tests/TriasDev.Tabular.Tests/Writing/TabularExportTests.cs new file mode 100644 index 0000000..f3b713a --- /dev/null +++ b/tests/TriasDev.Tabular.Tests/Writing/TabularExportTests.cs @@ -0,0 +1,320 @@ +using System.Runtime.CompilerServices; +using System.Text; + +using TriasDev.Tabular.Csv; +using TriasDev.Tabular.Tests.Fixtures; + +using Xunit; + +namespace TriasDev.Tabular.Tests.Writing; + +/// Writing an export: every kind of source, one sheet or a whole file, flushed as it goes. +public sealed class TabularExportTests +{ + private static CancellationToken Token => TestContext.Current.CancellationToken; + + private sealed record Item(long Id, string Name, decimal Amount); + + private static readonly TabularExport Export = TabularExport.For() + .Column("Id", i => i.Id) + .Column("Name", i => i.Name) + .Column("Amount", i => i.Amount) + .Build(); + + private static readonly TabularWriterOptions NoBom = new() { Csv = new CsvWriterOptions { ByteOrderMark = false } }; + + private static Item[] Items(int count) => [.. Enumerable.Range(1, count).Select(i => new Item(i, $"item {i}", i / 4m))]; + + private static async IAsyncEnumerable> Chunks(Item[] items, int size, [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + for (int at = 0; at < items.Length; at += size) + { + cancellationToken.ThrowIfCancellationRequested(); + await Task.Yield(); + yield return items[at..Math.Min(items.Length, at + size)]; + } + } + + private static async IAsyncEnumerable OneByOne(Item[] items) + { + foreach (Item item in items) + { + await Task.Yield(); + yield return item; + } + } + + private static string Csv(WriteTarget target) => Encoding.UTF8.GetString(target.ToArray()); + + [Fact] + public async Task WritesAHeaderAndARowPerItem() + { + WriteTarget target = new(); + + long rows = await Export.WriteAsync(target, TabularFormat.Csv, "data", Items(2), NoBom, Token); + + Assert.Equal(2, rows); + Assert.Equal("Id,Name,Amount\r\n1,item 1,0.25\r\n2,item 2,0.5\r\n", Csv(target)); + Assert.True(target.IsDisposed); + Assert.False(target.DisposedSynchronously); + } + + [Fact] + public async Task EveryKindOfSourceWritesTheSameFile() + { + Item[] items = Items(5_000); + + WriteTarget chunked = new(); + WriteTarget streamed = new(); + WriteTarget listed = new(); + WriteTarget selected = new(); + + Assert.Equal(5_000, await Export.WriteAsync(chunked, TabularFormat.Csv, "data", Chunks(items, 700, Token), NoBom, Token)); + Assert.Equal(5_000, await Export.WriteAsync(streamed, TabularFormat.Csv, "data", OneByOne(items), NoBom, Token)); + Assert.Equal(5_000, await Export.WriteAsync(listed, TabularFormat.Csv, "data", items, NoBom, Token)); + Assert.Equal(5_000, await Export.WriteAsync(selected, TabularFormat.Csv, "data", Chunks(items, 700, Token), chunk => chunk, NoBom, Token)); + + string expected = Csv(listed); + Assert.Equal(expected, Csv(chunked)); + Assert.Equal(expected, Csv(streamed)); + Assert.Equal(expected, Csv(selected)); + } + + [Fact] + public async Task FlushesAfterEveryChunk() + { + WriteTarget target = new(); + + await Export.WriteAsync(target, TabularFormat.Csv, "data", Chunks(Items(1_000), 100, Token), NoBom, Token); + + // Ten chunks, each flushed as it is written, and the file's completion. + Assert.True(target.AsyncWrites >= 10, $"{target.AsyncWrites} writes reached the target"); + } + + [Fact] + public async Task FlushesInsideALargeChunkWhenTheWriterRecommendsIt() + { + WriteTarget target = new(); + long reachedTheTarget = -1; + TabularExport export = TabularExport.For() + .Column("Id", i => + { + if (i.Id == 40_000) + { + reachedTheTarget = target.Length; + } + + return i.Id; + }) + .Column("Name", i => i.Name) + .Build(); + Item[] items = [.. Enumerable.Range(1, 60_000).Select(i => new Item(i, new string('x', 40), i))]; + + await export.WriteAsync(target, TabularFormat.Csv, "data", Chunks(items, items.Length, Token), NoBom, Token); + + // The target receives bytes only on a flush: some arrived before the chunk ended. + Assert.True(reachedTheTarget > 0, $"{reachedTheTarget} bytes had reached the target by row 40,000 of one chunk"); + } + + [Fact] + public async Task AFailureInsideASheetLeavesTheCallersWriterUnusable() + { + TabularExport export = TabularExport.For() + .Column("Id", i => i.Id > 4 ? throw new InvalidOperationException("boom") : i.Id) + .Build(); + WriteTarget target = new(); + + await using TabularWriter writer = TabularWriter.Create(target, TabularFormat.Csv); + + await Assert.ThrowsAsync(async () => await export.WriteSheetAsync(writer, "data", Items(10), Token)); + + Assert.Throws(writer.EndRow); + await Assert.ThrowsAsync(async () => await writer.CompleteAsync(Token)); + } + + [Fact] + public async Task WritesSeveralSheetsIntoOneWorkbook() + { + WriteTarget target = new(); + + await using (TabularWriter writer = TabularWriter.Create(target, TabularFormat.Xlsx)) + { + Assert.Equal(3, await Export.WriteSheetAsync(writer, "First", Items(3), Token)); + Assert.Equal(2, await Export.WriteSheetAsync(writer, "Second", Chunks(Items(2), 1, Token), Token)); + await writer.CompleteAsync(Token); + } + + using ITabularCursor cursor = TabularFile.Open(new MemoryStream(target.ToArray(), writable: false), "t.xlsx", cancellationToken: Token); + Assert.Equal(["First", "Second"], cursor.Sheets.Select(s => s.Name)); + } + + [Fact] + public async Task AValueTheFormatRefusesClosesTheStreamAndLeavesNoValidFile() + { + TabularExport export = TabularExport.For().Column("Id", i => i.Id).Column("Amount", i => i.Amount).Build(); + Item[] items = [.. Items(50_000)]; + items[39_999] = items[39_999] with { Amount = 1_234_567_890_123.456m }; + WriteTarget target = new(); + + TabularWriteException refused = await Assert.ThrowsAsync(async () => + await export.WriteAsync(target, TabularFormat.Xlsx, "data", Chunks(items, 10_000, Token), cancellationToken: Token)); + + Assert.Equal(ErrorCodes.Write.PrecisionLoss, refused.Code); + Assert.Equal(40_001, refused.RowNumber); + Assert.Equal("Amount", refused.Header); + Assert.True(target.IsDisposed); + Assert.False(target.DisposedSynchronously); + Assert.ThrowsAny(() => new TriasDev.Tabular.Xlsx.XlsxCursor(new MemoryStream(target.ToArray(), writable: false), cancellationToken: Token)); + } + + [Fact] + public async Task CancellationStopsBetweenChunksAndClosesTheStream() + { + using CancellationTokenSource cancel = CancellationTokenSource.CreateLinkedTokenSource(Token); + WriteTarget target = new(); + int chunksSeen = 0; + + bool tokenFlowed = false; + + // Deliberately called without a token: it can only arrive through the export's WithCancellation. + async IAsyncEnumerable> Source([EnumeratorCancellation] CancellationToken cancellationToken = default) + { + tokenFlowed = cancellationToken == cancel.Token; + + await foreach (IReadOnlyList chunk in Chunks(Items(1_000), 100, CancellationToken.None)) + { + if (++chunksSeen == 3) + { + await cancel.CancelAsync(); + } + + yield return chunk; + } + } + +#pragma warning disable xUnit1051, S8949 // The source is called without a token on purpose: the export must flow its own through WithCancellation. + await Assert.ThrowsAnyAsync(async () => + await Export.WriteAsync(target, TabularFormat.Csv, "data", Source(), NoBom, cancel.Token)); +#pragma warning restore xUnit1051, S8949 + + Assert.True(tokenFlowed, "the export did not enumerate the source WithCancellation"); + Assert.True(chunksSeen < 10); + Assert.True(target.IsDisposed); + } + + [Fact] + public async Task LeavesTheStreamOpenWhenAsked() + { + WriteTarget target = new(); + + await Export.WriteAsync(target, TabularFormat.Csv, "data", Items(1), new TabularWriterOptions { LeaveOpen = true }, Token); + + Assert.False(target.IsDisposed); + } + + [Fact] + public async Task RefusesANullSourceAndStillClosesTheStream() + { + WriteTarget target = new(); + + await Assert.ThrowsAsync(async () => + await Export.WriteAsync(target, TabularFormat.Csv, "data", (IEnumerable)null!, cancellationToken: Token)); + + Assert.True(target.IsDisposed); + } + + [Fact] + public async Task RefusesANullChunk() + { + static async IAsyncEnumerable> WithANullChunk() + { + await Task.Yield(); + yield return null!; + } + + ArgumentException refused = await Assert.ThrowsAsync(async () => + await Export.WriteAsync(new WriteTarget(), TabularFormat.Csv, "data", WithANullChunk(), cancellationToken: Token)); + Assert.Equal("chunks", refused.ParamName); + } + + [Fact] + public async Task ASelectorThatThrowsOrReturnsNullFaultsTheWriteAndClosesTheStream() + { + static async IAsyncEnumerable Messages() + { + for (int i = 0; i < 3; i++) + { + await Task.Yield(); + yield return i; + } + } + + WriteTarget throwing = new(); + await Assert.ThrowsAsync(async () => + await Export.WriteAsync(throwing, TabularFormat.Csv, "data", Messages(), m => m == 1 ? throw new InvalidOperationException("boom") : Items(2), NoBom, Token)); + Assert.True(throwing.IsDisposed); + + WriteTarget nulls = new(); + ArgumentException refused = await Assert.ThrowsAsync(async () => + await Export.WriteAsync(nulls, TabularFormat.Csv, "data", Messages(), m => m == 1 ? null! : Items(2), NoBom, Token)); + Assert.Equal("itemsOf", refused.ParamName); + Assert.True(nulls.IsDisposed); + + await using TabularWriter first = TabularWriter.Create(new WriteTarget(), TabularFormat.Csv); + await Assert.ThrowsAsync(async () => + await Export.WriteSheetAsync(first, "data", Messages(), m => m == 1 ? throw new InvalidOperationException("boom") : Items(2), Token)); + Assert.Throws(first.EndRow); + + await using TabularWriter second = TabularWriter.Create(new WriteTarget(), TabularFormat.Csv); + await Assert.ThrowsAsync(async () => + await Export.WriteSheetAsync(second, "data", Messages(), m => m == 1 ? null! : Items(2), Token)); + await Assert.ThrowsAsync(async () => await second.CompleteAsync(Token)); + } + + [Fact] + public async Task CancellationThroughPerItemAsyncSourceStopsTheWriteAndClosesTheStream() + { + using CancellationTokenSource cancel = CancellationTokenSource.CreateLinkedTokenSource(Token); + WriteTarget target = new(); + int seen = 0; + + async IAsyncEnumerable Source([EnumeratorCancellation] CancellationToken cancellationToken = default) + { + foreach (Item item in Items(1_000)) + { + if (++seen == 5) + { + await cancel.CancelAsync(); + } + + cancellationToken.ThrowIfCancellationRequested(); + await Task.Yield(); + yield return item; + } + } + +#pragma warning disable xUnit1051, S8949 // The source is called without a token on purpose: the export must flow its own through WithCancellation. + await Assert.ThrowsAnyAsync(async () => + await Export.WriteAsync(target, TabularFormat.Csv, "data", Source(), NoBom, cancel.Token)); +#pragma warning restore xUnit1051, S8949 + + Assert.True(seen < 100); + Assert.True(target.IsDisposed); + } + + [Fact] + public async Task ACompletedWritersFileIsNotReportedAsFailed() + { + await using TabularWriter writer = TabularWriter.Create(new WriteTarget(), TabularFormat.Csv); + await Export.WriteSheetAsync(writer, "data", Items(2), Token); + await writer.CompleteAsync(Token); + + InvalidOperationException first = await Assert.ThrowsAsync(async () => await Export.WriteSheetAsync(writer, "more", Items(1), Token)); + InvalidOperationException later = await Assert.ThrowsAsync(async () => await Export.WriteSheetAsync(writer, "more", Items(1), Token)); + + // "incomplete" also contains "complete": the faulted message must be ruled out by name. + Assert.StartsWith("The file is complete", first.Message, StringComparison.Ordinal); + Assert.StartsWith("The file is complete", later.Message, StringComparison.Ordinal); + Assert.DoesNotContain("failed earlier", later.Message, StringComparison.Ordinal); + } +} diff --git a/tests/TriasDev.Tabular.Tests/Writing/TextRulesTests.cs b/tests/TriasDev.Tabular.Tests/Writing/TextRulesTests.cs new file mode 100644 index 0000000..38f3b08 --- /dev/null +++ b/tests/TriasDev.Tabular.Tests/Writing/TextRulesTests.cs @@ -0,0 +1,47 @@ +using Xunit; + +namespace TriasDev.Tabular.Tests.Writing; + +/// The characters no format may carry, checked on every text cell. +/// Surrogates are built in code: a lone one in attribute data does not survive the test framework. +public sealed class TextRulesTests +{ + [Theory] + [InlineData("plain")] + [InlineData("tab\tand\r\nbreaks")] + public void AcceptsWritableText(string text) => Assert.Null(TextRules.Check(text)); + + [Fact] + public void AcceptsAWholePair() => Assert.Null(TextRules.Check(string.Concat("pair ", "\uD83D", "\uDE00", " whole"))); + + [Theory] + [InlineData(0x0007)] + [InlineData(0xFFFE)] + [InlineData(0xD83D)] + [InlineData(0xDE00)] + public void RefusesAForbiddenCharacter(int forbidden) => + Assert.Equal(ErrorCodes.Write.InvalidCharacter, TextRules.Check(string.Concat("text ", ((char)forbidden).ToString(), "."))); + + [Fact] + public void RefusesAPairInTheWrongOrder() => + Assert.Equal(ErrorCodes.Write.InvalidCharacter, TextRules.Check(string.Concat("reversed ", "\uDE00", "\uD83D"))); + + [Fact] + public void AllocatesNothingPerCall() + { + string text = string.Concat("Portfolio 12, \"quoted\" ", "\uD83D", "\uDE00"); + TextRules.Check(text); + + const int Calls = 100_000; + long before = GC.GetAllocatedBytesForCurrentThread(); + + for (int i = 0; i < Calls; i++) + { + TextRules.Check(text); + } + + // Less than a byte per call: a one-off from the runtime tiering the loop up is not a per-call + // allocation; the boxing this guards against cost 96 bytes on every call. + Assert.InRange(GC.GetAllocatedBytesForCurrentThread() - before, 0, Calls - 1); + } +}