fix(ast): indented formatting

README.md

@@ -8,13 +8,16 @@ LLT is designed to make prompt engineering and content generation as powerful an
 
 ## ✨ Key Features
 
-- **Razor-inspired DSL** with `@if`, `@foreach`, expressions, metadata, and inline variables  
-- **Message-oriented syntax** for LLM role structures (system, user, assistant)  
-- **Powerful metadata filtering** — select the best template by language, model, or custom qualifiers  
-- **Composable templates** — reuse and render nested templates inside others  
-- **Deterministic formatting** — preserves whitespace and indentation, ensuring predictable output  
-- **Expression evaluator** — supports arithmetic, logic, method access, array indexing, and ternary operators  
-- **Library-driven workflow** — import templates from assemblies, files, or strings  
+- **Razor-inspired DSL** with `@if`, `@foreach`, expressions, metadata, and inline variables
+- **Full expression evaluator** — arithmetic, logic, ternary, null-coalescing (`??`), property checks (`?:`), safe navigation (`?.`), array/object literals, and method calls
+- **Message-oriented syntax** for LLM role structures (system, user, assistant, tool)
+- **Powerful metadata filtering** — select the best template by language, model, or custom qualifiers
+- **Metadata fallback schemes** — hierarchical language fallback (`en-US` → `en` → sibling → default)
+- **Composable templates** — reuse and render nested templates inside others, with optional context override (`@render 'name' with ctx`)
+- **Extensible function system** — built-in functions plus custom ones registered from C#
+- **Deterministic formatting** — indentation is normalized during parsing, inner spacing is preserved, output is predictable
+- **Library-driven workflow** — import templates from assemblies, files, or strings
+- **Body parsing** — parse template bodies without the `@template` wrapper via `ParseTextTemplate` / `ParseMessagesTemplate`
 
 ## 📦 Installation
 
@@ -106,6 +109,14 @@ Console.WriteLine(template.Render()); // Hello GPT-4!
 ```
 
 Templates are resolved by **metadata specificity**, similar to CSS selector priority.
+If no template matches exactly, use fallback retrieval:
+
+```csharp
+// en-US → en → any sibling of the same language root → default language
+lib.SetLanguageFallbackScheme(new HierarchicalLanguageFallbackScheme("en"));
+
+var fallback = lib.RetrieveWithFallback("greeting", new LanguageMetadata("en-US"));
+```
 
 ## 💬 Message Templates for LLM Chats
 
@@ -134,59 +145,247 @@ LLT supports special `@messages` syntax for structured chat prompts:
 }
 ```
 
-**Rendered result:** a sequence of chat messages with proper roles (`system`, `user`, `assistant`).
-
-## 🧠 Supported Syntax
-
-- Inline expressions: `@(a + b * c)`
-- Conditionals:  
-  ```llt
-  @if cond {
-      ...
-  } else if other {
-      ...
-  } else {
-      ...
-  }
-  ```
-- Loops:  
-  ```llt
-  @foreach item in items {
-      @item
-  }
-  ```
-- Variables:  
-  ```llt
-  @let x = 5
-  @x = 10
-  @x
-  ```
-- Nested or external templates:  
-  ```llt
-  @render 'other_template'
-  ```
-- Comments:  
-  ```llt
-  @// line comment
-  @*
-    block comment
-  *@
-  ```
+**Rendered result:** a sequence of chat messages with proper roles (`system`, `user`, `assistant`, `tool`).
+Roles can be declared statically (`@system message`, `@user message`, `@assistant message`, `@tool message`)
+or dynamically via `@message { @role <expression> ... }`.
+
+## 🧠 Language Reference
+
+### Expressions
+
+#### Literals
+
+| Literal | Examples |
+|---|---|
+| Numbers | `42`, `3.14`, `-7` |
+| Strings | `'single'`, `"double"` |
+| Booleans | `true`, `false` |
+| Null | `null` |
+| Arrays | `[1, 2, 3]`, `['a', ctx?.value]`, `[]` (trailing commas allowed) |
+| Objects | `{ name: 'Fish', qty: 5 }`, `{ 'key with spaces': 1, [expr]: 2 }` |
+
+#### Access
+
+| Syntax | Meaning |
+|---|---|
+| `ctx` | The root of the rendering context |
+| `name` | Property of the root context (same as `ctx.name`) |
+| `?name` | **Safe** property access — returns `null` instead of throwing when missing |
+| `a.b` / `a.b.c` | Property access (chained) |
+| `a?.b` | **Safe navigation** — `null` if `a` is `null` or lacks `b` |
+| `a[expr]` | Index access (arrays, dictionaries, strings) |
+| `a?[expr]` | **Safe** index access |
+| `a.method(args)` | Method call (any public .NET method) |
+| `a?.method(args)` | **Safe** method call |
+| `func(args)` | Global template function (see [Functions](#-functions)) |
+
+#### Unary operators
+
+| Operator | Meaning | Example |
+|---|---|---|
+| `-` | Negation | `@(-x)` |
+| `!` | Logical NOT | `@(!flag)` |
+| `#` | Length (string / array / dictionary) | `@(#name)`, `@if #items > 0` |
+| `+` | No-op (kept for symmetry) | `@(+x)` |
+
+#### Binary operators (by precedence, high → low)
+
+| Precedence | Operators | Meaning |
+|---|---|---|
+| 1 | `*` `/` `%` | Multiplication, division, modulus |
+| 2 | `+` `-` | Addition, subtraction, string concatenation |
+| 3 | `<` `<=` `>` `>=` | Relational |
+| 4 | `?:` | **Has** operator — checks whether the left operand has the right property |
+| 5 | `==` `!=` | Equality |
+| 6 | `&&` | Logical AND |
+| 7 | `\|\|` | Logical OR |
+| 8 | `??` | **Null-coalescing** — right operand when the left is `null` or missing |
+| 9 | `? :` | Ternary conditional |
+
+#### Null-coalescing, Has and Safe navigation
+
+```llt
+@template t {
+    1: @(?value ?? 'No value')
+    2: @(ctx ?: 'value' ? value ?? 'Null' : 'No value')
+}
+```
+
+| Render context | Output |
+|---|---|
+| `new { value = "Hello" }` | `1: Hello` / `2: Hello` |
+| `new { value = (string?)null }` | `1: No value` / `2: Null` — the property *exists* but is `null` |
+| `new { }` | `1: No value` / `2: No value` — the property is *missing* |
+
+- `??` treats **missing** and `null` the same way.
+- `?:` (has) distinguishes them: it only checks **existence**, so it works great as the condition
+  when you need different fallbacks for `null` vs. missing:
+
+```llt
+@(user ?: 'name' ? user.name : 'Anonymous')
+@(user?.name ?? 'Anonymous')        @/ Safe navigation + coalescing
+```
+
+> [!TIP]
+> `@`-statements in plain text accept *simple* expressions (unary + member access).
+> Wrap binary expressions in parentheses: `@(a + b)` — inside `@if`, `@while` etc. full expressions are allowed.
+
+#### Formatting
+
+An expression can be followed by `:` and a format string:
+
+```llt
+Price: @price:'0.00'
+```
+
+### Statements
+
+| Statement | Syntax | Description |
+|---|---|---|
+| If | `@if cond { ... }` | Conditional block |
+| Else | `else { ... }`, `else if cond { ... }` or `@else { ... }` | Optional `@` before `else` is allowed |
+| Foreach | `@foreach item in items { ... }` | Iteration; loop variable is scoped to the block |
+| While | `@while cond { ... }` | Conditional loop |
+| Let | `@let x = expr` | Declares a new variable (lexically scoped) |
+| Assign | `@x = expr` | Assigns to an existing variable |
+| Render | `@render 'name'`, `@render 'name' with expr` | Renders another template, optionally with a new context |
+| Output | `@expr`, `@(expr)`, `@expr:'format'` | Prints an expression value |
+| Comment | `@/ line comment` | C#-style line comment — to the end of the line |
+| Comment | `@* block comment *@` | Razor-style block comment |
+
+Comments are skipped by the parser and can be placed wherever whitespace is allowed, including inside template bodies.
+
+Escapes: `@@` renders a literal `@`, `{{` renders `{`, `}}` renders `}`.
+
+Multi-line raw text can be wrapped in five backticks to avoid escaping:
+
+```llt
+@template code_sample
+{
+    `````
+    @if isNotParsed { this is raw text, not a statement }
+    `````
+}
+```
+
+### Metadata
+
+Attach metadata to a template with the `@metadata` block (only constant values are allowed):
+
+```llt
+@template greeting
+{
+    @metadata { lang: 'en', model: 'gpt-4', version: 2 }
+    Hello!
+}
+```
+
+Built-in metadata keys (parsed by the corresponding factories):
+
+| Key | Metadata type |
+|---|---|
+| `lang` | `LanguageMetadata` |
+| `model` | `TargetModelMetadata` |
+| `model_family` | `TargetModelFamilyMetadata` |
+| `version` | `VersionMetadata` |
+
+Pass factories to `Parse` so the keys are recognized; unknown keys become `AdditionalMetadata`:
+
+```csharp
+var parser = new LLTParser();
+var templates = parser.Parse(src, new MetadataFactory[]
+{
+    new LanguageMetadataFactory(),
+    new TargetModelMetadataFactory(),
+    new VersionMetadataFactory(),
+});
+```
+
+#### Custom metadata
+
+```csharp
+public class MyMetadata : IMetadata
+{
+    public MyMetadata(string value) => Value = value;
+    public string Value { get; }
+}
+
+public class MyMetadataFactory : MetadataFactory
+{
+    public override bool TryCreateMetadata(string key, TemplateDataAccessor value, out IMetadata metadata)
+    {
+        if (key == "my_key")
+        {
+            metadata = new MyMetadata(value.ToString());
+            return true;
+        }
+        metadata = null;
+        return false;
+    }
+}
+```
+
+#### Fallback schemes
+
+- `lib.SetLanguageFallbackScheme(new HierarchicalLanguageFallbackScheme("en"))` — walks up the language
+  hierarchy (`en-US` → `en`), then tries siblings, then the default language.
+- `lib.SetLanguageFallbackScheme(new MajorLanguageFallbackScheme())` — falls back to the major language group.
+- `lib.SetFallbackScheme(typeof(LanguageMetadata), scheme)` — register a custom `MetadataFallbackScheme<T>` for any metadata type.
+
+Retrieval API: `Retrieve`, `TryRetrieve`, `TryRetrieveBest`, `RetrieveWithFallback`, `RetrieveAll` —
+each with optional `identifier` and metadata arguments.
+
+## 🔧 Functions
+
+Built-in functions (`TemplateFunctions.All`):
+
+| Function | Description | Example |
+|---|---|---|
+| `type(x)` | Type name of the value (`string`, `number`, `boolean`, `object`, `array`, `null`, ...) | `@type(item)` |
+| `length(x)` | Length of a string, array or dictionary | `@length(items)` |
+| `strcat(a, b, ...)` | Concatenates values into a string | `@strcat('Hi, ', name)` |
+| `substr(s, start, len)` | Substring | `@substr(name, 0, 3)` |
+
+Register custom functions and pass them to `Render`:
+
+```csharp
+var functions = new TemplateFunctionSet(includeDefault: true,
+    new TemplateFunction("shout", (self, args) => args[0].ToString().ToUpperInvariant()));
+
+var rendered = template.Render(new { name = "Andrew" }, functions);
+```
+
+```llt
+Hello, @shout(name)!   @/ → Hello, ANDREW!
+```
 
 ## 🧾 Formatting & Scoping Rules
 
-- Whitespace and indentation in source templates are **fully preserved**
+- Leading indentation is **normalized during parsing**: every line loses up to `depth × TabSize` leading
+  whitespace, where `depth` is the block nesting level. You can write templates with comfortable
+  indentation — the common base indent is stripped and the output stays clean
+- Inner whitespace, line breaks and blank lines are preserved as written
+- Leading/trailing blank lines of blocks are trimmed; indentation before complex statements
+  (`@if`, `@foreach`, `@while`) is removed
+- Lines containing only non-rendering constructs — `@/` and `@* *@` comments, `@let` declarations,
+  variable assignments — are removed entirely: the surrounding line breaks are trimmed so they
+  leave no blank lines in the output
+- `TabSize` (default `4`) on `LLTParser` controls how many columns one indent level takes during refinement
 - `@let` variables are **lexically scoped**
 - Loop variables do **not leak outside** their block
 - Nested `@if` and `@foreach` blocks behave predictably, matching C#‑like logical semantics
 
 ## 🏗️ Architecture Overview
 
-- **LLTParser** – parses source text into ASTs (`TemplateNode`, `TemplateExpressionNode`, etc.)
-- **Template** – runtime object capable of rendering with dynamic context
-- **TemplateLibrary** – registry and loader for collections of templates with filtering
+- **LLTParser** – parses source text into ASTs (`TemplateNode`, `TemplateExpressionNode`, etc.);
+  supports full-file parsing (`Parse`) and body-only parsing (`ParseTextTemplate`, `ParseMessagesTemplate`)
+- **Template** – runtime object capable of rendering with dynamic context (`TextTemplate`, `MessagesTemplate`)
+- **TemplateLibrary** – registry and loader for collections of templates with filtering and fallbacks
 - **Metadata System** – extensible mechanism for attaching attributes such as language, model, or custom version
-- **ChatMessage Model** – unified representation for structured message templates (`system`, `user`, `assistant`)
+- **Data Accessors** – reflection / dictionary / array wrappers (`DataAccessorFactory`) with
+  `PropertiesToLowerCase`, `KeysToLowerCase` and `Snapshot` options
+- **Function System** – `TemplateFunctionSet` / `TemplateFunction` for extending template logic
+- **ChatMessage Model** – unified representation for structured message templates (`system`, `user`, `assistant`, `tool`)
 
 ## 🔖 License