fix: decline to flatten a heredoc whose interpolation spans lines (#347)

CHANGELOG.md

@@ -17,13 +17,31 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
   `python -m hcl2` are unaffected; only code importing `cli.hcl_to_json`, `cli.json_to_hcl`,
   `cli.hq`, or `cli.helpers` needs to add the `hcl2.` prefix. No compatibility shim ships,
   because a shim would still occupy the colliding name.
+
 - The redundant `cli/py.typed` marker is gone; `hcl2/py.typed` already covers `hcl2.cli`.
 
+- The `regex` package is no longer a dependency. Nothing imports it now that quoted strings are split by the span-aware template scanner; `lark` is the only runtime requirement.
+
 ### Added
 
 - Python 3.14 is now tested and declared as supported. No source changes were needed; the full
   suite passes on 3.14 as-is.
 
+### Fixed
+
+- A heredoc whose interpolation spans lines is not flattened. The quoted form cannot hold one: the newlines inside `${...}` are expression source, where OpenTofu rejects an escaped newline and a raw one makes the string span lines, which it also rejects. It used to emit the raw version -- output neither Terraform nor this library could read, written with no error -- and now hands the heredoc back in the form `preserve_heredocs=True` produces, which reads back as that heredoc. Declining is the only answer that does not change what the document means. The value form, which can carry such a body, measures a `<<-` margin on the lines that start with literal text: a line that begins inside `${...}` is expression source, and OpenTofu does not let its indent lower the margin of the rest. The same holds for a heredoc whose template uses a `~` strip marker: a heredoc body is lexed one line at a time, so the marker strips no further than its own line, while in the quoted string the same whitespace runs on into the next line's indent -- OpenTofu evaluates the loop `%{ for s in ["a", "b"] ~}\n  - ${s}\n%{ endfor ~}` to `  - a\n  - b\n` as a heredoc and to `- a\n- b\n` flattened. A declined heredoc written as a function argument goes back as the heredoc itself rather than as a quoted copy of its source. ([#347](https://github.com/amplify-education/python-hcl2/issues/347))
+- A quoted string can use a strip marker on an interpolation, `${~ ...}` or `${... ~}`. The grammar gave the marker to template directives only, so such a string did not parse at all, in `loads` or in `dumps` of a dict that carried one. OpenTofu evaluates `"a ${~ "b"} c"` to `ab c`; the marker is kept, spaced as a directive's is.
+- `$${` and `%%{` resolve to `${` and `%{` in the value form, in both quoted strings and heredocs. They are HCL's escapes for a literal sigil, exactly as `\"` is for a quote, and OpenTofu evaluates `"$${esc}"` to the six characters `${esc}`; returning them doubled made the value differ from the one Terraform reads, in the one mode that promises the value. The escapes written after one resolve as well, since the whole run is literal text: `"$${a\tb}"` is `${a<TAB>b}`, and so is the literal text between two directives. ([#336](https://github.com/amplify-education/python-hcl2/issues/336))
+- `strings_to_heredocs` resolves every escape the reader does. It knew `\n`, `\r`, `\"` and `\\`, so `"a\tb\n"` was written into the body as a backslash and a `t` -- two characters where Terraform reads one tab -- and `\uNNNN` fared the same. It now uses `process_escape_sequences`, the package's one implementation of that alphabet. ([#329](https://github.com/amplify-education/python-hcl2/issues/329))
+- Escapes are no longer added or resolved inside `${...}`. The text there is expression source, where a nested `"..."` is a string literal of its own: OpenTofu reads `"${upper("a")}"` as `A` and rejects `"${upper(\"a\")}"` outright, so escaping through an interpolation produced source the reference implementation will not parse, and resolving through one closed a nested literal early. Both directions now work on the spans the text is actually made of, and the scan knows the things inside an expression that can carry a non-structural brace: a string literal, HCL's `#`, `//` and `/* */` comments, and the nested expressions a string literal may itself contain -- OpenTofu evaluates `${1 /* } */ + 2}` to 3 and `"a ${upper("v${ "{" }w")} b"` to `a V{W b`, so counting either brace closed the expression inside itself. A heredoc body interprets no escape, so there a backslash is a character and the `${` after it still opens an expression: OpenTofu evaluates `<<EOF\nC:\${upper("a")}\nEOF` to `C:\A`. ([#339](https://github.com/amplify-education/python-hcl2/issues/339))
+- `dumps` writes a quoted string back split where the reader splits it. The regex it used knew nothing of string literals and then stripped a `"` from the edge of every piece rather than from the string once, so `"a \"${"b"}\" c"` came back as `"a \${"b"}\" c"`, which OpenTofu rejects with "Invalid escape sequence"; a flattened heredoc holding a JSON policy did the same, and a brace inside a nested literal raised `UnexpectedToken`. It now uses the same span scanner as the rest of this change.
+- Flattened heredoc bodies match the values Terraform and OpenTofu evaluate the same source to. Three things differed, all checked against OpenTofu v1.12.5 rather than read off the spec: the newline terminating the last content line was dropped (`<<EOT\nline\nEOT` returned `'line'`, not `'line\n'`); `<<-` measured its indent in spaces alone, so a tab-indented body was not dedented at all; and a whitespace-only line was excluded from the measurement but trimmed anyway. This is not a regression — 7.2.1 returned the same values — so it changes long-standing behaviour rather than restoring anything.
+- A carriage return in a flattened heredoc body is written as `\r` rather than left raw. `preserve_heredocs=False` returns quoted-string *source*, and a quoted string cannot hold a literal carriage return: OpenTofu rejects one with "No closing marker was found for the string". A heredoc read out of a CRLF file therefore flattened to source that would not parse again. The value form (`strip_string_quotes=True`) is unchanged and still hands back real carriage returns. `strings_to_heredocs` resolves `\r` when it writes a body, so the two halves stay each other's inverse: a heredoc interprets no escape, so a body carrying a backslash and an `r` would be those two characters rather than the carriage return the value held.
+- A `<<-` heredoc whose closing marker is indented with something other than spaces or tabs no longer appends that indentation to the value. The dedent already measured whitespace rather than spaces, matching OpenTofu, but the marker's own indent was stripped as `[ \t]*`, so a body indented with a non-breaking space, a vertical tab, a form feed or an ideographic space came back with one of those characters on the end. Whitespace here is Go's `unicode.IsSpace`, which OpenTofu measures with: the information separators U+001C to U+001F, which Python counts as whitespace, are content, so a line led by one sets the margin to zero rather than being dedented and losing them. Four such cases are now in the table that `bin/heredoc_ground_truth` re-derives from OpenTofu.
+- `strings_to_heredocs` leaves a value carrying a lone carriage return quoted. A heredoc body is read literally, so it can hold a `\r` only where one ends a line: OpenTofu rejects `<<EOF\nx\ry\nEOF` with "No closing marker was found for the string", while the quoted `"x\ry\n"` it came from is valid. Such a value stays quoted, for the same reason one that does not end in a newline does.
+- `strings_to_heredocs` picks a delimiter the body cannot close. It wrote `<<EOF` over every value, so a string holding a line reading `EOF` -- a log excerpt, a shell script, an embedded config, the payloads heredocs are for -- ended its own heredoc early and produced a file that no longer parsed. A numbered variant is used when the body occupies `EOF`, and ordinary values are written exactly as before. The lines that count as markers are Terraform's: OpenTofu ends a heredoc on the word padded with any whitespace at all, `EOF  `, a non-breaking space or a form feed included. A CRLF body counts too -- it is split on `\n`, so its lines carry their own `\r`, and OpenTofu ends a heredoc on `EOF\r` as readily as on `EOF `. ([#330](https://github.com/amplify-education/python-hcl2/issues/330))
+- `strings_to_heredocs` no longer adds a line to the body it writes. The value's own trailing newline is the one that precedes the closing marker, so a heredoc was being emitted one line longer than the string it came from. A value that does not end in a newline is now left as a quoted string, since no heredoc can express it. Flattening a document and restoring it now yields HCL that OpenTofu evaluates identically to the original; five of the eleven values in the round-trip fixture did not survive it before.
+
 ## \[8.1.4\] - 2026-09-08
 
 ### Fixed

CLAUDE.md

@@ -25,6 +25,7 @@ The **Direct** pipeline (`parse_to_tree` → `transform` → `to_lark` → `reco
 | `hcl2/builder.py` | Programmatic HCL document construction |
 | `hcl2/walk.py` | Generic tree-walking primitives for the LarkElement IR tree |
 | `hcl2/utils.py` | `SerializationOptions`, `SerializationContext`, string helpers |
+| `hcl2/template.py` | Splits template text into literal and `${...}`/`%{...}` spans, for quoted source or heredoc bodies |
 | `hcl2/const.py` | Constants: `IS_BLOCK`, `COMMENTS_KEY`, `INLINE_COMMENTS_KEY` |
 | `hcl2/cli/helpers.py` | File/directory/stdin conversion helpers |
 | `hcl2/cli/hcl_to_json.py` | `hcl2tojson` entry point |

bin/heredoc_ground_truth

@@ -0,0 +1,141 @@
+#!/usr/bin/env python
+"""Check the heredoc expectations in the test suite against Terraform itself.
+
+`test/unit/test_heredoc_matches_terraform.py` asserts what a heredoc body
+evaluates to. Those values did not come from this library or from reading the
+spec -- each one was produced by handing the same source to OpenTofu. That
+provenance is a docstring, which a reader has to take on trust and which
+nothing re-checks if the reference implementation ever moves.
+
+This script re-derives them. It reads the `CASES` table out of that test module,
+evaluates every source with `tofu console` (or `terraform console`), and
+compares. It is not part of the test run: the suite must not depend on a
+Terraform binary, and these values change about as often as the HCL spec does.
+
+Usage:
+    bin/heredoc_ground_truth            # verify; non-zero exit on any mismatch
+    bin/heredoc_ground_truth --print    # print the table as Python, to paste
+
+Requires `tofu` or `terraform` on PATH.
+"""
+
+import argparse
+import json
+import logging
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
+
+from test.unit.test_heredoc_matches_terraform import CASES  # noqa: E402
+
+BINARIES = ("tofu", "terraform")
+
+# Results go to stdout and diagnostics to stderr, so a piped run keeps the
+# table and its summary together while the binary in use stays visible.
+REPORT = logging.getLogger("heredoc_ground_truth.report")
+DIAGNOSTICS = logging.getLogger("heredoc_ground_truth.diagnostics")
+
+# Seconds one `console` evaluation may take. A binary waiting on a plugin cache
+# lock or a prompt would otherwise hang the run with nothing printed.
+TIMEOUT_SECONDS = 60
+
+
+def find_binary():
+    """Return the first Terraform-compatible binary on PATH, or None."""
+    for name in BINARIES:
+        path = shutil.which(name)
+        if path:
+            return path
+    return None
+
+
+def evaluate(binary, source):
+    """Return the value `binary` evaluates the given heredoc expression to.
+
+    The source is written as a local rather than an output so that nothing has
+    to be applied, and `jsonencode` is what carries the exact string back --
+    the console's own rendering escapes newlines for display.
+    """
+    with tempfile.TemporaryDirectory() as directory:
+        # newline="" so a case testing CRLF is written with the bytes it names.
+        with open(os.path.join(directory, "main.tf"), "w", encoding="utf-8", newline="") as handle:
+            handle.write("locals {\n  x = %s\n}\n" % source)
+        result = subprocess.run(
+            [binary, "console"],
+            cwd=directory,
+            input="jsonencode(local.x)\n",
+            capture_output=True,
+            text=True,
+            check=False,
+            timeout=TIMEOUT_SECONDS,
+        )
+        if result.returncode != 0:
+            raise RuntimeError(result.stderr.strip())
+        return json.loads(json.loads(result.stdout.strip().splitlines()[-1]))
+
+
+def configure_logging():
+    """Send each logger to its stream as bare messages, the lines the table is made of."""
+    for logger, stream in ((REPORT, sys.stdout), (DIAGNOSTICS, sys.stderr)):
+        handler = logging.StreamHandler(stream)
+        handler.setFormatter(logging.Formatter("%(message)s"))
+        logger.addHandler(handler)
+        logger.setLevel(logging.INFO)
+        logger.propagate = False
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    parser.add_argument(
+        "--print",
+        dest="print_table",
+        action="store_true",
+        help="print the evaluated table as Python instead of verifying",
+    )
+    args = parser.parse_args()
+
+    configure_logging()
+    binary = find_binary()
+    if binary is None:
+        DIAGNOSTICS.error("neither `tofu` nor `terraform` is on PATH")
+        return 2
+
+    DIAGNOSTICS.info("using %s\n", binary)
+    mismatches = 0
+    for source, expected in CASES:
+        # One case the binary rejects or stalls on is a result for that case,
+        # not a reason to lose the rest of the table and the summary.
+        try:
+            actual = evaluate(binary, source)
+        except subprocess.TimeoutExpired:
+            mismatches += 1
+            REPORT.error("ERR  %r\n       %s timed out after %ds", source, binary, TIMEOUT_SECONDS)
+            continue
+        except RuntimeError as error:
+            mismatches += 1
+            REPORT.error("ERR  %r\n       %s: %s", source, binary, error)
+            continue
+        if args.print_table:
+            REPORT.info("    (%r, %r),", source, actual)
+            continue
+        if actual == expected:
+            REPORT.info("ok   %r", source)
+        else:
+            mismatches += 1
+            REPORT.warning("BAD  %r\n       expected %r\n       %s says %r", source, expected, binary, actual)
+
+    if args.print_table:
+        return 1 if mismatches else 0
+
+    # stdout, so it lands after the per-case lines rather than ahead of them
+    # when the output is piped.
+    REPORT.info("\n%d of %d cases disagree", mismatches, len(CASES))
+    return 1 if mismatches else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

docs/01_getting_started.md

@@ -74,7 +74,7 @@ data = loads(text, serialization_options=SerializationOptions(
 | `wrap_objects` | `bool` | `False` | Wrap object values as inline HCL2 strings                                                                                                       |
 | `wrap_tuples` | `bool` | `False` | Wrap tuple values as inline HCL2 strings                                                                                                        |
 | `explicit_blocks` | `bool` | `True` | Add `__is_block__: True` markers to blocks. **Mandatory for JSON->HCL2 deserialization and reconstruction.**                                    |
-| `preserve_heredocs` | `bool` | `True` | Keep heredocs in their original form, markers included. When `False`, a heredoc becomes a quoted string with its newlines escaped (`'"a\nb"'`); combine with `strip_string_quotes` to get the body as a plain multi-line value instead. |
+| `preserve_heredocs` | `bool` | `True` | Keep heredocs in their original form, markers included. When `False`, a heredoc becomes a quoted string with its newlines escaped (`'"a\nb\n"'`); combine with `strip_string_quotes` to get the body as a plain multi-line value instead. Either way the body keeps the newline that terminates its last line, as Terraform's does. |
 | `force_operation_parentheses` | `bool` | `False` | Force parentheses around all operations                                                                                                         |
 | `preserve_scientific_notation` | `bool` | `True` | Keep scientific notation as-is                                                                                                                  |
 | `strip_string_quotes` | `bool` | `False` | Yield string *values* rather than source text: remove surrounding quotes (e.g. `"hello"` instead of `'"hello"'`) and resolve escape sequences (`"a\nb"` becomes a real newline). String literals inside expressions keep their quotes, so `upper("x")` stays `'${upper("x")}'`. **Breaks JSON->HCL2 deserialization and reconstruction.** |
@@ -127,7 +127,7 @@ text = dumps(data, deserializer_options=DeserializerOptions(
 | Field | Type | Default | Description |
 |---|---|---|---|
 | `heredocs_to_strings` | `bool` | `False` | Convert heredocs to plain strings |
-| `strings_to_heredocs` | `bool` | `False` | Convert strings with `\n` to heredocs |
+| `strings_to_heredocs` | `bool` | `False` | Convert newline-terminated strings to heredocs. A value that does not end in a newline is left as a quoted string, because a heredoc body always ends in one and writing it as a heredoc would change the value. |
 | `object_elements_colon` | `bool` | `False` | Use `:` instead of `=` in object elements |
 | `object_elements_trailing_comma` | `bool` | `True` | Add trailing commas in object elements |
 

docs/06_migrating_to_v8.md

@@ -215,17 +215,18 @@ This restores the v7 dict shape but disables round-trip support and comment pres
 
 ```python
 hcl2.loads('x = <<-EOT\n  line1\n  line2\n  EOT\n', serialization_options=V7_COMPAT)
-# {'x': 'line1\nline2'}
+# {'x': 'line1\nline2\n'}
 ```
 
-Note that `preserve_heredocs=False` on its own — without `strip_string_quotes` — produces the quoted *source* form with escaped newlines (`'"line1\\nline2"'`), because that output is meant to be reconstructable.
+Note that `preserve_heredocs=False` on its own — without `strip_string_quotes` — produces the quoted *source* form with escaped newlines (`'"line1\\nline2\\n"'`), because that output is meant to be reconstructable.
 
-Two details of heredoc values are easy to trip over, and both match how HCL itself behaves:
+Three details of heredoc values are easy to trip over, and all three match how HCL itself behaves:
 
+- **The body ends with a newline.** Every content line is terminated by its own newline, the last one included, so `<<EOT\nline\nEOT` is `'line\n'` — the same value Terraform evaluates it to. v7 returned `'line'`, and so did 8.1.x; both were wrong. Only an empty body has no trailing newline, because it has no content line.
 - **Backslash escapes are not interpreted in heredocs.** `strip_string_quotes` resolves `\n` inside a *quoted* string, but a heredoc body containing the two characters `\n` keeps them verbatim. HCL only processes escape sequences in quoted templates.
 - **Line endings come through as written.** A heredoc in a CRLF file yields a body with `\r\n`, because a carriage return inside the body is content rather than structure. Normalize on your side if you need `\n`.
 
 ```python
 hcl2.loads('x = <<EOT\na\\nb\nEOT\n', serialization_options=V7_COMPAT)
-# {'x': 'a\\nb'}  — the backslash and the "n" are two literal characters
+# {'x': 'a\\nb\n'}  — the backslash and the "n" are two literal characters
 ```

hcl2/cli/json_to_hcl.py

@@ -181,7 +181,7 @@ def main():  # pylint: disable=too-many-branches,too-many-statements,too-many-lo
     parser.add_argument(
         "--strings-to-heredocs",
         action="store_true",
-        help="Convert strings containing escaped newlines to heredocs",
+        help="Convert newline-terminated escaped strings to heredocs",
     )
 
     # FormatterOptions flags