fix: a heredoc ends its own line wherever it is written (#338)

CHANGELOG.md

@@ -24,6 +24,17 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 - Python 3.14 is now tested and declared as supported. No source changes were needed; the full
   suite passes on 3.14 as-is.
 
+### Fixed
+
+- Flattened heredoc bodies match the values Terraform and OpenTofu evaluate the same source to. Three things differed, all checked against OpenTofu v1.12.5 rather than read off the spec: the newline terminating the last content line was dropped (`<<EOT\nline\nEOT` returned `'line'`, not `'line\n'`); `<<-` measured its indent in spaces alone, so a tab-indented body was not dedented at all; and a whitespace-only line was excluded from the measurement but trimmed anyway. This is not a regression — 7.2.1 returned the same values — so it changes long-standing behaviour rather than restoring anything.
+- A carriage return in a flattened heredoc body is written as `\r` rather than left raw. `preserve_heredocs=False` returns quoted-string *source*, and a quoted string cannot hold a literal carriage return: OpenTofu rejects one with "No closing marker was found for the string". A heredoc read out of a CRLF file therefore flattened to source that would not parse again. The value form (`strip_string_quotes=True`) is unchanged and still hands back real carriage returns. `strings_to_heredocs` resolves `\r` when it writes a body, so the two halves stay each other's inverse: a heredoc interprets no escape, so a body carrying a backslash and an `r` would be those two characters rather than the carriage return the value held.
+- A heredoc written inside a list or an object ends its own line, `<<-` as well as `<<`. It ends at its closing marker, on a line of its own, so what follows has to start the next one -- `EOF,` closes nothing, and the file this library had just written did not parse, here or in Terraform. A top-level attribute survived only because the newline after it comes from the document. A heredoc built from a dict is now the same token a parsed one is: the grammar's terminal name, running through the newline after the marker, so it ends its line wherever it is written and reconstructing a parsed document is still byte for byte what it was. In an object that line break is the separator: OpenTofu rejects a comma at the start of the next line, so none is written there, whether the object is written out or inline inside an expression. ([#338](https://github.com/amplify-education/python-hcl2/issues/338))
+- A heredoc written as a function argument or an operand loads as the heredoc it is. It came back quoted, markers and all: `upper(<<EOF\nfoo\nEOF\n)` loaded as `${upper("<<EOF\nfoo\nEOF")}`, which `dumps` wrote as a multi-line quoted string that OpenTofu rejects with "Invalid multi-line string" and that was a different value besides. It now loads as `${upper(<<EOF\nfoo\nEOF\n)}`, keeping the newline after the closing marker so the `)` starts the next line, and `dumps` writes the source back. **This changes the default output** for any document with a heredoc inside an expression; a heredoc that is a whole attribute value is unchanged.
+- A `<<-` heredoc whose closing marker is indented with something other than spaces or tabs no longer appends that indentation to the value. The dedent already measured whitespace rather than spaces, matching OpenTofu, but the marker's own indent was stripped as `[ \t]*`, so a body indented with a non-breaking space, a vertical tab, a form feed or an ideographic space came back with one of those characters on the end. Whitespace here is Go's `unicode.IsSpace`, which OpenTofu measures with: the information separators U+001C to U+001F, which Python counts as whitespace, are content, so a line led by one sets the margin to zero rather than being dedented and losing them. Four such cases are now in the table that `bin/heredoc_ground_truth` re-derives from OpenTofu.
+- `strings_to_heredocs` leaves a value carrying a lone carriage return quoted. A heredoc body is read literally, so it can hold a `\r` only where one ends a line: OpenTofu rejects `<<EOF\nx\ry\nEOF` with "No closing marker was found for the string", while the quoted `"x\ry\n"` it came from is valid. Such a value stays quoted, for the same reason one that does not end in a newline does.
+- `strings_to_heredocs` picks a delimiter the body cannot close. It wrote `<<EOF` over every value, so a string holding a line reading `EOF` -- a log excerpt, a shell script, an embedded config, the payloads heredocs are for -- ended its own heredoc early and produced a file that no longer parsed. A numbered variant is used when the body occupies `EOF`, and ordinary values are written exactly as before. The lines that count as markers are Terraform's: OpenTofu ends a heredoc on the word padded with any whitespace at all, `EOF  `, a non-breaking space or a form feed included. A CRLF body counts too -- it is split on `\n`, so its lines carry their own `\r`, and OpenTofu ends a heredoc on `EOF\r` as readily as on `EOF `. ([#330](https://github.com/amplify-education/python-hcl2/issues/330))
+- `strings_to_heredocs` no longer adds a line to the body it writes. The value's own trailing newline is the one that precedes the closing marker, so a heredoc was being emitted one line longer than the string it came from. A value that does not end in a newline is now left as a quoted string, since no heredoc can express it. Flattening a document and restoring it now yields HCL that OpenTofu evaluates identically to the original; five of the eleven values in the round-trip fixture did not survive it before.
+
 ## \[8.1.4\] - 2026-09-08
 
 ### Fixed

bin/heredoc_ground_truth

@@ -0,0 +1,141 @@
+#!/usr/bin/env python
+"""Check the heredoc expectations in the test suite against Terraform itself.
+
+`test/unit/test_heredoc_matches_terraform.py` asserts what a heredoc body
+evaluates to. Those values did not come from this library or from reading the
+spec -- each one was produced by handing the same source to OpenTofu. That
+provenance is a docstring, which a reader has to take on trust and which
+nothing re-checks if the reference implementation ever moves.
+
+This script re-derives them. It reads the `CASES` table out of that test module,
+evaluates every source with `tofu console` (or `terraform console`), and
+compares. It is not part of the test run: the suite must not depend on a
+Terraform binary, and these values change about as often as the HCL spec does.
+
+Usage:
+    bin/heredoc_ground_truth            # verify; non-zero exit on any mismatch
+    bin/heredoc_ground_truth --print    # print the table as Python, to paste
+
+Requires `tofu` or `terraform` on PATH.
+"""
+
+import argparse
+import json
+import logging
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
+
+from test.unit.test_heredoc_matches_terraform import CASES  # noqa: E402
+
+BINARIES = ("tofu", "terraform")
+
+# Results go to stdout and diagnostics to stderr, so a piped run keeps the
+# table and its summary together while the binary in use stays visible.
+REPORT = logging.getLogger("heredoc_ground_truth.report")
+DIAGNOSTICS = logging.getLogger("heredoc_ground_truth.diagnostics")
+
+# Seconds one `console` evaluation may take. A binary waiting on a plugin cache
+# lock or a prompt would otherwise hang the run with nothing printed.
+TIMEOUT_SECONDS = 60
+
+
+def find_binary():
+    """Return the first Terraform-compatible binary on PATH, or None."""
+    for name in BINARIES:
+        path = shutil.which(name)
+        if path:
+            return path
+    return None
+
+
+def evaluate(binary, source):
+    """Return the value `binary` evaluates the given heredoc expression to.
+
+    The source is written as a local rather than an output so that nothing has
+    to be applied, and `jsonencode` is what carries the exact string back --
+    the console's own rendering escapes newlines for display.
+    """
+    with tempfile.TemporaryDirectory() as directory:
+        # newline="" so a case testing CRLF is written with the bytes it names.
+        with open(os.path.join(directory, "main.tf"), "w", encoding="utf-8", newline="") as handle:
+            handle.write("locals {\n  x = %s\n}\n" % source)
+        result = subprocess.run(
+            [binary, "console"],
+            cwd=directory,
+            input="jsonencode(local.x)\n",
+            capture_output=True,
+            text=True,
+            check=False,
+            timeout=TIMEOUT_SECONDS,
+        )
+        if result.returncode != 0:
+            raise RuntimeError(result.stderr.strip())
+        return json.loads(json.loads(result.stdout.strip().splitlines()[-1]))
+
+
+def configure_logging():
+    """Send each logger to its stream as bare messages, the lines the table is made of."""
+    for logger, stream in ((REPORT, sys.stdout), (DIAGNOSTICS, sys.stderr)):
+        handler = logging.StreamHandler(stream)
+        handler.setFormatter(logging.Formatter("%(message)s"))
+        logger.addHandler(handler)
+        logger.setLevel(logging.INFO)
+        logger.propagate = False
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    parser.add_argument(
+        "--print",
+        dest="print_table",
+        action="store_true",
+        help="print the evaluated table as Python instead of verifying",
+    )
+    args = parser.parse_args()
+
+    configure_logging()
+    binary = find_binary()
+    if binary is None:
+        DIAGNOSTICS.error("neither `tofu` nor `terraform` is on PATH")
+        return 2
+
+    DIAGNOSTICS.info("using %s\n", binary)
+    mismatches = 0
+    for source, expected in CASES:
+        # One case the binary rejects or stalls on is a result for that case,
+        # not a reason to lose the rest of the table and the summary.
+        try:
+            actual = evaluate(binary, source)
+        except subprocess.TimeoutExpired:
+            mismatches += 1
+            REPORT.error("ERR  %r\n       %s timed out after %ds", source, binary, TIMEOUT_SECONDS)
+            continue
+        except RuntimeError as error:
+            mismatches += 1
+            REPORT.error("ERR  %r\n       %s: %s", source, binary, error)
+            continue
+        if args.print_table:
+            REPORT.info("    (%r, %r),", source, actual)
+            continue
+        if actual == expected:
+            REPORT.info("ok   %r", source)
+        else:
+            mismatches += 1
+            REPORT.warning("BAD  %r\n       expected %r\n       %s says %r", source, expected, binary, actual)
+
+    if args.print_table:
+        return 1 if mismatches else 0
+
+    # stdout, so it lands after the per-case lines rather than ahead of them
+    # when the output is piped.
+    REPORT.info("\n%d of %d cases disagree", mismatches, len(CASES))
+    return 1 if mismatches else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

docs/01_getting_started.md

@@ -74,7 +74,7 @@ data = loads(text, serialization_options=SerializationOptions(
 | `wrap_objects` | `bool` | `False` | Wrap object values as inline HCL2 strings                                                                                                       |
 | `wrap_tuples` | `bool` | `False` | Wrap tuple values as inline HCL2 strings                                                                                                        |
 | `explicit_blocks` | `bool` | `True` | Add `__is_block__: True` markers to blocks. **Mandatory for JSON->HCL2 deserialization and reconstruction.**                                    |
-| `preserve_heredocs` | `bool` | `True` | Keep heredocs in their original form, markers included. When `False`, a heredoc becomes a quoted string with its newlines escaped (`'"a\nb"'`); combine with `strip_string_quotes` to get the body as a plain multi-line value instead. |
+| `preserve_heredocs` | `bool` | `True` | Keep heredocs in their original form, markers included. When `False`, a heredoc becomes a quoted string with its newlines escaped (`'"a\nb\n"'`); combine with `strip_string_quotes` to get the body as a plain multi-line value instead. Either way the body keeps the newline that terminates its last line, as Terraform's does. |
 | `force_operation_parentheses` | `bool` | `False` | Force parentheses around all operations                                                                                                         |
 | `preserve_scientific_notation` | `bool` | `True` | Keep scientific notation as-is                                                                                                                  |
 | `strip_string_quotes` | `bool` | `False` | Yield string *values* rather than source text: remove surrounding quotes (e.g. `"hello"` instead of `'"hello"'`) and resolve escape sequences (`"a\nb"` becomes a real newline). String literals inside expressions keep their quotes, so `upper("x")` stays `'${upper("x")}'`. **Breaks JSON->HCL2 deserialization and reconstruction.** |
@@ -127,7 +127,7 @@ text = dumps(data, deserializer_options=DeserializerOptions(
 | Field | Type | Default | Description |
 |---|---|---|---|
 | `heredocs_to_strings` | `bool` | `False` | Convert heredocs to plain strings |
-| `strings_to_heredocs` | `bool` | `False` | Convert strings with `\n` to heredocs |
+| `strings_to_heredocs` | `bool` | `False` | Convert newline-terminated strings to heredocs. A value that does not end in a newline is left as a quoted string, because a heredoc body always ends in one and writing it as a heredoc would change the value. |
 | `object_elements_colon` | `bool` | `False` | Use `:` instead of `=` in object elements |
 | `object_elements_trailing_comma` | `bool` | `True` | Add trailing commas in object elements |
 

docs/06_migrating_to_v8.md

@@ -215,17 +215,18 @@ This restores the v7 dict shape but disables round-trip support and comment pres
 
 ```python
 hcl2.loads('x = <<-EOT\n  line1\n  line2\n  EOT\n', serialization_options=V7_COMPAT)
-# {'x': 'line1\nline2'}
+# {'x': 'line1\nline2\n'}
 ```
 
-Note that `preserve_heredocs=False` on its own — without `strip_string_quotes` — produces the quoted *source* form with escaped newlines (`'"line1\\nline2"'`), because that output is meant to be reconstructable.
+Note that `preserve_heredocs=False` on its own — without `strip_string_quotes` — produces the quoted *source* form with escaped newlines (`'"line1\\nline2\\n"'`), because that output is meant to be reconstructable.
 
-Two details of heredoc values are easy to trip over, and both match how HCL itself behaves:
+Three details of heredoc values are easy to trip over, and all three match how HCL itself behaves:
 
+- **The body ends with a newline.** Every content line is terminated by its own newline, the last one included, so `<<EOT\nline\nEOT` is `'line\n'` — the same value Terraform evaluates it to. v7 returned `'line'`, and so did 8.1.x; both were wrong. Only an empty body has no trailing newline, because it has no content line.
 - **Backslash escapes are not interpreted in heredocs.** `strip_string_quotes` resolves `\n` inside a *quoted* string, but a heredoc body containing the two characters `\n` keeps them verbatim. HCL only processes escape sequences in quoted templates.
 - **Line endings come through as written.** A heredoc in a CRLF file yields a body with `\r\n`, because a carriage return inside the body is content rather than structure. Normalize on your side if you need `\n`.
 
 ```python
 hcl2.loads('x = <<EOT\na\\nb\nEOT\n', serialization_options=V7_COMPAT)
-# {'x': 'a\\nb'}  — the backslash and the "n" are two literal characters
+# {'x': 'a\\nb\n'}  — the backslash and the "n" are two literal characters
 ```

hcl2/cli/json_to_hcl.py

@@ -181,7 +181,7 @@ def main():  # pylint: disable=too-many-branches,too-many-statements,too-many-lo
     parser.add_argument(
         "--strings-to-heredocs",
         action="store_true",
-        help="Convert strings containing escaped newlines to heredocs",
+        help="Convert newline-terminated escaped strings to heredocs",
     )
 
     # FormatterOptions flags