diff --git a/CHANGELOG.md b/CHANGELOG.md index b8fda71d..85a94ad4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,13 +17,31 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. `python -m hcl2` are unaffected; only code importing `cli.hcl_to_json`, `cli.json_to_hcl`, `cli.hq`, or `cli.helpers` needs to add the `hcl2.` prefix. No compatibility shim ships, because a shim would still occupy the colliding name. + - The redundant `cli/py.typed` marker is gone; `hcl2/py.typed` already covers `hcl2.cli`. +- The `regex` package is no longer a dependency. Nothing imports it now that quoted strings are split by the span-aware template scanner; `lark` is the only runtime requirement. + ### Added - Python 3.14 is now tested and declared as supported. No source changes were needed; the full suite passes on 3.14 as-is. +### Fixed + +- A heredoc whose interpolation spans lines is not flattened. The quoted form cannot hold one: the newlines inside `${...}` are expression source, where OpenTofu rejects an escaped newline and a raw one makes the string span lines, which it also rejects. It used to emit the raw version -- output neither Terraform nor this library could read, written with no error -- and now hands the heredoc back in the form `preserve_heredocs=True` produces, which reads back as that heredoc. Declining is the only answer that does not change what the document means. The value form, which can carry such a body, measures a `<<-` margin on the lines that start with literal text: a line that begins inside `${...}` is expression source, and OpenTofu does not let its indent lower the margin of the rest. The same holds for a heredoc whose template uses a `~` strip marker: a heredoc body is lexed one line at a time, so the marker strips no further than its own line, while in the quoted string the same whitespace runs on into the next line's indent -- OpenTofu evaluates the loop `%{ for s in ["a", "b"] ~}\n - ${s}\n%{ endfor ~}` to ` - a\n - b\n` as a heredoc and to `- a\n- b\n` flattened. A declined heredoc written as a function argument goes back as the heredoc itself rather than as a quoted copy of its source. ([#347](https://github.com/amplify-education/python-hcl2/issues/347)) +- A quoted string can use a strip marker on an interpolation, `${~ ...}` or `${... ~}`. The grammar gave the marker to template directives only, so such a string did not parse at all, in `loads` or in `dumps` of a dict that carried one. OpenTofu evaluates `"a ${~ "b"} c"` to `ab c`; the marker is kept, spaced as a directive's is. +- `$${` and `%%{` resolve to `${` and `%{` in the value form, in both quoted strings and heredocs. They are HCL's escapes for a literal sigil, exactly as `\"` is for a quote, and OpenTofu evaluates `"$${esc}"` to the six characters `${esc}`; returning them doubled made the value differ from the one Terraform reads, in the one mode that promises the value. The escapes written after one resolve as well, since the whole run is literal text: `"$${a\tb}"` is `${ab}`, and so is the literal text between two directives. ([#336](https://github.com/amplify-education/python-hcl2/issues/336)) +- `strings_to_heredocs` resolves every escape the reader does. It knew `\n`, `\r`, `\"` and `\\`, so `"a\tb\n"` was written into the body as a backslash and a `t` -- two characters where Terraform reads one tab -- and `\uNNNN` fared the same. It now uses `process_escape_sequences`, the package's one implementation of that alphabet. ([#329](https://github.com/amplify-education/python-hcl2/issues/329)) +- Escapes are no longer added or resolved inside `${...}`. The text there is expression source, where a nested `"..."` is a string literal of its own: OpenTofu reads `"${upper("a")}"` as `A` and rejects `"${upper(\"a\")}"` outright, so escaping through an interpolation produced source the reference implementation will not parse, and resolving through one closed a nested literal early. Both directions now work on the spans the text is actually made of, and the scan knows the things inside an expression that can carry a non-structural brace: a string literal, HCL's `#`, `//` and `/* */` comments, and the nested expressions a string literal may itself contain -- OpenTofu evaluates `${1 /* } */ + 2}` to 3 and `"a ${upper("v${ "{" }w")} b"` to `a V{W b`, so counting either brace closed the expression inside itself. A heredoc body interprets no escape, so there a backslash is a character and the `${` after it still opens an expression: OpenTofu evaluates `<HCL2 deserialization and reconstruction.** | -| `preserve_heredocs` | `bool` | `True` | Keep heredocs in their original form, markers included. When `False`, a heredoc becomes a quoted string with its newlines escaped (`'"a\nb"'`); combine with `strip_string_quotes` to get the body as a plain multi-line value instead. | +| `preserve_heredocs` | `bool` | `True` | Keep heredocs in their original form, markers included. When `False`, a heredoc becomes a quoted string with its newlines escaped (`'"a\nb\n"'`); combine with `strip_string_quotes` to get the body as a plain multi-line value instead. Either way the body keeps the newline that terminates its last line, as Terraform's does. | | `force_operation_parentheses` | `bool` | `False` | Force parentheses around all operations | | `preserve_scientific_notation` | `bool` | `True` | Keep scientific notation as-is | | `strip_string_quotes` | `bool` | `False` | Yield string *values* rather than source text: remove surrounding quotes (e.g. `"hello"` instead of `'"hello"'`) and resolve escape sequences (`"a\nb"` becomes a real newline). String literals inside expressions keep their quotes, so `upper("x")` stays `'${upper("x")}'`. **Breaks JSON->HCL2 deserialization and reconstruction.** | @@ -127,7 +127,7 @@ text = dumps(data, deserializer_options=DeserializerOptions( | Field | Type | Default | Description | |---|---|---|---| | `heredocs_to_strings` | `bool` | `False` | Convert heredocs to plain strings | -| `strings_to_heredocs` | `bool` | `False` | Convert strings with `\n` to heredocs | +| `strings_to_heredocs` | `bool` | `False` | Convert newline-terminated strings to heredocs. A value that does not end in a newline is left as a quoted string, because a heredoc body always ends in one and writing it as a heredoc would change the value. | | `object_elements_colon` | `bool` | `False` | Use `:` instead of `=` in object elements | | `object_elements_trailing_comma` | `bool` | `True` | Add trailing commas in object elements | diff --git a/docs/06_migrating_to_v8.md b/docs/06_migrating_to_v8.md index b3ce260a..66f9b39c 100644 --- a/docs/06_migrating_to_v8.md +++ b/docs/06_migrating_to_v8.md @@ -215,17 +215,18 @@ This restores the v7 dict shape but disables round-trip support and comment pres ```python hcl2.loads('x = <<-EOT\n line1\n line2\n EOT\n', serialization_options=V7_COMPAT) -# {'x': 'line1\nline2'} +# {'x': 'line1\nline2\n'} ``` -Note that `preserve_heredocs=False` on its own — without `strip_string_quotes` — produces the quoted *source* form with escaped newlines (`'"line1\\nline2"'`), because that output is meant to be reconstructable. +Note that `preserve_heredocs=False` on its own — without `strip_string_quotes` — produces the quoted *source* form with escaped newlines (`'"line1\\nline2\\n"'`), because that output is meant to be reconstructable. -Two details of heredoc values are easy to trip over, and both match how HCL itself behaves: +Three details of heredoc values are easy to trip over, and all three match how HCL itself behaves: +- **The body ends with a newline.** Every content line is terminated by its own newline, the last one included, so `< bool: + r"""Whether *inner* could possibly resolve to a newline-terminated value. + + Only such a value can be written as a heredoc, and resolving the escapes to + find out costs a pass over the whole string -- for a document where nothing + ends in a newline, every one of those passes is discarded. The last two + characters answer it: a value ends with a newline only if its source ends + with one, escaped or real. + + Conservative on purpose. `"a\\n"` ends with a backslash and an `n` and + passes here, then resolves to those two characters and is rejected by the + check that actually matters. + """ + return inner.endswith("\\n") or inner.endswith("\n") + + +def _unescape_heredoc_body(inner: str) -> str: + r"""Resolve a quoted string's escapes for a body that interprets none. + + A heredoc body is read literally, so anything the quoted form spelled as + an escape has to become the character itself: `\t` a tab, `\u00e9` an + accented e. `process_escape_sequences` is the package's one implementation + of that alphabet, and using it here is what stops this path from resolving + a shorter list than the reader does. + + Only in literal spans. Inside `${...}` the text is expression source, and + an escape there belongs to a string literal written inside the expression: + OpenTofu reads `"${upper("a\"b")}"` as `A"B`, so resolving that `\"` would + close the nested literal early and change what the expression says. + """ + return map_literal_spans(inner, process_escape_sequences) + + +# A line that could end a heredoc: the delimiter word alone, give or take +# surrounding whitespace. Any whitespace, not just spaces and tabs: OpenTofu +# v1.12.6 ends `< str: + """Return a delimiter the body does not close on its own. + + `EOF` unless the body holds a line that would end the heredoc there, in + which case a numbered variant is used. The word matters: a log excerpt, a + shell script or an embedded config is exactly the sort of value people put + in a heredoc, and `EOF` is exactly the word such a payload tends to + contain. Writing one blindly produced a file that no longer parsed. + """ + occupied = set() + for line in content.split("\n"): + match = _CLOSING_MARKER_LINE.fullmatch(line) + if match is not None: + occupied.add(match.group(1)) + + if "EOF" not in occupied: + return "EOF" + + suffix = 1 + while f"EOF_{suffix}" in occupied: + suffix += 1 + return f"EOF_{suffix}" + + +# The grammar's ESCAPED_INTERPOLATION and ESCAPED_DIRECTIVE terminals. +_ESCAPED_MARKER = re.compile(r"\$\$\{[^}]*\}|%%\{[^}]*\}") + + +def _interpolation_spans(text: str, heredoc: bool = False) -> List[str]: + """The `${...}` and `%{...}` spans of *text*, in order.""" + return [chunk for kind, chunk in split_template(text, heredoc) if kind == INTERPOLATION] + + +def _expressible_as_heredoc(content: str, source: str) -> bool: + """Whether *content* can be a heredoc body without changing. + + A heredoc body is read literally, so it can hold a carriage return only + where one ends a line. A lone `\r` makes the file unreadable rather than + merely different: OpenTofu rejects `< LarkRule: if match: return self._deserialize_heredoc(value[1:-1], False) - if self.options.strings_to_heredocs: - inner = value[1:-1] - if "\\n" in inner: - return self._deserialize_string_as_heredoc(inner) + if self.options.strings_to_heredocs and _may_end_a_heredoc(value[1:-1]): + content = _unescape_heredoc_body(value[1:-1]) + # A heredoc's closing marker sits on a line of its own, so + # any body with content in it ends with a newline. A value + # that does not cannot be written as one without gaining + # that character, so it stays a quoted string. + # + # The empty string is the one value this excludes that a + # heredoc could in fact express -- `< StringRule: if "%{" in stripped: return self._deserialize_string_via_parser(value) + # Split where the reader would: `split_template` knows a string + # literal inside `${...}` is a template of its own, so a quote or a + # brace in one does not end the span. The regex this replaced did + # not, and it then stripped a `"` from the edge of every piece rather + # than from the string once -- `"a \"${"b"}\" c"` lost the quote of + # its `\"` to that and came back as source OpenTofu rejects. result = [] - # split string into individual parts based on lark grammar - # e.g. 'aaa$${bbb}ccc${"ddd-${eee}"}' -> ['aaa', '$${bbb}', 'ccc', '${"ddd-${eee}"}'] - # 'aa-${"bb-${"cc-${"dd-${5 + 5}"}"}"}' -> ['aa-', '${"bb-${"cc-${"dd-${5 + 5}"}"}"}'] - pattern = regex.compile(r"(\${1,2}\{(?:[^{}]|(?R))*\})") - parts = [part for part in pattern.split(value) if part != ""] - - for part in parts: - if part == '"': + for kind, chunk in split_template(inner): + if kind == INTERPOLATION: + result.append(self._deserialize_string_part(chunk)) continue - - if part.startswith('"'): - part = part[1:] - if part.endswith('"'): - part = part[:-1] - - string_part = self._deserialize_string_part(part) - result.append(string_part) - + # A literal stretch may hold the `$${...}`/`%%{...}` escapes, which + # the grammar tokenizes on their own. + position = 0 + for match in _ESCAPED_MARKER.finditer(chunk): + if match.start() > position: + result.append(self._deserialize_string_part(chunk[position : match.start()])) + result.append(self._deserialize_string_part(match.group())) + position = match.end() + if position < len(chunk): + result.append(self._deserialize_string_part(chunk[position:])) + + if not result: + # `""` keeps the one empty part it has always had. + result.append(self._deserialize_string_part("")) return StringRule([DBLQUOTE(), *result, DBLQUOTE()]) def _deserialize_string_via_parser(self, value: str) -> StringRule: @@ -245,10 +361,24 @@ def _deserialize_string_part(self, value: str) -> StringPartRule: if value.startswith("$${") and value.endswith("}"): return StringPartRule([ESCAPED_INTERPOLATION(value)]) + if value.startswith("%%{") and value.endswith("}"): + return StringPartRule([ESCAPED_DIRECTIVE(value)]) + if value.startswith("${") and value.endswith("}"): - return StringPartRule( - [InterpolationRule([INTERP_START(), self._deserialize_expression(value), RBRACE()])] - ) + # A strip marker sits against the braces -- `${~` and `~}` -- and + # is not part of the expression, which would not parse with it. + body = value[2:-1] + strip_open = body.startswith("~") + strip_close = body.endswith("~") + body = body[1 if strip_open else 0 : len(body) - 1 if strip_close else len(body)].strip() + children: List[Any] = [INTERP_START()] + if strip_open: + children.append(STRIP_MARKER()) + children.append(self._deserialize_expression("${" + body + "}")) + if strip_close: + children.append(STRIP_MARKER()) + children.append(RBRACE()) + return StringPartRule([InterpolationRule(children)]) return StringPartRule([STRING_CHARS(value)]) @@ -259,15 +389,10 @@ def _deserialize_heredoc( return HeredocTrimTemplateRule([HEREDOC_TRIM_TEMPLATE(value)]) return HeredocTemplateRule([HEREDOC_TEMPLATE(value)]) - def _deserialize_string_as_heredoc(self, inner: str) -> HeredocTemplateRule: - """Convert a quoted string with escaped newlines back into a heredoc.""" - # Single-pass unescape: \\n → \n, \\" → ", \\\\ → \ - content = re.sub( - r'\\(n|"|\\)', - lambda m: "\n" if m.group(1) == "n" else m.group(1), - inner, - ) - heredoc = f"< HeredocTemplateRule: + """Wrap an unescaped body, already newline-terminated, in heredoc syntax.""" + delimiter = _heredoc_delimiter(content) + heredoc = f"<<{delimiter}\n{content}{delimiter}" return HeredocTemplateRule([HEREDOC_TEMPLATE(heredoc)]) def _deserialize_expression(self, value: str) -> ExprTermRule: diff --git a/hcl2/hcl2.lark b/hcl2/hcl2.lark index 13ddb006..c99a62b9 100644 --- a/hcl2/hcl2.lark +++ b/hcl2/hcl2.lark @@ -141,7 +141,7 @@ string_part: STRING_CHARS // Expressions ?expression : or_expr QMARK new_line_or_comment? expression new_line_or_comment? COLON new_line_or_comment? expression -> conditional | or_expr -interpolation: INTERP_START expression RBRACE +interpolation: INTERP_START STRIP_MARKER? expression STRIP_MARKER? RBRACE // Template directives (flat rules — transformer assembles if/for structure) template_if_start: DIRECTIVE_START STRIP_MARKER? IF expression STRIP_MARKER? RBRACE diff --git a/hcl2/reconstructor.py b/hcl2/reconstructor.py index 166e6c58..30a659b7 100644 --- a/hcl2/reconstructor.py +++ b/hcl2/reconstructor.py @@ -66,6 +66,7 @@ def _reset_state(self): self._last_was_space = True self._current_indent = 0 self._last_token_name = None + self._token_before_last_name: Optional[str] = None self._last_rule_name = None # pylint:disable=R0911,R0912 @@ -85,6 +86,19 @@ def _should_add_space_before( if isinstance(current_node, Token): token_type = current_node.type + # `${~ expr ~}`: an interpolation's strip markers are spaced as a + # directive's are -- none against the braces, one toward the + # expression. The expression's first token belongs to another + # rule, so the opening marker is recognised by what preceded it. + strip = tokens.STRIP_MARKER.lark_name() + if ( + self._last_token_name == strip + and self._token_before_last_name == tokens.INTERP_START.lark_name() + ): + return True + if token_type == strip and parent_rule_name == "interpolation": + return self._last_token_name != tokens.INTERP_START.lark_name() + # Space before '{' in blocks if token_type == tokens.LBRACE.lark_name() and parent_rule_name == BlockRule.lark_name(): return True @@ -294,6 +308,7 @@ def _reconstruct_token(self, token: Token, parent_rule_name: Optional[str] = Non if self._should_add_space_before(token, parent_rule_name): result = " " + result + self._token_before_last_name = self._last_token_name self._last_token_name = token.type if len(token) != 0: self._last_was_space = result[-1].endswith(" ") or result[-1].endswith("\n") diff --git a/hcl2/rules/strings.py b/hcl2/rules/strings.py index 76ce0660..ced35e6a 100644 --- a/hcl2/rules/strings.py +++ b/hcl2/rules/strings.py @@ -2,9 +2,12 @@ import re import sys -from typing import Any, List, Tuple, Union +from typing import Any, List, Optional, Set, Tuple, Union + +from lark.tree import Meta from hcl2.rules.abstract import LarkRule +from hcl2.rules.directives import _insert_strip_optionals from hcl2.rules.expressions import ExpressionRule from hcl2.rules.tokens import ( DBLQUOTE, @@ -15,8 +18,10 @@ INTERP_START, RBRACE, STRING_CHARS, + STRIP_MARKER, TEMPLATE_STRING, ) +from hcl2.template import INTERPOLATION, LITERAL, map_literal_spans, split_template from hcl2.utils import ( HEREDOC_PATTERN, HEREDOC_TRIM_PATTERN, @@ -26,24 +31,142 @@ to_dollar_string, ) +# A run of the whitespace OpenTofu measures a heredoc with: Go's +# `unicode.IsSpace`, less the newline. That is Python's `str.isspace` without +# U+001C..U+001F, information separators Python counts as whitespace and Go +# does not -- OpenTofu leaves `<<-EOT\n a\n\x1c b\n EOT` undedented, +# where `lstrip()` measured a margin and dropped the separator with it. +_INDENT = re.compile(r"[^\S\n\x1c-\x1f]*") + -def _strip_closing_marker_line(text: str) -> str: - r"""Drop the closing marker line's indentation and the one newline before it. +def _strip_closing_marker_indent(text: str) -> str: + r"""Drop the whitespace indenting the closing marker on its own line. A heredoc body always ends ``...\n``, where ```` is the - whitespace preceding the closing marker on its own line. The spec allows - "an arbitrary number of spaces preceding it", and neither that indentation - nor the newline separating it from the last content line is part of the - value. The newline may be ``\r\n``, since heredocs parse in CRLF files. - - Everything else is: additional blank lines, and trailing spaces on a - content line. The latter are safe because a content line always ends with - its own newline, so the indentation match never reaches them. This replaces - a blanket ``rstrip("\n\t ")``, which could not tell the two apart and - discarded both. + whitespace preceding the closing marker. The spec allows "an arbitrary + number of spaces preceding it", and that indentation is not part of the + value. + + The newline before it *is*. The spec ends the template where the delimiter + "subsequently appears again on a line of its own", so every content line, + the last one included, is terminated by its own newline: ``< str: + """Resolve a literal stretch of quoted source into the characters it means.""" + return process_escape_sequences(text.replace("$${", "${").replace("%%{", "%{")) + + +def _body_spans(body: str, dedent: bool) -> List[Tuple[str, str]]: + """Split a heredoc *body* into its spans once, dedenting a `<<-` body in them. + + The margin is the smallest indent any content line carries. + + The spec measures "any literal string at the start of each line", so a + blank line offers no measurement. Counting it as zero would drag the + margin down and cancel the dedent for every other line. A line that + offered no measurement is left exactly as written -- OpenTofu keeps a + six-space line inside a four-space heredoc at six spaces rather than two. + + It also says "spaces", but the reference implementation does not read + that as narrowly: OpenTofu dedents a tab-indented `<<-` heredoc by one tab + per level. Measuring whitespace characters rather than spaces alone + matches it, and is identical to counting spaces on the space-indented + input that reading the letter of the spec would cover. + + A line that begins inside a `${...}` span is expression source, not a + literal line start, so it neither sets the margin nor loses it. OpenTofu + evaluates `<<-EOT\n a ${\n "b"\n }\n c\n EOT` to `a b\nc\n`: + the shallow ` "b"` does not drag the margin to two. + """ + spans = list(split_template(body, heredoc=True)) + if not dedent: + return spans + + in_span: Set[int] = set() + offset = 0 + for kind, chunk in spans: + if kind == INTERPOLATION: + in_span.update(offset + index + 1 for index, char in enumerate(chunk) if char == "\n") + offset += len(chunk) + + starts = [] + margin = sys.maxsize + position = 0 + for line in body.split("\n"): + indent = _INDENT.match(line).end() # type: ignore[union-attr] + if position not in in_span and indent != len(line): + starts.append(position) + margin = min(margin, indent) + position += len(line) + 1 + if margin in (0, sys.maxsize): + return spans + + # The indent a line loses is whitespace at a literal line start, so it + # always falls inside a literal span: a span opens with `$` or `%`. + cut: Set[int] = set() + for start in starts: + cut.update(range(start, start + margin)) + dedented = [] + offset = 0 + for kind, chunk in spans: + kept = chunk + if kind == LITERAL: + kept = "".join(char for index, char in enumerate(chunk, offset) if index not in cut) + dedented.append((kind, kept)) + offset += len(chunk) + return dedented + + +def _has_quoted_spelling(spans: List[Tuple[str, str]]) -> bool: + """Whether a heredoc of these *spans* can be written as a quoted string. + + Not when a `${...}` or `%{...}` runs across a line. The newlines inside + the span are expression source, where OpenTofu rejects an escaped one -- + "This character is not used within the language" -- and a raw one makes + the quoted string span lines, which it rejects as well. + + Nor when a span carries a `~` strip marker. A heredoc body is lexed one + line at a time, so a marker strips no further than its own line; in a + quoted string the same whitespace runs on through the newline into the + next line's indent. OpenTofu evaluates + `< str: + r"""Escape literal text so it can sit inside a quoted string. + + A carriage return is escaped alongside the newline: raw, it would break the + quoted string it is being written into. OpenTofu rejects `"ab"` with + "No closing marker was found for the string". + """ + return text.replace("\\", "\\\\").replace('"', '\\"').replace("\r", "\\r").replace("\n", "\\n") class InterpolationRule(LarkRule): @@ -51,10 +174,20 @@ class InterpolationRule(LarkRule): _children_layout: Tuple[ INTERP_START, + Optional[STRIP_MARKER], ExpressionRule, + Optional[STRIP_MARKER], RBRACE, ] + def __init__(self, children, meta: Optional[Meta] = None): + # `${~ ...}` and `${... ~}` strip the whitespace beside the + # interpolation, exactly as they do beside a directive: OpenTofu + # evaluates `"a ${~ "b"} c"` to `ab c`. A missing marker is a None + # placeholder, so the expression keeps one index either way. + _insert_strip_optionals(children, [1, 3]) + super().__init__(children, meta) + @staticmethod def lark_name() -> str: """Return the grammar rule name.""" @@ -63,12 +196,25 @@ def lark_name() -> str: @property def expression(self): """Return the interpolated expression.""" - return self.children[1] + return self.children[2] + + @property + def strip_open(self) -> bool: + """Whether a strip marker follows `${`.""" + return self.children[1] is not None + + @property + def strip_close(self) -> bool: + """Whether a strip marker precedes the closing `}`.""" + return self.children[3] is not None def serialize(self, options=SerializationOptions(), context=SerializationContext()) -> Any: - """Serialize to ${expression} string.""" + """Serialize to `${expression}`, spelling any strip marker as a directive does.""" with context.modify(inside_dollar_string=True): - return to_dollar_string(self.expression.serialize(options, context)) + expression = self.expression.serialize(options, context) + prefix = "~ " if self.strip_open else "" + suffix = " ~" if self.strip_close else "" + return to_dollar_string(f"{prefix}{expression}{suffix}") class StringPartRule(LarkRule): @@ -131,16 +277,33 @@ def serialize(self, options=SerializationOptions(), context=SerializationContext @staticmethod def _serialize_part_as_value(part, options, context) -> str: - """Serialize one part, resolving escapes in literal text only. + """Serialize one part into what the reader sees, by its terminal. + + Literal text has its escapes resolved. An interpolation is passed + through untouched: its text is expression source, not literal content, + so an escape inside it is not this string's to resolve. - Interpolations and escaped interpolation/directive markers are passed - through untouched: their text is expression source, not literal - content, so an escape inside them is not this string's to resolve. + `$${` and `%%{` are neither. They are escapes for a literal `${` and + `%{`, so the value carries the single sigil -- `"$${esc}"` is the six + characters `${esc}` to Terraform, not seven. """ serialized = part.serialize(options, context) - if part.content.lark_name() == "STRING_CHARS": + terminal = part.content.lark_name() + if terminal == "STRING_CHARS": return process_escape_sequences(serialized) - return serialized + if terminal in ("ESCAPED_INTERPOLATION", "ESCAPED_DIRECTIVE"): + # The token runs from the doubled sigil to the next `}`, and all + # of it is literal text: OpenTofu evaluates `"$${a\tb}"` to + # `${ab}`, so the escapes after the sigil resolve too. + return process_escape_sequences(serialized[1:]) + # Anything else is a nested template rule -- a directive and everything + # it encloses arrive as one part, so a marker written between `%{ if }` + # and `%{ endif }` never reaches the branch above and stayed doubled, + # while the same content in a heredoc resolved. The literal stretches + # are resolved like any other literal text -- markers and escapes both: + # OpenTofu evaluates `"%{ if true }$${z}\t%{ endif }"` to `${z}`. + # The directives themselves are expression source and left alone. + return map_literal_spans(serialized, _resolve_literal) class HeredocTemplateRule(LarkRule): @@ -150,6 +313,8 @@ class HeredocTemplateRule(LarkRule): # \r is trimmed alongside \n so a CRLF heredoc does not leave a stray # carriage return hanging off the closing marker. _trim_chars = "\r\n\t " + _pattern = HEREDOC_PATTERN + _dedents = False @staticmethod def lark_name() -> str: @@ -162,84 +327,69 @@ def heredoc(self): return self.children[0] def serialize(self, options=SerializationOptions(), context=SerializationContext()) -> Any: - """Serialize the heredoc, optionally stripping to a plain string.""" + """Serialize the heredoc, optionally flattening it to a quoted string or its value.""" heredoc = self.heredoc.serialize(options, context) - if not options.preserve_heredocs: - match = HEREDOC_PATTERN.match(heredoc) - if not match: - raise RuntimeError(f"Invalid Heredoc token: {heredoc}") - heredoc = _strip_closing_marker_line(match.group(2)) + if options.preserve_heredocs: + result = heredoc.rstrip(self._trim_chars) if options.strip_string_quotes: - # The caller asked for the value, so hand back the body as-is: - # real newlines, no escaping. The escaping below exists only to - # build the quoted-string *source* form returned otherwise. - return heredoc - heredoc = heredoc.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n") - return f'"{heredoc}"' + return result + return f'"{result}"' + + match = self._pattern.match(heredoc) + if not match: + raise RuntimeError(f"Invalid Heredoc token: {heredoc}") + # One scan of the body serves every question below: the `<<-` margin, + # whether a quoted spelling exists, and which stretches to escape. + spans = _body_spans(_strip_closing_marker_indent(match.group(2)), self._dedents) - result = heredoc.rstrip(self._trim_chars) if options.strip_string_quotes: - return result - return f'"{result}"' + # The caller asked for the value: real newlines, no escaping. + # `$${` and `%%{` are resolved, being escapes for a literal + # `${` and `%{` rather than characters of the value. + return "".join( + chunk.replace("$${", "${").replace("%%{", "%{") if kind == LITERAL else chunk + for kind, chunk in spans + ) + + if not _has_quoted_spelling(spans): + if context.inside_dollar_string: + # An argument is expression source, so the heredoc goes back as + # written. Its token carries the newline after the closing + # marker, which the `)` that follows needs to start its line. + return heredoc + # Handed back as written -- the same form `preserve_heredocs=True` + # produces, which reads back as this heredoc. + return f'"{heredoc.rstrip(self._trim_chars)}"' + + # Only the literal spans are escaped. Inside `${...}` the text is + # expression source, and escaping a quote there rewrites someone + # else's code: `${upper("a")}` would become `${upper(\\"a\\")}`, + # which OpenTofu rejects outright. + return '"' + "".join(_escape_for_quoted_source(c) if k == LITERAL else c for k, c in spans) + '"' class HeredocTrimTemplateRule(HeredocTemplateRule): - """Rule for indented heredoc template strings (<<-MARKER).""" + """Rule for indented heredoc template strings (<<-MARKER). + + Its body is dedented by the smallest indent any of its lines carries; see + https://github.com/hashicorp/hcl2/blob/master/hcl/hclsyntax/spec.md#template-expressions + and `_body_spans`. + """ _children_layout: Tuple[HEREDOC_TRIM_TEMPLATE] + _pattern = HEREDOC_TRIM_PATTERN + _dedents = True @staticmethod def lark_name() -> str: """Return the grammar rule name.""" return "heredoc_template_trim" - def serialize(self, options=SerializationOptions(), context=SerializationContext()) -> Any: - """Serialize the trim heredoc, stripping common leading whitespace.""" - # See https://github.com/hashicorp/hcl2/blob/master/hcl/hclsyntax/spec.md#template-expressions - # This is a special version of heredocs that are declared with "<<-" - # This will calculate the minimum number of leading spaces in each line of a heredoc - # and then remove that number of spaces from each line - - heredoc = self.heredoc.serialize(options, context) - - if not options.preserve_heredocs: - match = HEREDOC_TRIM_PATTERN.match(heredoc) - if not match: - raise RuntimeError(f"Invalid Heredoc token: {heredoc}") - heredoc = match.group(2) - - heredoc = _strip_closing_marker_line(heredoc) - lines = heredoc.split("\n") - - # calculate the min number of leading spaces in each line - # The spec measures "any literal string at the start of each line", so a - # blank line offers no measurement. Counting it as zero would drag the - # minimum down and cancel the dedent for every other line -- which only - # became reachable once blank lines stopped being stripped above. - min_spaces = sys.maxsize - for line in lines: - if not line.strip(): - continue - leading_spaces = len(line) - len(line.lstrip(" ")) - min_spaces = min(min_spaces, leading_spaces) - if min_spaces == sys.maxsize: - min_spaces = 0 - - # trim off that number of leading spaces from each line - lines = [line[min_spaces:] for line in lines] - - if not options.preserve_heredocs: - lines = [line.replace("\\", "\\\\").replace('"', '\\"') for line in lines] - - if options.strip_string_quotes: - # Value, not source: join with real newlines regardless of - # preserve_heredocs, and skip the escaping done for the quoted form. - return "\n".join(lines) - - sep = "\\n" if not options.preserve_heredocs else "\n" - inner = sep.join(lines) - return '"' + inner + '"' + @staticmethod + def _dedent(body: str) -> List[str]: + """Split *body* into lines and remove the common leading whitespace.""" + return "".join(chunk for _, chunk in _body_spans(body, True)).split("\n") class TemplateStringRule(LarkRule): diff --git a/hcl2/template.py b/hcl2/template.py new file mode 100644 index 00000000..df902401 --- /dev/null +++ b/hcl2/template.py @@ -0,0 +1,204 @@ +"""Splitting template text into the parts that mean different things. + +A quoted string or a heredoc body is not one run of literal characters. HCL +reads `${...}` and `%{...}` as expression source, and `$${`/`%%{` as escapes +for a literal `${`/`%{`. Anything that rewrites such text -- resolving escapes, +adding them, or turning one form into another -- has to know which span it is +looking at, or it corrupts the expression inside. + +The grammar already separates these for a quoted string, which is why +`StringRule._serialize_part_as_value` can do the right thing by asking each +part for its terminal. A heredoc body arrives as one opaque token, so the same +distinction has to be recovered from the text. That is what this does. + +The scan is a single left-to-right pass, because the two questions cannot be +answered separately: a splitter run before escapes are resolved would see the +`${` inside `$${` and open an interpolation that is not there. +""" + +from typing import Callable, Iterator, Tuple + +LITERAL = "literal" +INTERPOLATION = "interpolation" + +_OPENERS = {"$": "${", "%": "%{"} +_ESCAPES = {"$": "$${", "%": "%%{"} + + +def _skip_string(text: str, index: int) -> int: + """Return the index just past the string literal opening at *index*. + + A string literal inside an expression is itself a template, so a `${` or + `%{` in it opens a nested expression whose own literals may hold quotes + and braces. OpenTofu evaluates `"a ${upper("v${ "{" }w")} b"` to + `a V{W b`, so taking the next quote as the terminator ended this literal + at the one that *opens* the innermost one, and the brace after it was + then counted as structural. + + The escapes stay escapes here: `"a ${upper("v$${x}w")} b"` is `a V${X}W b`, + so `$${` does not open anything. + """ + length = len(text) + index += 1 + while index < length: + char = text[index] + if char == "\\": + index += 2 + continue + if char in _OPENERS: + if text.startswith(_ESCAPES[char], index): + index += len(_ESCAPES[char]) + continue + if text.startswith(_OPENERS[char], index): + end = _scan_expression(text, index + 1) + if end == -1: + # Unbalanced, so there is no literal to close either. + return length + index = end + continue + if char == '"': + return index + 1 + index += 1 + return length + + +def _skip_comment(text: str, index: int) -> int: + """Return the index just past the comment opening at *index*, or *index*. + + HCL writes them three ways, and all three may hold a brace: `#` and `//` + run to the end of the line, `/* */` to its terminator. OpenTofu evaluates + `${1 /* } */ + 2}` to 3, so a scan that counts that brace closes the + expression in the middle of itself. + """ + if text.startswith("/*", index): + end = text.find("*/", index + 2) + return len(text) if end == -1 else end + 2 + if text.startswith("//", index) or text[index] == "#": + end = text.find("\n", index) + return len(text) if end == -1 else end + return index + + +def _scan_expression(text: str, start: int) -> int: + """Return the index just past the `}` closing the span opened at *start*. + + Braces nest, and three things inside an expression may carry one that is + not structural: a string literal, a comment, and the body of a heredoc + written inline. The first two are skipped here. A brace in a heredoc body + is not, which is a known gap rather than an oversight -- recognising one + means matching its delimiter, and the case has not been seen in the wild. + """ + depth = 0 + index = start + length = len(text) + while index < length: + char = text[index] + if char == "\\": + # A backslash pair is one unit. Without this the scan enters string + # mode at the quote of a `\\"`, then reads the real closing quote as + # another escape and runs to the end of the text, swallowing + # everything after the expression into one span. + index += 2 + continue + if char == '"': + index = _skip_string(text, index) + continue + if char == "#" or text.startswith("//", index) or text.startswith("/*", index): + skipped = _skip_comment(text, index) + if skipped != index: + index = skipped + continue + if char == "{": + depth += 1 + elif char == "}": + depth -= 1 + if depth == 0: + return index + 1 + index += 1 + # Unbalanced. Reported as such rather than swallowed: handing the rest + # back as an expression means nothing escapes it, and the heredoc paths + # then emit raw newlines and unescaped quotes into what is supposed to be + # quoted-string source. Treated as literal it is at least well-formed. + return -1 + + +def split_template(text: str, heredoc: bool = False) -> Iterator[Tuple[str, str]]: + r"""Yield `(kind, chunk)` pairs covering *text* exactly once. + + `kind` is `LITERAL` for text the reader treats as characters, including + the `$${` and `%%{` escapes themselves, and `INTERPOLATION` for a `${...}` + or `%{...}` span, whose content is expression source. + + *heredoc* says the text is a heredoc body rather than quoted-string + source. A heredoc interprets no escape, so a backslash there is a + character and the `${` after it is live: OpenTofu evaluates + `< str: + """Resolve `$${` and `%%{` into the single sigil they stand for. + + Only in literal spans: inside `${...}` the same characters are expression + source, where `$${` does not mean a literal `${`. + """ + return "".join( + chunk.replace("$${", "${").replace("%%{", "%{") if kind == LITERAL else chunk + for kind, chunk in split_template(text, heredoc) + ) + + +def map_literal_spans(text: str, transform: Callable[[str], str], heredoc: bool = False) -> str: + """Apply *transform* to the literal spans of *text*, leaving the rest alone. + + An escape belongs to the string that carries it, not to an expression + written inside it: escaping or unescaping through an interpolation rewrites + someone else's source and changes what it means. + """ + return "".join( + transform(chunk) if kind == LITERAL else chunk for kind, chunk in split_template(text, heredoc) + ) diff --git a/pyproject.toml b/pyproject.toml index 6541b5d7..23f9e9e5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -28,7 +28,6 @@ requires-python = ">=3.8.0" dependencies = [ "lark>=1.1.5,<2.0", - "regex>=2024.4.16" ] dynamic = ["version"] diff --git a/requirements.txt b/requirements.txt index 3b25f45c..e0c904a3 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,3 @@ # Place dependencies in this file, following the distutils format: # http://docs.python.org/2/distutils/setupscript.html#relationships-between-distributions-and-packages lark>=1.1.5,<2.0 -regex>=2024.4.16 diff --git a/test/integration/specialized/heredocs_flattened.json b/test/integration/specialized/heredocs_flattened.json index 95fb4e55..43c6fd2c 100644 --- a/test/integration/specialized/heredocs_flattened.json +++ b/test/integration/specialized/heredocs_flattened.json @@ -1,16 +1,16 @@ { "locals": [ { - "simple": "\"hello world\"", - "multiline": "\"line1\\nline2\\nline3\"", - "with_quotes": "\"say \\\"hello\\\"\"", - "with_backslashes": "\"path\\\\to\\\\file\"", - "trimmed": "\"indented1\\nindented2\"", - "trimmed_mixed": "\"line1\\n line2\\nline3\"", - "json_content": "\"{\\\"key\\\": \\\"value\\\"}\"", + "simple": "\"hello world\\n\"", + "multiline": "\"line1\\nline2\\nline3\\n\"", + "with_quotes": "\"say \\\"hello\\\"\\n\"", + "with_backslashes": "\"path\\\\to\\\\file\\n\"", + "trimmed": "\"indented1\\nindented2\\n\"", + "trimmed_mixed": "\"line1\\n line2\\nline3\\n\"", + "json_content": "\"{\\\"key\\\": \\\"value\\\"}\\n\"", "empty": "\"\"", "empty_trimmed": "\"\"", - "blank_line_only": "\"\"", + "blank_line_only": "\"\\n\"", "after_empty": "\"still parsed\"", "__is_block__": true } diff --git a/test/integration/specialized/heredocs_restored.tf b/test/integration/specialized/heredocs_restored.tf index 05832d52..a7bad307 100644 --- a/test/integration/specialized/heredocs_restored.tf +++ b/test/integration/specialized/heredocs_restored.tf @@ -1,12 +1,18 @@ locals { - simple = "hello world" + simple = <b"` with "No + closing marker was found for the string", while `"a\rb"` evaluates to a + carriage return, which is what the heredoc body actually held. + + The value form is unaffected: it hands back the body, so its newlines and + carriage returns stay real characters. + """ + + FLAT = SerializationOptions(preserve_heredocs=False) + VALUE = SerializationOptions(preserve_heredocs=False, strip_string_quotes=True) + + def test_heredoc_source_form_escapes_carriage_returns(self): + source = loads("a = < str: + flattened = loads(source, serialization_options=self.FLAT) + return dumps(flattened, deserializer_options=self.HEREDOCS) + + def _round_trip(self, source: str) -> str: + restored = self._restore(source) + return loads(restored, serialization_options=self.VALUE)["a"] + + def test_the_value_is_unchanged(self): + self.assertEqual(self._round_trip("a = < str: + return dumps({"x": value}, deserializer_options=self.HEREDOCS) + + def _round_trip(self, value: str) -> str: + return loads(self._write(value), serialization_options=self.VALUE)["x"] + + def test_an_ordinary_body_still_uses_eof(self): + self.assertEqual(self._write(r'"plain\n"'), "x = < str: + return dumps({"a": value}, deserializer_options=HEREDOCS) + + def test_a_tab_becomes_a_tab(self): + self.assertEqual(self._body(r'"a\tb\n"'), "a = < str: + return dumps({"x": value}, deserializer_options=HEREDOCS) + + def test_a_synthesised_sigil_keeps_the_value_quoted(self): + # The value is `\u0024\u007bfoo\u007d\n`, spelled with chr() so this + # test's own source cannot be confused with the characters it means. + esc = chr(92) + "u" + source = '"' + esc + "0024" + esc + "007bfoo" + esc + "007d" + chr(92) + "n" + '"' + self.assertEqual(self._written(source), "x = " + source + "\n") + + def test_a_demoted_interpolation_keeps_the_value_quoted(self): + source = '"' + chr(92) + "u0024" + "${b}" + chr(92) + "n" + '"' + self.assertEqual(self._written(source), "x = " + source + "\n") + + def test_a_real_interpolation_still_converts(self): + self.assertEqual(self._written(r'"${b}\n"'), "x = < str: + return loads(f'a = "{body}"\n', serialization_options=QUOTED_VALUE)["a"] + + def _heredoc(self, body: str) -> str: + return loads(f"a = < list: + return [kind for kind, _ in split_template(body)] + + def _spans(self, body: str) -> list: + return list(split_template(body)) + + def test_a_brace_in_a_nested_expressions_string(self): + # tofu: "a ${upper("v${ "{" }w")} b" -> a V{W b + body = 'a ${upper("v${ "{" }w")} b' + self.assertEqual(self._kinds(body), [LITERAL, INTERPOLATION, LITERAL]) + + def test_that_span_covers_the_whole_expression(self): + body = 'a ${upper("v${ "{" }w")} b' + self.assertEqual(self._spans(body)[1], (INTERPOLATION, '${upper("v${ "{" }w")}')) + + def test_three_levels_of_nesting(self): + # tofu: "a ${upper("p${ lower("Q${ "{" }R") }s")} b" -> a PQ{RS b + body = 'a ${upper("p${ lower("Q${ "{" }R") }s")} b' + self.assertEqual(self._kinds(body), [LITERAL, INTERPOLATION, LITERAL]) + + def test_an_escape_in_a_nested_string_opens_nothing(self): + # tofu: "a ${upper("v$${x}w")} b" -> a V${X}W b + body = 'a ${upper("v$${x}w")} b' + self.assertEqual(self._spans(body)[1], (INTERPOLATION, '${upper("v$${x}w")}')) + + def test_a_directive_nested_in_a_string(self): + # tofu: "a ${upper("v%{ if true }y%{ endif }w")} b" -> a VYW b + body = 'a ${upper("v%{ if true }y%{ endif }w")} b' + self.assertEqual(self._kinds(body), [LITERAL, INTERPOLATION, LITERAL]) + + def test_an_unterminated_nested_expression_is_still_literal(self): + # No closing brace anywhere, so nothing is an interpolation. + self.assertEqual(self._kinds('a ${upper("v${ x") b'), [LITERAL]) + + +class TestABackslashInAHeredocIsLiteral(TestCase): + r"""A heredoc body interprets no escape, so a backslash there is a character. + + The scan treated `\` as escaping whatever followed it, which is right for a + quoted string and wrong for a heredoc: OpenTofu v1.12.6 evaluates + `<b}` and + `"%{ if true }$${z}\t%{ endif }"` to `${z}`. The value form dropped + the extra sigil but left `\t` as two characters in the first, and in a + directive resolved the marker but not the tab beside it. + """ + + def test_inside_an_escaped_interpolation(self): + self.assertEqual(loads('x = "$${a\\tb}"\n', serialization_options=QUOTED_VALUE)["x"], "${a\tb}") + + def test_inside_an_escaped_directive(self): + self.assertEqual(loads('x = "%%{a\\tb}"\n', serialization_options=QUOTED_VALUE)["x"], "%{a\tb}") + + def test_beside_a_marker_inside_a_directive(self): + result = loads('x = "%{ if c }$${z}\\t%{ endif }"\n', serialization_options=QUOTED_VALUE)["x"] + self.assertEqual(result, "%{ if c }${z}\t%{ endif }") + + def test_the_directive_itself_is_left_alone(self): + # A literal written inside the directive is expression source. + result = loads('x = "%{ if c == "a\\tb" }y%{ endif }"\n', serialization_options=QUOTED_VALUE)["x"] + self.assertEqual(result, '%{ if c == "a\\tb" }y%{ endif }') + + +class TestWritingAStringSplitsItOnItsOwnSpans(TestCase): + r"""Writing a quoted string back splits it where the reader would. + + `dumps` split a string with a regex that knew nothing of string literals, + then stripped a `"` from both edges of every piece rather than from the + string once. Beside an interpolation that ate the backslash-quote pair's + quote: `"a \"${"b"}\" c"` came back as `"a \${"b"}\" c"`, which OpenTofu + rejects with "Invalid escape sequence" -- and a flattened JSON policy, the + commonest heredoc there is, did the same. A brace in a nested literal + crashed the split outright. OpenTofu v1.12.6 evaluates each source below + and its rewritten form to the same value. + """ + + def test_an_escaped_quote_beside_an_interpolation(self): + source = 'x = "a \\"${"b"}\\" c"\n' + self.assertEqual(dumps(loads(source)), source) + + def test_a_flattened_json_policy(self): + source = 'x = <