colon fence syntax is broken by escape character

[arwedus]

Describe the problem

In case of a colon fence:

:::{NOTE}

README.md should contain the component or package name in it's title.
Using only README as title will lead to lots of "README" items in the outline of the generated documentation.
:::

mdformat produces:

:::\{NOTE}

README.md should contain the component or package name in it's title.
Using only README as title will lead to lots of "README" items in the outline of the generated documentation.
:::

Ideally, mdformat with mdformat_myst would respect this syntax and not escape it.

To me, it is not even clear why mdformat escapes curly brackets in the first place.

Link to your repository or website

No response

Steps to reproduce

1. "pip install mdformat_myst"
2. Put the above example in a .md file
3. run "mdformat filename.md"

The version of Python you're using

3.8

Your operating system

Ubuntu 20.04

Versions of your packages

mdformat==0.7.10
mdformat_frontmatter==0.4.1
mdformat_myst==0.1.4
mdformat_tables==0.4.1

Additional context

No response

[welcome]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.

If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).

Welcome to the EBP community! 🎉

[hukkin]

The plugin only supports syntax extensions that are on by default in MyST. Colon fence is not a supported syntax currently.

[adamgayoso]

Will this be supported in the future?

[verm]

The formater seems to have gotten more aggressive in its escaping. The default suggestion for https://github.com/executablebooks/rst-to-myst is to enable colon fences. Then using mdformat it proceeds to escape everything.

We're already using mdformat for our other documents but now converting our Sphinx+reST documents over the formatter will no longer work. Are there any plans to add support?

Thank you!

[paugier]

The plugin only supports syntax extensions that are on by default in MyST. Colon fence is not a supported syntax currently.

Colon fence are allowed by default in https://mystmd.org so I guess mdformat-myst has to support them and this issue has to be considered as a bug. @hukkin Do you think it makes sense?

The documentation of https://mystmd.org contains colon fence (see for example https://mystmd.org/guide/syntax-overview) and they are particularly useful for nested directives.

[hukkin]

Seems to me like syntax extensions, including colon fence, still remain disabled by default in the Python MyST parser. I realize there is a TypeScript parser now, owned by a different GitHub organization, and that parser may have different defaults. There also seems to be a new implementation agnostic MyST specification that makes no mention of the colon fence syntax extension whatsoever.

Requested by a member of the executablebooks organization, I have transferred this plugin to them and no longer maintain it, and also am not fully aware of the MyST project's state today (or future), so I won't take a stance on what makes sense for this plugin.

[nthiery]

@rowanc1, @choldgraf (e.g.) would you have an opinion on whether colon fence syntax should be supported by mdformat-myst?

[kdeldycke]

I hit a related case with backtick-fence directives (rather than colon fences) inside fenced code blocks. When documenting MyST directive syntax in a markdown code block example, mdformat restructures the fences and silently drops content.

Reproduction

Input:

Follow this template:

```markdown
## Excluded projects

```{note}
<Project name> is not included because <reason>.
```

## Project URLs

[^1]: [<url>](<url>)
```

Some text after.

After running mdformat 1.0.0:

Follow this template:

````markdown
## Excluded projects

```{note}
<Project name> is not included because <reason>.
````

## Project URLs

```

Some text after.
```

What went wrong

1. The outer fence is correctly upgraded from 3 to 4 backticks (to avoid ambiguity with the ```{note} content line). Good.
2. The original closing fence (the bare ``` after <reason>) is also upgraded to 4 backticks. Since it now matches the outer fence length, it closes the outer block too early. Everything after is now live markdown.
3. The [^1]: [<url>](<url>) line, now exposed as live markdown, is parsed as an unused link reference definition and silently dropped.
4. Some text after. gets pulled into a new code block.

The original markdown is valid CommonMark: the ```{note} line has trailing text after the backticks, so it is not a closing fence. Only the bare ``` closes the outer block. After reformatting, this structure is broken.

Workaround

Using tilde fences for the inner directive avoids the conflict:

````markdown
~~~{note}
<Project name> is not included because <reason>.
~~~
````

Not mdformat-myst specific

I tested with bare mdformat 1.0.0 (no plugins) and the behavior is identical, so this may warrant a separate issue on hukkin/mdformat. Posting here because the trigger is documenting MyST {directive} syntax, and this case could be useful for the test suite.

Environment

• mdformat 1.0.0
• mdformat_myst 0.3.0
• Python 3.13
• macOS