[girinataraj]

Description

Automatic links currently recognize only HTTP and FTP schemes. Other schemes
are emitted as raw markup, or incorrectly matched as email addresses when the
URL contains user information. For example:

<!-- Before -->
<p><ssh://example.org></p>
<p><a href="mailto:ssh://user@example.org/path">ssh://user@example.org/path</a></p>
<!-- After -->
<p><a href="ssh://example.org">ssh://example.org</a></p>
<p><a href="ssh://user@example.org/path">ssh://user@example.org/path</a></p>

The before example above decodes the HTML entities used to obfuscate email
addresses for readability.

Replace the hard-coded scheme alternatives with a letter-led scheme pattern,
allowing letters, digits, +, . and - after the first character. Keep the
existing :// delimiter, destination matching and link renderer. This follows
the URL expansion discussed in #1244; raw HTML handling and URI forms without
://, such as <urn:foo>, remain unchanged.

Add tests for SSH and Git URLs with user information, scheme variants,
query-string escaping, link boundaries and code spans. Update the syntax
documentation and add an unreleased changelog entry under Added.

Fixes #1244.

Validation on Windows, CPython 3.12.14:

• Focused autolink, link and raw HTML tests: 57 passed. The new user-information
  regression fails for both cases when run with the upstream pattern.
• Full suite: 1,144 tests run, 111 skipped, no failures or errors. The optional
  HTML Tidy and pinned Pygments environments were not exercised.
• Changed Python files pass Flake8. Repository-wide Flake8 7.2.0 reports four
  pre-existing E201/E202 errors in the untouched markdown/postprocessors.py:90.

Other Python versions and documentation/link/spelling builds were not run.

AI Assistance Disclosure

• No AI tools were used in preparing this PR.

• AI tools were used and I have disclosed which ones, and have fully reviewed and verified their output.

  OpenAI Codex assisted with issue research, reproduction, implementation,
  tests, documentation, changelog, commit message and this PR description.
  A Codex sub-agent reviewed the local patch; no automated AI review has
  been requested on this upstream PR.

  AI tools were used to assist with the following:

  • Code
  • Code Comments
  • Documentation
  • Changelog Entry
  • Commit Message
  • The Description Above
  • Other: regression tests and local patch review

Checklist

• This PR follows the contribution guidelines.
• The code follows the Code Style Guide.
• The commit message follows the Commit Message Style Guide.
• I have added or updated relevant docs, including release notes if applicable which follow the Documentation Style Guide.
• I have added or updated relevant tests.
• I have not requested, and will not request, an automated AI review for this PR.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

[waylan]

After reviewing the discussion in #1244 I can see you provided the fix I suggested. I am not yet convinced this is the best approach. I will have to think about it.

[girinataraj]

Understood. I’m happy to revise the implementation if you decide a different approach would fit the project better.

[girinataraj]

One more conservative option would be to preserve the existing HTTP/FTP matching and require a nonempty destination without whitespace for additional schemes. The current patch also recognizes <ssh://> and <ssh://example.org/a b> because it retains the existing destination pattern.

Would that address your concern, or are you reconsidering broader scheme recognition in the default parser?

[waylan]

The concern is that this will recognize anything as a scheme, including characters that are not a known scheme of any kind. The reasoning for the suggestion originally was to avoid needing to constantly add new hard-coded schemes to a whitelist as they come into existence. But now we have the opposite problem and will recognize non-schemes as valid schemes.

Suppose for a moment that a document author mistypes ftp as fto and doesn't notice. How should we handle that? With the current behavior, we don't recognize that as a valid scheme and it gets rendered in a way that will clue the author in that they made a mistake. With this fix, the author may never notice the mistake.

I'm not sure what the correct solution is for this.

[girinataraj]

That clarifies the concern. My earlier suggestion about whitespace would still recognize fto://example.org, so it wouldn’t address this.

Distinguishing known schemes from syntactically valid but unintended names seems to require an explicit list, whether maintained by the project or supplied by the user.

Would an opt-in extension with configurable schemes be a better fit, keeping the current default behavior? That would allow additional schemes without making arbitrary names automatic links.

[waylan]

I just need to make a decision. Any input @facelessuser or @mitya57? Do we whitelist specific schemes or let anything through that matches a pattern?

[mitya57]

I would not whitelist specific schemes, as that list will never be complete and people will keep requesting additions.

RFC 3986 also does not list schemas and provides a grammar instead. This PR seems to match that grammar exactly, so I think it's fine to merge.

[facelessuser]

I'm not sure I have strong opinions. More modern parsers seem to accept whatever: https://babelmark.github.io/?text=%3Cftodkssk%3A%2F%2Fexample.org%3E. Those that also accept bare URLs are far more strict when the angle brackets are removed, which isn't something Python Markdown supports.

I guess I'm fine with accepting whatever protocol, but I would not complain if we whitelisted specific ones. I know that isn't exactly helpful, but it is my honest take.

[facelessuser]

@mitya57 point about maintaining a list that may never be complete maybe pushes me over onto that side.

[waylan]

@mitya57 point about maintaining a list that may never be complete maybe pushes over onto that side.

It does me as well.