Content mappers

[Andrew Branch (andrewbranch)]

Implements #2824 (comment)

Overview

Content mappers are external integrations that allow TypeScript to include otherwise unsupported file types in a program. They transform a foreign file’s original text into valid TypeScript syntax and provide mappings between the original and transformed content.

Users specify a set of file extensions to be handled by a content mapper package in a tsconfig.json file:

{
  "compilerOptions": {
    // ...
  },
  "contentMappers": [
    {
      "package": "vue-content-mapper",
      "extensions": [".vue"],
      "options": {
        "strictTemplates": true
      }
    }
  ],
  "include": ["src"] // implicitly includes .vue as well as .ts
}

When contentMappers are specified, tsc must be run with --runExternalCode. VS Code passes --runExternalCode to tsc --lsp only in trusted workspaces; otherwise, contentMappers are ignored in the LSP server.

The package field will be resolved as a Node.js module name. The optional options field must be an object and is passed through to the mapper.

The package.json of the content mapper package must specify a typescript top-level field with a nested contentMapper field describing how to spawn the mapper process and what compiler options its transform requires. A mapper that reads additional project-specific configuration from external sources beyond those compiler options can additionally declare dynamicConfig: true:

{
  "name": "vue-content-mapper",
  "version": "1.0.0",
  "typescript": {
    "contentMapper": {
      "exec": ["node", "dist/server.js"],
      "compilerOptions": ["module", "jsx", "jsxImportSource"],
      "dynamicConfig": true
    }
  }
}

Note that the content mapper process need not be run with Node.js or implemented in JavaScript; package resolution serves as a convenient way to associate a content mapper with a versioned identity that can be managed alongside other dependencies in a project, but the exec field can specify any command.

VS Code extensions can also register content mapper integrations with the TypeScript extension. A registration always supplies the extensions that should trigger configured-project discovery, and may additionally provide an inline manifest and options for using the mapper in inferred projects. Configured projects continue to use only the contentMappers declared in their config files; extension-provided inferred-project mappers never modify configured project behavior.

Protocol

When constructing a program for a config that specifies contentMappers, module resolution recognizes file lookups for the specified extensions and requests transformed content from mapper processes over STDIO. Content mappers communicate with TypeScript over JSON-RPC. TypeScript sends all requests; mappers do not send requests or notifications. All mappers handle initialize and transform. Mappers declaring dynamicConfig: true additionally handle openProject and closeProject.

type PositionEncoding = "utf-8" | "utf-16";

interface InitializeParams {
    protocolVersion: 1;
    /** The position encodings supported by TypeScript. The mapper must choose one of these encodings. */
    positionEncodings: PositionEncoding[];
    /** BCP 47 locale requested for diagnostics. */
    locale?: string;
}

interface InitializeResult {
    /** Must match the protocolVersion sent in InitializeParams. */
    protocolVersion: 1;
    /** The position encoding the mapper will use for all span mapping positions and diagnostic positions. */
    positionEncoding: PositionEncoding;
    /**
     * The source identifier displayed for mapper-produced diagnostics.
     * Must not be "ts", "tsc", "typescript", or any file extension TypeScript understands.
    */
    diagnosticSource: string;
}

/** This request is sent only to mappers that declare `dynamicConfig: true`. */
interface OpenProjectParams {
    /** Absolute tsconfig path, or an empty string for a project without a config file. */
    configFileName: string;
    /** Opaque process-local handle assigned by TypeScript. */
    projectHandle: string;
    /** Object from the contentMappers entry, when specified. */
    options?: Record<string, unknown>;
    /** The project's effective compiler options. */
    compilerOptions: CompilerOptions;
}

/** This response is required only from mappers that declare `dynamicConfig: true`. */
interface OpenProjectResult {
    /**
     * Stable fingerprint of all dynamically discovered configuration that can affect transforms.
     */
    configIdentity: string;
    /**
     * Absolute file names whose changes may alter configIdentity or transform output.
     * May only be returned when the package declares `dynamicConfig: true`. Do not include
     * the files being transformed; those are watched separately.
     */
    watchedFiles?: string[];
}

interface TransformParams {
    fileName: string;
    /** Original content of the file to be transformed. */
    content: string;
    /** Object from the contentMappers entry, when specified. */
    options?: Record<string, unknown>;
    /** Project handle supplied in openProject. Absent for mappers without `dynamicConfig: true`. */
    projectHandle?: string;
    /** The subset of compiler options that the mapper requested in its package.json. */
    compilerOptions: CompilerOptions;
}

interface MappedOutput {
    /** Valid JS, JSX, TS, TSX, or JSON text that TypeScript can parse. */
    text: string;
    /** The virtual file extension that determines how TypeScript parses this output. */
    extension: ".js" | ".jsx" | ".mjs" | ".cjs" | ".ts" | ".tsx" | ".mts" | ".cts" | ".json";
    /** Mappings between the original and transformed content. */
    mappings?: SpanMapping[];
    /** Framework-specific directives that suppress TypeScript diagnostics in virtual ranges. */
    diagnosticDirectives?: DiagnosticDirectives;
}

enum DiagnosticDirectivePolicy {
    Ignore = 0,
    Expect = 1,
}

interface UnusedExpectDirectiveDiagnostic {
    /** Diagnostic code reported when an `Expect` directive suppresses no diagnostics. */
    code: number;
    /** Diagnostic text reported when an `Expect` directive suppresses no diagnostics. */
    messageText: string;
}

interface DiagnosticDirectives {
    /** Shared diagnostics reported for unused `Expect` directives. */
    unusedExpectDirectiveDiagnostics: UnusedExpectDirectiveDiagnostic[];
    directives: MappedDiagnosticDirective[];
}

/** Positions and lengths are in the specified `positionEncoding`. */
type MappedDiagnosticDirective = [
    /** Location of the framework directive in the original source. */
    originalStart: number,
    originalLength: number,
    /** Region of virtual code affected by the directive. */
    virtualStart: number,
    virtualEnd: number,
    policy: DiagnosticDirectivePolicy,
    /**
     * Index into `unusedExpectDirectiveDiagnostics`. Required for `Expect` directives
     * when the array contains more than one entry.
     */
    unusedExpectDirectiveIndex?: number,
];

interface TransformResult extends MappedOutput {
    /** Parse errors in the original content. */
    diagnostics?: MapperDiagnostic[];
    /** Additional virtual files associated with this input. */
    supplemental?: MappedOutput[];
}

/** This request is sent only to mappers that declare `dynamicConfig: true`. */
interface CloseProjectParams {
    /** Project handle supplied in openProject. */
    projectHandle: string;
}

/** Positions and lengths are in the specified `positionEncoding`. */
type SpanMapping = [
    virtualStart: number,
    virtualLength: number,
    originalStart: number,
    originalLength: number,
    kind: SpanMapKind,
    features?: SpanMapFeature,
];

enum SpanMapKind {
    /** Verbatim spans in virtual text have the same length and content as their counterparts in original text. */
    Verbatim = 0,
    /** Atom spans in virtual text may have different length and content than their counterparts in the original text. */
    Atom = 1,
    /** Alias spans in virtual text may have different length and content than their counterparts in the original text, but diagnostics display their original text. */
    Alias = 2,
}

/** Controls which TypeScript language service features may use a span. */
enum SpanMapFeature {
    None = 0,
    Hover = 1 << 0,
    SignatureHelp = 1 << 1,
    Completion = 1 << 2,
    Definition = 1 << 3,
    TypeDefinition = 1 << 4,
    Implementation = 1 << 5,
    References = 1 << 6,
    DocumentHighlights = 1 << 7,
    Rename = 1 << 8,
    CallHierarchy = 1 << 9,
    CodeActions = 1 << 10,
    Formatting = 1 << 11,
    InlayHints = 1 << 12,
    SemanticTokens = 1 << 13,
    FoldingRanges = 1 << 14,
    SelectionRanges = 1 << 15,
    LinkedEditing = 1 << 16,
    AutoInsert = 1 << 17,
    DocumentSymbols = 1 << 18,
    CodeLens = 1 << 19,
    /** Enables every language service feature. This is the default when `features` is omitted. */
    All = (CodeLens << 1) - 1,
}

/** Start and length are in the specified `positionEncoding`. */
interface MapperDiagnostic {
    messageText: string;
    start: number;
    length: number;
    code?: number;
}

A mapper may return supplemental outputs when a file contributes more than one TypeScript or JavaScript file, such as an Astro component containing multiple script blocks. TypeScript automatically includes these outputs in the same program as the canonical output, so they participate in binding and type checking without needing to be imported. Supplemental outputs receive compiler-assigned virtual file names based on their order and extension, but those names are not module resolution targets and cannot be imported directly. Imports written inside supplemental outputs resolve relative to the directory containing the original file.

Span maps

For a content mapper to be useful, it needs to provide a mapping between the transformed output and the original content. In the CLI, these mappings are used to show TypeScript-generated diagnostics in the original, non-TypeScript content. Take a simple example:

// original content:
(+ 1 2 "oops")

// transformed content:
add(1, 2, "oops");

// span mapping:
add(1, 2, "oops");
^^^                 [0, 3)   → [1, 2) + atom
    ^               [4, 5)   → [3, 4) 1 verbatim
       ^            [7, 8)   → [5, 6) 2 verbatim
          ^^^^^^    [10, 16) → [7, 13) "oops" verbatim

TypeScript sees and checks the transformed content, in this case producing a diagnostic for the string literal "oops" because it is not a number. The span mapping allows TypeScript to report the diagnostic in the original content, at the correct location of the string literal ([7, 13), instead of [10, 16)).

In this example, add mapped to + with SpanMapKind.Atom, indicating a correspondence between the two spans, but with different lengths and content. If the name add failed to resolve, the displayed diagnostic range would cover +, but the message would still reference the identifier add:

add.lisp:1:2 - error TS2304: Cannot find name 'add'.

1 (+ 1 2 "oops")
   ~

The mapper can use SpanMapKind.Alias instead of SpanMapKind.Atom to indicate that the virtual and original text name the same entity. When the diagnostic is rendered, the original text of the alias span (+) will be substituted for the virtual text (add) in the diagnostic message:

add.lisp:1:2 - error TS2304: Cannot find name '+'.
1 (+ 1 2 "oops")
   ~

Gaps in the span map are treated as fully synthesized content and cannot be mapped to a location in the original text. Unlike in Volar, diagnostics in unmappable regions are not discarded. In the CLI, they cause a short snippet of the transformed content to be shown with the diagnostic. A common case may be a content mapper that synthesizes an import statement at the top of the file used in scaffolding. If that import fails to resolve, the user will see:

app.vue:1:26 - error TS2307: Cannot find module '@vue/content-mapper-utils' or its corresponding type declarations.
  This location is in code generated by the content mapper '@vue/content-mapper@1.0.0' and has no corresponding location in the original file.

1 import { scaffolding } from "@vue/content-mapper-utils";
                              ~~~~~~~~~~~~~~~~~~~~~~~~~~~

Spans in the virtual text must not overlap, but multiple may map to the same span in the original content. In other words, one range in the original content can map to multiple ranges in the transformed content. This can be useful in the language server when combined with SpanMapFeature and SpanMapKind. Broadly speaking, when a language server request is received for a position in a content-mapped file, the handler maps it to every projection whose feature mask includes the requested operation, performs analysis on the transformed content, and maps visible results back through spans that participate in the same feature. This lets a mapper independently select, for example, one projection for hover and another for definitions or references.

The language server currently supports the following features for content-mapped files:

• Diagnostics - always mapped to original content where possible; diagnostics in synthesized regions are collected and reported at the top of the file. Declared-but-not-used diagnostics are automatically suppressed in synthesized regions. Diagnostics are intentionally not represented by a span-map feature flag. Framework-specific semantic diagnostic suppression is instead expressed explicitly through diagnosticDirectives.
• Position-based features - hover, signature help, completions, definitions, type definitions, implementations, source definitions, references, document highlights, rename, call hierarchy, code actions, formatting, linked editing, and auto-insert map incoming positions or ranges through spans participating in their corresponding SpanMapFeature flag.
• Document-wide features - inlay hints, semantic tokens, folding ranges, selection ranges, document symbols, and CodeLens map visible results back only through spans participating in their corresponding flag.
• Text edits - feature participation does not make a mapping edit-safe. Rename, code action, completion, and formatting edits may be written back only through exact, length-preserving SpanMapKind.Verbatim mappings.

Language service requests and visible results can be disabled independently for any span by clearing the corresponding bits, or disabled for all features with SpanMapFeature.None. If features is omitted from the span mapping tuple, it defaults to SpanMapFeature.All, enabling every supported language service feature for that span.

Note

Unlike with Volar, feature participation must be statically determined by the content mapper during transformation. This level of LSP feature mapping is not intended to replace fully custom language servers. TypeScript’s goal in providing language service support for content-mapped files is to support a good editing experience inside <script> blocks or similar verbatim ranges that embed normal TypeScript or JavaScript code without a third-party language server needing to proxy every request unchanged. We expect that ecosystems implementing complex transforms may still want to implement their own language servers alongside TypeScript’s, and either augment or replace TypeScript’s implementation of these language service features. Content mappers provide a baseline editing experience, but they also provide the API foundation for more specialized language servers to build on. Vue tooling, for example, may choose to enable TypeScript features only for selected projections while a separate language server handles the rest, accessing the AST, type, and symbol information of transformed content through an API connection to TypeScript’s language server.

Diagnostic directives

Virtual text can include // @ts-ignore or // @ts-expect-error directives to suppress a TypeScript diagnostic on the next line. However, Vue supports its own diagnostic directives, which have different scope than TypeScript’s:

// Original:
<!-- @vue-expect-error -->
<div
  :id="firstError"
  :title="secondError"
></div>

// Virtual:
__VLS_asFunctionalElement1(
    __VLS_intrinsics.div,
    __VLS_intrinsics.div,
)({
    id: (__VLS_ctx.firstError),
    title: (__VLS_ctx.secondError),
});

The expect-error directive in Vue applies to errors on every line of the <div> element in the original text, which are broken up into multiple lines in the virtual text too. This behavior can’t be replicated with any number of // @ts-expect-error directives. Instead, the content mapper can return diagnosticDirectives in the transform result, which specify a virtual range and a policy of either DiagnosticDirectivePolicy.Expect or DiagnosticDirectivePolicy.Ignore. TypeScript will suppress bind/check diagnostics in that virtual range according to the policy, and report unused Expect directives as diagnostics in the original content. Unused-directive diagnostics are stored once in unusedExpectDirectiveDiagnostics and referenced by index from directive tuples.

Note

In the same way that it’s technically possible for a mapper to put a // @ts-ignore comment between every line of its output, it’s also possible for a mapper to synthesize ignore regions without a corresponding directive in the original content, but this is not recommended. It’s not currently possible to filter diagnostics by code. This is a break from Volar; I want to see if mappers can get around this with different code generation strategies rather than relying on filtering before considering a broad diagnostic filtering feature. As a type checker implementer, we really prefer to report all the type errors we see.

Failure handling

Mappers return diagnostics for unparseable content, and errors in the transformed text itself are handled by TypeScript like any other file. If the mapper fails in an unexpected way (e.g., crashes or doesn’t conform to the protocol), TypeScript reports a localized diagnostic and treats the file as an empty TypeScript file. After five failures in a single project, TypeScript stops attempting to transform files with that content mapper and issues a final diagnostic reporting the failure.

LSP activation

TypeScript’s language server can only know to care about the file extensions registered in contentMappers once the server is running and has discovered a tsconfig.json that specifies them. In the case where a user opens a directory in VS Code and opens a single .vue file, the TypeScript VS Code extension hasn’t even activated, much less spawned a server that knows about a contentMappers registration. To address this, third-party VS Code extensions need to explicitly activate the TypeScript extension and register their content mapper contributions:

const extension = vscode.extensions.getExtension("TypeScriptTeam.native-preview");
const api = await extension?.activate();

const registration = api?.registerContentMappers(
    "publisher.vue-language-features",
    [{ extensions: [".vue"] }],
);

Registering extensions causes TypeScript to inspect matching documents that are already open and discover any configured projects that provide a mapper for them. The returned disposable removes the contribution.

An extension may also provide a mapper for files that do not belong to a configured project by including an inline inferred-project contribution:

const registration = api?.registerContentMappers(
    "publisher.vue-language-features",
    [{
        extensions: [".vue"],
        inferredProjectContribution: {
            options: { strictTemplates: true }, // corresponds to tsconfig.json contentMappers options
            manifest: {                         // corresponds to a content mapper's package.json
                name: "vue-content-mapper",
                version: "1.0.0",
                exec: [process.execPath, mapperEntryPoint],
                cwd: extension.extensionUri,
                compilerOptions: ["module", "jsx", "jsxImportSource"],
                dynamicConfig: true,
            },
        },
    }],
);

It's recommended that extensions always provide an inferredProjectContribution, and to supply a manifest built from resolving the workspace-installed content mapper package, falling back to a bundled version if the package is not installed. But ultimately, the extension is responsible for content mapper resolution and version/fallback policy.

Emit

Content-mapped files are not emitted to JavaScript. When --declaration is enabled, however, declaration files are emitted from the transformed content. The declaration file name for App.svelte is App.d.svelte.ts. Declaration files for supplemental outputs of a file named App.svelte are emitted as App.svelte.0.d.ts, App.svelte.1.d.ts, etc., and are automatically referenced by App.d.svelte.ts. Declaration maps are currently not supported.

Incremental, build, watch, and process consolidation

Content mappers are supported in --incremental, --build, and --watch modes. Each project records sorted mapper transform identities in .tsbuildinfo and compares them during up-to-date checks. Changing an identity forces files handled by that mapper to be transformed again.

For a mapper without dynamicConfig: true, the transform identity is computed without starting its process. It includes the resolved package name and version, the tsconfig entry’s options, and the values of compiler options named by typescript.contentMapper.compilerOptions. Consequently, incremental and solution-build status checks do not spawn processes for mappers with static configuration.

For a mapper declaring dynamicConfig: true, TypeScript sends openProject to obtain configIdentity before an up-to-date decision. The mapper is responsible for changing configIdentity whenever dynamically discovered configuration that can affect transforms changes.

TypeScript watches the absolute paths returned in watchedFiles. A change invalidates only projects that reported that path, closes their current mapper project configuration, obtains a fresh identity and watch set, and performs a normal project rebuild. Other projects using the same mapper package continue using their existing project handles and the shared process.

Note

Modifying a static-config content mapper implementation during local development will not change its identity, so you’ll need to bump the local package.json version, or use --force or --clean to clear cached outputs if testing with --incremental or --build.

In --build mode with project references, and in some instances in the language server, it’s possible to have a project graph with many projects all defining the same content mapper. To avoid excessive spawning of child processes, TypeScript deduplicates content mapper processes by resolved package name and version. For a mapper with dynamicConfig: true, one process may have many open project handles. Dynamic-config mappers must isolate project-specific state by projectHandle, accept requests for different projects in any order, and release that state on closeProject. Static-config mappers receive transforms without a project handle. Processes remain alive while any project using that package is retained.

API integration

Content-mapped SourceFiles can be inspected by the JavaScript API. For a content-mapped SourceFile, file.text is the transformed text, file.originalText is the original text, and file.spanMap exposes an API for mapping between the two. Regardless of the positionEncoding used by the content mapper, accessing the span map through the JavaScript API always yields UTF-16 positions.

const mapped = file.spanMap.virtualToOriginalPosition(10);
// { position, fidelity }
// See _packages/native-preview/src/ast/spanMap.ts for details.

If the content mapper provided supplemental outputs for a file, the file names are set on file.supplementalOutputs and can be retrieved with program.getSourceFile(fileName).

Debugging

On the CLI, when the TS_CONTENT_MAPPER_DEBUG environment variable is set, JSON-RPC communication is logged and the mapper process’s STDERR is captured and redirected to tsc’s STDERR.

In the LSP, when the LSP log level is set to Trace, JSON-RPC communication and mapper STDERR flow to the LSP client (you can see the content mapper debug logs in the “TypeScript 7” output channel in VS Code, for example). At lower log levels, mapper STDERR is discarded.

I also have a prototype of a VS Code extension that lets you see the virtual content and span mappings, which I’ll share after this PR is merged.

Later follow-up

• Investigate if declaration maps can work by double-mapping back to original text
• Provide a JavaScript library for implementing the content mapper protocol

[Andrew Branch (andrewbranch)]

I cloned vuejs/language-tools and had Copilot take its best shot at making a real content mapper from @vue/language-core. The repo contains a corpus of 222 Vue project fixtures. 210 produce the same output with the content mapper and with vue-tsc. The scaffolding generated by @vue/language-core does not type check clean and sometimes doesn’t even parse without syntax errors (e.g. emitting TypeScript syntax while claiming the virtual extension is .js), and Volar happily swallows these. I did the thing I advised not doing to get most of the superfluous errors to disappear:

In the same way that it’s technically possible for a mapper to put a // @ts-ignore comment between every line of its output, it’s also possible for a mapper to synthesize ignore regions without a corresponding directive in the original content, but this is not recommended.

This intentionally doesn’t silence parse errors, so I think most of the remaining errors are things that should at least be investigated as a fix in the language core.

The result is 2.4x faster than vue-tsc with TypeScript 6 and 1.4x faster than vue-tsc with typescript-native-bridge, while using 24% less peak memory than typescript-native-bridge. (On some earlier more contrived test corpuses (corpi?), I saw a pretty decent performance boost by using Node.js worker threads to parallelize transform requests (since tsc is trying to parse in parallel, it pays to be able to serve multiple requests at once), but that didn’t make any difference in this benchmark, but that architecture is preserved in the implementation.)

I did this just to help myself prove out the TypeScript implementation; I looked at the AI code only enough to know the results weren’t completely fabricated. I don’t plan to PR it but the Vue team is welcome to reference it or pull from it insofar as it’s useful. I’m pushing my branch only to back up my claims that it’s mostly working and it’s pretty fast: https://github.com/vuejs/language-tools/compare/master...andrewbranch:language-tools:feat/vue-content-mapper?expand=1

[Remco Haszing (remcohaszing)]

One more thing I forgot to mention earlier, is that it would be nice to have the ability to have some logging, both in the editor and in the CLI.

I imagine a tsc --verbose flag could log all JSON RPC communication. And a TypeScript LSP setting could add logging in the language server, similar to ${id}.trace.server.* options in vscode-languageclient.

But also a server → client log notification type would be nice, since content mappers can’t log to stdout. In my testing I worked around this by logging to a file, which works, but is inconvenient.

[Andrew Branch (andrewbranch)]

Bikeshed request: I don’t love the name "tsContentMapper" for the mapper package.json key. Any better ideas?

[Remco Haszing (remcohaszing)]

How about:

{
  "typescript": {
    "contentMapper": {
      // …
    }
  }
}

This provides a namespace that can be reused for other purposes in future TypeScript versions.

[Andrew Branch (andrewbranch)]

• TS_CONTENT_MAPPER_DEBUG environment variable logs JSON-RPC communication as well as the mapper’s STDERR; setting LSP log level to Trace has same effect
• --loadExternalPlugins renamed to --runExternalCode
• tsContentMapper renamed to a nested object of typescript.contentMapper

[Andrew Branch (andrewbranch)]

I’m making one more change to the shape of diagnosticDirectives (logging with my Vue prototype made it apparent how un-compact it is compared to the rest of the protocol), and then I think that will be all.

[Remco Haszing (remcohaszing)]

I ported most of the MDX Volar mapper to a content mapper, and it works well. What remains mostly is a lot of tests.

One specific mapping broke. MDX supports ESM syntax, so auto-imports are useful. However, the file might not have imports yet. As a workaround, I added a strategically placed empty mapping next to a generated import in the virtual content. The TypeScript content mapper breaks on this empty mapping. It would be useful if content mappers can provide explicit locations where auto-imports may be inserted.

[Andrew Branch (andrewbranch)]

I was thinking there might be a fairly easy change for the code that decides where it wants to insert imports to skip over incompatible spans when it's working in a virtual text file. I'll look into that.

[Andrew Branch (andrewbranch)]

With the more compact diagnosticDirectives format, my Vue content mapper is now about 1.6x faster than vue-tsc with typescript-native-bridge, using 27% less peak memory. (Interestingly, this time, using four workers did improve things a little. I'm running multiple trials to get these numbers but still on a noisy dev machine with other incidental things going on in the CPU. The error margin is probably pretty large across different days here.)

Sorry, this run was polluted by memory sampling. It’s actually 2x faster than typescript-native-bridge currently.

[Andrew Branch (andrewbranch)]

vuejs/language-tools@76d07d9 shows how extensions can contribute a JSON schema for their content mapper options that will get merged into the tsconfig schema.