Line breakpoints never verify in UTF-16 encoded source files #1726

Line breakpoints in Java source files encoded as UTF-16 LE with BOM stay unverified forever, even after the class is loaded and the line contains code. The same files debug fine in Eclipse. Our sources are UTF-16 because the ERP development environment we work with generates them that way, so we cannot switch to UTF-8.

Environment

• Operating System: Windows Server 2022
• JDK version: JRE 25 embedded in the Java extension (language server); debuggee runs on JDK 11
• Visual Studio Code version: 1.144.0
• Java extension version: 1.58.0
• Java Debugger extension version: 0.60.2026102705 (pre-release, bundles com.microsoft.java.debug.plugin 0.53.2)

Steps To Reproduce

1. Create demo/A.java as UTF-16 LE with BOM, for example in PowerShell:

   New-Item -ItemType Directory -Force demo | Out-Null
   [IO.File]::WriteAllText("$PWD\demo\A.java", "package demo;`r`n`r`npublic class A {`r`n    public static void main(String[] a) {`r`n        int x = 1;`r`n        System.out.println(x);`r`n    }`r`n}`r`n", [Text.UnicodeEncoding]::new($false, $true))

   Content of the file:

   package demo;
   
   public class A {
       public static void main(String[] a) {
           int x = 1;
           System.out.println(x);
       }
   }

2. Open the folder in VS Code and set a line breakpoint on int x = 1;
3. Debug demo.A

Current Result

The breakpoint stays grey ("unverified") and the program runs through without stopping. We see this for every line breakpoint in every UTF-16 file of our real project.

Expected Result

The breakpoint is verified and execution stops, as it does for UTF-8 files.

Additional Informations

I think the cause is in microsoft/java-debug, JdtSourceLookUpProvider.readFile: it reads the file with new InputStreamReader(new FileInputStream(filePath)), i.e. with the JVM default charset (UTF-8). For a UTF-16 file the ASTParser then finds no package and no type, so no class name is resolved and the breakpoint can never be bound. Calling readFile from the shipped plugin jar (0.53.2) on the file above and parsing the result the same way gives no package and no type; with the BOM-aware version below it gives package demo, type A.

Workaround I tried: -Dfile.encoding=UTF-16 in java.jdt.ls.vmargs fixes the parsing, but then the language server does not start (ExceptionInInitializerError in org.eclipse.core.net, caused by NullPointerException: key can't be null while loading a native library).

A small change that honours the BOM fixed it for me (tested locally by rebuilding the plugin jar: breakpoints in UTF-16 files verify and stop, UTF-8 files with and without BOM behave as before):

private static String readFile(String filePath) {
    try {
        byte[] bytes = Files.readAllBytes(Paths.get(filePath));
        if (bytes.length >= 3 && (bytes[0] & 0xFF) == 0xEF && (bytes[1] & 0xFF) == 0xBB && (bytes[2] & 0xFF) == 0xBF) {
            return new String(bytes, 3, bytes.length - 3, StandardCharsets.UTF_8);
        }
        if (bytes.length >= 2 && (bytes[0] & 0xFF) == 0xFF && (bytes[1] & 0xFF) == 0xFE) {
            return new String(bytes, 2, bytes.length - 2, StandardCharsets.UTF_16LE);
        }
        if (bytes.length >= 2 && (bytes[0] & 0xFF) == 0xFE && (bytes[1] & 0xFF) == 0xFF) {
            return new String(bytes, 2, bytes.length - 2, StandardCharsets.UTF_16BE);
        }
        return new String(bytes, Charset.defaultCharset());
    } catch (IOException e) {
        return "";
    }
}

Using the resource charset (IFile.getCharset()) where the file belongs to a workspace project might be even better. I'm happy to open a PR in microsoft/java-debug if that helps.