Skip to content

fix(java): allocate non-static inner classes without a constructor in generated codecs - #4204

Merged
chaokunyang merged 2 commits into
apache:mainfrom
CCCLL:fix-java-inner-class-jit-allocation
Nov 4, 2026
Merged

chaokunyang merged 2 commits into
apache:mainfrom
CCCLL:fix-java-inner-class-jit-allocation

Conversation

@CCCLL

@CCCLL CCCLL commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Why?

On JDK 8-24, a generated serializer cannot be created for a non-static inner class. In the default native configuration, serialization succeeds and deserialization fails:

org.apache.fory.exception.DeserializationException: Deserialize failed, read objects are: [InnerClassRepro$Holder@6bb2d00b, null]
Caused by: org.apache.fory.codegen.CodegenException: outerPointer can't be null when InnerClassRepro$Holder$Inner is instance inner class

CodecBuilder.newBean() returns new Expression.NewInstance(beanType) for a source-public class on JDK < 25, and the NewInstance constructor rejects a non-static member class that has no outer pointer. An inner class has no no-arg constructor, so NewInstance would allocate it without calling a constructor anyway; the outer pointer is never used on this path. On JDK 25+ newBean() already takes the ObjectInstantiator branch for classes without a public no-arg constructor, so the same objects round-trip there.

Which paths are affected on JDK < 25:

  • Native compatible mode, which has been the default since 1.2.0 (feat: default compatible mode for native serialization #3742): deserializing any non-static inner class instance. Schema-consistent native mode is not affected, because CodegenSerializer.supportCodegenForJavaSerialization keeps inner classes on the interpreter ObjectSerializer.
  • With reference tracking enabled, this also covers PriorityBlockingQueue, LinkedBlockingDeque and other Serializable classes that hold a Condition: their JDK serialization form contains an AbstractQueuedSynchronizer.ConditionObject, which is a public inner class. (Without reference tracking these still fail later with IllegalMonitorStateException, because the lock and its Condition's outer instance are restored as two objects; that is unchanged by this PR and matches the interpreter path and JDK 25.)
  • xlang mode, both schema-consistent and compatible: a registered inner class fails to serialize.
  • Registering an inner class for GraalVM build-time codec generation (ensureSerializersCompiled()).

Reproducer (fails on 1.7.7 with JDK 21, works with this change):

import java.util.Arrays;
import java.util.Collections;
import java.util.concurrent.PriorityBlockingQueue;
import org.apache.fory.Fory;

public class InnerClassRepro {
  public static class Holder {
    public int value = 1;
    public Inner inner = new Inner();

    public class Inner {
      public int value = 2;
    }
  }

  public static void main(String[] args) {
    Fory fory = Fory.builder().withXlang(false).withRefTracking(true).requireClassRegistration(false).build();
    Holder holder = (Holder) fory.deserialize(fory.serialize(new Holder()));
    System.out.println("inner class: " + holder.inner.value);
    PriorityBlockingQueue<Integer> queue = new PriorityBlockingQueue<>(4, Collections.reverseOrder());
    queue.addAll(Arrays.asList(1, 3, 2));
    PriorityBlockingQueue<?> copy = (PriorityBlockingQueue<?>) fory.deserialize(fory.serialize(queue));
    System.out.println("PriorityBlockingQueue head: " + copy.peek());
  }
}

What does this PR do?

CodecBuilder.newBean() no longer builds a NewInstance for a non-static member class. Such classes take the existing no-constructor allocation branch (Unsafe.allocateInstance on JDK < 25, ObjectInstantiator on JDK 25+) that non-public classes already use. Every class that could be generated before still takes the same branch, and the wire format is unchanged; nothing changes on JDK 25+.

Routing inner classes to the interpreter CompatibleSerializer in TypeResolver (as schema-consistent mode does) would fix only the native compatible read path, leave xlang broken, and move JDK 25+ off the generated serializer that already handles these classes, so I kept the fix in newBean().

CodegenCompatibleSerializerTest#testNonStaticInnerClass round-trips a user inner class with and without reference tracking (checking the restored outer instance) and a PriorityBlockingQueue with a comparator.

Verification:

  • New test on main with only the test added: fails on JDK 8, 11, 17 and 21 with the exception above; passes on JDK 25 and 26, which are not affected.
  • New test with the change: passes on JDK 8, 11, 17, 21, 25 and 26.
  • fory-core tests with the change on JDK 8, 11, 17, 21, 25 and 26 (JDK 25+ with the CI JDK_JAVA_OPTIONS): 2352 tests, 0 failures each. fory-json (1285 tests) and fory-format (146 tests), which also inherit CodecBuilder.newBean(): 0 failures on JDK 21. The xlang peer suites were skipped locally (no peer runtimes).
  • xlang round trip of a registered inner class (schema-consistent and compatible), and ensureSerializersCompiled() with -Dorg.graalvm.nativeimage.imagecode=buildtime, checked with standalone programs on JDK 21: both fail on main and work with the change. I did not build a native image.
  • mvn -T16 spotless:check and mvn -T16 checkstyle:check: pass.

Related issues

None found.

Does this PR introduce any user-facing change?

  • Does this PR introduce any public API change?
  • Does this PR introduce any binary protocol compatibility change?

Benchmark

Not applicable: only non-static inner classes, which previously failed to generate, get different generated code.

@CCCLL
CCCLL requested a review from chaokunyang as a code owner November 3, 2026 13:33
@chaokunyang
chaokunyang merged commit bb0b749 into apache:main Nov 4, 2026
70 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants