Skip to content

Run a .NET constructor once when it is instantiated from Python - #166

Draft
quantconnect-kronos[bot] wants to merge 4 commits into
masterfrom
kronos/bug-constructor-runs-twice
Draft

quantconnect-kronos[bot] wants to merge 4 commits into
masterfrom
kronos/bug-constructor-runs-twice

Conversation

@quantconnect-kronos

@quantconnect-kronos quantconnect-kronos Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

What changed and why

Constructing a .NET class from Python ran a constructor twice on the same object whenever the type had a public constructor callable with no arguments (parameterless, or all parameters optional). Stock pythonnet runs it once; the extra run comes from #67. It was also reported upstream as pythonnet#2555.

  1. MetaType.tp_call calls tp_new, then Init.
  2. ClassObject.tp_new_impl binds zero arguments against the type's constructors and invokes the match on the new object. Bug constructor default call #67 added this so that a Python subclass whose __init__ never calls super().__init__() still gets its managed base constructor.
  3. Init calls obj.__init__(*args, **kw), which binds and invokes the matching constructor on the same object again.

Field initializers allocated twice, and constructor side effects (event subscriptions, registrations, counters) happened twice.

The fix:

  • ClassObject.tp_new_impl: when the type being constructed is the reflected class itself, no constructor runs in tp_new, because tp_call always follows with __init__. Subclasses that declare __namespace__ keep the current path: their generated .NET type is registered as its own reflected type, but they're Python subclasses. Until a constructor runs on that object, none of its field initializers have run. So tp_new suppresses its finalization (GC.SuppressFinalize, marked by CLRObject.awaitingCtor), and MethodBinder.Invoke registers it again (GC.ReRegisterForFinalize) right before the first constructor runs on it. A failed T(bad args) therefore no longer leaves the uninitialized object to its finalizer. A constructor that throws still gets its object finalized, as with a .NET new.
  • Python subclasses still get the base constructor in tp_new, before their own __init__ runs. That keeps Bug constructor default call #67's guarantee, and the base constructor still runs before the subclass's __init__. Running it after __init__ instead would reset managed state that __init__ sets without calling super(), such as a property assigned in __init__. tp_new now records the constructor it ran on the object (CLRObject.ctorRanInNew), and MethodBinder.Invoke checks it. It skips the call when a zero-argument base __init__() binds that same constructor. An explicitly selected overload such as super().__init__.__overloads__[System.String]() binds a different constructor, so it still runs. For a __namespace__ subclass it also skips the base type's parameterless constructor: the generated type mirrors each base constructor and chains to it, so the parameterless constructor tp_new ran has already run the base one. Any constructor call through the binder clears the mark, and so does ClassBase.Init once that construction's __init__ returns. So a later explicit obj.__init__() still runs the constructor, as it does today.

Results, with each constructor recording its calls (Python 3.11, .NET 10):

Python code before after
SomeClass() () + () ()
SomeClass(1, "str") () + (int, string) (int, string)
Python subclass, no __init__ () + () ()
subclass calling super().__init__() () + () ()
subclass calling super().__init__(1, "s") () + (int, string) () + (int, string) (unchanged)
subclass __init__ not calling super() () ()
__namespace__ subclass, no __init__ () + () ()
__namespace__ subclass calling super().__init__() () + () ()
OnlyDefault() (only a parameterless ctor) () + () ()
OnlyArgs(1, "s") (only an (int, string) ctor) (int, string) (int, string)
OptionalArgs(5) (only an (int a = 0) ctor) (0) + (5) (5)
OptionalArgs() (0) + (0) (0)

Known limitation (accepted): one double run remains, for a Python subclass whose __init__ calls super().__init__(args) with arguments: () in tp_new, then the constructor with arguments. In tp_new there's no way to know whether __init__ will call a base constructor at all, so the parameterless one runs first. Deferring it until after __init__ would reset managed state that subclasses set in __init__ without calling super(), which changes how existing Lean algorithms behave. Maintainers decided to keep this behaviour (option A), and OptionalArgs_ValueConstructor pins it.

Tests

  • QCTest (src/embed_tests/QCTest.cs): the expected counts now match a single run. PureCSharpConstruction expects CalledDefault == 0, and NoArgs_DefaultConstructor, NoArgs_NoConstructor and WithArgs_DefaultConstructor expect CalledDefault == 1 (it was 2). New tests:

    • PureCSharpDefaultConstruction, PureCSharpOptionalArgsConstruction and PureCSharpOptionalArgsDefaultConstruction construct the type directly.
    • NoArgs_DefaultConstructorAfterOtherConstruction constructs another object inside the subclass's __init__ before calling super().__init__().
    • ReinitAfterConstructionRunsConstructorAgain calls __init__() explicitly after construction.
    • OptionalArgs_DefaultConstructor and OptionalArgs_ValueConstructor cover Python subclasses of a type whose only constructor has all-optional parameters, calling super().__init__() (1 run) and super().__init__(5) (2 runs, the accepted limitation below).
    • Namespaced_NoInit and Namespaced_DefaultConstructor cover __namespace__ subclasses.
    • SelectedOverload_OptionalArgsConstructor: a subclass calling super().__init__.__overloads__[System.String]() on a type with () and (string value = "optional") ends with the string constructor's state.
  • New test_construction_finalization (tests/test_constructors.py, fixture Python.Test.FinalizedConstructorTest). The fixture has a field initializer, a finalizer that uses the field and counts its runs, and a constructor that throws. In a subprocess, the test calls FinalizedConstructorTest("unsupported") (catches the TypeError), FinalizedConstructorTest(), and the throwing constructor (catches its exception). It then forces finalization and asserts a normal exit with exactly 2 finalizer runs: the constructed instance and the one whose constructor threw, but not the never-constructed one. With the GC.ReRegisterForFinalize call removed, it counts 0.

    Without the runtime change, 9 of the original 16 QCTests fail. Without the __namespace__ condition, Namespaced_DefaultConstructor fails. On 5757cec, the failed construction aborts the subprocess with NullReferenceException in FinalizedConstructorTest.Finalize, and SelectedOverload_OptionalArgsConstructor gets "default". With the full change, all 21 QCTests and the subprocess test pass.

  • Ran CI's suites on Python 3.11, on this branch and on master:

    • dotnet test --runtime any-x64 src/embed_tests/ --filter "FullyQualifiedName!~SetPythonPath": branch Passed 1047, Skipped 8, Failed 0 (master: Passed 1037, Skipped 8; the difference is the 10 new tests).
    • dotnet test --runtime any-x64 src/embed_tests/ --filter "FullyQualifiedName~SetPythonPath": Passed 1.
    • pytest --runtime netcore tests: 486 passed, 23 skipped (master: 485; +1 new test).
    • dotnet test --runtime any-x64 src/python_tests_runner/: Passed 2.
  • The Lean Python unit and regression test workflows run in this PR's CI.

Constructing a reflected class from Python ran a constructor twice on the
same object whenever the type had a constructor callable with no
arguments: tp_new invoked the zero-argument one, then __init__ bound and
invoked the matching constructor again. Field initializers allocated
twice and constructor side effects (event subscriptions, registrations,
counters) happened twice.

- tp_new no longer runs a constructor when the type being constructed is
  the reflected class itself: MetaType.tp_call always follows with
  __init__, which runs the matching one.
- A Python subclass still gets its base constructor in tp_new, before its
  own __init__ runs, so an __init__ that never calls the base __init__
  keeps working. When that __init__ then calls the base __init__() with no
  arguments, the binder resolves the constructor tp_new already ran, so it
  is not run a second time. The mark tp_new leaves on the object only
  lasts for that construction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@quantconnect-kronos quantconnect-kronos Bot added kronos:develop:working develop: in progress kronos:review Request review labels Nov 12, 2026
@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added kronos:review:queued review: queued kronos:review:working review: in progress and removed kronos:review Request review kronos:review:queued review: queued labels Nov 12, 2026

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review

CI is green: Build and Test on Python 3.8 to 3.14, plus the Lean Python unit and regression workflows. The root cause is right, and the approach fixes every case in the description's table. I built the head commit and master and ran the same probe scripts on both (Python 3.11, .NET 10):

Python code master this PR
Counted(v=4) (kwargs) 2 1
Python subclass of a Derived : Base type, super().__init__() Derived 2, Base 2 1, 1
two-level Python subclass, each calling super().__init__() 2 1
subclass calling Counted.__init__(self) explicitly 2 1
subclass of an (int value = 0) type, super().__init__() (2, 0) (1, 0)
subclass of an (int value = 0) type, super().__init__(5) (2, 5) (2, 5)
__namespace__ subclass, no __init__ 2 1
__namespace__ subclass, super().__init__() 2 2
System.Exception('m').Message, List[int]() ok ok
Counted.__new__(Counted) constructed, Calls == 1 uninitialized: field initializers not run, Calls == 0

The last row matches stock pythonnet. copy.copy, copy.deepcopy and pickle of a .NET object raise cannot pickle on both builds, so they don't reach the uninitialized path.

I checked the reason for keeping the base constructor before __init__, and it holds. Lean's PythonIndicator() is : base(""), and CustomSimpleMovingAverage.__init__ in CustomIndicatorWithExtensionAlgorithm.py sets self.name without calling super(). If the base constructor ran after __init__, it would reset that name.

The new checks on the construction path cost a ConstructorInfo type test in MethodBinder.Invoke and a GetManagedObject in ClassBase.Init. Exact-type construction also stops building a MethodBinder in tp_new. Net, construction does less work than before.

Findings

  1. Needs a decision: the remaining double run for super().__init__(args). A Python subclass whose __init__ calls the base __init__ with arguments still runs () + (int, string). Stock pythonnet runs only (int, string), and that is the result this fix is meant to reach. The description's question is the right one: in tp_new there's no way to know which constructor __init__ will call. Removing the extra run means moving the base constructor after __init__, which breaks the pattern above. A QuantConnect maintainer needs to accept this remaining case on the PR, or pick the trade-off. Once that's settled, this finding is resolved either way.
  2. __namespace__ subclasses calling super().__init__() still run the constructor twice (inline on MethodBinder.cs). Either cover the case or list it in the description as unchanged.
  3. Test gap (inline on QCTest.cs): nothing covers the skip for a Python subclass of a type whose only constructor has all-optional parameters.

Risk

I set risk:high. The change affects how every .NET object is constructed from Python: every Lean Python algorithm, custom data type and indicator goes through it. It also changes what T.__new__(T) returns. That blast radius is the high-risk trait, and it stays even with the green Lean workflows.

Other repositories

No other repository needs a code change. Lean picks this up with the next pythonnet package version.

Comment thread src/runtime/MethodBinder.cs Outdated
co.ranZeroArgCtorInNew = false;
// A zero-argument base __init__() call binds the constructor tp_new already ran on this
// Python subclass instance (same type's constructors, same zero arguments): don't run it twice.
if (binding.info.DeclaringType == co.inst.GetType()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For a subclass that declares __namespace__, co.inst.GetType() is the generated .NET type. Its super().__init__() binds a constructor declared on the reflected base type, so this comparison fails, the mark is cleared, and the base constructor runs a second time. I measured 2 runs on both master and this branch for:

class A(Counted):
    __namespace__ = 'probe.ns'
    def __init__(self):
        super().__init__()

The same subclass without __init__ now runs once. This isn't a regression, but it's the same double run the PR fixes. Please check whether the generated zero-argument constructor already chains to the base one that super().__init__() binds. If it does, comparing against the base type for an IPythonDerivedType instance would cover this case; add a QCTest for it. If not, list it in the description as unchanged.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Covered in 5757cec. The generated type mirrors each base constructor and its body calls the base one (_Base__cinit__ → call ctor), so the parameterless constructor tp_new runs on the generated type has already run the base parameterless one. The skip now also accepts the base type's parameterless constructor when the instance is an IPythonDerivedType. It requires binding.args.Length == 0 because mirrors don't keep default values, so an all-optional base constructor never gets the mark from tp_new. New Namespaced_DefaultConstructor (1 run; fails without the condition) and Namespaced_NoInit.

Comment thread src/embed_tests/QCTest.cs Resolved
@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added risk:high High risk and removed kronos:review:working review: in progress labels Nov 12, 2026
A __namespace__ subclass's generated .NET type mirrors each base
constructor and chains to it, so the parameterless constructor tp_new
runs on it has already run the base type's parameterless one. A
zero-argument super().__init__() binds that base constructor, which was
then run a second time. Also pin the all-optional-parameter constructor
cases for Python subclasses in QCTest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@quantconnect-kronos

Copy link
Copy Markdown
Author

Addressed the review in 5757cec:

  • __namespace__ subclasses calling super().__init__() now run the base constructor once. The generated type's constructors chain to the base ones, so the skip also accepts the base type's parameterless constructor for an IPythonDerivedType instance. Added Namespaced_DefaultConstructor (fails without the change) and Namespaced_NoInit.
  • All-optional-parameter constructor: added OptionalArgs_DefaultConstructor (1 run) and OptionalArgs_ValueConstructor (2 runs, pinning the open case).
  • super().__init__(args) with arguments: still open for a maintainer to decide. The question is in the description.

CI suites on Python 3.11: embed tests passed 1046, skipped 8, failed 0; SetPythonPath 1/1; pytest --runtime netcore tests 485 passed, 23 skipped; python_tests_runner 2/2. The description's table and test list are updated.

@quantconnect-kronos quantconnect-kronos Bot added the kronos:review Request review label Nov 12, 2026
@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added kronos:review:queued review: queued kronos:review:working review: in progress and removed kronos:review Request review kronos:review:queued review: queued labels Nov 12, 2026

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of 5757cec

CI is green on this commit: Build and Test on Python 3.8 to 3.14, plus the Lean Python unit and regression workflows.

What changed since my last review

  • __namespace__ subclasses calling super().__init__(): resolved. I checked the claim that the generated type's constructors chain to the base ones. ClassDerived.AddConstructor (src/runtime/Types/ClassDerived.cs:307-359) emits each mirrored constructor as a call to InvokeCtor, which calls _<Base>__cinit__, and that method calls the base constructor. So the parameterless constructor tp_new runs on the generated type has already run the base one, and skipping the base () in MethodBinder.Invoke is correct. The binding.args.Length == 0 guard is also correct: mirrored constructors don't keep default values, so an all-optional base constructor never gets the mark.
  • Optional-argument subclasses: resolved. OptionalArgs_DefaultConstructor and OptionalArgs_ValueConstructor pin both cases.

I ran the 20 QCTests on the head commit locally, and all pass. I also probed the new __namespace__ path for cases the tests don't cover (Python 3.11, .NET 10):

Python code constructor runs
__namespace__ subclass of (int value = 0), super().__init__() 1, Value == 0
same, super().__init__(5) 1, Value == 5
__namespace__ subclass, then explicit obj.__init__() 1, then 2 (the re-init still runs, as on master)
plain Python subclass of a __namespace__ class, with and without __init__ 1
__namespace__ subclass calling Base.__init__(self) explicitly 1
__namespace__ subclass constructing another instance in __init__ before super().__init__() 1 and 1
type with both () and (int a = 7): direct, Python subclass and __namespace__ subclass, each calling () () once, (int) never

None of these regress.

Findings

  1. Still open, needs a decision: the remaining double run for super().__init__(args). A Python subclass whose __init__ calls the base __init__ with arguments still runs () + (int, string). Stock pythonnet runs only (int, string), and that's the result the original report asks for. The only way to get there is to defer the base constructor until after __init__. That breaks subclasses that set managed state in __init__ without calling super(), which I verified last time against a real indicator pattern. A QuantConnect maintainer needs to accept this remaining case or pick the trade-off on this PR. Once they do, this finding is resolved either way, and OptionalArgs_ValueConstructor already pins whichever way it goes.

Risk

risk:high stays. The change affects how every .NET object is constructed from Python, including every Python algorithm, custom data type and indicator, and it changes what T.__new__(T) returns. Green Lean workflows and the probes above don't shrink that blast radius.

Other repositories

No other repository needs a code change. Downstream picks this up with the next pythonnet package version.

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot removed the kronos:review:working review: in progress label Nov 12, 2026
@quantconnect-kronos

Copy link
Copy Markdown
Author

@Martin-Molinero, one decision is needed before this can go further. The rest of the review is resolved, and there are no code changes in this round.

A Python subclass whose __init__ calls super().__init__(args) with arguments still runs the parameterless base constructor first (in tp_new), then the one with arguments: () + (int, string). Stock pythonnet runs only (int, string).

  • Option A (this PR as is): keep the base constructor before __init__. The remaining double run is limited to this one case. A subclass whose __init__ never calls super() still has its managed state initialized before its __init__ body runs.
  • Option B: defer the base constructor until after __init__, and run it only when no constructor ran during __init__. That matches stock pythonnet in every case. But a subclass that sets managed state in __init__ without calling super() would have it reset afterwards. Lean's CustomSimpleMovingAverage in CustomIndicatorWithExtensionAlgorithm.py does this: it sets self.name, and PythonIndicator() : base("") would then overwrite it. Its __init__ could also call managed members before the base constructor has run.

OptionalArgs_ValueConstructor pins the current behaviour (2 runs). Which option do you want?

@jhonabreul

Copy link
Copy Markdown
Collaborator

About the decision, does stock Pythonnet has the same limitation as option B?

@quantconnect-kronos

Copy link
Copy Markdown
Author

No. Stock pythonnet has a different limitation: it never runs a base constructor unless __init__ calls super().__init__(...). So it can't reset anything after __init__, but a subclass that doesn't call super() gets an object whose constructor never ran: field initializers unset, constructor body skipped. That is what #67 fixed.

I measured it with a .NET class Indicator that has Name, a field initializer List<int> Window = new List<int>(), Indicator() { Name = ""; } and Indicator(string name). Python 3.11, .NET 10:

Python subclass stock 3.0.5 and 3.2.1 this PR (option A) option B (expected)
__init__ sets self.Name = name, no super() 0 constructor runs, Name == "sma", Window is None 1 run, Name == "sma", Window set 1 run after __init__: Window set, but Name reset to ""
__init__ calls a member using Window, no super() NullReferenceException 1 run, works NullReferenceException (the constructor hasn't run yet when __init__ calls the member)
__init__ calls super().__init__("sma") 1 run 2 runs (() + (string)) 1 run

So the three behave differently:

  • Stock: one constructor run at most, but none at all without super().
  • Option B: stock plus a parameterless constructor after __init__. It also covers the no-super() case, at the cost of overwriting what __init__ set and failing if __init__ uses managed members.
  • Option A (this PR): always initializes the object before __init__, at the cost of the extra run when super().__init__(args) has arguments.

No code changes in this round.

@jhonabreul

Copy link
Copy Markdown
Collaborator

Go with Option A, the PR as it is with this one only limitation so we do not change current behaviour in existing Lean algorithms

@Martin-Molinero

Copy link
Copy Markdown
Member

Go for Option A, the PR as it is.

@quantconnect-kronos

Copy link
Copy Markdown
Author

Thanks, going with option A: the PR stays as it is. I replaced the open question in the description with the decision, as a "Known limitation (accepted)" note. It says a subclass calling super().__init__(args) with arguments still runs () before that constructor, which keeps existing Lean algorithms behaving as they do today. OptionalArgs_ValueConstructor pins that behaviour. No code changes; the head is still 5757cec3.

@quantconnect-kronos quantconnect-kronos Bot added the kronos:review Request review label Nov 12, 2026
@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added kronos:review:queued review: queued kronos:review:working review: in progress and removed kronos:review Request review kronos:review:queued review: queued labels Nov 12, 2026
@quantconnect-kronos-reviewer
quantconnect-kronos-reviewer Bot marked this pull request as ready for review November 12, 2026 20:58

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of 5757cec

The code hasn't changed since my last review, and CI is still green on this commit: Build and Test on Python 3.8 to 3.14, plus the Lean Python unit and regression workflows.

Findings

  1. The remaining double run for super().__init__(args): resolved by decision. Two maintainers chose option A on this PR: keep the base constructor in tp_new so existing algorithms behave as they do today, and accept the extra () run before a base constructor that __init__ calls with arguments. The description now lists it as a known limitation, and OptionalArgs_ValueConstructor pins it.

Nothing else is open. My earlier probes and the 20 QCTests on this commit cover the rest of the description's table, including the __namespace__ paths.

Risk

risk:high stays. Every .NET object constructed from Python goes through this path, including every Python algorithm, custom data type and indicator, and T.__new__(T) now returns an object whose constructor hasn't run, as in stock pythonnet. The green Lean workflows and the tests support approving it, but they don't shrink that blast radius, so a person should merge it.

Other repositories

No other repository needs a code change. Downstream gets the fix with the next pythonnet package version.

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot removed the kronos:review:working review: in progress label Nov 12, 2026
@quantconnect-kronos quantconnect-kronos Bot added the kronos:review Request review label Nov 12, 2026
@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added kronos:review:queued review: queued and removed kronos:review Request review labels Nov 12, 2026
@quantconnect-kronos-reviewer-2 quantconnect-kronos-reviewer-2 Bot added kronos:review:working review: in progress and removed kronos:review:queued review: queued labels Nov 12, 2026

@quantconnect-kronos-reviewer-2 quantconnect-kronos-reviewer-2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of 5757cec

The earlier findings are resolved, including the maintainers' choice of option A. I am not asking to revisit that decision. However, independent base/head probes found two additional regressions that block this commit:

  1. Failed constructor binding can terminate the host during finalization. An unsupported argument to an otherwise parameterless, finalizable CLR class now leaves an allocated object whose field initializers never ran. Python catches the expected TypeError, but the object's finalizer subsequently dereferences an uninitialized field and aborts the process. The same probe on master catches the error and exits normally.
  2. An explicitly selected optional-parameter constructor is silently skipped. For a class exposing both C() and C(string value = "optional"), a Python subclass calling super().__init__.__overloads__[System.String]() previously finishes with the string constructor's state. This commit leaves the parameterless constructor's state instead. The skip recognizes the declaring type, not the constructor that actually ran.

Both reproductions and the requested regression coverage are in the inline comments.

Verification

  • CI is green: Build and Test on Python 3.8–3.14 and both Lean Python workflows.
  • Locally, dotnet test --no-build --runtime any-x64 src/embed_tests/ --filter FullyQualifiedName~QCTests passed all 20 tests with the installed Python shared library configured.
  • Separate .NET 10 / Python 3.11 probe processes loaded builds of master and this head. The failed-binding/finalizer probe exited 0 on master, 134 on this head, with an unhandled NullReferenceException in the finalizer. The explicit-overload probe returned optional on master, default on this head.

Risk and consumers

risk:high remains: this changes the common CLR construction path, and the failed-binding probe demonstrates a new host-termination path. No separate consumer code change is needed; downstream adoption should wait for these runtime regressions to be fixed and covered by tests.

Comment thread src/runtime/Types/ClassObject.cs Resolved
Comment thread src/runtime/MethodBinder.cs Outdated
// A __namespace__ subclass's generated type mirrors each base constructor and chains to it, so
// its parameterless constructor has already run the base type's parameterless one.
var instType = co.inst.GetType();
if ((binding.info.DeclaringType == instType

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Compare the constructor that ran, not only its declaring type. Empty Python arguments do not imply that binding selected the same constructor as tp_new: explicit overload selection can choose a different all-optional constructor. I tested a CLR Counted exposing Counted() (sets Selected = "default") and Counted(string value = "optional") (sets Selected = value) with:

import System

class S(Counted):
    def __init__(self):
        super().__init__.__overloads__[System.String]()

obj = S()

Master runs the explicitly selected string constructor and produces obj.Selected == "optional"; this head silently skips it and produces "default". Both constructors have the same declaring type, and the original Python tuple/kwargs are empty, so this condition suppresses a constructor that has not run. Please retain/compare the actual constructor identity or signature, including any generated-to-base mapping, and add a test preserving the explicitly selected overload's state.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6089cc0. tp_new now records the constructor it ran (CLRObject.ctorRanInNew holds the MethodBase instead of a bool). The skip requires binding.info == ctorRanInNew; for a __namespace__ subclass it instead requires the base type's parameterless constructor that the generated parameterless one chains to. Your __overloads__[System.String]() case binds a different constructor, so it runs. New SelectedOverload_OptionalArgsConstructor (() plus (string value = "optional")) asserts Selected == "optional"; it gets "default" on 5757cec.

@quantconnect-kronos-reviewer-2 quantconnect-kronos-reviewer-2 Bot removed the kronos:review:working review: in progress label Nov 12, 2026
…tor tp_new ran

- tp_new hands out the reflected type's object without running a
  constructor, so a failed __init__ (no matching overload) left an object
  with no field initializers run to its finalizer, which could throw and
  terminate the process. tp_new now suppresses finalization of that object,
  and the binder registers it again right before a constructor runs on it,
  as for a .NET 'new'.
- The skip of a repeated base constructor compared only the declaring type,
  so an explicitly selected overload such as
  super().__init__.__overloads__[System.String]() was skipped too. tp_new now
  records the constructor it ran, and only that constructor (or, for a
  __namespace__ subclass, the base parameterless constructor its generated
  one chains to) is skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@quantconnect-kronos

Copy link
Copy Markdown
Author

Addressed both regressions from the second review in 6089cc0:

  • Failed construction and finalization (P1): tp_new suppresses finalization of the reflected type's object until a constructor runs on it. The binder re-registers it right before the first constructor, so a failed T(bad args) no longer leaves an uninitialized object to its finalizer. New subprocess test test_failed_construction_does_not_finalize_uninitialized_object aborts on 5757cec and passes now.
  • Explicitly selected overload (P2): the skip now compares the constructor tp_new actually ran (CLRObject.ctorRanInNew), not its declaring type. New SelectedOverload_OptionalArgsConstructor gets "default" on 5757cec and "optional" now.

CI suites on Python 3.11 at 6089cc0:

Suite Passed Skipped Failed
embed tests 1047 8 0
SetPythonPath 1 0 0
pytest --runtime netcore tests 486 23 0
python_tests_runner 2 0 0

The description is updated with the new mechanism and tests. Option A, as decided, is unchanged.

@quantconnect-kronos
quantconnect-kronos Bot marked this pull request as draft November 12, 2026 21:25
@quantconnect-kronos quantconnect-kronos Bot added the kronos:review Request review label Nov 12, 2026
@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added kronos:review:queued review: queued kronos:review:working review: in progress and removed kronos:review Request review kronos:review:queued review: queued labels Nov 12, 2026

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of 6089cc0

CI is green on this commit: Build and Test on Python 3.8 to 3.14, plus the Lean Python unit and regression workflows. The CI logs show that test_failed_construction_does_not_finalize_uninitialized_object runs on every Python version and isn't skipped.

What changed since the last reviews

Both regressions from the second review on 5757cec are fixed.

  • Failed construction and the finalizer: resolved. I built the head and ran probes with a finalizable class that has a field initializer and a finalizer that uses it, each scenario 100 times (Python 3.11, .NET 10). Each probe ran gc.collect(), then GC.Collect() and GC.WaitForPendingFinalizers() twice:

    Python code constructors run finalizers run exit
    Fin() 100 100 0
    Fin(i) 100 100 0
    FinNoDefault(i) (no parameterless constructor) 100 100 0
    Fin("x"), constructor throws 100 100 0
    Fin(1.5, 2), no matching overload, TypeError caught 0 0 0
    Fin.__new__(Fin), then o.__init__(3) 100 100 0
    Fin.__new__(Fin) only 0 0 0
    Python subclass calling super().__init__(2) 200 (accepted limitation) 100 0

    A successful construction is finalized exactly once, not twice. After SuppressFinalize, ReRegisterForFinalize only clears the suppressed bit and doesn't queue a second entry. A constructor that throws is still finalized, as with a .NET new.

  • Explicitly selected overload: resolved. The skip now compares the constructor tp_new ran (binding.info == ranInNew). For a __namespace__ subclass it accepts only the base parameterless constructor, and only when the generated constructor tp_new ran is the parameterless one. SelectedOverload_OptionalArgsConstructor pins it. __init__ overloads go through MethodBinder.Invoke too, so no other constructor path skips the re-register.

The extra cost per construction is a GetManagedObject and the SuppressFinalize/ReRegisterForFinalize pair. Both calls only flip a bit in the object header, so this isn't a hot-path concern.

Findings

  1. No test covers the half of the new finalization mechanism that keeps today's behaviour (inline on MethodBinder.cs). Every object of the reflected type now starts with finalization suppressed. GC.ReRegisterForFinalize in the binder is the only thing that gives it back. If that line is dropped or moved behind a branch that doesn't run, no finalizable .NET object constructed from Python is ever finalized again, and no test fails. The new subprocess test only asserts the failure path: that the finalizer doesn't run. The only fixture with a finalizer is FinalizedConstructorTest, and nothing in tests/ or src/embed_tests/ checks that a .NET object constructed from Python still gets finalized. The TestFinalizer tests cover PyObject wrappers. Please extend the subprocess test, for example with a static counter that the fixture's finalizer increments. It should assert that a successful FinalizedConstructorTest() is finalized once, and that an instance whose constructor throws is finalized, as the description claims.

Risk

risk:high stays. Every .NET object constructed from Python goes through this path, including every Python algorithm, custom data type and indicator. This commit adds a new way for that path to go wrong: finalization is now suppressed and re-enabled on every construction. The probes above show it behaves correctly today, but finding 1 is what would keep it that way.

Other repositories

No other repository needs a code change. Downstream gets the fix with the next pythonnet package version.

// tp_new suppressed finalization until a constructor runs; register it before the
// constructor body, as for a .NET 'new', so the object is finalized even if it throws.
co.awaitingCtor = false;
GC.ReRegisterForFinalize(co.inst);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is now the only thing that makes a .NET object constructed from Python finalizable, because tp_new suppresses finalization for every reflected-type instance. Nothing tests it. If it's removed, every test still passes, including the new subprocess test, which only asserts that the finalizer does not run after a failed bind. I checked the head with a finalizable probe class: Fin() and a throwing Fin("x") are each finalized once, so the code is right. Please pin it: in the subprocess script, construct FinalizedConstructorTest() successfully, and also call a constructor that throws, then assert both were finalized, for example through a static counter the finalizer increments.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pinned in 2692e63. FinalizedConstructorTest now has a static Finalized counter that its finalizer increments, and a constructor that throws a given exception. The subprocess test, renamed test_construction_finalization, runs three constructions: a failed bind (TypeError caught), a successful FinalizedConstructorTest(), and the throwing constructor (exception caught). It then forces finalization and asserts the counter is exactly 2. With this GC.ReRegisterForFinalize line commented out, the test fails with 0 != 2.

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot removed the kronos:review:working review: in progress label Nov 12, 2026
tp_new suppresses finalization of the reflected type's object until a
constructor runs, and the binder re-registers it, so the subprocess test
now also asserts that a successfully constructed instance, and one whose
constructor throws, are each finalized once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@quantconnect-kronos

Copy link
Copy Markdown
Author

Addressed the review of 6089cc0 in 2692e63 (tests only, no runtime change). The subprocess test, now test_construction_finalization, asserts the re-enable half of the finalization mechanism too. A successfully constructed FinalizedConstructorTest() and one whose constructor throws are each finalized once, and the never-constructed object from a failed bind isn't. That is exactly 2 finalizer runs. Removing GC.ReRegisterForFinalize from MethodBinder.Invoke makes it fail with 0.

CI suites on Python 3.11 at 2692e63:

Suite Passed Skipped Failed
embed tests 1047 8 0
SetPythonPath 1 0 0
pytest --runtime netcore tests 486 23 0
python_tests_runner 2 0 0

The description is updated.

@quantconnect-kronos quantconnect-kronos Bot added the kronos:review Request review label Nov 12, 2026
@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added kronos:review:queued review: queued kronos:review:working review: in progress and removed kronos:review Request review kronos:review:queued review: queued labels Nov 12, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kronos:develop:working develop: in progress kronos:review:working review: in progress risk:high High risk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants