fix(gpt_oss): free quantized expert tensors per-layer to reduce peak memory

[stanley1208]

Summary

Step 1 of #1274.

_dequantize_block_scale_tensors was holding every layer's quantized blocks/scales alive until the dequant loop finished, doubling peak memory during GPT-OSS load. This pops each entry from the local mapping right before dequantizing. No behavior change.

for layer_name in list(layer_name_to_quantized_weights.keys()):
    quantized_dict = layer_name_to_quantized_weights.pop(layer_name)
    dequantized_weights = self._convert_moe_packed_tensors(
        quantized_dict["blocks"], quantized_dict["scales"]
    )
    del quantized_dict
    state_dict[layer_name] = dequantized_weights

Originally I had a Step 2 too (CPU dequant fallback), but @akoumpa flagged the runtime cost on large models and that it doesn't help on DGX Spark unified memory. Dropped.

Won't fully close the 2× 48GB GPT-OSS 20B OOM on its own, dequantized BF16 tensors still accumulate in state_dict. I'll measure post-merge and come back with a streaming approach if needed.

Tests

26 existing tests in tests/unit_tests/models/gpt_oss/test_gptoss_state_dict_adapter.py pass. New test_dequantize_frees_quantized_between_iterations uses weakref to assert layer N's tensors are GC-able when layer N+1 starts. Confirmed it fails without the fix.

Refs #1274

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[akoumpa]

/ok to test b471951

[akoumpa]

Hi @stanley1208 ,

sorry to be bothering you with linting, but can you run

ruff format nemo_automodel/components/models/gpt_oss/state_dict_adapter.py

commit + push? so that the lint does not compalin.

Thank you & sorry for the trouble.

[akoumpa]

/ok to test 16fded5

[akoumpa]

/ok to test 5941f96

[stanley1208]

@akoumpa heads up: the CI failures look like a workflow bug, not a code issue. The launch banner shows test: is blank and the failing-jobs list includes literal matrix.test-name and gb200_${{ matrix.test-name }}, which means the matrix template isn't expanding. Probably something the recent main merge brought in. Re-running won't help until that's fixed on main.