Conversation
There was a problem hiding this comment.
drop it, unnecessary changes.
There was a problem hiding this comment.
drop it, unnecessary changes.
| # PR9; the model's `mtp_config=None` here means none should appear) but | ||
| # require ≥ 90% coverage for the main stack. | ||
| coverage_ratio = covered / len(xtuner_names) | ||
| assert coverage_ratio >= 0.90, ( |
There was a problem hiding this comment.
If it is only to skip the mtp check, please explicitly ensure that all parameters before mtp are fully loaded, rather than using magic numbers like 90%, and indicate that the mtp test is a TODO item
| assert cfg.mtp_config is not None | ||
| assert cfg.mtp_config.num_layers == 1 | ||
|
|
||
| def test_to_hf_key_list_coverage(self) -> None: |
There was a problem hiding this comment.
The testing method for parameter alignment is too indirect. It is recommended to adopt the following testing approach:
Load a DeepSeekv4-flash model with only one layer （num_hidden_layers=1）, then execute load and save, and test whether the keys obtained from load and save align with the original weights at the bitwisze level
There was a problem hiding this comment.
Combine this test and the layer_parity test into one file
| with torch.no_grad(): | ||
| hf_out = _run_hf_layer(hf_model, layer_idx, hidden_states, input_ids, position_ids) | ||
| xt_out = _run_xtuner_layer(xtuner_layer, hidden_states, input_ids, position_ids, hf_model) | ||
| torch.testing.assert_close(xt_out, hf_out, atol=atol, rtol=atol) |
There was a problem hiding this comment.
Here are a few DeepSeekv4 tests:
- Run hash router layer alone
- Run csa alone
- Run hca alone
- Run a minimum DeepSeekv4-flash, these layers all have
Finally, all tests should go through the HF path to ensure bitwise alignment!
| xt_out = _run_xtuner_layer(xtuner_layer, hidden_states, input_ids, position_ids, hf_model) | ||
| torch.testing.assert_close(xt_out, hf_out, atol=atol, rtol=atol) | ||
|
|
||
| def test_subcomponent_probe(self, capsys) -> None: |
There was a problem hiding this comment.
I completely don't get the point of this test? If it's just for temporary debugging to show where it forked, please comment it out
There was a problem hiding this comment.
All op and submodule level tests have the only reference being the Hugging Face implementation, and then control the absolute and relative errors
| TARGET_DEVICE = get_device() | ||
|
|
||
|
|
||
| def _patch_sympy_mod_eval_negative_subs() -> None: |
There was a problem hiding this comment.
Double check is this necessary
| _patch_sympy_mod_eval_negative_subs() | ||
|
|
||
|
|
||
| def _patch_triton_max_block_xblock() -> None: |
…scoring, dual RoPE
… sparse-attn + varlen triton kernels
1 participant
No description provided.