From fd15d1beef73842742737beb5283897eee5ba266 Mon Sep 17 00:00:00 2001 From: Philip Washington Sorst Date: Mon, 19 Oct 2026 17:07:54 +0200 Subject: [PATCH 1/2] fix: forward ref_image_args to the video pipeline --ref-image-args was only plumbed through sd_img_gen_params_t, so every key was silently dropped in vid_gen mode. For MiniMax-H3 Ref2VA this meant references were always rescaled to the canvas area, and the documented resize_before_vae=false escape hatch had no effect. Generation metadata still recorded the value. Add ref_image_args to sd_vid_gen_params_t (appended last to keep struct offsets stable), forward it from the CLI and server, and resolve it in generate_video() instead of using a default-constructed RefImageParams. AnimateDiff, which reuses the image path, forwards the value too. Add a ref_image_size key with match (default) and max modes, matching ComfyUI's nodes_minimax_h3.py. match reproduces the previous arithmetic exactly, so existing renders are unchanged; max sizes references by a 2048 px short edge for better identity fidelity. Neither mode upscales. Both keep the per-axis 32-px rounding that MiniMax-H3's patchify_3d requires. The new RefImageParams member is part of ConditioningCache::same_params() so that requests differing only by this setting do not reuse stale conditioning. H3 consumes references through its own packed sequence rather than the generic conditioner/DiT reference path, so the remaining keys cannot take effect there. resolve_ref_image_params() now warns for those instead of dropping them silently, and warns that resize_before_vae=false is unsupported rather than ignoring it. Invalid values are reported once and not repeated by the per-model warnings. Also log the effective reference dimensions per image and warn when reference patch tokens exceed 4x the generated video's tokens. Verification: built stable-diffusion, sd-cli and sd-server and ran MiniMax-H3 Ref2VA with and without ref_image_size=max; match keeps the previous reference dimensions, max produces the 2048-short-edge sizing. The match branch was additionally checked against the previous inline formula with no mismatches. --- docs/edit.md | 3 ++ docs/image_preprocessing.md | 3 +- docs/minimax_h3.md | 9 ++++ examples/common/common.cpp | 1 + examples/server/api.md | 6 ++- examples/server/routes_sdcpp.cpp | 1 + include/stable-diffusion.h | 1 + src/conditioning/conditioning_cache.h | 3 +- src/model/diffusion/model.hpp | 6 +++ src/pipeline/diffusion_engine.cpp | 53 ++++++++++++++++++++ src/pipeline/video.cpp | 71 ++++++++++++++++++++++----- src/stable-diffusion.cpp | 1 + 12 files changed, 144 insertions(+), 14 deletions(-) diff --git a/docs/edit.md b/docs/edit.md index 62ecff2af5..c3b0c5212c 100644 --- a/docs/edit.md +++ b/docs/edit.md @@ -78,6 +78,9 @@ If presets are insufficient, you can manually configure the following parameters | `vlm_max_size` | int | Maximum VLM input size; interpreted according to `vlm_resize_mode`. | Integer | | `vlm_min_size` | int | Minimum VLM input size; interpreted according to `vlm_resize_mode`. | Integer | | `vlm_size` | int | Shortcut to set both VLM min and max size to the same value. | Integer | +| `ref_image_size` | string | MiniMax-H3 reference image sizing before VAE encoding. `match` scales each reference down to the generation's pixel area; `max` uses a 2048 px short edge. Both only downscale. | `match`, `max` | + +`ref_image_size` applies to `vid_gen` with MiniMax-H3 Ref2VA. Reference latents are attended to at every sampling step, so `max` can be several times slower. ### Preset Default Values diff --git a/docs/image_preprocessing.md b/docs/image_preprocessing.md index 719781750e..1ca6460e8a 100644 --- a/docs/image_preprocessing.md +++ b/docs/image_preprocessing.md @@ -128,7 +128,8 @@ To disable reference resizing before VAE encoding, use `--ref-image-args "resize_before_vae=false"` or the server field `"ref_image_args": "resize_before_vae=false"`. This is separate from `target=ref,mode=none`, which only skips input geometry. Model constraints -still apply. +still apply; MiniMax-H3 requires 32-pixel-aligned references and warns instead +of honoring the switch. ## Server requests diff --git a/docs/minimax_h3.md b/docs/minimax_h3.md index 64eb78f539..c7811a94fa 100644 --- a/docs/minimax_h3.md +++ b/docs/minimax_h3.md @@ -83,6 +83,15 @@ share the same timeline. Ref2VA cannot be combined with `--init-img` or Reference images keep their aspect ratio and are only downscaled when their pixel area exceeds the requested generation canvas. +Pass `--ref-image-args "ref_image_size=max"` (or `ref_image_size` in +`sd_vid_gen_params_t.ref_image_args`) to size references by a 2048 px short edge +instead, which preserves more identity detail at a higher sampling cost. Both +modes only downscale: references at or below the limit keep their size. Either +way the per-axis result is rounded to the 32-pixel grid, so a dimension can grow +by up to 16 pixels, and very small references are widened to 32 pixels. +`resize_before_vae=false` is not supported here; MiniMax-H3 keeps references on +the 32-pixel grid and reports a warning. + The C API exposes the same inputs through `ref_images`, `ref_videos`, and `ref_audios` in `sd_vid_gen_params_t`. Each `sd_ref_video_t` supplies its own frame rate and optional soundtrack; non-24-fps inputs are resampled internally. diff --git a/examples/common/common.cpp b/examples/common/common.cpp index 9e7851fef1..6922c50a4b 100644 --- a/examples/common/common.cpp +++ b/examples/common/common.cpp @@ -2910,6 +2910,7 @@ sd_vid_gen_params_t SDGenerationParams::to_sd_vid_gen_params_t() { params.circular_x = circular || circular_x; params.circular_y = circular || circular_y; params.image_preprocess = {image_preprocess.c_str()}; + params.ref_image_args = ref_image_args.c_str(); return params; } diff --git a/examples/server/api.md b/examples/server/api.md index bb42523661..429aa4fe6a 100644 --- a/examples/server/api.md +++ b/examples/server/api.md @@ -901,6 +901,7 @@ LTX and Wan preserve causal state between temporal tiles. Hunyuan Video and TAEH | --- | --- | | `video_frames` | `integer` | | `fps` | `integer` | +| `ref_image_args` | `string` | | `moe_boundary` | `number` | | `vace_strength` | `number` | | `high_noise_sample_params` | `object` | @@ -1394,7 +1395,7 @@ Compared with `img_gen`, the `vid_gen` request body: - `vid_gen` is a single video sequence job, so `batch_count` is not part of the request schema - `mask_image`, `control_image`, `control_strength`, `ip_adapter_image`, `ip_adapter_strength`, and `embed_image_metadata` are not part of the request schema - `ref_images` is accepted for MiniMax-H3 Ref2VA conditioning; other video model families currently ignore it -- `vid_gen` adds `end_image`, `control_frames`, `high_noise_sample_params`, `video_frames`, `fps`, `moe_boundary`, and `vace_strength` +- `vid_gen` adds `end_image`, `control_frames`, `high_noise_sample_params`, `video_frames`, `fps`, `ref_image_args`, `moe_boundary`, and `vace_strength` Example: @@ -1409,6 +1410,7 @@ Example: "seed": -1, "video_frames": 33, "fps": 16, + "ref_image_args": "", "moe_boundary": 0.875, "vace_strength": 1.0, @@ -1490,6 +1492,7 @@ Example: - Images retain request order and correspond to ``, ``, and so on in the prompt. - For MiniMax-H3, non-empty `ref_images` cannot be combined with `init_image` or `end_image`. A conflicting API request fails during generation; the WebUI checks this before submission. - MiniMax-H3 does not support `control_frames`; leave that array empty. +- `ref_image_args` applies to `vid_gen`; MiniMax-H3 honors `ref_image_size` there (`match` default, `max` for a 2048 px short edge). Both modes only downscale references. - Other video model families currently ignore `ref_images`. - `features_by_mode.vid_gen` does not currently advertise `ref_images`; its absence is not an indication that MiniMax-H3 reference images are unsupported. @@ -1540,6 +1543,7 @@ Top-level scalar fields: | `seed` | `integer` | | `video_frames` | `integer` | | `fps` | `integer` | +| `ref_image_args` | `string` | | `moe_boundary` | `number` | | `vace_strength` | `number` | | `preview` | `string` | diff --git a/examples/server/routes_sdcpp.cpp b/examples/server/routes_sdcpp.cpp index 7e93e5edbf..514114e1ad 100644 --- a/examples/server/routes_sdcpp.cpp +++ b/examples/server/routes_sdcpp.cpp @@ -178,6 +178,7 @@ static json make_vid_gen_defaults_json(const SDGenerationParams& defaults, const {"seed", defaults.seed}, {"video_frames", defaults.video_frames}, {"image_preprocess", defaults.image_preprocess}, + {"ref_image_args", defaults.ref_image_args}, {"fps", defaults.fps}, {"moe_boundary", defaults.moe_boundary}, {"vace_strength", defaults.vace_strength}, diff --git a/include/stable-diffusion.h b/include/stable-diffusion.h index 404d8cbddc..c4863c90e6 100644 --- a/include/stable-diffusion.h +++ b/include/stable-diffusion.h @@ -456,6 +456,7 @@ typedef struct { bool circular_x; bool circular_y; sd_image_preprocess_params_t image_preprocess; + const char* ref_image_args; } sd_vid_gen_params_t; typedef struct sd_ctx_t sd_ctx_t; diff --git a/src/conditioning/conditioning_cache.h b/src/conditioning/conditioning_cache.h index 6e6cbcaf77..34648c9ad1 100644 --- a/src/conditioning/conditioning_cache.h +++ b/src/conditioning/conditioning_cache.h @@ -48,7 +48,8 @@ class ConditioningCache { return std::tie(p.text, p.clip_skip, p.width, p.height, p.zero_out_masked, r.pass_to_vlm, r.pass_to_dit, r.ref_index_mode, r.force_ref_timestep_zero, r.resize_before_vae, r.vae_input_max_pixels, - r.vlm_resize_mode, r.vlm_min_size, r.vlm_max_size, r.resize_vae_to_target); + r.vlm_resize_mode, r.vlm_min_size, r.vlm_max_size, r.resize_vae_to_target, + r.ref_image_size); }; if (fields(a) != fields(b) || (a.ref_images == nullptr) != (b.ref_images == nullptr) || diff --git a/src/model/diffusion/model.hpp b/src/model/diffusion/model.hpp index 955c5277a7..19b3ee2f5e 100644 --- a/src/model/diffusion/model.hpp +++ b/src/model/diffusion/model.hpp @@ -16,6 +16,11 @@ enum class RefImageResizeMode { AREA, }; +enum class RefImageSizeMode { + MATCH, + MAX, +}; + struct RefImageParams { bool pass_to_vlm = false; bool pass_to_dit = true; @@ -27,6 +32,7 @@ struct RefImageParams { int vlm_min_size = -1; int vlm_max_size = -1; bool resize_vae_to_target = false; + RefImageSizeMode ref_image_size = RefImageSizeMode::MATCH; }; const std::unordered_map REF_IMAGE_PRESETS = { diff --git a/src/pipeline/diffusion_engine.cpp b/src/pipeline/diffusion_engine.cpp index 22ef3ac459..4a24bd06d2 100644 --- a/src/pipeline/diffusion_engine.cpp +++ b/src/pipeline/diffusion_engine.cpp @@ -3237,6 +3237,9 @@ std::string StableDiffusionGGML::get_default_ref_image_preset(SDVersion version) RefImageParams StableDiffusionGGML::resolve_ref_image_params(const char* ref_image_args) const { RefImageParams params; std::string preset_name = get_default_ref_image_preset(version); + // Keys whose value was rejected. They are already reported, so per-model "no effect" + // warnings must not repeat them. + std::set rejected_keys; for (const auto& [key, value] : parse_key_value_args(ref_image_args, "reference image args")) { if (key == "preset") { @@ -3249,6 +3252,7 @@ RefImageParams StableDiffusionGGML::resolve_ref_image_params(const char* ref_ima valid_list += (valid_list.empty() ? "" : ", ") + name; } LOG_WARN("ignoring invalid reference image preset '%s'. Valid options: [%s]", value.c_str(), valid_list.c_str()); + rejected_keys.insert(key); } break; } @@ -3262,10 +3266,12 @@ RefImageParams StableDiffusionGGML::resolve_ref_image_params(const char* ref_ima if (key == "pass_to_vlm") { if (!parse_strict_bool(value, params.pass_to_vlm)) { LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); + rejected_keys.insert(key); } } else if (key == "pass_to_dit") { if (!parse_strict_bool(value, params.pass_to_dit)) { LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); + rejected_keys.insert(key); } } else if (key == "ref_index_mode") { if (value == "fixed") { @@ -3276,18 +3282,22 @@ RefImageParams StableDiffusionGGML::resolve_ref_image_params(const char* ref_ima params.ref_index_mode = Rope::RefIndexMode::DECREASE; } else { LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); + rejected_keys.insert(key); } } else if (key == "force_ref_timestep_zero") { if (!parse_strict_bool(value, params.force_ref_timestep_zero)) { LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); + rejected_keys.insert(key); } } else if (key == "resize_before_vae") { if (!parse_strict_bool(value, params.resize_before_vae)) { LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); + rejected_keys.insert(key); } } else if (key == "vae_input_max_pixels") { if (!parse_strict_int(value, params.vae_input_max_pixels)) { LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); + rejected_keys.insert(key); } } else if (key == "vlm_resize_mode") { if (value == "longest_side") { @@ -3298,14 +3308,31 @@ RefImageParams StableDiffusionGGML::resolve_ref_image_params(const char* ref_ima params.vlm_resize_mode = RefImageResizeMode::NONE; } else { LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); + rejected_keys.insert(key); } } else if (key == "vlm_max_size") { if (!parse_strict_int(value, params.vlm_max_size)) { LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); + rejected_keys.insert(key); } } else if (key == "vlm_min_size") { if (!parse_strict_int(value, params.vlm_min_size)) { LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); + rejected_keys.insert(key); + } + } else if (key == "ref_image_size") { + bool valid = true; + if (value == "match") { + params.ref_image_size = RefImageSizeMode::MATCH; + } else if (value == "max") { + params.ref_image_size = RefImageSizeMode::MAX; + } else { + LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); + rejected_keys.insert(key); + valid = false; + } + if (valid && !sd_version_is_minimax_h3(version)) { + LOG_WARN("ref_image_size is only supported by MiniMax-H3 for now"); } } else if (key != "preset" && key != "vlm_size") { LOG_WARN("ignoring unknown reference image arg '%s'", key.c_str()); @@ -3316,6 +3343,7 @@ RefImageParams StableDiffusionGGML::resolve_ref_image_params(const char* ref_ima int vlm_size; if (!parse_strict_int(value, vlm_size)) { LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); + rejected_keys.insert(key); } else { LOG_INFO("vlm_size override: setting both min and max size to %ld", (long)vlm_size); params.vlm_min_size = vlm_size; @@ -3327,6 +3355,31 @@ RefImageParams StableDiffusionGGML::resolve_ref_image_params(const char* ref_ima if (params.force_ref_timestep_zero && !sd_version_is_krea2(version)) { LOG_WARN("force_ref_timestep_zero is only supported by Krea2 architecture for now"); } + if (sd_version_is_minimax_h3(version)) { + // H3 consumes references as VAE latents in its own packed sequence rather than + // through the generic conditioner/DiT reference path, so only sizing args apply. + // force_ref_timestep_zero is already covered by the Krea2-only warning above. + static const std::set h3_noop_keys = { + "preset", + "pass_to_vlm", + "pass_to_dit", + "ref_index_mode", + "vae_input_max_pixels", + "vlm_resize_mode", + "vlm_min_size", + "vlm_max_size", + "vlm_size", + }; + for (const auto& arg : parse_key_value_args(ref_image_args, "reference image args")) { + if (h3_noop_keys.count(arg.first) > 0 && rejected_keys.count(arg.first) == 0) { + LOG_WARN("reference image arg '%s' has no effect for MiniMax-H3", arg.first.c_str()); + } + } + + if (!params.resize_before_vae) { + LOG_WARN("MiniMax-H3 requires 32-pixel-aligned reference images and cannot skip resizing before VAE encoding; ignoring resize_before_vae=false"); + } + } return params; } diff --git a/src/pipeline/video.cpp b/src/pipeline/video.cpp index 2fcff4c9d4..bdef0902ea 100644 --- a/src/pipeline/video.cpp +++ b/src/pipeline/video.cpp @@ -4,6 +4,7 @@ #include #include #include +#include #include "conditioning/wan_audio.h" #include "core/rng.hpp" @@ -461,9 +462,33 @@ namespace sd::pipeline { return window; } + // ComfyUI-compatible reference sizing: per-axis rounding to 32 px keeps the reference + // divisible by VAE 16 x patch 2 without padding in the H3 graph. + static std::pair minimax_h3_reference_size(int source_w, + int source_h, + int target_w, + int target_h, + RefImageSizeMode mode) { + constexpr double kCanvasMultiple = 32.0; + constexpr double kRefImageShortEdge = 2048.0; + double scale = 1.0; + if (mode == RefImageSizeMode::MAX) { + scale = std::min(1.0, kRefImageShortEdge / std::min(source_w, source_h)); + } else { + double source_area = static_cast(source_w) * source_h; + double target_area = static_cast(target_w) * target_h; + scale = std::min(1.0, std::sqrt(target_area / source_area)); + } + auto align32 = [](double value) { + return std::max(32, static_cast(std::round(value / kCanvasMultiple)) * 32); + }; + return {align32(source_w * scale), align32(source_h * scale)}; + } + static std::optional prepare_video_generation_latents(StableDiffusionGGML* sd, const sd_vid_gen_params_t* sd_vid_gen_params, - GenerationRequest* request) { + GenerationRequest* request, + const RefImageParams& ref_image_params) { ImageGenerationLatents latents; int64_t prepare_start_ms = ggml_time_ms(); @@ -576,6 +601,7 @@ namespace sd::pipeline { sd_vid_gen_params->ref_videos_count, sd_vid_gen_params->ref_audios_count); + int64_t ref_image_tokens = 0; for (int i = 0; i < sd_vid_gen_params->ref_images_count; ++i) { auto image = ensure_image_tensor_channels( sd_image_to_tensor(sd_vid_gen_params->ref_images[i]), @@ -586,12 +612,15 @@ namespace sd::pipeline { } int source_w = static_cast(image.shape()[0]); int source_h = static_cast(image.shape()[1]); - double source_area = static_cast(source_w) * source_h; - double target_area = static_cast(request->width) * request->height; - double scale = std::min(1.0, std::sqrt(target_area / source_area)); - int width = std::max(32, static_cast(std::round(source_w * scale / 32.f)) * 32); - int height = std::max(32, static_cast(std::round(source_h * scale / 32.f)) * 32); - image = resize_reference(image, width, height); + auto [width, height] = + minimax_h3_reference_size(source_w, source_h, request->width, request->height, ref_image_params.ref_image_size); + LOG_INFO("MiniMax-H3 reference image %d: %dx%d -> %dx%d", + i + 1, + source_w, + source_h, + width, + height); + image = resize_reference(image, width, height); auto latent = sd->encode_first_stage(image); if (latent.empty()) { LOG_ERROR("failed to encode MiniMax-H3 reference image %d", i + 1); @@ -602,6 +631,10 @@ namespace sd::pipeline { latents.minimax_reference_blocks.push_back({MiniMaxH3ReferenceKind::IMAGE, video_index, -1}); + // Latents are patched by 2x2 in the DiT, so the token count is a quarter of the + // 16-pixel latent grid. + ref_image_tokens += static_cast(latents.ref_latents.back().shape()[0] / 2) * + (latents.ref_latents.back().shape()[1] / 2); MiniMaxH3PresentationItem item; item.kind = MiniMaxH3PresentationKind::IMAGE; item.frames.push_back(std::move(image)); @@ -711,6 +744,18 @@ namespace sd::pipeline { -1, encoded_index}); } + + if (ref_image_tokens > 0) { + int64_t target_tokens = sd->get_image_seq_len(request->width, request->height) / 4 * + sd->video_frames_to_latent_frames(request->frames); + if (target_tokens > 0 && ref_image_tokens > target_tokens * 4) { + LOG_WARN("MiniMax-H3 reference images add %" PRId64 + " tokens versus %" PRId64 + " for the generated video; sampling cost scales with every attended token", + ref_image_tokens, + target_tokens); + } + } } if (!has_references && (!start_image.empty() || !end_image.empty())) { @@ -1147,7 +1192,8 @@ namespace sd::pipeline { static std::optional prepare_video_generation_embeds(StableDiffusionGGML* sd, const sd_vid_gen_params_t* sd_vid_gen_params, const GenerationRequest& request, - const ImageGenerationLatents& latents) { + const ImageGenerationLatents& latents, + const RefImageParams& ref_image_params) { ConditionerRunnerEndOnExit conditioner_runner_end{sd->cond_stage_model.get()}; ImageGenerationEmbeds embeds; @@ -1157,6 +1203,7 @@ namespace sd::pipeline { condition_params.zero_out_masked = true; condition_params.ref_images = &latents.ref_images; condition_params.minimax_h3_references = &latents.minimax_presentation_refs; + condition_params.ref_image_params = ref_image_params; if (sd_version_is_lingbot_video(sd->version) || sd_version_is_minimax_h3(sd->version)) { condition_params.ref_image_params.vlm_resize_mode = RefImageResizeMode::AREA; } @@ -1526,6 +1573,7 @@ namespace sd::pipeline { img_gen_params.circular_x = sd_vid_gen_params->circular_x; img_gen_params.circular_y = sd_vid_gen_params->circular_y; img_gen_params.image_preprocess = sd_vid_gen_params->image_preprocess; + img_gen_params.ref_image_args = sd_vid_gen_params->ref_image_args; sd->animatediff_num_frames = n_frames; bool ok = generate_image(sd, &img_gen_params, frames_out, num_frames_out); @@ -1550,7 +1598,7 @@ namespace sd::pipeline { sd->reset_cancel_flag(); - const RefImageParams ref_image_params; + const RefImageParams ref_image_params = sd->resolve_ref_image_params(sd_vid_gen_params->ref_image_args); int64_t t0 = ggml_time_ms(); sd->vae_tiling_params = sd_vid_gen_params->vae_tiling_params; @@ -1589,7 +1637,7 @@ namespace sd::pipeline { sd->reset_generation_extensions(); SamplePlan plan(sd, sd_vid_gen_params, request); - auto latent_inputs_opt = prepare_video_generation_latents(sd, sd_vid_gen_params, &request); + auto latent_inputs_opt = prepare_video_generation_latents(sd, sd_vid_gen_params, &request, ref_image_params); if (!latent_inputs_opt.has_value()) { return false; } @@ -1598,7 +1646,8 @@ namespace sd::pipeline { auto embeds_opt = prepare_video_generation_embeds(sd, sd_vid_gen_params, request, - latents); + latents, + ref_image_params); if (!embeds_opt) { return false; } diff --git a/src/stable-diffusion.cpp b/src/stable-diffusion.cpp index d476f92d6b..c283520bb5 100644 --- a/src/stable-diffusion.cpp +++ b/src/stable-diffusion.cpp @@ -633,6 +633,7 @@ void sd_vid_gen_params_init(sd_vid_gen_params_t* sd_vid_gen_params) { sd_vid_gen_params->hires.custom_sigmas_count = 0; sd_vid_gen_params->circular_x = false; sd_vid_gen_params->circular_y = false; + sd_vid_gen_params->ref_image_args = ""; sd_cache_params_init(&sd_vid_gen_params->cache); } From 826a2ef778e8ef346355fd4b211853d648e628ee Mon Sep 17 00:00:00 2001 From: Philip Washington Sorst Date: Tue, 20 Oct 2026 20:35:33 +0000 Subject: [PATCH 2/2] docs: clarify ref_image_size rounding behavior The ref_image_size docs claimed both modes only downscale references. Neither mode scales a reference up beyond its limit, but the per-axis snap to the 32 px grid can still grow an axis by up to 16 px, so the statement was wrong: a 48 px axis becomes 64. Describe the actual behavior in docs/edit.md, docs/minimax_h3.md and the server API docs, and record the reasoning at the call site. Flooring max instead of rounding it would under-size in-limit references and diverge from ComfyUI's nodes_minimax_h3.py, which rounds in both branches, so the rounding is kept for both modes. Addresses the ref_image_size=max review comment on #2177. --- docs/edit.md | 2 +- docs/minimax_h3.md | 9 +++++---- examples/server/api.md | 2 +- src/pipeline/video.cpp | 4 +++- 4 files changed, 10 insertions(+), 7 deletions(-) diff --git a/docs/edit.md b/docs/edit.md index c3b0c5212c..631b91a14b 100644 --- a/docs/edit.md +++ b/docs/edit.md @@ -78,7 +78,7 @@ If presets are insufficient, you can manually configure the following parameters | `vlm_max_size` | int | Maximum VLM input size; interpreted according to `vlm_resize_mode`. | Integer | | `vlm_min_size` | int | Minimum VLM input size; interpreted according to `vlm_resize_mode`. | Integer | | `vlm_size` | int | Shortcut to set both VLM min and max size to the same value. | Integer | -| `ref_image_size` | string | MiniMax-H3 reference image sizing before VAE encoding. `match` scales each reference down to the generation's pixel area; `max` uses a 2048 px short edge. Both only downscale. | `match`, `max` | +| `ref_image_size` | string | MiniMax-H3 reference image sizing before VAE encoding. `match` scales each reference down to the generation's pixel area; `max` uses a 2048 px short edge. Only references above the limit are downscaled; each axis is then snapped to the nearest 32 px multiple (at most +16 px, minimum 32 px). | `match`, `max` | `ref_image_size` applies to `vid_gen` with MiniMax-H3 Ref2VA. Reference latents are attended to at every sampling step, so `max` can be several times slower. diff --git a/docs/minimax_h3.md b/docs/minimax_h3.md index c7811a94fa..7a48e3f1be 100644 --- a/docs/minimax_h3.md +++ b/docs/minimax_h3.md @@ -85,10 +85,11 @@ pixel area exceeds the requested generation canvas. Pass `--ref-image-args "ref_image_size=max"` (or `ref_image_size` in `sd_vid_gen_params_t.ref_image_args`) to size references by a 2048 px short edge -instead, which preserves more identity detail at a higher sampling cost. Both -modes only downscale: references at or below the limit keep their size. Either -way the per-axis result is rounded to the 32-pixel grid, so a dimension can grow -by up to 16 pixels, and very small references are widened to 32 pixels. +instead, which preserves more identity detail at a higher sampling cost. Neither +mode scales a reference up beyond its limit, but each axis is snapped to the +nearest 32-pixel multiple, so an axis can grow by up to 16 pixels past the scaled +size (a 48-pixel axis becomes 64) and references are never narrower than 32 +pixels. `resize_before_vae=false` is not supported here; MiniMax-H3 keeps references on the 32-pixel grid and reports a warning. diff --git a/examples/server/api.md b/examples/server/api.md index 429aa4fe6a..4dc03251dc 100644 --- a/examples/server/api.md +++ b/examples/server/api.md @@ -1492,7 +1492,7 @@ Example: - Images retain request order and correspond to ``, ``, and so on in the prompt. - For MiniMax-H3, non-empty `ref_images` cannot be combined with `init_image` or `end_image`. A conflicting API request fails during generation; the WebUI checks this before submission. - MiniMax-H3 does not support `control_frames`; leave that array empty. -- `ref_image_args` applies to `vid_gen`; MiniMax-H3 honors `ref_image_size` there (`match` default, `max` for a 2048 px short edge). Both modes only downscale references. +- `ref_image_args` applies to `vid_gen`; MiniMax-H3 honors `ref_image_size` there (`match` default, `max` for a 2048 px short edge). Only references above the limit are downscaled; each axis is then snapped to the nearest 32 px multiple. - Other video model families currently ignore `ref_images`. - `features_by_mode.vid_gen` does not currently advertise `ref_images`; its absence is not an indication that MiniMax-H3 reference images are unsupported. diff --git a/src/pipeline/video.cpp b/src/pipeline/video.cpp index bdef0902ea..5309a101b8 100644 --- a/src/pipeline/video.cpp +++ b/src/pipeline/video.cpp @@ -463,7 +463,9 @@ namespace sd::pipeline { } // ComfyUI-compatible reference sizing: per-axis rounding to 32 px keeps the reference - // divisible by VAE 16 x patch 2 without padding in the H3 graph. + // divisible by VAE 16 x patch 2 without padding in the H3 graph. Both modes round rather + // than floor: flooring would under-size in-limit references and diverge from the + // upstream node, which rounds in both branches. static std::pair minimax_h3_reference_size(int source_w, int source_h, int target_w,