Add ADetailer support to server image generation

docs/adetailer.md

@@ -1,9 +1,10 @@
 # ADetailer
 
 `sd-cli` can run a YOLOv8 object detector on an existing or newly generated
-image and perform a cropped inpaint pass for every detected object. The first
-implementation supports YOLOv8 detection checkpoints. YOLOv8 segmentation and
-MediaPipe models are not supported yet.
+image, and `sd-server` can run it after image generation. Both perform a cropped
+inpaint pass for every detected object. The implementation supports YOLOv8
+detection checkpoints. YOLOv8 segmentation and MediaPipe models are not
+supported yet.
 
 ## Convert a detector
 
@@ -68,6 +69,36 @@ ADetailer can also run automatically after normal image generation:
   --extra-ad-args "confidence=0.3,denoising_strength=0.4,inpaint_width=512,inpaint_height=512"
 ```
 
+The same options can be supplied when starting `sd-server`. When `--ad-model` is set, its detector runs after OpenAI-compatible and native asynchronous image generation. SDAPI requests enable ADetailer through `alwayson_scripts`, as described below. If no object is detected, the original image is returned unchanged. RGBA output keeps its original alpha channel, including when detected regions are repaired. Detector initialization or detail-pass failures are reported as request failures.
+
+Server image-generation requests limit base, hires, and detail-pass dimensions to 4096 pixels per side. ADetailer step overrides are limited to 1–100; `steps=0` in `extra_ad_args` keeps inheriting the main generation steps. These limits also apply to inherited dimensions and `extra_ad_args` overrides. Requests exceeding the limits return a parameter error before generation. CLI limits are unchanged.
+
+### SDAPI requests
+
+`/sdapi/v1/txt2img` and `/sdapi/v1/img2img` accept the WebUI ADetailer argument format:
+
+```json
+{
+  "prompt": "portrait photo",
+  "alwayson_scripts": {
+    "ADetailer": {
+      "args": [true, false, {
+        "ad_model": "face_yolov8n.pt",
+        "ad_prompt": "[PROMPT], detailed face",
+        "ad_use_steps": true,
+        "ad_steps": 20
+      }]
+    }
+  }
+}
+```
+
+The detector must be configured at server startup with `--ad-model`. `ad_model` must match that file's basename; the original `.pt` basename is also accepted for a converted detector. Requests cannot load arbitrary detector paths. Omitting the script, setting its first argument to `false`, selecting `"None"`, or setting `ad_tab_enable` to `false` disables detailing. Dictionary-only argument arrays are also accepted. Multiple enabled dictionaries run in order using the configured detector.
+
+Supported settings include prompts, confidence, mask filtering by area or confidence, mask count and area ratios, dilation/erosion, offsets, mask merging/inversion, blur, denoising strength, masked or full-image inpainting, padding, and optional overrides for inpaint dimensions, steps, CFG, sampler/scheduler, and CLIP skip. The corresponding `ad_use_*` flags control whether overrides apply. Otherwise dimensions, steps, CFG, sampler, scheduler, and CLIP skip inherit the main request. Denoising strength defaults to `0.4`. Pixel mask settings are limited to the larger output dimension, with a minimum limit of 32 and a maximum of 16384 pixels.
+
+Model/VAE switching, non-default noise multipliers, face restoration, ControlNet, and skipping the initial img2img pass are not supported; active requests for these features return an error. Unknown settings and invalid active values also return errors. During detailing, `/sdapi/v1/progress` remains active with an `adetailer` job; its percentage and preview stay fixed until detailing finishes.
+
 An empty ADetailer prompt inherits the main prompt. `[PROMPT]` inserts the main
 prompt, `[SEP]` assigns different prompts to consecutive masks, and `[SKIP]`
 skips the corresponding mask.
@@ -82,6 +113,7 @@ All settings other than the detector path and prompts are passed through
 | `nms` | `0.45` | NMS IoU threshold |
 | `max_detections` | `100` | Maximum detections retained after NMS |
 | `mask_k_largest` | `0` | Keep only the largest K masks; zero keeps all |
+| `mask_filter_method` | `area` | Rank retained masks by `area` or `confidence` |
 | `mask_min_ratio` | `0` | Minimum bbox area relative to the image |
 | `mask_max_ratio` | `1` | Maximum bbox area relative to the image |
 | `dilate_erode` | `4` | Positive values dilate; negative values erode |
@@ -90,6 +122,7 @@ All settings other than the detector path and prompts are passed through
 | `merge_masks`, `invert_mask` | `false` | Boolean alternatives to `mask_mode` |
 | `mask_blur` | `4` | Final composite feather radius |
 | `inpaint_padding` | `32` | Padding around the detected region |
+| `inpaint_only_masked` | `true` | Crop to the detected region; `false` uses the full image with the detection mask |
 | `inpaint_width`, `inpaint_height` | mode-specific | `512x512` after generation; input/output size in `adetailer` mode |
 | `denoising_strength` | mode-specific | `0.4` after generation; inherits `--strength` in `adetailer` mode |
 | `steps` | `0` | Detail steps; zero inherits the main generation |

examples/common/common.cpp

@@ -2219,7 +2219,7 @@ bool SDGenerationParams::from_json_str(
         if (target_custom_sigmas != nullptr &&
             sample_json.contains("custom_sigmas") &&
             sample_json["custom_sigmas"].is_array() &&
-            !sample_json["custom_sigmas"].empty()) { // empty [] = "not specified", keep server/CLI default
+            !sample_json["custom_sigmas"].empty()) {  // empty [] = "not specified", keep server/CLI default
             *target_custom_sigmas = sample_json["custom_sigmas"].get<std::vector<float>>();
         }
         if (sample_json.contains("sample_method") && sample_json["sample_method"].is_string()) {

examples/server/async_jobs.cpp

@@ -244,6 +244,10 @@ bool execute_img_gen_job(ServerRuntime& runtime,
         }
         clear_preview_callback();
         results.adopt(raw_results, num_results);
+        if (!results.empty() && !apply_server_adetailer(runtime, job.img_gen, params, results)) {
+            error_message = "ADetailer failed";
+            return false;
+        }
     }
 
     const int num_results = results.count();

examples/server/routes_openai.cpp

@@ -75,7 +75,7 @@ static bool build_openai_generation_request(const httplib::Request& req,
         error_message = "invalid params";
         return false;
     }
-    return true;
+    return validate_server_img_gen_request(runtime, request, error_message);
 }
 
 static bool build_openai_edit_request(const httplib::Request& req,
@@ -158,22 +158,22 @@ static bool build_openai_edit_request(const httplib::Request& req,
     request.gen_params.height      = height;
     request.gen_params.batch_count = n;
 
-    std::string sd_cpp_extra_args_str          = extract_and_remove_sd_cpp_extra_args(request.gen_params.prompt);
-    const json extra_args                      = sd_cpp_extra_args_str.empty()
-                                                     ? json::object()
-                                                     : json::parse(sd_cpp_extra_args_str, nullptr, false);
-    const bool has_extra_init_image_override   = extra_args.is_object() && extra_args.contains("init_image");
-    const bool has_extra_ref_images_override   = extra_args.is_object() && extra_args.contains("ref_images");
-    const bool has_extra_strength_override     = extra_args.is_object() &&
-                                                 extra_args.contains("strength") &&
-                                                 extra_args["strength"].is_number();
-    const bool has_extra_mask_image            = extra_args.is_object() &&
-                                                 extra_args.contains("mask_image") &&
-                                                 !extra_args["mask_image"].is_null();
+    std::string sd_cpp_extra_args_str        = extract_and_remove_sd_cpp_extra_args(request.gen_params.prompt);
+    const json extra_args                    = sd_cpp_extra_args_str.empty()
+                                                   ? json::object()
+                                                   : json::parse(sd_cpp_extra_args_str, nullptr, false);
+    const bool has_extra_init_image_override = extra_args.is_object() && extra_args.contains("init_image");
+    const bool has_extra_ref_images_override = extra_args.is_object() && extra_args.contains("ref_images");
+    const bool has_extra_strength_override   = extra_args.is_object() &&
+                                             extra_args.contains("strength") &&
+                                             extra_args["strength"].is_number();
+    const bool has_extra_mask_image = extra_args.is_object() &&
+                                      extra_args.contains("mask_image") &&
+                                      !extra_args["mask_image"].is_null();
     const bool supports_ref_image_conditioning = sd_ctx_supports_ref_image_conditioning(runtime.sd_ctx);
     const bool may_need_uploaded_init_image    = !has_extra_init_image_override &&
-                                                 (!supports_ref_image_conditioning || !mask_bytes.empty() ||
-                                                  has_extra_mask_image || has_extra_strength_override || has_extra_ref_images_override);
+                                              (!supports_ref_image_conditioning || !mask_bytes.empty() ||
+                                               has_extra_mask_image || has_extra_strength_override || has_extra_ref_images_override);
     SDImageOwner uploaded_image_for_init;
     bool uploaded_image_was_first_reference = false;
     for (auto& bytes : images_bytes) {
@@ -252,7 +252,7 @@ static bool build_openai_edit_request(const httplib::Request& req,
         return false;
     }
 
-    return true;
+    return validate_server_img_gen_request(runtime, request, error_message);
 }
 
 static bool execute_sync_img_gen_request(ServerRuntime& runtime,
@@ -270,6 +270,10 @@ static bool execute_sync_img_gen_request(ServerRuntime& runtime,
             num_results = 0;
         }
         results.adopt(raw_results, num_results);
+        if (!results.empty() && !apply_server_adetailer(runtime, request, img_gen_params, results)) {
+            error_message = "ADetailer failed";
+            return false;
+        }
     }
 
     if (results.empty()) {