fix: keep knowledge base uploads working when the embedding service caps inputs

[Mola-maker]

Problem

Uploading a document to a knowledge base could fail with the message quoted in #10109:

存储失败：文本块已生成，但写入知识库索引时出错。

Nothing is wrong with storage. The dashboard always sends batch_size=32 (DocumentsTab.vue → knowledge_base_service.py), while DashScope text-embedding-v3/v4 — one of AstrBot's default embedding templates — reject any request carrying more than 10 inputs:

HTTP 400  InvalidParameter  Value error, batch size is invalid, it should not be larger than 10.: input.contents

That error is deterministic, so the retry loop in EmbeddingProvider.get_embeddings_batch can never recover: each batch spends its 3 retries, the upload aborts, and the knowledge base ends up with doc_count=0 chunk_count=0.

Because the upload runs as a background task, the browser never sees an HTTP error — only the misleading log line, which is what made this hard to diagnose.

Reproduction (before)

Driving the real KBHelper.upload_document path with a stub embedding backend that emulates DashScope's 10-input cap, at the dashboard's batch_size=32:

[Core] [WARN] [knowledge_base.kb_helper:499]: 上传文档失败: 存储失败：文本块已生成，但写入知识库索引时出错。
>>> KB after uploads: doc_count=0 chunk_count=0

Fix

Three layers, all on the provider request boundary.

1. Declared per-request input limit. EmbeddingProvider.get_max_batch_size() (default None = unknown) is consulted once in get_embeddings_batch, and the caller's batch_size is clamped to it — the doomed request is never built, at zero cost. DashScopeEmbeddingProvider declares the per-model limit (v1/v2 → 25, v3/v4 → 10, multimodal → a conservative 10). OpenAIEmbeddingProvider infers it from the configured base URL host, the same way it already infers embedding_dimensions, since DashScope is commonly reached through its OpenAI-compatible endpoint. Any other gateway whose limit the adapter cannot know can set embedding_max_batch_items in its provider config (schema entry added; combined with min(), so config can only lower a known limit).

2. Adaptive splitting as the runtime safety net. A declared limit is only as good as the table behind it, so a rejection that actually blames the number of inputs is handled where it happens: the batch is split and its pieces retried, instead of re-sending the same oversized request. When the provider states a limit (should not be larger than 10), that number is used exactly — 32 inputs become 10/10/10/2 rather than repeated halving — and cached on the provider for the rest of the upload and later ones. When it does not, the batch halves. Splitting is triggered only when an HTTP status in {400, 413, 414, 422} (or no status at all) agrees with a narrow size-semantics pattern and no token/context/rate-limit pattern is present, so unrelated failures keep their existing behaviour. Deterministic 4xx failures also no longer burn three exponential-backoff retries. Result ordering, the concurrency semaphore, the progress callback and the aggregate 有 N 个批次处理失败 error are unchanged.

3. Honest error attribution. FaissVecDB.insert_batch called the provider outside any try, so the raw error escaped unlabelled and kb_helper relabelled it as a storage failure — the source of the reported message. That call is now wrapped as KnowledgeBaseUploadError(stage="embedding", ...) with the real cause, so any residual embedding failure (bad key, unknown model, network) is legible instead of being blamed on storage.

Reproduction (after)

Same script, same batch_size=32, same 10-input cap:

########## UPLOAD big.xlsx via dashboard defaults (batch_size=32) ##########
  -> OK chunk_count=100
########## UPLOAD big.docx via dashboard defaults (batch_size=32) ##########
  -> OK chunk_count=36
>>> KB after uploads: doc_count=2 chunk_count=136

And a provider that fails for an unrelated reason now reports the real one instead of the storage message:

上传文档失败: 向量化失败：调用嵌入模型时出错。原因：有 1 个批次处理失败: 批次 0 处理失败: HTTP 401: Incorrect API key provided: [REDACTED]
stage='embedding'

Credential redaction

Carrying the provider's text through the new stage="embedding" error also meant carrying whatever the provider quoted from the request, API key included — into the upload log (kb_helper logs details) and into the dashboard's failure list. The text now goes through the existing redact_sensitive_text at both sites that carry it (vec_db.py, and kb_helper's storage-stage handler, which passes the same cause through the same log line). Non-secret context is kept, so the message stays actionable; details["cause"] stays complete, only user_message is bounded to 300 chars.

That redactor had a gap this exposed: its sk- pattern only accepted an alphanumeric body, so dash-segmented vendor keys (sk-proj-..., sk-ant-api03-...) passed through untouched. It is widened here, and its named-credential pattern now also accepts a quoted value (api_key="..."), which the =-style form previously missed. Short stand-ins such as sk-test are still left readable on purpose.

Tests

New tests/unit/test_embedding_batch_split.py (37 cases): declared-clamp and config-declared limits, split-and-retry with order preservation, exact use of a stated limit, halving when none is stated, learned-limit reuse across calls, a limit learned without a stated value not being persisted, retryable failures still retried, a single rejected text surfacing the provider error, classifier false-positive exclusions (context length, max_tokens, rate limit, auth, 5xx), a real openai.BadRequestError shape, per-model DashScope limits, host sniffing, the OpenAI-compatible DashScope endpoint end-to-end, an end-to-end FaissVecDB.insert_batch of 40 chunks against a 10-input provider, and redaction of a key-bearing provider error.

New tests/unit/test_error_redaction.py (18 cases) covers the redactor: the vendor key formats, named credentials in JSON/query/header form, safe_error, and short placeholders deliberately left alone.

Verification:

• pytest tests/unit and pytest tests --ignore=tests/unit --ignore=tests/integration — 1325 + 2041 passed. The 11 failures in those runs (grep tool limits, dashboard config update, KOOK client, file-URI helpers, ...) reproduce identically on a clean HEAD worktree and are unrelated to this change.
• ruff format --check . and ruff check . — clean.

Not in this PR

The dashboard discards the per-file reason the server already returns in result.failed[].error (DocumentsTab.vue → only counts are shown), so a failed upload still surfaces no reason in the UI. Making that visible means a new UI surface plus new locale keys in four languages, so it belongs in a separate UX PR.

Fixes #10109

🤖 Generated with Claude Code

Summary by Sourcery

Prevent knowledge-base upload failures caused by embedding providers that cap the number of inputs per request.

New Features:

• Support provider-specific maximum input counts through adapter detection or the embedding_max_batch_items configuration option.

Bug Fixes:

• Keep knowledge-base uploads working when embedding services reject oversized batches by clamping declared limits and adaptively splitting rejected requests.
• Report embedding-provider failures as embedding errors instead of misleading storage failures.

Enhancements:

• Reuse provider-reported batch limits, preserve embedding order and progress reporting, and avoid retrying deterministic client-side batch-size errors.
• Redact API keys and other credentials from embedding failure messages and upload error details.

Tests:

• Add comprehensive coverage for declared limits, adaptive splitting, failure classification, provider-specific detection, end-to-end vector-store uploads, error attribution, and credential redaction.

[sourcery-ai]

Hey - I've found 1 issue

Prompt for AI Agents

Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="astrbot/core/db/vec_db/faiss_impl/vec_db.py" line_range="142-149" />
<code_context>
+            # Attach the embedding stage here: otherwise the raw provider error
+            # escapes unlabelled and gets reported downstream as a storage
+            # failure, hiding the real cause (see kb_helper's stage mapping).
+            cause = str(exc).strip() or type(exc).__name__
+            if len(cause) > 300:
+                cause = cause[:300] + "…"
+            raise KnowledgeBaseUploadError(
+                stage="embedding",
+                user_message=f"向量化失败：调用嵌入模型时出错。原因：{cause}",
+                details={
+                    "cause": str(exc),
+                    "error_type": type(exc).__name__,
+                    "provider": type(self.embedding_provider).__name__,
</code_context>
<issue_to_address>
**🚨 issue (security):** The raw embedding-provider exception is copied into both the user-facing `user_message` and `details["cause"]` without redaction. Provider authentication errors can include the API key, as shown by the reported `Incorrect API key provided: sk-xxx` output, so credentials are exposed in upload logs or returned error data.

**Triggers:** When an embedding service includes credentials or other sensitive request data in its exception text.

**Suggested fix:** Sanitize provider exception text before putting it in `user_message` or `details`, and preserve the full exception only in a properly protected diagnostic log.
</issue_to_address>

Sourcery assessment

Needs a human reviewer. 1 finding to address first, and if the new batching or error-classification logic is wrong, an upload could fail after some vectors or documents have already been persisted, leaving a bounded partial knowledge base that needs cleanup or a rerun; reverting will not remove those records. The change does not move money or expose access, and the remaining impact is normally repairable.

Blocking findings: astrbot/core/db/vec_db/faiss_impl/vec_db.py:149

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

[Mola-maker]

Valid — and the fix uncovered a gap in the redactor we already have for this.

redact_sensitive_text (astrbot/core/utils/error_redaction.py) is the repo's primitive for exactly this, but it could not catch the reported shape: its sk- pattern only accepted an alphanumeric body, so dash-segmented vendor keys (sk-proj-… from OpenAI, sk-ant-api03-… from Anthropic, DashScope's own sk-…) passed through untouched. Extended that pattern, and made the named-credential pattern accept a quoted value so api_key="…" is covered too (the =-style form previously missed it).

Both sites that carry the provider's text are now redacted:

• astrbot/core/db/vec_db/faiss_impl/vec_db.py:149 — the flagged line. The user-facing summary is still bounded to 300 chars; details["cause"] keeps the redacted error in full for diagnostics.
• astrbot/core/knowledge_base/kb_helper.py:435 — the storage-stage handler passes the same cause through the same kb_helper.py:499 log line, so it had the same exposure.

The full exception still travels on the raise … from exc chain for debugging, but nothing logs it: kb_helper logs KnowledgeBaseUploadError at logger.warning without exc_info.

Verified end to end — the same script that previously printed the raw key now prints:

上传文档失败: 向量化失败：调用嵌入模型时出错。原因：有 1 个批次处理失败: 批次 0 处理失败: HTTP 401: Incorrect API key provided: [REDACTED]
stage='embedding'
details={'cause': '... Incorrect API key provided: [REDACTED]', ...}

tests/unit/test_error_redaction.py covers the redactor (key formats, named credentials, short placeholders like sk-test deliberately left readable), plus key-bearing assertions through FaissVecDB.insert_batch and through upload_document's storage stage.