Skip to content

docs: stable nodepool tolerations and observability resource profiles - #191

Open
Guimove wants to merge 3 commits into
mainfrom
docs/karpenter-stable-toleration-and-metrics-profiles
Open

Guimove wants to merge 3 commits into
mainfrom
docs/karpenter-stable-toleration-and-metrics-profiles

Conversation

@Guimove

@Guimove Guimove commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Two documentation gaps surfaced by a customer support ticket (Pylon #4898). Both changes are corrections and additions to existing pages, no new page and no navigation change, so docs.json is untouched.

1. deployment.affinity.node.required and the stable node pool

docs/configuration/service-advanced-settings.mdx

The reference entry is badged for Application, Container, Cronjob and Job, and offers an "AWS — Karpenter on-demand stable pool" example:

{ "karpenter.sh/capacity-type": "on-demand", "karpenter.sh/nodepool": "stable" }

The stable Karpenter NodePool carries a nodepool/stable:NoSchedule taint, and the existing <Warning> only covers the case where no node matches the labels. The engine injects the matching toleration for Applications and Containers, but Jobs, Cronjobs and Terraform services have no stable branch: they only ever build tolerations for the gpu and cronjob pools. Following the documented example on a Job therefore applies a hard node affinity toward a tainted node with no toleration, and the pods stay Pending.

The new subsection documents:

  • the taint, and the fact that this setting only controls the affinity half
  • automatic toleration on Applications and Containers, and automatic pinning for container-mode databases
  • a <Warning> that Jobs and Cronjobs are not covered
  • an <Info> on the automatic stable targeting when min_instances is 1 or the service has storage
  • a pointer to the existing Helm section, which already documents both halves correctly

2. Observability resource profiles

docs/configuration/integrations/observability/qovery-observe.mdx

metrics_parameters.configuration.resource_profile accepts LOW, NORMAL and HIGH, and no page described what each level sets. Customers were being asked to change a cluster-wide observability setting blind, with no way to tell whether it degraded retention or scrape interval. It does neither: the profile only moves CPU and memory requests and limits.

Adds a per-component table for the three levels next to the existing Data Retention section, an <Info> clarifying that retention is identical on every profile, and a <Warning> that LOW drops Prometheus and the Thanos Compactor to 1Gi, which can cause OOMKills during compaction on clusters with many series.

Review notes

The figures need a maintainer's confirmation before merge. AGENTS.md lists the OpenAPI spec, the CLI repo and existing documentation pages as the sources of truth. The resource profile values are not in any of them; they were read from the engine source, and the OpenAPI spec only carries the enum. The one figure that could be cross-checked, Thanos retention at 15/30/30 days, matches the Data Retention section already published on the same page.

The Job and Cronjob behaviour is arguably a bug rather than intended design. This PR documents it as it stands today. If the engine is fixed so that Jobs and Cronjobs get the toleration like Applications and Containers, the <Warning> should be removed rather than kept.

Local render not verified. The Mintlify CLI is not available in this environment. Tag balance, heading depth, internal link format and the #data-retention anchor were checked manually.

Opened as a draft for those reasons.


Summary by cubic

Fixes two documentation gaps on existing pages found through a customer support ticket (Pylon #4898): the stable Karpenter node pool toleration behavior and the observability resource profiles.

  • service-advanced-settings.mdx documents the nodepool/stable:NoSchedule taint and that deployment.affinity.node.required only controls the affinity half.
  • Qovery adds the matching toleration for Applications and Containers and targets stable automatically when min_instances is 1 or storage is attached.
  • Jobs and Cronjobs do not receive the toleration and stay Pending on stable; the optional cronjob pool has its own taint and toleration.
  • qovery-observe.mdx lists per-component CPU and memory requests and limits for the LOW, NORMAL, and HIGH resource profiles.
  • Profiles change only resource reservations; retention, scrape interval, replica counts, alerting, and log and event collection stay the same.
  • LOW drops Prometheus and the Thanos Compactor to 1Gi, which can cause OOMKills on clusters with many series.

Review notes

  • Resource profile figures come from the engine source and need a maintainer's confirmation before merge.
  • Job and Cronjob behavior is documented as-is; if the engine later adds the toleration, remove the warning.
  • Local render was not verified; tag balance, heading depth, internal link format, and the #data-retention anchor were checked manually.

Written for commit 9176414. Summary will update on new commits.

Review in cubic

@mintlify

mintlify Bot commented Sep 17, 2026

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
qovery 🟢 Ready View Preview Sep 17, 2026, 9:47 AM

💡 Tip: Enable Automations to automatically generate PRs for you.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 17, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
qovery-doc-mintlify-proxy 9176414 Sep 17 2026, 09:46 AM

… profiles

The deployment.affinity.node.required reference showed a `stable` node pool
example badged for Applications, Containers, Jobs and Cronjobs, without
mentioning that the pool carries a nodepool/stable:NoSchedule taint. Qovery
injects the matching toleration for Applications and Containers only, so
following that example on a Job or Cronjob leaves pods in Pending. Documents
which service types are covered, the automatic targeting rules, and points
Helm users to the existing Helm section.

The observability resource_profile levels (LOW, NORMAL, HIGH) were not
documented anywhere, leaving no way to know what each level sets before
changing a cluster-wide setting. Adds the per-component CPU and memory table
and states what the profile does not change, notably retention.

Reported through customer support (Pylon #4898).
@Guimove
Guimove force-pushed the docs/karpenter-stable-toleration-and-metrics-profiles branch from 65f40c4 to c516bac Compare September 17, 2026 09:12
@Guimove
Guimove marked this pull request as ready for review September 17, 2026 09:22
@Guimove
Guimove requested a review from a team September 17, 2026 09:22

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread docs/configuration/service-advanced-settings.mdx Outdated
Comment thread docs/configuration/service-advanced-settings.mdx Outdated
Comment thread docs/configuration/integrations/observability/qovery-observe.mdx Outdated
Comment thread docs/configuration/integrations/observability/qovery-observe.mdx Outdated
Qualify the cronjob node pool as optional and stop restating its behavior:
the `stable` toleration gap is the subject here, so point to the existing
Dedicated Cronjob Node Pool section instead of duplicating it.

State the EKS with Karpenter condition explicitly on the automatic stable
placement rule.

Stop repeating the memory figures in the LOW warning; refer to the table so
the values live in one place.
Loki, Promtail and the Qovery Event Logger are listed on the same page as
part of the stack but are sized independently, so name them explicitly
rather than letting "the observability components" cover them.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant