From f6538ce410d9b6391f06ad44e6c690d1786adf50 Mon Sep 17 00:00:00 2001 From: Marcel Rebro Date: Thu, 17 Sep 2026 13:19:31 +0200 Subject: [PATCH 1/4] docs: repoint docs-tests baseline at moved account settings doc Closes #2951. #2919 moved sources/platform/account/settings.md to account/settings/index.md, but pages.json and the stored account-settings baseline still referenced the flat path, so every assertion's doc back-reference pointed at a file that no longer exists and the doc-side integrity suite failed on all 16 of them. Fifteen assertions needed only the path plus a +6 line shift. The sixteenth, settings-tab-side-menu, lost its justifying sentence when #2919 rewrote the Account section, so it is re-quoted against the new intro ("In the left-side panel, go to **Settings**."). Same target, same route, so the UI-side check is unchanged. No Console UI claim changes here, and there is no genuine UI drift behind #2951: the filed report's "32 failing / 48 of 92 passed" is 16 doc-side failures counted twice by the CI retry (#2911), against 22 integrity + 26 UI passes and the usual 12 fixture skips. Verified against staging: 64 passed, 0 failed, 12 skipped. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01FSj1mTUbMKgmu2JoGJLSKD --- docs-tests/assertions/account-settings.json | 36 ++++++++++----------- docs-tests/pages.json | 2 +- 2 files changed, 19 insertions(+), 19 deletions(-) diff --git a/docs-tests/assertions/account-settings.json b/docs-tests/assertions/account-settings.json index 2a29818631..af80c9ff1c 100644 --- a/docs-tests/assertions/account-settings.json +++ b/docs-tests/assertions/account-settings.json @@ -1,5 +1,5 @@ { - "source_file": "sources/platform/account/settings.md", + "source_file": "sources/platform/account/settings/index.md", "assertions": [ { "id": "settings-tab-side-menu", @@ -7,8 +7,8 @@ "target": "Settings", "at": "/settings", "page_context": "Side menu (global navigation) leading to the Account settings page", - "source_quote": "By clicking the **Settings** tab on the side menu, you will be presented with an Account page", - "source_line": 11, + "source_quote": "1. In the left-side panel, go to **Settings**.", + "source_line": 12, "needs_auth": true }, { @@ -18,7 +18,7 @@ "at": "/settings/security", "page_context": "Settings page tabs", "source_quote": "The **Login & Privacy** tab (**Security & Privacy** for organization accounts) contains sensitive settings", - "source_line": 21, + "source_line": 27, "needs_auth": true }, { @@ -27,7 +27,7 @@ "target": "Security & Privacy", "page_context": "Settings page tabs, label shown instead of Login & Privacy for organization accounts", "source_quote": "(**Security & Privacy** for organization accounts)", - "source_line": 21, + "source_line": 27, "needs_auth": true }, { @@ -37,7 +37,7 @@ "at": "/settings/security", "page_context": "Login & Privacy tab > Session section", "source_quote": "In the **Session** section, you can adjust the session configuration.", - "source_line": 35, + "source_line": 41, "needs_auth": true }, { @@ -47,7 +47,7 @@ "at": "/settings/integrations", "page_context": "Settings page tabs", "source_quote": "The **API & Integrations** tab provides essential tools for accessing the Apify platform programmatically.", - "source_line": 39, + "source_line": 45, "needs_auth": true }, { @@ -57,7 +57,7 @@ "at": "/settings/integrations", "page_context": "API & Integrations tab", "source_quote": "you can manage your **API tokens**, which are necessary for using the", - "source_line": 39, + "source_line": 45, "needs_auth": true }, { @@ -67,7 +67,7 @@ "at": "/settings/integrations", "page_context": "API & Integrations tab", "source_quote": "**account-level integrations**, and **Actor OAuth accounts**.", - "source_line": 39, + "source_line": 45, "needs_auth": true }, { @@ -77,7 +77,7 @@ "at": "/settings/integrations", "page_context": "API & Integrations tab", "source_quote": "and **Actor OAuth accounts**.", - "source_line": 39, + "source_line": 45, "needs_auth": true }, { @@ -87,7 +87,7 @@ "at": "/settings/integrations", "page_context": "API & Integrations tab > MCP connectors section", "source_quote": "The **MCP connectors** section lets you authorize third-party MCP servers", - "source_line": 43, + "source_line": 49, "needs_auth": true }, { @@ -97,7 +97,7 @@ "at": "/settings/integrations", "page_context": "API & Integrations tab > MCP connectors section", "source_quote": "Open **Settings > API & Integrations > MCP connectors** and select **Add connector**.", - "source_line": 47, + "source_line": 53, "needs_auth": true }, { @@ -106,7 +106,7 @@ "target": "Own OAuth client", "page_context": "MCP connector creation modal, authentication method selection, API & Integrations tab", "source_quote": "In the connector creation modal, select **Own OAuth client** and provide:", - "source_line": 79, + "source_line": 85, "needs_auth": true }, { @@ -115,7 +115,7 @@ "target": "Authorize", "page_context": "MCP connector card, API & Integrations tab (shown when reauthorization is needed)", "source_quote": "the **Authorize** button appears on the connector card.", - "source_line": 89, + "source_line": 95, "needs_auth": true }, { @@ -124,7 +124,7 @@ "target": "Delete", "page_context": "MCP connector card, API & Integrations tab", "source_quote": "Click **Delete** on the connector card.", - "source_line": 93, + "source_line": 99, "needs_auth": true }, { @@ -134,7 +134,7 @@ "at": "/settings/organizations", "page_context": "Settings page tabs", "source_quote": "The **Organizations** tab is where you can view your accounts' current organizations", - "source_line": 97, + "source_line": 103, "needs_auth": true }, { @@ -144,7 +144,7 @@ "at": "/settings/notifications", "page_context": "Settings page tabs", "source_quote": "The **Notifications** tab allows you to customize your notification preferences.", - "source_line": 101, + "source_line": 107, "needs_auth": true }, { @@ -153,7 +153,7 @@ "target": "Referrals", "page_context": "Settings page tabs (no dedicated route mapping known)", "source_quote": "The **Referrals** tab lets you share Apify with others and earn rewards.", - "source_line": 105, + "source_line": 111, "needs_auth": true } ] diff --git a/docs-tests/pages.json b/docs-tests/pages.json index 5e35fe8805..10373ab552 100644 --- a/docs-tests/pages.json +++ b/docs-tests/pages.json @@ -1,7 +1,7 @@ { "pages": [ "sources/platform/account/console.md", - "sources/platform/account/settings.md", + "sources/platform/account/settings/index.md", "sources/platform/account/billing/index.md" ] } From 103b6878d0da8c007a334dac085a4bef217e0181 Mon Sep 17 00:00:00 2001 From: Marcel Rebro Date: Thu, 17 Sep 2026 16:03:00 +0200 Subject: [PATCH 2/4] docs: extend docs-tests coverage to storage and the rest of the account section MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Widens the docs-tests baseline from 3 pages / 38 assertions to 11 pages / 87 assertions, adding the storage section and the account pages that were never covered. New pages: - storage: overview, datasets, key-value stores, request queues - account: two-factor authentication, promo codes, general resource access, organization usage Harness fixes this surfaced: - scripts/extract.sh derived the slug with `s#\.md$##`, so an .mdx page kept its extension and an index page became `account-settings-index` rather than overwriting `account-settings` — a re-extraction would have written a second baseline file beside the existing one and both suites would have loaded them. Now strips .mdx/.md and a trailing /index. The expression uses two plain substitutions instead of `\.mdx\?$`, which matches nothing under BSD sed (macOS, where the extractor is run). - prompts/extract-system.md pointed the storage tabs at `/storage?tab=datasets`, `?tab=keyValueStores` and `?tab=requestQueues`. Console serves `/storage/datasets`, `/storage/key-value-stores` and `/storage/request-queues`; the query-param URLs return 200 but redirect to the datasets tab, so every re-extraction would have regenerated an `at` route that lands on the wrong page. Route table corrected. The same stale URLs appear in the storage docs themselves, where they send readers to the wrong tab. That is genuine drift and a prose fix, so it is left to its own PR rather than bundled here. Curation notes for the new baseline: - Left-nav items (Storage, Proxy) are anchors, not buttons, and there is no element_link kind — dropped rather than mis-typed. - "Include unnamed storages" is a checkbox label, not a button. - 2FA Disable/Recovery settings describe the enabled state; the seeded user has 2FA off, so they are fixture-gated (no `at`) instead of failing. - Assertions already covered by the account baseline were not duplicated, and generic targets ("ID", "Actions") that match arbitrary page text were removed. Verified against staging: 132 passed, 0 failed, 42 skipped, 0 issues. The skips are element assertions with no landing route — the pre-existing detail-page-fixture gap, now larger because storage detail pages carry many of the documented labels. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01FSj1mTUbMKgmu2JoGJLSKD --- docs-tests/README.md | 22 +++- .../account-billing-promo-codes.json | 35 ++++++ ...collaboration-general-resource-access.json | 34 ++++++ ...collaboration-organization-how_to_use.json | 14 +++ ...nt-settings-two-factor-authentication.json | 106 ++++++++++++++++++ docs-tests/assertions/storage-dataset.json | 96 ++++++++++++++++ .../assertions/storage-key_value_store.json | 60 ++++++++++ .../assertions/storage-request_queue.json | 60 ++++++++++ docs-tests/assertions/storage.json | 87 ++++++++++++++ docs-tests/pages.json | 10 +- docs-tests/prompts/extract-system.md | 6 +- docs-tests/scripts/extract.sh | 16 ++- 12 files changed, 533 insertions(+), 13 deletions(-) create mode 100644 docs-tests/assertions/account-billing-promo-codes.json create mode 100644 docs-tests/assertions/account-collaboration-general-resource-access.json create mode 100644 docs-tests/assertions/account-collaboration-organization-how_to_use.json create mode 100644 docs-tests/assertions/account-settings-two-factor-authentication.json create mode 100644 docs-tests/assertions/storage-dataset.json create mode 100644 docs-tests/assertions/storage-key_value_store.json create mode 100644 docs-tests/assertions/storage-request_queue.json create mode 100644 docs-tests/assertions/storage.json diff --git a/docs-tests/README.md b/docs-tests/README.md index 11f6a14028..1a5365b755 100644 --- a/docs-tests/README.md +++ b/docs-tests/README.md @@ -46,7 +46,7 @@ produce and a human commits. | Kind | Checks | | ---------------- | ------------------------------------------------------------------ | -| `route` | Documented path is reachable (HTTP < 400) | +| `route` | Documented path is reachable (HTTP < 400) — status only, see below | | `element_tab` | Documented tab label exists on the page named in `at` | | `element_button` | Documented button label exists on the page named in `at` | | `element_text` | Documented heading/label/field is visible on the page named in `at` | @@ -181,11 +181,14 @@ throwaway branch). ## Known gaps (deferred) -- **Coverage is a starting slice.** `pages.json` covers the account section - (`account/console` dashboard, `account/settings`, `account/billing`) — routes - and landing-page elements. (`console/store.md` was dropped: it has no bold UI - element labels to test.) Widening to more pages is a follow-up: add to - `pages.json`, re-extract, review, commit. +- **Coverage is the Console-surface slice.** `pages.json` covers the account + section (dashboard, settings, 2FA, billing, promo codes, general resource + access, organization usage) and storage (overview, datasets, key-value + stores, request queues) — routes and landing-page elements. (`console/store.md` + was dropped: it has no bold UI element labels to test.) Widening further is a + follow-up: add to `pages.json`, re-extract, review, commit. Actors, tasks and + schedules are the obvious next section, but most of their labels sit on detail + pages or inside modals — see the two gaps below. - **Detail-page fixtures.** Assertions about Actor-detail, Schedule-detail, etc. need a known fixture to navigate to. The runner currently *skips* element assertions with no `at` route — surfacing the gap without false negatives. @@ -195,6 +198,13 @@ throwaway branch). Actors/…) are a Console-wide check, not a per-page claim — not modeled yet. - **Multi-step flows.** The schema only supports atomic claims (one navigate-then-check). "Click X, then Y, then Z" sequences are not modeled. +- **`route` checks are status-only.** A `route` assertion passes on any + HTTP < 400, so a documented URL that *resolves* but no longer lands where the + doc says still passes. Console's storage tabs are the live example: the + documented `/storage?tab=keyValueStores` returns 200 but redirects to + `/storage/datasets?tab=keyValueStores`, silently showing the Datasets tab. + Only the `element_*` assertions pinned to that page catch it. Prefer pairing a + route with at least one element claim from the page it should land on. - **Surface mismatch.** Some docs describe the public marketing site (e.g. `apify.com/store`) while the harness tests the Console (`/store`); the two can use different labels, so those element claims may not map. Check the surface a diff --git a/docs-tests/assertions/account-billing-promo-codes.json b/docs-tests/assertions/account-billing-promo-codes.json new file mode 100644 index 0000000000..fd1f2ff24f --- /dev/null +++ b/docs-tests/assertions/account-billing-promo-codes.json @@ -0,0 +1,35 @@ +{ + "source_file": "sources/platform/account/billing/promo-codes.mdx", + "assertions": [ + { + "id": "subscription-tab", + "kind": "element_tab", + "target": "Subscription", + "at": "/billing/subscription", + "page_context": "Billing page", + "source_quote": "Select the **Subscription** tab.", + "source_line": 16, + "needs_auth": true + }, + { + "id": "special-offers-section", + "kind": "element_text", + "target": "Special offers", + "at": "/billing/subscription", + "page_context": "Billing page > Subscription tab", + "source_quote": "In the **Special offers** section, select **Add a promo code**.", + "source_line": 17, + "needs_auth": true + }, + { + "id": "add-promo-code-button", + "kind": "element_button", + "target": "Add a promo code", + "at": "/billing/subscription", + "page_context": "Billing page > Subscription tab > Special offers section", + "source_quote": "In the **Special offers** section, select **Add a promo code**.", + "source_line": 17, + "needs_auth": true + } + ] +} diff --git a/docs-tests/assertions/account-collaboration-general-resource-access.json b/docs-tests/assertions/account-collaboration-general-resource-access.json new file mode 100644 index 0000000000..fc8f4d4341 --- /dev/null +++ b/docs-tests/assertions/account-collaboration-general-resource-access.json @@ -0,0 +1,34 @@ +{ + "source_file": "sources/platform/account/collaboration/general-resource-access.md", + "assertions": [ + { + "id": "general-resource-access-setting-name", + "kind": "element_text", + "target": "General resource access", + "at": "/settings/security", + "page_context": "Settings page > Login & Privacy tab (Security & Privacy for organization accounts) > account resource access setting", + "source_quote": "**General resource access** is an account setting that defines the default access policy at the account level.", + "source_line": 13, + "needs_auth": true + }, + { + "id": "restricted-default-option", + "kind": "element_text", + "target": "Restricted", + "page_context": "Settings page > Login & Privacy tab > General resource access setting options", + "source_quote": "**Restricted (default)**: With this setting, only signed-in users with an explicit access to the resources can access them.", + "source_line": 16, + "needs_auth": true + }, + { + "id": "share-run-data-with-developers-section", + "kind": "element_text", + "target": "Share run data with developers", + "at": "/settings/security", + "page_context": "Settings page > Login & Privacy tab > Share run data with developers section", + "source_quote": "can be enabled under **Settings → Login & Privacy** in the **Share run data with developers** section", + "source_line": 82, + "needs_auth": true + } + ] +} diff --git a/docs-tests/assertions/account-collaboration-organization-how_to_use.json b/docs-tests/assertions/account-collaboration-organization-how_to_use.json new file mode 100644 index 0000000000..d1483a86e3 --- /dev/null +++ b/docs-tests/assertions/account-collaboration-organization-how_to_use.json @@ -0,0 +1,14 @@ +{ + "source_file": "sources/platform/account/collaboration/organization/how_to_use.md", + "assertions": [ + { + "id": "settings-organizations-route", + "kind": "route", + "target": "/settings/organizations", + "page_context": "Account page > Organizations tab, listing organizations the user is a member of", + "source_quote": "from the **Organizations** tab on your [account page](https://console.apify.com/settings/organizations)", + "source_line": 29, + "needs_auth": true + } + ] +} diff --git a/docs-tests/assertions/account-settings-two-factor-authentication.json b/docs-tests/assertions/account-settings-two-factor-authentication.json new file mode 100644 index 0000000000..735c57b09b --- /dev/null +++ b/docs-tests/assertions/account-settings-two-factor-authentication.json @@ -0,0 +1,106 @@ +{ + "source_file": "sources/platform/account/settings/two-factor-authentication.md", + "assertions": [ + { + "id": "two-factor-authentication-section", + "kind": "element_text", + "target": "Two-factor authentication", + "at": "/settings/security", + "page_context": "Settings > Login & Privacy tab", + "source_quote": "In **Two-factor authentication**, select **Enable**.", + "source_line": 29, + "needs_auth": true + }, + { + "id": "enable-2fa-button", + "kind": "element_button", + "target": "Enable", + "at": "/settings/security", + "page_context": "Settings > Login & Privacy tab, Two-factor authentication section", + "source_quote": "In **Two-factor authentication**, select **Enable**.", + "source_line": 29, + "needs_auth": true + }, + { + "id": "verify-code-field", + "kind": "element_text", + "target": "Verify the code from the app", + "page_context": "2FA setup modal shown after clicking Enable on Settings > Login & Privacy", + "source_quote": "In the **Verify the code from the app** field, enter the 6-digit code generated by your app.", + "source_line": 39, + "needs_auth": true + }, + { + "id": "continue-button-setup", + "kind": "element_button", + "target": "Continue", + "page_context": "2FA setup modal, after entering the verification code", + "source_quote": "Select **Continue**.", + "source_line": 40, + "needs_auth": true + }, + { + "id": "setup-key-link", + "kind": "element_button", + "target": "Setup key", + "page_context": "2FA setup modal, manual setup without QR code", + "source_quote": "In the modal window, select the **Setup key** link.", + "source_line": 47, + "needs_auth": true + }, + { + "id": "phone-number-field", + "kind": "element_text", + "target": "Phone number", + "page_context": "2FA recovery information configuration step", + "source_quote": "**Phone number**. Apify will only use your phone number to verify your identity during the recovery process.", + "source_line": 67, + "needs_auth": true + }, + { + "id": "personal-information-field", + "kind": "element_text", + "target": "Personal information", + "page_context": "2FA recovery information configuration step", + "source_quote": "**Personal information**. Make sure the information you provide is secure and easy to remember.", + "source_line": 68, + "needs_auth": true + }, + { + "id": "continue-button-recovery-info", + "kind": "element_button", + "target": "Continue", + "page_context": "2FA recovery information configuration step, after filling phone number and personal information", + "source_quote": "Once you complete both fields, select **Continue**.", + "source_line": 70, + "needs_auth": true + }, + { + "id": "recovery-settings-button", + "kind": "element_button", + "target": "Recovery settings", + "page_context": "Settings > Login & Privacy tab, Two-factor authentication section", + "source_quote": "In **Two-factor authentication**, select **Recovery settings**.", + "source_line": 96, + "needs_auth": true + }, + { + "id": "disable-2fa-button", + "kind": "element_button", + "target": "Disable", + "page_context": "Settings > Login & Privacy tab, Two-factor authentication section", + "source_quote": "In **Two-factor authentication**, select **Disable**.", + "source_line": 106, + "needs_auth": true + }, + { + "id": "remove-app-button", + "kind": "element_button", + "target": "Remove app", + "page_context": "2FA disable confirmation modal", + "source_quote": "Select **Remove app**.", + "source_line": 108, + "needs_auth": true + } + ] +} diff --git a/docs-tests/assertions/storage-dataset.json b/docs-tests/assertions/storage-dataset.json new file mode 100644 index 0000000000..c7ffbda248 --- /dev/null +++ b/docs-tests/assertions/storage-dataset.json @@ -0,0 +1,96 @@ +{ + "source_file": "sources/platform/storage/dataset/index.md", + "assertions": [ + { + "id": "storage-section-route", + "kind": "route", + "target": "/storage", + "page_context": "Storage section of Apify Console", + "source_quote": "you can view your datasets in the [Storage](https://console.apify.com/storage) section under the [Datasets](https://console.apify.com/storage?tab=datasets) tab.", + "source_line": 46, + "needs_auth": true + }, + { + "id": "datasets-tab-route", + "kind": "route", + "target": "/storage?tab=datasets", + "page_context": "Storage page, Datasets tab", + "source_quote": "under the [Datasets](https://console.apify.com/storage?tab=datasets) tab.", + "source_line": 46, + "needs_auth": true + }, + { + "id": "dataset-id-column", + "kind": "element_text", + "target": "Dataset ID", + "at": "/storage/datasets", + "page_context": "Storage page > Datasets tab, list of datasets", + "source_quote": "Click on its **Dataset ID** to open the dataset detail page.", + "source_line": 52, + "needs_auth": true + }, + { + "id": "dataset-table-view-tab", + "kind": "element_tab", + "target": "Table", + "page_context": "Dataset detail page, data view selector", + "source_quote": "Browse the data in **Table** or **JSON** view.", + "source_line": 53, + "needs_auth": true + }, + { + "id": "dataset-json-view-tab", + "kind": "element_tab", + "target": "JSON", + "page_context": "Dataset detail page, data view selector", + "source_quote": "Browse the data in **Table** or **JSON** view.", + "source_line": 53, + "needs_auth": true + }, + { + "id": "dataset-export-button", + "kind": "element_button", + "target": "Export", + "page_context": "Dataset detail page", + "source_quote": "Click **Export** to download the data in your preferred format.", + "source_line": 54, + "needs_auth": true + }, + { + "id": "dataset-actions-menu", + "kind": "element_button", + "target": "Actions", + "page_context": "Dataset detail page, menu for renaming and access rights", + "source_quote": "Utilize the **Actions** menu to modify the dataset's name", + "source_line": 56, + "needs_auth": true + }, + { + "id": "dataset-api-button", + "kind": "element_button", + "target": "API", + "page_context": "Dataset detail page", + "source_quote": "The **API** button allows you to explore and test the dataset's [API endpoints](/api/v2/storage-datasets).", + "source_line": 56, + "needs_auth": true + }, + { + "id": "clean-items-option", + "kind": "element_text", + "target": "Clean items", + "page_context": "Dataset export option in Apify Console (Datasets tab / dataset detail page)", + "source_quote": "can be downloaded from the [Apify Console](https://console.apify.com/storage?tab=datasets) using the **Clean items** option.", + "source_line": 339, + "needs_auth": true + }, + { + "id": "dataset-share-button", + "kind": "element_button", + "target": "Share", + "page_context": "Dataset detail page, sharing access rights", + "source_quote": "through the **Share** button under the **Actions** menu.", + "source_line": 421, + "needs_auth": true + } + ] +} diff --git a/docs-tests/assertions/storage-key_value_store.json b/docs-tests/assertions/storage-key_value_store.json new file mode 100644 index 0000000000..5bed3ec817 --- /dev/null +++ b/docs-tests/assertions/storage-key_value_store.json @@ -0,0 +1,60 @@ +{ + "source_file": "sources/platform/storage/key_value_store/index.md", + "assertions": [ + { + "id": "storage-overview-route", + "kind": "route", + "target": "/storage", + "page_context": "Storage section of Apify Console, reached from the key-value store docs page", + "source_quote": "in the [Storage](https://console.apify.com/storage) section", + "source_line": 37, + "needs_auth": true + }, + { + "id": "key-value-stores-tab-route", + "kind": "route", + "target": "/storage?tab=keyValueStores", + "page_context": "Key-value stores tab within the Storage section of Apify Console", + "source_quote": "under the [Key-value stores](https://console.apify.com/storage?tab=keyValueStores) tab.", + "source_line": 37, + "needs_auth": true + }, + { + "id": "store-id-link", + "kind": "element_text", + "target": "Store ID", + "at": "/storage/key-value-stores", + "page_context": "Key-value stores list page, Storage section > Key-value stores tab", + "source_quote": "To view a key-value store's content, click on its **Store ID**.", + "source_line": 41, + "needs_auth": true + }, + { + "id": "api-button-kv-detail", + "kind": "element_button", + "target": "API", + "page_context": "Key-value store detail page (Storage > Key-value stores > individual store)", + "source_quote": "Click on the **API** button to view and test a store's", + "source_line": 42, + "needs_auth": true + }, + { + "id": "share-button-kv-detail", + "kind": "element_button", + "target": "Share", + "page_context": "Key-value store detail page, under the Actions menu", + "source_quote": "through the **Share** button under the **Actions** menu.", + "source_line": 279, + "needs_auth": true + }, + { + "id": "actions-menu-kv-detail", + "kind": "element_button", + "target": "Actions", + "page_context": "Key-value store detail page, menu containing Share and rename options", + "source_quote": "through the **Share** button under the **Actions** menu.", + "source_line": 279, + "needs_auth": true + } + ] +} diff --git a/docs-tests/assertions/storage-request_queue.json b/docs-tests/assertions/storage-request_queue.json new file mode 100644 index 0000000000..85695c337f --- /dev/null +++ b/docs-tests/assertions/storage-request_queue.json @@ -0,0 +1,60 @@ +{ + "source_file": "sources/platform/storage/request_queue.md", + "assertions": [ + { + "id": "storage-section-route", + "kind": "route", + "target": "/storage", + "page_context": "Apify Console > Storage overview section", + "source_quote": "you can view your request queues in the [Storage](https://console.apify.com/storage) section under the [Request queues]", + "source_line": 35, + "needs_auth": true + }, + { + "id": "request-queues-tab-route", + "kind": "route", + "target": "/storage?tab=requestQueues", + "page_context": "Apify Console > Storage > Request queues tab", + "source_quote": "under the [Request queues](https://console.apify.com/storage?tab=requestQueues) tab", + "source_line": 35, + "needs_auth": true + }, + { + "id": "queue-id-column-list", + "kind": "element_text", + "target": "Queue ID", + "at": "/storage/request-queues", + "page_context": "Storage > Request queues tab list, identifying column/link to open a queue", + "source_quote": "To view a request queue, click on its **Queue ID**.", + "source_line": 39, + "needs_auth": true + }, + { + "id": "queue-actions-menu", + "kind": "element_button", + "target": "Actions", + "page_context": "Request queue detail page, menu for renaming and sharing the queue", + "source_quote": "Under the **Actions** menu, you can rename your queue's name", + "source_line": 40, + "needs_auth": true + }, + { + "id": "queue-share-button", + "kind": "element_button", + "target": "Share", + "page_context": "Request queue detail page, Actions menu option to manage access rights", + "source_quote": "and [access rights](/account/collaboration) using the **Share** button.", + "source_line": 41, + "needs_auth": true + }, + { + "id": "queue-api-button", + "kind": "element_button", + "target": "API", + "page_context": "Request queue detail page, button to view and test API endpoints", + "source_quote": "Click on the **API** button to view and test a queue's [API endpoints]", + "source_line": 42, + "needs_auth": true + } + ] +} diff --git a/docs-tests/assertions/storage.json b/docs-tests/assertions/storage.json new file mode 100644 index 0000000000..79318a1e3f --- /dev/null +++ b/docs-tests/assertions/storage.json @@ -0,0 +1,87 @@ +{ + "source_file": "sources/platform/storage/index.md", + "assertions": [ + { + "id": "storage-overview-route", + "kind": "route", + "target": "/storage", + "page_context": "Storage overview page in Apify Console", + "source_quote": "To view your storages in [Apify Console](https://console.apify.com/storage):", + "source_line": 42, + "needs_auth": true + }, + { + "id": "storage-detail-api-button", + "kind": "element_button", + "target": "API", + "page_context": "Storage detail page (dataset/key-value store/request queue), top right corner", + "source_quote": "To view the related API endpoints, select **API** in the top right corner.", + "source_line": 48, + "needs_auth": true + }, + { + "id": "include-unnamed-storages-label", + "kind": "element_text", + "target": "Include unnamed storages", + "at": "/storage", + "page_context": "Storage overview page, toggle for unnamed storages", + "source_quote": "Use the **Include unnamed storages** checkbox to either display or hide unnamed storages.", + "source_line": 54, + "needs_auth": true + }, + { + "id": "storage-actions-menu", + "kind": "element_button", + "target": "Actions", + "page_context": "Storage detail page, menu for renaming/sharing/deleting a store", + "source_quote": "To rename a store, open the **Actions** menu and select **Rename**.", + "source_line": 58, + "needs_auth": true + }, + { + "id": "storage-rename-option", + "kind": "element_button", + "target": "Rename", + "page_context": "Storage detail page, Actions menu option", + "source_quote": "To rename a store, open the **Actions** menu and select **Rename**.", + "source_line": 58, + "needs_auth": true + }, + { + "id": "storage-share-option", + "kind": "element_button", + "target": "Share", + "page_context": "Storage detail page, Actions menu option for sharing a store", + "source_quote": "To share a storage, select **Share** in the **Actions** menu and provide an email, username, or user ID.", + "source_line": 60, + "needs_auth": true + }, + { + "id": "run-dataset-tab", + "kind": "element_tab", + "target": "Dataset", + "page_context": "Actor run details page, storage type tab", + "source_quote": "Open your run's details and select the **Dataset**, **Key-value store**, or **Request queue** tab as appropriate.", + "source_line": 111, + "needs_auth": true + }, + { + "id": "run-kv-store-tab", + "kind": "element_tab", + "target": "Key-value store", + "page_context": "Actor run details page, storage type tab", + "source_quote": "Open your run's details and select the **Dataset**, **Key-value store**, or **Request queue** tab as appropriate.", + "source_line": 111, + "needs_auth": true + }, + { + "id": "run-request-queue-tab", + "kind": "element_tab", + "target": "Request queue", + "page_context": "Actor run details page, storage type tab", + "source_quote": "Open your run's details and select the **Dataset**, **Key-value store**, or **Request queue** tab as appropriate.", + "source_line": 111, + "needs_auth": true + } + ] +} diff --git a/docs-tests/pages.json b/docs-tests/pages.json index 10373ab552..62d2d918de 100644 --- a/docs-tests/pages.json +++ b/docs-tests/pages.json @@ -2,6 +2,14 @@ "pages": [ "sources/platform/account/console.md", "sources/platform/account/settings/index.md", - "sources/platform/account/billing/index.md" + "sources/platform/account/settings/two-factor-authentication.md", + "sources/platform/account/billing/index.md", + "sources/platform/account/billing/promo-codes.mdx", + "sources/platform/account/collaboration/general-resource-access.md", + "sources/platform/account/collaboration/organization/how_to_use.md", + "sources/platform/storage/index.md", + "sources/platform/storage/dataset/index.md", + "sources/platform/storage/key_value_store/index.md", + "sources/platform/storage/request_queue.md" ] } diff --git a/docs-tests/prompts/extract-system.md b/docs-tests/prompts/extract-system.md index 33decb1bde..08b56020b1 100644 --- a/docs-tests/prompts/extract-system.md +++ b/docs-tests/prompts/extract-system.md @@ -57,9 +57,9 @@ map the page_context to a known route. | Actor analytics | `/actors/insights/analytics` | | Store | `/store` | | Storage overview | `/storage` | -| Datasets tab | `/storage?tab=datasets` | -| Key-value stores tab | `/storage?tab=keyValueStores` | -| Request queues tab | `/storage?tab=requestQueues` | +| Datasets tab | `/storage/datasets` | +| Key-value stores tab | `/storage/key-value-stores` | +| Request queues tab | `/storage/request-queues` | | Schedules | `/schedules` | | Proxy | `/proxy` | | Proxy groups | `/proxy/groups` | diff --git a/docs-tests/scripts/extract.sh b/docs-tests/scripts/extract.sh index 15e1406aea..45c2ca1dd7 100755 --- a/docs-tests/scripts/extract.sh +++ b/docs-tests/scripts/extract.sh @@ -23,9 +23,19 @@ if [[ ! -f "$ABS_DOC" ]]; then exit 1 fi -# Slug for the stored file: strip the sources/platform/ prefix and .md suffix, -# turn path separators into dashes. e.g. console/settings.md -> console-settings -SLUG=$(printf '%s' "$DOC_PATH" | sed -e 's#^sources/platform/##' -e 's#\.md$##' -e 's#[/ ]#-#g') +# Slug for the stored file: strip the sources/platform/ prefix, the .md/.mdx +# suffix and a trailing /index, then turn path separators into dashes. +# e.g. account/console.md -> account-console +# account/billing/index.md -> account-billing (not account-billing-index) +# account/billing/promo.mdx -> account-billing-promo +# Dropping /index keeps the slug stable when a flat page becomes a directory, +# so a re-extraction overwrites the existing baseline instead of writing a +# second file beside it (which the suites would then both load). +# `.mdx` before `.md`, as two plain expressions: BSD sed (macOS, where TWs run +# the extractor) has no `\?` in a basic regex, so `\.mdx\?$` silently matches +# nothing there while working under GNU sed. +SLUG=$(printf '%s' "$DOC_PATH" \ + | sed -e 's#^sources/platform/##' -e 's#\.mdx$##' -e 's#\.md$##' -e 's#/index$##' -e 's#[/ ]#-#g') OUTPUT_FILE="assertions/$SLUG.json" mkdir -p assertions From d5a1f7639d158441baa8e1d622ceca7bc891de4e Mon Sep 17 00:00:00 2001 From: Marcel Rebro Date: Thu, 17 Sep 2026 16:12:50 +0200 Subject: [PATCH 3/4] docs: assign at-routes to 7 previously-skipped docs-tests assertions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Folds in the work from docs/docs-tests-detail-fixtures, which was pushed 2026-07-28 and never opened as a PR. That branch predates both the IA v3 path refresh and this coverage expansion, so it conflicts with all three account baselines; the at-routes are reapplied here rather than rebased. Gives a landing route to the dashboard sections (/), the billing Pricing and Invoices tabs, and the Settings Referrals tab, so they evaluate instead of skipping. All 7 verified against staging — including the billing routes, which were never checked when the original branch was written. Skips drop from 42 to 35; passes rise from 132 to 139, 0 failures. The remaining 35 are element assertions with no atomic landing route: storage and Actor detail pages, connector modals, and the 2FA enabled-state controls. Those need the seeded-user fixtures from #2671, not an at-route. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01FSj1mTUbMKgmu2JoGJLSKD --- docs-tests/assertions/account-billing.json | 6 ++++-- docs-tests/assertions/account-console.json | 12 ++++++++---- docs-tests/assertions/account-settings.json | 3 ++- 3 files changed, 14 insertions(+), 7 deletions(-) diff --git a/docs-tests/assertions/account-billing.json b/docs-tests/assertions/account-billing.json index 201d4cf821..7cc109c40d 100644 --- a/docs-tests/assertions/account-billing.json +++ b/docs-tests/assertions/account-billing.json @@ -117,7 +117,8 @@ "page_context": "Billing page > Pricing tab", "source_quote": "The **Pricing** tab offers a way to quickly check all unit pricing for various platform services", "source_line": 61, - "needs_auth": true + "needs_auth": true, + "at": "/billing/pricing" }, { "id": "invoices-tab", @@ -126,7 +127,8 @@ "page_context": "Billing page > Invoices tab", "source_quote": "The **Invoices** tab is where you can find your current and previous invoices for Apify platform usage.", "source_line": 67, - "needs_auth": true + "needs_auth": true, + "at": "/billing/invoices" }, { "id": "limits-tab", diff --git a/docs-tests/assertions/account-console.json b/docs-tests/assertions/account-console.json index 95e0f7fa3b..6120ca3c65 100644 --- a/docs-tests/assertions/account-console.json +++ b/docs-tests/assertions/account-console.json @@ -44,7 +44,8 @@ "page_context": "Landing page after signing in to Apify Console", "source_quote": "you land on the **Dashboard** section that provides an overview of your account", "source_line": 46, - "needs_auth": true + "needs_auth": true, + "at": "/" }, { "id": "dashboard-recently-viewed", @@ -53,7 +54,8 @@ "page_context": "Dashboard overview section", "source_quote": "**Recently viewed**: Displays Actors you have recently accessed.", "source_line": 48, - "needs_auth": true + "needs_auth": true, + "at": "/" }, { "id": "dashboard-suggested-actors", @@ -62,7 +64,8 @@ "page_context": "Dashboard overview section", "source_quote": "**Suggested Actors for you**: Recommends Actors that might interest you", "source_line": 50, - "needs_auth": true + "needs_auth": true, + "at": "/" }, { "id": "dashboard-actor-runs", @@ -71,7 +74,8 @@ "page_context": "Dashboard overview section", "source_quote": "**Actor runs**: Shows your recent Actor runs, as well as scheduled runs and tasks.", "source_line": 52, - "needs_auth": true + "needs_auth": true, + "at": "/" } ] } diff --git a/docs-tests/assertions/account-settings.json b/docs-tests/assertions/account-settings.json index af80c9ff1c..d87fbe0d29 100644 --- a/docs-tests/assertions/account-settings.json +++ b/docs-tests/assertions/account-settings.json @@ -154,7 +154,8 @@ "page_context": "Settings page tabs (no dedicated route mapping known)", "source_quote": "The **Referrals** tab lets you share Apify with others and earn rewards.", "source_line": 111, - "needs_auth": true + "needs_auth": true, + "at": "/settings/referrals" } ] } From 114ddb812911e81f7fb57e31abfb9aef117194c2 Mon Sep 17 00:00:00 2001 From: Marcel Rebro Date: Thu, 17 Sep 2026 16:41:18 +0200 Subject: [PATCH 4/4] docs: record third-party integration pages as out of scope for docs-tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit sources/platform/integrations/** documents third-party interfaces, not Console. Those pages carry the highest bold-label density in the docs, so any scan looking for testable UI labels ranks them first — and every one of them is a dead end: the harness can't reach those products, their UIs change on someone else's schedule, and a failure would be unactionable. Writes the rule into the "adding a new test case" guide next to the existing marketing-site exclusion, and notes that this caps what the harness can ever cover — about half of sources/platform/ — so coverage is not misread as a share of all docs. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01FSj1mTUbMKgmu2JoGJLSKD --- docs-tests/README.md | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/docs-tests/README.md b/docs-tests/README.md index 1a5365b755..a8a502d1b1 100644 --- a/docs-tests/README.md +++ b/docs-tests/README.md @@ -114,8 +114,21 @@ adding a single claim. 1. Add the doc's repo-relative path to **`pages.json`**, e.g. `"sources/platform/account/notifications.md"`. Only add pages that document - the **Console** UI — not the public marketing site (see the surface-mismatch - gap below). + the **Console** UI. + + Two whole categories are out of scope, permanently: + + - **`sources/platform/integrations/**`** — these pages document *third-party* + interfaces (HubSpot, n8n, Make, Power Automate, …), not Console. They have + the highest bold-label density in the docs, so a naive scan ranks them + first; ignore them. The harness has no access to those products, their UIs + change on someone else's schedule, and a failure there would be + unactionable. This is roughly half of `sources/platform/`, and it is a + permanent ceiling on what this harness can cover — worth stating plainly + whenever coverage is reported as a number. + - **The public marketing site** (`apify.com/store` and friends) — the harness + points at Console, and the two use different labels (see the + surface-mismatch gap below). 2. Run `pnpm extract sources/platform/account/notifications.md` (one page) or `pnpm extract:all` (everything). This writes `assertions/account-notifications.json`. 3. **Review the diff** — this is the real work. The extractor is a first draft;