Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 31 additions & 8 deletions docs-tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ produce and a human commits.

| Kind | Checks |
| ---------------- | ------------------------------------------------------------------ |
| `route` | Documented path is reachable (HTTP < 400) |
| `route` | Documented path is reachable (HTTP < 400) — status only, see below |
| `element_tab` | Documented tab label exists on the page named in `at` |
| `element_button` | Documented button label exists on the page named in `at` |
| `element_text` | Documented heading/label/field is visible on the page named in `at` |
Expand Down Expand Up @@ -114,8 +114,21 @@ adding a single claim.

1. Add the doc's repo-relative path to **`pages.json`**, e.g.
`"sources/platform/account/notifications.md"`. Only add pages that document
the **Console** UI — not the public marketing site (see the surface-mismatch
gap below).
the **Console** UI.

Two whole categories are out of scope, permanently:

- **`sources/platform/integrations/**`** — these pages document *third-party*
interfaces (HubSpot, n8n, Make, Power Automate, …), not Console. They have
the highest bold-label density in the docs, so a naive scan ranks them
first; ignore them. The harness has no access to those products, their UIs
change on someone else's schedule, and a failure there would be
unactionable. This is roughly half of `sources/platform/`, and it is a
permanent ceiling on what this harness can cover — worth stating plainly
whenever coverage is reported as a number.
- **The public marketing site** (`apify.com/store` and friends) — the harness
points at Console, and the two use different labels (see the
surface-mismatch gap below).
2. Run `pnpm extract sources/platform/account/notifications.md` (one page) or
`pnpm extract:all` (everything). This writes `assertions/account-notifications.json`.
3. **Review the diff** — this is the real work. The extractor is a first draft;
Expand Down Expand Up @@ -181,11 +194,14 @@ throwaway branch).

## Known gaps (deferred)

- **Coverage is a starting slice.** `pages.json` covers the account section
(`account/console` dashboard, `account/settings`, `account/billing`) — routes
and landing-page elements. (`console/store.md` was dropped: it has no bold UI
element labels to test.) Widening to more pages is a follow-up: add to
`pages.json`, re-extract, review, commit.
- **Coverage is the Console-surface slice.** `pages.json` covers the account
section (dashboard, settings, 2FA, billing, promo codes, general resource
access, organization usage) and storage (overview, datasets, key-value
stores, request queues) — routes and landing-page elements. (`console/store.md`
was dropped: it has no bold UI element labels to test.) Widening further is a
follow-up: add to `pages.json`, re-extract, review, commit. Actors, tasks and
schedules are the obvious next section, but most of their labels sit on detail
pages or inside modals — see the two gaps below.
- **Detail-page fixtures.** Assertions about Actor-detail, Schedule-detail, etc.
need a known fixture to navigate to. The runner currently *skips* element
assertions with no `at` route — surfacing the gap without false negatives.
Expand All @@ -195,6 +211,13 @@ throwaway branch).
Actors/…) are a Console-wide check, not a per-page claim — not modeled yet.
- **Multi-step flows.** The schema only supports atomic claims (one
navigate-then-check). "Click X, then Y, then Z" sequences are not modeled.
- **`route` checks are status-only.** A `route` assertion passes on any
HTTP < 400, so a documented URL that *resolves* but no longer lands where the
doc says still passes. Console's storage tabs are the live example: the
documented `/storage?tab=keyValueStores` returns 200 but redirects to
`/storage/datasets?tab=keyValueStores`, silently showing the Datasets tab.
Only the `element_*` assertions pinned to that page catch it. Prefer pairing a
route with at least one element claim from the page it should land on.
- **Surface mismatch.** Some docs describe the public marketing site (e.g.
`apify.com/store`) while the harness tests the Console (`/store`); the two can
use different labels, so those element claims may not map. Check the surface a
Expand Down
35 changes: 35 additions & 0 deletions docs-tests/assertions/account-billing-promo-codes.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
{
"source_file": "sources/platform/account/billing/promo-codes.mdx",
"assertions": [
{
"id": "subscription-tab",
"kind": "element_tab",
"target": "Subscription",
"at": "/billing/subscription",
"page_context": "Billing page",
"source_quote": "Select the **Subscription** tab.",
"source_line": 16,
"needs_auth": true
},
{
"id": "special-offers-section",
"kind": "element_text",
"target": "Special offers",
"at": "/billing/subscription",
"page_context": "Billing page > Subscription tab",
"source_quote": "In the **Special offers** section, select **Add a promo code**.",
"source_line": 17,
"needs_auth": true
},
{
"id": "add-promo-code-button",
"kind": "element_button",
"target": "Add a promo code",
"at": "/billing/subscription",
"page_context": "Billing page > Subscription tab > Special offers section",
"source_quote": "In the **Special offers** section, select **Add a promo code**.",
"source_line": 17,
"needs_auth": true
}
]
}
6 changes: 4 additions & 2 deletions docs-tests/assertions/account-billing.json
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,8 @@
"page_context": "Billing page > Pricing tab",
"source_quote": "The **Pricing** tab offers a way to quickly check all unit pricing for various platform services",
"source_line": 61,
"needs_auth": true
"needs_auth": true,
"at": "/billing/pricing"
},
{
"id": "invoices-tab",
Expand All @@ -126,7 +127,8 @@
"page_context": "Billing page > Invoices tab",
"source_quote": "The **Invoices** tab is where you can find your current and previous invoices for Apify platform usage.",
"source_line": 67,
"needs_auth": true
"needs_auth": true,
"at": "/billing/invoices"
},
{
"id": "limits-tab",
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
{
"source_file": "sources/platform/account/collaboration/general-resource-access.md",
"assertions": [
{
"id": "general-resource-access-setting-name",
"kind": "element_text",
"target": "General resource access",
"at": "/settings/security",
"page_context": "Settings page > Login & Privacy tab (Security & Privacy for organization accounts) > account resource access setting",
"source_quote": "**General resource access** is an account setting that defines the default access policy at the account level.",
"source_line": 13,
"needs_auth": true
},
{
"id": "restricted-default-option",
"kind": "element_text",
"target": "Restricted",
"page_context": "Settings page > Login & Privacy tab > General resource access setting options",
"source_quote": "**Restricted (default)**: With this setting, only signed-in users with an explicit access to the resources can access them.",
"source_line": 16,
"needs_auth": true
},
{
"id": "share-run-data-with-developers-section",
"kind": "element_text",
"target": "Share run data with developers",
"at": "/settings/security",
"page_context": "Settings page > Login & Privacy tab > Share run data with developers section",
"source_quote": "can be enabled under **Settings → Login & Privacy** in the **Share run data with developers** section",
"source_line": 82,
"needs_auth": true
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"source_file": "sources/platform/account/collaboration/organization/how_to_use.md",
"assertions": [
{
"id": "settings-organizations-route",
"kind": "route",
"target": "/settings/organizations",
"page_context": "Account page > Organizations tab, listing organizations the user is a member of",
"source_quote": "from the **Organizations** tab on your [account page](https://console.apify.com/settings/organizations)",
"source_line": 29,
"needs_auth": true
}
]
}
12 changes: 8 additions & 4 deletions docs-tests/assertions/account-console.json
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,8 @@
"page_context": "Landing page after signing in to Apify Console",
"source_quote": "you land on the **Dashboard** section that provides an overview of your account",
"source_line": 46,
"needs_auth": true
"needs_auth": true,
"at": "/"
},
{
"id": "dashboard-recently-viewed",
Expand All @@ -53,7 +54,8 @@
"page_context": "Dashboard overview section",
"source_quote": "**Recently viewed**: Displays Actors you have recently accessed.",
"source_line": 48,
"needs_auth": true
"needs_auth": true,
"at": "/"
},
{
"id": "dashboard-suggested-actors",
Expand All @@ -62,7 +64,8 @@
"page_context": "Dashboard overview section",
"source_quote": "**Suggested Actors for you**: Recommends Actors that might interest you",
"source_line": 50,
"needs_auth": true
"needs_auth": true,
"at": "/"
},
{
"id": "dashboard-actor-runs",
Expand All @@ -71,7 +74,8 @@
"page_context": "Dashboard overview section",
"source_quote": "**Actor runs**: Shows your recent Actor runs, as well as scheduled runs and tasks.",
"source_line": 52,
"needs_auth": true
"needs_auth": true,
"at": "/"
}
]
}
106 changes: 106 additions & 0 deletions docs-tests/assertions/account-settings-two-factor-authentication.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
{
"source_file": "sources/platform/account/settings/two-factor-authentication.md",
"assertions": [
{
"id": "two-factor-authentication-section",
"kind": "element_text",
"target": "Two-factor authentication",
"at": "/settings/security",
"page_context": "Settings > Login & Privacy tab",
"source_quote": "In **Two-factor authentication**, select **Enable**.",
"source_line": 29,
"needs_auth": true
},
{
"id": "enable-2fa-button",
"kind": "element_button",
"target": "Enable",
"at": "/settings/security",
"page_context": "Settings > Login & Privacy tab, Two-factor authentication section",
"source_quote": "In **Two-factor authentication**, select **Enable**.",
"source_line": 29,
"needs_auth": true
},
{
"id": "verify-code-field",
"kind": "element_text",
"target": "Verify the code from the app",
"page_context": "2FA setup modal shown after clicking Enable on Settings > Login & Privacy",
"source_quote": "In the **Verify the code from the app** field, enter the 6-digit code generated by your app.",
"source_line": 39,
"needs_auth": true
},
{
"id": "continue-button-setup",
"kind": "element_button",
"target": "Continue",
"page_context": "2FA setup modal, after entering the verification code",
"source_quote": "Select **Continue**.",
"source_line": 40,
"needs_auth": true
},
{
"id": "setup-key-link",
"kind": "element_button",
"target": "Setup key",
"page_context": "2FA setup modal, manual setup without QR code",
"source_quote": "In the modal window, select the **Setup key** link.",
"source_line": 47,
"needs_auth": true
},
{
"id": "phone-number-field",
"kind": "element_text",
"target": "Phone number",
"page_context": "2FA recovery information configuration step",
"source_quote": "**Phone number**. Apify will only use your phone number to verify your identity during the recovery process.",
"source_line": 67,
"needs_auth": true
},
{
"id": "personal-information-field",
"kind": "element_text",
"target": "Personal information",
"page_context": "2FA recovery information configuration step",
"source_quote": "**Personal information**. Make sure the information you provide is secure and easy to remember.",
"source_line": 68,
"needs_auth": true
},
{
"id": "continue-button-recovery-info",
"kind": "element_button",
"target": "Continue",
"page_context": "2FA recovery information configuration step, after filling phone number and personal information",
"source_quote": "Once you complete both fields, select **Continue**.",
"source_line": 70,
"needs_auth": true
},
{
"id": "recovery-settings-button",
"kind": "element_button",
"target": "Recovery settings",
"page_context": "Settings > Login & Privacy tab, Two-factor authentication section",
"source_quote": "In **Two-factor authentication**, select **Recovery settings**.",
"source_line": 96,
"needs_auth": true
},
{
"id": "disable-2fa-button",
"kind": "element_button",
"target": "Disable",
"page_context": "Settings > Login & Privacy tab, Two-factor authentication section",
"source_quote": "In **Two-factor authentication**, select **Disable**.",
"source_line": 106,
"needs_auth": true
},
{
"id": "remove-app-button",
"kind": "element_button",
"target": "Remove app",
"page_context": "2FA disable confirmation modal",
"source_quote": "Select **Remove app**.",
"source_line": 108,
"needs_auth": true
}
]
}
Loading
Loading