Improve docs indexing, accessibility, and initial load performance

.github/workflows/integration_tests.yml

@@ -168,6 +168,10 @@ jobs:
         working-directory: ./docs/app
         run: uv run --active --no-sync pytest --runxfail tests/test_doc_links.py -v
 
+      - name: Run docs frontend regression tests
+        working-directory: ./docs/app
+        run: uv run --active --no-sync node --test tests/frontend_quality.test.mjs
+
       - name: Upload Socket.dev Firewall report
         if: always()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1

docs/advanced_onboarding/code_structure.md

@@ -91,7 +91,7 @@ def template(page: Callable[[], rx.Component]) -> rx.Component:
     )
 ```
 
-The `@template` decorator should appear below the `@rx.page` decorator and above the page-returning function. See the [Posts Page](#a-post-page-example_big_apppagespostspy) code for an example.
+The `@template` decorator should appear below the `@rx.page` decorator and above the page-returning function. See the [Posts Page](#a-post-page:-example_big_app/pages/posts.py) code for an example.
 
 ## State Management
 

docs/app/README.md

@@ -40,3 +40,25 @@ WHITELISTED_PAGES = [
 - Paths are prefix-matched, so `"/components"` will include all pages under that section.
 
 After editing the whitelist, restart the dev server for changes to take effect.
+
+## Production quality checks
+
+Build the complete documentation app before auditing SEO or load performance:
+
+```bash
+uv run reflex export --no-zip
+node --test tests/frontend_quality.test.mjs
+uv run pytest tests
+```
+
+If the build uses a custom `REFLEX_WEB_WORKDIR`, pass that environment variable to both test commands. The Python link validator reads that build's sitemap. The frontend tests use the build's installed React and bundler to check server-rendered code, highlight invalidation, and removal of unused components.
+
+The `reflex-docs` integration CI jobs run the frontend tests after building the production site, using the installed React and bundler dependencies.
+
+Breadcrumbs and canonical URLs use `deploy_url` and `frontend_path` from the app config. Local runs use the framework's localhost default. Deployment jobs must set `REFLEX_DEPLOY_URL` to the origin serving that build (for example, `https://reflex.dev` in production or the staging origin).
+
+The docs app serves permanent HTTP 301 redirects for its legacy URLs when the Reflex backend serves the frontend. In development or when HTML is hosted separately, requests reach the frontend instead: the redirect pages retain client navigation and prerendered HTML includes an immediate refresh, canonical link, noindex directive, and a usable destination link. That fallback navigates readers but returns HTTP 200. For HTTP 301 semantics on a separate frontend/CDN, configure redirects at that host's edge using the `redirects` list in `reflex_docs/reflex_docs.py`; backend middleware alone cannot redirect requests it never receives.
+
+Docs pages intentionally omit the marketing site's pixels and session recording scripts. Search, examples, newsletter signup, and status information remain available.
+
+The docs config enables `frontend_lazy_bundled_libraries`. Optional libraries registered for dynamic components load on the first dynamic-component evaluation, while React and the shared runtime stay available immediately. This prevents the full Radix namespace from being imported on every page. The framework default remains `False`; custom scripts that read optional libraries from `window.__reflex` directly should retain that default or await `window.__reflex_load()` first.

docs/app/news/+docs-search-and-loading.docs.md

@@ -0,0 +1 @@
+Improve documentation search metadata, structured breadcrumbs, legacy redirects, responsive reference tables, and accessible examples. Remove marketing trackers from documentation pages.

docs/app/reflex_docs/pages/docs/__init__.py

@@ -179,6 +179,8 @@ def get_image_from_frontmatter(filepath: str) -> str | None:
 
 
 manual_titles = {
+    "docs/ai_builder/apis.md": "APIs",
+    "docs/ai_builder/urls.md": "URLs",
     "docs/database/overview.md": "Database Overview",
     "docs/custom-components/overview.md": "Custom Components Overview",
     "docs/custom-components/command-reference.md": "Custom Component CLI Reference",
@@ -268,7 +270,7 @@ def extract_doc_description(
     Returns:
         A cleaned, truncated description, or None.
     """
-    min_len = 120
+    min_len = 40
     if metadata:
         for key in ("meta_description", "description"):
             value = metadata.get(key)
@@ -316,8 +318,8 @@ def extract_doc_description(
             *(f"{n}." for n in range(1, 10)),
         )
         # Accumulate prose across paragraph breaks until the description is
-        # substantial (~120 chars) so a short opening sentence doesn't become a
-        # too-short meta description. Stop at the first structural line
+        # a useful summary (~40 chars) so a short opening sentence doesn't become a
+        # generic meta description. Stop at the first structural line
         # (heading/list/code) once some prose has been collected.
         for raw in text.splitlines():
             line = raw.strip()

docs/app/reflex_docs/pages/docs/metadata.py

@@ -19,3 +19,44 @@ def truncate_meta_description(description: str, max_len: int = 155) -> str:
     # exceeds max_len, even when the leading slice has no word boundary.
     truncated = description[: max_len - 1].rsplit(" ", 1)[0].rstrip(",.;:")
     return f"{truncated}…"
+
+
+def docs_metadata(path: str, title: str, description: str | None) -> tuple[str, str]:
+    """Build distinct search snippets using the page's product hierarchy.
+
+    Args:
+        path: The app-relative documentation route.
+        title: The page heading.
+        description: A summary extracted from the document, when available.
+
+    Returns:
+        The search title and description.
+    """
+    acronyms = {
+        "ai": "AI",
+        "api": "API",
+        "cli": "CLI",
+        "html": "HTML",
+        "css": "CSS",
+        "mcp": "MCP",
+        "sdk": "SDK",
+    }
+    parents = [
+        " ".join(acronyms.get(word, word.capitalize()) for word in part.split("-"))
+        for part in path.strip("/").split("/")[:-1]
+    ]
+    if title.lower() == "index" and parents:
+        title = " ".join(
+            acronyms.get(word, word.capitalize())
+            for word in path.strip("/").split("/")[-1].split("-")
+        )
+    title = " ".join(acronyms.get(word.lower(), word) for word in title.split())
+    context = [
+        parent for parent in reversed(parents) if parent.lower() != title.lower()
+    ]
+    subject = " · ".join([title, *context])
+    summary = (
+        description
+        or f"{subject}: documentation, examples, and reference for building Python web applications with Reflex."
+    )
+    return f"{subject} · Reflex Docs", truncate_meta_description(summary)

docs/app/reflex_docs/pages/docs/source.py

@@ -30,49 +30,39 @@ def stacked_description_rows(
     description: Callable[[], rx.Component],
     breakpoint: str,
     description_cell_class: str = "",
-) -> tuple[rx.Component, rx.Component]:
+) -> tuple[rx.Component, ...]:
     """Build a table row whose description column stacks on narrow containers.
 
     On containers at least as wide as the Tailwind ``breakpoint``, the
     description renders as the last cell of a single row. Below that, it
-    drops to a second full-width row so the table never needs horizontal
+    stacks below the other cells so the table never needs horizontal
     scrolling just to read descriptions. The nearest ancestor with the
     ``@container`` class defines the measured width.
 
     Args:
         leading_cells: The non-description cells as (content, extra cell classes).
-        description: Description factory, called once per rendered copy.
+        description: Description factory, called once.
         breakpoint: Tailwind container breakpoint (e.g. ``"4xl"``) above which
             the description stays in-row.
         description_cell_class: Extra classes for the in-row description cell.
 
     Returns:
-        The main row and the narrow-only description row.
+        A single row whose description spans the grid on narrow containers.
     """
     return (
         rx.table.row(
             *[
                 rx.table.cell(
                     content,
-                    # When stacked, the description row below carries the divider.
-                    class_name=f"{extra_class} @max-{breakpoint}:shadow-none".strip(),
+                    class_name=f"{extra_class} @max-{breakpoint}:shadow-none @max-{breakpoint}:min-w-0 @max-{breakpoint}:h-auto".strip(),
                 )
                 for content, extra_class in leading_cells
-            ]
-            + [
-                rx.table.cell(
-                    description(),
-                    class_name=f"{description_cell_class_name} {description_cell_class} hidden @{breakpoint}:table-cell",
-                ),
-            ]
-        ),
-        rx.table.row(
+            ],
             rx.table.cell(
                 description(),
-                col_span=len(leading_cells),
-                class_name=description_cell_class_name,
+                class_name=f"{description_cell_class_name} {description_cell_class} @max-{breakpoint}:col-span-full @max-{breakpoint}:h-auto [&_p:last-child]:mb-0",
             ),
-            class_name=f"@{breakpoint}:hidden",
+            class_name=f"@max-{breakpoint}:grid @max-{breakpoint}:grid-cols-{len(leading_cells)}",
         ),
     )
 
@@ -119,7 +109,7 @@ def format_fields(
     if env_var_prefix is not None:
         headers = [headers[0], "Environment Variable", *headers[1:]]
 
-    def field_rows(field: FieldDocumentation) -> tuple[rx.Component, rx.Component]:
+    def field_rows(field: FieldDocumentation) -> tuple[rx.Component, ...]:
         leading_cells = [(format_field(field), "")]
         if env_var_prefix is not None:
             leading_cells.append((

docs/app/reflex_docs/pages/docs_landing/views/ai_builder.py

@@ -88,7 +88,11 @@ def card(
             class_name="flex flex-col gap-2 p-8",
         ),
         content,
-        rx.el.a(href=href, class_name="absolute inset-0"),
+        rx.el.a(
+            href=href,
+            aria_label=title,
+            class_name="absolute inset-0 rounded-xl focus-visible:outline-2 focus-visible:-outline-offset-4 focus-visible:outline-primary-9",
+        ),
         class_name="flex flex-col bg-secondary-1/96 backdrop-blur-[16px] rounded-xl relative cursor-pointer transition-colors overflow-hidden shadow-[0_0_0_1px_rgba(0,0,0,0.04),0_12px_24px_0_rgba(0,0,0,0.08),0_1px_1px_0_rgba(0,0,0,0.01),0_4px_8px_0_rgba(0,0,0,0.03)] dark:shadow-none dark:border dark:border-secondary-4",
     )
 

docs/app/reflex_docs/pages/docs_landing/views/link_item.py

@@ -43,7 +43,11 @@ def link_item(
             description,
             class_name="text-secondary-11 text-sm font-[475] text-start",
         ),
-        rx.el.a(to=href, class_name="absolute inset-0"),
+        rx.el.a(
+            to=href,
+            aria_label=title,
+            class_name="absolute inset-0 rounded-xl focus-visible:outline-2 focus-visible:outline-offset-4 focus-visible:outline-primary-9",
+        ),
         class_name=ui.cn(
             "flex flex-col gap-2 pr-8 py-8 group border-r border-b border-secondary-4 relative max-lg:p-6 hover:bg-[linear-gradient(243deg,var(--secondary-2)_0%,var(--secondary-1)_100%)]",
             "lg:pl-8 pl-6" if has_padding_left else "",

docs/app/reflex_docs/redirects.py

@@ -0,0 +1,46 @@
+"""Permanent redirects for renamed documentation pages."""
+
+from collections.abc import Sequence
+
+from starlette.responses import RedirectResponse
+from starlette.types import ASGIApp, Receive, Scope, Send
+
+
+class DocsRedirectMiddleware:
+    """Resolve legacy documentation URLs before rendering the frontend."""
+
+    def __init__(
+        self,
+        app: ASGIApp,
+        redirects: Sequence[tuple[str, str]],
+        frontend_path: str,
+    ) -> None:
+        """Build a lookup of public legacy URLs and their canonical destinations.
+
+        Args:
+            app: The wrapped application.
+            redirects: App-relative source and destination routes.
+            frontend_path: The public mount prefix.
+        """
+        self.app = app
+        prefix = frontend_path.rstrip("/")
+        self.redirects = {
+            prefix + source.rstrip("/"): prefix + target for source, target in redirects
+        }
+
+    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
+        """Redirect known GET/HEAD routes and pass other requests through.
+
+        Args:
+            scope: The ASGI connection scope.
+            receive: The ASGI receive callable.
+            send: The ASGI send callable.
+        """
+        if scope["type"] == "http" and scope["method"] in {"GET", "HEAD"}:
+            target = self.redirects.get(scope["path"].rstrip("/"))
+            if target is not None:
+                if query := scope.get("query_string", b""):
+                    target += "?" + query.decode("latin-1")
+                await RedirectResponse(target, status_code=301)(scope, receive, send)
+                return
+        await self.app(scope, receive, send)

docs/app/reflex_docs/reflex_docs.py

@@ -1,22 +1,26 @@
 """The main Reflex website."""
 
+import json
 import os
 import sys
+from functools import partial
 
 import reflex as rx
 import reflex_enterprise as rxe
 from reflex_site_shared import styles
 from reflex_site_shared.backend.status import monitor_checkly_status
-from reflex_site_shared.constants import REFLEX_ASSETS_CDN, REFLEX_DOMAIN_URL
+from reflex_site_shared.constants import REFLEX_ASSETS_CDN
 from reflex_site_shared.meta.meta import (
     ONE_LINE_DESCRIPTION,
     create_meta_tags,
     favicons_links,
     to_cdn_image_url,
 )
-from reflex_site_shared.telemetry import get_pixel_website_trackers
+from reflex_site_shared.utils.url import public_url
 
 from reflex_docs.pages import page404, routes
+from reflex_docs.redirects import DocsRedirectMiddleware
+from reflex_docs.templates.docpage.docpage import breadcrumb_data
 from reflex_docs.whitelist import _check_whitelisted_path
 
 # This number discovered by trial and error on Windows 11 w/ Node 18, any
@@ -48,7 +52,7 @@ def _llms_txt_directive() -> rx.Component:
         radius="large",
         accent_color="violet",
     ),
-    head_components=get_pixel_website_trackers() + favicons_links(),
+    head_components=favicons_links(),
 )
 
 app.register_lifespan_task(monitor_checkly_status)
@@ -82,7 +86,7 @@ def _canonical_url(path: str) -> str:
     # "/docsoverview/" instead of "/docs/overview/".
     if not path.startswith("/"):
         path = "/" + path
-    url = REFLEX_DOMAIN_URL.rstrip("/") + _FRONTEND_PATH + path
+    url = public_url(path)
     return url if url.endswith("/") else url + "/"
 
 
@@ -135,6 +139,16 @@ def _canonical_url(path: str) -> str:
             # can't be emitted, so keep any page-provided meta as-is.
             canonical = None
             meta = list(route.meta) if route.meta is not None else []
+        if canonical is not None:
+            meta.append(
+                rx.el.script(
+                    json.dumps(
+                        breadcrumb_data(route.path, head_title.split(" · ")[0]),
+                        ensure_ascii=False,
+                    ).replace("<", "\\u003c"),
+                    type="application/ld+json",
+                )
+            )
         meta.append({"name": "theme-color", "content": route.background_color})
 
         # Reflex's compiler always renders exactly one og:image from add_page's
@@ -188,21 +202,50 @@ def _canonical_url(path: str) -> str:
 ])
 
 
-def _redirect_page():
+def _redirect_page(target: str):
+    """Render a usable destination link for static hosting.
+
+    Args:
+        target: The app-relative destination, prefixed by the router.
+
+    Returns:
+        The static redirect page content.
+    """
     return rx.fragment(
-        rx.el.h1("Redirecting", class_name="sr-only"),
+        rx.el.h1("This page has moved"),
+        rx.el.a("Continue to the documentation", href=target),
     )
 
 
 for source, target in redirects:
     if _check_whitelisted_path(target):
         app.add_page(
-            _redirect_page,
+            partial(_redirect_page, target),
             route=source,
             title="Redirecting - Reflex Web Framework",
             description="You are being redirected to the requested page.",
             on_load=rx.redirect(target),
             context={"sitemap": None},
+            meta=[
+                rx.el.link(rel="canonical", href=_canonical_url(target)),
+                {"name": "robots", "content": "noindex, follow"},
+                rx.el.meta(
+                    http_equiv="refresh", content=f"0;url={_FRONTEND_PATH}{target}"
+                ),
+            ],
         )
 
 app.add_page(page404.component, route=page404.path)
+
+
+# HTTP 301 applies when page requests reach this backend. Separate frontend
+# hosts use the redirect pages above and need edge rules for HTTP 301 semantics.
+app.api_transformer = partial(
+    DocsRedirectMiddleware,
+    redirects=[
+        (source, target)
+        for source, target in redirects
+        if _check_whitelisted_path(target)
+    ],
+    frontend_path=_FRONTEND_PATH,
+)