ENT-14135: Added broken internal link checker (3.24)

content/examples/tutorials/file_comparison.markdown

@@ -6,7 +6,7 @@ aliases:
   - "/examples-tutorials-file_comparison.html"
 ---
 
-1. Add the [policy contents][File comparison#Full policy] (also can be downloaded from <a href="file_compare_test.cf">file_compare_test.cf</a>) to a new file, such as /var/cfengine/masterfiles/file_test.cf.
+1. Add the [policy contents][File comparison#Full policy] (also can be downloaded from <a href="../file_compare_test.cf">file_compare_test.cf</a>) to a new file, such as /var/cfengine/masterfiles/file_test.cf.
 2. Run the following commands as root on the command line:
    ```console
    export AOUT_BIN="a.out"

content/getting-started/developing-modules.markdown

@@ -137,6 +137,6 @@ There are several places to look for more information or inspiration when writin
 
 * [The real git promise type code](https://github.com/cfengine/modules/tree/c3b7329b240cf7ad062a0a64ee8b607af2cb912a/promise-types/git/)
 * [HTTP promise type module](https://github.com/cfengine/modules/tree/c861789d4b376147d904fccd76963a92e65eaa97/promise-types/http/)
-* [CFEngine custom promise type specification](./reference-promise-types-custom.html)
+* [CFEngine custom promise type specification](/reference/promise-types/custom/)
 * [Blog post: How to implement CFEngine Custom Promise types in Python](https://cfengine.com/blog/2020/how-to-implement-cfengine-custom-promise-types-in-python/)
 * [Blog post: How to implement CFEngine Custom Promise types in Bash](https://cfengine.com/blog/2021/how-to-implement-cfengine-custom-promise-types-in-bash/)

content/getting-started/installation/_index.markdown

@@ -40,7 +40,7 @@ This is also possible, for example using Vagrant and VirtualBox, and we will pro
 
 With the Linux VM there are 2 machines we will be talking about:
 
-![](machines.png)
+![](../machines.png)
 
 The **CFEngine hub** is the aforementioned Ubuntu 20.04 VM.
 We will access this via SSH, and install CFEngine there.
@@ -153,7 +153,7 @@ Spawn an Ubuntu 20.04 Linux Virtual Machine using the web GUI (in Digital Ocean,
 Find the IP address of your virtual machine, and the username so you can log in with SSH.
 For example, in Digital Ocean, the username is `root`, and the IP might be `128.199.44.119` (found in top left of droplet screen as "ipv4"):
 
-![](digital-ocean.png)
+![](../digital-ocean.png)
 
 **Note:** In the rest of this tutorial, replace the IP address we use in the examples, `192.168.56.2` with that IP.
 
@@ -229,7 +229,7 @@ In the future you might want to set up a DNS entry for your hub and give it a pr
 After this, you should see the CFEngine Enterprise login screen.
 Log in with username admin, password admin, and you will be asked to change the password.
 
-![](mp-login.png)
+![](../mp-login.png)
 
 After changing the password to something more secure, you should be able to log in and see the dashboard.
 

content/reference/promise-types/files/_index.markdown

@@ -101,7 +101,7 @@ The diagram below shows the ordering. Notice that the same ordering
 applies regardless of file type (plain-file or directory). Note also
 that file editing is done "atomically".
 
-![The normal ordering of file operators in CFEngine 3](filelogic.png)
+![The normal ordering of file operators in CFEngine 3](../filelogic.png)
 
 The pseudo-code for this logic is shown in the diagram and below:
 

content/reference/promise-types/files/edit_line/insert_lines.markdown

@@ -360,7 +360,7 @@ insert_if_not_contains_from_list => { "find_me_1", "find_me_2" };
 
 **Type:** `body location`
 
-**See also:** [Common body attributes][Promise types#Common body attributes], [`location` bodies in the standard library](reference-masterfiles-policy-framework-lib-files.html#location-bodies), [`start` location body in the standard library](reference-masterfiles-policy-framework-lib-files.html#location-bodies), [`before(srt)` location body in the standard library](reference-masterfiles-policy-framework-lib-files.html#before), [`after(srt)` location body in the standard library](reference-masterfiles-policy-framework-lib-files.html#after)
+**See also:** [Common body attributes][Promise types#Common body attributes], [`location` bodies in the standard library](/reference/masterfiles-policy-framework/lib-files#location-bodies), [`start` location body in the standard library](/reference/masterfiles-policy-framework/lib-files#location-bodies), [`before(srt)` location body in the standard library](/reference/masterfiles-policy-framework/lib-files#before), [`after(srt)` location body in the standard library](/reference/masterfiles-policy-framework/lib-files#after)
 
 #### before_after
 

content/web-ui/settings.markdown

@@ -138,7 +138,7 @@ status (on or off).
 
 ## Version control repository
 
-<img src="settings-vcs.png" alt="Version control repository" width="700px">
+<img src="hub_administration/settings-vcs.png" alt="Version control repository" width="700px">
 
 The repository holding the organization's masterfiles can be adjusted
 on the Version control repository screen.

generator/_references.md

@@ -11,57 +11,58 @@
 [github documentation]: https://github.com/cfengine/documentation "Documentation repository on GitHub"
 [github design-center]: https://github.com/cfengine/design-center "Design-Center repository on GitHub"
 [evaluate cfengine]: https://cfengine.com/evaluate-enterprise "Evaluate CFEngine"
-[cfengine getting started]: https://cfengine.com/enterprise-getting-started "Get started in 10 minutes"
 [LMDB]: http://symas.com/mdb/ "Symas Lightning Memory-Mapped Database"
-[vim_cf3]: https://github.com/neilhwatson/vim_cf3 "CFEngine 3 vim integration"
 [community package repositories]: http://cfengine.com/cfengine-linux-distros "Community package repositories"
 [community download page]: https://cfengine.com/community/download/ "Community package download page"
 [enterprise software download page]: https://cfengine.com/product/free-download/ "Enterprise download page"
-[boolean]: reference-language-concepts-promises.html#promise-attributes
-[clist]: reference-language-concepts-promises.html#promise-attributes
-[anchored]: reference-language-concepts-pattern-matching-and-referencing.html#anchored-vs-unanchored-regular-expressions
-[unanchored]: reference-language-concepts-pattern-matching-and-referencing.html#anchored-vs-unanchored-regular-expressions
-[datatypes]: reference-language-concepts-variables.html
-[utility functions]: tags-utility-functions.html
-[data functions]: tags-data-functions.html
-[system functions]: tags-system-functions.html
-[io functions]: tags-io-functions.html
-[comm functions]: tags-communication-functions.html
-[files functions]: tags-files-functions.html
-[body classes]: reference-promise-types.html#classes
-[body common]: reference-components.html#common-control
-[body file]: reference-language-concepts-namespaces.html
-[body hub control port]: reference-components-cf-hub.html#port
-[sys.policy_entry_dirname]: reference-special-variables-sys.html#sys-policy_entry_dirname
-[sys.policy_entry_filename]: reference-special-variables-sys.html#sys-policy_entry_filename
+[boolean]: /reference/language-concepts/promises#promise-attributes
+[clist]: /reference/language-concepts/promises#promise-attributes
+[anchored]: /reference/language-concepts/pattern-matching-and-referencing#anchored-vs-unanchored-regular-expressions
+[unanchored]: /reference/language-concepts/pattern-matching-and-referencing#anchored-vs-unanchored-regular-expressions
+[datatypes]: /reference/language-concepts/variables
+[body classes]: /reference/promise-types#classes
+[body common]: /reference/components#common-control
+[body file]: /reference/language-concepts/namespaces
+[body hub control port]: /reference/components/cf-hub#port
+[sys.policy_entry_dirname]: /reference/special-variables/sys#syspolicy_entry_dirname
+[sys.policy_entry_filename]: /reference/special-variables/sys#syspolicy_entry_filename
 [supported versions]: https://cfengine.com/extended-support/
-[Inventory API]: api-enterprise-api-ref-inventory.html
-[sys.uqhost]: reference-special-variables-sys.html#sys-uqhost
-[sys.policy_hub]: reference-special-variables-sys.html#sys-policy_hub
-[seed_cp]: reference-masterfiles-policy-framework-lib-files.html#seed_cp
-[Append to inputs used by main policy]: reference-masterfiles-policy-framework.html#append-to-inputs-used-by-main-policy
-[mpf_extra_autorun_inputs]: reference-masterfiles-policy-framework.html#additional-automatically-loaded-inputs
-[Append to inputs used by update policy]: reference-masterfiles-policy-framework.html#append-to-inputs-used-by-update-policy
-[Classes and decisions]: reference-language-concepts-classes.html
-[language-concepts-classes-hard]: reference-language-concepts-classes.html#hard-classes.html "Language concepts -> Classes and decisions: Hard classes"
-[lib/files.cf]: reference-masterfiles-policy-framework-lib-files
-[lib/packages.cf]: reference-masterfiles-policy-framework-lib-packages
-[stdlib-mog]: reference-masterfiles-policy-framework-lib-files.html#mog
+[Inventory API]: /api/enterprise-api-ref/inventory
+[sys.uqhost]: /reference/special-variables/sys#sysuqhost
+[sys.policy_hub]: /reference/special-variables/sys#syspolicy_hub
+[seed_cp]: /reference/masterfiles-policy-framework/lib-files#seed_cp
+[Masterfiles Policy Framework]: /reference/masterfiles-policy-framework
+[Append to inputs used by main policy]: /reference/masterfiles-policy-framework#append-to-inputs-used-by-main-policy
+[mpf_extra_autorun_inputs]: /reference/masterfiles-policy-framework#additional-automatically-loaded-inputs
+[Append to inputs used by update policy]: /reference/masterfiles-policy-framework#add-additional-policy-files-for-update-inputs
+[Classes and decisions]: /reference/language-concepts/classes
+[language-concepts-classes-hard]: /reference/language-concepts/classes#hard-classes "Language concepts -> Classes and decisions: Hard classes"
+[lib/files.cf]: /reference/masterfiles-policy-framework/lib-files
+[lib/packages.cf]: /reference/masterfiles-policy-framework/lib-packages
+[stdlib-mog]: /reference/masterfiles-policy-framework/lib-files#mog
 [jq-project]: https://stedolan.github.io/jq/ "jq is a lightweight and flexible command-line JSON processor. Try online at jqplay.org!"
-[Using Vagrant]: getting-started-installation-general-installation-installation-enterprise-vagrant.html "The CFEngine Vagrant environment provides an easy way to test and explore CFEngine Enterprise."
-[type]: reference-functions-type.html
-[promise-type-measurements]: reference-promise-types-measurements.html
-[promise-type-custom]: reference-promise-types-custom.html
-[promise-type-custom-protocol]: reference-promise-types-custom.html#protocol
-[component-cf-monitord]: reference-components-cf-monitord.html
-[cf-hub]: reference-components-cf-hub.html
-[cf-hub#hub_schedule]: reference-components-cf-hub.html#hub_schedule
-[cf-hub#exclude_hosts]: reference-components-cf-hub.html#exclude_hosts
-[cf-hub#control-promises]: reference-components-cf-hub.html#control-promises
-[mpf-configure-measurement-collection]: reference-masterfiles-policy-framework.html#configure-enterprise-measurement-monitoring-collection.html
-[mpf-configure-component-restart]: reference-masterfiles-policy-framework.html#configure-mpf-to-automatically-restart-components-on-relevant-data-change.html
-[mpf-classification-bundles]: reference-masterfiles-policy-framework.html#classification-bundles-before-autorun.html
-[mpf-services-autorun]: reference-masterfiles-policy-framework-services-autorun.html
-[package-modules-the-api]: reference-language-concepts-modules-package-module-api.html#the-api
-[Functions#collecting functions]: /reference/functions/#collecting-functions
-[High availability]: /examples/tutorials/high-availability/ "CFEngine High availability overview"
+[Using Vagrant]: /getting-started/installation/general-installation/installation-enterprise-vagrant "The CFEngine Vagrant environment provides an easy way to test and explore CFEngine Enterprise."
+[type]: /reference/functions/type
+[promise-type-measurements]: /reference/promise-types/measurements
+[promise-type-custom]: /reference/promise-types/custom
+[promise-type-custom-protocol]: /reference/promise-types/custom#protocol
+[component-cf-monitord]: /reference/components/cf-monitord
+[cf-hub]: /reference/components/cf-hub
+[cf-hub#hub_schedule]: /reference/components/cf-hub#hub_schedule
+[cf-hub#exclude_hosts]: /reference/components/cf-hub#exclude_hosts
+[cf-hub#control-promises]: /reference/components/cf-hub#control-promises
+[mpf-configure-measurement-collection]: /reference/masterfiles-policy-framework#configure-enterprise-measurementmonitoring-collection
+[mpf-configure-component-restart]: /reference/masterfiles-policy-framework#configure-mpf-to-automatically-restart-components-on-relevant-data-change
+[mpf-classification-bundles]: /reference/masterfiles-policy-framework#classification-bundles-before-autorun
+[mpf-services-autorun]: /reference/masterfiles-policy-framework/services-autorun
+[package-modules-the-api]: /reference/language-concepts/modules/package-module-api#the-api
+[Functions#collecting functions]: /reference/functions#collecting-functions
+[guest_environments]: /reference/promise-types/guest_environments
+[defaults]: /reference/promise-types/defaults
+[getgrgid()]: https://linux.die.net/man/3/getgrgid
+[getgrnam()]: https://linux.die.net/man/3/getgrnam
+[getpwuid()]: https://linux.die.net/man/3/getpwuid
+[getpwnam()]: https://linux.die.net/man/3/getpwnam
+[select()]: https://linux.die.net/man/3/select
+[search_up]: /reference/functions/findfiles_up
+[High availability]: /examples/tutorials/high-availability "CFEngine High availability overview"

generator/_scripts/cfdoc_link_checker.py

@@ -0,0 +1,126 @@
+import html.parser
+import os
+import re
+import sys
+import urllib.parse
+
+# Tags/attributes that carry a URL we should be able to resolve to a file.
+LINK_ATTRS = {
+    "a": "href",
+    "img": "src",
+    "link": "href",
+    "script": "src",
+}
+
+# Schemes that never point at a file in the built site.
+SKIPPED_SCHEMES = ("mailto:", "tel:", "javascript:", "data:")
+
+# The version switcher (lts_versions_list.html / versions_list.html) links to
+# sibling docs builds for other branches/versions, e.g. "../../docs/3.27/",
+# "../../docs/lts/" or "../../docs/archive/index.html". Each version/archive
+# is built and deployed separately, so those paths never exist in this
+# build's own _site and aren't broken links.
+VERSION_LINK_RE = re.compile(
+    r"(?:^|/)docs/(?:master|lts|archive|\d+(?:\.\d+){1,2})(?:/|$)"
+)
+
+
+class _LinkExtractor(html.parser.HTMLParser):
+    """Collects (url, line) for every href/src found in one HTML page."""
+
+    def __init__(self):
+        super().__init__(convert_charrefs=True)
+        self.links = []
+
+    def handle_starttag(self, tag, attrs):
+        attr_name = LINK_ATTRS.get(tag)
+        if attr_name is None:
+            return
+        url = dict(attrs).get(attr_name)
+        if url:
+            line, _col = self.getpos()
+            self.links.append((url, line))
+
+
+def _is_checkable(url):
+    """Only internal links/assets are worth resolving against the filesystem."""
+    url = url.strip()
+    if not url or url.startswith("#"):
+        return False
+    if url.startswith(SKIPPED_SCHEMES):
+        return False
+    if VERSION_LINK_RE.search(url):
+        return False
+    parsed = urllib.parse.urlsplit(url)
+    if parsed.scheme or parsed.netloc:
+        return False  # external, e.g. http://, https://, //host/...
+    return True
+
+
+def _resolve(site_root, html_file, url):
+    """Map an internal href/src to the file it should point at on disk.
+
+    Returns None for links that don't reference a separate file (e.g. a bare
+    "?query" link on the current page).
+    """
+    path = url.split("#", 1)[0].split("?", 1)[0]
+    if not path:
+        return None
+
+    if path.startswith("/"):
+        target = os.path.join(site_root, path.lstrip("/"))
+    else:
+        target = os.path.normpath(os.path.join(os.path.dirname(html_file), path))
+
+    # Hugo renders pretty URLs as a directory containing index.html; a link
+    # is valid whether or not it has the trailing slash the directory implies.
+    if os.path.isdir(target):
+        target = os.path.join(target, "index.html")
+
+    return target
+
+
+def _find_html_files(site_root):
+    for dirpath, _dirnames, filenames in os.walk(site_root):
+        for filename in filenames:
+            if filename.endswith(".html"):
+                yield os.path.join(dirpath, filename)
+
+
+def run(config):
+    """Checks every internal link/image/script src in the built site and
+    exits non-zero, listing what's broken and where, if any target is
+    missing.
+
+    Runs against the generated HTML in CFE_DIR (after the Hugo build) rather
+    than the markdown source, so it validates Hugo's actual routing (pretty
+    URLs, aliases, page bundles) instead of a reimplementation of it.
+    """
+    site_root = os.path.join(config["project_directory"], config["CFE_DIR"])
+    if not os.path.isdir(site_root):
+        sys.stderr.write("ERROR: built site not found at %s\n" % site_root)
+        sys.exit(1)
+
+    broken = []
+    for html_file in _find_html_files(site_root):
+        parser = _LinkExtractor()
+        with open(html_file, "r", encoding="utf-8", errors="replace") as f:
+            parser.feed(f.read())
+
+        for url, line in parser.links:
+            if not _is_checkable(url):
+                continue
+            target = _resolve(site_root, html_file, url)
+            if target is None:
+                continue
+            if not os.path.exists(target):
+                broken.append((os.path.relpath(html_file, site_root), line, url))
+
+    if broken:
+        sys.stderr.write(
+            "ERROR: %d broken internal link(s)/asset reference(s) found in the built site:\n"
+            % len(broken)
+        )
+        for page, line, url in broken:
+            sys.stderr.write("  %s:%d: %s\n" % (page, line, url))
+        sys.exit(1)

generator/_scripts/cfdoc_postprocess.py

@@ -24,6 +24,7 @@
 
 import cfdoc_environment as environment
 import cfdoc_sourcelinks as sourcelinks
+import cfdoc_link_checker as link_checker
 
 import sys
 
@@ -35,4 +36,9 @@
     print(sys.exc_info())
     exit(1)
 
+# Runs outside the try/except above: it exits directly (like
+# cfdoc_references_resolver.run) with a clean broken-link report, rather than
+# letting the generic exception handler above swallow that message.
+link_checker.run(config)
+
 exit(0)