diff --git a/sales/preheat/README.md b/sales/preheat/README.md new file mode 100644 index 0000000..63d670d --- /dev/null +++ b/sales/preheat/README.md @@ -0,0 +1,129 @@ +# Preheat + +See from Jamf Pro whether each Apple content cache is working, which devices it serves, and whether it already holds the OS update they are about to ask for. Preheat it when it does not. + +> **Status: early, working, looking for testers.** Proven end to end in one lab: one cache +> server, three Macs, three iPads and an iPhone, through three real point releases. Not yet +> done: an update enforced by a Blueprint across many devices, and a second cache server in a +> second office. The full account is in +> [What is proven, and what is not](docs/FINDINGS.md#what-is-proven-and-what-is-not). +> Issues and pull requests welcome — open one at https://github.com/masterstompie/preheat/issues. + +![Jamf Pro dashboard with the content cache Smart Groups](docs/images/dashboard-smart-groups.png) + +*The day-to-day view: one cache server, ready for the next OS push, serving two Macs and four mobile devices. "Wiped recently" reads 1 because this cache's drive was erased the evening before: the tile to alert on, doing its job. "On but not chosen" and "Chosen but not caching" are the two that should always read 0. (Two captures of one page, joined. The "Preheat test" groups belong to the lab.)* + +## What it tells you +Apple content caching keeps one copy of each update in the office, and saves the internet link +every time a second device asks for it. When it works it needs no help. But Apple gives an +administrator no way to see whether it is working, what it holds, or which devices can reach it. + +Preheat answers one question per cache server, judged only against the devices that use that +server: does it already hold the OS update each of them is about to download? + +| Verdict | Meaning | +|---|---| +| READY | every file those devices will ask for is in the cache, complete | +| PARTIAL | some of them are | +| NOT READY | none of them is; or the cache is down; or the server has gone silent | + +The verdict is an extension attribute on the server's record, so Smart Groups, the dashboard and +email alerts work on it like on anything else in Jamf Pro. Beside it, Preheat shows: + +- what each cache holds, by name: "iOS 27.0 (24A437) delta from 26.7 for iPhone17,3" +- which cache each Mac, iPhone, iPad and Apple TV would use +- whether the cache is earning its keep: the share of what it delivered that did not come from Apple +- whether a cache was wiped, is low on space, or is caching when nobody chose it to + +When a cache is not ready, Preheat can fill it before the devices ask, with no test device at +that site: see [Preheating](docs/PREHEATING.md). + +## How it works +- Jamf Pro's inventory says which hardware and which OS build every device has. +- Apple's own lookup service, the one every device asks, says which file that hardware and build + would download next. +- An extension attribute on each cache server lists the files its cache holds. +- An extension attribute on each Mac says which cache that Mac would use, as macOS itself decides. +- `admin/readiness-check.py` puts the four together and writes the verdict. + +Nothing with a password runs on a device, and nothing is installed by hand on any Mac. More in +[Reference](docs/REFERENCE.md). + +## What you need +- Jamf Pro 11.31 or later, with computers enrolled and sending inventory, the cache servers + among them. Mobile devices too, if you want iPhone, iPad, Apple TV or Vision Pro readiness. +- A Jamf Pro API role and client. +- An admin Mac, or a scheduled job, with Python 3.8 or later and no extra packages. +- One or more Macs running content caching, each with a fixed address and the Xcode Command Line + Tools. +- macOS 27 on the cache servers, if you want their live status and to configure them by Blueprint. + +The full list, with the API privileges each feature needs, is in +[Requirements](docs/SETUP.md#requirements). + +## Setup, in short +| Do this | With | +|---|---| +| Create the API role and client, and store the credentials in your keychain | `python3 admin/readiness-check.py --store-credentials` | +| Create the extension attributes | `python3 admin/sync-eas.py --create`, and two text fields by hand | +| Run the check | `python3 admin/readiness-check.py`, and with `--write` to store the verdict | +| Create the Smart Groups | `python3 admin/create-smart-groups.py` | +| Have each cache server name its files | `python3 admin/create-labels-policy.py` | +| Check your work | `python3 admin/doctor.py` | +| Keep it current | run the check with `--write` on a schedule; hourly is plenty | + +Each step, with its tables, is in [Jamf setup](docs/SETUP.md#jamf-setup). The same installation +as Terraform is in [`terraform/`](terraform/README.md). Every script answers `--help` and changes +nothing when asked: [docs/USAGE.md](docs/USAGE.md). + +## Test without Jamf + python3 admin/readiness-check.py --inventory-json admin/sample-inventory.json + python3 admin/readiness-check.py --inventory-json admin/sample-inventory-large-site.json --platforms all # two public addresses, two peer caches + +Each sample carries the answers Apple gave when it was recorded (2026-09-28), so it gives the same +result whatever Apple has released since. `--record-answers` records them again. + +![Readiness check output for the sample fleet](docs/images/readiness-check-output.png) + +*Report for the sample two-office fleet: Office X ready, Office A missing the Mac14,2 asset.* + +## Read more +| Page | What is in it | +|---|---| +| [Setup](docs/SETUP.md) | Requirements, the extension attributes as Jamf's form asks for them, setup step by step, where things live | +| [Preheating](docs/PREHEATING.md) | Filling a cache before the devices ask: OS updates, hardware not yet enrolled, installers, Xcode, Aerials, and what cannot be preheated | +| [Blueprints](docs/BLUEPRINTS.md) | Declarative updates in two waves, a group that holds on the previous OS, update settings, and configuring the cache servers themselves | +| [Findings](docs/FINDINGS.md) | What was measured: what is proven, what one release weighs, what a full cache does, thin links and outages | +| [Caveats](docs/CAVEATS.md) | What can go wrong, each one seen in the lab; and two scripts for when clients do not see the cache | +| [Reference](docs/REFERENCE.md) | Every piece, where each fact comes from, large sites, mobile devices, live status, why a Mac downloaded at 1 pm, hardware coverage, the rest of the Jamf platform | +| [Usage](docs/USAGE.md) | Every script and every option | + +## Prior art +Charles Edge's [precache](https://github.com/krypted/precache) (2016 to 2020) pre-filled the old +macOS Server Caching Service with installers, IPSWs and App Store apps, and could take its device +list from Jamf Pro. Content caching has since moved into macOS itself and updates have become +per-model deltas. Preheat starts from the same idea, that a cache can be filled before devices ask, +and adds the parts that did not exist then: knowing what a cache holds, and whether it is ready for +the devices that use it. + +## Bill of Materials + +| Component | Kind | Version | License | +|---|---|---|---| +| [jamf/jamfplatform](https://registry.terraform.io/providers/jamf/jamfplatform) | Terraform provider | ≥ 0.29 | Jamf — verify before redistribution | + +Python dependencies: none (stdlib only). Shell dependencies: `/usr/bin/curl` (system, macOS built-in). + +## Privacy + +Preheat does not collect telemetry, send analytics, or phone home. It reads your Jamf Pro inventory +and queries Apple's public software update API (`gdmf.apple.com`), three other Apple catalogs for +file names (`swscan.apple.com`, `configuration.apple.com`, `devimages-cdn.apple.com`) and AppleDB +(`api.appledb.dev`). No device data leaves your network except in those outbound lookup calls, which carry no personally +identifiable information. See [Jamf's Privacy Policy](https://www.jamf.com/legal/privacy-policy/). + +## License + +Copyright © 2026 Jamf Software LLC. All rights reserved. + +Distributed under the Jamf Concepts Use Agreement. See LICENSE for the full terms. diff --git a/sales/preheat/admin/cache-collect.sh b/sales/preheat/admin/cache-collect.sh new file mode 100755 index 0000000..e0707f3 --- /dev/null +++ b/sales/preheat/admin/cache-collect.sh @@ -0,0 +1,154 @@ +#!/bin/bash +# Copyright 2026, Jamf Software LLC. +# This work is licensed under the terms of the Jamf Source Available License +# https://github.com/jamf/scripts/blob/main/LICENCE.md +# cache-collect.sh -- READ-ONLY inspection of an Apple content caching server. +# +# Purpose: gather everything needed to design a Jamf Pro Extension Attribute that +# reports which macOS updates are cached. Makes NO changes: the cache database is +# copied to a temp folder before it is queried, and the service is never touched. +# +# Run on the caching Mac: sudo bash admin/cache-collect.sh +# Output: /Users/Shared/cache-report-.txt (plus a .json of macOS-update rows) + +set -u +case "${1:-}" in -h|--help) /usr/bin/sed -n '2,/^set -u/p' "$0" | /usr/bin/sed -e '/^set -u/d' -e 's/^# \{0,1\}//'; exit 0;; esac +if [[ $EUID -ne 0 ]]; then echo "Run with sudo: sudo bash $0"; exit 1; fi + +STAMP=$(date +%Y%m%d-%H%M%S) +REPORT="/Users/Shared/cache-report-$STAMP.txt" +JSON="/Users/Shared/cache-macos-assets-$STAMP.json" +WORK=$(mktemp -d /tmp/cache-collect.XXXXXX) +trap 'rm -rf "$WORK"' EXIT + +exec > >(tee "$REPORT") 2>&1 +section(){ printf '\n\n===== %s =====\n' "$1"; } + +section "SYSTEM" +sw_vers +echo "model: $(sysctl -n hw.model)" +echo "board id: $(sysctl -n hw.target)" +echo "uptime: $(uptime)" +echo "hostname: $(scutil --get LocalHostName 2>/dev/null)" + +section "CONTENT CACHE STATUS (AssetCacheManagerUtil status)" +AssetCacheManagerUtil status 2>&1 + +section "CONTENT CACHE STATUS JSON" +AssetCacheManagerUtil -j status 2>&1 + +section "CONTENT CACHE SETTINGS JSON" +SETTINGS_JSON=$(AssetCacheManagerUtil -j settings 2>&1) +echo "$SETTINGS_JSON" + +section "LAUNCHD SERVICE STATE" +launchctl print system/com.apple.AssetCache 2>&1 | head -40 +echo "--- processes ---" +pgrep -lf AssetCache 2>&1 + +# ---- locate the data folder (default or custom DataPath) ---- +DATA_PATH=$(echo "$SETTINGS_JSON" | python3 -c ' +import json,sys +try: + d=json.load(sys.stdin); r=d.get("result",d) + print(r.get("DataPath") or "") +except Exception: print("")' 2>/dev/null) +[[ -z "$DATA_PATH" ]] && DATA_PATH="/Library/Application Support/Apple/AssetCache/Data" +DB="$DATA_PATH/AssetInfo.db" + +section "DATA FOLDER: $DATA_PATH" +ls -la "$DATA_PATH" 2>&1 | head -40 +echo "--- disk usage of data folder ---" +du -sh "$DATA_PATH" 2>&1 +echo "--- largest 25 files under data folder ---" +find "$DATA_PATH" -type f -size +100M -exec ls -l {} \; 2>/dev/null | sort -k5 -n -r | head -25 + +if [[ ! -f "$DB" ]]; then + echo "AssetInfo.db not found at $DB -- stopping database section." +else + # ---- copy DB (and WAL/SHM if present) so we never touch the live file ---- + cp "$DB" "$WORK/AssetInfo.db" + [[ -f "$DB-wal" ]] && cp "$DB-wal" "$WORK/AssetInfo.db-wal" + [[ -f "$DB-shm" ]] && cp "$DB-shm" "$WORK/AssetInfo.db-shm" + Q(){ sqlite3 -readonly "$WORK/AssetInfo.db" "$@"; } + + section "DATABASE SCHEMA" + Q ".schema" + + section "ZASSET COLUMNS" + Q -header -column "PRAGMA table_info(ZASSET);" + + section "ROW COUNTS PER TABLE" + for t in $(Q "select name from sqlite_master where type='table';"); do + printf '%-20s %s\n' "$t" "$(Q "select count(*) from $t;")" + done + + section "ASSET SUMMARY BY URI PREFIX (first 2 path components)" + Q -header -column " + select substr(ZURI,1,instr(substr(ZURI,2),'/')+1) as prefix, + count(*) as n, + round(sum(ZTOTALBYTES)/1073741824.0,2) as GB + from ZASSET group by prefix order by GB desc limit 40;" + + section "ASSET SUMMARY BY FILE EXTENSION" + Q -header -column " + select case when instr(ZURI,'.')=0 then '(none)' else lower(replace(ZURI, rtrim(ZURI, replace(ZURI,'.','')), '')) end as ext, + count(*) as n, round(sum(ZTOTALBYTES)/1073741824.0,2) as GB + from ZASSET group by ext order by GB desc limit 30;" + + section "SAMPLE OF 15 URIS (to see exact format)" + Q "select ZURI from ZASSET order by ZTOTALBYTES desc limit 15;" + + section "macOS UPDATE CANDIDATES (all columns, line mode)" + MATCH="ZURI like '%MacSoftwareUpdate%' or ZURI like '%InstallAssistant%' or ZURI like '%.ipsw%' or ZURI like '%macOSUpd%' or ZURI like '%SoftwareUpdate%' or ZURI like '%.aea%' or ZURI like '%FCS/%'" + Q -header -line "select *, + datetime(ZCREATIONDATE + 978307200,'unixepoch') as created_utc, + datetime(ZLASTACCESSED + 978307200,'unixepoch') as last_accessed_utc + from ZASSET where $MATCH order by ZTOTALBYTES desc;" + + section "macOS UPDATE CANDIDATES (compact)" + Q -header -column "select round(ZTOTALBYTES/1073741824.0,2) as GB, + datetime(ZCREATIONDATE + 978307200,'unixepoch') as created_utc, + ZURI + from ZASSET where $MATCH order by ZTOTALBYTES desc;" + + section "ON-DISK SIZE CHECK FOR CANDIDATES (is the file fully present?)" + # Try to relate DB rows to files on disk by GUID / checksum-looking columns. + Q -json "select * from ZASSET where $MATCH;" > "$JSON" + [[ -s "$JSON" ]] || echo "[]" > "$JSON" + echo "JSON written to $JSON ($(python3 -c "import json;print(len(json.load(open('$JSON'))))") rows)" + python3 - "$JSON" "$DATA_PATH" <<'PY' +import json,sys,os +rows=json.load(open(sys.argv[1])); data=sys.argv[2] +# Layout learned 2026-09-17: each asset's bytes live in // +for r in rows: + uri=r.get("ZURI",""); total=r.get("ZTOTALBYTES") or 0; guid=r.get("ZGUID") or "" + d=os.path.join(data,guid); ondisk=0; files=0 + if guid and os.path.isdir(d): + for root,_,fs in os.walk(d): + for f in fs: + try: ondisk+=os.path.getsize(os.path.join(root,f)); files+=1 + except OSError: pass + pct=(100.0*ondisk/total) if total else 0 + state="COMPLETE" if total and ondisk>=total else ("PARTIAL" if ondisk else "NO DATA ON DISK") + print(f"\n{uri}\n guid={guid} files={files} ondisk={ondisk} total={total} ({pct:.1f}%) -> {state}") + print(f" ZMD5OFFSET={r.get('ZMD5OFFSET')} ZCHECKSUM={r.get('ZCHECKSUM')!r}") +PY + + section "OTHER TABLES (first 20 rows each, may explain completeness/state)" + for t in $(Q "select name from sqlite_master where type='table' and name not in ('ZASSET');"); do + echo "--- $t ---"; Q -header -column "select * from $t limit 20;" + done +fi + +section "RECENT CONTENT CACHE LOG (last 30 min, errors/faults + status summaries)" +log show --last 30m --predicate 'subsystem == "com.apple.AssetCache"' --style compact 2>/dev/null \ + | grep -Ei 'error|fault|fail|shut|deactivat|activat|start|stopp|status|registration' | tail -80 + +section "RECENT MACOS UPDATE REQUESTS SEEN BY THE CACHE (last 24h, if any)" +log show --last 24h --predicate 'subsystem == "com.apple.AssetCache"' --style compact 2>/dev/null \ + | grep -Ei 'MacSoftwareUpdate|InstallAssistant|ipsw|\.zip|\.aea|FCS' | tail -40 + +section "DONE" +echo "Report: $REPORT" +echo "JSON: $JSON" diff --git a/sales/preheat/admin/create-labels-policy.py b/sales/preheat/admin/create-labels-policy.py new file mode 100644 index 0000000..4ac85ba --- /dev/null +++ b/sales/preheat/admin/create-labels-policy.py @@ -0,0 +1,52 @@ +#!/usr/bin/env python3 +# Copyright 2026, Jamf Software LLC. +# This work is licensed under the terms of the Jamf Source Available License +# https://github.com/jamf/scripts/blob/main/LICENCE.md +""" +Put admin/preheat-labels.py into Jamf Pro as a script, and create the policy that runs it on cache servers: + General: trigger Recurring Check-in, frequency Ongoing Scripts: preheat-labels.py + Maintenance: Update Inventory Scope: Smart Group "Content Caching - Servers" +So at every check-in a cache server names any new files it holds, then sends inventory, and the decoded EA +(ea-content-cache-decoded.sh) is current without anyone running the admin script. Disable the policy to stop. +Run again after editing preheat-labels.py: the script in Jamf is updated, the policy is left alone. +Needs API privileges: Create/Read/Update Scripts, Create/Read Policies, Read Smart Computer Groups. +Usage: python3 admin/create-labels-policy.py [--group "Content Caching - Servers"] [--disabled] +""" +import os, sys, argparse, urllib.error, urllib.request +from xml.sax.saxutils import escape +sys.path.insert(0, os.path.dirname(__file__)); rc = __import__("readiness-check") +SCRIPT_NAME = "preheat-labels.py"; POLICY_NAME = "Preheat - Refresh cache labels" + +def main(): + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--group", default="Content Caching - Servers"); ap.add_argument("--disabled", action="store_true", help="create the policy switched off") + args = ap.parse_args() + j = rc.jamf_from_credentials() + local = open(os.path.join(os.path.dirname(os.path.abspath(__file__)), SCRIPT_NAME)).read() + scripts = j.call("/api/v1/scripts?page=0&page-size=500").get("results", []) + hit = next((s for s in scripts if s["name"] == SCRIPT_NAME), None) + body = {"name": SCRIPT_NAME, "info": "Names the OS update files a content cache holds. No Jamf credentials. See the preheat README.", + "notes": "Managed by admin/create-labels-policy.py", "priority": "AFTER", "scriptContents": local} + if hit: + sid = hit["id"] + if (hit.get("scriptContents") or "").replace("\r\n", "\n").strip() == local.strip(): print(f"script {SCRIPT_NAME}: up to date (id {sid})") + else: j.call(f"/api/v1/scripts/{sid}", "PUT", {**hit, **body}); print(f"script {SCRIPT_NAME}: updated (id {sid})") + else: + sid = j.call("/api/v1/scripts", "POST", body)["id"]; print(f"script {SCRIPT_NAME}: created (id {sid})") + if any(p["name"] == POLICY_NAME for p in j.call("/JSSResource/policies")["policies"]): + print(f"policy '{POLICY_NAME}': already exists, left alone"); return + groups = j.call("/api/v2/computer-groups/smart-groups?page=0&page-size=500").get("results", []) + gid = next((g["id"] for g in groups if g["name"] == args.group), None) + if not gid: sys.exit(f"Smart Group '{args.group}' not found; run admin/create-smart-groups.py first") + xml = f"""{escape(POLICY_NAME)}{str(not args.disabled).lower()} +CHECKINtrueOngoing +{gid} +1 +true""" + req = urllib.request.Request(j.base + "/JSSResource/policies/id/0", data=xml.encode(), method="POST", + headers={"Authorization": "Bearer " + j.token, "Content-Type": "application/xml"}) + try: + with urllib.request.urlopen(req, timeout=60) as r: print(f"policy '{POLICY_NAME}': created, scoped to '{args.group}'", r.read().decode()[-40:]) + except urllib.error.HTTPError as e: sys.exit(f"policy create refused HTTP {e.code}: {e.read().decode()[:300]}") + +if __name__ == "__main__": main() diff --git a/sales/preheat/admin/create-smart-groups.py b/sales/preheat/admin/create-smart-groups.py new file mode 100644 index 0000000..98cfef2 --- /dev/null +++ b/sales/preheat/admin/create-smart-groups.py @@ -0,0 +1,98 @@ +#!/usr/bin/env python3 +# Copyright 2026, Jamf Software LLC. +# This work is licensed under the terms of the Jamf Source Available License +# https://github.com/jamf/scripts/blob/main/LICENCE.md +""" +Create the Smart Computer Groups this project recommends, skipping any that already exist. +Needs API client privileges: Read / Create Smart Computer Groups. +Credentials: login keychain (readiness-check.py --store-credentials) or env JAMF_URL, JAMF_CLIENT_ID, JAMF_CLIENT_SECRET. +Usage: python3 admin/create-smart-groups.py [--dry-run] [--update] + python3 admin/create-smart-groups.py --served-by "Office X" # add a per-office group +""" +import os, sys, json, urllib.error, argparse +sys.path.insert(0, os.path.dirname(__file__)); rc = __import__("readiness-check") + + +def crit(name, search, value, and_or="and", prio=0): + return {"name": name, "searchType": search, "value": value, "andOr": and_or, "priority": prio, + "openingParen": False, "closingParen": False} + +GROUPS = [ + ("Content Caching - Servers", "Macs running Apple content caching, serving or not (status EA starts with Active or Inactive).", + [crit(rc.EA_STATUS, "like", "Active")]), + ("Content Caching - Ready for OS push", "Cache servers whose local devices' next OS update is fully cached (readiness EA READY).", + [crit(rc.EA_READY, "like", "READY"), crit(rc.EA_READY, "not like", "NOT READY", "and", 1)]), + ("Content Caching - Partially ready", "Cache servers with some, not all, needed OS assets cached.", + [crit(rc.EA_READY, "like", "PARTIAL")]), + ("Content Caching - Not ready", "Cache servers missing every needed OS asset.", + [crit(rc.EA_READY, "like", "NOT READY")]), + ("Content Caching - Inactive or broken", "Caching turned on but not serving (registration failed, low space, data path missing).", + [crit(rc.EA_STATUS, "like", "Inactive")]), + ("Content Caching - Low on space", "Caches that report LOWSPACE: the cache volume is nearly full, missing or renamed. A cache that loses its volume switches itself off, and still shows here.", + [crit(rc.EA_STATUS, "like", "LOWSPACE")]), + ("Content Caching - Low benefit", "Cache servers that re-serve under 30% of what they download: few devices use them, or devices cannot reach them (VPN, wrong network).", + [crit(rc.EA_EFFECT, "like", "LOW ")]), + ("Content Caching - Wiped recently", "Cache servers that lost more than 90% of their content in the last 3 days: almost always a failed registration with Apple (VPN on the server, blocked egress, changed public IP).", + [crit(rc.EA_EFFECT, "like", "WIPED ")]), + ("Macs with no content cache", "Macs that will download OS updates straight from Apple (locator found no cache).", + [crit(rc.EA_FOUND, "is", "None found")]), +] + +MOBILE_GROUPS = [ + ("Mobile devices with no content cache", "iPads, iPhones and Apple TVs that will download OS updates straight from Apple.", + [crit(rc.EA_MOBILE_SERVER, "is", "None found")]), +] + +def create_mobile(j, groups, dry): + try: + existing = {g["groupName"] for g in j.call("/api/v2/mobile-device-groups/smart-groups?page=0&page-size=1000").get("results", [])} + except urllib.error.HTTPError as e: + print(f"mobile groups skipped (HTTP {e.code}; needs Read Smart Mobile Device Groups)"); return + for name, desc, criteria in groups: + if name in existing: print(f"exists: {name}"); continue + body = {"groupName": name, "groupDescription": desc, "criteria": criteria, "siteId": "-1"} + if dry: print(f"would create (mobile): {name}"); continue + for path in ("/api/v2/mobile-device-groups/smart-groups", "/api/v1/mobile-device-groups/smart-groups"): + try: j.call(path, "POST", body); print(f"created: {name} (mobile)"); break + except urllib.error.HTTPError as e: + if path.endswith("v1/mobile-device-groups/smart-groups") or e.code != 404: + print(f"FAILED: {name}: HTTP {e.code} {e.read().decode()[:200]}"); break + +def main(): + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--dry-run", action="store_true", help="print what would be created or updated") + ap.add_argument("--update", action="store_true", help="also bring existing groups of these names to the rules in this script (for installations made before a rule changed)") + ap.add_argument("--served-by", nargs=2, metavar=("LABEL", "GUID"), help="also create 'Macs served by