Reduce binary footprint: minimal SQLite and selectable curl TLS backend#1499
Reduce binary footprint: minimal SQLite and selectable curl TLS backend#1499bmehta001 wants to merge 14 commits into
Conversation
On the CPP11/curl path (non-Apple, non-Windows), the built-in libcurl HTTP client was always compiled and curl was a hard find_package(CURL REQUIRED) dependency -- pulling in curl and a TLS backend (OpenSSL/mbedTLS) even for hosts that already have their own HTTP stack. Add option(BUILD_CURL_HTTP_CLIENT ON). When OFF, the curl block is skipped (no find_package(CURL), no link, no -DHAVE_MAT_CURL_HTTP_CLIENT) and the build instead defines -DMATSDK_NO_DEFAULT_HTTP_CLIENT. mat/config.h then undefines HAVE_MAT_DEFAULT_HTTP_CLIENT centrally (regardless of the config preset), which the SDK already handles end-to-end: HttpClientFactory and HttpClient_Curl.cpp compile out, and LogManagerImpl's existing !HAVE_MAT_DEFAULT_HTTP_CLIENT branch requires the host to supply an IHttpClient via CFG_MODULE_HTTP_CLIENT. Default ON keeps existing behavior unchanged. Apple/Windows are unaffected (they use native HTTP stacks and never enter the curl block). Validated on WSL x64-linux: with OFF, libmat has no curl symbols and a consumer links with no -lcurl/-lTLS (1.43 MB stripped, vs 4.39 MB with curl+mbedTLS and 10.65 MB with curl+OpenSSL). Files changed: - CMakeLists.txt: BUILD_CURL_HTTP_CLIENT option + gating - lib/include/mat/config.h: central HAVE_MAT_DEFAULT_HTTP_CLIENT opt-out Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tprint The SDK uses SQLite only for its offline event-storage cache (plain tables, transactions, WAL, autovacuum/VACUUM, a few PRAGMAs, and one custom UTF-8 function), so most SQLite subsystems are dead weight. Add an option to compile a private SQLite from the vendored amalgamation with a set of amalgamation-safe strip flags (single source of truth: MATSDK_SQLITE_MINIMAL_DEFS), removing the external sqlite3 dependency and shrinking SQLite ~10.2% (.text) / ~12.5% (object). - Root CMakeLists.txt: add option(MATSDK_MINIMAL_SQLITE) (default OFF). In vcpkg mode, skip find_package(unofficial-sqlite3) when bundling, and emit a clear FATAL_ERROR pointing at the system-sqlite/minimal-sqlite features when neither provides SQLite (e.g. a bare [core] install). - lib/CMakeLists.txt: define MATSDK_SQLITE_MINIMAL_DEFS, compute MATSDK_BUNDLE_SQLITE (minimal OR vendored-Android), and build a single sqlite3_bundled. The strip flags are applied ONLY when MATSDK_MINIMAL_SQLITE is ON, so the default Android legacy build keeps its existing unstripped bundled SQLite. Warnings are disabled on the vendored target (/w on MSVC, -w on GCC/Clang for the stripped build) so the SDK's -Werror/-WX does not fire on amalgamation code. A static mat propagates the PRIVATE sqlite3_bundled through its link interface, so export+install it. - MSTelemetryConfig.cmake.in: skip find_dependency(unofficial-sqlite3) when bundled. - vcpkg port: add a minimal-sqlite feature (-DMATSDK_MINIMAL_SQLITE=ON) and move sqlite3 into a default system-sqlite feature so [core,minimal-sqlite] drops it. - docs/building-with-vcpkg.md: document the feature, the size win, and the static-absorption symbol-visibility caveat. SQLITE_OMIT_AUTOINIT and SQLITE_DEFAULT_MEMSTATUS=0 are deliberately NOT stripped: the former because skipSqliteInitAndShutdown lets the host skip the SDK's explicit sqlite3_initialize() (which the host cannot do against a private SQLite), the latter because the SDK arms a soft heap limit via sqlite3_soft_heap_limit64() that is only enforced while memory statistics are enabled. Validated: vendored Linux Debug (77 offline-storage/SQLite unit tests pass on the debug amalgamation), vcpkg [core,minimal-sqlite] consumer (links MSTelemetry::sqlite3_bundled, runs 10/10, external sqlite3 dropped), default vcpkg path regression (system-sqlite intact), and MSVC compile/link of sqlite3_bundled+mat. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR introduces opt-in build knobs intended to reduce the SDK’s footprint in host-embedded scenarios by (1) allowing builds without the built-in libcurl HTTP client and (2) optionally replacing the external SQLite dependency with a private, feature-stripped bundled SQLite.
Changes:
- Added
BUILD_CURL_HTTP_CLIENTto omit the built-in curl/TLS HTTP client and require a host-suppliedIHttpClient. - Added
MATSDK_MINIMAL_SQLITEand corresponding vcpkgminimal-sqlitefeature to build a stripped, bundled SQLite instead of depending on vcpkg’ssqlite3. - Updated the vcpkg port and documentation to reflect the new SQLite feature split and minimal-SQLite option.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/ports/cpp-client-telemetry/vcpkg.json | Moves sqlite3 into a default system-sqlite feature and adds minimal-sqlite. |
| tools/ports/cpp-client-telemetry/portfile.cmake | Wires vcpkg minimal-sqlite feature into -DMATSDK_MINIMAL_SQLITE=ON. |
| lib/include/mat/config.h | Adds MATSDK_NO_DEFAULT_HTTP_CLIENT override to disable HAVE_MAT_DEFAULT_HTTP_CLIENT. |
| lib/CMakeLists.txt | Implements bundled/minimal SQLite build, link selection, and installation/export logic. |
| docs/building-with-vcpkg.md | Documents the new vcpkg minimal-sqlite feature and behavior. |
| CMakeLists.txt | Adds MATSDK_MINIMAL_SQLITE and BUILD_CURL_HTTP_CLIENT; adjusts vcpkg SQLite discovery flow. |
| cmake/MSTelemetryConfig.cmake.in | Skips unofficial-sqlite3 dependency discovery when SQLite is bundled. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # A static libmat propagates its PRIVATE static dependencies through its link | ||
| # interface, so the bundled SQLite must be part of the same export set and be | ||
| # installed alongside mat for downstream find_package() consumers to link. | ||
| set(MATSDK_INSTALL_TARGETS mat) | ||
| if(MATSDK_BUNDLE_SQLITE AND TARGET sqlite3_bundled) | ||
| list(APPEND MATSDK_INSTALL_TARGETS sqlite3_bundled) | ||
| endif() |
| find_package(ZLIB REQUIRED) | ||
| find_package(nlohmann_json CONFIG REQUIRED) | ||
| message(STATUS "Using vcpkg-provided sqlite3, zlib, nlohmann-json") | ||
| message(STATUS "Using vcpkg-provided zlib, nlohmann-json") |
- lib/CMakeLists.txt: only add sqlite3_bundled to the install/export set when mat is a STATIC_LIBRARY. A shared mat absorbs the private SQLite into libmat and does not propagate the PRIVATE dependency, so exporting the separate archive there was unnecessary and could let a consumer link a second SQLite copy. For a static mat the archive must stay exported because the static library propagates its PRIVATE dependency through its link interface (\$<LINK_ONLY:...>). - CMakeLists.txt: make the vcpkg dependency-mode status message reflect whether the external sqlite3 package or the private minimal SQLite is used. Verified with an isolated CMake export test: static mat exports m+sq (consumer linking only the namespaced lib resolves sq); shared mat exports only m and install(EXPORT) succeeds with sq excluded. Re-ran the vcpkg [core,minimal-sqlite] consumer (static x64-linux): 10/10. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Correct the inline comment: a PRIVATE link of the bundled SQLite suppresses propagation of its include dirs / compile definitions, but a static mat still propagates the archive for linking via \$<LINK_ONLY:...> (hence it is exported for static builds); a shared mat absorbs it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…efault-http-client feature Make the HTTP-client footprint a consumer choice instead of hardcoding curl[openssl]: - vcpkg.json: replace the base curl[openssl] dependency with three features -- curl-openssl (default; libcurl + OpenSSL), curl-mbedtls (libcurl + mbedTLS), and no-default-http-client (omit the built-in client). curl-openssl is a default feature so a plain install keeps current behavior; [core,no-default-http-client] drops curl entirely. - portfile.cmake: map the no-default-http-client feature to -DBUILD_CURL_HTTP_CLIENT=OFF via INVERTED_FEATURES. - CMakeLists.txt: when the built-in client is enabled in vcpkg mode but libcurl is not found, emit a clear FATAL_ERROR pointing at the curl-openssl/curl-mbedtls/ no-default-http-client features (instead of a bare find_package failure). - docs: document the size ladder (OpenSSL ~10.6MB / mbedTLS ~4.4MB / no-curl ~1.4MB) and the exact mbedTLS recipe -- crucially, the consumer must ALSO list curl with default-features:false at the top level, because vcpkg only honors curl's default-features:false for top-level dependencies (otherwise curl's ssl default pulls OpenSSL in transitively alongside mbedTLS). Validated on WSL with vcpkg: default resolves curl[openssl]+sqlite3; the documented mbedTLS recipe builds with mbedTLS only (no libssl/libcrypto, libcurl carries no OpenSSL symbols) and the consumer runs; [core,no-default-http-client] drops curl from the graph; [core,minimal-sqlite,no-default-http-client] drops curl and the external sqlite3. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ard (Copilot review) - CMakeLists.txt: in vcpkg mode use find_package(CURL CONFIG QUIET) and gate on TARGET CURL::libcurl. Forcing CONFIG selects the vcpkg-provided CURLConfig (which defines the imported target) rather than the module FindCURL, which on some CMake versions does not define CURL::libcurl and would fail at link. - portfile.cmake: fail fast when more than one of curl-openssl/curl-mbedtls/ no-default-http-client is selected. vcpkg cannot express mutual exclusivity, so a consumer requesting e.g. curl-mbedtls without [core] keeps the default curl-openssl and would union both TLS backends; the guard now errors with guidance to use the [core,...] form. Validated: the guard passes single selections and fires on curl-openssl+curl-mbedtls and curl-openssl+no-default-http-client; the default (curl-openssl) vcpkg consumer still configures via CURL CONFIG, links, and runs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Drop the ability to build without the built-in libcurl HTTP client. That option only benefited consumers that already ship their own IHttpClient; the SDK's named consumers (and Apple/Windows, which use NSURLSession/WinInet) never needed it, and it added a fragile feature plus a config-flow opt-out. The TLS-backend selection (curl-openssl default / curl-mbedtls) and minimal-SQLite remain. - CMakeLists.txt: remove option(BUILD_CURL_HTTP_CLIENT) and the no-curl else branch; the curl HTTP client is always built on the CPP11/curl path again (keeping the find_package(CURL CONFIG) + TARGET CURL::libcurl hardening). - lib/include/mat/config.h: remove the MATSDK_NO_DEFAULT_HTTP_CLIENT -> HAVE_MAT_DEFAULT_HTTP_CLIENT opt-out. - vcpkg.json: remove the no-default-http-client feature. - portfile.cmake: remove the INVERTED_FEATURES mapping; the mutual-exclusivity guard now covers just curl-openssl vs curl-mbedtls. - docs: drop the no-curl row/section; note the size figures are worst-case (without consumer-side --gc-sections). Validated: vcpkg.json parses, CMake configures cleanly, and the mat target builds and links with the curl client compiled in. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Updated scope: removed the no-curl ( |
…kg packages
macOS/iOS ship libsqlite3 and libz as system libraries, so pulling and statically
linking the vcpkg sqlite3 + zlib packages added ~1 MB of redundant code to Apple
binaries. Link the system libraries instead -- consistent with the SDK's own Swift
Package (which links .linkedLibrary("sqlite3"/"z")) and with how analogous telemetry
SDKs (e.g. sentry-native) gate these deps off Apple platforms.
- vcpkg.json: gate the zlib dependency and the system-sqlite feature's sqlite3
dependency to "!osx & !ios" so they are not installed on Apple.
- CMakeLists.txt: on APPLE in vcpkg mode, find_package(SQLite3)/find_package(ZLIB)
(CMake's modules resolve to the OS libraries) and set MATSDK_APPLE_SYSTEM_DEPS.
- lib/CMakeLists.txt: link SQLite::SQLite3 + ZLIB::ZLIB on Apple; never bundle a
private SQLite on Apple (MATSDK_MINIMAL_SQLITE is a no-op there since the system
lib is already smaller).
- MSTelemetryConfig.cmake.in: re-find system SQLite3 on Apple, the vcpkg
unofficial-sqlite3 elsewhere.
- docs: note the Apple system-lib behavior.
Validated: non-Apple paths unchanged -- Linux vendored mat builds, and the Linux
vcpkg consumer's generated config resolves unofficial-sqlite3 (if(OFF)) and runs.
The Apple build itself needs validation on macOS/iOS CI (no Mac available here);
the risk is whether find_package(SQLite3) resolves the system lib under the vcpkg
Apple triplets' find-root settings.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| message(FATAL_ERROR | ||
| "libcurl was not found. The vcpkg port provides the curl HTTP client " | ||
| "through the curl-openssl (default) or curl-mbedtls feature. Install " | ||
| "cpp-client-telemetry with its default features or with [core,curl-mbedtls].") |
The curl-openssl/curl-mbedtls guidance in the port's fatal-error messages and docs recommended cpp-client-telemetry[core,curl-mbedtls], but the [core,...] form (default-features:false) drops ALL default features -- including system-sqlite -- not just curl-openssl. That example yields a config-time failure with no SQLite backend selected. Update both FATAL_ERROR messages (portfile.cmake mutual-exclusivity guard, CMakeLists.txt libcurl-not-found) and the docs prose to show a complete, working feature set ([core,curl-mbedtls,system-sqlite]) and to note that [core,...] also drops system-sqlite, so a SQLite backend must be re-selected. Files: tools/ports/cpp-client-telemetry/portfile.cmake, CMakeLists.txt, docs/building-with-vcpkg.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two doc/manifest accuracy fixes from Copilot review: - The dependency table labeled libcurl 'optional' for non-Windows/non-Apple, but since the no-curl option was removed, Linux/Android vcpkg builds always require curl (only the TLS backend is selectable). Relabel as required. - The curl-mbedtls feature description recommended [core,curl-mbedtls], which drops all defaults (incl. system-sqlite); note that a SQLite backend must be re-selected to avoid a configure-time failure. Files: docs/building-with-vcpkg.md, tools/ports/cpp-client-telemetry/vcpkg.json Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| message(FATAL_ERROR | ||
| "SQLite was not found and the minimal SQLite is not enabled. The vcpkg " | ||
| "port provides SQLite through one of two features: 'system-sqlite' " | ||
| "(default, links the external sqlite3 package) or 'minimal-sqlite' " | ||
| "(builds a private feature-stripped SQLite). Install " | ||
| "cpp-client-telemetry with its default features, or with " | ||
| "[core,minimal-sqlite]. For a direct CMake build, pass " | ||
| "-DMATSDK_MINIMAL_SQLITE=ON or ensure unofficial-sqlite3 is discoverable.") |
| if(@MATSDK_APPLE_SYSTEM_DEPS@) | ||
| find_dependency(SQLite3) | ||
| elseif(NOT @MATSDK_BUNDLE_SQLITE@) | ||
| find_dependency(unofficial-sqlite3 CONFIG) | ||
| endif() |
Five fixes from the Copilot review on the curl/SQLite feature interactions (all stem from vcpkg's [core,...] form dropping ALL default features, not just one): - portfile.cmake: fail fast on Linux/Android when no curl TLS backend is selected (verified: a real vcpkg install of [core,minimal-sqlite] now stops at the portfile with a complete [core,curl-openssl,system-sqlite] example, instead of a later, opaque libcurl-not-found error). - CMakeLists.txt libcurl message: show how to re-select a curl backend (not only mbedTLS) under [core,...], alongside a SQLite backend. - CMakeLists.txt SQLite message: include the valid [core,system-sqlite] path, not only [core,minimal-sqlite]. - MSTelemetryConfig.cmake.in: find_dependency(CURL CONFIG) so the exported package config uses the vcpkg CURLConfig that defines CURL::libcurl (the target MSTelemetryTargets references), matching the unofficial-sqlite3/ nlohmann_json CONFIG siblings and the root CMakeLists CURL CONFIG lookup. - docs: minimal-sqlite manifest example re-selects curl-openssl so the Linux/Android manifest actually configures. Files: CMakeLists.txt, cmake/MSTelemetryConfig.cmake.in, docs/building-with-vcpkg.md, tools/ports/cpp-client-telemetry/portfile.cmake Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The mutual-exclusivity check (curl-openssl vs curl-mbedtls) previously ran on all platforms. Since curl-openssl is a default feature and the curl dependency is platform-filtered to linux|android, a cross-platform manifest that enables curl-mbedtls without [core] would falsely fail the port on Windows/macOS/iOS -- where curl is not used (WinInet / Apple HTTP) and neither feature pulls curl. Wrap both the mutual-exclusivity (count>1) and no-curl (count==0) checks in a single VCPKG_TARGET_IS_LINUX/ANDROID block so they only fire where the curl backend selection is actually meaningful. Verified on x64-linux: [core,minimal- sqlite] still fails with the no-curl message, and [curl-mbedtls] (no core) still fails with the mutual-exclusivity message. Files: tools/ports/cpp-client-telemetry/portfile.cmake Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The port tests built the SDK from the portfile's pinned vcpkg_from_github REF
(v3.10.161.1), so they never exercised the PR's own source -- and the macOS/iOS
jobs failed because this PR's manifest drops the Apple sqlite3/zlib packages
while the old pinned source still calls find_package(unofficial-sqlite3)
unconditionally (the Apple system-libs branch only exists in the PR source).
Add an opt-in MATSDK_VCPKG_SOURCE_DIR hook to portfile.cmake: when set, the port
builds that local source; when unset (production installs), the pinned release is
downloaded as before, so the published port behavior is unchanged. The five
tests/vcpkg/* scripts set it to the repo root so the port tests validate the
actual source + manifest together.
Verified on Linux (x64-linux): the port now builds the working-tree SDK and the
consumer passes 10/10; the macOS/iOS jobs will exercise the Apple system-libs
branch (find_package(SQLite3)/ZLIB) instead of the dropped vcpkg packages.
Files: tools/ports/cpp-client-telemetry/portfile.cmake,
tests/vcpkg/test-vcpkg-{linux,macos,ios,android}.sh,
tests/vcpkg/test-vcpkg-windows.ps1
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
On Windows, vcpkg runs portfiles in a sanitized environment and strips custom variables unless allow-listed via VCPKG_KEEP_ENV_VARS. Without it the portfile never saw MATSDK_VCPKG_SOURCE_DIR and silently fell back to the pinned release (v3.10.161.1), so the Windows port test validated the old release instead of the PR source while still reporting PASS. Allow-list the variable so the test builds the working tree, matching the Linux/macOS scripts (POSIX vcpkg passes the variable through, so they need no change). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Summary
Three changes that reduce the SDK's binary footprint and dependency surface when it is embedded into a host application (e.g. ONNX Runtime / Foundry Local). All are opt-in or platform-scoped, so the default vcpkg install and the default CMake build are unchanged (built-in curl HTTP client + external
sqlite3).1.
MATSDK_MINIMAL_SQLITE(default OFF) — private feature-stripped SQLiteThe SDK uses SQLite only for its offline event-storage cache (plain tables, transactions, WAL, autovacuum/
VACUUM, a few PRAGMAs, and one custom UTF-8 function).MATSDK_MINIMAL_SQLITE=ONcompiles a private SQLite from the vendored amalgamation with a set of amalgamation-safe strip flags (single source of truth:MATSDK_SQLITE_MINIMAL_DEFS), removing the externalsqlite3dependency and shrinking SQLite ~10.2% (.text) / ~12.5% (stripped object).minimal-sqlitefeature;sqlite3moves into a defaultsystem-sqlitefeature, so[core,minimal-sqlite]drops the external dependency.SQLITE_OMIT_AUTOINIT(theskipSqliteInitAndShutdownconfig lets the host skip the SDK's explicitsqlite3_initialize(), which the host cannot do against a private SQLite) andSQLITE_DEFAULT_MEMSTATUS=0(the SDK arms a soft heap limit viasqlite3_soft_heap_limit64()that is only enforced while memory statistics are on).matpropagates the private SQLite through its link interface, so it is exported/installed asMSTelemetry::sqlite3_bundled(static-only; a sharedmatabsorbs it). The footprint doc notes the static-absorption symbol-visibility caveat.2. Selectable curl TLS backend —
curl-openssl(default) /curl-mbedtlsOn Linux/Android the built-in HTTP client links libcurl, which pulls a TLS backend. The port now exposes the choice as two mutually-exclusive vcpkg features:
curl-openssl(default) —curl[core,openssl], unchanged behavior.curl-mbedtls—curl[core,mbedtls]for a smaller TLS footprint.Because vcpkg only honors
default-features: falsefor top-level dependencies, selecting mbedTLS requires the[core,curl-mbedtls,...]form plus a top-levelcurl[core,mbedtls]override in the consumer manifest (documented, with a verified recipe). The port fails fast on Linux/Android if both or neither curl feature is selected; these guards are scoped to Linux/Android, since Windows (WinInet) and Apple (NSURLSession) do not link curl.3. Apple: link the system
libsqlite3/libzon macOS/iOSmacOS/iOS ship
libsqlite3andlibz, so the port no longer pulls the vcpkgsqlite3/zlibpackages there (gated!osx & !iosin the manifest). The build usesfind_package(SQLite3)/find_package(ZLIB)(SQLite::SQLite3/ZLIB::ZLIB) — the same system libraries the SDK's Swift Package already links — removing ~1 MB of redundant bundled binaries on Apple.minimal-sqlitetherefore has no effect on Apple.Footprint impact
Validation
MATSDK_MINIMAL_SQLITE=ON[core,minimal-sqlite]consumerMSTelemetry::sqlite3_bundled, runs 10/10; external sqlite3 droppedsystem-sqlite+curl-openssl)curl-mbedtlsrecipevcpkg install --dry-run+ full build resolve tocurl[core,mbedtls], no OpenSSL built[core,minimal-sqlite]fails fast (no curl backend);[curl-mbedtls]w/o[core]fails (mutual-exclusive); both messages give complete examplessqlite3_bundled+matcompile/link.text/ −12.5% stripped objectNote: the Apple system-libs path uses CMake's standard
FindSQLite3/FindZLIB; it is validated by the macOS/iOS CI jobs (cannot be built locally without a Mac).Reviewed for correctness, security (the SQLite strip is security-neutral-to-positive —
SQLITE_DQS=0hardens identifier handling), and design; driven through the Copilot review loop.Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com