Add index schema 2.1 with delta support

[JohnMcPMS]

📖 Description

Adds a new schema version with support for generating and consuming delta databases. When using the new schema, the general flow is:

1. Update the index as normal; when appropriate, mark a baseline.
2. When making future updates, provide the baseline and delta output locations before packaging prep.
3. This will produce the normal package ready index and a delta from the baseline.

This allows the existing flow to be maintained while also producing deltas. A future change will introduce the public surface to do those interactions.

The spec (#6500) has more details, but the basic concept is that we record the difference between the baseline and the new index, keeping package rows fixed so that we can merge them easily at runtime. The delta uses new table names and creates views that merge the tables from the delta and baseline. This means that the code to read the index doesn't need to change.

🔗 References

Spec: #6500

🔍 Validation

Added significant new test coverage for delta scenarios.

Microsoft Reviewers: Open in CodeFlow

[github-actions]

check-spelling-bot Report

🔴 Please review

See the 📂 files view, the 📜action log, or 📝 job summary for details.

Unrecognized words (30)

alsokeep
ATTAC
brandnew
changednew
changeseq
cmdadded
cmddrop
cmdgone
cmdkeep
cmdnew
cmdold
commontag
createdatabase
deltatest
diffed
firstcmd
markasbaseline
newcmd
NEWPC
oldcmd
OLDPC
othercmd
othertest
ownertombstone
preparetables
recased
simpleview
unbackfilled
updatetracking
valuetombstone

These words are not needed and should be removed

AAD ABCD abi ACL'd AMap Amd appdata ARMNT asan Baz bitmask bluetooth boundparms brk Buf certs cgi CMSG codepage commandline constexpr Cov cswinrt CTL Dbg Dcom decompressor dedupe DEFT devhome Dns dsc ERANGE errcode errmsg errstr filemode Finalizers FULLWIDTH fuzzer GES github Hackathon HINSTANCE hlocal hmac Hyperlink ICONDIR icu idx img inet Intelli iwr JDK LCID lhs LONGLONG LPBYTE LPCWSTR LPDWORD LPSTR LPVOID LPWSTR MAJORVERSION MAXLENGTH maxvalue MDs MINORVERSION mta nlohmann NONAME NOUPDATE NTFS ofile oid oop OPTOUT outfile OUTOFMEMORY PARAMETERMAP pdb PDWORD pid PKCS pkix placeholders positionals posix pscustomobject pseudocode PSHOST publickey qword redirector regexes remoting reparse REQS rhs rowid RTTI runspace runtimes SARL savepoint Scm sid sqlite subdir subkey trimstart ttl typedef uninitialize uninstallation UNMARSHALING userprofile versioned Webserver website wildcards winreg WMI workaround Wpp wsl

Some files were automatically ignored 🙈

These sample patterns would exclude them:

^\Q.github/workflows/duplicate-surfacing.lock.yml\E$
^\Q.github/workflows/issue-closure-recommendation.lock.yml\E$

You should consider adding them to:

.github/actions/spelling/excludes.txt

File matching is via Perl regular expressions.

To check these files, more of their words need to be in the dictionary than not. You can use patterns.txt to exclude portions, add items to the dictionary (e.g. by adding them to allow.txt), or fix typos.

To accept these unrecognized words as correct, update file exclusions, and remove the previously acknowledged and now absent words, you could run the following commands

... in a clone of the git@github.com:JohnMcPMS/winget-cli.git repository
on the delta-repo branch (ℹ️ how do I use this?):

curl -s -S -L 'https://raw.githubusercontent.com/check-spelling/check-spelling/cfb6f7e75bbfc89c71eaa30366d0c166f1bd9c8c/apply.pl' |
perl - 'https://github.com/microsoft/winget-cli/actions/runs/34880367118/attempts/1' &&
git commit -m 'Update check-spelling metadata'

Forbidden patterns 🙅 (1)

In order to address this, you could change the content to not match the forbidden patterns (comments before forbidden patterns may help explain why they're forbidden), add patterns for acceptable instances, or adjust the forbidden patterns themselves.

These forbidden patterns matched content:

Should be case-(in)sensitive

\bcase (?:in|)sensitive\b

Pattern suggestions ✂️ (2)

You could add these patterns to .github/actions/spelling/patterns.txt:

# Automatically suggested patterns

# hit-count: 1 file-count: 1
# assign regex
= /[^*].*?(?:[a-z]{3,}|[A-Z]{3,}|[A-Z][a-z]{2,}).*/[gi]?(?=\W|$)

# hit-count: 1 file-count: 1
# regex choice
\(\?:[^)]+\|[^)]+\)

Alternatively, if a pattern suggestion doesn't make sense for this project, add a # to the beginning of the line in the candidates file with the pattern to stop suggesting it.

Errors, Warnings, and Notices ❌ (3)

See the 📂 files view, the 📜action log, or 📝 job summary for details.

❌ Errors, Warnings, and Notices	Count
ℹ️ candidate-pattern	2
❌ forbidden-pattern	1
⚠️ slow-file	2

See ❌ Event descriptions for more information.

If the flagged items are 🤯 false positives

If items relate to a ...

• binary file (or some other file you wouldn't want to check at all).

  Please add a file path to the excludes.txt file matching the containing file.

  File paths are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your files.

  ^ refers to the file's path from the root of the repository, so ^README\.md$ would exclude README.md (on whichever branch you're using).

• well-formed pattern.

  If you can write a pattern that would match it,
  try adding it to the patterns.txt file.

  Patterns are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your lines.

  Note that patterns can't match multiline strings.

[ranm-msft]

This had no reviews yet, so flagging that I have read it - focused on the merge/view layer (DeltaViews.cpp, DeltaGeneration.cpp) and the builder/wrapper changes rather than all 36 files.

Three things I went looking for and was glad not to find problems with:

• ATTACH binds the path as a parameter rather than interpolating it, so a baseline filename is not an injection surface.
• DatabaseSpecifier is a nice solve for the disposition problem. Having one type render the URI means the ATTACH target and the sqlite3_open_v2 target cannot drift apart, which is the bug I went looking for first.
• Association suppression is row-level rather than package-level, so a delta that changes one mapping for a package does not silently drop that package's other baseline associations. That one is easy to get wrong.

I also convinced myself the writtenRowIds collision check is not dead code - it is what lets a removed-then-re-added package reclaim its rowid - and that validating baseline affinity before attaching is deliberate, so a failed validation does not leave the caller's connection carrying a half-configured schema.

The only thing I would reconcile is on the spec side rather than here: the view SQL in #6500 no longer matches these definitions. I left the detail there.

One question: is the expectation always baseline plus a single delta, or could deltas chain? The rowid allocation reads like it assumes one delta over one baseline, and I could not tell from here whether chaining is out of scope by design.