Use magic for diff classification

[andrew]

Use github.com/git-pkgs/magic to classify complete extracted file content before generating line diffs. Keep the existing is_binary result and restrict line diffs to UTF-8 text, so binary signatures, malformed text, disallowed controls, late NUL bytes, and UTF-16 are not converted to strings.

Add coverage for Unicode and BOM-marked UTF-8 text, PDF signatures without NUL, invalid UTF-8, controls, early and late NUL bytes, and UTF-16.

[Copilot]

Pull request overview

This PR updates the archive diffing logic to use github.com/git-pkgs/magic for classifying full extracted file contents before generating line diffs, aiming to only produce line diffs for UTF-8 text and avoid string conversion for binary signatures and malformed/unsupported text.

Changes:

• Add github.com/git-pkgs/magic and use it to decide whether a file is diffable as UTF-8 text before generating diffs.
• Replace the previous NUL-byte heuristic with magic-based detection (isDiffableText).
• Expand tests to cover Unicode, UTF-8 BOM, PDF signature without NUL, invalid UTF-8, disallowed controls, early/late NUL, and UTF-16.

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated no comments.

File	Description
go.mod	Adds the github.com/git-pkgs/magic dependency and converts the single require into a require block.
go.sum	Records checksums for the new magic dependency.
diff/diff.go	Switches binary/text detection to magic and gates diff generation based on diffable UTF-8 text.
diff/diff_test.go	Adds/updates test cases for the new detection behavior and binary-file diff suppression.

Suppressed comments (2)

diff/diff.go:157

• Same is_binary semantic issue in the modified-file path: IsBinary is set when content is not diffable text, which includes UTF-16 and magic.KindUnknown (invalid UTF-8 without NUL). If is_binary is intended to mean actual binary content (as in the PR description), set it only when magic.Detect(...).Kind == magic.KindBinary, and separately decide whether to generate a line diff based on KindText + UTF-8.

		if !isDiffableText(oldContent) || !isDiffableText(newContent) {
			fd.IsBinary = true
		} else {
			diffText, added, deleted := generateUnifiedDiff(path, oldContent, newContent)
			fd.Diff = diffText

diff/diff.go:137

• PR description says to keep the existing is_binary result while restricting line diffs to UTF-8. The current logic sets IsBinary=true for any non-diffable content (including magic.KindUnknown invalid UTF-8 without NUL and UTF-16 text), which changes the meaning of is_binary from “binary” to “diff suppressed”. Consider only setting IsBinary when magic.Detect(content).Kind == magic.KindBinary, and separately gating diff generation on KindText + UTF-8.

This issue also appears on line 153 of the same file.

			if !isDiffableText(content) {
				fd.IsBinary = true
			} else {
				fd.Diff = generateAddedDiff(path, content)
				fd.LinesAdded = countLines(content)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.