Fix sync loop for too long path

[harminius]

Fixes MerginMaps/qgis-plugin#932

This adds an extended-path prefix to all Python filesystem operations on Windows so that it can work with files&folder > 260 characters. The prefix is not in the responses and won't reach the server DB.

[harminius]

client-log.txt

[harminius]

db paths look fine

[coveralls]

Coverage Report for CI Build 34469353898

Coverage decreased (-0.02%) to 83.371%

Details

• Coverage decreased (-0.02%) from the base build.
• Patch coverage: 13 uncovered changes across 5 files (112 of 125 lines covered, 89.6%).
• 3 coverage regressions across 2 files.

Uncovered Changes

File	Changed	Covered	%
mergin/utils.py	19	12	63.16%
mergin/client.py	11	9	81.82%
mergin/client_pull.py	17	15	88.24%
mergin/fs.py	26	25	96.15%
mergin/merginproject.py	42	41	97.62%
Total (7 files)	125	112	89.6%

Coverage Regressions

3 previously-covered lines in 2 files lost coverage.

File	Lines Losing Coverage	Coverage
mergin/client_pull.py	2	82.57%
mergin/client.py	1	88.65%

---

Coverage Stats

Relevant Lines:	5713
Covered Lines:	4763
Line Coverage:	83.37%
Coverage Strength:	0.83 hits per line

---

💛 - Coveralls

[varmar05]

So this is only for local handling of files but it will never end up in relative paths for files in DB (we need this to work for all clients)?

[harminius]

yes, rel_path() handles the prefixed path correctly and removes it

[varmar05]

Using some patch function for a lot of os.paths hanging around in the code seems fragile to me. It can easily slip and we might forgot to use in future. I think we need to come up with some generic handler, single place which will make sure all paths are correct in whole code base.

[harminius]

Introduced wrapper to geodiff in bcc7ae6
and put all file system calls in one place 2051d4f

[harminius]

Let's try to use \\?\-prefixed paths for geodiff calls and other (all) file I/O.

[harminius]

PR history:

Debugging info:
Pull works as py-client saves the diffs to tmp dir
However, there is always a check if there's something to push after the pull.

Push treats every gpkg with editing mode enabled as well as with original vs. current gpkg checksum mismatch (which is quite common) as potentially updated, and we rely on geodiff to calculate the real changes. Geodiff creates an empty diff file even when there is no change.
Geodiff's create_changeset() fails because it's not possible to create a file due to the Windows path length limit. It then pushes the whole file. Then we want to apply the push changes to the basefile. So geodiff removes the original geopackage and copies the current copy, which apparently changes the checksum, and we're caught in the loop. ➿

Site	Feature
merginproject.py get_push_changes	push
merginproject.py get_local_delta	pull, local changes detection
client_pull.py get_download_diff_files	pull, v1 fallback
client_pull.py pull_project_async v2 loop	pull, v2
client_pull.py download_diffs_async	get_file_diff / version viewer / reports

This is a minimal fix for the issue. There are other enhancements/issue mitigations:

1. Shorten diff name
   Store diff files locally under a short, flat name, decoupled from the file's path in the project — e.g. .mergin/diffs/ instead of .mergin/-diff-. Fixes works esp. for files nested deep in the subfolders. This does not require server changes - the server doesn't care what we call the local file.

Path	Length
Live file <rel>	P + R + 1
Basefile .mergin<rel> — already mandatory today	P + R + 9
Diff today .mergin<rel>-diff-	P + R + 51
Diff flattened .mergin\diffs<uuid>	P + 51 (constant)
Same, 16-char id	P + 31 (constant)

2. Safeguard for any infinite sync loop
   sync_project_generator's while has_changes has no iteration limit. Track, per file, how many consecutive iterations it was pushed as a full upload with no diff; if the same versioned file does this N times in a row (e.g. 2–3), raise a ClientError naming the file instead of continuing. This is cause-agnostic - it also protects against any other future reason a changeset can't be computed.

3. Long path error workaround
   Use \\?\-prefixed paths for geodiff calls and other file I/O.

4. Use a temp folder for push diffs, as it is used for pull diffs already.