bug: MVStore backend can grow uncontrollably

[DarkAtra]

Nitrite's MVStore backend can grow to hundreds of MB or even GB despite containing only a few hundred KB of live data. Frequent updates to existing documents leave thousands of partially-live chunks with mostly obsolete pages. Since Nitrite sets autoCompactFillRate(0), these chunks are never automatically compacted, so the database file keeps growing during long-running applications. Rebuilding/copying the same database reduces the file size drastically (down to a few KB).

In my project, database records are updated every 5 seconds and the database file grew to ~800MB over the past few months. The MVStore's layoutMap looks like this after a few days:

chunk:128037f,block:d2,len:15,pages:83,livePages:1,max:16950,liveMax:800,map:11,next:e7,root:4a00dfc0004e0403,time:a8fa9573c,unusedAtVersion:128038f,version:128037f,toc:141cb,occupancy:ffffffff7fffffffffffffffffffffff07

chunk:155513d,block:46,len:6,pages:57,livePages:9,max:5fc0,liveMax:830,map:12,next:4c,root:55544f4000106a85,time:c8795d349,unusedAtVersion:1555146,version:155513d,toc:4eb6,occupancy:ffffffffffffffff8fe363

chunk:155513e,block:4c,len:b,pages:71,livePages:1,max:c920,liveMax:200,map:12,next:57,root:55544f8000289d05,time:c8795f5ee,unusedAtVersion:1555147,version:155513e,toc:a96f,occupancy:ffffffffffffffffffffffffffbf01

chunk:155513f,block:e7,len:10,pages:a2,livePages:1,max:12030,liveMax:30,map:12,next:f7,root:55544fc0003cd345,time:c8795fa58,unusedAtVersion:1555147,version:155513f,toc:f9da,occupancy:ffffffffffffffffffffffffffffffffffffff7f03

[...]

The layout contains around ~6400 chunks and there are barely any live pages in the chunks. The occupancy metric also suggests that most of the chunk is just obsolete data. My database actually only contains around 100 live documents with barely any content. The expected database file size would be less than 1MB.

AI suggested that the issue might be related to the disabled auto compaction builder.autoCompactFillRate(0) in MVStoreUtils (#41). Since i couldn't find any other form of chunk compaction in the nitrite code, this explanation sounds plausible to me.

Reproducer: https://github.com/DarkAtra/nitrite-mvstore-file-size-issue

Nitrite Version: 4.4.3
MVStore Version: 2.4.240