hindsight export currently loads ALL entries from the source db into memory before dumping them into the destination.
This is reeeal slow on a large dataset (1+ months).
proposed solution: query 1000 arbs at a time, export them as you go; maximize parallelization of load/save
- file export will need modification to append to the file, also thread-safe lock on the file ofc
- db ops can likely be done w/ parallel connections, as there won't be any write conflicts coming from the source
hindsight exportcurrently loads ALL entries from the source db into memory before dumping them into the destination.This is reeeal slow on a large dataset (1+ months).
proposed solution: query 1000 arbs at a time, export them as you go; maximize parallelization of load/save