More efficient alternative to PQgetCopyData.
#1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jtv
force-pushed
the
PQgetNextCopyData
branch
from
8e7c171 to
15a8886
Compare
jtv
force-pushed
the
PQgetNextCopyData
branch
2 times, most recently
from
4f33275 to
c31215c
Compare
jtv
changed the title
PQgetNextCopyData.PQgetCopyData.
jtv
force-pushed
the
PQgetNextCopyData
branch
4 times, most recently
from
9f17c54 to
1d05c52
Compare
Saves CPU time (and power usage) by skipping the allocation and deallocation of buffers for each row. The downside is that the data is only valid for one iteration, but that's probably all that most users of COPY OUT need anyway. The callback-based `PQhandleCopyData` reduces CPU usage by more than half in my toy benchmark. Wall-clock time stayed about the same as it was. The disadvantage is that this breaks with existing libpq API style. But when performance matters, that might be worth it.
Pass the buffer to the callback as `const`. Instead of copying `pqGetCopyData3` as `pqHandleCopyData3`, re-implement `PQgetCopyData` on top of the callback-based API, now renamed to `pqGetCopyData3`. Use `fwrite` instead of `printf` so we don't need zero-terminated strings.
jtv
changed the title
PQgetCopyData.PQgetCopyData.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Saves CPU time (and power usage) by skipping the allocation and
deallocation of buffers for each row. The downside is that the data
is only valid for one iteration, but that's probably all that most
users of COPY OUT need anyway.
The new function,
PQhandleCopyData, reduces CPU usage to lessthan half of what
PQgetCopyDatauses.The disadvantage is that it breaks with existing libpq API style.
But when performance matters, that might be worth it.