Fix progress bar on native uploads and proxy test implementation #48
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduces a new integration test to verify native upload functionality with a large file. Also refactors file list formatting for improved readability in existing tests.
Added the proxy parameter to calls to upload_files and upload_files_parallel in DVUploader to ensure proxy settings are used during uploads. Also improved import ordering for consistency.
Introduced _ProgressFileWrapper to wrap file-like objects and update a rich progress bar during file uploads. Improved error handling in upload retries and added an assertion for file handler presence. Refactored code for clarity and consistency.
Expanded the testing section to include instructions for running all tests, specific tests, and non-expensive tests using pytest. This provides clearer guidance for contributors on how to execute different test scenarios.
Introduces a pytest fixture to start a local HTTP proxy using proxy.py for tests requiring proxy support. Updates the native upload integration test to use the new fixture, improving reliability and isolation of proxy-dependent tests.
Included proxy.py version 2.4.4 in the main dependencies to support proxy-related functionality.
Removed unused AsyncGenerator import from nativeupload.py and cleaned up unused stdout variable in tests/conftest.py for better code clarity.
Introduces an optional proxy parameter to dataset file retrieval functions and methods in dvuploader.py and utils.py, allowing HTTP requests to be routed through a specified proxy. This enhances flexibility for users operating behind network proxies.
Introduces a 'proxy' parameter to native_upload, _update_metadata, and _retrieve_file_ids functions, allowing API requests to be routed through a specified proxy. This enhances flexibility for deployments requiring network routing via proxies.
Eliminates the Squid service setup from the GitHub Actions test workflow, simplifying the build job configuration.
Deleted the http_proxy_server fixture and its helper from conftest.py, and commented out the proxy upload integration test in test_native_upload.py. The proxy functionality has been verified manually, but the automated test setup was unreliable.
Cleaned up the tests/conftest.py file by removing unused imports: signal, socket, subprocess, sys, and time.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces several enhancements and fixes across the codebase, mainly focusing on improving upload progress tracking, proxy support for testing, code organization, and documentation. The changes also include new test coverage for large file uploads and proxy scenarios.
Native upload progress tracking:
_ProgressFileWrapperclass innativeupload.pyto wrap file-like objects and update the Rich progress bar during file uploads, allowing accurate progress reporting for large files. This wrapper is now used in the native upload flow. [1] [2]_single_native_uploadto handle additional network exceptions (httpx.ReadError,httpx.RequestError), improving robustness against transient network failures.Proxy support for integration tests:
http_proxy_serverpytest fixture inconftest.pythat launches a local HTTP proxy usingproxy.pyfor tests requiring proxy support. This enables reliable testing of proxy scenarios.proxy.pyto the test dependencies inpyproject.toml.Closing issues