Skip to content

Mark a test which we know fails when connected to the test server #1787

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
PGijsbers merged 5 commits into from
Nov 5, 2026
Merged

Mark a test which we know fails when connected to the test server #1787

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5ece582
Mark a test which we know fails when connected to the test server
PGijsbers Nov 4, 2026
463b5dc
use file upload instead of url upload
PGijsbers Nov 5, 2026
3bf61b1
use dataset upload instead of url upload
PGijsbers Nov 5, 2026
3c92640
use services with MinIO instead of SeaweedFS
PGijsbers Nov 5, 2026
b6b329e
Update dataset name to reflect data uploaded
PGijsbers Nov 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,9 @@ jobs:
if: matrix.os == 'ubuntu-latest'
id: clone-services
run: |
git clone --depth 1 https://github.com/openml/services.git
# 2026-11-05: Services switched to SeaweedFS and the MinIO client cannot handle the new URLs.
# We use the old stack until we have migrated away from the MinIO client
git clone --revision dd8b98a9e0450bf487f973e1711c8b00dc022294 --depth 1 https://github.com/openml/services.git

- name: Start Docker Services
id: start-services
Expand Down
21 changes: 17 additions & 4 deletions tests/test_api/test_datasets.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,14 +126,27 @@ def test_v1_get_qualities(dataset_v1):
reason="Test requires admin key. Set OPENML_TEST_SERVER_ADMIN_KEY environment variable.",
)
@pytest.mark.test_server
def test_v1_status_update(dataset_v1):
def test_v1_status_update(dataset_v1, static_cache_dir):
openml.config.apikey = TestBase.admin_key
arff_file_path = (
static_cache_dir
/ "org"
/ "openml"
/ "test"
/ "data"
/ "download"
/ "1666876"
/ "phpFsFYVN"
/ "body.arff"
)
new_dataset = OpenMLDataset(
f"TEST-{time.time()!s}-UploadTestWithURL",
f"TEST-{time.time()!s}-UploadTestWithFile",
"test",
"ARFF",
data_format="arff",
version=1,
url="https://www.openml.org/data/download/61/dataset_61_iris.arff",
licence="public",
default_target_attribute="class",
data_file=arff_file_path,
)
Comment on lines +129 to 150

@PGijsbers PGijsbers Nov 5, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

URL uploading is now disabled on the server. This test doesn't really care about the particulars of the dataset being uploaded, so we just switch it to file upload. (I think this test isn't even really appropriate for openml-python to begin with, but that's not something I want to discuss as part of the PR but rather look at sometime separately).

new_dataset.publish()
_status_update_check(dataset_v1, new_dataset.dataset_id, "deactivated")
Expand Down
13 changes: 11 additions & 2 deletions tests/test_datasets/test_dataset_functions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -547,10 +547,18 @@ def _assert_status_of_dataset(self, *, did: int, status: str):
@pytest.mark.test_server
def test_data_status(self):
arff_file_path = (
self.static_cache_dir / "org" / "openml" / "test" / "datasets" / "2" / "dataset.arff"
self.static_cache_dir
/ "org"
/ "openml"
/ "test"
/ "data"
/ "download"
/ "1666876"
/ "phpFsFYVN"
/ "body.arff"
)
dataset = OpenMLDataset(
"anneal",
f"TEST-{time.time()!s}-UploadTestWithFile",
"test",
data_format="arff",
version=1,
Expand All @@ -559,6 +567,7 @@ def test_data_status(self):
data_file=arff_file_path,
)
dataset.publish()
dataset.publish()
TestBase._mark_entity_for_removal("data", dataset.dataset_id)
TestBase.logger.info(
f"collected from {__file__.split('/')[-1]}: {dataset.dataset_id}",
Expand Down
13 changes: 12 additions & 1 deletion tests/test_flows/test_flow.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,7 @@ def test_get_structure():


def test_tagging():
# Not possible as a marker since autouse fixtures change the configuration
flows = openml.flows.list_flows(size=1)
flow_id = flows["id"].iloc[0]
flow = openml.flows.get_flow(flow_id)
Expand All @@ -136,7 +137,17 @@ def test_tagging():
tag = f"test_tag_TestFlow_{unique_indicator}"
flows = openml.flows.list_flows(tag=tag)
assert len(flows) == 0
flow.push_tag(tag)
try:
flow.push_tag(tag)
except openml.exceptions.OpenMLServerException as e:
# pytest.mark.xfail doesn't work because the autouse fixtures change server config
reason = "There is an issue with Elasticsearch on the test server since Oct 2026."
test_server = openml.config.get_server_base_url() == "https://test.openml.org"
elastic_search_error = "Elastic Search" in e.message
if test_server and elastic_search_error:
pytest.xfail(reason)
raise

flows = openml.flows.list_flows(tag=tag)
assert len(flows) == 1
assert flow_id in flows["id"].values
Expand Down
Loading