Define the get_instance_status and get_job_status() methods of the BenchmarkLauncher

[R-Palazzo]

Resolve #570
CU-86b8tmh3y

If you’d like to test the feature end to end, here’s a code snippet you can use. You only need to update the credentials and the output_destination so it doesn’t point to the internal benchmark destination.
To use SDV Enterprise synthesizers, you’ll need to have sdv-enterprise installed in the local environment where you run the code.

from sdgym._benchmark_launcher import BenchmarkLauncher
from sdgym.run_benchmark.run_benchmark import _get_config
import os
import json

service_account = {
    "type": "service_account",
    "project_id": "sdgym-337614",
    "private_key_id": "",
    "private_key": "",
    "client_email": "",
    "client_id": "",
    "auth_uri": "",
    "token_uri": "",
    "auth_provider_x509_cert_url": "",
    "client_x509_cert_url": "",
    "universe_domain": "",
    "gcp_project": "sdgym-337614",
    "gcp_zone": "us-central1-a"
}

os.environ["GCP_SERVICE_ACCOUNT_JSON"] = json.dumps(service_account)
os.environ["AWS_ACCESS_KEY_ID"] = ''
os.environ["AWS_SECRET_ACCESS_KEY"] = ''
os.environ['SDV_ENTERPRISE_USERNAME'] = ''
os.environ['SDV_ENTERPRISE_LICENSE_KEY'] = ''

aws_access_key_id = os.getenv('AWS_ACCESS_KEY_ID')
aws_secret_access_key = os.getenv('AWS_SECRET_ACCESS_KEY')
modality = 'single_table'

config = _get_config(modality)
launcher = BenchmarkLauncher(config)
launcher.launch()
job_status = launcher.get_job_status()
instance_status = launcher.get_instance_status()

[sdv-team]

Task linked: CU-86b8tmh3y SDGym - Define the get_instance_status and get_job_status() methods of the BenchmarkLauncher #570

[codecov]

Codecov Report

❌ Patch coverage is 97.58065% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.98%. Comparing base (52f57fb) to head (195d83b).

Files with missing lines	Patch %	Lines
sdgym/_benchmark_launcher/benchmark_launcher.py	96.77%	3 Missing ⚠️

Additional details and impacted files

@@                      Coverage Diff                       @@
##           feature/benchmark_launcher     #583      +/-   ##
==============================================================
+ Coverage                       84.58%   84.98%   +0.39%     
==============================================================
  Files                              40       40              
  Lines                            3418     3529     +111     
==============================================================
+ Hits                             2891     2999     +108     
- Misses                            527      530       +3     

Flag	Coverage Δ
integration	45.31% <0.00%> (-1.48%)	⬇️
unit	80.61% <97.58%> (+0.54%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[amontanez24]

don't we just need to do this once in init?

[R-Palazzo]

Good point. Since we don’t expose any method to update the config after init, validating it once there should be enough. Updated in c33a1e5

[amontanez24]

what's the scenario where invalid instance names are being provided?

[R-Palazzo]

We raise an error for invalid instance names currently:

SDGym/sdgym/_benchmark_launcher/benchmark_launcher.py

Line 166 in 92f8165

raise ValueError(