feat: split the sources in modules so it become very verbose the source of the database - #339
Open
luabida wants to merge 17 commits into
Open
feat: split the sources in modules so it become very verbose the source of the database#339luabida wants to merge 17 commits into
luabida wants to merge 17 commits into
Conversation
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #339 +/- ##
=======================================
Coverage ? 94.88%
=======================================
Files ? 177
Lines ? 21380
Branches ? 0
=======================================
Hits ? 20286
Misses ? 1094
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…ce of the database
luabida
force-pushed
the
split-sources-verbose
branch
from
August 31, 2026 12:05
325ad3c to
5ba7f3c
Compare
- Lead all docs (README, quickstart, data-sources, api, tutorials, migration, guides) with the origin-namespaced API - Add flat -> namespaced migration guide with deprecation warning text - Re-execute getting_started_pysus.ipynb with pysus.ftp.sinasc(...) - info_table()/search() now show per-origin fetch hints (pysus.ftp.sinan) that only appear when a real namespaced fetcher exists - Add tests for the new info() call-hint column
Saude (dadosabertos.saude.gov.br) resources are frequently Latin-1 even when advertised as UTF-8. pd.read_csv assumed UTF-8, so every resource failed to parse and _fetch_saude returned an empty DataFrame for any as_dataframe=True call (both flat pysus.arboviroses and the namespaced pysus.saude.arboviroses). Extract _saude_csv_to_frame() helper that sniffs the delimiter and falls back across encodings, and add regression tests covering UTF-8, Latin-1, and corrupt inputs.
Add a download parameter (default True) threaded through fetch(), _fetch_catalog(), _fetch_origin_direct(), _fetch_ducklake() and _fetch_saude(). When download=False each fetcher resolves which files would be fetched and returns their remote paths (or Saude CSV resource URLs) without downloading anything. as_dataframe is ignored in that case because there is no local data to build a DataFrame from. Also fetch each Saude CKAN package exactly once, sharing the resolved CSV resources between listing and downloading. Ignore roadmap markdown files via the lowercase roadmap*.md pattern.
Namespaced fetchers (pysus.ftp.*, pysus.dadosgov.*, pysus.saude.*) now return a high-level, synchronous FileBag instead of a raw list of paths: - download=False -> remote FileBag of BaseRemoteFile entities (reports "(remote)" in repr; as_dataframe ignored), with _RemoteURL stand-ins for URL-only origins such as Saude. - download=True + as_dataframe=False -> local FileBag of downloaded files. - as_dataframe=True -> unchanged concatenated DataFrame. FileBag supports len/iter/getitem (incl. slice), paths/kind/first, download()/download_one(), and to_dataframe()/df. The flat, deprecated fetchers keep their historic list[str] | pd.DataFrame return type. Thread a private _bag flag through fetch -> _fetch_catalog / _fetch_origin_direct / _fetch_ducklake / _fetch_saude, and coerce the result via _coerce_bag. 14 new tests plus 2 updated origin tests; docs updated to describe the new return type.
DuckDataset and File inherited Pydantic's default repr, which dumped every field including private internals (e.g. border=<DatasetAdapter object>). Add concise __repr__ implementations that surface the meaningful fields: DuckDataset shows its record name; File shows path, type, optional group, and dataset. No serialization behavior changes.
Add a full walkthrough of the origin-namespaced FileBag workflow (list with download=False, inspect, download/download_one, to_dataframe) to tutorials.rst, with an explicit anchor target; reference it from data-sources.rst and files-and-formats.rst so all guides show the download=False -> FileBag -> download() -> to_dataframe() path rather than only the DataFrame form.
info() now prints a footer tip reminding users that namespaced fetchers return a FileBag, accept download=False to list files without fetching, and as_dataframe=True for a DataFrame. Add the same guidance to the info_table docstring.
Saude's CNES was declared in the catalog (DatasetSpec) but had no namespaced fetcher, so pysus.info() showed an empty Call for it. Unlike the FTP/DadosGov cnes (monthly state/year/month dumps), Saude serves CNES as CKAN resources (/cnes/estabelecimentos, /cnes/tipounidades) through _fetch_saude. - Add saude_cnes() in _impl.databases: a **kwargs fetcher routing to _fetch_data(dataset="cnes", origin="Saude"). - Add a per-origin function alias map in _impl.source so build_origin_module binds SAUDE's "cnes" to saude_cnes while FTP/DadosGov keep the shared monthly-dump cnes. - Register "cnes" in APPLICABILITY["SAUDE"]; update saude module docstring (19 themes) with the new fetcher. - Tests: saude.cnes routes to _fetch_saude and returns a remote FileBag on download=False; info() now shows pysus.saude.cnes(...) (no dataset lacks a hint, so the omission test now exercises _fetcher_hint directly). Full suite: 1626 passed, 2 skipped.
Replace the hard-coded _SAUDE_GROUP_MAP resolution in _fetch_saude with spec-driven enumeration (DatasetSpec ckan_group + slug_patterns) so slug-only themes like CNES, SISVAN and OUTROSTEMAS resolve correctly instead of returning nothing. Filter CSV resources by the format metadata (capturing *_csv.zip multi-format archives) rather than by a bare .csv URL suffix, and unwrap downloaded zips when building a DataFrame.
luabida
force-pushed
the
split-sources-verbose
branch
from
August 31, 2026 18:50
99a2152 to
fe51573
Compare
luabida
marked this pull request as ready for review
August 31, 2026 18:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.