feat: split the sources in modules so it become very verbose the source of the database

README.md

@@ -11,12 +11,18 @@ It downloads, converts, and analyses datasets from four independent sources —
 (OpenDataSUS), and **DuckLake** (S3 mirror) — and exposes them through a
 single, DataFrame-first API.
 
+The data source is a first-class part of the public API: you reach each origin
+through its own namespace (`pysus.ftp.*`, `pysus.dadosgov.*`,
+`pysus.saude.*`), so the source of every dataset is explicit and impossible to
+ignore.
+
 ## Key features
 
-- **One-line downloads** — `sinan("DENG", 2024, as_dataframe=True)` returns a
-  `pandas.DataFrame` in a single call.
-- **Four data sources** — FTP, DadosGov, Saude (OpenDataSUS), DuckLake; the
-  orchestrator picks the best route automatically.
+- **One-line downloads** — `pysus.ftp.sinan("DENG", 2024, as_dataframe=True)`
+  returns a `pandas.DataFrame` in a single call.
+- **Origin-namespaced API** — `pysus.ftp`, `pysus.dadosgov`, and
+  `pysus.saude` make the data source explicit: `pysus.ftp.sinan(...)`,
+  `pysus.dadosgov.sinasc(...)`, `pysus.saude.arboviroses(...)`.
 - **Data quality** — `missing_values()`, `validate_data()`, `quality_score()`,
   and `profile_report()` give instant insight into completeness and schema
   integrity.
@@ -76,19 +82,59 @@ docker compose down
 
 ### Download a dataset (one-liner)
 
+The recommended way to fetch data is through an origin namespace. Each origin
+exposes the same per-database fetchers:
+
+```python
+import pysus
+
+# DATASUS FTP (served from the S3 catalog mirror by default)
+df = pysus.ftp.sinan(disease="deng", year=2024, as_dataframe=True)
+
+# dados.gov.br (CKAN)
+df = pysus.dadosgov.sinasc(state="SP", year=[2020, 2021, 2022, 2023], as_dataframe=True)
+
+# dadosabertos.saude.gov.br (theme datasets)
+df = pysus.saude.arboviroses(year=2024, as_dataframe=True)
+```
+
+By default every namespace reads the S3/Parquet mirror (`source="catalog"`).
+To query the origin server directly, pass `source="origin"`:
+
 ```python
-from pysus import sinan, sinasc, sim, sih, sia, pni, ibge, cnes, ciha
+df = pysus.ftp.sinan(disease="deng", year=2024, source="origin", as_dataframe=True)
+```
+
+The legacy flat functions (`pysus.sinan`, `pysus.arboviroses`, ...) still
+work unchanged but emit a deprecation warning pointing you to the namespaced
+call.
 
-# Returns a list of local Parquet paths
-parquet_files = sinan(disease="deng", year=2024)
+### What a namespaced fetcher returns
 
-# Get a DataFrame directly
-df = sinan(disease="deng", year=2024, as_dataframe=True)
+Namespaced fetchers return either a high-level `FileBag` or a `DataFrame`:
 
-# Multiple years, filtered by state
-df = sinasc(state="SP", year=[2020, 2021, 2022, 2023], as_dataframe=True)
+- `download=False` → a **remote** `FileBag` listing what would be fetched
+  (nothing is downloaded; `as_dataframe` is ignored here). Call
+  `bag.download()`, `bag.download_one(i)`, or `bag[i].download()`.
+- `download=True` (default) + `as_dataframe=False` → a **local** `FileBag` of
+  downloaded files.
+- `as_dataframe=True` → a single concatenated `pandas.DataFrame`.
+
+A `FileBag` is synchronous; `repr` lists each file (remote files are marked
+`(remote)`), and `to_dataframe()`/`df` concat local tabular files:
+
+```python
+import pysus
+
+bag = pysus.saude.arboviroses(download=False)
+# Files[fa_casoshumanos_1994-2026.csv (remote), fa_epizpnh_1994-2026.csv (remote)]
+local = bag.download()      # -> FileBag of downloaded local files
+df = local.to_dataframe()   # -> concatenated pandas.DataFrame
 ```
 
+The legacy flat fetchers keep their historic `list[str] | pd.DataFrame`
+return type.
+
 ### Browse available datasets
 
 ```python
@@ -99,6 +145,16 @@ search("sinan")                 # fuzzy search across FTP, Saude, DadosGov
 list_files("SINAN")             # list files within a dataset
 ```
 
+Discovery is also scoped per origin:
+
+```python
+import pysus
+
+pysus.ftp.info()                                  # datasets on the FTP origin
+pysus.ftp.list_files("SINAN", year=2024, state="RJ")
+pysus.dadosgov.get_origin_meta()                  # origin metadata
+```
+
 ### The PySUS client (full control)
 
 ```python
@@ -274,24 +330,32 @@ Precedence: explicit argument > environment variable > TOML file > default.
 
 ## Data sources
 
-| Dataset | Description | FTP | DadosGov | Saude | DuckLake |
-|---------|-------------|:---:|:--------:|:-----:|:--------:|
-| SINAN | Disease notifications | x | x | x | x |
-| SIM | Mortality | x | x | x | x |
-| SINASC | Births | x | x | x | x |
-| SIH | Hospitalisations | x | | | x |
-| SIA | Ambulatory procedures | x | | | x |
-| CIHA | Hospital admissions | x | | | x |
-| CNES | Health facilities | x | x | x | x |
-| PNI | Immunisations | x | x | x | x |
-| IBGE | Geographic data | x | | | x |
-| COVID19 | COVID-19 confirmed cases | x | x | x | x |
-| Arboviroses | Arboviral diseases | | | x | |
-| AssistenciaSaude | Health assistance | | | x | |
-| AtencaoPrimaria | Primary care | | | x | |
-| Vacinacao | Vaccination | | | x | |
-| SisAgua | Water surveillance | | | x | |
-| Sisvan | Nutritional surveillance | | | x | |
+PySUS reads from three **origins** (FTP DataSUS, dados.gov.br, OpenDataSUS),
+plus a shared **DuckLake/S3 mirror** that serves as the default cache. The
+`DuckLake` mark below means the dataset is served from the S3 catalog mirror
+by default via that origin namespace.
+
+| Dataset | Description | `pysus.ftp` | `pysus.dadosgov` | `pysus.saude` |
+|---------|-------------|:---:|:--------:|:-----:|
+| SINAN | Disease notifications | x | x | |
+| SIM | Mortality | x | x | |
+| SINASC | Births | x | x | |
+| SIH | Hospitalisations | x | | |
+| SIA | Ambulatory procedures | x | | |
+| CIHA | Hospital admissions | x | | |
+| CNES | Health facilities | x | x | |
+| PNI | Immunisations | x | x | |
+| IBGE | Geographic data | x | | |
+| COVID19 | COVID-19 confirmed cases | x | x | |
+| Arboviroses | Arboviral diseases | | | x |
+| AssistenciaSaude | Health assistance | | | x |
+| AtencaoPrimaria | Primary care | | | x |
+| Vacinacao | Vaccination | | | x |
+| SisAgua | Water surveillance | | | x |
+| Sisvan | Nutritional surveillance | | | x |
+
+> **Note on Saude:** the Saude portal has no catalog mirror, so `pysus.saude.*`
+> always queries the CKAN portal directly regardless of `source`.
 
 ## Architecture
 

docs/source/api.rst

@@ -3,7 +3,8 @@ API Reference
 
 The ``pysus.api`` package provides a layered architecture for discovering,
 downloading, and reading data from Brazilian public health databases
-(DATASUS). It supports four remote data sources.
+(DATASUS). It supports four remote data sources: FTP DataSUS, dados.gov.br,
+OpenDataSUS (dadosabertos.saude.gov.br), and the DuckLake/S3 catalog mirror.
 
 Architecture Overview
 ---------------------
@@ -31,12 +32,13 @@ concrete implementations::
 Quick Start
 -----------
 
-The simplest way to use PySUS is via the high-level convenience
-functions::
+The simplest way to use PySUS is through an origin namespace, which makes the
+data source explicit::
 
-    from pysus import sinan
+    import pysus
 
-    df = sinan(disease="dengue", year=2023)
+    df = pysus.ftp.sinan(disease="dengue", year=2023)
+    df = pysus.saude.arboviroses(disease="dengue", year=2023)
 
 Or with the async API::
 
@@ -48,6 +50,14 @@ Or with the async API::
             await pysus.download(f)
 
 
+FileBag
+-------
+
+.. automodule:: pysus.api.bag
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
 Main Client
 -----------
 

docs/source/databases/data-sources.rst

@@ -7,80 +7,116 @@ Data Sources
 
    getting_started_pysus
 
-PySUS provides simplified functions that return pandas DataFrames directly:
+PySUS provides simplified, origin-namespaced functions that return pandas
+DataFrames directly. Each origin is reachable through its own namespace —
+``pysus.ftp.*``, ``pysus.dadosgov.*``, ``pysus.saude.*`` — so the data source
+is explicit:
 
 .. code-block:: python
 
-    from pysus import sinan, sinasc, sim, sih, sia, pni, ibge, cnes, ciha
+    import pysus
 
-    # Download SINAN Dengue data
-    df = sinan(disease="deng", year=2024)
+    # Download SINAN Dengue data (DATASUS FTP, via the S3 catalog mirror)
+    df = pysus.ftp.sinan(disease="deng", year=2024)
 
     # Multiple years
-    df = sinan(disease="deng", year=[2023, 2024])
+    df = pysus.ftp.sinan(disease="deng", year=[2023, 2024])
 
-    # SINASC births for São Paulo
-    df = sinasc(state="SP", year=2024)
+    # SINASC births for São Paulo (dados.gov.br)
+    df = pysus.dadosgov.sinasc(state="SP", year=2024)
 
-    # SIM mortality data
-    df = sim(state="SP", year=2024)
+    # SIM mortality data — query the origin server directly
+    df = pysus.ftp.sim(state="SP", year=2024, source="origin")
 
     # SIH hospitalizations
-    df = sih(state="SP", year=2024, month=[1, 2, 3])
+    df = pysus.ftp.sih(state="SP", year=2024, month=[1, 2, 3])
 
     # CNES health facilities
-    df = cnes(state="SP", year=2024, month=1)
+    df = pysus.ftp.cnes(state="SP", year=2024, month=1)
 
 OpenDataSUS (Saude) functions
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. code-block:: python
 
-    from pysus import arboviroses, vacinacao, assistencia_saude
+    import pysus
 
     # Dengue/Chik/Zika notifications
-    df = arboviroses(disease="dengue", year=2024)
+    df = pysus.saude.arboviroses(disease="dengue", year=2024)
 
     # Vaccination coverage
-    df = vacinacao(state="SP", year=2024)
+    df = pysus.saude.vacinacao(state="SP", year=2024)
 
     # Hospital and health facility data
-    df = assistencia_saude(state="SP", year=2024)
+    df = pysus.saude.assistencia_saude(state="SP", year=2024)
+
+The legacy flat functions (``pysus.sinan``, ``pysus.arboviroses``, ...) still
+work but emit a deprecation warning pointing to the namespaced form.
+
+Working with files (FileBag)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Besides returning a ``DataFrame`` directly, namespaced fetchers can hand you a
+``FileBag`` to inspect or download files explicitly:
+
+.. code-block:: python
+
+    import pysus
+
+    # Nothing is downloaded here — just a listing of remote files
+    bag = pysus.ftp.sinan(disease="deng", year=2020, download=False)
+    print(bag)                       # Files[DENGBR20.parquet (remote)]
+    print(bag[0].path)               # public/data/ftp/sinan/DENG/2020/_/BR/DENGBR20.parquet
+
+    # Download them all (or bag.download_one(0)) -> a local FileBag
+    local = bag.download()           # Files[DENGBR20.parquet]
+
+    # Concatenate the downloaded files into a single DataFrame
+    df = local.to_dataframe()        # same as local.df
+
+Return type by keyword:
+
+* ``download=False`` → remote ``FileBag`` (listings; ``as_dataframe`` ignored)
+* ``download=True`` (default) + ``as_dataframe=False`` → local ``FileBag``
+* ``as_dataframe=True`` → concatenated ``pandas.DataFrame``
 
 Function Reference
 ^^^^^^^^^^^^^^^^^^
 
+Namespaced fetchers per origin. Both ``pysus.ftp.*`` and
+``pysus.dadosgov.*`` read the S3 catalog mirror by default.
+
 .. list-table::
     :header-rows: 1
 
     * - Function
       - Dataset
       - Parameters
-    * - ``sinan(disease, year)``
+    * - ``ftp.sinan(...)`` / ``dadosgov.sinan(...)``
       - Disease Notifications
       - disease (e.g., "DENG", "ZIKA"), year
-    * - ``sinasc(state, year, group)``
+    * - ``ftp.sinasc(...)`` / ``dadosgov.sinasc(...)``
       - Births
-      - state, year, group (optional)
-    * - ``sim(state, year, group)``
+      - state, year
+    * - ``ftp.sim(...)`` / ``dadosgov.sim(...)``
       - Mortality
-      - state, year, group (optional)
-    * - ``sih(state, year, month, group)``
+      - state, year
+    * - ``ftp.sih(...)``
       - Hospitalizations
-      - state, year, month, group (optional)
-    * - ``sia(state, year, month, group)``
+      - state, year, month
+    * - ``ftp.sia(...)``
       - Ambulatory
-      - state, year, month, group (optional)
-    * - ``pni(state, year, group)``
+      - state, year, month
+    * - ``ftp.pni(...)`` / ``dadosgov.pni(...)``
       - Immunizations
-      - state, year, group (optional)
-    * - ``ibge(year, group)``
+      - state, year
+    * - ``ftp.ibge(...)``
       - IBGE
-      - year, group (optional)
-    * - ``cnes(state, year, month, group)``
+      - year
+    * - ``ftp.cnes(...)`` / ``dadosgov.cnes(...)``
       - Health Facilities
-      - state, year, month, group (optional)
-    * - ``ciha(state, year, month)``
+      - state, year, month
+    * - ``ftp.ciha(...)``
       - Hospital Admissions
       - state, year, month
 
@@ -93,34 +129,36 @@ OpenDataSUS (Saude) Functions
     * - Function
       - Dataset
       - Parameters
-    * - ``arboviroses(**kwargs)``
+    * - ``saude.arboviroses(**kwargs)``
       - Arboviroses (Dengue/Chik/Zika/YF)
       - disease, state, year (via kwargs)
-    * - ``vacinacao(**kwargs)``
+    * - ``saude.vacinacao(**kwargs)``
       - Vaccination Coverage
       - state, year (via kwargs)
-    * - ``assistencia_saude(**kwargs)``
+    * - ``saude.assistencia_saude(**kwargs)``
       - Hospital/Facility Data
       - state, year (via kwargs)
-    * - ``atencao_primaria(**kwargs)``
+    * - ``saude.atencao_primaria(**kwargs)``
       - Primary Care (Previne Brasil)
       - state, year (via kwargs)
-    * - ``sisvan(**kwargs)``
+    * - ``saude.sisvan(**kwargs)``
       - Nutrition Surveillance
       - state, year (via kwargs)
-    * - ``sisagua(**kwargs)``
+    * - ``saude.sisagua(**kwargs)``
       - Water Quality
       - state, year (via kwargs)
-    * - ``covid19(**kwargs)``
-      - COVID-19
-      - state, year (via kwargs)
-    * - ``bnafar(**kwargs)``
+    * - ``saude.bnafar(**kwargs)``
       - Pharmaceutical Assistance
       - state, year (via kwargs)
-    * - ``saude_indigena(**kwargs)``
+    * - ``saude.saude_indigena(**kwargs)``
       - Indigenous Health
       - state, year (via kwargs)
 
+The ``source`` parameter is accepted everywhere: ``source="catalog"`` (default)
+serves the S3/Parquet mirror; ``source="origin"`` queries the origin server
+directly.  The Saude portal has no catalog mirror, so ``saude.*`` always
+queries the CKAN portal.
+
 Using the PySUS Client
 ^^^^^^^^^^^^^^^^^^^^^^