Skip to content

fix(cli): make pysus ftp files and download work - #354

Open
devgtv wants to merge 2 commits into
AlertaDengue:mainfrom
devgtv:fix/cli-ftp-await-and-list-files
Open

devgtv wants to merge 2 commits into
AlertaDengue:mainfrom
devgtv:fix/cli-ftp-await-and-list-files

Conversation

@devgtv

@devgtv devgtv commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Both commands called the async FTP client without awaiting it, used a method that doesn't exist, never connected, and never wrote any file:

ftp = _get_ftp()
try:
    datasets = ftp.datasets()          # async def — coroutine, never awaited
    ...
    remote_files = target.get_files()  # AttributeError: no such method
    ...
    ftp.download(f, out_dir)           # async def — coroutine, never awaited
finally:
    ftp.close()                        # async def — coroutine, never awaited

connect, datasets, download and close are all async def in pysus/api/ftp/client.py. Datasets expose their files through the content/search contract (BaseRemoteDataset), not get_files() — which appears nowhere else in the codebase. connect() was also never called, and datasets() itself raises ConnectionError when the client is not connected.

Impact

  • pysus ftp files SINAN → TypeError: 'coroutine' object is not iterable
  • pysus ftp download SINAN → printed Downloading 3 file(s)... and Done. Files saved to ... with exit code 0, having written zero bytes

Two of the command's four behaviors were entirely non-functional, and the download path reported false success.

Fix

  • Both commands now run inside an async helper driven by _run_sync, the same bridge the dadosgov CLI uses (so they also work under Jupyter, where asyncio.run would fail)
  • await ftp.connect() opens the session; await ftp.close() runs in finally
  • get_files() → await target.search(), which recurses through groups to the files
  • filters compare against str(f.path) so a non-str path cannot raise
  • download awaits each transfer and writes out_dir / <basename> rather than handing the full remote path to open()
  • the duplicated filter block moved into a shared _select_files, and the class-name lookup into _resolve_dataset

Tests

The existing test_ftp.py only covered the bad-slug path, so nothing ever reached these lines. Added 7 tests with a fake async client covering: listing files, the --year filter, no-match exiting 0, close() on error, all 3 transfers being awaited, writing basenames rather than full remote paths, and no false "Done." on no-match.

Verified all 7 fail against the previous code (TypeError: 'coroutine' object is not iterable plus coroutine ... was never awaited warnings).

  • 1740 passed, 6 skipped
  • black and isort clean

Both commands called the async FTP client without awaiting it, used a
`Dataset.get_files()` method that does not exist, never connected, and
never wrote the files:

    ftp = _get_ftp()
    try:
        datasets = ftp.datasets()      # coroutine, never awaited
        ...
        remote_files = target.get_files()   # AttributeError
        ...
        ftp.download(f, out_dir)       # coroutine, never awaited
    finally:
        ftp.close()                    # coroutine, never awaited

`connect`, `datasets`, `download` and `close` are all `async def`, and
datasets expose their files through the `content`/`search` contract
rather than `get_files()`. `pysus ftp files SINAN` died with
"TypeError: 'coroutine' object is not iterable", and `pysus ftp download`
printed "Downloading N file(s)..." and "Done. Files saved to ..." with
exit code 0 while transferring nothing.

The two commands now run inside an async helper driven by `_run_sync`,
the same bridge the dadosgov CLI uses, so they also work under Jupyter.
The FTP session is opened with `connect()` and closed in `finally`, and
the dataset is located by class name. `get_files()` is replaced with
`search()`, which recurses through groups to the files. Filters compare
against `str(f.path)` so a non-str path cannot raise, and `download` now
awaits each transfer and writes `out_dir / <basename>` instead of
handing the whole remote path to `open()`.

The duplicated filter block moved into a shared `_select_files`, and
dataset lookup into `_resolve_dataset`.
@devgtv
devgtv force-pushed the fix/cli-ftp-await-and-list-files branch from df49e9c to bdbba0b Compare October 2, 2026 19:18
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 98.82353% with 1 line in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@9db5915). Learn more about missing BASE report.

Files with missing lines Patch % Lines
pysus/tests/cli/test_ftp.py 98.82% 1 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #354   +/-   ##
=======================================
  Coverage        ?   97.21%           
=======================================
  Files           ?      180           
  Lines           ?    23371           
  Branches        ?        0           
=======================================
  Hits            ?    22721           
  Misses          ?      650           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants