diff --git a/.rules.md b/.rules.md index f8ec46a7..bba36456 100644 --- a/.rules.md +++ b/.rules.md @@ -2,122 +2,43 @@ This file provides guidance to programming agents when working with code in this repository. -## Project Overview +Python client for the [Apify API](https://docs.apify.com/api/v2), published on PyPI as `apify-client`. `ApifyClient` (sync) and `ApifyClientAsync` (async) are the entry points. -Python client for the [Apify API](https://docs.apify.com/api/v2). Provides `ApifyClient` (sync) and `ApifyClientAsync` (async) classes to interact with the Apify platform. Published on PyPI as `apify-client`. +Read [CONTRIBUTING.md](CONTRIBUTING.md) first. It covers setup, tasks, tests, code style, generated models, documentation, and commits, and all of it applies to agents too. -## Development Commands - -Uses [uv](https://docs.astral.sh/uv/) for project management and [Poe the Poet](https://poethepoet.natn.io/) as task runner. Requires Python 3.11+. - -```bash -uv run poe install-dev # Install dev deps + git hooks -uv run poe check-code # Run all checks (lint, type-check, unit-tests, docstring check) -uv run poe lint # Ruff format check + ruff check -uv run poe format # Auto-fix lint issues and format -uv run poe type-check # Run ty type checker -uv run poe unit-tests # Run unit tests -uv run poe check-docstrings # Verify async docstrings match sync -uv run poe fix-docstrings # Auto-fix async docstrings -uv run poe generate-models # Regenerate _models.py, _typeddicts.py, _literals.py from the published spec -uv run poe generate-models-from-file # Regenerate from a local OpenAPI spec file - -# Run a single test -uv run pytest tests/unit/test_file.py -uv run pytest tests/unit/test_file.py::test_name -``` - -Integration tests require `APIFY_TEST_USER_API_TOKEN` and `APIFY_TEST_USER_2_API_TOKEN` env vars. +@CONTRIBUTING.md ## Architecture -### Client Hierarchy +### Client hierarchy -`ApifyClient`/`ApifyClientAsync` are the entry points. They provide methods that return resource sub-clients: +Entry points return resource sub-clients, which can create child clients (e.g. `ActorClient.builds()` -> `BuildCollectionClient`): ```text ApifyClient -├── .actor(id) → ActorClient (single resource operations) +├── .actor(id) → ActorClient (single resource operations) ├── .actors() → ActorCollectionClient (list/create) -├── .dataset(id) → DatasetClient -├── .datasets() → DatasetCollectionClient ├── .run(id) → RunClient -├── .runs() → RunCollectionClient -└── ... (schedules, tasks, webhooks, key-value stores, request queues, etc.) +└── ... (datasets, schedules, tasks, webhooks, key-value stores, request queues, etc.) ``` -Each resource client can create child clients (e.g., `ActorClient.builds()` → `BuildCollectionClient`). - -### Sync/Async Symmetry - -Every resource client has both sync and async variants in the **same file**. For example, `actor.py` contains both `ActorClient` and `ActorClientAsync`. Both share the same interface — async versions use `async/await`. - -Docstrings are written on sync clients and **automatically copied** to async clients via `scripts/check_docstrings.py`. When modifying docstrings, edit only the sync client and run `uv run poe fix-docstrings`. +`ClientRegistry` / `ClientRegistryAsync` (`_client_registry.py`) hold the resource client classes and are passed to every client, so clients can create child clients without importing them (avoids circular imports). All resource clients derive from `ResourceClientBase` (`_resource_clients/_resource_client.py`), whose `WithLogDetailsClient` metaclass wraps public methods for structured logging. -### Dependency Injection via ClientRegistry +### Sync/async symmetry -`ClientRegistry`/`ClientRegistryAsync` (in `_client_registry.py`) holds references to all resource client classes and is passed to each client. This avoids circular imports and lets resource clients create child clients without directly importing them. +Every resource client in `src/apify_client/_resource_clients/` has its sync and async variant in the **same file** (e.g. `ActorClient` and `ActorClientAsync` in `actor.py`) with the same interface. Docstrings are written on the sync client only (see [Documentation](CONTRIBUTING.md#documentation)). -### HTTP Client Abstraction +### HTTP layer -- `HttpClient`/`HttpClientAsync` — base classes in `http_clients/_base.py` holding the shared request pipeline (retries, - timeouts, API errors); transports implement the `send_request`, error-classification, and lifecycle hooks -- `ImpitHttpClient`/`ImpitHttpClientAsync` — default implementation (Rust-based Impit) -- `Httpx2HttpClient`/`Httpx2HttpClientAsync` — built-in alternative behind the `httpx2` optional extra -- `HttpResponse` — Protocol (not a concrete class) for response objects -- Users can plug in custom HTTP clients via `ApifyClient.with_custom_http_client()` +`HttpClient` / `HttpClientAsync` (`http_clients/_base.py`) hold the shared request pipeline (retries, timeouts, API errors); transports (`Impit*` default, `Httpx2*` behind the `httpx2` extra, or a custom one via `ApifyClient.with_custom_http_client()`) implement sending, error classification, and lifecycle hooks. Request-body compression lives in `http_compressors/`, selected by the `compression` constructor argument. -### Base Classes +### Generated models -- `ResourceClientBase` — shared URL building, parameter handling, metaclass-based logging -- `ResourceClient` / `ResourceClientAsync` — sync/async base classes with HTTP call methods -- `WithLogDetailsClient` metaclass — auto-wraps public methods for structured logging +`_models.py`, `_typeddicts.py`, and `_literals.py` are **generated**; never edit them by hand (see [Generated models](CONTRIBUTING.md#generated-models)). Each input-side TypedDict ships in a snake_case and a camelCase variant (`RequestResourceDict`, `RequestResourceCamelDict`); the camel variants are synthesized in the post-processing script, and resource-client signatures accept either casing or the Pydantic model. -### Data Models +### Checking API response shapes -`src/apify_client/_models.py` and `src/apify_client/_typeddicts.py` are **auto-generated** — do not edit them manually. Every Pydantic model and TypedDict comes from the OpenAPI spec. +The OpenAPI spec can lag behind the API and omit fields, and a live response only shows the fields that happen to be set. If the private `apify-core` repository is available locally, use it to confirm which fields an endpoint returns: -Each input-side TypedDict ships in two casings: snake_case (`RequestResourceDict`) and camelCase (`RequestResourceCamelDict`). The camel variants are synthesized in `scripts/postprocess_generated_models.py` from the Pydantic `Field(alias=...)` map; resource-client signatures union both with the Pydantic model so users can pass either casing without losing type-checker support. - -- Generated by `datamodel-code-generator` from the OpenAPI spec at `https://docs.apify.com/api/openapi.json` (config in `pyproject.toml` under `[tool.datamodel-codegen]`, aliases in `datamodel_codegen_aliases.json`) -- After generation, `scripts/postprocess_generated_models.py` is run to apply additional fixes -- To regenerate locally: - - From the live published spec: `uv run poe generate-models` - - From a local spec file: `uv run poe generate-models-from-file path/to/openapi.json` -- `scripts/openapi_spec.py` downloads the spec once into git-ignored `tmp/openapi.json` — both codegen passes read that copy, so a spec redeployed mid-run can't leave the generated files describing two different inputs — then records its `info.version` in `pyproject.toml` under `[tool.apify.openapi-spec]` -- That version is a **coarse marker, not a content identity**: the spec is served latest-only, and apify-docs bumps `components/version.yaml` in a follow-up `[skip ci]` commit, so a deploy can publish new content under the old stamp. A moved value proves the spec changed; an unchanged value proves nothing -- Recording is unconditional; the nightly workflow keeps it honest by committing nothing unless the models changed. A local `generate-models` that only moves that line should be dropped, not committed alone. `generate-models-from-file` leaves the line alone, so models from a candidate spec must not be committed -- In CI, the `Regenerate models` workflow (`on_schedule_regenerate_models.yaml`) runs nightly at 02:00 UTC and opens a PR when the models change. It always generates on master and rebuilds its `ci/regenerate-models` branch from master instead of appending, so the PR diff is always "current spec vs current master". Retitle the `chore:` PR to `fix:`/`feat:` when the diff is user-facing, so it lands in the changelog and triggers a release -- The gate is the **generated models**, not the spec version — which moves on rebuilds regardless of client impact, and doesn't reliably move when the content does -- Manual regeneration is also possible from the GitHub Actions UI (`Regenerate models` workflow) - -## Code Conventions - -- **Line length**: 120 characters -- **Linting/formatting**: Ruff with nearly all rules enabled (see `pyproject.toml`) -- **Type checking**: ty (Astral's type checker) -- **Docstrings**: Google style format, single backticks for inline code references (`` `ApifyClient` `` not ``` ``ApifyClient`` ```) -- **Imports**: `from __future__ import annotations` used throughout -- **Commits**: [Conventional Commits](https://www.conventionalcommits.org/) format. Choose the type based on *what* changed, not just *why*: - - `feat:` / `fix:` / `perf:` / `refactor:` / `style:` — **source code only**; these trigger a release and appear in the changelog - - `test:` — test additions or changes (no release triggered) - - `docs:` — documentation changes; also triggers a doc release on master - - `ci:` — CI/workflow changes - - `chore:` — dependency bumps, tooling, generated files (e.g. model regeneration PRs), and other housekeeping - - `build:` — build system changes - -## Testing - -- **Unit tests** (`tests/unit/`): Use `pytest-httpserver` to mock HTTP. No network required. -- **Integration tests** (`tests/integration/`): Hit the live Apify API. -- **Async**: `asyncio_mode = "auto"` — async tests run automatically without markers. -- **Parallelism**: Tests run in parallel via `pytest-xdist`. - -Unit test pattern: - -```python -def test_example(httpserver: HTTPServer) -> None: - httpserver.expect_request('/v2/endpoint').respond_with_json({'data': ...}) - client = ApifyClient(token='test', api_url=httpserver.url_for('/').removesuffix('/')) - # assert client behavior -``` +- A single resource's fields are in `PUBLISHED_*_FIELDS` in `src/packages/-server/src//.server.ts`. +- A list endpoint has its own, usually smaller, `API_PUBLISHED_*_LIST_FIELDS` in `src/api/src/routes//_list.ts`, and often builds each item inline. Don't assume a list item has the same fields as the single resource. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 815e8fb1..b6638ea4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,36 +4,15 @@ Here you'll find a contributing guide to get started with development. ## Environment -For local development, it is required to have Python 3.11 (or a later version) installed. +For local development, you need a Python version that satisfies `requires-python` in `pyproject.toml`. We use [uv](https://docs.astral.sh/uv/) for project management. Install it and set up your IDE accordingly. -We use [Poe the Poet](https://poethepoet.natn.io/) as a task runner, similar to npm scripts in `package.json`. -All tasks are defined in `pyproject.toml` under `[tool.poe.tasks]` and can be run with `uv run poe `. - -### Available tasks - -| Task | Description | -| ---- | ----------- | -| `install-dev` | Install development dependencies | -| `check-code` | Run lint, type-check, unit-tests, and docstring check | -| `lint` | Run linter | -| `format` | Fix lint issues and format code | -| `type-check` | Run type checker | -| `unit-tests` | Run unit tests | -| `unit-tests-cov` | Run unit tests with coverage | -| `integration-tests` | Run integration tests | -| `integration-tests-cov` | Run integration tests with coverage | -| `check-docstrings` | Check async client docstrings | -| `fix-docstrings` | Fix async client docstrings | -| `build-docs` | Build documentation website | -| `run-docs` | Run documentation website locally | -| `build` | Build package | -| `clean` | Remove build artifacts and clean caches | +We use [Poe the Poet](https://poethepoet.natn.io/) as a task runner, similar to npm scripts in `package.json`. All tasks are defined in `pyproject.toml` under `[tool.poe.tasks]`. Run `uv run poe` to list them with their descriptions, and `uv run poe ` to run one. ## Dependencies -To install this package and its development dependencies, run: +To install this package with its development dependencies and the pre-commit hooks, run: ```sh uv run poe install-dev @@ -41,79 +20,74 @@ uv run poe install-dev ## Code checking -To execute all code checking tools together, run: +To run all code checks (lint, type check, async docstring check, and unit tests), run: ```sh uv run poe check-code ``` -### Linting +We use [ruff](https://docs.astral.sh/ruff/) for linting and formatting, [ty](https://docs.astral.sh/ty/) for type checking, and [pytest](https://docs.pytest.org/) for tests. All of them are configured in `pyproject.toml`, and each has its own task (`lint`, `format`, `type-check`, `unit-tests`). -We utilize [ruff](https://docs.astral.sh/ruff/) for linting, which analyzes code for potential issues and enforces consistent style. Refer to `pyproject.toml` for configuration details. +## Tests -To run linting: +- **Unit tests** (`tests/unit`) are isolated from the Apify API. See [`tests/unit/README.md`](tests/unit/README.md) for the rules they follow. Run them with `uv run poe unit-tests`, or a single test with `uv run pytest tests/unit/.py::`. +- **Integration tests** (`tests/integration`) call the real Apify API. See [`tests/integration/README.md`](tests/integration/README.md) for the tokens they need. Run them with `uv run poe integration-tests`. -```sh -uv run poe lint -``` +## Code style -### Formatting +### Readability -Our automated code formatting also leverages [ruff](https://docs.astral.sh/ruff/), ensuring uniform style and addressing fixable linting issues. Configuration specifics are outlined in `pyproject.toml`. +- Prefer a few simple statements over one dense line. Give intermediate values a name instead of nesting calls, comprehensions, and unpacking into a single long expression. +- When replacing a call with an equivalent one, change only the call and keep the surrounding statements as they were. -To run formatting: +### Comments and docstrings -```sh -uv run poe format -``` - -### Type checking - -Type checking is handled by [ty](https://docs.astral.sh/ty/), verifying code against type annotations. Configuration settings can be found in `pyproject.toml`. - -To run type checking: - -```sh -uv run poe type-check -``` +- Every module starts with `from __future__ import annotations`. +- Comments and docstrings describe what the code does now. The bug a change fixed, what a PR changed, and contrasts with removed behavior (`instead of ...`, `no longer ...`) belong in the commit message. +- Don't explain language features in comments. A comment earns its place with a domain reason, an invariant the types can't express, or a non-obvious consequence for callers. +- Use single backticks for code references, never RST-style double backticks. +- Wrap docstrings and prose comments to the full line length configured in `pyproject.toml`. +- Write only ASCII in source files: `->` instead of an arrow, `-` instead of a dash, straight quotes. +- Write `RQ` for the request queue. Log messages and public `.mdx` docs spell out the full term. -### Unit tests +### Tests -We use [pytest](https://docs.pytest.org/) as a testing framework with many plugins. Check `pyproject.toml` for configuration details and installed plugins. +- Write flat `def test_*()` functions, never `class Test*`, and no section-header comments between tests. +- Almost every test gets a one-line docstring stating what it verifies, without a `Regression test:`-style prefix. +- In `@pytest.mark.parametrize`, wrap each case in `pytest.param(..., id='...')` with an id that describes the case. +- Use `Mock` and the `monkeypatch` fixture for test doubles, not hand-rolled stubs, counters, or flags. +- A test that changes process-wide or event-loop-wide state restores it. Prefer `monkeypatch`; for APIs it can't patch (`loop.set_exception_handler()`, `signal.signal()`, logging config, `os.chdir()`), capture the original value and restore it in a `finally`. +- On Windows with Python 3.12 or older, `datetime.now()` ticks in 1-15 ms steps, so two timestamps taken a few milliseconds apart can be equal. A test that asserts one timestamp is later than another injects a `Mock` clock instead of sleeping. +- For a TCP endpoint that must refuse connections, bind a socket to `('127.0.0.1', 0)` and hold it open without calling `listen()`. A hard-coded port can be taken by a server in a parallel xdist worker. Test servers use `127.0.0.1`, since on Windows `localhost` resolves to both IPv4 and IPv6. -To run unit tests: +### YAML -```sh -uv run poe unit-tests -``` +Use double quotes for quoted YAML values (`retries: "3"`). Keep single quotes when the value itself contains double quotes, and for string literals inside GitHub Actions expressions (`${{ ... }}`, `if:` conditions), where double quotes are invalid. -To run unit tests with XML coverage report: +## Generated models -```sh -uv run poe unit-tests-cov -``` +`src/apify_client/_models.py`, `_typeddicts.py`, and `_literals.py` are generated from the Apify OpenAPI specification by [datamodel-code-generator](https://github.com/koxudaxi/datamodel-code-generator) and post-processed by `scripts/postprocess_generated_models.py`. Never edit them by hand; make lasting fixes in the post-processing script. The generator is configured under `[tool.datamodel-codegen]` in `pyproject.toml`, with field aliases in `datamodel_codegen_aliases.json`. -## Integration tests +- `uv run poe generate-models` regenerates them from the published specification. `scripts/openapi_spec.py` describes how the specification version is recorded. Don't commit a regeneration that only moves that version. +- `uv run poe generate-models-from-file ` generates them from a local specification, for trying out unpublished changes. Don't commit the result. +- The nightly `Regenerate models` workflow opens a `chore:` PR when the models change. If the diff is user-facing, retitle it to `fix:` or `feat:` so it lands in the changelog and triggers a release. -We have integration tests which interact with the Apify platform. +## Documentation -Prerequisites: +We adhere to the [Google docstring format](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) for documenting our codebase. Every user-facing class or method is documented. Documentation standards are enforced using [Ruff](https://docs.astral.sh/ruff/). -- Set `APIFY_TEST_USER_API_TOKEN` to your Apify API token -- Set `APIFY_TEST_USER_2_API_TOKEN` for storage access tests (requires two accounts) -- Optionally set `APIFY_INTEGRATION_TESTS_API_URL` to use a different Apify API environment +Resource client docstrings follow one pattern: -To run integration tests: +- The first line is `Sub-client for managing a specific .` or `Sub-client for the collection.` +- The second paragraph names two or three things the client does (not every method) and says to obtain an instance via an appropriate method on the parent client class, without naming that method. -```sh -uv run poe integration-tests -``` +Docstrings are written on the sync clients only. After editing one, run `uv run poe fix-docstrings` to copy it to the async client; `uv run poe check-docstrings` fails when the two differ. -## Documentation +Our API documentation is generated from these docstrings using [pydoc-markdown](https://pypi.org/project/pydoc-markdown/) with additional post-processing. Markdown files in the `docs/` folder complement the autogenerated content. Final documentation is rendered using [Docusaurus](https://docusaurus.io/) and published to GitHub Pages. -We adhere to the [Google docstring format](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) for documenting our codebase. Every user-facing class or method is documented. Documentation standards are enforced using [Ruff](https://docs.astral.sh/ruff/). +A user-facing feature ships with its documentation in the same PR: a page in `docs/02_concepts/` that explains the mechanism and, where it helps, a page in `docs/03_guides/` that shows a realistic use. -Our API documentation is generated from these docstrings using [pydoc-markdown](https://pypi.org/project/pydoc-markdown/) with additional post-processing. Markdown files in the `docs/` folder complement the autogenerated content. Final documentation is rendered using [Docusaurus](https://docusaurus.io/) and published to GitHub Pages. +Code examples in `docs/` come in `_sync.py` / `_async.py` pairs and have their own lint settings in `docs/pyproject.toml`. Never edit `website/versioned_docs/`: those snapshots are produced by the release automation and frozen against the code that shipped. Fix `docs/`, and the next release snapshot carries the change. To run the documentation locally (requires Node.js): @@ -140,23 +114,25 @@ Doc images are committed as optimized `.webp`. To convert a new image, run ## Commits -We use [Conventional Commits](https://www.conventionalcommits.org/) format for commit messages. This convention is used to automatically determine version bumps during the release process. - -### Available commit types +We use [Conventional Commits](https://www.conventionalcommits.org/) format for commit messages. The release process reads them to determine the version bump and build the changelog. Pick the type by what the commit changes, not by why: | Type | Description | | ---- | ----------- | -| `feat` | A new feature | -| `fix` | A bug fix | -| `docs` | Documentation only changes | -| `style` | Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc) | -| `refactor` | A code change that neither fixes a bug nor adds a feature | -| `perf` | A code change that improves performance | -| `test` | Adding missing tests or correcting existing tests | -| `build` | Changes that affect the build system or external dependencies (example scopes: gulp, broccoli, npm) | -| `ci` | Changes to our CI configuration files and scripts (example scopes: Travis, Circle, BrowserStack, SauceLabs) | -| `chore` | Other changes that don't modify src or test files | -| `revert` | Reverts a previous commit | +| `feat` | A new feature in the source code | +| `fix` | A bug fix in the source code | +| `perf` | A source code change that improves performance | +| `refactor` | A source code change that neither fixes a bug nor adds a feature | +| `style` | A source code change that doesn't affect its meaning (white-space, formatting) | +| `test` | Adding or changing tests | +| `docs` | Documentation changes | +| `ci` | CI workflow changes | +| `build` | Build system changes | +| `chore` | Dependency bumps, tooling, generated files, and other housekeeping | +| `revert` | Reverting a previous commit | + +`feat`, `fix`, `perf`, `refactor`, and `style` trigger a release and appear in the changelog, so use them for source code changes only. `docs` triggers a docs release on master. + +Logging changes are breaking: changing a log level, a logger name, or the wording of a log message, or removing one, breaks downstream log-based tooling. Keep the accurate type, add `!` after it, and list the changes in a `BREAKING CHANGE:` footer. PRs are squash-merged, so the `!` must also be in the PR title. ## Release process diff --git a/README.md b/README.md index c9a6950a..9cb58d86 100644 --- a/README.md +++ b/README.md @@ -224,12 +224,7 @@ The full documentation lives at **[docs.apify.com/api/client/python](https://doc ## Contributing -Bug reports, fixes, and improvements are welcome! See [CONTRIBUTING.md](./CONTRIBUTING.md) for the development setup, coding standards, testing, and the release process. The repo uses [uv](https://docs.astral.sh/uv/) for project management and [Poe the Poet](https://poethepoet.natn.io/) as a task runner; the typical loop is: - -```bash -uv run poe install-dev # install dev deps and git hooks -uv run poe check-code # lint, type-check, unit tests, docstring check -``` +Bug reports, fixes, and improvements are welcome! See [CONTRIBUTING.md](./CONTRIBUTING.md) for the development setup, coding standards, testing, and the release process. ## License diff --git a/pyproject.toml b/pyproject.toml index 607d487a..73ebe368 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -267,39 +267,45 @@ impit = false # Run tasks with: uv run poe [tool.poe.tasks] -clean = "rm -rf .coverage .pytest_cache .ruff_cache .ty_cache build dist htmlcov" -install-sync = "uv sync --all-extras" -build = "uv build" -publish-to-pypi = "uv publish --token ${APIFY_PYPI_TOKEN_CRAWLEE}" -type-check = "uv run ty check" -unit-tests = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} tests/unit" -unit-tests-cov = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} --cov=src/apify_client --cov-report=xml:coverage-unit.xml tests/unit" -integration-tests = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} tests/integration" -integration-tests-cov = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} --cov=src/apify_client --cov-report=xml:coverage-integration.xml tests/integration" -check-docstrings = "uv run python -m scripts.check_docstrings" -fix-docstrings = "uv run python -m scripts.fix_docstrings" -check-code = ["lint", "type-check", "check-docstrings", "unit-tests"] +clean = { cmd = "rm -rf .coverage .pytest_cache .ruff_cache .ty_cache build dist htmlcov", help = "Remove build artifacts and clean caches" } +install-sync = { cmd = "uv sync --all-extras", help = "Sync dependencies with all extras" } +build = { cmd = "uv build", help = "Build the package" } +publish-to-pypi = { cmd = "uv publish --token ${APIFY_PYPI_TOKEN_CRAWLEE}", help = "Publish the package to PyPI" } +type-check = { cmd = "uv run ty check", help = "Run the type checker" } +unit-tests = { cmd = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} tests/unit", help = "Run unit tests" } +unit-tests-cov = { cmd = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} --cov=src/apify_client --cov-report=xml:coverage-unit.xml tests/unit", help = "Run unit tests with an XML coverage report" } +integration-tests = { cmd = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} tests/integration", help = "Run integration tests" } +integration-tests-cov = { cmd = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} --cov=src/apify_client --cov-report=xml:coverage-integration.xml tests/integration", help = "Run integration tests with an XML coverage report" } +check-docstrings = { cmd = "uv run python -m scripts.check_docstrings", help = "Check that async client docstrings match their sync counterparts" } +fix-docstrings = { cmd = "uv run python -m scripts.fix_docstrings", help = "Copy sync client docstrings to their async counterparts" } +check-code = { sequence = ["lint", "type-check", "check-docstrings", "unit-tests"], help = "Run lint, type check, docstring check, and unit tests" } [tool.poe.tasks.install-dev] shell = "uv sync --all-extras && uv run pre-commit install" +help = "Install development dependencies and pre-commit hooks" [tool.poe.tasks.lint] shell = "uv run ruff format --check && uv run ruff check" +help = "Run the linter and check formatting" [tool.poe.tasks.format] shell = "uv run ruff check --fix && uv run ruff format" +help = "Fix lint issues and format code" [tool.poe.tasks.update-docs-theme] shell = "pnpm update @apify/docs-theme" cwd = "website" +help = "Update the documentation website theme" [tool.poe.tasks.build-docs] shell = "./build_api_reference.sh && pnpm install && uv run pnpm build" cwd = "website" +help = "Build the documentation website" [tool.poe.tasks.run-docs] shell = "./build_api_reference.sh && pnpm install && uv run pnpm start" cwd = "website" +help = "Run the documentation website locally" # Both passes read one downloaded copy, so a specification redeployed mid-run can't leave the generated files # describing two different inputs. The version is recorded last, once generation succeeded. @@ -318,6 +324,7 @@ uv run python -m scripts.openapi_spec fetch \ && python scripts/postprocess_generated_models.py \ && uv run python -m scripts.openapi_spec record-version """ +help = "Regenerate models from the published OpenAPI specification" # For a candidate specification that isn't published yet. The recorded version deliberately stays put - it names # the published specification - so models generated this way must not be committed, hence the warning. @@ -333,3 +340,4 @@ uv run datamodel-codegen --input $input_file \ && echo "Generated from $input_file - the version in [tool.apify.openapi-spec] still names the published specification, so do not commit these models." """ args = [{ name = "input-file", positional = true, required = true }] +help = "Generate models from a local OpenAPI specification file (don't commit the result)" diff --git a/tests/integration/README.md b/tests/integration/README.md index 92c31a40..afde0f52 100644 --- a/tests/integration/README.md +++ b/tests/integration/README.md @@ -6,4 +6,6 @@ They require valid API tokens provided via environment variables: - `APIFY_TEST_USER_API_TOKEN` - `APIFY_TEST_USER_2_API_TOKEN` (used for permission-related tests) +To run them against a different Apify API environment, also set `APIFY_INTEGRATION_TESTS_API_URL`. + Tests that do not call the Apify API should be implemented as unit tests instead. diff --git a/tests/unit/README.md b/tests/unit/README.md index dd7d16ff..f2c7ea02 100644 --- a/tests/unit/README.md +++ b/tests/unit/README.md @@ -9,3 +9,12 @@ They must not require: - network access at all Any test that calls the Apify API belongs to integration tests. + +Mock HTTP with the `httpserver` fixture from `pytest-httpserver`: + +```python +def test_example(httpserver: HTTPServer) -> None: + httpserver.expect_request('/v2/endpoint').respond_with_json({'data': ...}) + client = ApifyClient(token='test', api_url=httpserver.url_for('/').removesuffix('/')) + # assert client behavior +```