diff --git a/.rules.md b/.rules.md index 4edef808..a91d9d86 100644 --- a/.rules.md +++ b/.rules.md @@ -2,138 +2,28 @@ This file provides guidance to programming agents when working with code in this repository. -## Project Overview +The Apify SDK for Python (`apify` on PyPI) is the library for building [Apify Actors](https://docs.apify.com/platform/actors) in Python. It builds on [Crawlee](https://crawlee.dev/python) and the [Apify API client](https://docs.apify.com/api/client/python). -The Apify SDK for Python (`apify` package on PyPI) is the official library for creating [Apify Actors](https://docs.apify.com/platform/actors) in Python. It provides Actor lifecycle management, storage access (datasets, key-value stores, request queues), event handling, proxy configuration, and pay-per-event charging. It builds on top of the [Crawlee](https://crawlee.dev/python) web scraping framework and the [Apify API Client](https://docs.apify.com/api/client/python). Supports Python 3.11–3.14. Build system: uv_build. +Read [CONTRIBUTING.md](CONTRIBUTING.md) first. It covers setup, tasks, tests, code style, documentation, and commits, and all of it applies to agents too. -## Common Commands +@CONTRIBUTING.md -```bash -# Install dependencies (including dev) -uv sync --all-extras +## Pydantic models -# Install dev dependencies + pre-commit hooks -uv run poe install-dev - -# Format code (also auto-fixes lint issues via ruff check --fix) -uv run poe format - -# Lint (format check + ruff check) -uv run poe lint - -# Type check -uv run poe type-check - -# Run all checks (lint + type-check + unit tests) -uv run poe check-code - -# Unit tests (no API token needed) -uv run poe unit-tests - -# Run a single test file -uv run pytest tests/unit/actor/test_actor_lifecycle.py - -# Run a single test by name -uv run pytest tests/unit/actor/test_actor_lifecycle.py -k "test_name" - -# Integration tests (needs APIFY_TEST_USER_API_TOKEN) -uv run poe integration-tests - -# E2E tests (needs APIFY_TEST_USER_API_TOKEN, builds/deploys Actors on platform) -uv run poe e2e-tests -``` - -## Code Style - -- **Formatter/Linter**: Ruff (line length 120, single quotes for inline, double quotes for docstrings) -- **Type checker**: ty (targets Python 3.11) -- **All ruff rules enabled** with specific ignores — see `pyproject.toml` `[tool.ruff.lint]` for the full ignore list -- Tests are exempt from docstring rules (`D`), assert warnings (`S101`), and private member access (`SLF001`) -- Unused imports are allowed in `__init__.py` files (re-exports) -- **Pre-commit hooks**: lint check + type check run automatically on commit -- **Commits**: [Conventional Commits](https://www.conventionalcommits.org/) format. Choose the type based on *what* changed, not just *why*: - - `feat:` / `fix:` / `perf:` / `refactor:` / `style:` — **source code only**; these trigger a release and appear in the changelog - - `test:` — test additions or changes (no release triggered) - - `docs:` — documentation changes; also triggers a doc release on master - - `ci:` — CI/workflow changes - - `chore:` — dependency bumps, tooling, and other housekeeping - - `build:` — build system changes - -### Pydantic aliases +Don't introduce a shared Pydantic base model to centralize `model_config`. Each model inherits `pydantic.BaseModel` and declares `model_config = ConfigDict(...)` inline, which also keeps ruff's flake8-type-checking and `RUF012` checks working, since they only recognize a literal `pydantic.BaseModel` base. `model_dump(by_alias=True)` serializes every model under camelCase names. Keep it that way: -- **API / event / data models** (camelCase JSON round-trip with the platform) — set - `model_config = ConfigDict(alias_generator=to_camel)`, so both validation and serialization use the - camelCase name. -- **`Configuration`** — validated from environment variables, so each field carries a `validation_alias` - (a plain string, or `AliasChoices(...)` for several legacy env names). Serialization is handled by the - model-level `alias_generator=AliasGenerator(serialization_alias=to_camel)`, which derives the camelCase - name from the Python field name. Don't use `alias=` on a `Configuration` field — it pins serialization - to the env-var name and defeats the camelCase generator. -- Set an explicit `serialization_alias` only when the wire name isn't the `to_camel` derivation of the - field name (e.g. `RequestQueueHead.lock_time` -> `lockSecs`). +- **API / event / data models** (camelCase JSON round-trip with the platform) — set `model_config = ConfigDict(alias_generator=to_camel)`, so both validation and serialization use the camelCase name. +- **`Configuration`** — validated from environment variables, so each field carries a `validation_alias` (a plain string, or `AliasChoices(...)` for several legacy env names). Serialization is handled by the model-level `alias_generator=AliasGenerator(serialization_alias=to_camel)`. Don't use `alias=` on a `Configuration` field — it pins serialization to the env-var name and defeats the camelCase generator. +- Set an explicit `serialization_alias` only when the wire name isn't the `to_camel` derivation of the field name (e.g. `RequestQueueHead.lock_time` -> `lockSecs`). ## Architecture -### Core (`src/apify/`) - -- **`_actor.py`** — The `_ActorType` class is the central API. `Actor` is a lazy-object-proxy (`lazy-object-proxy.Proxy`) wrapping `_ActorType` — it acts as both a class (e.g. `Actor.is_at_home()`) and an instance-like context manager (`async with Actor:`). On `__aenter__`, the proxy's `__wrapped__` is replaced with the active `_ActorType` instance. It manages the full Actor lifecycle (`init`, `exit`, `fail`), provides access to storages (`open_dataset`, `open_key_value_store`, `open_request_queue`), handles events, proxy configuration, charging, and platform API operations (`start`, `call`, `metamorph`, `reboot`). - -- **`_configuration.py`** — `Configuration` extends Crawlee's `Configuration` with Apify-specific settings (API URL, token, Actor run metadata, proxy settings, charging config). Configuration is populated from environment variables (`APIFY_*`). - -- **`_charging.py`** — Pay-per-event billing system. `ChargingManager` / `ChargingManagerImplementation` handle charging events against pricing info fetched from the API. - -- **`_proxy_configuration.py`** — `ProxyConfiguration` manages Apify proxy setup (residential, datacenter, groups, country targeting). - -- **`_webhook.py`** — The `Webhook` dataclass (ad-hoc / persistent webhook definition) and the `to_client_representations` helper. Response and data models are no longer defined in the SDK — they come from `apify-client` v3 (e.g. `Run`, the Actor pricing-info models). - -### Storage Clients (`src/apify/storage_clients/`) - -Four storage client implementations, all implementing Crawlee's abstract storage client interface: - -- **`_apify/`** — `ApifyStorageClient`: talks to the Apify API for dataset, key-value store, and request queue operations (separate sub-clients for single vs. shared request queues). Used when running on the Apify platform. -- **`_file_system/`** — `FileSystemStorageClient` (alias `ApifyFileSystemStorageClient`): extends Crawlee's file system client with Apify-specific key-value store behavior. -- **`_smart_apify/`** — `SmartApifyStorageClient`: hybrid client that writes to both API and local file system for resilience. -- **`MemoryStorageClient`** — re-exported from Crawlee for in-memory storage. - -### Storages (`src/apify/storages/`) - -Re-exports Crawlee's `Dataset`, `KeyValueStore`, and `RequestQueue` classes. - -### Events (`src/apify/events/`) - -- **`_apify_event_manager.py`** — `ApifyEventManager` extends Crawlee's event system with platform-specific events received via WebSocket connection. - -### Request Loaders (`src/apify/request_loaders/`) - -- **`_apify_request_list.py`** — `ApifyRequestList` creates request lists from Actor input URLs (supports both direct URLs and "requests from URL" sources). - -### Scrapy Integration (`src/apify/scrapy/`) - -Optional integration (`apify[scrapy]` extra) providing Scrapy scheduler, middlewares, pipelines, and extensions for running Scrapy spiders as Apify Actors. - -### Key Dependencies - -- **`crawlee`** — Base framework providing storage abstractions, event system, configuration, service locator pattern -- **`apify-client`** — HTTP client for the Apify API (`ApifyClientAsync`); also the source of response and data models (`Run`, pricing info, webhook representations) - -The SDK no longer depends on `apify-shared`. The platform env-var enums (`ApifyEnvVars`, `ActorEnvVars`) are vendored in `apify._consts` and re-exported from the top-level `apify` package. - -## Testing - -Three test levels in `tests/`: - -- **`unit/`** — Fast tests with no external dependencies. Use mocked API clients (`ApifyClientAsyncPatcher` fixture). Run with `uv run poe unit-tests`. -- **`integration/`** — Tests making real Apify API calls but not deploying Actors. Requires `APIFY_TEST_USER_API_TOKEN`. Run with `uv run poe integration-tests`. -- **`e2e/`** — Full end-to-end tests that build and deploy Actors on the platform. Slowest. Requires `APIFY_TEST_USER_API_TOKEN`. Use `make_actor` and `run_actor` fixtures. Run with `uv run poe e2e-tests`. - -All test levels use `pytest-asyncio` with `asyncio_mode = "auto"` (no need for `@pytest.mark.asyncio`). Tests run in parallel via `pytest-xdist` (`--numprocesses`). Each test gets isolated state via the autouse `_isolate_test_environment` fixture which resets `Actor`, `service_locator`, and `AliasResolver` state. Conftest files live in each subdirectory (`tests/unit/conftest.py`, etc.) — there is no top-level `tests/conftest.py`. - -### Key Test Fixtures - -- **`apify_client_async_patcher`** (unit) — `ApifyClientAsyncPatcher` instance for mocking `ApifyClientAsync` methods. Patch by `method`/`submethod`, tracks call history in `.calls`. -- **`make_httpserver`/`httpserver`** (unit) — session-scoped `HTTPServer` via `pytest-httpserver` for HTTP interception. -- **`apify_client_async`** (integration/e2e) — real `ApifyClientAsync` using `APIFY_TEST_USER_API_TOKEN`. -- **`make_actor`** (e2e) — creates a temporary Actor on the platform from a function, `main_py` string, or source files dict; cleans up after the session. -- **`run_actor`** (e2e) — calls an Actor and waits up to 10 minutes for completion. +- **`Actor`** (`_actor.py`) is a `lazy-object-proxy` wrapping `_ActorType`. It works both as a class (`Actor.is_at_home()`) and as a context manager (`async with Actor:`); on `__aenter__` the proxy's `__wrapped__` is replaced with the active `_ActorType` instance. +- **`Configuration`** (`_configuration.py`) extends Crawlee's `Configuration` and is populated from `APIFY_*` environment variables. +- **Named child runs** (`_child_runs.py`) — runs started with `run_name` are persisted in the default key-value store so they can be reattached after a migration or resurrection; exposed through `Actor.child_runs`. +- **Storage clients** (`storage_clients/`) implement Crawlee's storage client interface: `ApifyStorageClient` talks to the Apify API (separate sub-clients for single vs. shared request queues), `FileSystemStorageClient` adds Apify-specific key-value store behavior, and `SmartApifyStorageClient` delegates to the cloud client on the platform and to the local one elsewhere. +- **Events** — `ApifyEventManager` receives platform events over a WebSocket. +- **Errors** — `apify.errors` only re-exports the exceptions from `apify_client.errors`; the SDK defines no error classes of its own, and API calls raise the client exceptions unchanged. +- Response and data models (`Run`, pricing info, webhook representations) come from `apify-client`; the SDK doesn't define its own. The platform env-var enums (`ApifyEnvVars`, `ActorEnvVars`) live in `apify._consts`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 85c11e28..d4e10e7c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,34 +4,15 @@ Here you'll find a contributing guide to get started with development. ## Environment -For local development, it is required to have Python 3.11 (or a later version) installed. +For local development, you need a Python version that satisfies `requires-python` in `pyproject.toml`. We use [uv](https://docs.astral.sh/uv/) for project management. Install it and set up your IDE accordingly. -We use [Poe the Poet](https://poethepoet.natn.io/) as a task runner, similar to npm scripts in `package.json`. -All tasks are defined in `pyproject.toml` under `[tool.poe.tasks]` and can be run with `uv run poe `. - -### Available tasks - -| Task | Description | -| ---- | ----------- | -| `install-dev` | Install development dependencies | -| `check-code` | Run lint, type-check, and unit-tests | -| `lint` | Run linter | -| `format` | Fix lint issues and format code | -| `type-check` | Run type checker | -| `unit-tests` | Run unit tests | -| `unit-tests-cov` | Run unit tests with coverage | -| `integration-tests` | Run integration tests | -| `integration-tests-cov` | Run integration tests with coverage | -| `build-docs` | Build documentation website | -| `run-docs` | Run documentation website locally | -| `build` | Build package | -| `clean` | Remove build artifacts and clean caches | +We use [Poe the Poet](https://poethepoet.natn.io/) as a task runner, similar to npm scripts in `package.json`. All tasks are defined in `pyproject.toml` under `[tool.poe.tasks]`. Run `uv run poe` to list them with their descriptions, and `uv run poe ` to run one. ## Dependencies -To install this package and its development dependencies, run: +To install this package with its development dependencies and the pre-commit hooks, run: ```sh uv run poe install-dev @@ -39,72 +20,47 @@ uv run poe install-dev ## Code checking -To execute all code checking tools together, run: +To run all code checks (lint, type check, and unit tests), run: ```sh uv run poe check-code ``` -### Linting - -We utilize [ruff](https://docs.astral.sh/ruff/) for linting, which analyzes code for potential issues and enforces consistent style. Refer to `pyproject.toml` for configuration details. - -To run linting: - -```sh -uv run poe lint -``` - -### Formatting - -Our automated code formatting also leverages [ruff](https://docs.astral.sh/ruff/), ensuring uniform style and addressing fixable linting issues. Configuration specifics are outlined in `pyproject.toml`. +We use [ruff](https://docs.astral.sh/ruff/) for linting and formatting, [ty](https://docs.astral.sh/ty/) for type checking, and [pytest](https://docs.pytest.org/) for tests. All of them are configured in `pyproject.toml`, and each has its own task (`lint`, `format`, `type-check`, `unit-tests`). -To run formatting: +## Tests -```sh -uv run poe format -``` +Each test level has a README describing its fixtures, isolation, and required tokens. -### Type checking +- **Unit tests** (`tests/unit`) mock the Apify API and need no credentials. See [`tests/unit/README.md`](tests/unit/README.md). Run them with `uv run poe unit-tests`, or a single test with `uv run pytest tests/unit/.py::`. +- **Integration tests** (`tests/integration`) call the real Apify API without deploying Actors. See [`tests/integration/README.md`](tests/integration/README.md). Run them with `uv run poe integration-tests`. +- **E2E tests** (`tests/e2e`) build and run Actors on the Apify platform. See [`tests/e2e/README.md`](tests/e2e/README.md). Run them with `uv run poe e2e-tests`. -Type checking is handled by [ty](https://docs.astral.sh/ty/), verifying code against type annotations. Configuration settings can be found in `pyproject.toml`. +## Code style -To run type checking: +### Readability -```sh -uv run poe type-check -``` +- Prefer a few simple statements over one dense line. Give intermediate values a name instead of nesting calls, comprehensions, and unpacking into a single long expression. +- When replacing a call with an equivalent one, change only the call and keep the surrounding statements as they were. -### Unit tests +### Comments and docstrings -We use [pytest](https://docs.pytest.org/) as a testing framework with many plugins. Check `pyproject.toml` for configuration details and installed plugins. +- Comments and docstrings describe what the code does now. The bug a change fixed, what a PR changed, and contrasts with removed behavior (`instead of ...`, `no longer ...`) belong in the commit message. +- Don't explain language features in comments. A comment earns its place with a domain reason, an invariant the types can't express, or a non-obvious consequence for callers. +- Use single backticks for code references, never RST-style double backticks. +- Wrap docstrings and prose comments to the full line length configured in `pyproject.toml`. +- Write only ASCII in source files: `->` instead of an arrow, `-` instead of a dash, straight quotes. +- Write `RQ` for the request queue. Log messages and public `.mdx` docs spell out the full term. -To run unit tests: +### Tests -```sh -uv run poe unit-tests -``` - -To run unit tests with XML coverage report: - -```sh -uv run poe unit-tests-cov -``` - -## Integration tests - -We have integration tests which build and run Actors using the Python SDK on the Apify platform. - -Prerequisites: - -- Set `APIFY_TEST_USER_API_TOKEN` to your Apify API token -- Optionally set `APIFY_INTEGRATION_TESTS_API_URL` to use a different Apify API environment - -To run integration tests: - -```sh -uv run poe integration-tests -``` +- Write flat `def test_*()` functions, never `class Test*`, and no section-header comments between tests. +- Almost every test gets a one-line docstring stating what it verifies, without a `Regression test:`-style prefix. +- In `@pytest.mark.parametrize`, wrap each case in `pytest.param(..., id='...')` with an id that describes the case. +- Use `Mock` and the `monkeypatch` fixture for test doubles, not hand-rolled stubs, counters, or flags. +- A test that changes process-wide or event-loop-wide state restores it. Prefer `monkeypatch`; for APIs it can't patch (`loop.set_exception_handler()`, `signal.signal()`, logging config, `os.chdir()`), capture the original value and restore it in a `finally`. +- On Windows with Python 3.12 or older, `datetime.now()` ticks in 1-15 ms steps, so two timestamps taken a few milliseconds apart can be equal. A test that asserts one timestamp is later than another injects a `Mock` clock instead of sleeping. +- For a TCP endpoint that must refuse connections, bind a socket to `('127.0.0.1', 0)` and hold it open without calling `listen()`. A hard-coded port can be taken by a server in a parallel xdist worker. Test servers use `127.0.0.1`, since on Windows `localhost` resolves to both IPv4 and IPv6. ## Documentation @@ -112,6 +68,10 @@ We adhere to the [Google docstring format](https://sphinxcontrib-napoleon.readth Our API documentation is generated from these docstrings using [pydoc-markdown](https://pypi.org/project/pydoc-markdown/) with additional post-processing. Markdown files in the `docs/` folder complement the autogenerated content. Final documentation is rendered using [Docusaurus](https://docusaurus.io/) and published to GitHub Pages. +Code examples in `docs/` have their own lint settings in `docs/pyproject.toml`. CI lints and type-checks them but doesn't run them, so run the standalone ones yourself after changing them, with `APIFY_LOCAL_STORAGE_DIR` pointing to a temporary directory. Guide titles in `docs/03_guides/` describe the task, such as "Scraping with BeautifulSoup and HTTPX2" or "Browser automation with Playwright". Don't change a page's `id:`, since it is part of the page URL and existing links would break. + +Never edit `website/versioned_docs/`: those snapshots are produced by the release automation and frozen against the code that shipped. Fix `docs/`, and the next release snapshot carries the change. + To run the documentation locally (requires Node.js): ```sh @@ -137,23 +97,25 @@ Doc images are committed as optimized `.webp`. To convert a new image, run ## Commits -We use [Conventional Commits](https://www.conventionalcommits.org/) format for commit messages. This convention is used to automatically determine version bumps during the release process. - -### Available commit types +We use [Conventional Commits](https://www.conventionalcommits.org/) format for commit messages. The release process reads them to determine the version bump and build the changelog. Pick the type by what the commit changes, not by why: | Type | Description | | ---- | ----------- | -| `feat` | A new feature | -| `fix` | A bug fix | -| `docs` | Documentation only changes | -| `style` | Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc) | -| `refactor` | A code change that neither fixes a bug nor adds a feature | -| `perf` | A code change that improves performance | -| `test` | Adding missing tests or correcting existing tests | -| `build` | Changes that affect the build system or external dependencies (example scopes: gulp, broccoli, npm) | -| `ci` | Changes to our CI configuration files and scripts (example scopes: Travis, Circle, BrowserStack, SauceLabs) | -| `chore` | Other changes that don't modify src or test files | -| `revert` | Reverts a previous commit | +| `feat` | A new feature in the source code | +| `fix` | A bug fix in the source code | +| `perf` | A source code change that improves performance | +| `refactor` | A source code change that neither fixes a bug nor adds a feature | +| `style` | A source code change that doesn't affect its meaning (white-space, formatting) | +| `test` | Adding or changing tests | +| `docs` | Documentation changes | +| `ci` | CI workflow changes | +| `build` | Build system changes | +| `chore` | Dependency bumps, tooling, generated files, and other housekeeping | +| `revert` | Reverting a previous commit | + +`feat`, `fix`, `perf`, `refactor`, and `style` trigger a release and appear in the changelog, so use them for source code changes only. `docs` triggers a docs release on master. + +Logging changes are breaking: changing a log level, a logger name, or the wording of a log message, or removing one, breaks downstream log-based tooling. Keep the accurate type, add `!` after it, and list the changes in a `BREAKING CHANGE:` footer. PRs are squash-merged, so the `!` must also be in the PR title. ## Release process diff --git a/README.md b/README.md index a08accc1..b73357dd 100644 --- a/README.md +++ b/README.md @@ -227,12 +227,7 @@ The full SDK documentation lives at **[docs.apify.com/sdk/python](https://docs.a ## Contributing -Bug reports, fixes, and improvements are welcome! See [CONTRIBUTING.md](./CONTRIBUTING.md) for the development setup, coding standards, testing, and release process. The project uses [uv](https://docs.astral.sh/uv/) for project management and [Poe the Poet](https://poethepoet.natn.io/) as a task runner; the typical loop is: - -```bash -uv run poe install-dev # install dev dependencies and git hooks -uv run poe check-code # lint, type-check, and unit tests -``` +Bug reports, fixes, and improvements are welcome! See [CONTRIBUTING.md](./CONTRIBUTING.md) for the development setup, coding standards, testing, and release process. ## License diff --git a/pyproject.toml b/pyproject.toml index bfb018bb..d72c6ee1 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -241,36 +241,42 @@ crawlee = false # Run tasks with: uv run poe [tool.poe.tasks] -clean = "rm -rf .coverage .pytest_cache .ruff_cache .ty_cache build dist htmlcov" -install-sync = "uv sync --all-extras" -build = "uv build" -publish-to-pypi = "uv publish --token ${APIFY_PYPI_TOKEN_CRAWLEE}" -type-check = "uv run ty check" -unit-tests = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} tests/unit" -unit-tests-cov = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} --cov=src/apify --cov-report=xml:coverage-unit.xml tests/unit" -integration-tests = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} tests/integration" -integration-tests-cov = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} --cov=src/apify --cov-report=xml:coverage-integration.xml tests/integration" -e2e-tests = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} tests/e2e" -e2e-tests-cov = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} --cov=src/apify --cov-report=xml:coverage-e2e.xml tests/e2e" -check-code = ["lint", "type-check", "unit-tests"] +clean = { cmd = "rm -rf .coverage .pytest_cache .ruff_cache .ty_cache build dist htmlcov", help = "Remove build artifacts and clean caches" } +install-sync = { cmd = "uv sync --all-extras", help = "Sync dependencies with all extras" } +build = { cmd = "uv build", help = "Build the package" } +publish-to-pypi = { cmd = "uv publish --token ${APIFY_PYPI_TOKEN_CRAWLEE}", help = "Publish the package to PyPI" } +type-check = { cmd = "uv run ty check", help = "Run the type checker" } +unit-tests = { cmd = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} tests/unit", help = "Run unit tests" } +unit-tests-cov = { cmd = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} --cov=src/apify --cov-report=xml:coverage-unit.xml tests/unit", help = "Run unit tests with an XML coverage report" } +integration-tests = { cmd = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} tests/integration", help = "Run integration tests" } +integration-tests-cov = { cmd = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} --cov=src/apify --cov-report=xml:coverage-integration.xml tests/integration", help = "Run integration tests with an XML coverage report" } +e2e-tests = { cmd = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} tests/e2e", help = "Run E2E tests" } +e2e-tests-cov = { cmd = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} --cov=src/apify --cov-report=xml:coverage-e2e.xml tests/e2e", help = "Run E2E tests with an XML coverage report" } +check-code = { sequence = ["lint", "type-check", "unit-tests"], help = "Run lint, type check, and unit tests" } [tool.poe.tasks.install-dev] shell = "uv sync --all-extras && uv run pre-commit install" +help = "Install development dependencies and pre-commit hooks" [tool.poe.tasks.lint] shell = "uv run ruff format --check && uv run ruff check" +help = "Run the linter and check formatting" [tool.poe.tasks.format] shell = "uv run ruff check --fix && uv run ruff format" +help = "Fix lint issues and format code" [tool.poe.tasks.update-docs-theme] shell = "pnpm update @apify/docs-theme" cwd = "website" +help = "Update the documentation website theme" [tool.poe.tasks.build-docs] shell = "./build_api_reference.sh && pnpm install && uv run pnpm build" cwd = "website" +help = "Build the documentation website" [tool.poe.tasks.run-docs] shell = "./build_api_reference.sh && pnpm install && uv run pnpm start" cwd = "website" +help = "Run the documentation website locally" diff --git a/tests/integration/README.md b/tests/integration/README.md index 4e39bcbd..ce49e197 100644 --- a/tests/integration/README.md +++ b/tests/integration/README.md @@ -27,3 +27,9 @@ To run against a different environment, also set `APIFY_INTEGRATION_TESTS_API_UR - **`apify_client_async`** — A session-scoped `ApifyClientAsync` instance configured with the test token and API URL. - **`request_queue_apify`** — Creates a parametrized (`single`/`shared` access mode) Apify request queue on the platform, yields it, and drops it after the test. Defined in `conftest.py`. - **`prepare_test_env`** / **`_isolate_test_environment`** (autouse) — Resets global state and sets `APIFY_LOCAL_STORAGE_DIR` to a temporary directory before each test. + +## Platform behavior to account for + +- Right after a request is added to a shared RQ, the head can still come back empty. Poll negative assertions (such as `is_finished()`) as well as positive ones. +- In the response of `update_request`, `wasAlreadyHandled` reflects the request's state before the update. +- RQ metadata counts (`handledRequestCount`, `pendingRequestCount`) lag a few seconds behind the operations that change them. diff --git a/tests/unit/README.md b/tests/unit/README.md index 24b8461d..fb8b8c90 100644 --- a/tests/unit/README.md +++ b/tests/unit/README.md @@ -13,3 +13,7 @@ uv run poe unit-tests - **`prepare_test_env`** / **`_isolate_test_environment`** (autouse) — Resets global state (Actor initialization, service locator, storage) and sets `APIFY_LOCAL_STORAGE_DIR` to a temporary directory before each test. - **`apify_client_async_patcher`** — Helper for patching `ApifyClientAsync` methods to return fixed values or replacement functions, with automatic call tracking. - **`httpserver`** — Local HTTP server (via `pytest-httpserver`) for testing HTTP interactions without real network calls. + +## Troubleshooting + +If tests fail with `SystemExit: 0` raised from `_actor.py` when `async with Actor:` exits, the virtual environment lacks the optional extras. Without Scrapy installed, the Actor exits the process on teardown. Run `uv sync --all-extras` (or `uv run poe install-dev`).