diff --git a/docs/architecture/background-jobs.md b/docs/architecture/background-jobs.md new file mode 100644 index 000000000..b33f7456c --- /dev/null +++ b/docs/architecture/background-jobs.md @@ -0,0 +1,420 @@ +--- +title: Background jobs +sidebar_position: 6 +--- + +# Background jobs + +Some work does not have to happen while a customer waits for a page: handing a mail to the +mail server, writing a large export, pushing stock to a marketplace. Thelia runs that work +through [Symfony Messenger](https://symfony.com/doc/current/messenger.html). Code dispatches a +message, a handler does the work, and the configuration of the shop decides whether the +handler runs at once or later, in a worker. + +This page explains the model. The configuration, the commands and a complete module example +are in the [background jobs reference](../reference/background-jobs.md); running the workers +on a server is covered in [Running the workers](../getting-started/background-workers.md). + +## What the core ships + +The core requires `symfony/messenger`, `symfony/doctrine-messenger`, `doctrine/dbal` and +`symfony/scheduler`. It does not require the Doctrine ORM or DoctrineBundle: DBAL is only +there to serve the queue kept in the shop database. + +There is one bus, `messenger.bus.default`, and four transports: + +| Transport | What it carries | Where it lives | +| --- | --- | --- | +| `async` | The jobs of the shop: mail delivery, the messages of modules | `MESSENGER_TRANSPORT_DSN`, synchronous when empty | +| `async_heavy` | Back-office exports and imports | Derived from `MESSENGER_TRANSPORT_DSN`, or `MESSENGER_HEAVY_TRANSPORT_DSN` | +| `failed` | The jobs that failed every attempt (the failure transport) | The shop database by default, whatever the queue is; `MESSENGER_FAILURE_TRANSPORT_DSN` overrides it | +| `scheduler_thelia` | The recurring tasks of the `thelia` schedule | Generated by the schedule, nothing to configure | + +## Nothing waits by default + +`MESSENGER_TRANSPORT_DSN` is shipped empty in `.env`. An empty value makes `async` and +`async_heavy` fall back to `sync://`: every message is handled immediately, in the request or the command that +dispatched it, exactly as if the handler had been called directly. Nothing is stored and +nothing waits for a worker, so a shop that runs no worker loses no job. A failure in the +handler surfaces in the code that dispatched the message, as any exception would. + +The shop opts in to a real queue by naming one: + +```bash +# .env.local +MESSENGER_TRANSPORT_DSN=doctrine://default +``` + +From then on, messages are stored and wait for `messenger:consume`. Setting the variable +without running a worker leaves mails undelivered and exports and imports pending, so the two +go together. + +| DSN | Queue | +| --- | --- | +| empty | No queue, every job runs at once | +| `doctrine://default` | Table `messenger_messages` of the shop database | +| `redis://host:6379/messages` | A Redis stream, once `symfony/redis-messenger` is installed | +| `amqp://user:password@host:5672/%2f/messages` | An AMQP broker such as RabbitMQ, once `symfony/amqp-messenger` is installed | + +The core only ships the Doctrine transport. A Redis queue needs +`composer require symfony/redis-messenger` and the `redis` PHP extension; an AMQP queue needs +`composer require symfony/amqp-messenger` and the `amqp` extension. + +### Choosing a transport + +Start with `doctrine://default`. It needs nothing to install and no server to run: the queue is +a table of the shop database, saved with its backups, and the failed jobs are kept in that +database whatever the queue is. Workers on several servers can share it, since they all reach +the shop database. The jobs of a shop (mails, back-office exports and imports, the messages of a +few modules) are well within what a table can serve. + +Move to another transport for one of these reasons: + +- the volume: modules that dispatch many messages, a push per product combination on a large + catalog for instance, keep the workers reading and writing the table all day. Redis or a + broker takes that load off the database the shop serves its pages from, +- the hosting already runs a Redis server or a RabbitMQ broker, with its monitoring and its + backups: use it rather than adding a queue to the database, +- the team already operates AMQP for other applications and wants the shop jobs in the same + place. + +Redis is the shorter step: one package, one PHP extension, and the heavy queue is derived from +the DSN. AMQP also needs `MESSENGER_HEAVY_TRANSPORT_DSN` set, see below. Whatever the transport, +a Redis or AMQP server holds personal data and must stay private, see +[Personal data](../security/personal-data.md#background-jobs). + +### A second queue for heavy jobs + +Exports and imports are routed to `async_heavy`, so a large import never holds up the mails +waiting on `async`. Its DSN is derived from `MESSENGER_TRANSPORT_DSN`: + +| `MESSENGER_TRANSPORT_DSN` | `async_heavy` | +| --- | --- | +| empty | `sync://`, the job runs at once | +| `doctrine://...` | The same table, with `queue_name=heavy` | +| `redis://host:port//...` | The stream `_heavy`, or `messages_heavy` when the DSN names no stream | +| anything else | The same DSN: both transports share one queue | + +`MESSENGER_HEAVY_TRANSPORT_DSN` overrides the derived value. On AMQP, or any queue other than +Doctrine and Redis, set it: sharing the queue of `async`, the exports and imports are read by +the worker of `async` and tried again three times like its jobs. A worker started with +`messenger:consume async async_heavy` takes the mails first, then the heavy jobs; a shop can +also run one worker per transport. + +## The queue in the shop database + +`doctrine://default` opens a DBAL connection with the settings of the Propel connection, its +PDO options and attributes included (TLS certificates, for instance), so the queue lives in the +same database as the shop, with no extra service. Only the connection name `default` is +accepted. + +The `messenger_messages` table is created by the installer and by the 3.3.0 update script. +The transport never creates it on the fly. + +The DBAL connection is a second connection to the database, independent from Propel. A +message dispatched inside a Propel transaction is written to the queue at once, and stays +there even if that transaction is rolled back. A worker can then pick up a message about a +row that was never committed. Dispatch after the commit, never inside the transaction. + +## Events or messages + +Thelia events and messages answer different needs. + +An event, dispatched through the event dispatcher, tells every listener that something +happens, and every listener runs before `dispatch()` returns. Listeners can change the event, +stop its propagation, or make the action fail. Events stay synchronous whatever the +configuration of the queue: the event dispatcher is not a message bus. See the +[events reference](../reference/events.md). + +A message asks for one piece of work to be done by its handler, now or later, without the +caller depending on the result. Use a message when: + +- the work is slow or calls a remote service (a mail server, a marketplace, an ERP), +- the caller does not need its result to answer, +- the work can be retried on its own if it fails. + +The two combine. A listener that reacts to an event but does not need to do the work in the +request dispatches a message, and the handler does the work: + +```php +public function onStockUpdated(ProductSaleElementUpdateEvent $event): void +{ + $this->bus->dispatch(new PushStock($event->getProductSaleElementId())); +} +``` + +The listener stays fast, and the handler gets the retries and the failure handling described +below. + +## Retries and failed jobs + +When a handler throws on the `async` transport, the message is retried three times, after +30 seconds, 2 minutes and 8 minutes (a multiplier of 4, plus the random jitter Symfony adds). +After the last attempt it goes to the `failed` transport. A message is never lost and never +retried in a loop. `async_heavy` has no retries (`max_retries: 0`): a message that fails there +goes to `failed` at once, since running a whole export or import again unchanged would fail the +same way. + +A handler that knows a retry cannot help, for instance because the remote service rejected +the data, throws `Symfony\Component\Messenger\Exception\UnrecoverableMessageHandlingException` +and the message goes to `failed` at once. + +`failed` is `MESSENGER_FAILURE_TRANSPORT_DSN`, which defaults to +`doctrine://default?queue_name=failed`: by default the failed jobs are kept in the shop +database, whatever queue `async` uses, so the back office has a single place to read them from. +They stay there until someone replays them, removes them, or a purge deletes them after 30 +days (`thelia:messenger:purge-failed` and `maintenance:purge`). When `failed` is in the shop +database, replaying or deleting a job from the back office acts once, even when two +administrators, or a double click, act on the same job at the same moment. Replaying the same +job from the back office and with `messenger:failed:retry` at the same moment is not guarded: +see [Known limits](../reference/background-jobs.md#known-limits). + +Messenger writes the exception of a failed or retried job to the log of the workers, and a +database error quotes the values of a row. The core takes the exception out of these log lines, +on the `messenger` channel, and names it by its class, code and place in the code instead. The +back office shows the reason of a failed job only when it was written for the administrator, or +when it is the answer of the mail server with its credentials hidden; anything else reads as a +server error. The rule is detailed in the [reference](../reference/background-jobs.md#back-office). + +Because a failed job can be replayed, and because a retry runs a handler that may have done +part of its work, a handler must be idempotent: run twice, its effect happens once. The +reference shows how in [Idempotence and replay](../reference/background-jobs.md#idempotence-and-replay). + +## What a worker sees + +A handler running in a worker serves no page: there is no visitor, no cart and no HTTP +request of a browser. Each message gets a request of its own on the address of the shop +(`DEFAULT_URI`, its folder included), with an empty session, so the loops of a template a +handler renders, a mail of a module for instance, can run. Nothing a page sets up runs for it: +the session language is the default language, the request is not one of the back office, and +no customer or administrator is restored from a cookie. The request goes once the message is +handled or has failed, and a message the worker skipped never hands its request to the next. + +The handler still runs in a long process that handles one message after another, so the core +resets the state of the shop before each message: + +- settings are read again from the database, so a change made in the back office since the + previous message is seen, +- the active languages, the default country and currency, the module configuration, the + rewritten URLs, the price currency, the taxes of a rule, the return statuses and the results + of the loops are forgotten, +- the translator is set back to the default language. + +Before each message, the Propel connection is also checked and reopened if MySQL closed it +after its `wait_timeout`. Propel instance pooling is disabled for the whole life of the worker, +so a handler never reads a model object kept in memory by a previous message. + +Once a message is handled or has failed, and before the worker acknowledges it, the DBAL +connection of the queue in the shop database is closed. The acknowledgement, +or the retry, goes through a connection opened afresh, which MySQL cannot have closed while a +long job kept the worker busy. + +Symfony also resets the services tagged `kernel.reset` between messages, unless the worker is +started with `--no-reset`. + +What a worker does not read again is its container: the mail settings and the handlers of the +modules active when it started. Clearing the cache of the shop (saving the mail settings, +turning a module on or off, `thelia:cache:clear`) therefore asks every worker to stop once its +current message is done, as `messenger:stop-workers` does, and its supervisor starts it again on the new cache. +A clear asked for by a job (a module handler that saves a setting), or by the command of a +recurring task the worker runs, stops the worker running it once the job is acknowledged, and +the cache is cleared when that worker's command ends, an exception included: clearing it in +the middle would delete the files the worker still loads its own listeners from. A worker +stopped by a fatal error or killed loses the clear: run `php Thelia thelia:cache:clear` then. +A clear of the image or document cache leaves the workers alone. The signal is written in the +application cache pools, so workers running on another server only see it when +`THELIA_CACHE_DSN` points to a cache server they share; otherwise, run +`php Thelia messenger:stop-workers` on their server. + +A handler therefore never relies on what a visitor would have set. It receives ids, reads its +data again, chooses the locale explicitly, and builds absolute URLs from `DEFAULT_URI`. + +## What the core sends through the queue + +The core routes three jobs: mails to `async`, exports and imports to `async_heavy`. + +### E-mails + +`Symfony\Component\Mailer\Messenger\SendEmailMessage` is routed to `async`. The mail is +rendered in the request, where the language and the shop URL are known; only the delivery to +the mail server is deferred. With a queue, a slow or unreachable mail server no longer holds +up the page that sends the mail, and a mail that could not be delivered ends up in `failed`, +ready to be sent again. + +The order history line `email_sent` is written when the mail server accepted the mail. A +listener on the Symfony Mailer `SentMessageEvent` reads the `X-Thelia-Order-Id` and +`X-Thelia-Message-Code` headers of the mail. These headers travel through the queue with the +mail and are removed right before it is handed to the mail server, so the customer never +receives them. With a worker, the line appears when the worker delivered the mail, and its +author is `system`. + +A queue may deliver a job twice: when a worker stops after the mail server accepted the mail +but before the job was acknowledged, the transport redelivers the job once its +`redeliver_timeout` is over. `MailerFactory` gives each mail it sends an identifier in an +`X-Thelia-Mail-Id` header, a new one on each send, even when the same `Email` object is sent +again. A worker keeps the identifiers of the mails the server accepted for a week, and does not +send a mail whose identifier it kept. Workers on several servers share them when +`THELIA_CACHE_DSN` points to a cache server they share. A mail sent with the Symfony mailer +directly, not through `MailerFactory`, is not guarded: see +[Known limits](../reference/background-jobs.md#known-limits). + +With a queue, `MailerFactory::send*OrFail()` only throws when the mail could not be built or +queued. A delivery failure happens later, in the worker. `MailerFactory::sendNow()` hands a mail +to the mail server at once, queue or not: the back office uses it to test the mail +configuration, whose only point is the answer of the server. + +A queued mail holds everything it is sent with, and names no file of the server: + +- an attachment given by its path (`attachFromPath()`) is read when the mail is queued and + stored in the queue with its content. It is held in the memory of the request, and a large + one is bounded by the `max_allowed_packet` of the database, +- a mail is read back from the queue only when its class is exactly `RawMessage`, `Message` or + `Email`, or a `TemplatedEmail` or `NotificationEmail` already rendered, and when every part is + one Symfony builds a mail with. A part that names a file, in the mail or in a mail it carries + (`MessagePart`), a part of another class, or a part that throws when it is looked at, makes + the job an `UndecodableJob`, never sent, +- a mail that would be refused is refused when it is queued, with a `LogicException`. Symfony + queues a `TemplatedEmail` before rendering it, and the worker renders nothing: render it + first (`BodyRenderer`), or send it without a queue. + +### Back-office exports and imports + +An export or an import started from the back office is a job. The launcher +(`Thelia\Domain\DataTransfer\Job\ExportJobLauncher` or `ImportJobLauncher`) records a row in +`export_job` or `import_job` and dispatches `RunExportJob` or `RunImportJob`, routed to +`async_heavy`. The row is what the back office reads. Without a queue, the job runs in the +request and the page shows its outcome at once. With a queue, the page of the job follows it: +waiting, running, done, or failed with its reason. + +An import file is checked in the request, before anything is queued, and moved to +`var/data-transfer/import/`. It is kept outside the cache on purpose: a deployment empties the +cache, and a queued import would lose its file. + +An import runs in a single database transaction. Stopped half way, by an error, a killed +worker or a deployment, it leaves the catalog as it was, and its outcome is written to its row +in the same transaction, so the catalog and the row never disagree. The cost is that the rows it +writes stay locked until it ends: an order or a back-office edit touching one of them waits, then +fails after `innodb_lock_wait_timeout`. Run a large stock or price import outside trading hours. +When the code that launches an import already holds a transaction, the import works inside it +and leaves the commit or the rollback to that code, so do not launch an import inside a +transaction of your own. The listeners of `IMPORT_BEGIN`, `IMPORT_FINISHED` and `IMPORT_SUCCESS` +run inside the transaction too: what they write is cancelled with the import, what they send is +not, and a job they queue in the shop database is committed at once, so a worker may run it +before the import is committed. + +A job never runs twice at the same time, and a finished job never runs again. A worker claims +the row of the job before running it. A message that finds its job still running, because +another worker holds it or because its worker died, does not run it: it is queued again to look +at the job later. The job is taken over only once its row has shown no progress for an hour, so +a long export or import that is still working is never started a second time. What this means +for an operator after a worker is killed is in +[Running the workers](../getting-started/background-workers.md#when-a-worker-stops). + +A job that fails goes to `failed` at once, with no retry: running a whole export or import +again unchanged would fail the same way. Replaying it from the failed jobs starts it over, an +import from its first row. + +The reason a job failed is shown to the administrator only when it was written for them, by an +exception implementing `Thelia\Exception\UserFacingFailure`. Any other failure may quote SQL, +the values of a row, paths or host names: the administrator reads that the job failed because of +a server error, and the log names the exception by its class, code and place in the code, never +by its message. + +An export or an import is one job for one file. It is not split into smaller jobs: a file is +written by a single writer, and an import has to be atomic. Splitting work into smaller jobs +fits work made of independent items, as in a module that pushes one message per product +combination. It does not fit the rows of a file: do not dispatch one message per row. + +The launchers, the rights, the limits of an uploaded file and the retention of the jobs and their +files are in the [reference](../reference/background-jobs.md#back-office-exports-and-imports). + +## Recurring tasks + +The recurring tasks of the shop are declared in a Symfony Scheduler schedule named `thelia` +(`Thelia\Scheduler\TheliaSchedule`). A worker consuming `scheduler_thelia` runs them; until +one does, nothing runs, and a shop can keep its crontab instead. The schedule is stateful and +locked through `LOCK_DSN`, so two workers never run a task twice, and a worker that restarts +after a missed run catches up the last one only. Across servers this holds only when both are +shared: `LOCK_DSN` must name a store every worker reaches (not `flock` nor `semaphore`, which +lock one server only), and +`THELIA_CACHE_DSN` a cache server they all use; otherwise each server runs the tasks on its own, +and a cache clear forgets the last runs. The tasks and their settings are listed in +the [reference](../reference/background-jobs.md#recurring-tasks). + +A task of the schedule that fails does not go to the failure transport: Messenger only sends +there what came from a queue, and the task runs again at its next time anyway. The core logs the +failure and keeps the last one of each task, in the application cache, until a run of that task +goes through, or for a month after the date of that failure, so a task taken off the schedule +does not stay listed forever. The Background jobs screen lists them under "Recurring tasks that +failed", see the [reference](../reference/background-jobs.md#back-office). + +## Which classes a queue accepts + +A queued message is data that is turned back into objects when a worker reads it, and whoever +can write to the queue would choose their classes. Thelia never uses PHP serialization for +this. Envelopes are JSON, written and read by the Symfony serializer through +`Thelia\Messenger\Serializer\AllowedClassesSerializer`, which checks the classes before +building anything: + +- a message class must be under `Thelia\`, under the namespace of an active module (the + namespace named after the module code, such as `MyModule\Message\SyncStock`), + `Symfony\Component\Mailer\Messenger\SendEmailMessage`, or listed by its exact name in the + container parameter `thelia.messenger.allowed_message_classes`, +- outside that parameter, a handler of the application must also take the message class, or + one of its parents or interfaces. The handled classes are collected when the container is + built, so no other class of those namespaces is ever built from a queue. A handler that takes any message (`*` or `object`) does not count, +- a stamp must be a Messenger stamp or come from those same namespaces, and `SerializerStamp` + is refused, as it would let the queue change how the rest of the envelope is read. +- a message that runs a console command, a process, an HTTP request or a scheduled service + call, or that dispatches another message (the classes under `Symfony\Component\Console\Messenger\`, + `Process\Messenger\`, `HttpClient\Messenger\`, `Scheduler\Messenger\`, `Cache\Messenger\` and + `Messenger\Message\`), is never built from a queue, even when a project lists it: whoever writes + to the queue would run anything on the server. The schedule of the shop is not affected, as + its messages never go through a queue. + +Whether a message class belongs to an active module is read from the database each time. When +MySQL closed the connection of the worker during a long wait, the Propel connection is opened +again once and the module is read again. + +The check runs when a message is read from a queue and when it is written to one. A module +dispatching to a real queue a class the workers would refuse gets a `LogicException` ("is not +one the shop queues") at dispatch time, not a queue that never empties. Without a queue +(`sync://`), the message is handled in the same call and never serialized, so the check does +not run: a message that no handler takes fails with Symfony's `NoHandlerForMessageException` +instead, also at dispatch time. + +A job already queued can still become unreadable: its class is refused by the check, or it can +no longer be built because its module was turned off, its class was removed, or its content no +longer fits the class. The worker then reads it as `Thelia\Messenger\Message\UndecodableJob`, +which keeps the original class (`originalType`), the reason and the original body, instead of +letting it be dropped. Whatever the inner serializer throws while building the message, the +reason is a fixed one, `Its content no longer fits the class .`, and the exception is +named in the log by its class, code and place: a message that cannot be built never stops the +worker, however often it is delivered. The handler of an `UndecodableJob` sends the job to +`failed` at once. The Background jobs screen lists it as `Unreadable job: ` with its +original class, and it can be deleted there. + +For a module author, this means: + +- message classes live in the module namespace, +- each message class has a handler: a message that no handler takes fails at dispatch, with + a `LogicException` when it goes to a queue, with a `NoHandlerForMessageException` without one, +- they can be written and read as JSON by the Symfony serializer: public or promoted + constructor properties with scalar types, such as ids and codes, +- they never carry a Propel model. The handler reads the row again from its id, +- they implement `Thelia\Messenger\Message\DescribedJob`, so the back office lists a failed job + by a line that says what it was, with no personal data in it. See + [the message of the example module](../reference/background-jobs.md#the-message). + +A class from outside these namespaces, such as an application class under `App\`, is +accepted only once a project adds its exact name to `thelia.messenger.allowed_message_classes`. +The check for files a mail would make the worker read only applies to Symfony's +`SendEmailMessage`: a listed class that carries an `Email` of its own must refuse attachments by +path itself. + +## Personal data + +A queue holds personal data: a queued mail carries the address of its recipient and the content +of the order, and a failed job keeps everything it was dispatched with. How long each is kept, +and how to protect a Redis or AMQP server, is in +[Personal data](../security/personal-data.md#background-jobs). diff --git a/docs/back-office/order-history.md b/docs/back-office/order-history.md index a65fc220b..1ca6d537b 100644 --- a/docs/back-office/order-history.md +++ b/docs/back-office/order-history.md @@ -144,8 +144,12 @@ The recorder: The listeners that feed the journal are `RecordOrderHistoryListener` (order creation, payment, status, delivery and transaction references, address) and `RecordOrderReturnHistoryListener`. Invoice numbering is recorded where the reference is allocated, by the automatic path -(`invoice_ref_auto`) and by the `allocate_invoice_ref` status action. `MailerFactory` records -`email_sent` when the message parameters carry `order_id` or `order_ref`. +(`invoice_ref_auto`) and by the `allocate_invoice_ref` status action. `email_sent` is recorded +by `Thelia\Mailer\EventListener\OrderEmailHistoryListener` on the Symfony Mailer +`SentMessageEvent`, once the mail server has accepted the mail; with a queue, the worker +writes it, and its author is `system`. `MailerFactory` only names the order in the +`X-Thelia-Order-Id` and `X-Thelia-Message-Code` headers of the mail, when the message +parameters carry `order_id` or `order_ref`. A payment module that dispatches an `OrderEvent` should name itself with `OrderEvent::setSourceModuleCode()`, so the line is authored by the module. The base payment diff --git a/docs/getting-started/background-workers.md b/docs/getting-started/background-workers.md new file mode 100644 index 000000000..ef4b7e411 --- /dev/null +++ b/docs/getting-started/background-workers.md @@ -0,0 +1,301 @@ +--- +title: Running the workers +sidebar_position: 4.6 +--- + +# Running the workers + +A shop that names a queue in `MESSENGER_TRANSPORT_DSN` needs at least one worker: a long +`messenger:consume` process that handles the queued jobs. Without a queue, nothing waits for +a worker and this page only matters for the recurring tasks. The model is explained in +[Background jobs](../architecture/background-jobs.md), the settings in the +[background jobs reference](../reference/background-jobs.md). + +## The workers to run + +```bash +php Thelia messenger:consume async --time-limit=3600 --memory-limit=256M +php Thelia messenger:consume async_heavy --time-limit=3600 --memory-limit=256M +php Thelia messenger:consume scheduler_thelia --time-limit=3600 --memory-limit=256M +``` + +Mails and the messages of modules go to `async`, back-office exports and imports to +`async_heavy`, so a large import does not hold up the mails. One worker per transport keeps +them fully apart: a mail never waits behind an import, and the scheduled tasks never wait +behind either. A smaller setup can run a single worker on +`messenger:consume async async_heavy`, which takes the mails first, then the heavy jobs, plus +the scheduler worker. + +Do not consume `scheduler_thelia` when the same commands already run from a crontab +(`maintenance:purge`, `sale:check-activation`, `thelia:messenger:purge-failed`): each would run +twice. See the warning in the [recurring tasks reference](../reference/background-jobs.md#recurring-tasks). + +With workers on several servers, run the scheduler worker on one of them, or share the lock +and the cache: `LOCK_DSN` must name a store every server reaches and `THELIA_CACHE_DSN` a cache +server they all use, which keeps the last run of each task. `flock`, the default of the core, and +`semaphore`, the default of a project created from `thelia/thelia-project`, both lock one server +only. Otherwise each server runs every task. + +The queue in the shop database is the one to start with; when to prefer Redis or AMQP is +explained in [Choosing a transport](../architecture/background-jobs.md#choosing-a-transport). +A Redis or AMQP queue needs its Messenger bridge, which the core does not ship: +`composer require symfony/redis-messenger` (with the `redis` PHP extension) or +`composer require symfony/amqp-messenger` (with the `amqp` extension). The queue in the shop +database (`doctrine://default`) needs nothing more. On AMQP, also set +`MESSENGER_HEAVY_TRANSPORT_DSN` to a queue of its own: otherwise `async_heavy` shares the queue +of `async`, its exports and imports are read by the `async` worker and tried again three times +like the mails. + +A worker on `async` or `async_heavy` is only useful once `MESSENGER_TRANSPORT_DSN` names a queue. Started on a +shop without a queue, it sits idle and does nothing, because every job already ran in the +request that dispatched it: harmless, but useless. `php Thelia messenger:stats` and the +Background jobs screen of the back office ("Queue: None") tell whether a queue is configured. +The scheduler worker is useful either way, if the shop runs its recurring tasks from the +schedule rather than from a crontab. + +`--time-limit` and `--memory-limit` make the worker exit cleanly after an hour or once it uses +256 MB, and the process manager starts a fresh one. A PHP process that runs for days grows in +memory, and recycling the worker keeps it in check. A connection that MySQL closed after its +`wait_timeout` is reopened before the next message. The connection of the queue in the shop +database is closed after each message, before the worker acknowledges it, so a long export or +import never leaves the acknowledgement on a connection MySQL has dropped. + +## When a worker stops + +A worker stopped cleanly, by `SIGTERM` or `SIGINT` from its process manager, by +`php Thelia messenger:stop-workers`, or for a deployment, finishes the message in hand, then +stops. Nothing is lost: the messages it did not take wait in the queue for the next worker. + +A worker stopped hard (`SIGKILL`, out of memory, a server that goes down) leaves the message it +was handling marked as delivered. The Doctrine transport hands it out again once its +`redeliver_timeout` has passed: 3600 seconds by default, set in the DSN, for instance +`MESSENGER_TRANSPORT_DSN=doctrine://default?redeliver_timeout=600`. The heavy queue is derived +from the same DSN and takes the same value. The handler then runs a second time, which is one +reason handlers must be [idempotent](../reference/background-jobs.md#idempotence-and-replay). + +An export or an import is not taken over that fast. Its job row records its progress, and the +job is only started again once that row has shown no progress for an hour. A redelivered message +that finds the row still fresh looks again 10 minutes later. An export or an import cut by a hard +stop therefore starts over about an hour after the worker died, an hour and ten minutes at most, +and a shorter `redeliver_timeout` does not make it sooner. This is on purpose: a long export or +import that is still working on another worker is never started a second time. A job still held +after 12 hours of such checks is sent to the failure transport, `failed`, where it can be replayed. + +## Supervisor + +```ini +; /etc/supervisor/conf.d/thelia-worker.conf +[program:thelia-async] +command=php /var/www/shop/Thelia messenger:consume async --time-limit=3600 --memory-limit=256M +directory=/var/www/shop +user=www-data +numprocs=1 +process_name=%(program_name)s_%(process_num)02d +autostart=true +autorestart=true +startsecs=0 +stopsignal=TERM +stopwaitsecs=300 +redirect_stderr=true +stdout_logfile=/var/www/shop/var/log/worker-async.log + +[program:thelia-async-heavy] +command=php /var/www/shop/Thelia messenger:consume async_heavy --time-limit=3600 --memory-limit=256M +directory=/var/www/shop +user=www-data +numprocs=1 +process_name=%(program_name)s_%(process_num)02d +autostart=true +autorestart=true +startsecs=0 +stopsignal=TERM +stopwaitsecs=300 +redirect_stderr=true +stdout_logfile=/var/www/shop/var/log/worker-async-heavy.log + +[program:thelia-scheduler] +command=php /var/www/shop/Thelia messenger:consume scheduler_thelia --time-limit=3600 --memory-limit=256M +directory=/var/www/shop +user=www-data +numprocs=1 +process_name=%(program_name)s_%(process_num)02d +autostart=true +autorestart=true +startsecs=0 +stopsignal=TERM +stopwaitsecs=300 +redirect_stderr=true +stdout_logfile=/var/www/shop/var/log/worker-scheduler.log +``` + +```bash +supervisorctl reread +supervisorctl update +supervisorctl status +``` + +`stopwaitsecs` is how long supervisor waits for the current message before killing the +process. Set it above the duration of your longest job: on `async_heavy`, a large export or +import. + +## systemd + +```ini +# /etc/systemd/system/thelia-worker@.service +[Unit] +Description=Thelia worker (%i) +# Name the unit of your database server, or drop it when the database runs elsewhere. +After=network.target mariadb.service + +[Service] +User=www-data +WorkingDirectory=/var/www/shop +ExecStart=/usr/bin/php /var/www/shop/Thelia messenger:consume %i --time-limit=3600 --memory-limit=256M +Restart=always +RestartSec=5 +KillSignal=SIGTERM +TimeoutStopSec=300 + +[Install] +WantedBy=multi-user.target +``` + +```bash +systemctl daemon-reload +systemctl enable --now thelia-worker@async thelia-worker@async_heavy thelia-worker@scheduler_thelia +``` + +`Restart=always` starts a new worker each time one exits on its time or memory limit. + +## Shared hosting: the crontab fallback + +A hosting that does not allow long processes can still consume the queue from cron: + +```bash +* * * * * cd /var/www/shop && php Thelia messenger:consume async async_heavy scheduler_thelia --time-limit=55 +``` + +Drop `scheduler_thelia` from this line if `maintenance:purge` and the other tasks stay in your +crontab: each would run twice. Each run consumes for 55 seconds, then exits before the next one starts. Jobs wait up to a +minute before they run. Two runs that overlap are safe: each message is handed to one worker +only. + +## DDEV + +In development, the synchronous default is usually all you need: mails reach Mailpit during +the request, exports are served and imports run at once. To try the queue, set +`MESSENGER_TRANSPORT_DSN=doctrine://default` in `.env.local` and let DDEV run the worker. + +The `thelia/thelia` repository ships `.ddev/config.worker.yaml` for that. A project created +with `composer create-project thelia/thelia-project` has no `.ddev/` directory: after +`ddev config`, create the file with this content: + +```yaml +# .ddev/config.worker.yaml +web_extra_daemons: + - name: messenger-worker + command: "php bin/console messenger:consume async async_heavy --time-limit=3600 --memory-limit=256M" + directory: /var/www/html +``` + +Run `ddev restart` to start it. Without a queue the worker waits and does nothing. It leaves +out `scheduler_thelia` on purpose, since the recurring tasks would purge the carts and the logs of +the development database at night; run them by hand with +`ddev exec php bin/console messenger:consume scheduler_thelia`. + +Set `MESSENGER_TRANSPORT_DSN` in `.env.local`, not under `web_environment` in +`.ddev/config.yaml`: a value set there wins over the `.env` files, for every command run in the +container. The worker runs the code it loaded at start: after changing a handler, run +`ddev exec php Thelia messenger:stop-workers` so it restarts on the new code. To stop the worker +for a while, run `ddev exec "supervisorctl stop webextradaemons:messenger-worker"`. + +To kill workers by hand, as when testing a hard stop, match the command with a pattern that +does not match itself: `pkill -f '[m]essenger:consume'`. A plain `pkill -f messenger:consume` +run through a shell also matches that shell, and kills it. + +## Monitoring + +`messenger:stats` prints the number of messages waiting in each transport: + +```bash +php Thelia messenger:stats +php Thelia messenger:stats async async_heavy failed +``` + +Wire it into the monitoring of the hosting and alert on two signals: + +- `async` above a threshold for several minutes in a row (for instance more than 100 messages + for 10 minutes): the workers are stopped or cannot keep up. `async_heavy` holds few, long + jobs; alert when a job waits there longer than your longest export or import should take, +- `failed` above zero: a job failed every attempt and needs someone to read its reason + (`php Thelia messenger:failed:show`) and replay or remove it. + +The back office shows the same counts under Configuration > System > Background jobs, where +the waiting count adds both queues, or counts them once when `async` and `async_heavy` share the +same DSN. The same screen lists the recurring tasks whose last run failed: a failed task of +`scheduler_thelia` never reaches `failed`, so `messenger:stats` does not count it. + +## Deploying + +A worker runs the code it loaded when it started. Left running through a deployment, it +consumes the messages of the new version with the old code. Stop the workers before touching +the cache, and start them once the new release is ready: + +1. Stop the workers through the process manager, which sends `SIGTERM` and waits for the + current message: `supervisorctl stop thelia-async:* thelia-async-heavy:* thelia-scheduler:*`, or + `systemctl stop thelia-worker@async thelia-worker@async_heavy thelia-worker@scheduler_thelia`. +2. Deploy the code and run the update script (see [Updating](../upgrading/update.md)). +3. Rebuild the cache: remove `var/cache/prod` and `var/propel/prod`, then run + `php Thelia cache:warmup --env=prod`. Never run `cache:clear` while a worker runs. +4. Start the workers: `supervisorctl start thelia-async:* thelia-async-heavy:* thelia-scheduler:*`, or + `systemctl start thelia-worker@async thelia-worker@async_heavy thelia-worker@scheduler_thelia`. + +When the release is switched atomically (a symlink pointing to the new release directory), +the workers can instead be restarted after the switch with +`php Thelia messenger:stop-workers`: each worker stops after its current message, and the +process manager starts new ones on the new release. The command leaves its signal in the +application cache, and a worker only sees it in the cache it reads: when each release has its +own `var/` directory, run the command from the release the workers were started from. + +Outside a deployment, a cache clear made by the shop (`php Thelia thelia:cache:clear`, saving +the mail settings, turning a module on or off in the back office) stops the workers by itself +once their current message is done, and the process manager starts them again on the new cache. +Removing `var/cache/prod` by hand sends no such signal: stop the workers first, as above. + +Messages queued during the deployment wait in the queue and are handled once the workers are +back. The file of a queued import is kept in `var/data-transfer/import/`, outside the cache, +so rebuilding the cache does not lose it. + +The web servers and the workers must share two directories: `var/data-transfer/import/`, where +an uploaded import waits for its worker, and `var/cache/export/`, where a worker writes the +export file the back office serves. When each release has its own `var/` directory, or the +workers run on another machine, point both at the same storage, and never empty them in a +deployment while jobs wait. + +Run the web server, the workers and any crontab or script that exports or runs +`maintenance:purge` under the same system user, or under users that share a group. The shop +creates the exports, their archives and the uploaded imports readable by their owner and their +group only (mode 0640), since they hold the data of the customers, in folders it makes writable +by their group (mode 0770), since a worker deletes the import the web server stored and the purge +removes what either left; the group reads a file, it never rewrites one. A folder made before keeps +its mode: when the users differ, check `var/data-transfer/import` and the export folder of the +cache, and give them `chmod 0770` if they were made more open or more closed. +An export written by a worker under a user outside that group could not be served by the back +office, nor an import it stored read by a worker. + +## Personal data in the queue + +Queued mails carry the recipient address and the content of the order, and a failed job keeps +everything it was dispatched with. The failed jobs are deleted after 30 days by +`thelia:messenger:purge-failed` and `maintenance:purge`: a shop that does not consume the +schedule keeps both in its crontab. The retention periods, and how to protect a Redis or AMQP +server, are in [Personal data](../security/personal-data.md#background-jobs). + +## Sizing + +- One worker per transport is enough for most shops. Add `async` or `async_heavy` workers + (`numprocs` in supervisor) when `messenger:stats` shows that queue growing. +- Every worker is a PHP process. Count them against the PHP processes the server can run and + the connections MySQL accepts: workers that take every slot leave the web requests waiting. +- Keep `--memory-limit` below the `memory_limit` of PHP CLI, so the worker exits on its own + limit rather than being killed in the middle of a message. diff --git a/docs/getting-started/configuration.md b/docs/getting-started/configuration.md index 15c447662..62c14d23a 100644 --- a/docs/getting-started/configuration.md +++ b/docs/getting-started/configuration.md @@ -42,8 +42,32 @@ JWT_PASSPHRASE=your-passphrase # Lock LOCK_DSN=flock + +# Background jobs: empty means no queue, every job runs at once +MESSENGER_TRANSPORT_DSN= ``` +### Background jobs and recurring tasks + +These variables decide whether mails, exports, imports and the jobs of modules wait in a queue +for a worker, and when the recurring tasks run. Only `MESSENGER_TRANSPORT_DSN` is written in +`.env`; the others take the default below when they are not set. + +| Variable | Default | Effect | +| --- | --- | --- | +| `MESSENGER_TRANSPORT_DSN` | empty | The queue of the jobs. Empty, every job runs at once in the request that dispatched it. `doctrine://default` keeps the queue in the shop database. | +| `MESSENGER_HEAVY_TRANSPORT_DSN` | derived from `MESSENGER_TRANSPORT_DSN` | The queue of the back-office exports and imports. | +| `MESSENGER_FAILURE_TRANSPORT_DSN` | `doctrine://default?queue_name=failed` | Where the jobs that failed every attempt are kept. | +| `THELIA_SCHEDULE_SALE_CHECK` | `* * * * *` | When `sale:check-activation` runs. Empty disables it. | +| `THELIA_SCHEDULE_MAINTENANCE_PURGE` | `30 3 * * *` | When `maintenance:purge` runs. Empty disables it. | +| `THELIA_SCHEDULE_FAILED_JOBS_PURGE` | `0 4 * * *` | When `thelia:messenger:purge-failed` runs. Empty disables it. | +| `THELIA_SCHEDULE_CURRENCY_RATES` | empty | When `currency:update-rates` runs. Off by default. | + +The recurring tasks only run when a worker consumes the `scheduler_thelia` transport. Naming a +queue in `MESSENGER_TRANSPORT_DSN` without running a worker leaves the mails undelivered. See the +[background jobs reference](../reference/background-jobs.md#configuration) for the details of +each variable, and [Running the workers](./background-workers.md) to run them. + ### Database configuration Database credentials are set as individual environment variables (not a `DATABASE_URL` string): diff --git a/docs/getting-started/index.md b/docs/getting-started/index.md index 10c23b8ff..1102bf7b7 100644 --- a/docs/getting-started/index.md +++ b/docs/getting-started/index.md @@ -26,6 +26,12 @@ falls back on generated recipes without reporting anything, the `config/packages Thelia expects are never written, and the install fails much later on an unrelated message, most often `You must either configure a "public_key" or a "secret_key"`. +Another symptom of a missing token: Flex applies the Symfony recipe of `symfony/messenger` +instead of the Thelia one. It writes `config/packages/messenger.yaml` and adds +`MESSENGER_TRANSPORT_DSN=doctrine://default?auto_setup=0` to `.env`, so every mail waits in a +queue for a worker that does not run, and no mail leaves the shop. Delete that file and that +line, as described in [Updating from 3.2 to 3.3](../upgrading/from-3.2-to-3.3.md#symfony-recipes). + Create a token on [github.com/settings/tokens](https://github.com/settings/tokens). No scope is needed, the recipes are public. Give it to Composer once: diff --git a/docs/reference/background-jobs.md b/docs/reference/background-jobs.md new file mode 100644 index 000000000..8e29e12f6 --- /dev/null +++ b/docs/reference/background-jobs.md @@ -0,0 +1,1139 @@ +--- +title: Background jobs +sidebar_position: 5 +--- + +# Background jobs reference + +This page lists the settings and commands of the background jobs, then walks through a +complete module that queues its own work. The model behind it (transports, retries, the +classes a queue accepts) is explained in [Background jobs](../architecture/background-jobs.md). +Running the workers in production is covered in +[Running the workers](../getting-started/background-workers.md). + +## Configuration + +Every setting is an environment variable, set in `.env.local` or in the environment of the +hosting. + +| Variable | Default | Effect | +| --- | --- | --- | +| `MESSENGER_TRANSPORT_DSN` | empty | The queue of the `async` transport, from which the `async_heavy` queue is derived. Empty means no queue: every job runs at once. `doctrine://default`, a Redis DSN or an AMQP DSN names a queue that a worker consumes. | +| `MESSENGER_HEAVY_TRANSPORT_DSN` | derived | Overrides the queue of `async_heavy`. Set it on AMQP: otherwise the exports and imports share the queue of `async` and its three retries. | +| `MESSENGER_FAILURE_TRANSPORT_DSN` | `doctrine://default?queue_name=failed` | Where the jobs that failed every attempt are kept. | +| `THELIA_SCHEDULE_SALE_CHECK` | `* * * * *` | Cron expression of `sale:check-activation`. Empty disables the task. | +| `THELIA_SCHEDULE_MAINTENANCE_PURGE` | `30 3 * * *` | Cron expression of `maintenance:purge`. Empty disables the task. | +| `THELIA_SCHEDULE_FAILED_JOBS_PURGE` | `0 4 * * *` | Cron expression of `thelia:messenger:purge-failed`. Empty disables the task. | +| `THELIA_SCHEDULE_CURRENCY_RATES` | empty | Cron expression of `currency:update-rates`. Off by default. | +| `LOCK_DSN` | `flock` in the `.env` of the core, `semaphore` in the `.env.dist` of `thelia-project` | The lock store that keeps two workers from running the same scheduled task. | + +`doctrine://default` is the only Doctrine connection name accepted: it is the shop database, +reached with the settings of the Propel connection. The table `messenger_messages` is created +by the installer and by the 3.3.0 update script. + +A Redis or AMQP DSN needs its Messenger bridge, which the core does not ship: +`composer require symfony/redis-messenger` (with the `redis` PHP extension) or +`composer require symfony/amqp-messenger` (with the `amqp` extension). + +How `async_heavy` derives its queue from `MESSENGER_TRANSPORT_DSN` is shown in +[A second queue for heavy jobs](../architecture/background-jobs.md#a-second-queue-for-heavy-jobs). +The options of the Symfony transport go in the DSN, such as +`doctrine://default?redeliver_timeout=600`; the heavy queue derived from it keeps them. What +`redeliver_timeout` changes, and what it does not, is explained in +[When a worker stops](../getting-started/background-workers.md#when-a-worker-stops). + +The core configures Messenger for the whole application: its serializer, its bus, its +`failure_transport` and the routing of its messages are loaded before the configuration of the +project. The container parameter `thelia.messenger.allowed_message_classes` (an array, empty by +default) lists the exact names of the message classes that a project queues from outside +`Thelia\` and the namespaces of the active modules: an application that queues its own +`App\Message\…` classes gets a `LogicException` on dispatch until they are listed there. The +core declares it empty (`core/lib/Thelia/Config/Resources/parameters/messenger.php`). Any other +message class must also be taken by a handler, or have a parent or an interface a handler +takes: the parameter `thelia.messenger.handled_message_classes` is filled when the container is +built and is not meant to be set by hand. A handler that takes any message (`*` or `object`) +does not count. + +A project sets the list in `config/services.yaml`, which is loaded after the configuration of +the core: + +```yaml +# config/services.yaml +parameters: + thelia.messenger.allowed_message_classes: + - App\Message\SyncInvoices +``` + +### Transports + +| Transport | Retries | Notes | +| --- | --- | --- | +| `async` | 3, after 30 s, 2 min and 8 min (multiplier 4, with jitter) | Then the message goes to `failed`. | +| `async_heavy` | none (`max_retries: 0`) | Back-office exports and imports. A job that fails goes to `failed` at once. | +| `failed` | none | Replay with `messenger:failed:retry` or from the back office. | +| `scheduler_thelia` | none | The recurring tasks of the `thelia` schedule. | + +The core routes `Symfony\Component\Mailer\Messenger\SendEmailMessage` to `async`, and +`Thelia\Domain\DataTransfer\Job\RunExportJob` and `Thelia\Domain\DataTransfer\Job\RunImportJob` +to `async_heavy`. + +## Commands + +| Command | Use | +| --- | --- | +| `messenger:consume async async_heavy --time-limit=3600 --memory-limit=256M` | Run a worker. It takes the mails first, then the exports and imports. | +| `messenger:consume scheduler_thelia --time-limit=3600 --memory-limit=256M` | Run the recurring tasks. | +| `messenger:stats` | Count the messages waiting in each transport. | +| `messenger:failed:show` | List the failed jobs, or show one with its error. | +| `messenger:failed:retry` | Replay failed jobs. | +| `messenger:failed:remove` | Delete failed jobs. | +| `messenger:stop-workers` | Ask every worker to stop after its current message. | +| [`thelia:messenger:purge-failed`](./cli/thelia_messenger_purge_failed.md) | Delete the jobs in the failure transport for more than `--older-than` days (30 by default). `maintenance:purge` deletes those older than 30 days too. | + +All of them run through `php Thelia`, for example `php Thelia messenger:stats`. + +## Back office + +Configuration > System > Background jobs (`/admin/configuration/background-jobs`) shows: + +- whether a queue is configured, and how many jobs wait in it, +- how many jobs failed, and the list of failed jobs, the newest first, with their description, + the reason of the failure, the date and the number of attempts. The description is the + `describe()` of a message implementing `Thelia\Messenger\Message\DescribedJob`, the subject + and the recipients of a mail, or else the short name of the message class, +- the recurring tasks whose last run failed, under "Recurring tasks that failed", +- the last 20 exports and the last 20 imports of the administrator looking at it, of every + administrator for a super-administrator. + +The reason of a failed job is shown when it was written for the administrator (an exception +implementing `Thelia\Exception\UserFacingFailure`) or when it is the answer of the mail server, +with the credentials of any URL in it hidden (a password holding an `@` is hidden whole). +Either is cut to 2000 characters. Any other reason reads as a server error. + +The waiting count adds the jobs of `async` and of `async_heavy`, and counts them once when both +transports have the same DSN. When `failed` is in the shop database, the default, the list is +read in SQL, so the newest failures show however many there are. + +Each failed job can be replayed or deleted. Replay takes the job out of `failed` first, then +puts it back on the transport it failed on; if that fails, the job goes back to `failed`. A job +that implements `Thelia\Messenger\Message\ReplayableJob` is put back as `forReplay()` returns +it, so it starts afresh, and the same goes for a job replayed with `messenger:failed:retry`. A +double click cannot send it twice. Without a queue, a replayed job runs at once, and a job that fails +again stays in the failed list. A replay that fails shows "The job failed again. The details are +in the server log." and the reason goes to the log. If putting it back in `failed` fails too (queue and +database both down), the job is lost from the queues and a critical line of the log names it by +its id and its class, and the two failures by their class, code and place: its description and +its content may name a customer, and the log is kept far longer than the failed jobs. + +A job the workers could no longer read (module turned off, class removed, content that no longer +fits) is listed as `Unreadable job: ` with its original class. Replayed, it would fail +the same way, so it has no Replay button; delete it once you +know what it was. + +A recurring task of the `thelia` schedule that fails does not go to `failed`: it runs again +at its next time. Its last failure is kept in the application cache until a run of the task +goes through, or for a month after the date of that failure, and the screen lists it with the +task, the reason and the date. A command that exits with an error code gives +`Command "" exited with code "".` as its reason. + +The "Details" link of an export or import is only shown to the administrator who started the +job and to super-administrators. + +The screen answers to the resource `admin.configuration.background-jobs`, given to no existing +profile by the update: until it is granted, only super-administrators see the screen. Grant `VIEW` to read +it, `UPDATE` to replay and `DELETE` to delete. The description of a mail names its recipient +and a failure reason may quote personal data, so give it only to the profiles that need it. + +## Back-office exports and imports + +Exports and imports started from the back office are jobs. How they run, and why an import is +a single transaction, is explained in +[Background jobs](../architecture/background-jobs.md#back-office-exports-and-imports). This +section lists what they expose and what they accept. + +| | Export | Import | +| --- | --- | --- | +| Transport | `async_heavy` | `async_heavy` | +| Launcher | `ExportJobLauncher::launch(Export $export, string $serializerId, ?string $archiverId = null, ?Lang $language = null, bool $includeImages = false, bool $includeDocuments = false, ?array $rangeDate = null, ?int $adminId = null): ExportJob` | `ImportJobLauncher::launch(Import $import, File $file, string $originalName, ?Lang $language = null, ?int $adminId = null): ImportJob` | +| Message | `RunExportJob(int $exportJobId)` | `RunImportJob(int $importJobId)` | +| Row | `export_job` | `import_job`: status, file name, rows imported, refused rows with their reason, error | +| Without a queue | Runs in the request, the file is served at once | Runs in the request, the page shows the rows imported and the rows refused | +| With a queue | `/admin/export/job/{id}`: waiting, running with the rows written, done with the download, failed with the reason | `/admin/import/job/{id}`: waiting, running, done with the rows changed and refused, failed with the reason | +| Replay of a failed job | Starts the export over | Starts over from the first row | +| Files | Deleted by `maintenance:purge` once older than a day | Kept in `var/data-transfer/import//` until the import is done, kept while it may be replayed | +| `maintenance:purge` | Deletes the done jobs older than 7 days, any other after 31 days | Deletes the done jobs older than 7 days, any other after 31 days, with their files | +| Right to launch | `VIEW` on the export resource | `UPDATE` on the import resource; `VIEW` only shows the imports and their jobs | +| Job page access | The administrator who started it and super-administrators, download included | The administrator who started it and super-administrators | + +The launchers and the messages live in `Thelia\Domain\DataTransfer\Job\`. A job has one of the +statuses of the enum `Thelia\Domain\DataTransfer\Job\JobStatus`: `queued`, `running`, `done`, +`failed`. Both messages implement `Thelia\Messenger\Message\DescribedJob`, and are listed as +`Export #` and `Import #` among the failed jobs. + +### The job pages + +A job page refreshes itself every 3 seconds, at most 100 times, then stops and offers a link to +refresh it by hand; a "Stop refreshing" link stops it earlier. Other administrators than the one +who started the job get a 403, even with the export or import right, and the lists of recent +exports and imports only show their own jobs, except to super-administrators. + +An administrator can launch 10 exports and imports in 10 minutes, the two counted together +(rate limiter `admin_data_transfer_launch`, declared in the core `framework` configuration). A +launch is only counted once its form is valid: a form sent without a file, with a file the +server did not receive whole, with a file whose name or content the import refuses, or with an +invalid option is not counted. Past the limit, the launch is refused with a message asking to +wait a few minutes. + +Without a queue, an import that refused more than 10 rows leads to its job page, which lists +them all; with fewer, the form page shows them in its message. The description of an export or +an import, written in HTML by its module, keeps only `p`, `br`, `strong`, `b`, `em`, `i`, `u`, +`ul`, `ol`, `li`, `code`, `pre`, `blockquote`, `span` and `a` (with `href` and `title`, and only +`http`, `https` and `mailto` links). Anything else, scripts, styles, media, `id` and `name` +attributes included, is removed. + +### The reason of a failure + +The reason an administrator reads, on the job page and on the Background jobs screen, is the +message of an exception implementing `Thelia\Exception\UserFacingFailure`, found anywhere in the +chain of the failure, cut to 2000 characters. The core ones live in +`Thelia\Domain\DataTransfer\Exception`: `UploadRefusedException` (an uploaded file refused), +`DataTransferNoDataFoundException`, `HandlerUnavailableException`, `MissingColumnsException` and +`JobRefusedException`. `Thelia\Form\Exception\FormValidationException` itself does not implement +the interface. + +Any other failure reads "The job failed because of a server error. The details are in the server +log.", and the log line of the job names the exception by its class, code and place in the code. +The job page passes the reason through the translator, so the fixed messages are shown in the +language of the administrator. + +A module whose export or import throws an exception meant for the administrator implements +the interface on it: + +```php +final class WarehouseRefusedException extends \RuntimeException implements \Thelia\Exception\UserFacingFailure +{ +} +``` + +Some refusals of the core, all `JobRefusedException`: + +- an export whose year is not four digits or whose month is not 1 to 12: "The dates of the + export are not valid.", +- an export whose format is no longer available on the server: + `The format "" is no longer available on this server.`, +- an import row whose save leaves the transaction unable to commit, even when a module catches + the failure: `Row could not be saved: nothing was imported.` + +A job the queue refuses at dispatch is recorded as failed with "The job could not be queued. The +details are in the server log.", and the file of such an import is deleted. A job whose row was +deleted fails for good and stays in `failed`. A job deleted from the failed jobs before it ever +ran is marked failed with "Deleted from the failed jobs before it ran.", so its row no longer +reads as waiting for a worker. + +### What an import accepts + +The file is checked in the request, before anything is queued. Its extension must be one the +import handlers accept, and its content is read: an archive must match the type of its +archiver, anything else must be text. A file that does not parse in its format is refused with +a message asking to check its content. + +An archive is looked into before it is extracted, both when it is uploaded and when it is +imported (`Thelia\Domain\DataTransfer\ArchiveInspector`). It is refused when it holds more than +1000 entries, folders included, more than 512 MB once extracted, a name that is absolute or +contains `..`, or a link, zip and tar alike. What the extraction really wrote is measured against +the same 512 MB, and the extracted copy is removed once the import is over, whatever came of it. +`ImportHandler` takes the `ArchiveInspector` as a required constructor argument: code that builds +or extends it passes one. + +A row the import refuses (a missing combination, an invalid GTIN) is listed with the job as a +refusal, and the other rows are kept. A refusal that is not valid UTF-8 is stored with the +invalid bytes replaced. The refusals are kept up to 60,000 bytes; past that, the first ones are +kept and a last line says ` more rows were refused.` A stock or a price must be a number, +given as text or as a number, below 10^10 in absolute value: a list, an object, a missing price +or anything else refuses the row with its reason, and the price of a row is checked before +anything is written for it. + +The file is moved out of the upload directory to `var/data-transfer/import/`, not to the cache, +which a deployment empties. Its path is stored relative to the project, its name is cut to 100 +characters, and the import only reads a file that lies in that directory. + +### Files and retention + +- An import file is deleted once the import is done. Without a queue a failed import can never + be replayed, so its file is deleted at once; with a queue it is kept while the job may be + replayed. +- Deleting a failed import from the back office deletes its file too + (`Thelia\Messenger\Event\FailedJobRemovedEvent`), unless the import is still marked running. + `messenger:failed:remove` and `thelia:messenger:purge-failed` leave the file to + `maintenance:purge`. +- `maintenance:purge` deletes the `done` jobs created more than 7 days ago, and any other job + (failed, queued, running) after 31 days: a day past the 30 days a failed message is kept, so a + replayed job still finds its row. The file of an import goes with its row only when it lies + inside `var/data-transfer/import`. The purge also removes the files of that directory older + than 31 days, such as the extracted copy of an archive left by a killed worker, and its empty + directories once they are more than a day old. +- A file that cannot be deleted is logged as a warning and left to the purge; the job is still + reported as it ended. + +## Recurring tasks + +The `thelia` schedule runs these commands when a worker consumes `scheduler_thelia`: + +| Command | Variable | Default | +| --- | --- | --- | +| `sale:check-activation` | `THELIA_SCHEDULE_SALE_CHECK` | every minute | +| `maintenance:purge` | `THELIA_SCHEDULE_MAINTENANCE_PURGE` | 03:30 | +| `thelia:messenger:purge-failed` | `THELIA_SCHEDULE_FAILED_JOBS_PURGE` | 04:00 | +| `currency:update-rates` | `THELIA_SCHEDULE_CURRENCY_RATES` | off | + +`currency:update-rates` is off because it overwrites every rate, including those set by hand. +`sale:check-activation` switches a sale back on when it was turned off by hand inside its +dates, as the command always did. + +A shop that keeps these commands in its crontab must not also consume `scheduler_thelia`, or +each task runs twice. Choose one of the two. + +A module adds a task to the same schedule with a Symfony Scheduler attribute and +`schedule: 'thelia'`: + +- `#[AsCronTask('0 2 * * *', schedule: 'thelia')]` for a cron expression, +- `#[AsPeriodicTask('1 hour', schedule: 'thelia')]` for an interval. + +The attribute goes on a console command class, or on a service class with `method:` naming the +method to call. The example below schedules a command. + +## A complete module: StockSync + +`StockSync` pushes the stock of each product combination to a remote warehouse API. When a +combination is saved in the back office, a listener dispatches a message, and the handler +calls the API. A nightly task pushes every combination again, to catch changes made elsewhere. + +With an empty `MESSENGER_TRANSPORT_DSN`, the push happens during the save. With a queue, the +save returns at once and a worker does the push, with retries if the API is down. + +Without a queue, an exception thrown by the handler reaches the code that dispatched the +message: the save of the combination in the back office fails with it. A shop that relies on a +remote API this way should run a queue. + +The first line of each block below names the file it goes in; it is not part of the file. + +``` +local/modules/StockSync/ +├── StockSync.php +├── composer.json +├── Config/ +│ ├── config.xml +│ ├── module.xml +│ ├── schema.xml +│ └── TheliaMain.sql # generated by module:generate:sql +├── Command/ +│ └── PushAllStockCommand.php +├── EventListener/ +│ └── StockChangedListener.php +├── Message/ +│ └── PushStock.php +├── MessageHandler/ +│ └── PushStockHandler.php +├── Model/ # generated by module:generate:model +└── Tests/ + ├── PushStockHandlerTest.php + └── StockChangedListenerTest.php +``` + +### The module class + +`configureServices()` registers every class of the module, which makes the handler, the +listener and the command discovered. `configureContainer()` routes the message to `async`, in the +same way the core routes its own messages. `postActivation()` creates the tables. + +```php +// local/modules/StockSync/StockSync.php +load(self::getModuleCode().'\\', __DIR__) + ->exclude([ + __DIR__.'/I18n/*', + __DIR__.'/Model/*', + __DIR__.'/Message/*', + __DIR__.'/Tests/*', + __DIR__.'/StockSync.php', + ]) + ->autowire() + ->autoconfigure(); + } + + public static function configureContainer(ContainerConfigurator $containerConfigurator): void + { + $containerConfigurator->extension('framework', [ + 'messenger' => [ + 'routing' => [ + PushStock::class => 'async', + ], + ], + ], prepend: true); + } + + public function postActivation(?ConnectionInterface $con = null): void + { + if (!self::getConfigValue('is_initialized', false)) { + (new Database($con))->insertSql(null, [__DIR__.'/Config/TheliaMain.sql']); + + self::setConfigValue('is_initialized', true); + } + } +} +``` + +`Message/` is excluded from the service scan: a message is data, not a service. `Tests/` is +excluded too: a test class is not a service, and PHPUnit is not installed in production. The message +class may also carry the route itself, with `#[AsMessage('async')]`. A message routed nowhere runs +in the request that dispatched it. + +A shop can also route the message itself, in `config/packages/messenger.yaml`, which takes +precedence over the prepended module configuration: + +```yaml +# config/packages/messenger.yaml +framework: + messenger: + routing: + 'StockSync\Message\PushStock': async +``` + +### A queue of its own + +Routed to `async`, the pushes wait behind the mails of the shop. A module whose work is +plentiful can give it a transport of its own, still synchronous by default: + +```php +public static function configureContainer(ContainerConfigurator $containerConfigurator): void +{ + $containerConfigurator->parameters()->set('stock_sync.inline_transport_dsn', 'sync://'); + $containerConfigurator->extension('framework', [ + 'messenger' => [ + 'transports' => [ + 'stock_sync' => '%env(default:stock_sync.inline_transport_dsn:STOCK_SYNC_TRANSPORT_DSN)%', + ], + 'routing' => [ + PushStock::class => 'stock_sync', + ], + ], + ], prepend: true); +} +``` + +With `STOCK_SYNC_TRANSPORT_DSN` empty or unset, the pushes stay synchronous. A shop that wants +them queued sets, for instance, `STOCK_SYNC_TRANSPORT_DSN=doctrine://default?queue_name=stock_sync` +and runs a worker of its own with `php Thelia messenger:consume stock_sync`: the workers of the +shop do not consume it. Pushes that fail every +attempt still go to the `failed` transport of the shop, and the back office lists them there, +but it does not count the pushes waiting in `stock_sync`: watch that queue with +`messenger:stats stock_sync`. + +### Metadata and dependencies + +```xml + + + + StockSync\StockSync + + Stock sync + Pushes the stock of each combination to a warehouse API + + + en_US + + 1.0.0 + + Your Name + you@example.com + + classic + 3.3.0 + prod + +``` + +```json +{ + "name": "your-vendor/stock-sync-module", + "type": "thelia-module", + "license": "MIT", + "require": { + "thelia/installer": "~1.1", + "symfony/http-client": "^7.4" + }, + "extra": { + "installer-name": "StockSync" + }, + "autoload": { + "psr-4": { + "StockSync\\": "" + } + } +} +``` + +`Config/config.xml` declares nothing, since `configureServices()` registers the services, but +the `module:generate:*` commands refuse a module without it: + +```xml + + + + +``` + +### The tables + +`stock_sync_state` remembers the last quantity pushed for each combination; it is what makes +the handler idempotent. `stock_sync_log` keeps one line per push. The foreign key points to a +core table, so the core schema is declared as an external schema, for reference only: without +it, `module:generate:sql` cannot resolve `product_sale_elements`. + +```xml + + + + + + + + + + + + + + + +
+ + + + + + +
+
+``` + +Generate the models and the SQL file: + +```bash +php Thelia module:generate:model StockSync +php Thelia module:generate:sql StockSync +``` + +### The message + +The message carries an id, nothing else. It is a class of the module namespace, so the queue +accepts it, and its only property is a promoted scalar, so the Symfony serializer writes and +reads it as JSON. + +```php +// local/modules/StockSync/Message/PushStock.php +productSaleElementsId); + } +} +``` + +Never put a Propel model in a message. It cannot be serialized as JSON, and by the time the +worker reads the message the row may have changed or been deleted. + +`Thelia\Messenger\Message\DescribedJob` gives the job the line the Background jobs screen shows +for it among the failed jobs. A message that does not implement it is listed by the short name +of its class (`PushStock`), which tells the administrator nothing about which combination +failed. `describe()` runs when the screen is displayed, on the message read back from the +queue. + +The description is shown to every administrator who may read the screen, and it stays as long +as the failed job does. Never put personal data in it: no name, e-mail address or postal +address of a customer, no order content. Give ids and references, which the administrator can +look up with their own rights. + +`Thelia\Messenger\Message\ReplayableJob` is for a message that carries a state a replay must +reset, such as the number of times it was postponed: its `forReplay()` returns the message to +put back on the queue when the job is replayed, from the back office or with +`messenger:failed:retry`. `PushStock` carries only an id, so it has nothing to reset and does +not implement it. + +### The handler + +The handler re-reads the combination, compares it with what was pushed last time, calls the +API with the absolute quantity, then records the push. It assumes nothing about a request: the +shop URL comes from `DEFAULT_URI`, and the locale of the product title is the default language +of the shop, chosen explicitly. + +```php +// local/modules/StockSync/MessageHandler/PushStockHandler.php +findPk($message->productSaleElementsId); + + // Deleted since the message was dispatched: nothing left to push. + if (null === $productSaleElements) { + return; + } + + $quantity = (float) $productSaleElements->getQuantity(); + $state = StockSyncStateQuery::create()->findPk($productSaleElements->getId()); + + // Already pushed with this quantity: a replay, or a second message for the same change. + if (null !== $state && $state->getPushedQuantity() === $quantity) { + return; + } + + $apiUrl = (string) StockSync::getConfigValue('api_url'); + + if ('' === $apiUrl) { + throw new UnrecoverableMessageHandlingException('StockSync has no api_url configured.'); + } + + $locale = Lang::getDefaultLanguage()->getLocale(); + $product = $productSaleElements->getProduct(); + $product->setLocale($locale); + + $response = $this->httpClient->request('PUT', rtrim($apiUrl, '/').'/stock/'.rawurlencode((string) $productSaleElements->getRef()), [ + 'auth_bearer' => (string) StockSync::getConfigValue('api_token'), + 'json' => [ + 'quantity' => $quantity, + 'title' => $product->getTitle(), + 'locale' => $locale, + 'source' => rtrim($this->shopUrl, '/'), + ], + ]); + + $status = $response->getStatusCode(); + + // The API refused the data: retrying would get the same answer. + if ($status >= 400 && $status < 500 && 429 !== $status) { + throw new UnrecoverableMessageHandlingException(\sprintf('The warehouse refused %s with HTTP %d.', $productSaleElements->getRef(), $status)); + } + + // A 5xx or a 429: throw, so Messenger retries later. + if ($status >= 300) { + throw new \RuntimeException(\sprintf('The warehouse answered HTTP %d for %s.', $status, $productSaleElements->getRef())); + } + + $this->recordPush($productSaleElements->getId(), $quantity, $state); + } + + private function recordPush(int $productSaleElementsId, float $quantity, ?StockSyncState $state): void + { + $connection = Propel::getConnection('TheliaMain'); + $connection->beginTransaction(); + + try { + $state ??= (new StockSyncState())->setProductSaleElementsId($productSaleElementsId); + $state + ->setPushedQuantity($quantity) + ->setPushedAt(new \DateTimeImmutable()) + ->save($connection); + + (new StockSyncLog()) + ->setProductSaleElementsId($productSaleElementsId) + ->setQuantity($quantity) + ->setCreatedAt(new \DateTimeImmutable()) + ->save($connection); + + $connection->commit(); + } catch (\Throwable $exception) { + $connection->rollBack(); + + throw $exception; + } + } +} +``` + +`StockSync::getConfigValue()` is read again for every message, because the worker forgets the +module configuration between messages: a token changed in the back office is used by the next +push without restarting the worker. + +The two writes of `recordPush()` share an explicit transaction on the Propel connection, so +the state and the log never disagree. + +### The listener + +The core saves a combination in its `PRODUCT_UPDATE_PRODUCT_SALE_ELEMENT` listener, at +priority 128, and commits its transaction there. The module listens to the same event at a +lower priority, so it runs after the commit, and dispatches the message. + +```php +// local/modules/StockSync/EventListener/StockChangedListener.php + ['onStockChanged', 0], + ]; + } + + public function onStockChanged(ProductSaleElementUpdateEvent $event): void + { + $this->bus->dispatch(new PushStock($event->getProductSaleElementId())); + } +} +``` + +The queue in the shop database uses its own connection, so a message dispatched before the +commit stays queued even if the transaction is rolled back. Dispatching after the action has +committed avoids a push for a change that never happened. If your own code wraps the action +in a larger transaction, dispatch after that outer commit instead. + +Stock also changes outside this event, when an order is placed for instance. The nightly task +below covers those changes; a module that needs them sooner listens to the events concerned +in the same way. + +### The recurring task + +`#[AsCronTask]` adds the command to the `thelia` schedule. It runs at 02:00 when a worker +consumes `scheduler_thelia`, and can be run by hand like any command. + +```php +// local/modules/StockSync/Command/PushAllStockCommand.php +select('Id')->find(); + + foreach ($ids as $id) { + $this->bus->dispatch(new PushStock((int) $id)); + } + + $output->writeln(\sprintf('%d stock push(es) queued.', \count($ids))); + + return self::SUCCESS; + } +} +``` + +The command only dispatches messages. With a queue, the pushes are spread over the `async` +worker; without one, they run during the command. Combinations whose quantity did not change +are skipped by the handler, so the nightly run only calls the API for real changes. + +### Trying it + +With the files above in `local/modules/StockSync/`, a local stand-in for the warehouse API is +enough to watch a push go through the queue. + +1. Install and activate the module. Activation creates its two tables: + + ```bash + composer dump-autoload + php Thelia module:refresh + php Thelia module:activate StockSync + ``` + +2. Start a fake warehouse that answers `200` and keeps what it receives: + + ```bash + mkdir -p /tmp/warehouse + echo ' /tmp/warehouse/index.php + php -S 127.0.0.1:8099 -t /tmp/warehouse /tmp/warehouse/index.php + ``` + +3. Give the module its two settings. The example has no configuration page; a module setting + is stored in `module_config` and its value in `module_config_i18n`, under `en_US` when no + locale is given: + + ```sql + INSERT INTO module_config (module_id, name, created_at, updated_at) + SELECT m.id, s.name, NOW(), NOW() + FROM module m, (SELECT 'api_url' AS name UNION ALL SELECT 'api_token') s + WHERE m.code = 'StockSync' + AND NOT EXISTS (SELECT 1 FROM module_config c WHERE c.module_id = m.id AND c.name = s.name); + INSERT INTO module_config_i18n (id, locale, value) + SELECT c.id, 'en_US', IF(c.name = 'api_url', 'http://127.0.0.1:8099', 'test-token') + FROM module_config c JOIN module m ON m.id = c.module_id + WHERE m.code = 'StockSync' AND c.name IN ('api_url', 'api_token') + ON DUPLICATE KEY UPDATE value = VALUES(value); + ``` + + Run again, it changes the two values without adding rows. + +4. With `MESSENGER_TRANSPORT_DSN=doctrine://default` in `.env.local` and no worker running, + change the stock of a combination in the back office. The page answers at once and the + messages wait: "Save all combinations" saves every row of the product, so one message per + combination is queued, and the handler skips those whose quantity did not change: + + ```bash + php Thelia messenger:stats async + ``` + +5. Run a worker until the queue is empty, then read what the warehouse received and what the + module recorded: + + ```bash + php Thelia messenger:consume async --time-limit=10 -vv + cat /tmp/warehouse/received.log + ``` + + ```sql + SELECT * FROM stock_sync_log ORDER BY id DESC LIMIT 1; + ``` + +Without a queue, the push happens during the save of the stock, in the same request. Stop the +fake warehouse and change the stock again to see a retry: the worker logs the failure, waits 30 +seconds, and sends the message to `failed` after three more attempts. + +### Turning the module off + +A queue accepts the messages of active modules only. Messages of `StockSync` still waiting when +the module is turned off can no longer be read: the worker reads each of them as +`Thelia\Messenger\Message\UndecodableJob` and sends it to `failed`. Nothing is lost and +nothing loops. The Background jobs screen lists them as `Unreadable job: ` with the +original class `StockSync\Message\PushStock`. Delete them there, and let the nightly task push +the stock again once the module is back. + +To avoid this, wait for `php Thelia messenger:stats async` to reach zero before turning the +module off. The same happens when a message class is renamed or removed, or when its +constructor changes so that queued content no longer fits: keep the old class until the queue +is empty. + +## Idempotence and replay + +A handler can run more than once for the same message: + +- a retry after a failure that happened halfway, for instance when the API accepted the + push but the database write that follows failed, +- a replay from the failed jobs, by `messenger:failed:retry` or from the back office, +- a worker killed hard (`SIGKILL`, out of memory) while handling the message: the Doctrine + transport redelivers it after `redeliver_timeout`, one hour by default, +- two messages for the same row, dispatched by two saves in a row. + +Write every handler so that running it twice has the effect of running it once: + +- carry ids in the message, and read the current state of the row in the handler, +- check whether the work is already done before doing it (`stock_sync_state` above), +- send absolute values to remote services (a quantity, a status), never deltas such as + "remove 2", which a replay would apply twice, +- record what was done in the same transaction as the other writes of the handler, +- return without error when the row no longer exists or is in a state that makes the job + pointless. Throwing would only send the message to `failed` for nothing. + +Use `UnrecoverableMessageHandlingException` when a retry cannot succeed (data refused, +configuration missing). The message goes to `failed` at once, where it can be replayed once +the cause is fixed. + +Test the replay: run the handler twice in a test and check that the effect happened once, as +in [Testing a message, its handler and its dispatch](#testing-a-message-its-handler-and-its-dispatch). + +### Known limits + +Two cases are not guarded by the core in this version: + +- A failed job replayed from the back office and with `messenger:failed:retry` at the same + moment may be sent twice: the command reads the job without holding it while it asks what to + do with it. Replay a job from one place at a time. +- A mail sent with the Symfony mailer directly, instead of `MailerFactory`, carries no + identifier: when the queue hands its job out again (a worker stopped between the mail server + accepting the mail and the job being acknowledged), it is sent a second time. Send the mails + of a module through `MailerFactory`, which a worker never delivers twice. + +Both are a reason more to keep every handler idempotent. + +## Testing a message, its handler and its dispatch + +Three things are worth a test in a module that queues work: that the right message is +dispatched, that the handler does its work, and that running the handler twice has the effect +of running it once. A fourth test checks that the queue of the shop accepts the message, which +otherwise only shows when a real queue refuses it. + +Test them apart. The tests of the core run without a queue: `tests/bootstrap.php` empties +`MESSENGER_TRANSPORT_DSN` and `MESSENGER_HEAVY_TRANSPORT_DSN`, so `async` is synchronous and a +message dispatched through the bus of the shop is handled at once, by the real handler. A module +test that goes through that bus cannot tell "dispatched" from "handled", and its handler calls +the real API. Give the listener a bus of its own instead, and call the handler directly. + +### The dispatch, without a database + +The listener receives a `MessageBus` that sends `PushStock` to an `InMemoryTransport` and has no +handler. `getSent()` returns what was queued; nothing ran. + +```php +// local/modules/StockSync/Tests/StockChangedListenerTest.php + ['stock_sync']], + new ServiceLocator(['stock_sync' => static fn (): InMemoryTransport => $queue]), + )), + ]); + + (new StockChangedListener($bus))->onStockChanged(new ProductSaleElementUpdateEvent(new Product(), 42)); + + $sent = $queue->getSent(); + self::assertCount(1, $sent); + self::assertEquals(new PushStock(42), $sent[0]->getMessage()); + } +} +``` + +### The handler and its replay, with the test database + +The handler reads the database, so its test extends `Thelia\Test\IntegrationTestCase`, which +runs each test in a transaction rolled back at the end (see [Testing](../testing/index.md)). The +handler is built by hand with a `MockHttpClient`, which answers in place of the warehouse and +counts the requests. Calling it twice with the same message is the replay: the second call must +not reach the warehouse, and only one push is logged. + +```php +// local/modules/StockSync/Tests/PushStockHandlerTest.php +createFixtureFactory(); + $product = $factory->product($factory->category(), $factory->taxRule(), $factory->currency()); + + $this->combination = $product->getDefaultSaleElements(); + $this->combination->setQuantity(7)->save($this->getPropelConnection()); + + StockSync::setConfigValue('api_url', 'https://warehouse.test'); + StockSync::setConfigValue('api_token', 'test-token'); + } + + public function testThePushSendsTheQuantityOfTheCombination(): void + { + $warehouse = new MockHttpClient(new MockResponse('', ['http_code' => 200])); + + $this->handler($warehouse)(new PushStock($this->combination->getId())); + + self::assertSame(1, $warehouse->getRequestsCount()); + self::assertSame(1, $this->pushesLogged()); + } + + public function testAReplayCallsTheWarehouseOnce(): void + { + $warehouse = new MockHttpClient([ + new MockResponse('', ['http_code' => 200]), + new MockResponse('', ['http_code' => 200]), + ]); + $handler = $this->handler($warehouse); + + $handler(new PushStock($this->combination->getId())); + $handler(new PushStock($this->combination->getId())); + + self::assertSame(1, $warehouse->getRequestsCount()); + self::assertSame(1, $this->pushesLogged()); + } + + public function testARefusalIsNotRetried(): void + { + $warehouse = new MockHttpClient(new MockResponse('', ['http_code' => 422])); + + $this->expectException(UnrecoverableMessageHandlingException::class); + + $this->handler($warehouse)(new PushStock($this->combination->getId())); + } + + public function testTheQueueOfTheShopAcceptsTheMessage(): void + { + $serializer = $this->getService(AllowedClassesSerializer::class); + + $decoded = $serializer->decode($serializer->encode(new Envelope(new PushStock(42)))); + + self::assertEquals(new PushStock(42), $decoded->getMessage()); + } + + private function handler(MockHttpClient $warehouse): PushStockHandler + { + return new PushStockHandler($warehouse, 'https://shop.test'); + } + + private function pushesLogged(): int + { + return StockSyncLogQuery::create() + ->filterByProductSaleElementsId($this->combination->getId()) + ->count(); + } +} +``` + +`testTheQueueOfTheShopAcceptsTheMessage()` writes and reads the message with the serializer of +the queues, `Thelia\Messenger\Serializer\AllowedClassesSerializer`. It fails with +`The message class "StockSync\Message\PushStock" is not one the shop queues` when the message +has no handler, lives outside the namespace of the module, or the module is not active in the +test database. + +### Running the tests + +In a checkout of `thelia/thelia`, the module in `local/modules/StockSync` is registered and +activated in the test database by `test:prepare`, which also runs its `postActivation()` and so +creates its tables. Prepare the test database once the module is there, then run its tests with +the same wrapper as the core suites: + +```bash +ddev exec composer test:prepare +ddev exec composer phpc ./vendor/bin/phpunit local/modules/StockSync/Tests +``` + +`phpc` sets `APP_ENV=test` and unsets the `DATABASE_*` and `MESSENGER_*` variables DDEV injects, +so the tests use the `test` database and no queue (see [Testing](../testing/index.md#run-the-tests)). + +A project created with `thelia/thelia-project` ships `bin/test-prepare`, but neither PHPUnit nor +a PHPUnit configuration. To run the tests of a module there, require `phpunit/phpunit` as a dev +dependency and give the project a `phpunit.xml.dist` and a `tests/bootstrap.php` like those of +`thelia/thelia`. + +Check that a test can fail before trusting it: remove the "already pushed" check of the handler +and `testAReplayCallsTheWarehouseOnce()` fails; remove the `dispatch()` of the listener and +`testSavingACombinationQueuesOnePushForIt()` fails. diff --git a/docs/reference/cli/index.md b/docs/reference/cli/index.md index 1761c0d7b..5930bbb31 100644 --- a/docs/reference/cli/index.md +++ b/docs/reference/cli/index.md @@ -105,6 +105,21 @@ with an existing invoice numbering to continue. `maintenance:purge`, `customer:anonymize` and `customer:export-personal-data` still declare their name with `setName()` in `configure()` rather than the `#[AsCommand]` attribute. Both styles work; new commands should use the attribute. ::: +### Background jobs + +| Command | Description | +| --- | --- | +| `messenger:consume` | Run a worker on one or more transports (`async`, `async_heavy`, `scheduler_thelia`). | +| `messenger:stats` | Count the messages waiting in each transport. | +| `messenger:failed:show` | List the failed jobs, or show one with its error. | +| `messenger:failed:retry` | Replay failed jobs. | +| `messenger:failed:remove` | Delete failed jobs. | +| `messenger:stop-workers` | Ask the workers to stop after their current message. | +| [`thelia:messenger:purge-failed`](./thelia_messenger_purge_failed.md) | Delete the failed jobs set aside for more than a number of days. | + +The `messenger:*` commands come from Symfony Messenger. See [Background jobs](../background-jobs.md) +and [Running the workers](../../getting-started/background-workers.md). + ### Customers and personal data | Command | Description | diff --git a/docs/reference/cli/maintenance_purge.md b/docs/reference/cli/maintenance_purge.md index 2493710eb..2193704c3 100644 --- a/docs/reference/cli/maintenance_purge.md +++ b/docs/reference/cli/maintenance_purge.md @@ -38,6 +38,17 @@ At the end of its run the command dispatches `TheliaEvents::MAINTENANCE_PURGE`: listens to it, purges its own tables, and appends a line to the report with `$event->addResult()`. +The core listens to the same event to purge the back-office exports and imports, through +`Thelia\Domain\DataTransfer\Service\DataTransferJobPurger`: the done jobs older than 7 days, +any other (failed, queued, running) after 31 days, a day longer than the failed jobs they can be +replayed from, the uploaded file of each deleted import when it lies inside +`var/data-transfer/import`, and the files of that directory older than 31 days. The same listener removes the export files of the cache older +than a day; with `--dry-run`, it counts them and deletes none. These periods are fixed. + +The failed jobs set aside in `failed` for more than 30 days are deleted on the same run +(`Thelia\Messenger\EventListener\FailedJobsMaintenancePurgeListener`), as +[`thelia:messenger:purge-failed`](./thelia_messenger_purge_failed.md) does with its default. See [Background jobs](../background-jobs.md#back-office-exports-and-imports). + ## Examples See what would be removed: ```shell diff --git a/docs/reference/cli/thelia_messenger_purge_failed.md b/docs/reference/cli/thelia_messenger_purge_failed.md new file mode 100644 index 000000000..f1bba6c93 --- /dev/null +++ b/docs/reference/cli/thelia_messenger_purge_failed.md @@ -0,0 +1,48 @@ +--- +title: thelia:messenger:purge-failed +--- + +## Description +Delete the background jobs kept in the failure transport for longer than a number of days. + +A job that failed every attempt is kept in the `failed` transport, with everything it was +dispatched with, until someone replays or removes it (`messenger:failed:show`, +`messenger:failed:retry`, `messenger:failed:remove`). A failed order mail holds the address of +the customer and the content of the order, so these jobs must not be kept forever. This command +deletes the old ones. + +## Usage +```shell + thelia:messenger:purge-failed [options] +``` + +## Options + - `--older-than=DAYS` Age in days from which a failed job is deleted. Defaults to `30`. Takes a whole number of days, from 1 to 3650; any other value ends with the `INVALID` exit code. + - `--dry-run` Count the jobs that would be deleted, and delete nothing. + +The age of a job is counted from the date it was sent to the failure transport. When `failed` is in the shop +database, the default (`doctrine://default?queue_name=failed`), that is the `created_at` of its +row, and the old jobs are counted or deleted in one SQL statement however many there are. With +any other failure transport, every job is read and dated by its last `RedeliveryStamp`; a job +that carries none is kept. The default of `--older-than` is +`Thelia\Messenger\FailedMessagePurger::RETENTION_DAYS`. + +The `thelia` schedule runs the command every day at 04:00 when a worker consumes +`scheduler_thelia` (`THELIA_SCHEDULE_FAILED_JOBS_PURGE`, default `0 4 * * *`). A shop that runs +its tasks from a crontab adds it there. + +## Examples +Count the failed jobs older than 30 days: +```shell +php Thelia thelia:messenger:purge-failed --dry-run +``` + +Delete the failed jobs kept for more than 14 days: +```shell +php Thelia thelia:messenger:purge-failed --older-than=14 +``` + +:::tip +Record the retention period in the GDPR register. See +[Background jobs](../background-jobs.md) and [Personal data](../../security/personal-data.md). +::: diff --git a/docs/reference/index.md b/docs/reference/index.md index df210f318..1319d660a 100644 --- a/docs/reference/index.md +++ b/docs/reference/index.md @@ -33,6 +33,12 @@ Transactional emails and order documents (invoice, delivery slip) are Twig theme See [Emails and PDF](./emails-and-pdf.md) for the theme reference and the preview commands. +### Background jobs + +Mail delivery, back-office exports and imports, and the work of modules can run in a queue consumed by a worker, and recurring tasks run from a schedule. + +See [Background jobs](./background-jobs.md) for the settings, the commands and a complete module example. + ## CLI commands Thelia provides console commands for common operations such as cache management, module handling, and database operations. @@ -47,6 +53,7 @@ See the [CLI reference](./cli/index.md) for all available commands. | [Forms](./forms.md) | Form creation and validation | | [Internationalization](./internationalization.md) | Multi-language support | | [Emails and PDF](./emails-and-pdf.md) | Email and PDF Twig themes | +| [Background jobs](./background-jobs.md) | Queues, workers and recurring tasks | | [CLI Commands](./cli/index.md) | Console commands | ## Legacy references diff --git a/docs/security/personal-data.md b/docs/security/personal-data.md index 3d2d38b83..fb42a9b38 100644 --- a/docs/security/personal-data.md +++ b/docs/security/personal-data.md @@ -156,6 +156,64 @@ purge cannot hand a blocked IP address a fresh set of attempts. See [`maintenance:purge`](../reference/cli/maintenance_purge.md). +### Background jobs + +Exports and imports started from the back office, and jobs that failed in the queue, hold +personal data too: an import file can list customers, and a failed order mail keeps the address +of the customer and the content of the order. Their retention periods are fixed in the code. + +| Data | Purged by | Kept | +| --- | --- | --- | +| Export and import jobs that are done | `maintenance:purge` | 7 days | +| Other export and import jobs (failed, queued, running) | `maintenance:purge` | 31 days, a day longer than the failed jobs they can be replayed from | +| Uploaded import files | The import itself, then `maintenance:purge` | Deleted once the import is done, or once it failed when there is no queue to replay it from, or once its failed job is deleted from the back office (not by `messenger:failed:remove` nor `thelia:messenger:purge-failed`, which leave it to `maintenance:purge`, 31 days at most); a file kept for a replay goes with its job, only when it lies under `var/data-transfer/import` | +| Files left in `var/data-transfer/import` | `maintenance:purge` | 31 days | +| Failed jobs of the queue | `thelia:messenger:purge-failed`, and `maintenance:purge` | 30 days; `--older-than` changes it for `thelia:messenger:purge-failed` | + +The 31-day sweep of `var/data-transfer/import` removes what no job row points to any more, such +as the extracted copy of an archive left by a worker that was killed, or a file whose row was +deleted by hand. The `thelia` schedule runs `thelia:messenger:purge-failed` every day at 04:00; +a shop that runs its tasks from a crontab adds it there. Record both retentions in the GDPR +register. + +The reason an administrator reads of a failed export or import, on its job page and on the +Background jobs screen, does not quote the data: unless the exception was written for the +administrator, it reads as a server error, and the error line Thelia writes to the log names it +by its class, code and place in the code rather than by its message. The failed job kept in +`failed` stores only that text: the original exception is not chained to it, since Symfony +keeps the whole chain of an exception it sends to the failure transport. This holds for the exports and imports of +the back office; a failed mail or module job keeps the exception its handler threw, but the +Background jobs screen only shows its reason when it was written for the administrator or is +the answer of the mail server, credentials hidden, and the log of the workers names its +exception by class, code and place (`Thelia\Messenger\Log\FailedJobLogProcessor`). + +A queued mail stores the content of its attachments in the queue, an attachment given by its +path included: the queue, and `failed` for a mail that could not be delivered, hold the +documents a mail carries, such as an invoice. See +[`thelia:messenger:purge-failed`](../reference/cli/thelia_messenger_purge_failed.md). + +The description of a failed job on the Background jobs screen names the recipient of a mail. +Give the screen (`admin.configuration.background-jobs`) only to the profiles that need it. A +module describes its own jobs with ids and references, never personal data, see +[the message of the example module](../reference/background-jobs.md#the-message). + +#### The queue server + +Treat the queue like the database: + +- A Redis or AMQP server must not be reachable from the internet. Bind it to the private + network of the hosting and protect it with a password. The workers read its queue as the + queue of the shop: whoever can write to it can send mails through the mail server of the shop + and replay the exports and imports it names. This is an accepted risk, and the reason such a + server must stay private. A queue in the shop database adds nothing to what access to that + database already gives. +- Put the DSN and its credentials in `.env.local` or in the secrets of the hosting, never in a + committed file. +- Do not share one Redis between environments unless each one has its own stream: a staging + worker would otherwise send the mails of production. Give each environment its own stream + name in the DSN (`redis://host:6379/messages_staging`, whose heavy jobs then go to the stream + `messages_staging_heavy`) or its own Redis database. + ## Adding a module purge to the same run `maintenance:purge` dispatches `TheliaEvents::MAINTENANCE_PURGE` at the end of its run. A @@ -188,4 +246,5 @@ To be explicit, so nobody promises it: - [Security policy](./security-policy.md) - [`customer:anonymize`](../reference/cli/customer_anonymize.md), [`customer:export-personal-data`](../reference/cli/customer_export_personal_data.md), - [`maintenance:purge`](../reference/cli/maintenance_purge.md) + [`maintenance:purge`](../reference/cli/maintenance_purge.md), + [`thelia:messenger:purge-failed`](../reference/cli/thelia_messenger_purge_failed.md) diff --git a/docs/testing/index.md b/docs/testing/index.md index 3cd80f454..dd36faef7 100644 --- a/docs/testing/index.md +++ b/docs/testing/index.md @@ -277,6 +277,8 @@ It performs the following: It does not set templates, create an admin, or import demo data. It is the minimal setup a test run needs. +Tests always run without a queue: `tests/bootstrap.php` empties `MESSENGER_TRANSPORT_DSN` and `MESSENGER_HEAVY_TRANSPORT_DSN` after the `.env` files are read, so every background job runs at once, whatever queue the machine is configured with. A message dispatched through the bus of the shop in a test is therefore handled at once, by its real handler. To test a module that queues work (the dispatch with an in-memory transport, the handler and its replay with the test database), see [Testing a message, its handler and its dispatch](../reference/background-jobs.md#testing-a-message-its-handler-and-its-dispatch). + :::caution Stale Propel cache If tests suddenly hit the wrong database after switching branches or environments, delete `var/propel/test/` and re-run `composer test:prepare`. Step 5 above does this automatically, but a manual run is the quickest fix when a cache survives. ::: diff --git a/docs/upgrading/breaking-changes.md b/docs/upgrading/breaking-changes.md index 6ac0da8ef..1e8b46179 100644 --- a/docs/upgrading/breaking-changes.md +++ b/docs/upgrading/breaking-changes.md @@ -9,6 +9,48 @@ This page lists, release by release, what a module or a theme has to change when For the update procedure itself, see [Update](./update.md). +## Thelia 3.3 + +The update gestures (the Symfony Messenger recipe, the shared directories, the queue of the heavy jobs) are in [Updating from 3.2 to 3.3](./from-3.2-to-3.3.md). See [Background jobs](../architecture/background-jobs.md) for the model. + +### Doctrine DBAL 4 + +The core requires `doctrine/dbal ^4.2`, for the queue in the shop database (DBAL alone, no ORM). A project or a module held on DBAL 3 must move to DBAL 4 before it updates. + +### Back-office theme + +The `default-twig` back-office theme requires `thelia/core ^3.3` in its `composer.json`. Update the core and the theme together. + +### Export and import handlers + +`ExportHandler::export()`, `ImportHandler::import()` and `ImportHandler::validateUpload()` take a new optional last argument. A module that overrides one of them has to declare it. + +| Method | New last argument | +|--------|-------------------| +| `Thelia\Domain\DataTransfer\ExportHandler::export()` | `?\Closure $onProgress = null`, told the rows written | +| `Thelia\Domain\DataTransfer\ImportHandler::import()` | `?\Closure $onProgress = null`, told the rows read | +| `Thelia\Domain\DataTransfer\ImportHandler::validateUpload()` | `?File $file = null`, the uploaded file whose content is checked | + +`processExport()` and `processImport()` keep their signature. `ImportHandler::import()` now extracts an archive through a private method, so an override of `extractArchive()` is no longer called by it. + +`ImportHandler` takes a fourth constructor argument, `Thelia\Domain\DataTransfer\ArchiveInspector`: a module that builds or extends it passes it. `AbstractArchiver` implements the new `Thelia\Core\Archiver\ClosableArchiverInterface` (`close(): bool`, `discard(): void`): an archiver of a module that declares either method with another signature must align it. + +### Mails + +- The order history line `email_sent` is written once the mail server has taken the mail, by `Thelia\Mailer\EventListener\OrderEmailHistoryListener` on the Symfony Mailer `SentMessageEvent`, and no longer by `MailerFactory` right after handing the mail over. Without a queue this happens in the same request; with one, it is the worker that writes it, later, and the author of the line is `system`. +- `MailerFactory` names the order in two headers of the mail, `X-Thelia-Order-Id` and `X-Thelia-Message-Code`, and gives every mail an identifier in `X-Thelia-Mail-Id`. They travel through the queue and are removed right before the mail is handed to the mail server. A worker keeps the identifiers of the mails the server accepted for a week, and does not send a mail whose identifier it kept: the queue may hand a job out twice when a worker stopped before acknowledging it. +- `MailerFactory` no longer takes an `OrderHistoryRecorder`. Its fourth constructor argument is now the required `Symfony\Component\Mailer\Transport\TransportInterface` (the `mailer.transports` service), used by `sendNow()`. A module that builds it with `new` and still passes the recorder gets a `TypeError`. +- With a queue, `sendEmailMessageOrFail()` and the methods built on it only throw when the mail could not be built or queued. A delivery failure no longer reaches the caller: the mail goes to the failure transport, `failed`. +- A `TemplatedEmail` that is not rendered yet is refused when it is queued, with a `LogicException`: the worker renders nothing. Render it before sending it (`BodyRenderer`), or send it without a queue. + +### Messenger configuration + +The core configures Messenger for the whole application: its serializer, its bus, its `failure_transport` and the routing of its messages come before the configuration of the project. An application or a module that already queued messages of its own (`App\Message\…`) gets a `LogicException` on dispatch until their classes are listed in the `thelia.messenger.allowed_message_classes` parameter, or live in the namespace of an active module and have a handler. See [Which classes a queue accepts](../architecture/background-jobs.md#which-classes-a-queue-accepts). + +### Import right + +Launching an import from the back office takes the `UPDATE` right on `admin.import`. The `VIEW` right used to be enough; it now only shows the imports and their jobs. Review the profiles of the administrators who import. + ## Thelia 3.2 A class you read from the container needs nothing when its constructor gains arguments. The "has to pass" cases below concern a module that builds the class with `new`. diff --git a/docs/upgrading/from-3.2-to-3.3.md b/docs/upgrading/from-3.2-to-3.3.md new file mode 100644 index 000000000..0eee26cef --- /dev/null +++ b/docs/upgrading/from-3.2-to-3.3.md @@ -0,0 +1,55 @@ +--- +title: Updating from 3.2 to 3.3 +sidebar_position: 2.3 +--- + +# Updating from 3.2 to 3.3 + +This page lists what is specific to the move from Thelia 3.2 to 3.3. The procedure itself +(backup, Composer, database script, cache, assets, modules) is on [Update](./update.md); the +steps below come on top of it. If you start from 3.1, apply +[Updating from 3.1 to 3.2](./from-3.1-to-3.2.md) first. + +Thelia 3.3 ships Symfony Messenger and the [background jobs](../architecture/background-jobs.md). +Nothing waits for a worker after the update: `MESSENGER_TRANSPORT_DSN` stays empty and every job +runs at once, as before, until you name a queue. + +## Symfony recipes + +- The core requires `symfony/messenger`. When Composer applies the Symfony recipe of that + package instead of the Thelia one, it writes `config/packages/messenger.yaml` and adds + `MESSENGER_TRANSPORT_DSN=doctrine://default?auto_setup=0` to `.env`, and every mail then waits + for a worker. Delete the `config/packages/messenger.yaml` the Symfony recipe wrote: the core + configures Messenger itself. A `messenger.yaml` of your own, such as one that routes a module + message, stays. Unless a worker runs, also delete that line, so `async` stays synchronous. +- The Symfony recipe of `symfony/scheduler` writes `src/Schedule.php`. Delete it too: the core + declares its own `thelia` schedule. + +## Queues and workers + +- Going back from a queue to no queue leaves the jobs still waiting in `messenger_messages`. + Let a worker empty the queues before you empty `MESSENGER_TRANSPORT_DSN`. +- The core configures Messenger for the whole application: its serializer, its bus, its + `failure_transport` and the routing of its messages come before the configuration of the + project. An application that already queued messages of its own (`App\Message\…`) gets a + `LogicException` on dispatch until their classes are listed in the + `thelia.messenger.allowed_message_classes` parameter. A module message is accepted when it + lives in the namespace of an active module and has a handler. +- With a queue, the web servers and the workers must share `var/data-transfer/import/` and + `var/cache/export/`, and a deployment must not empty them while jobs wait. See + [Running the workers](../getting-started/background-workers.md#deploying). +- On an AMQP queue, set `MESSENGER_HEAVY_TRANSPORT_DSN`: otherwise the exports and imports share + the queue of `async`, are read by its worker and tried again three times like its jobs. + +## Rights + +- Launching an import from the back office takes the `UPDATE` right on `admin.import`; `VIEW` + now only shows the imports and their jobs. Review the profiles of the administrators who + import. +- The new screen Configuration > System > Background jobs answers to the resource + `admin.configuration.background-jobs`, which the update gives to no profile: grant it to the + profiles that follow the jobs. Until then, only super-administrators see it. + +## Dependencies + +- The core requires `doctrine/dbal ^4.2`: a project or a module held on DBAL 3 moves to DBAL 4. diff --git a/docs/upgrading/update.md b/docs/upgrading/update.md index 48653481e..ef1aeca4f 100644 --- a/docs/upgrading/update.md +++ b/docs/upgrading/update.md @@ -26,6 +26,8 @@ Read the release notes of the version you move to, and the matching page in relies on, and a module or a template you depend on may need its own bump in `composer.json`. +If the shop runs background workers, stop them before you start and restart them at the end, as described in [Deploying](../getting-started/background-workers.md#deploying). + ## 1. Update the code A project installed with `composer create-project thelia/thelia-project` depends on @@ -140,9 +142,10 @@ run the module's own `update()` method: ```bash php Thelia module:refresh -php Thelia cache:clear ``` +Then rebuild the cache as in [step 3](#3-rebuild-the-cache), workers stopped. + ## Notes for each version Each release can ask for something the generic procedure above does not cover: a setting to @@ -151,6 +154,7 @@ cross, in order: - [Updating from 3.0 to 3.1](./from-3.0-to-3.1.md), patch releases 3.1.1, 3.1.2, 3.1.3 and 3.1.4 included. - [Updating from 3.1 to 3.2](./from-3.1-to-3.2.md), patch releases 3.2.1, 3.2.2 and 3.2.3 included. +- [Updating from 3.2 to 3.3](./from-3.2-to-3.3.md): the background jobs. ## Recommendations