add the ability to create internal queues as quorum type instead of hard-coded classic

[Michalis-Apostolou]

Need

We need to upgrade the rmq cluster to version 4+. in this version High Availability (HA) policy with mirror queues has been deprecated and completely removed https://www.rabbitmq.com/docs/3.13/ha . So we need to replace all classic queue definitions with quorum

Problem

Rabbit-queue creates internal queues to implement scheduled publishing for a message

1. there is a queue created on app connect {prefix}_delay_reply
2. The publisher for a delayed messaged for XXX amount creates the queue {prefix}_delay_{XXX}

Example:

Since the library does not define the type of the queue , the are created to what is default on the virtual host.

If can't/don't want to change the default queue type of the virtual host , you should have the ability to define if these queues should be created as quorum or not.

Solution

By utilizing a new option available in the package scheduledPublishQueuesAsQuorum on rabbit config, this will passed around during the creating of the queues.
In order to avoid conflicts between already created classic queues and newly created quorum queues , the naming convention changes to

1. {prefix}_delay_quorum_reply
2. {prefix}_delay_quorum_{XXX}

[iliasbibas]

I think we need to look a bit deeper on how quorum queues handle dead lettering, a policy might be needed if we need the messages to end up in a DLQ instead of silently dropped.

[Michalis-Apostolou]

I think we need to look a bit deeper on how quorum queues handle dead lettering, a policy might be needed if we need the messages to end up in a DLQ instead of silently dropped.

@iliasbibas Are you referring to enable the at-least-once guarantee ? https://www.rabbitmq.com/docs/quorum-queues#activating-at-least-once-dead-lettering. to override the default at-most-once ?

[iliasbibas]

I think we need to look a bit deeper on how quorum queues handle dead lettering, a policy might be needed if we need the messages to end up in a DLQ instead of silently dropped.

@iliasbibas Are you referring to enable the at-least-once guarantee ? https://www.rabbitmq.com/docs/quorum-queues#activating-at-least-once-dead-lettering. to override the default at-most-once ?

Yes, for the delay queues I believe at-least-once and overflow: reject-publish should be set so that the delayed job end up in the delay_reply queue (I think that the delay_reply is already set as the target for these messages).

[iliasbibas]

Btw @Michalis-Apostolou , the _quorum_ addition in the queue name should perhaps be optional right? Otherwise it break compatibility, ie will doubly create queues even if an existing quorum queue without the name addition exists.

[DimitrK]

💭 hold the naming order of quorum until SREs verify what regexps are using for alerts on delay queues (cc @a-margaritis )

[Michalis-Apostolou]

Btw @Michalis-Apostolou , the _quorum_ addition in the queue name should perhaps be optional right? Otherwise it break compatibility, ie will doubly create queues even if an existing quorum queue without the name addition exists.

it was intentional , because if there was already existing a classic queue with the same name. and you opt-in to create the same queue with different config (i.e. quorum queue type) it will conflict ungracefully and will cause to disconnect.

Then there is the issue on how to resolve such conflict. you can't update a queue config. you have to destroy and recreate the queue. During the recreation of the queue in a high traffic topic may lead to message loss

[iliasbibas]

Btw @Michalis-Apostolou , the _quorum_ addition in the queue name should perhaps be optional right? Otherwise it break compatibility, ie will doubly create queues even if an existing quorum queue without the name addition exists.

it was intentional , because if there was already existing a classic queue with the same name. and you opt-in to create the same queue with different config (i.e. quorum queue type) it will conflict ungracefully and will cause to disconnect.

Then there is the issue on how to resolve such conflict. you can't update a queue config. you have to destroy and recreate the queue. During the recreation of the queue in a high traffic topic may lead to message loss

I understand that, I'm saying that in a cluster that already has a quorum queue example_queue (e.g. vhost default) , creating a new example_quorum_queue by default will not be correct.
Also an introduction of new naming conventions will require updating the existing policies, most of which rely on regex.