diff --git a/messenger.rst b/messenger.rst index db280b9cc4d..19572eb127e 100644 --- a/messenger.rst +++ b/messenger.rst @@ -623,6 +623,32 @@ config and start your workers: See the `Supervisor docs`_ for more details. + +.. tip:: + + To get a deterministic worker name like for e.g. the Redis transport, you may pass the process number as an + environment variable. + + .. code-block:: ini + + ;/etc/supervisor/conf.d/messenger-worker.conf + [program:messenger-consume] + […] + environment = REDIS_CONSUMER=consumer-%(process_num)d + + You can then use it in your messenger configuration: + + .. configuration-block:: + + .. code-block:: yaml + + # config/packages/messenger.yaml + framework: + messenger: + transports: + redis: 'redis://localhost:6379/messages/symfony/%env(REDIS_CONSUMER)%' + + .. _messenger-retries-failures: Retries & Failures @@ -900,6 +926,40 @@ serializer How to serialize the final payload ``Redis::SERIALIZER_PHP` ``Redis::OPT_SERIALIZER`` option) ================== =================================== ======= +.. tip:: + + At some point you'll likely want to scale the number of workers working on your queue. + Make sure you assign the correct ``consumer`` and ``group`` values in that case. + The more likely case is that you want every worker to work on the queue independently, reducing + the time needed to process the pending messages. In that case, every single worker + must have a different ``consumer`` option value so Redis can identify the different workers. + Your workers should thus be configured e.g. as follows: + + * Worker 1: redis://localhost:6379/messages/symfony/consumer-1 + * Worker 2: redis://localhost:6379/messages/symfony/consumer-2 + * Worker 3: redis://localhost:6379/messages/symfony/consumer-3 + * […] + + The less likely case would be if you wanted to have every single worker to process every single message. + That means messages would be processed multiple times. In that case, you must have different ``group`` + configurations: + + * Worker 1: redis://localhost:6379/messages/symfony-1/consumer + * Worker 2: redis://localhost:6379/messages/symfony-2/consumer + * Worker 3: redis://localhost:6379/messages/symfony-3/consumer + * […] + +.. caution:: + + Be careful about the ``consumer`` and ``group`` names you choose. It might be tempting to use e.g. the ``HOSTNAME`` + environment variable in orchestrated, containerized environments (Docker Swarm, Kubernetes, etc.) to get a unique + host name. However, they usually contain a random unique identifier which means if you destroy a container while it was + working on a message, this message will remain in pending state forever as it is very unlikely there's ever going + to be another worker with exactly the same ``HOSTNAME`` as the one you destroyed. In other words, you have to + make sure you're using deterministic values such as ``worker-1``, ``worker-2`` etc. + In case you are using Kubernetes to orchestrate your containers, consider using a ``StatefulSet`` rather than + a ``Deployment`` for example. + In Memory Transport ~~~~~~~~~~~~~~~~~~~