Skip to content

Commit 1da2c37

Browse files
authored
[Messenger] More on unique Redis consumer names
1 parent 80cd5b8 commit 1da2c37

File tree

1 file changed

+18
-4
lines changed

1 file changed

+18
-4
lines changed

messenger.rst

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -619,7 +619,17 @@ times:
619619
process_name=%(program_name)s_%(process_num)02d
620620
621621
Change the ``async`` argument to use the name of your transport (or transports)
622-
and ``user`` to the Unix user on your server. Next, tell Supervisor to read your
622+
and ``user`` to the Unix user on your server.
623+
624+
If you use the Redis Transport, note that each worker needs a unique consumer name to
625+
avoid the same message being handled by multiple workers. One way to achieve this is
626+
to set an environment variable in the Supervisor configuration file, which you can
627+
then refer to in `messenger.yaml` (see Redis section above):
628+
629+
.. code-block:: ini
630+
environment=MESSENGER_CONSUMER_NAME=%(program_name)s_%(process_num)02d
631+
632+
Next, tell Supervisor to read your
623633
config and start your workers:
624634

625635
.. code-block:: terminal
@@ -1209,9 +1219,13 @@ claim_interval Interval on which pending/abandoned ``60000`` (1 Minute)
12091219
.. caution::
12101220

12111221
There should never be more than one ``messenger:consume`` command running with the same
1212-
config (stream, group and consumer name) to avoid having a message handled more than once.
1213-
Using the ``HOSTNAME`` as the consumer might often be a good idea. In case you are using
1214-
Kubernetes to orchestrate your containers, consider using a ``StatefulSet``.
1222+
combination of ``stream``, ``group`` and ``consumer``, or messages could end up being
1223+
handled more than once. If you run multiple queue workers, ``consumer` can be set to an
1224+
environment variable (like ``%env(MESSENGER_CONSUMER_NAME)%`)` set by Supervisor
1225+
(example below) or whatever service used to manage the worker processes.
1226+
In a container environment, the ``HOSTNAME`` can be used as the consumer name, since
1227+
there is only one worker per container/host. If using Kubernetes to orchestrate the
1228+
containers, consider using a ``StatefulSet`` to have stable names.
12151229

12161230
.. tip::
12171231

0 commit comments

Comments
 (0)