Skip to content

[Messenger] More on unique Redis consumer names #14888

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 21, 2021
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 19 additions & 4 deletions messenger.rst
Original file line number Diff line number Diff line change
Expand Up @@ -619,7 +619,18 @@ times:
process_name=%(program_name)s_%(process_num)02d

Change the ``async`` argument to use the name of your transport (or transports)
and ``user`` to the Unix user on your server. Next, tell Supervisor to read your
and ``user`` to the Unix user on your server.

If you use the Redis Transport, note that each worker needs a unique consumer name to
avoid the same message being handled by multiple workers. One way to achieve this is
to set an environment variable in the Supervisor configuration file, which you can
then refer to in `messenger.yaml` (see Redis section above):

.. code-block:: ini

environment=MESSENGER_CONSUMER_NAME=%(program_name)s_%(process_num)02d

Next, tell Supervisor to read your
config and start your workers:

.. code-block:: terminal
Expand Down Expand Up @@ -1209,9 +1220,13 @@ claim_interval Interval on which pending/abandoned ``60000`` (1 Minute)
.. caution::

There should never be more than one ``messenger:consume`` command running with the same
config (stream, group and consumer name) to avoid having a message handled more than once.
Using the ``HOSTNAME`` as the consumer might often be a good idea. In case you are using
Kubernetes to orchestrate your containers, consider using a ``StatefulSet``.
combination of ``stream``, ``group`` and ``consumer``, or messages could end up being
handled more than once. If you run multiple queue workers, ``consumer` can be set to an
environment variable (like ``%env(MESSENGER_CONSUMER_NAME)%`)` set by Supervisor
(example below) or whatever service used to manage the worker processes.
In a container environment, the ``HOSTNAME`` can be used as the consumer name, since
there is only one worker per container/host. If using Kubernetes to orchestrate the
containers, consider using a ``StatefulSet`` to have stable names.

.. tip::

Expand Down