Skip to content

Documented redis consumer group relation #11869

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

Documented redis consumer group relation #11869

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ffe9dff
Documented redis consumer group relation
Toflar Jul 3, 2019
525dfdc
Clarified HOSTNAME environment variable
Toflar Jul 3, 2019
dc557d5
Tried to carify even more and incorporated the Supervisor example for…
Toflar Jul 8, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions messenger.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -900,6 +900,39 @@ serializer How to serialize the final payload ``Redis::SERIALIZER_PHP`
``Redis::OPT_SERIALIZER`` option)
================== =================================== =======

.. tip::

At some point you'll likely want to scale the number of workers working on your queue.
Make sure you assign the correct ``consumer`` and ``group`` values in that case.
The more likely case is that you want every worker to work on the queue independently, reducing
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it likely?

reducing the time needed to process the pending messages

I just googled about redis stream and their consumer groups. Maybe a link to redis docs should be added here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I was hesitant getting too much into how Redis streams work because that's already documented in the Redis docs (e.g. https://redis.io/topics/streams-intro). Actually that's the first link you get when you google for "redis streams" :)

Why is it likely?

Why scaling is likely? If one worker was enough, a simple database queue would probably be enough. I don't think one would choose Redis if the load wasn't bigger :)

the time needed to process the pending messages. In that case, every single worker
must have a different ``consumer`` option value so Redis can identify the different workers.
When working with Docker containers one idea might be to use the ``HOSTNAME`` environment variable:

.. configuration-block::

.. code-block:: yaml

# config/packages/messenger.yaml
framework:
messenger:
transports:
redis: 'redis://localhost:6379/messages/symfony/%env(HOSTNAME)%'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would we use here if we weren’t using Docker?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You just have to make sure to pass individual consumer names. In whatever way you do that. So I can't tell you, it depends on what you use :)

Copy link
Contributor

@alexander-schranz alexander-schranz Jul 3, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hostname ( php -r "var_dump(gethostname());" ) would also work in many cases for none docker setup as its unique in its network. If not because servers are not in the same network it I think he need to set it manually like the example I posted above or find another variable e.g.:

# Worker 1
DSN=redis://localhost:6379/messages/symfony/consumer-1
# Worker 2
DSN=redis://localhost:6379/messages/symfony/consumer-2

Or find other case uniqueness like a domain prefix.

Copy link
Contributor Author

@Toflar Toflar Jul 3, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

E.g. for supervisor we need to find out how we can somehow pass on the numprocs as individual identifiers. I haven't tested but something like this might be working:

[program:messenger-consume]
command=php /path/to/your/app/bin/console messenger:consume async --time-limit=3600
user=ubuntu
numprocs=2
autostart=true
autorestart=true
process_name=%(program_name)s_%(process_num)02d
environment = PROCESS_NUM=%(process_num)d

And then maybe use

redis: 'redis://localhost:6379/messages/symfony/consumer-%env(PROCESS_NUM)%'

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool! Can we try that? To create a generic solution, we could say to use some generic env var like REDIS_CONSUMER - the difference would just be then how you set that, depending on your setup.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried now but I'm having some issues with finding the right spot 😄 It's not anything strictly related to the Redis transport. It's for any transport that needs deterministic values so I figured I'd place it as a tip where it imho belongs: the supervisor config part 😄


The less likely case would be if you wanted to have every single worker to process every single message.
That means messages would be processed multiple times. In that case, you must have different ``group``
configurations.

.. caution::

Be careful when using the ``HOSTNAME`` environment variable in orchestrated environments such as Kubernetes or
Docker Swarm. It usually contains a random unique identifier which means if you destroy a container while it was
working on a message, this message will remain in pending state forever as it is very unlikely there's ever going
to be another worker with exactly the same ``HOSTNAME`` as the one you destroyed. In other words, you have to
make sure you're using deterministic ``HOSTNAME`` values such as ``worker-1``, ``worker-2`` etc.
In case you are using Kubernetes to orchestrate your containers, consider using a ``StatefulSet`` rather than
a ``Deployment`` for example.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let’s shorten this :). These cautions are great for the few people who have this case, but they tend to “weigh down” the docs over time.

I think adding a comment in the code blocks above might be enough:

For Kubernetes, use a StatefulSet to avoid randomly host names

Copy link
Contributor

@toooni toooni Jul 30, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This caution block can be removed since abandoned messages are now claimed: symfony/symfony#35384


In Memory Transport
~~~~~~~~~~~~~~~~~~~

Expand Down