-
-
Notifications
You must be signed in to change notification settings - Fork 5.2k
Documented redis consumer group relation #11869
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 2 commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -900,6 +900,39 @@ serializer How to serialize the final payload ``Redis::SERIALIZER_PHP` | |
``Redis::OPT_SERIALIZER`` option) | ||
================== =================================== ======= | ||
|
||
.. tip:: | ||
|
||
At some point you'll likely want to scale the number of workers working on your queue. | ||
Make sure you assign the correct ``consumer`` and ``group`` values in that case. | ||
The more likely case is that you want every worker to work on the queue independently, reducing | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why is it likely?
I just googled about redis stream and their consumer groups. Maybe a link to redis docs should be added here? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah I was hesitant getting too much into how Redis streams work because that's already documented in the Redis docs (e.g. https://redis.io/topics/streams-intro). Actually that's the first link you get when you google for
Why scaling is likely? If one worker was enough, a simple database queue would probably be enough. I don't think one would choose Redis if the load wasn't bigger :) |
||
the time needed to process the pending messages. In that case, every single worker | ||
must have a different ``consumer`` option value so Redis can identify the different workers. | ||
When working with Docker containers one idea might be to use the ``HOSTNAME`` environment variable: | ||
|
||
.. configuration-block:: | ||
|
||
.. code-block:: yaml | ||
|
||
# config/packages/messenger.yaml | ||
framework: | ||
messenger: | ||
transports: | ||
redis: 'redis://localhost:6379/messages/symfony/%env(HOSTNAME)%' | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What would we use here if we weren’t using Docker? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You just have to make sure to pass individual consumer names. In whatever way you do that. So I can't tell you, it depends on what you use :) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. hostname (
Or find other case uniqueness like a domain prefix. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. E.g. for
And then maybe use
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Cool! Can we try that? To create a generic solution, we could say to use some generic env var like REDIS_CONSUMER - the difference would just be then how you set that, depending on your setup. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I tried now but I'm having some issues with finding the right spot 😄 It's not anything strictly related to the Redis transport. It's for any transport that needs deterministic values so I figured I'd place it as a tip where it imho belongs: the supervisor config part 😄 |
||
|
||
The less likely case would be if you wanted to have every single worker to process every single message. | ||
That means messages would be processed multiple times. In that case, you must have different ``group`` | ||
configurations. | ||
|
||
.. caution:: | ||
|
||
Be careful when using the ``HOSTNAME`` environment variable in orchestrated environments such as Kubernetes or | ||
Docker Swarm. It usually contains a random unique identifier which means if you destroy a container while it was | ||
working on a message, this message will remain in pending state forever as it is very unlikely there's ever going | ||
to be another worker with exactly the same ``HOSTNAME`` as the one you destroyed. In other words, you have to | ||
make sure you're using deterministic ``HOSTNAME`` values such as ``worker-1``, ``worker-2`` etc. | ||
In case you are using Kubernetes to orchestrate your containers, consider using a ``StatefulSet`` rather than | ||
a ``Deployment`` for example. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Let’s shorten this :). These cautions are great for the few people who have this case, but they tend to “weigh down” the docs over time. I think adding a comment in the code blocks above might be enough:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This caution block can be removed since abandoned messages are now claimed: symfony/symfony#35384 |
||
|
||
In Memory Transport | ||
~~~~~~~~~~~~~~~~~~~ | ||
|
||
|
Uh oh!
There was an error while loading. Please reload this page.