Pipelined usage of Redis Cluster with Lettuce fails for some commands that return `PipelinedRedisFuture`

[EMDavl]

Hello!
I encountered an issue while using RedisTemplate.executePipelined with Redis Cluster. I'm attaching a Minimal Reproducible Example (MRE) for your reference. This issue occurs both in a locally deployed cluster and in a cluster deployed in a test environment.

The problem arises when calling mSet with keys that map to different nodes. The data itself gets written to Redis, but then an error is thrown. It seems that the cause is here. In this case, resultHolder is a PipelinedRedisFuture, which cannot be cast to RedisCommand.

It reproduces since 2.7.11

Here's the stack trace.

org.springframework.data.redis.RedisSystemException: Unknown redis exception
	at org.springframework.data.redis.FallbackExceptionTranslationStrategy.getFallback(FallbackExceptionTranslationStrategy.java:49) ~[spring-data-redis-3.2.4.jar:3.2.4]
	at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:39) ~[spring-data-redis-3.2.4.jar:3.2.4]
	at org.springframework.data.redis.connection.lettuce.LettuceConnection.convertLettuceAccessException(LettuceConnection.java:306) ~[spring-data-redis-3.2.4.jar:3.2.4]
	at org.springframework.data.redis.connection.lettuce.LettuceConnection.lambda$doInvoke$1(LettuceConnection.java:414) ~[spring-data-redis-3.2.4.jar:3.2.4]
	at org.springframework.data.redis.connection.lettuce.LettuceInvoker$Synchronizer.invoke(LettuceInvoker.java:673) ~[spring-data-redis-3.2.4.jar:3.2.4]
	at org.springframework.data.redis.connection.lettuce.LettuceInvoker$DefaultSingleInvocationSpec.get(LettuceInvoker.java:589) ~[spring-data-redis-3.2.4.jar:3.2.4]
	at org.springframework.data.redis.connection.lettuce.LettuceStringCommands.mSet(LettuceStringCommands.java:152) ~[spring-data-redis-3.2.4.jar:3.2.4]
	at org.springframework.data.redis.connection.DefaultedRedisConnection.mSet(DefaultedRedisConnection.java:354) ~[spring-data-redis-3.2.4.jar:3.2.4]
	at org.springframework.data.redis.connection.DefaultStringRedisConnection.mSet(DefaultStringRedisConnection.java:645) ~[spring-data-redis-3.2.4.jar:3.2.4]
	at ru.emdavl.Runner.lambda$run$0(Main.java:35) ~[classes/:na]
	at org.springframework.data.redis.core.RedisTemplate.lambda$executePipelined$1(RedisTemplate.java:475) ~[spring-data-redis-3.2.4.jar:3.2.4]
	at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:396) ~[spring-data-redis-3.2.4.jar:3.2.4]
	at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:363) ~[spring-data-redis-3.2.4.jar:3.2.4]
	at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:350) ~[spring-data-redis-3.2.4.jar:3.2.4]
	at org.springframework.data.redis.core.RedisTemplate.executePipelined(RedisTemplate.java:471) ~[spring-data-redis-3.2.4.jar:3.2.4]
	at org.springframework.data.redis.core.RedisTemplate.executePipelined(RedisTemplate.java:465) ~[spring-data-redis-3.2.4.jar:3.2.4]
	at ru.emdavl.Runner.run(Main.java:34) ~[classes/:na]
	at org.springframework.boot.SpringApplication.lambda$callRunner$5(SpringApplication.java:790) ~[spring-boot-3.2.4.jar:3.2.4]
	at org.springframework.util.function.ThrowingConsumer$1.acceptWithException(ThrowingConsumer.java:83) ~[spring-core-6.1.5.jar:6.1.5]
	at org.springframework.util.function.ThrowingConsumer.accept(ThrowingConsumer.java:60) ~[spring-core-6.1.5.jar:6.1.5]
	at org.springframework.util.function.ThrowingConsumer$1.accept(ThrowingConsumer.java:88) ~[spring-core-6.1.5.jar:6.1.5]
	at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:798) ~[spring-boot-3.2.4.jar:3.2.4]
	at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:789) ~[spring-boot-3.2.4.jar:3.2.4]
	at org.springframework.boot.SpringApplication.lambda$callRunners$3(SpringApplication.java:774) ~[spring-boot-3.2.4.jar:3.2.4]
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) ~[na:na]
	at java.base/java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357) ~[na:na]
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:510) ~[na:na]
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[na:na]
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[na:na]
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[na:na]
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:na]
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[na:na]
	at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:774) ~[spring-boot-3.2.4.jar:3.2.4]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:341) ~[spring-boot-3.2.4.jar:3.2.4]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1354) ~[spring-boot-3.2.4.jar:3.2.4]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1343) ~[spring-boot-3.2.4.jar:3.2.4]
	at ru.emdavl.Main.main(Main.java:20) ~[classes/:na]
Caused by: java.lang.ClassCastException: class io.lettuce.core.cluster.PipelinedRedisFuture cannot be cast to class io.lettuce.core.protocol.RedisCommand (io.lettuce.core.cluster.PipelinedRedisFuture and io.lettuce.core.protocol.RedisCommand are in unnamed module of loader 'app')
	at org.springframework.data.redis.connection.lettuce.LettuceResult.<init>(LettuceResult.java:54) ~[spring-data-redis-3.2.4.jar:3.2.4]
	at org.springframework.data.redis.connection.lettuce.LettuceResult$LettuceResultBuilder.build(LettuceResult.java:149) ~[spring-data-redis-3.2.4.jar:3.2.4]
	at org.springframework.data.redis.connection.lettuce.LettuceConnection.newLettuceResult(LettuceConnection.java:467) ~[spring-data-redis-3.2.4.jar:3.2.4]
	at org.springframework.data.redis.connection.lettuce.LettuceConnection.lambda$doInvoke$1(LettuceConnection.java:411) ~[spring-data-redis-3.2.4.jar:3.2.4]
	... 33 common frames omitted

[mp911de]

Thanks for the report. We actually never intended to support Cluster via pipelining due to the fact that Cluster consists of multiple connections and we would kind-of create a asynchronous command queue that is distributed across nodes.

The Jedis implementation doesn't consider cluster pipelining at all. Some Lettuce commands are implemented in a synchronous way (e.g. bRPopLPush, sort) to emulate cross-slot commands across multiple nodes.

However, it makes sense to allow commands that Lettuce returns as PipelinedRedisFuture and not to fail. In that sense, we should also go over the commands that we emulate in a synchronous way and to investigate which commands can be utilized from Lettuce so that we can rely on Lettuce's async implementation.

[EMDavl]

Could you please clarify how I can use pipelining considering that we have Redis in cluster mode, and we're using spring-data-redis with lettuce? What would be the best practice?

Currently, I just replaced mset with setex, and everything started working fine.

[mp911de]

We intend to refine what we support and what is out of support. Cluster was never built with pipelining in mind, also transactions are not supported. Back in the day when Redis Cluster was implemented, Jedis didn't support pipelining at all.

Pipelining with Lettuce works by coincidence as Lettuce has no specific pipelining API, we leverage pipelining by using its async API.

You can find a branch holding some changes around support and documentation at #2889.

Going forward, we could revisit our Jedis arrangement and investigate how to enable pipelining for commands that do not require cross-slot-specific handling.