Support for Redis Enterprise Labs (RLEC)

[Noxaro]

Expected Behavior

I can use a sharded redis enterprise labs cluster with spring session data redis without getting crossslot violations because of unallowed key renames.

Current Behavior

When i use spring session redis with redis enterprise labs it comes to cross slot violations when someone tries to login.
This ends up in 500 internal server error as the session key can not be changed.

org.springframework.dao.InvalidDataAccessApiUsageException: CROSSSLOT Keys in request don't hash to the same slot (command='RENAME', original slot=3229, wrong slot=8403, first key='redis:sessions:cbfa98af-b6ee-43ce-85e9-7013a0ebf0b3', violating key='redis:sessions:0122c855-b661-4cd4-aab9-4361a0ebc406'); nested exception is redis.clients.jedis.exceptions.JedisDataException: CROSSSLOT Keys in request don't hash to the same slot (command='RENAME', original slot=3229, wrong slot=8403, first key='redis:sessions:cbfa98af-b6ee-43ce-85e9-7013a0ebf0b3', violating key='redis:sessions:0122c855-b661-4cd4-aab9-4361a0ebc406')
	at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:69)
	at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:42)
	at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44)
	at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:42)
	at org.springframework.data.redis.connection.jedis.JedisConnection.convertJedisAccessException(JedisConnection.java:135)
	at org.springframework.data.redis.connection.jedis.JedisKeyCommands.rename(JedisKeyCommands.java:284)
	at org.springframework.data.redis.connection.DefaultedRedisConnection.rename(DefaultedRedisConnection.java:125)
	at org.springframework.data.redis.core.RedisTemplate.lambda$rename$16(RedisTemplate.java:936)
	at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:228)
	at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:188)
	at org.springframework.data.redis.core.RedisTemplate.rename(RedisTemplate.java:935)
	at org.springframework.session.data.redis.RedisIndexedSessionRepository$RedisSession.saveChangeSessionId(RedisIndexedSessionRepository.java:831)
	at org.springframework.session.data.redis.RedisIndexedSessionRepository$RedisSession.save(RedisIndexedSessionRepository.java:782)
	at org.springframework.session.data.redis.RedisIndexedSessionRepository$RedisSession.access$000(RedisIndexedSessionRepository.java:670)
	at org.springframework.session.data.redis.RedisIndexedSessionRepository.save(RedisIndexedSessionRepository.java:398)
	at org.springframework.session.data.redis.RedisIndexedSessionRepository.save(RedisIndexedSessionRepository.java:249)
	at org.springframework.session.web.http.SessionRepositoryFilter$SessionRepositoryRequestWrapper.commitSession(SessionRepositoryFilter.java:225)
	at org.springframework.session.web.http.SessionRepositoryFilter$SessionRepositoryRequestWrapper.access$100(SessionRepositoryFilter.java:192)
	at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:144)
	at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:103)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712)
	at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461)
	at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:384)
	at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312)
	at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:394)
	at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:253)
	at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:348)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:173)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1594)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: redis.clients.jedis.exceptions.JedisDataException: CROSSSLOT Keys in request don't hash to the same slot (command='RENAME', original slot=3229, wrong slot=8403, first key='redis:sessions:cbfa98af-b6ee-43ce-85e9-7013a0ebf0b3', violating key='redis:sessions:0122c855-b661-4cd4-aab9-4361a0ebc406')
	at redis.clients.jedis.Protocol.processError(Protocol.java:132)
	at redis.clients.jedis.Protocol.process(Protocol.java:166)
	at redis.clients.jedis.Protocol.read(Protocol.java:220)
	at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:318)
	at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:236)
	at redis.clients.jedis.BinaryJedis.rename(BinaryJedis.java:427)
	at org.springframework.data.redis.connection.jedis.JedisKeyCommands.rename(JedisKeyCommands.java:282)
	... 39 common frames omitted

Context
We built an oauth2 authorization server based on the spring boot stack and redis. As our service grew we needed to scale it out.
Initially we used redis open source which made it hard for us to meet our scaling requirements.
To solve this we migrated to Redis Labs Enterprise Cluster (used through Google Cloud Platform).
While this worked out very well scaling wise we now have the issue that the only working session driver redisson is not as stable as we'd like it to be.
Our first choice would be Spring Session as it's integrated well with the Spring Boot stack but unfortunately there's no support for Redis Labs Enterprise Cluster (RLEC).
As fas as we know, the reason for this is that RLEC behaves like a standalone redis server but doesn't support multi-key commands such as rename.
The point is that within Spring Session the redis rename command is being used in RedisSessionStore/RedisIndexedSessionStore::saveChangedSessionId()(https://github.com/spring-projects/spring-session/blob/master/spring-session-data-redis/src/main/java/org/springframework/session/data/redis/RedisIndexedSessionRepository.java#L831) to rename the key (which includes the current session key).
The question here is where do you see the support for RLEC? Spring Session or Spring Data Redis?
I am asking because we're currently facing the issue as described above and would like to contribute.
Thank you for your feedback and guidance!

Best Regards
Noxaro

[eleftherias]

Thanks for getting in touch @Noxaro.

There have been some similar issues reported in the Spring Data Redis project.

Could you take a look at https://jira.spring.io/browse/DATAREDIS-1154 and https://jira.spring.io/browse/DATAREDIS-1142 and see if either of those applies to your situation?

[Noxaro]

Hi @eleftherias

redis enterprise handles all the clustering stuff internaly.
So our application connects to a single instance endpoint and redis enterprise does the clustering magic.
But a sharded redis enterprise cluster comes also with some trade offs like that multi key operations like RENAMEare not supported.
=> https://docs.redislabs.com/latest/rc/concepts/clustering/

With that said our application connects to redis enterprise with a single instance configuration, as this is the recommended way.
I've tested spring session data redis with a little spring boot app to make sure spring-security-oauth has no impact on the described problem.
The relevant configuration part here is the application.yml which contains the following:

spring:
  session:
    store-type: redis
    redis:
      namespace: redis
  redis:
    host: redis-12345.******.europe-west4-mz.gcp.cloud.rlrcp.com
    port: 12345
    password: ********

With that configuration the same exception is thrown as described above.

Also a key rename is not possible when you try it on the redis-cli. Here is the output of it:

redis-12345.******.europe-west4-mz.gcp.cloud.rlrcp.com:12345> SET test value
OK
redis-12345.******.europe-west4-mz.gcp.cloud.rlrcp.com:12345> RENAME test test1
(error) CROSSSLOT Keys in request don't hash to the same slot (command='RENAME', original slot=6918, wrong slot=4768, first key='test', violating key='test1')

With that in mind i think both tickets are describing another problem then we have.

Best Regards
Noxaro

[rwinch]

@Noxaro Thanks for the additional details. Given that Spring Security must invoke HttpServletRequest.changeSessionId() to change the session id at log in to prevent session fixation attacks, what would you expect Spring Session to do differently?

[Noxaro]

Hi @rwinch,

the reason for this issue was to get an idea of where to implement a fix for the redis enterprise key rename handling.

Yesterday i took a look into the source code of spring-session and spring-data-redis.
I think you are right, spring-session can't do anything in this case.
The fix for the multi key operations should be implemented in spring-data-redis.

With this information i think we can close this issue.
We are going forward to get in touch with the spring-data-redis project.

Best Regards
Noxaro

[rwinch]

Thanks for the response. I'll close this issue.

For what it is worth, you could work around this by providing a custom implementation of RedisOperations that delegates to an existing implementation for operations that work, but then does something custom for operations like rename.