Null Elements Returned By CrudRepository.findAll() for Expired Keys [DATAREDIS-570]

[spring-projects-issues]

Brian Marcey opened DATAREDIS-570 and commented

The Redis documentation states that "if a key expires while the application is down the expiry event will not be processed which may lead to secondary indexes containing still references to the expired object."

I would expect this behavior and have indeed seen it while testing my code.

However, I have also noticed that if I restart my application after the above scenario occurs,
CrudRepostitory.findAll() will return a null value for each of the orphaned Ids. I didn't expect that and I can work around it, but it would be nice if SDR could detect this condition and not even include them in the returned Iterable. Note that CrudRepository.count() also returns an incorrect value under this scenario.

I can include more information if the above is not clear. Thank you!

---

Affects: 1.7.4 (Hopper SR4)

6 votes, 8 watchers

[spring-projects-issues]

arpan2501 commented

Team any updates on this, as we are still facing this issue.

We are retrieving null values with keys expiry time set to 5 minutes but data retrieved is days old which is null

[spring-projects-issues]

kubav182 commented

Can you pls fix this bug?

[spring-projects-issues]

Enoobong Ibanga commented

Hi Christoph Strobl any updates on this please?

[spring-projects-issues]

Eugene Zrazhevsky commented

Still being critical

[bebers22]

We are also facing that.
Any news for the fix?

[bergerst]

This bug not only affects findAll(), all the other query methods will also return incorrect data.

For example, findAll(example, PageRequest.ofSize(1)) will often return an empty Page with getTotal() returning a number > 0. count() will just measure the size of the Redis Set, including the expired ids.

I was able to implement a workaround to get rid of the garbage.

First, use @EnableRedisRepositories(enableKeyspaceEvents = ON_STARTUP) on a @Configuration bean so the expiration listener is active.

Then do something like this:

@Component
@RequiredArgsConstructor
public class RedisCleanupRunner implements ApplicationRunner {
  private final RedisTemplate<String, String> template;

  @Override
  public void run(ApplicationArguments args) throws Exception {
    SetOperations<String, String> setOps = template.opsForSet();
    HashOperations<String, Object, Object> hashOps = template.opsForHash();
    private final String redisKey = "yourKeySpace";

    Set<String> ids = setOps.members(redisKey);
    for (String id : ids) {
      String key = redisKey + ":" + id;

      if (!TRUE.equals(template.hasKey(key))) {
          hashOps.put(key, "foo", "bar");
          template.expire(key, Duration.ofMillis(1));
      }
    }
  }
}

The expiration of the newly created entries will trigger the MappingExpirationListener to delete all the phantoms and indexes and remove the ID from the Set.

If you have multiple RedisHash classes, run the above code in a loop for each one.

[german1608]

Hello. Is this fixed? I'm dealing with this problem :(

[Orahpajo]

I'm joining the 'would be great if this would be fixed' crowd. But so far the workaround of @bergerst is working.

[cgpoh]

I tried @bergerst workaround but it is not working for me. The issue is very random.

[bergerst]

@cgpoh Did you also use my workaround in #2200 ?

[cgpoh]

@bergerst , thanks! I tried your workaround in #2200 by setting notify-keyspace-events Ex in the redis.conf file but I still get the null pointer exception when doing the query.