Add `cursorId` to `ScanOptions`

[wendao8469759]

I have searched all the information and found that scan cannot set cursor and setting count is invalid. What is the correct way to write it here?

[wendao8469759]

private Set<String> scanTargetKeys(String keyWords, Integer pageNum,Integer pageSize) {
        if (StringUtil.isEmpty(keyWords)) {
            keyWords = "*";
        }
        String finalKeyWords = keyWords;
        return redisTemplate.execute((RedisCallback<Set<String>>) connection -> {
            Set<String> keysTmp = new HashSet<>();
            ScanOptions.ScanOptionsBuilder scanOptionsBuilder = ScanOptions.scanOptions();
            scanOptionsBuilder.match(finalKeyWords);
            scanOptionsBuilder.count(pageSize);

            Cursor<byte[]> cursor = connection.scan(scanOptionsBuilder.build());
            int i = 0;
            while (cursor.hasNext()) {
                String key = new String(cursor.next());
                if (i >= (pageNum - 1) * pageSize && i < pageNum * pageSize) {
                    keysTmp.add(key);
                }
                if (i >= pageNum * pageSize) {
                    break;
                }
                i++;
            }

            return keysTmp;
        });
    }

[jxblum]

You might want to post this "question" on StackOverflow (here) to the [Spring Data] Redis community at large.

GitHub Issues for Spring Data Redis are for bugs, improvements and new features in SD Redis itself, or changes to the documentation, whether in the reference doc or Javadoc. This is not a QA or support forum.

Having said this, I am curious why you chose to perform a SCAN inside a RedisCallback using the RedisTemplate rather than using the RedisTemplate.scan(:ScanOptions) method (Javadoc), specifically?

The primary difference in approach here is that with RedisTemplate.scan(:ScanOptions) method (unlike your (manually executed SCAN above) will configure and use a "sticky" connection, which is likely necessary for the application to remember the prior Cursor location on subsequent CURSOR command invocations.

If you simply invoke RedisTemplate.execute(:RedisCallback<?>), then it is possible that a new connection will be acquired. Even if it is an existing (likely "pooled") connection, it is not necessarily the same connection used to perform the initial SCAN and therefore no prior state of the Cursors location would have been retained on the subsequent call.

I would simply change your code to do the following:

private Set<String> scanTargetKeys(String keyWords, Integer pageNum,Integer pageSize) {

    // ...

    ScanOptions scanOptions = ScanOptions.scanOptions()
        .match(finalKeyWords)
        .count(pageSize)
        .build()

    Cursor<byte[]> cursor = redisTemplate.scan(scanOptions);

    // iterator over the Cursor as before

}    

Then when your scanTargetKeys(..) method is invoked again, this should give you the result your looking for???

If not, please provide a reproducer of the issue you are experiencing (preferably) as minimal test case or using a simple example.

[wendao8469759]

There may be some issues with my description, but I actually want to implement the Redis command: scan cursor [MATCH pattern] [COUNT count]
use cursor. next() is called, it starts iterating from the position of the previous cursor, rather than starting from 0. Currently, I use

Cursor<String> cursor = redisTemplate.scan(scanOptions);
cursor.next();

I found that every iteration starts from 0. If I have many redis keys, does it mean that I need to iterate from scratch to obtain the subsequent keys?

[wendao8469759]

    private Set<String> scanTargetKeys2(String keyWords, Long pageNum, Long pageSize) {
        Set<String> keysTmp = new HashSet<>();
        if (StringUtil.isEmpty(keyWords)) {
            keyWords = "*";
        }
        String finalKeyWords = keyWords;

        ScanOptions scanOptions = ScanOptions.scanOptions()
                .match(finalKeyWords)
                .count(pageSize)
                .build();
        Cursor<String> cursor = redisTemplate.scan(scanOptions);
        while (cursor.hasNext()) {
            long cursorId = cursor.getCursorId();
            long position = cursor.getPosition();
            String key = cursor.next();
            System.out.println("cursorId = " + cursorId + " position = " + position);
            if (position >= (pageNum - 1) * pageSize && position < pageNum * pageSize) {
                keysTmp.add(key);
                if(keysTmp.size() == pageSize){
                    break;
                }
            }
            if (position >= pageNum * pageSize) {
                break;
            }
        }
        return keysTmp;
    }

When my pageSize=1 and pageNum=10; My loop has been executed 10 times.
When my pageSize=2 and pageNum=10; My loop has been executed 20 times.
By analogy, when my pageSize=100, my loop executes pageSize * pageNum times.
Is this correct?

[jxblum]

I will need to think on this more...

I was re-reading about the Redis SCAN command (doc) and came across a few more interesting tidbits. Search for:

• "Number of elements returned at every SCAN call"
• "The COUNT option" (logically following the first read, and it seems COUNT may only be a "hint")
• "Why SCAN may return all the items of an aggregate data type in a single call?"
• ...

Although, it seems like SCAN itself should do the right thing (because SCAN always uses a key space with hash tables, unlike SSCAN, HSCAN and ZSCAN)??

I need to write a test case to verify some things.

[wendao8469759]

Yes, according to the explanation in the official website doc。
Time complexity: O(1) for every call. O(N) for a complete iteration, including enough command calls for the cursor to return back to 0. N is the number of elements inside the collection.

[jxblum]

PART 1 (of 2):

After a more careful and thorough analysis, I have learned more about Redis's behavior as well as limitations (questionably, and arguably, a bug) in Spring Data Redis.

NOTE: This will need the extended Spring Data team's attention and review before we move forward on this ticket.

The limitation (bug?) in Spring Data Redis (SD Redis) is that it does not take the current "cursor" position into account when subsequently running and repeating the Redis SCAN command.

If we trace the SCAN down from the RedisTemplate, starting here, first you see that it uses a Redis connection object to run the Redis SCAN command (here). Ultimately, SD Redis is going to use a KeyCommands implementation (either from Jedis or Lettuce, thus the "driver") to execute the Redis SCAN command using the Jedis or Lettuce connection, as seen here.

JEDIS

If we trace from Jedis, we see this block of code runs the SCAN using Jedis (Javadoc), for instance, here (typeless call).

We notice that the "cursorId" is the "cursor" argument to the Jedis.scan(cursor:byte[], :ScanOptions) call in this case. It is this "cursor" that will set the beginning for the next, or subsequent scan.

However, nothing calls this SD Redis JedisKeyCommands.scan(..) method other than the overloaded method above, which simply passes 0 for the current "cursor" position, thereby always causing the scan to start at the beginning.

LETTUCE

A similar outcome is revealed when tracing from Lettuce. Again, we see this block of code.

Ultimately, the limitation is induced by the SD Redis ScanOptions type (Javadoc), which does not give you an "option" for setting the current "cursor" position (a marker from a previous scan). This is important since the SD Redis ScanOptions gets "converted" to Lettuce ScanArgs (Javadoc) before executing the SCAN, invoked from here, and as seen here.

As you can see in the conversion, only the "pattern", "count" (LIMIT) and (optionally) type are taken into consideration, which is arguably a limitation derived from Lettuce's KeyScanArgs type (Javadoc) itself, which has no option to set the current "cursor" position.

We actually see the same limitation with Jedis when converting the SD Redis ScanOptions to Jedis ScanParams (Javadoc).

Although, like Jedis, which allows the current "cursor" position to be passed in the call to Jedis.scan(..), again as seen here, Lettuce provides the means to pass the current "cursor" position into the Lettuce driver when the SCAN is invoked, by "customizing" (or reconstructing) the already constructed Lettuce ScanCursor object (Javadoc) using the SD Redis ScanOptions object, from here to here.

SUMMARY

So, in summary, it is arguable and possible that Spring Data Redis could set the current position of the "cursor" for subsequent SCANs.

However, the question is, does it matter?