Provide Redis metadata store that writes key/values instead of single hash [INT-4065]

[spring-operator]

Aaron Loes opened INT-4065 and commented

Took a look at the RedisMetadataStore for use with the IdempotentReceiverInterceptor for deduplication for our high throughput data streams. We noticed that it maintains a single hash object for the metadata. For this use case, having a single hash object for all metadata doesn't really work as we would blow out the memory of any server pretty quickly. Since we really don't care about data after a deterministic amount of time because we're really only trying to avoid reprocessing data we've already seen due to a replay scenario, ideally we would want the metadata to expire.

Being that hash keys cannot have an expiration on them (i believe) and because the current implementation doesn't allow for metadata expiration, a version that doesn't use a hash but rather individual key/value pairs for each metadata record would suffice as we could turn expiration from the server. But more ideal would be to set a TTL for records individually for differing needs.

I could see this in a few different forms.

1. an alternative implementation of the MetadataStore (RedisKeyValueMetadataStore) that uses key/value instead of hash and that also allows for TTL
   or
2. an implementation of the existing RedisMetadataStore that's takes in some sort of strategy for writing data to redis where one strategy would be to use a single hash and another would use key/value pairs prefixed with a key and accounted for TTL

---

Reference URL: https://stackoverflow.com/questions/45495329/using-redismetadatastore-redisproperties-with-a-redis-cluster

2 votes, 4 watchers

[spring-operator]

Artem Bilan commented

+1 for RedisKeyValueMetadataStore and maybe with the configurable prefix to distinguish pairs for this instance from others on Redis server.

[spring-operator]

Artem Bilan commented

See the Reference URL for the problem in Redis cluster.

So, that might make sense to revise this issue and implement it finally.