Set default consistency at project level, per profile

[JesusTheHun]

According to the documentation, the default consistency level for Repositories is NOT_BOUNDED ( https://docs.spring.io/spring-data/couchbase/docs/current/reference/html/#couchbase.repository.consistency ).
This is fine for some environment, but not for others. Typically during tests, most of them fail because they try to read their own writes, and it fails.
With a previous version of Spring Data Couchbase based on SDK 2.x we could extend getDefaultConsistency() from AbstractCouchbaseConfiguration. This method is now gone and as we are migrating, it breaks our app. No green tests, no delivery.

[mikereiche]

I'm not sure why it was removed - it was before my time. I suspect it's because it was being set (for testing), forgotten about and left that way for production - slowing down every query. I'll ask about adding it back.

As of 4.3.0-M3, the ScanConsistency annotation can go on entity classes.

@ScanConsistency(query = QueryScanConsistency.REQUEST_PLUS)

[JesusTheHun]

So what are my options right now ? Update my entities/repos to use request plus and once you make a new release we can go NOT_BOUNDED ?

Edit : is there a way with the SDK to programmatically say "wait all my inserts" ?
In some business cases it can be required to wait for inserts to be indexed, example :

repo1.save(doc1);
repo1.save(doc2);
repo2.save(doc3);
// await index to persist
repo3.findAllUsingSomePredicate() // some computation query that requires doc1 to doc3 to be indexed. If this query is annotated with @ScanConsistency(query = QueryScanConsistency.REQUEST_PLUS), will it see the doc1 to doc3 ?

I also have some concerns about the order of indexation. Using the outbox pattern and a message broker :

// on the business node
start tx
insert 1000 doc
insert a message
end tx

// on the event processing node
watchingNewMessage
onNewMessage fetchThe1000Docs // is this guaranteed to see the docs inserted in the tx ?

[mikereiche]

1. add the annotation to such repo methods in the repository interface:

@ScanConsistency(query = QueryScanConsistency.REQUEST_PLUS)
List<Airport> findAllUsingSomePredicate();

2. that we restore getDefaultConsistency() to 4.2.x. You could test with 4.2.6-SNAPSHOT version starting tomorrow (it's updated each day). The release of 4.2.6 is October 14. We don't make the releases - they are done by the spring data framework team ( https://calendar.spring.io/ it's labeled 2021.0.6 == 4.2.6). There. are instructions on how to use snapshot and milestones here - https://docs.spring.io/spring-data/couchbase/docs/4.2.5/reference/html/#reference (although I believe the repos are now https://repo.spring.io/snapshot ( snapshot vs libs-snapshot, milestone vs libs-milestone)).

3. use 4.3.0-M3 and put annotation on the entity classes. See(2) for using milestone builds. Release date is November 12.

Edit : is there a way with the SDK to programmatically say "wait all my inserts" ?
I believe scanConsistency REQUEST_PLUS will do this - it works off the time of the request. So that the request does not run until all the inserts that occurred up to the time the request arrived have been indexed.

Regarding the transaction example - kv-inserts are instantaneous - they will be in-order. It's the indexing that is async. If the application needs to wait until all the indexing is complete, any query with scanConsistency(REQUEST_PLUS) will wait until the indexing of all documents inserted at the time of that query is complete.

[JesusTheHun]

1. Ok
2. October 14 is close enough for us. We will use the snapshot version in the meanwhile.
3. How stable is this version atm ?

If the application needs to wait until all the indexing is complete, any query with scanConsistency(REQUEST_PLUS) will wait until the indexing of all documents inserted at the time of that query is complete.

I need to see the documents, for that I need them to be in the index, right ? Using n1ql, not KV.
It's not optimal as I could be processing the message after a big insert, but even in the worst case scenario (for our app) it's not such a big deal. It would be nice to have some kind of "markers" on the documents processed by a tx so you could wait only until the tx documents are indexed.
The workaround if I need maximum speed, would be to include all the doc ids in the broker message, so the event processing function will be able to fetch them with KV. If I need to use n1ql, I would have to fetch manually, add the tx/message documents to the list using KV. It would still not work for aggregation though, that's the limit of the workaround.

[mikereiche]

3. How stable is this version atm ?

4.2.6-SNAPSHOT is currently identical to 4.2.5 release.

"It would be nice to have some kind of "markers" on the documents processed by a tx so you could wait only until the tx documents are indexed."

That's QueryOptions.consistentWith(MutationState), where MutationState is a collection of MutationToken from MutationResults from inserts/upserts/replaces/removes. None of that is exposed in spring data couchbase. (the consistentWith() method on templates is something else).

" but even in the worst case scenario (for our app) it's not such a big deal"
I did a quick test on a very simple case and it took 2.5x as long.
(Also - query took 80x as long as kv. So if you can fetch via kv instead of query with REQUEST_PLUS, you can be 200x faster)

[JesusTheHun]

That's QueryOptions.consistentWith(MutationState), where MutationState is a collection of MutationToken from MutationResults from inserts/upserts/replaces/removes. None of that is exposed in spring data couchbase. (the consistentWith() method on templates is something else).

Wow, this is amazing. Do you plan to expose it in spring data ? How can I use that in the meanwhile ? Why isn't this documented ?
For a small amount of data, sharing those token in a broker message is a solution, but what about thousands of documents, as a result of a batch for example, is it possible by any chance that those tokens have a common root so you only need to share one token ?

[mikereiche]

but what about thousands of documents

I believe just the last one should be sufficient, as REQUEST_PLUS works off the time - and that would only work if they are processed in the order they arrive (I think - not sure).

is it possible by any chance that those tokens have a common root so you only need to share one token ?

To use the feature, all the MutationTokens go in a single MutationState object. You'd still be passing the same amount of data, though.

Do you plan to expose it in spring data ?

The issue is with the spring data model, where an insert/updates return the entity being inserted/updated. There's no room to return the MutationToken. UNLESS... we do something like we do for meta().cas -> @Version annotated field and meta().expiry() -> @expiry annotated field. I don't see anything wrong with returning the MutationToken in an @MutationToken field. To pass the MutationState in to queries, the QueryOptions are exposed to template and repository APIs in 4.3 (November 12).

I created #1245 to investigate.

[JesusTheHun]

I believe just the last one should be sufficient, as REQUEST_PLUS works off the time - and that would only work if they are processed in the order they arrive (I think - not sure).

Can you check with the core team? If that's the case things become a LOT easier!

UNLESS... we do something like we do for meta().cas -> @Version annotated field and meta().expiry() -> @expiry annotated field. I don't see anything wrong with returning the MutationToken in an @MutationToken field.

This would be amazing! Does this still requires EnhanceDurability? I've seen that on the forum. Is it possible to activate that with the latest sdk version?

[mikereiche]

I have a question in about using only the last mutation token (note to self - they refer to the list of mutation tokens as a "scan_vector").

Mutation Tokens (for enhanced durability) are enabled by default in SDK 3.x https://docs.couchbase.com/java-sdk/current/ref/client-settings.html . The comments about enabling them are likely from SDK 2.x

[mikereiche]

I have a question in about using only the last mutation token

Answer: "There is no coordination for mutations across documents". So it's not a possibility. The sequence number in a Mutation Token is per-document.

[JesusTheHun]

Ok! REQUEST_PLUS it will be in case of large messages then :)
Thank you so much for all those information and answers !

[JesusTheHun]

So just to be clear and make a recap, for now it is available only in the SDK. You said earlier it's not like the template consistency, how do I get the SDK from spring?