Bucket may not be opened when applications starts

[edudar]

I noticed this while working with the integration tests. @SpringBootTest + @Testcontainers. Some builds on CI were failing with timeout exception when trying to do repository.save() and CB error was

om.couchbase.client.core.error.AmbiguousTimeoutException: UpsertRequest, Reason: TIMEOUT
{"cancelled":true,"completed":true,"coreId":"0x3b57a1700000003","idempotent":false
,"reason":"TIMEOUT","requestId":161,"requestType":"UpsertRequest","retried":19
,"retryReasons":["BUCKET_NOT_AVAILABLE"], ...

I've added buckets printout right before save operation like this:

var couchbaseClientFactory = context.getApplicationContext().getBean(CouchbaseClientFactory.class);
couchbaseClientFactory.getCluster().buckets().getAllBuckets()
    .forEach((bucket, settings) -> log.info("Test Bucket: [{}]; Settings: [{}]; Healthy: [{}]", bucket, settings, settings.healthy()));

and it showed that the bucket is there and is healthy. But save() fails.

Following through the code from the client factory down to where the bucket opens, it looks like opening a bucket is an async operation that is not guaranteed to finish (or rather guaranteed not to finish) when the method exits:

SimpleCouchbaseClientFactory:
this.bucket = cluster.get().bucket(bucketName);

Cluster:
return bucketCache.computeIfAbsent(bucketName, n -> new Bucket(asyncCluster.bucket(n)));

AsyncCluster:
return bucketCache.computeIfAbsent(bucketName, n -> {
  core.openBucket(n);
  return new AsyncBucket(n, core, environment.get());
});

Core:
configurationProvider
  .openBucket(name)
  .subscribe(v -> {}, t -> {}, () -> eventBus.publish(new BucketOpenedEvent(Duration.ofNanos(System.nanoTime() - start), coreContext, name)));

.subscribe is async here so the client factory may very well get a bucket instance that has not been opened yet.

I tried couchbaseClientFactory.getBucket().waitUntilReady(...) but it does not seem to work here and I don't see any code in waitUntilReady() that would ensure that bucket is open.

A workaround I came up with is the following:

couchbaseClientFactory.getCluster().core().configurationProvider().openBucket(BUCKET_NAME).block(WAIT_TIME);

It's not very nice and I block on Mono but it works for tests where context starts in less than a second and I need to make sure bucket is available when test cases start executing.

It'd be good to ensure in SimpleCouchbaseClientFactory that the bucket is indeed opened when context starts.

[mikereiche]

I tried couchbaseClientFactory.getBucket().waitUntilReady(...)

This waits for the cluster to reach a specific state. The method is on Bucket for historical reasons - which is misleading.
Blocking on openBucket() as you have done is probably the best that you can do. I'm sorry you had to arrive at that on your own.

[mikereiche]

There is a change to the Java SDK that should address this - JVMCBC-950

[mikereiche]

They can add this to their class that extends AbstractCouchbaseConfiguration.
it calls waitUntilReady(). The fix in to make it more resilient to newly-created buckets JVMCBC-950 (not sure this is involved in their case), is in core-io/2.1.4 (which goes with java-client/3.1.4)
it calls openBucket() with a wait.
Then it tries a 'get' on the bucket in a loop.

/**
 * Override couchbaseClientFactory of AbstractCouchbaseConfiguration. After creating the client factory, it calls
 * waitUntilRead() and openBucket() to ensure the bucket is ready to use before returning
 */
@Bean
@Override
public CouchbaseClientFactory couchbaseClientFactory(final Cluster couchbaseCluster) {
	CouchbaseClientFactory factory = super.couchbaseClientFactory(couchbaseCluster);
	factory.getCluster().waitUntilReady(Duration.ofMinutes(1));
	factory.getCluster().core().configurationProvider().openBucket(getBucketName()).block(Duration.ofMinutes(1));
	tryGet(factory, 60);
	return factory;
}
private void tryGet(CouchbaseClientFactory factory, int maxAttempts) {
	int attempts = 0;
	do {
		attempts++;
		try {
			factory.getCluster().bucket(getBucketName()).defaultCollection().get("x");
		} catch (Exception e) {
			if (e instanceof com.couchbase.client.core.error.DocumentNotFoundException) {
				break;
			}
			try {
				Thread.sleep(1000);
			} catch (InterruptedException interruptedException) {}
		}
	} while (attempts < 60);
}