Cannot retrieve map that contains BigInteger using SDC.

[mmonti]

Hi!

I am running into an issue when i try to retrieve a document that contains a key with a BigInteger type value.
The error I am getting is a bit missleading since the document is stored correctly but fail to retrieve:

java.lang.IllegalArgumentException: Attribute of type java.math.BigInteger cannot be stored and must be converted.

This seems to be a limitation only on the SDC sdk since I tried to store and retrieve the document with the Couchbase Driver class (Cluster) and I was able to do it without any issues.

I did some debugging in the code and it looks like there is check in the CouchbaseDocument.java class (line 126) that uses CouchbaseSimpleTypes to test if the type of the value that is beign loaded matches certain types: CouchbaseDocument.class, CouchbaseList.class.

 private void verifyValueType(final Object value) {
        if (value != null) {
            Class<?> clazz = value.getClass();
            if (!CouchbaseSimpleTypes.DOCUMENT_TYPES.isSimpleType(clazz)) {
                throw new IllegalArgumentException("Attribute of type " + clazz.getCanonicalName() + " cannot be stored and must be converted.");
            }
        }
    }

I created a repository with a couple test to repro the issue: https://github.com/mmonti/sdc-biginteger-issue/blob/master/src/test/java/com/example/biginteger/BigIntegerIssue.java

I am using spring-data-couchbase 4.4.5 in the repro project, but i can confirm that the issue is also present in spring-data-couchbase 3.1.18.

I just wanted to know if this is a known issue and if there is any possible workaround for it.
Thanks!

[mikereiche]

The work-around would be to save BigIntegers as strings when using the java sdk directly.

Edit: Although the following is mostly correct, numbers greater than 53 bits will have the lower-bits truncated when retrieved through a n1ql query. Thus the 12345678901234567890 used in the example will be returned as 12345678901234567000 from a n1ql query. It is better to store BigIntegers as strings (SDC does this automatically).

It's odd that this hasn't come up earlier. The issue stems from org.springframework.data.mapping.model omitting BigInteger (it also omits BigDecimal) - and CouchbaseSimpleTypes does not add it.
The ObjectMapper used by the couchbase java SDK de/serializes BigInteger as a json number (just digits, no quotes). The mapper used by spring-data-couchbase serializes BigInteger as a string (digits surrounded by quotes), and puts it into a CouchbaseDocument that gets written to the db. When it is read from the db, the value is put() into a CouchbaseDocument as a String, and then deserialized from the CouchbaseDocument into the entity as a BigInteger.

A better solution would be to add NumberToBigInteger/BigIntegerToNumber converters to OtherConverters - but this would make data written by the "fixed" SDC incompatible for reading with an un-fixed version SDC.

I've re-worked the provided test cases to use the BigIntegerEntity from MappingCouchbaseConverterTests.java.

	private BigIntegerEntity bigIntegerEntity = new BigIntegerEntity("1", new BigInteger("12345678901234567890"));
	static class BigIntegerEntity  {
		private final String id;
		private final BigInteger big;

		public BigIntegerEntity(String id, BigInteger big) {
			this.id=id;
			this.big = big;
		}
	}

	Map<String,Object> map = Map.of("big", bigIntegerEntity.big);

	@BeforeEach
	public void setUp() {
		var savedNestedMap = couchbaseTemplate.getCollection("_default").upsert(bigIntegerEntity.id, map);
		log.info("savedNestedMap = {}", savedNestedMap.mutationToken());
	}

	@Test
	void test_store_and_retrieve_using_couchbase_driver_directly() {
		var bucket = couchbaseTemplate.getCouchbaseClientFactory().getBucket();
		String id="1";
		Map<String,Object> map = Map.of("big", new BigInteger("12345678901234567890"));
		var savedNestedMap = couchbaseTemplate.getCollection("_default").upsert(id, map);
		final GetResult resultMap = bucket.defaultCollection().get(id);
		final BigInteger bigInteger = (BigInteger) resultMap.contentAs(Map.class).get("big");
		log.info("Big integer value = {}", bigInteger);
		assertEquals(map.get("big"), bigInteger);
		QueryResult queryResult = couchbaseTemplate.getCouchbaseClientFactory().getCluster().query(
				"SELECT `"+bucketName()+"`.big from `"+bucketName()+"` where meta().id=\""+bigIntegerEntity.id+"\"",
										QueryOptions.queryOptions().scanConsistency(QueryScanConsistency.REQUEST_PLUS));
		for(JsonObject bigJson: queryResult.rowsAsObject()){
			log.info("big: "+bigJson.get("big"));
			assertEquals(map.get("big"), bigJson.getNumber("big")); // <<<< WILL FAIL
		}

	@Test
	void test_store_and_retrieve_using_spring_data_couchbase_directly() {
		var savedNestedMap = couchbaseTemplate.save(bigIntegerEntity);

		final var result = couchbaseTemplate.findById(BigIntegerEntity.class).one(bigIntegerEntity.id);
		log.info("Big integer value = {}", result.big);
		assertEquals(bigIntegerEntity.big, result.big);
	}
	@Test
	void test_fail_on_retrieve_saved_map_with_big_integer() {
		var retrievedMap = couchbaseTemplate.findById(BigIntegerEntity.class).one(bigIntegerEntity.id);
		log.info("Big integer value = {}", retrievedMap.big);
	}

	@Test
	void test_success_on_retrieve_saved_map_when_loading_custom_couchbase_simple_type() {
		// 1.- Before running, uncomment CouchbaseSimpleTypes.java in the test folder.

		var retrievedMap = couchbaseTemplate.findById(BigIntegerEntity.class).one(bigIntegerEntity.id);
		log.info("Big integer value = {}", retrievedMap.big);
		assertEquals(bigIntegerEntity.big, retrievedMap.big);
	}

	@Test
	void test_fail_on_retrieve_nested_map_with_big_integer_key() {
		// 1.- Before running, uncomment CouchbaseSimpleTypes.java in the test folder.

		// Try to retrieve the nested map.
		var retrievedMap = couchbaseTemplate.findById(BigIntegerEntity.class).one(bigIntegerEntity.id);
		log.info("Big integer value = {}", (retrievedMap.big));
		assertEquals(bigIntegerEntity.big, retrievedMap.big);
	}

The error I am getting is a bit missleading since the document is stored correctly but fail to retrieve

Yes. But on retrieval it is being stored (put) in a CouchbaseDocument, which is decoded into the entity.

[vforchi]

Are there any news on this issue? I have the same problem, and I don't understand if there is a way forward.

A better solution would be to add NumberToBigInteger/BigIntegerToNumber converters to OtherConverters - but this would make data written by the "fixed" SDC incompatible for reading with an un-fixed version SDC.

I don't think this is an argument against adding these converters: as long as large numbers in a document can be mapped to BigDecimal/BigInteger, I think SDC should handle them. They could come from any source, not necessarily SDC.

[mikereiche]

SDC serialize/deserialezes BigIntegers as strings because Integers larger than 2^64 do not fit in longs. What do you want the behavior to be? Fail on Bigintegers larger than 2^64 ?

[vforchi]

I don't understand: is it doing it already or not? Because in your first comment you said this would be a workaround.

[mikereiche]

I have the same problem,

is it doing it already or not?

If you have the "same problem", then it is not doing "it".

This work-around is a work-around. The java sdk does not do this. The application using the java sdk would need to do this. It's up to the application on how to implement that.

I don't understand if there is a way forward.

What do you want the "way forward" to be? Given that there already documents with BigIntegers written as integer numbers in documents by the Java SDK, and also documents with BigIntegers written as strings by SDC, and that any "way forward" cannot break anything that already works.

I think SDC should handle them

You can provide your own customer converters to handle any conversion. Do you have converters to propose that wouldn't break on existing documents?

I don't think this is an argument against adding these converters

Breaking existing functionality is indeed an argument.

[vforchi]

I understand your point, but how can I read a document created with another interface, which puts the numbers in as they are? Is there a way to support both formats?

[mikereiche]

Including Number as a valid simple type for DOCUMENT_TYPES will result in BigInteger and BigDecimal (and any Number) to be written (and read) in the form of a number (no quotes). I don't know why this was not allowed from the beginning.

CouchbaseSimpleTypes.java:

	public static final SimpleTypeHolder DOCUMENT_TYPES = new SimpleTypeHolder(
			Stream.of(CouchbaseDocument.class, CouchbaseList.class,  Number.class).collect(toSet()), true);

This also solves the issue of BigIntegers and BigDecimals that have already been written as Strings. The only problem is that that using query truncates (rounds-down) over 63 bits down to 53 bits. https://issues.couchbase.com/browse/MB-24464