[DE-793] Missing jackson-dataformat-cbor dependency when deploying in AWS MSK

[jordanauge]

Hello team,

When deploying the released jar file as a custom plugin in AWS MSK, we get the following error

[Worker-0565c2ad5fa4d38cd] [2024-03-18 19:26:32,973] INFO [kafka-connect-arangodb|task-1] task config: com.arangodb.kafka.config.ArangoSinkConfig@17a7f69d (com.arangodb.kafka.ArangoSinkTask:48)
[Worker-0565c2ad5fa4d38cd] [2024-03-18 19:26:32,975] ERROR [kafka-connect-arangodb|task-1] WorkerSinkTask{id=kafka-connect-arangodb-1} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:191)
[Worker-0565c2ad5fa4d38cd] java.util.ServiceConfigurationError: com.fasterxml.jackson.core.JsonFactory: Provider com.fasterxml.jackson.dataformat.cbor.CBORFactory not found
[Worker-0565c2ad5fa4d38cd]  at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:589)
[Worker-0565c2ad5fa4d38cd]  at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1212)
[Worker-0565c2ad5fa4d38cd]  at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1221)
[Worker-0565c2ad5fa4d38cd]  at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265)
[Worker-0565c2ad5fa4d38cd]  at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300)
[Worker-0565c2ad5fa4d38cd]  at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385)
[Worker-0565c2ad5fa4d38cd]  at com.arangodb.serde.jackson.internal.JacksonMapperProvider.of(JacksonMapperProvider.java:30)
[Worker-0565c2ad5fa4d38cd]  at com.arangodb.serde.jackson.JacksonSerde.of(JacksonSerde.java:23)
[Worker-0565c2ad5fa4d38cd]  at com.arangodb.serde.jackson.json.JacksonJsonSerdeProvider.create(JacksonJsonSerdeProvider.java:11)
[Worker-0565c2ad5fa4d38cd]  at com.arangodb.internal.config.ArangoConfig.getUserDataSerde(ArangoConfig.java:253)
[Worker-0565c2ad5fa4d38cd]  at com.arangodb.internal.config.ArangoConfig.getInternalSerde(ArangoConfig.java:260)
[Worker-0565c2ad5fa4d38cd]  at com.arangodb.internal.net.Communication.<init>(Communication.java:34)
[Worker-0565c2ad5fa4d38cd]  at com.arangodb.http.HttpCommunication.<init>(HttpCommunication.java:37)
[Worker-0565c2ad5fa4d38cd]  at com.arangodb.http.HttpProtocolProvider.createProtocol(HttpProtocolProvider.java:28)
[Worker-0565c2ad5fa4d38cd]  at com.arangodb.ArangoDB$Builder.build(ArangoDB.java:374)
[Worker-0565c2ad5fa4d38cd]  at com.arangodb.kafka.config.ArangoSinkConfig.createCollection(ArangoSinkConfig.java:727)
[Worker-0565c2ad5fa4d38cd]  at com.arangodb.kafka.ArangoSinkTask.start(ArangoSinkTask.java:49)
[Worker-0565c2ad5fa4d38cd]  at org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:308)
[Worker-0565c2ad5fa4d38cd]  at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:196)
[Worker-0565c2ad5fa4d38cd]  at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:189)
[Worker-0565c2ad5fa4d38cd]  at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:238)
[Worker-0565c2ad5fa4d38cd]  at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[Worker-0565c2ad5fa4d38cd]  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[Worker-0565c2ad5fa4d38cd]  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[Worker-0565c2ad5fa4d38cd]  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[Worker-0565c2ad5fa4d38cd]  at java.base/java.lang.Thread.run(Thread.java:829)
[Worker-0565c2ad5fa4d38cd] [2024-03-18 19:26:32,975] INFO [kafka-connect-arangodb|task-1] stopping ArangoSinkTask (com.arangodb.kafka.ArangoSinkTask:64)

Adding the following dependency in the pom.xml file, and rebuilding with mvn clean package -P standalone solved the issue:

        <dependency>
            <groupId>com.fasterxml.jackson.dataformat</groupId>
            <artifactId>jackson-dataformat-cbor</artifactId>
        </dependency>

Furthermore it seems recommended to add versioning to dependencies, in that case, we might go for 2.17.0 as found in https://mvnrepository.com/artifact/com.fasterxml.jackson.dataformat/jackson-dataformat-cbor

Cheers

[rashtao]

This error is not caused by kafka-connect-arangodb, but by some wrong SPI mapping present in your classpath. You might want to open an issue in the related library causing this.

Additionally, we could implement a more tolerant approach to load SPI providers in the ArangoDB Java driver, i.e. catching ServiceConfigurationError.

[rashtao]

The next release of ArangoDB Java driver will be tolerant wrt these errors, see arangodb/arangodb-java-driver#552

[rashtao]

Closing as clarified, please reopen in case of further problems.