Implement enhancement for ArangoCollection.importDocuments to batch in threads

[rkhaja]

The ArangoDB-Java-Driver should have bulk import methods that can take as parameters, batchSize and threads, to achieve similar functionality as the arangoimport command line utility that has options:

• --batch-size <uint64> size for individual data batches (in bytes) (default: 1048576)
• --threads <uint32> Number of parallel import threads. Most useful for the rocksdb engine (default: 2)

[joerg84]

Just curious: What would be the use case to use that from java rather than arangoimport?
Is that a frequent operation triggered from your application?

[rkhaja]

The Java driver already has the ability to perform bulk imports which is documented here: https://www.arangodb.com/docs/stable/drivers/java-reference-collection-bulk-import.html

The current method signatures in the ArangoDB-Java-Driver are the following:

• ArangoCollection.importDocuments(Collection<?> values, DocumentImportOptions options) : DocumentImportEntity

• ArangoCollection.importDocuments(String values, DocumentImportOptions options) : DocumentImportEntity

Where values is:

• Collection<?>: A list of Objects that will be stored as documents
• String: JSON-encoded array of objects that will be stored as documents

As I mentioned, the current API methods are missing the ability to partition values into smaller batches and run in multiple threads. Further, if the values is too big to be sent in a single request, it would fail.

What would be the use case to use that from java rather than arangoimport?

Usage of arangoimport to bulk load data, means that the data has to be serialized on disk in a format that can be read by arangoimport prior to bulk import. No in memory manipulations are possible.

Usage of the the Java API methods allows the ability to manipulate the input data or values programmatically in memory prior to bulk import, so provides a convenient functionality.

Is that a frequent operation triggered from your application?

Yes. I am/will be refreshing/reloading/replacing approximately 25 billion edge collection documents per day.

[rkhaja]

Here is a Gist for ArangoCollection.importDocuments to batch in threads.

[hkernbach]

Closing this now. PR merged (#276).