Is there a replacement for the BulkProcessor?

[tony-hizzle]

With RHLC, one could use the BulkProcessor API to batch IndexRequests and DeleteRequests. Is there a recommended replacement for BulkProcessor when migrating to elasticsearch-java?

https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-docs-bulk-processor.html

[swallez]

There's no equivalent for now, but we're working on a replacement that will land in the near future.

[jerryguowei]

Is there any update on this feature?

[jetpack1116]

Any update on BulkProcessors for the new ES java api?

[jetpack1116]

if there is no equivalent for BulkProcessor in the new elasticsearch java api client, how will we migrate a BulkProcessor code?
Are the new elasticsearch java api client BulkRequest, BulkOperation, BulkResponse object enough?

Is there a sample just to point to the right direction?

[divadpoc]

any update would be much appreciated @swallez

[swallez]

Sorry, no update yet. I'll raise this internally as a topic that needs to be prioritized.

[antondollmaier]

Hi folks - thanks for the already open issue.

We've recently migrated from the now deprecated ES7 client to the new API Client in v8 and "stumbled" the very hard way on bulk requests.
ElasticSearch rejects our bulk requests due to max_content_length exceeded - which is plausible, as the bulk() client is now not doing any batching.

Instead of having the "old" way (as well as the way of the clients in, e.g., Python), where the ES client is splitting the bulk request in smaller chunks based on (byte)size or document count, we now have to reside to manually splitting the data ourselves. Seems a step backward to me.

Is there a way to speed up the development of a "new" BulkProcessor, aside from "implement it yourselves"? ("buy Enterprise support", "send us chocolate", "talk to and bribe a member of the Evangelist team", ...)

Thanks! :)

Best,
Anton

[sat245]

Do we have any deadline on update of this issue ?
Any documentation coming soon ?

[frank-montyne]

Any reason why this is taking so long? Why even introduce a new API without this core component for ingestion? This issue should be at the top of the list of things to solve in my opinion. An update and preferably timeline would be greatly appreciated.

Thanks and best regards,
Frank