Skip to content

for bulk operations, don't raise an exception if any of the ingested items fails #99

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 5, 2022

Conversation

philvarner
Copy link
Collaborator

Related Issue(s):

  • n/a

Description:

The default behavior of the bulk api (which streams the individual operations in batches) is to immediately stop and raise an exception if any of the operations in a batch fail. Instead, don't raise on that error, and just keep ingesting the items. This is so that if there's a bad item that's being ingested, it doesn't prevent the others in the batch from being ingested.

TBD is better reporting back to the caller as to which items succeeded and failed.

PR Checklist:

  • Code is formatted and linted (run pre-commit run --all-files)
  • Tests pass (run make test)
  • Documentation has been updated to reflect changes, if applicable, and docs build successfully (run make docs)
  • Changes are added to the changelog

@philvarner philvarner requested a review from gadomski May 4, 2022 18:50
Copy link
Member

@gadomski gadomski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth making this an optional argument to bulk_async in case someone else wants different behavior?

@philvarner
Copy link
Collaborator Author

Is it worth making this an optional argument to bulk_async in case someone else wants different behavior?

Yes, but not as part of this work -- right now, it's just blowing up because the exception is getting thrown, but there should really be a more formalized response as part of the Transactions spec, similarly to how ES handles it by returning a result for each document that's indexed.

@philvarner philvarner merged commit c0f875d into main May 5, 2022
@jonhealy1 jonhealy1 deleted the pv/bulk-error-ignore branch January 14, 2023 07:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants