Index aware document writes

[costin]

Currently elasticsearch-hadoop supports only document creation (id is always generated by ES).

This area needs to be improved to support the following strategies:

1. create/put-if-absent (applies also when there's no id)
2. index op-type=index or create document if missing, replace if present
3. upsert or create document if missing, _update if present

[vvakar]

+1 for upsert support.

To add some color around that, let's say we're tracking sales data for electronic devices: there's a data stream from brick-and-mortar stores, another stream from web sales, etc - we would like to accumulate all that data as it becomes available under each device's own document in ES. For example:
{
"device": "iPhone",
"sales_data": {
"January": {"brick_and_mortar": 305, "website": 298 },
"February": { "brick_and_mortar": 225, "website": 168 },
"March": {...}
}

That way we can process various aspects of the data using a decoupled SOA architecture and trust that it all gets merged under its corresponding document in ES. In the above example, getting the most up-to-date sales information is a matter of looking up the document in ES and aggregation/facets can be built accordingly.