Map index name to document

[mirco-romagnoli]

We're using a date index name processor to point a document to the right index based on a document's date field value.
After saving the document we need to get it again from Elasticsearch to read some fields that are filled by an ingestion pipeline.
The issue is that after the save we don't know the index name where the document has been saved.
Could it be possible to add a @Index annotation that maps the document's index name on the Java field?

[sothawo]

Not clear what mean with

a date index name processor to point a document to the right index

How are these index names constructed? Can't you search with a wildcard or index aliases?

[mirco-romagnoli]

The processor points the document to an index with the structure yyyy-MM-dd, the date is determined by the value inside a field given as a parameter to the processor.
The issue here is that I have to directly get the document instead of searching for it, otherwise I have to wait for the translog to be flushed at each indexing request, which is not ideal.

[sothawo]

even if it would be possible to write the index name to the document during writing, you would not have it back in the application. And even if so - if storing an entity in index-a and returning the information as result of the save or storing it in the entity on save would work - how would you later have this information to get the entity? That would mean to store the combination of entity id and index somewhere else.

And how would an @Index annotation help? What should be done with that?

[mirco-romagnoli]

even if it would be possible to write the index name to the document during writing, you would not have it back in the application.

I have an ingestion pipeline associated to an index, when we put a document into that index instead of being written on that specific index it gets routed to another index based on the date value contained in the said document. I've successfully done that using the RestHighLevelClient and the actual index where the document has been written is included in the response.

That would mean to store the combination of entity id and index somewhere else.

Yes, and that's where an @Index annotation would be useful, the "_index" field of the response should be written in the annotated field of the mapped model.
For instance this is what I'm actually doing with a RestHighLevelClient to add a Model object to index my-index:

Model model = new Model();
Map<String, Object> map = objectMapper.convertValue(model, new TypeReference<>() {});
IndexRequest request = new IndexRequest("my-index");
request.source(map);
IndexResponse response = restHighLevelClient.index(request, RequestOptions.DEFAULT);
String index = response.getIndex();
String id = response.getId();

index value is not "my-index", instead is "my-index-yyyy-MM-dd" where the date is based on a field contained in the Model object.

By introducing an @Index annotation I could add an index field to the Model class, annotate it and index the document by doing the following:

Model model = new Model();
Model saved = elasticsearchOperations.save(model, IndexCoordinates.of("my-index"));

After that I expect that calling saved.getIndex() should return the same value contained in the index variable of the previous example.

[sothawo]

ok, I got it. Although I am not sure if @Index would be the right name, it might lead to the misconception that one could use this field to specify where the entity should be stored, we'll probably need a different name for that.

[mirco-romagnoli]

Yes, I agree with you. Maybe @ResponseIndex or @RetrievedIndex could be better?