Description
We allow for patterns to be used in our resource objects so that document fields can inform which index to be written to at runtime. These index patterns usually require a different approach to how the connector runs. For instance, index patterns should only be used when auto creating indices is allowed. If index creation is disabled, we check in the job configuration stage to see if the index exists. We have no data to expand a pattern at this point, and so the pattern is treated like a plain index name. Another such example would be in the Spark SQL integration - If a patterned index name is targeted for an overwrite operation, the underlying documents are not removed before indexing occurs because once again there is no data to expand the index pattern, thus it is treated as a plain index name. I'm sure there are other edge cases in the project.
Some things that might be good to consider for the project:
- It would be great if there was a way to detect when we're trying to use an unexpanded index pattern in place of a plain index.
- We would benefit from more validation of index names against the other settings in the connector (like auto creation)