Migration issues Templates and Dynamic queries

[franquiroz]

Hi, i'm trying to migrate my app from NEST to Elastic.Clients.Elasticsearch, so, i have found some blocking issues

1. i Cant create PutTemplateAsync with Analisis

NEST: await client.Indices.PutTemplateAsync(templateName,t => t.Settings(s => s.Analysis(a => a.Analyzers(an => an
.Custom("ana_s_lc", ca => ca.Tokenizer("standard").Filters("lowercase")))
.Normalizers(nm => nm.Custom("nor_lc", c => c.Filters("lowercase")))
.Tokenizers(t => t.PathHierarchy("tok_path", p => p.Delimiter('\')))
.TokenFilters(t => t.Lowercase("lowercase"))));

Elastic.Clients.Elasticsearch: Settings(Func<FluentDictionary<string, object> ??????????????)

2. In NEST i can create a dynamic QueryContainer like that:

a. queryfilters &= (new QueryContainerDescriptor<FileEvent>().Term(t => t.Field(f => f.AgentUUID).Value(item.Value)));
b. queryfilters &= (new QueryContainerDescriptor<FileEvent>().Term(t => t.Field(f => f.ProxyConfigurationId).Value(Guid.Empty)));
c. queryfilters &= (new QueryContainerDescriptor<FileEvent>().Bool(b => b.MustNot(m => m.Exists(e => e.Field(f => f.FileName)))));

implementing the options (a, b, c) dynamically if some conditions are complied...
and so on, with more complex filters like:

void ElasticFilter(ref QueryContainer queryfilters, CustomFilter item, Expression<Func<T, object>> field) where T : class
{
QueryContainer qc = null;
var allValues = item.Value.Split("||").Where(w => !string.IsNullOrEmpty(w));
switch (item.Filter)
{
case StringFilterType.Contains:
foreach (var wc in allValues.Select(f => new QueryContainerDescriptor().Wildcard(t => t.Field(field).Value("" + f.Replace("", "").Replace("\", "\\").Trim() + ""))))
{
qc |= wc;
}
queryfilters &= new QueryContainerDescriptor().Bool(b => b.Must(m => qc));
break;
case StringFilterType.NotContains:
foreach (var wc in allValues.Select(f => new QueryContainerDescriptor().Wildcard(t => t.Field(field).Value("" + f.Replace("", "").Replace("\", "\\").Trim() + ""))))
{
qc |= wc;
}
queryfilters &= new QueryContainerDescriptor().Bool(b => b.MustNot(m => qc));
break;
case StringFilterType.Equals:
foreach (var wc in allValues.Select(f => new QueryContainerDescriptor().Term(t => t.Field(field).Value(f.Trim()))))
{
qc |= wc;
}
queryfilters &= new QueryContainerDescriptor().Bool(b => b.Must(m => qc));
break;
case StringFilterType.NotEquals:
foreach (var wc in allValues.Select(f => new QueryContainerDescriptor().Term(t => t.Field(field).Value(f.Trim()))))
{
qc |= wc;
}
queryfilters &= new QueryContainerDescriptor().Bool(b => b.MustNot(m => qc));
break;
case StringFilterType.StartWith:
foreach (var wc in allValues.Select(f => new QueryContainerDescriptor().Prefix(t => t.Field(field).Value(f.Trim()))))
{
qc |= wc;
}
queryfilters &= new QueryContainerDescriptor().Bool(b => b.Must(m => qc));
break;
case StringFilterType.NotStartWith:
foreach (var wc in allValues.Select(f => new QueryContainerDescriptor().Prefix(t => t.Field(field).Value(f.Trim()))))
{
qc |= wc;
}
queryfilters &= new QueryContainerDescriptor().Bool(b => b.MustNot(m => qc));
break;

 default:
     break;

}
}

I CAN'T REACH ANY ALTERNATIVE SOLUTION FOR ANY OF MY ISSUES

i hope anyone can help me

THX SO MUCH

[MarkMcDonald]

Been covering this myself and been very reliant on the source code to guide. If it helps though I can share the approach we've been using to refactor existing code.

As far as I can make out the concept of ANDs and ORs you had in the NEST client has been remove - and I actually I think this may be a good thing because I don't think they exist conceptually in the search language.

So below is an example of how we refactored from using the operators to Action delegates so we didn't have to refactor existing if statements to much, same can be done with for statements and sorts and other things.

        private static Action<QueryDescriptor<MySearchModel>> BuildQuery([NotNull] MyRequest request)
        {
            var musts = new List<Action<QueryDescriptor<MySearchModel>>>();
            var mustNots = new List<Action<QueryDescriptor<MySearchModel>>>();
            var shoulds = new List<Action<QueryDescriptor<MySearchModel>>>();

            if (request.TagIds?.Any() ?? false)
            {
                musts
                    .Add(m => m
                        .Terms(t => t
                            .Field(f => f.Tags)
                            .Terms(new TermsQueryField(request.TagIds.Select(x => FieldValue.Long(x)).ToArray()))
                        )
                    );
            }

            if (request.DocumentTypeId != null && request.DocumentTypeId != 0)
            {
                musts
                    .Add(m => m
                        .Term(t => t
                            .Field("docTypeId")
                            .Value(FieldValue.Long(request.DocumentTypeId ?? 0))
                        )
                    );
            }

            mustNots
                .Add(m => m
                    .Term(t => t
                        .Field(f => f.HideInSearchResults)
                        .Value(FieldValue.Boolean(true))
                    )
                );

            if (request.ExcludedDocumentTypeIds != null && (request.ExcludedDocumentTypeIds?.Any() ?? false))
            {
                mustNots
                    .Add(m => m
                        .Terms(t => t
                            .Field("docTypeId")
                            .Terms(new TermsQueryField(request.ExcludedDocumentTypeIds.Select(x => FieldValue.Long(x)).ToArray()))
                        )
                    );
            }
            if (request.ExcludedPageIds != null && (request.ExcludedPageIds?.Any() ?? false))
            {
                mustNots
                    .Add(m => m
                        .Terms(t => t
                            .Field("id")
                            .Terms(new TermsQueryField(request.ExcludedPageIds.Select(x => FieldValue.Long(x)).ToArray()))
                        )
                    );
            }

            if (request.PrimaryTagId.HasValue && request.PrimaryTagId != 0)
            {
                shoulds
                    .Add(s => s
                        .Term(t => t
                            .Field("primaryTagId")
                            .Value(FieldValue.Long(request.PrimaryTagId.Value))
                        )
                    );
            }

            return query => query
                .Bool(b => b
                    .Must(musts.ToArray())
                    .MustNot(mustNots.ToArray())
                    .Should(shoulds.ToArray())
                );
        }

We can then refactor / create the search result descriptor as below:

                Action<SearchRequestDescriptor<MySearchModel>> searchRequest = s => s
                                        .Index("my-index")
                                        .Query(BuildQuery(request))
                                        .Sort(s => s.Field(f => f.UpdatedOn, fs => fs.Order(SortOrder.Desc)))
                                        .From(0)
                                        .Size(20);

Multiple sorting fields can be addressed in a similar manner:

        private static Action<SortOptionsDescriptor<MySearchModel>>[] BuildSort(string sortOption)
        {
            return sortOption.ToLower() switch
            {
                "sort2" => new Action<SortOptionsDescriptor<MySearchModel>>[]
                    {
                        s => s.Field(f => f.MyField1, fs => fs.Order(SortOrder.Asc)),
                        s => s.Field(f => f.MyField2, fs => fs.Order(SortOrder.Desc)),
                    },
                "sort3" => new Action<SortOptionsDescriptor<MySearchModel>>[]
                    {
                        s => s.Field(f => f.MyField3, fs => fs.Order(SortOrder.Desc)),
                        s => s.Score(new ScoreSort() { Order = SortOrder.Desc })
                    },
                _ => new Action<SortOptionsDescriptor<MySearchModel>>[]
                    {
                        s => s.Field(f => f.MyField4, fs => fs.Order(SortOrder.Desc)),
                        s => s.Field(f => f.Dates.FirstOrDefault().StartDate, fs => fs.Order(SortOrder.Asc)),
                        s => s.Field(f => f.Dates.FirstOrDefault().EndDate, fs => fs.Order(SortOrder.Asc)),
                    },
            };
        }

Your Sort line would then become:

                Action<SearchRequestDescriptor<MySearchModel>> searchRequest = s => s
                                        .Index("my-index")
                                        .Query(BuildQuery(request))
                                        .Sort(BuildSort(sortOption))
                                        .From(0)
                                        .Size(20);

I can't vouch if this is 100% the approach that should be taken. The documentation is very thin on client usage, and doesn't seem to be up to date with the source code or NuGet packages. But it is working for us so far.

Based on using upgrading from NEST 7.17.5 to Elastic.Clients.Elasticsearch 8.13.12.

Hope people find this helpful.

[franquiroz]

Hi @MarkMcDonald, Thanks so much for your reply. I understand that my current code does not have a solution in the new DLL, for me it is not viable make a migration because my project is very big and a have a lot of indexes with dynamic querys and nested querys... i'm really disapointed and indignant....

by the way, do you know something about my first question about Analysis templates?

thx again!!

[flobernd]

@franquiroz

Could you please post the complete NEST code for what you are trying to achieve? I want to make sure I fully understand your question.

The && and || operators are currently not implemented for QueryDescriptor (NEST name: QueryContainerDescriptor). The non descriptor equivalent Query (NEST name: QueryContainer) still supports these operations. I will probably re-introduce the operators for the descriptor as well, but this is not on the roadmap right now. In the meantime, you should either switch to using Query or take a look at the above example code from @MarkMcDonald .

@MarkMcDonald Thank you for providing this example code! This is exactly how it should be done in the current client 🙂

[franquiroz]

hi, @flobernd

1. the piece of code is up the post. I cannot implement the Settings => Analisis, because doest exists...

await client.Indices.PutTemplateAsync(templateName,t => t.Settings(s => s.Analysis(a => a.Analyzers(an => an
.Custom("ana_s_lc", ca => ca.Tokenizer("standard").Filters("lowercase")))
.Normalizers(nm => nm.Custom("nor_lc", c => c.Filters("lowercase")))
.Tokenizers(t => t.PathHierarchy("tok_path", p => p.Delimiter('')))
.TokenFilters(t => t.Lowercase("lowercase"))));

3. Ok, I understand, I hope you can re-introduce it, like I said before my project is too big to rethink and redo all the querys from the scratch. i need to make the migration the less painful as possible.

thx so much

[flobernd]

Hi @franquiroz,

it seems like these settings are incorrectly modelled in our specification. I'll have a look and try to find out if this is intended or not.