[5.1] Returned values of runtime fields are null for every field except the last one

[shahzebklp]

I am adding runtime fields to my NativeQuery object like this:

public void addRuntimeFieldsToQuery(NativeQuery query) {
  String field1 = "runtimeField1";
  String field2 = "runtimeField2";

  query.addRuntimeField(new RuntimeField(field1, "keyword", getScript(field1)));
  query.addRuntimeField(new RuntimeField(field2, "keyword", getScript(field2)));

  query.addFields(field1, field2);
}

When I do the query I can see in my debugger that both fields and runtimeFields lists are populated just like I expect them to be in the query. However, when I get the search result only the last field's, i.e. field2, value is non-null – the rest are null. If I change the order of the last line to be query.addFields(field2, field1), then only field1's value is non-null.

This started happening after upgrading to Spring Boot 3 and the new client libraries

import org.springframework.data.elasticsearch.client.elc.NativeQuery;
import co.elastic.clients.elasticsearch._types.query_dsl.*;

I am also using import org.springframework.data.elasticsearch.core.ElasticsearchOperations for doing the search.
I am on Spring Boot version 3.1.0 and spring-data-elasticsearch 5.1.0.

At this point I am suspecting something in the spring api is behaving differently than what it used to in Spring Boot 2.x. I have tested this for both SB2 and SB3 and Elasticsearch 7.17.7 and 8.10.3, and this issue is only showing up in SB3.

[sothawo]

I cannot reproduce that. I just added this test to org.springframework.data.elasticsearch.core.query.scriptedandruntimefields.ScriptedAndRuntimeFieldsIntegrationTests

	@Test // #2727
	@DisplayName("should return runtime fields values")
	void shouldReturnRuntimeFieldsValues() {

		var entity = new SAREntity();
		entity.setId("42");
		entity.setValue(3);

		repository.save(entity);

		var runtimeField1 = getRuntimeField("scriptedValue1", 2);
		var runtimeField2 = getRuntimeField("scriptedValue2", 3);

		var query = CriteriaQuery.builder(Criteria.where("value").is(3)).build();
		query.addRuntimeField(runtimeField1);
		query.addRuntimeField(runtimeField2);
		query.addSourceFilter(new FetchSourceFilterBuilder().withIncludes("*").build());
		query.addFields("scriptedValue1", "scriptedValue2");
		var searchHits = operations.search(query, SAREntity.class);

		assertThat(searchHits.getTotalHits()).isEqualTo(1);
		var foundEntity = searchHits.getSearchHit(0).getContent();
		assertThat(foundEntity.value).isEqualTo(3);
		assertThat(foundEntity.getScriptedValue1()).isEqualTo(6);
		assertThat(foundEntity.getScriptedValue2()).isEqualTo(9);
	}

and no matter in which order I add the values in addFields, it works.

The entity is defined as:

	@Document(indexName = "#{@indexNameProvider.indexName()}-sar")
	public static class SAREntity {
		@Nullable private String id;
		@Field(type = FieldType.Integer)
		@Nullable Integer value;
		@ScriptedField
		@Nullable Integer scriptedValue1;
		@ScriptedField
		@Nullable Integer scriptedValue2;
                // getter and setter omitted
}

the method creating the runtime fields:

private static RuntimeField getRuntimeField(String fieldName, int factor) {
		return new RuntimeField(
				fieldName,
				"long",
				String.format("emit(doc['value'].size() > 0 ? doc['value'].value * %d : 0)", factor));
	}

The request body sent to elElasticsearch:

{
	"_source": {
		"excludes": [],
		"includes": [
			"*"
		]
	},
	"fields": [
		{
			"field": "scriptedValue1"
		},
		{
			"field": "scriptedValue2"
		}
	],
	"from": 0,
	"query": {
		"bool": {
			"must": [
				{
					"query_string": {
						"default_operator": "and",
						"fields": [
							"value"
						],
						"query": "3"
					}
				}
			]
		}
	},
	"runtime_mappings": {
		"scriptedValue1": {
			"script": {
				"source": "emit(doc['value'].size() > 0 ? doc['value'].value * 2 : 0)"
			},
			"type": "long"
		},
		"scriptedValue2": {
			"script": {
				"source": "emit(doc['value'].size() > 0 ? doc['value'].value * 3 : 0)"
			},
			"type": "long"
		}
	},
	"size": 10,
	"track_scores": false,
	"version": true
}

the response:

{
	"took": 352,
	"timed_out": false,
	"_shards": {
		"total": 1,
		"successful": 1,
		"skipped": 0,
		"failed": 0
	},
	"hits": {
		"total": {
			"value": 1,
			"relation": "eq"
		},
		"max_score": 1.0,
		"hits": [
			{
				"_index": "scripted-runtime-1-sar",
				"_id": "42",
				"_version": 1,
				"_score": 1.0,
				"_source": {
					"_class": "org.springframework.data.elasticsearch.core.query.scriptedandruntimefields.ScriptedAndRuntimeFieldsIntegrationTests$SAREntity",
					"id": "42",
					"value": 3
				},
				"fields": {
					"scriptedValue1": [
						6
					],
					"scriptedValue2": [
						9
					]
				}
			}
		]
	}
}

[sothawo]

Just had another look, this is a bug in the 5.1 branch, for the upcoming 5.2 version this was fixed with #2663

[shahzebklp]

I see you added this to the 5.1.6 milestone, does that mean the fix will be part of that release?