[Serverless]: Details on what settings and system indices (if any) are available on Serverless

[ppf2]

Serverless Docs

Elasticsearch

Description

https://www.elastic.co/docs/current/serverless/elasticsearch/differences

On this page, we made the statement:

Certain APIs, API parameters, index, cluster and node level settings are not available. Refer to our REST API references for a list of available APIs.

The provided link is only about APIs. Where are the details on which specific "index, cluster and node level settings" can be modified on Serverless? 😄 If a setting is available, what is the hard default on Serverless (esp. if it's not modifiable so that users will know what the default behavior is)? Also, it will be helpful to document what system indices (if any) users can still see on Serverless.

One way to approach this is to consolidate Serverless and non-Serverless API and setting guides and use tagging to inform the users if a specific setting/API call is available on all deployment types vs. not on Serverless, etc..

Resources and additional context

https://www.elastic.co/docs/current/serverless/elasticsearch/differences

[shainaraskas]

related: https://github.com/elastic/platform-docs-team/issues/395

[kilfoyle]

For settings docs specifically, converting the source to YAML is still very much a work in progress, and we've started with Kibana settings for the PoC. But, we do have this draft schema that allows tagging for deployment types, default values, etc.

[maggieghamry]

Thanks for creating this @ppf2 ! I would request the priority bumped on this issue if possible as there is currently no way for folks migrating from on-prem or cloud to Serverless to have a comprehensive understanding of what settings may not be available in the managed environment. IMHO we need a Serverless setting lookup to search for any setting and find out if it's supported in Serverless (and if not, if there is a potential for ER or if they are blocked intentionally for x purpose), and have that visible for all end users. 🙏

[lcawl]

IMHO we need a Serverless setting lookup to search for any setting and find out if it's supported in Serverless (and if not, if there is a potential for ER or if they are blocked intentionally for x purpose), and have that visible for all end users

I think a helpful first step would be to consider whether the schema David mentioned above would be sufficient for Elasticsearch settings as well or whether there are additional metadata that our users will need. For example I think all the fields like applies_to, datatype, default, description (as seen in https://github.com/elastic/kibana/blob/main/docs/settings-gen/source/kibana-alert-action-settings.yml for example) are applicable. Perhaps some way of indicating whether the setting is dynamic or static and whether it's an index or cluster or data stream-level setting? Once we have consensus on what all the important facets are the next step would be to follow up on where/who will be involved in filling in all that missing information.

[jakommo]

Just want to add that we should make sure such future page shows up if a user searched for Settings [setting.name] are not available when running in serverless mode.

[kunisen]

FWIW, I also raised an internal discussion on a tangentially but strongly related topic - https://github.com/elastic/sdh-elasticsearch/issues/9039#issuecomment-2964851474 with ES dev friend (@john-wagster).

Key point is what @jakommo says, we want to have a generic page covering why some settings are designed / marked as not available on serverless (not specific to one or two settings, but in general), and link this page to API response to show up the doc.

---

To be clear, solving the stemmed topic in #79 (comment) and this comment, doesn't solve the original one - they are two different things.

---

Pius, if you think we should separate this topic out, I will file a new one specifically for this generic page topic.

[ppf2]

As part of this work (once we have tagged all the settings that are available/unavailable), let's also update the differences page to provide instructions on how users can tell which setting is available/unavailable. Thx!