Tweak search toc, add apis and tools page, move stuff (#604)

Closes elastic/developer-docs-team#264

- move near-real-time-search.md to manage-data as it's es internal
- Add apis and tools page
- Tweak search toc
- Update applies to stuff
- Other minor cosmetic stuff

Author: leemthompo

deploy-manage/tools/snapshot-and-restore.md

@@ -106,7 +106,7 @@ In Elasticsearch 8.0 and later versions, feature states are the only way to back
 
 ## How snapshots work
 
-Snapshots are **automatically deduplicated** to save storage space and reduce network transfer costs. To back up an index, a snapshot makes a copy of the index’s [segments](/solutions/search/search-approaches/near-real-time-search.md) and stores them in the snapshot repository. Since segments are immutable, the snapshot only needs to copy any new segments created since the repository’s last snapshot.
+Snapshots are **automatically deduplicated** to save storage space and reduce network transfer costs. To back up an index, a snapshot makes a copy of the index’s [segments](/manage-data/data-store/near-real-time-search.md) and stores them in the snapshot repository. Since segments are immutable, the snapshot only needs to copy any new segments created since the repository’s last snapshot.
 
 Each snapshot is **logically independent**. When you delete a snapshot, Elasticsearch only deletes the segments used exclusively by that snapshot. Elasticsearch doesn’t delete segments used by other snapshots in the repository.
 

explore-analyze/query-filter/tools.md

@@ -8,16 +8,16 @@ mapped_pages:
 
 # Query tools [devtools-kibana]
 
-Elasticsearch offers tools that you can use to query your data, manage those queries, and optimize them to be as efficient as possible.
+Access these specialized tools in Kibana and the Serverless UI to develop, debug, and refine your search queries while monitoring their performance and efficiency.
 
-|     |     |
-| --- | --- |
-| [Saved queries](tools/saved-queries.md)  |  Save your searches and queries to reuse them later.   |
+| Tool | Function |
+|------|----------|
+| [Saved queries](tools/saved-queries.md) | Save your searches and queries to reuse them later. |
 | [Console](tools/console.md) | Interact with the REST APIs of {{es}} and {{kib}}, including sending requests and viewing API documentation. |
 | [{{searchprofiler}}](tools/search-profiler.md) | Inspect and analyze your search queries. |
-| [Grok Debugger   ](tools/grok-debugger.md) | Build and debug grok patterns before you use them in your data processing pipelines. |
-| [Painless Lab](../scripting/painless-lab.md) | [beta] Test and debug Painless scripts in real-time. |
-| [Playground](tools/playground.md)   |  Combine your Elasticsearch data with the power of large language models (LLMs) for retrieval augmented generation (RAG), using a chat interface.   |
+| [Grok Debugger](tools/grok-debugger.md) | Build and debug grok patterns before you use them in your data processing pipelines. |
+| [Painless Lab](../scripting/painless-lab.md) | [beta] Test and debug Painless scripts in real-time. |
+| [Playground](tools/playground.md) | Combine your Elasticsearch data with the power of large language models (LLMs) for retrieval augmented generation (RAG), using a chat interface. |
 
 
 

solutions/search/search-approaches/near-real-time-search.md renamed to manage-data/data-store/near-real-time-search.md

@@ -3,7 +3,6 @@ mapped_pages:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/near-real-time.html
 applies_to:
   stack:
-  serverless:
 ---
 
 # Near real-time search [near-real-time]
@@ -14,15 +13,15 @@ Lucene, the Java libraries on which {{es}} is based, introduced the concept of p
 
 Sitting between {{es}} and the disk is the filesystem cache. Documents in the in-memory indexing buffer ([Figure 1](#img-pre-refresh)) are written to a new segment ([Figure 2](#img-post-refresh)). The new segment is written to the filesystem cache first (which is cheap) and only later is it flushed to disk (which is expensive). However, after a file is in the cache, it can be opened and read just like any other file.
 
-:::{image} ../../../images/elasticsearch-reference-lucene-in-memory-buffer.png
+:::{image} /images/elasticsearch-reference-lucene-in-memory-buffer.png
 :alt: A Lucene index with new documents in the in-memory buffer
 :title: A Lucene index with new documents in the in-memory buffer
 :name: img-pre-refresh
 :::
 
 Lucene allows new segments to be written and opened, making the documents they contain visible to search ​without performing a full commit. This is a much lighter process than a commit to disk, and can be done frequently without degrading performance.
 
-:::{image} ../../../images/elasticsearch-reference-lucene-written-not-committed.png
+:::{image} /images/elasticsearch-reference-lucene-written-not-committed.png
 :alt: The buffer contents are written to a segment, which is searchable, but is not yet committed
 :title: The buffer contents are written to a segment, which is searchable, but is not yet committed
 :name: img-post-refresh

manage-data/toc.yml

@@ -4,6 +4,7 @@ toc:
   - file: data-store.md
     children:
       - file: data-store/index-basics.md
+      - file: data-store/near-real-time-search.md
       - file: data-store/data-streams.md
         children:
           - file: data-store/data-streams/set-up-data-stream.md

raw-migrated-files/docs-content/serverless/elasticsearch-dev-tools.md

@@ -62,7 +62,7 @@ In {{es}} 8.0 and later versions, feature states are the only way to back up and
 
 ## How snapshots work [how-snapshots-work]
 
-Snapshots are automatically deduplicated to save storage space and reduce network transfer costs. To back up an index, a snapshot makes a copy of the index’s [segments](../../../solutions/search/search-approaches/near-real-time-search.md) and stores them in the snapshot repository. Since segments are immutable, the snapshot only needs to copy any new segments created since the repository’s last snapshot.
+Snapshots are automatically deduplicated to save storage space and reduce network transfer costs. To back up an index, a snapshot makes a copy of the index’s [segments](../../../manage-data/data-store/near-real-time-search.md) and stores them in the snapshot repository. Since segments are immutable, the snapshot only needs to copy any new segments created since the repository’s last snapshot.
 
 Each snapshot is also logically independent. When you delete a snapshot, {{es}} only deletes the segments used exclusively by that snapshot. {{es}} doesn’t delete segments used by other snapshots in the repository.
 

raw-migrated-files/elasticsearch/elasticsearch-reference/snapshot-restore.md

@@ -160,7 +160,6 @@ toc:
       - file: docs-content/serverless/connect-to-byo-llm.md
       - file: docs-content/serverless/cspm-required-permissions.md
       - file: docs-content/serverless/detections-logsdb-index-mode-impact.md
-      - file: docs-content/serverless/elasticsearch-dev-tools.md
       - file: docs-content/serverless/elasticsearch-differences.md
       - file: docs-content/serverless/elasticsearch-explore-your-data.md
       - file: docs-content/serverless/elasticsearch-http-apis.md

raw-migrated-files/toc.yml

@@ -1,4 +1,5 @@
 redirects:
+  'solutions/search/search-approaches/near-real-time-search.md': '!manage-data/data-store/near-real-time-search.md'
   'get-started/installing-elastic-stack.md': '!deploy-manage/deploy/self-managed.md'
   'deploy-manage/deploy/elastic-cloud/ec-configure-deployment-settings.md': '!deploy-manage/deploy/elastic-cloud/ec-customize-deployment-components.md'
   'deploy-manage/users-roles/cluster-or-deployment-auth/user-authentication.md':

redirects.yml

@@ -9,10 +9,67 @@ applies_to:
 
 # APIs and tools
 
-% What needs to be done: Write from scratch
+This page is handy list of the most important APIs and tools you need to build, test, and manage your search app built with {{es}}.
 
-% Use migrated content from existing pages that map to this page:
+## API endpoints
 
-% - [ ] ./raw-migrated-files/docs-content/serverless/elasticsearch-dev-tools.md
-% - [ ] https://www.elastic.co/guide/en/enterprise-search/current/search-ui.html
-%      Notes: mostly redirect purposes, this page wasn't migrated, but you can pull from the live URL if needed.
+### Query & search APIs
+
+| Endpoint | Function |
+|----------|----------|
+| [`_search`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/group/endpoint-search) | Searches and aggregations written in [Query DSL](/explore-analyze/query-filter/languages/querydsl.md) and [retrievers](retrievers-overview.md) syntax |
+| [`_query`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/group/endpoint-esql)| Endpoint for [{{esql}}](/explore-analyze/query-filter/languages/esql.md) queries |
+| [`_explain`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-explain) | Provides detailed explanation of how a specific document matches a query with scoring breakdown |
+| [`_count`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-count) | Returns count of documents matching a query without retrieving results |
+| [`_validate/query`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-indices-validate-query) | Validates query syntax without executing the search |
+| [`_analyze`](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-analyze.html) | Performs analysis for [full-text search](./full-text.md) on a text string and returns the resulting tokens. |
+
+### Ingestion & mapping APIs
+
+| Endpoint | Function |
+|----------|----------|
+| [`_mapping`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-indices-get-field-mapping) | Retrieves or updates field mappings with options for specific field inspection |
+| [`_reindex`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-reindex) | Copies documents from one index to another, useful for mapping changes |
+| [`_update_by_query`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-update-by-query) | Updates documents matching a query without reindexing |
+| [`_bulk`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-bulk-1) | Performs multiple index/update/delete operations in a single request |
+| [`_refresh`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-indices-refresh-3) | Forces a refresh to make recent operations searchable |
+| [`_ingest/pipeline`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/group/endpoint-ingest) | Creates and manages document processing pipelines before indexing |
+
+### Search optimization APIs
+
+| Endpoint | Function |
+|----------|----------|
+| [`_rank_eval`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-rank-eval)| Evaluates search quality against known relevant documents |
+| [`_settings`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-indices-get-settings-1) | Configures settings including slow logs, refresh intervals, and replicas (only index-level settings available in serverless) |
+| [`_scripts`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/group/endpoint-script) | Creates or updates stored scripts for reuse in queries and aggregations |
+
+
+## UI tools [elasticsearch-dev-tools]
+
+### Dev tools
+
+Access these specialized tools in Kibana and the Serverless UI to develop, debug, and refine your search queries while monitoring their performance and efficiency.
+
+These tools are documented in the **Explore & Analyze** section:
+
+| Tool | Function |
+|------|----------|
+| [Saved queries](/explore-analyze/query-filter/tools/saved-queries.md) | Save your searches and queries to reuse them later. |
+| [Console](/explore-analyze/query-filter/tools/console.md) | Interact with the REST APIs of {{es}} and {{kib}}, including sending requests and viewing API documentation. |
+| [{{searchprofiler}}](/explore-analyze/query-filter/tools/search-profiler.md) | Inspect and analyze your search queries. |
+| [Grok Debugger](/explore-analyze/query-filter/tools/grok-debugger.md) | Build and debug grok patterns before you use them in your data processing pipelines. |
+| [Painless Lab](/explore-analyze/scripting/painless-lab.md) | Test and debug Painless scripts in real-time. |
+
+### Playground
+
+Use [Playground](rag/playground.md) to combine your {{es}} data with the power of large language models (LLMs) for retrieval augmented generation (RAG), using a chat interface. Playground is also very useful for testing and debugging your {{es}} queries, using the [retrievers](retrievers-overview.md) syntax with the `_search` endpoint.
+
+### Search UI
+
+[Elastic Search UI](./site-or-app/search-ui.md) is a library of JavaScript and React tools for building search experiences, optimized for use with {{es}}.
+
+:::{tip}
+Check out the Elasticsearch Labs [blog](https://www.elastic.co/search-labs) to learn how to use Elastic to build advanced search experiences including generative AI, embedding models, reranking capabilities and more.
+
+The accompanying [GitHub repository](https://www.github.com/elastic/elasticsearch-labs) contains hands-on Python notebooks and sample apps to help you get started with these advanced search features.
+:::

solutions/search/apis-and-tools.md

@@ -3,12 +3,10 @@ navigation_title: "Search Applications"
 mapped_pages:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/search-application-overview.html
 applies_to:
-  stack:
-  serverless:
+  stack: beta
+  serverless: beta
 ---
 
-
-
 # Search applications [search-application-overview]
 
 
@@ -32,14 +30,15 @@ The Search Applications feature was introduced in Elastic version **8.8.0**.
 
 ::::{note} 
 Search Applications is a beta feature. Beta features are subject to change and are not covered by the support SLA of general release (GA) features. Elastic plans to promote this feature to GA in a future release.
-
 ::::
 
 
-This feature is available to all **Elastic Cloud** deployments.
+This feature is available to all **{{ech}}** deployments.
 
 This feature is also available to **self-managed** deployments when Elastic subscription requirements are satisfied. View the requirements for this feature under the **Elastic Search** section of the [Elastic Stack subscriptions](https://www.elastic.co/subscriptions) page.
 
+For Serverless users, this is an API-only feature. You can create and manage search applications using the [Search Application APIs](https://www.elastic.co/docs/api/doc/elasticsearch-serverless/group/endpoint-search_application).
+
 Your deployment must include the {{es}} and {{kib}} services.
 
 Managing search applications requires the `manage_search_application` cluster privilege, and also requires the `manage` [index privilege](../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices) on all indices associated with the search application.
@@ -57,6 +56,10 @@ Search Applications use [search templates](search-templates.md) to simplify the
 
 ### Option 1: Get started in the UI [search-application-overview-get-started-ui] 
 
+```{applies_to}
+serverless: unavailable
+```
+
 You can create build, and manage your search applications directly in the {{kib}} UI under **Search**. Make sure you have at least one {{es}} index to work with on your deployment. The indices underlying your search application are searched together, similar to how an [alias](../../manage-data/data-store/aliases.md) searches over multiple indices.
 
 To create a new search application in {{kib}}:

solutions/search/search-applications.md

@@ -3,8 +3,8 @@ navigation_title: "Search API and templates"
 mapped_pages:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/search-application-api.html
 applies_to:
-  stack:
-  serverless:
+  stack: beta
+  serverless: beta
 ---
 
 

solutions/search/search-applications/search-application-api.md

@@ -3,8 +3,8 @@ navigation_title: "Search Application client guide"
 mapped_pages:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/search-application-client.html
 applies_to:
-  stack:
-  serverless:
+  stack: beta
+  serverless: beta
 ---
 
 

solutions/search/search-applications/search-application-client.md

@@ -3,8 +3,8 @@ navigation_title: "Security"
 mapped_pages:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/search-application-security.html
 applies_to:
-  stack:
-  serverless:
+  stack: beta
+  serverless: beta
 ---
 
 

solutions/search/search-applications/search-application-security.md

@@ -644,6 +644,9 @@ toc:
             children:
               - file: search/ranking/learning-to-rank-model-training.md
               - file: search/ranking/learning-to-rank-search-usage.md
+      - file: search/cross-cluster-search.md
+        children:
+          - file: search/using-resolve-cluster-endpoint-before-cross-cluster-search.md
       - file: search/querying-for-search.md
         children:
           - file: search/the-search-api.md
@@ -661,8 +664,4 @@ toc:
               - file: search/search-applications/search-application-api.md
               - file: search/search-applications/search-application-security.md
               - file: search/search-applications/search-application-client.md
-      - file: search/apis-and-tools.md
-      - file: search/search-approaches/near-real-time-search.md
-      - file: search/cross-cluster-search.md
-        children:
-          - file: search/using-resolve-cluster-endpoint-before-cross-cluster-search.md
+      - file: search/apis-and-tools.md