Merge branch 'cocoindex-io:main' into expr-union-type-impl

Author: chardoncs

docs/docs/core/basics.md

@@ -1,17 +1,17 @@
 ---
-title: Basics
-description: "CocoIndex basic concepts: indexing flow, data, operations, data updates, etc."
+title: Indexing Basics
+description: "CocoIndex basic concepts for indexing: indexing flow, data, operations, data updates, etc."
 ---
 
-# CocoIndex Basics
+# CocoIndex Indexing Basics
 
 An **index** is a collection of data stored in a way that is easy for retrieval.
 
-CocoIndex is an ETL framework for building indexes from specified data sources, a.k.a. indexing. It also offers utilities for users to retrieve data from the indexes.
+CocoIndex is an ETL framework for building indexes from specified data sources, a.k.a. **indexing**. It also offers utilities for users to retrieve data from the indexes.
 
-## Indexing flow
+An **indexing flow** extracts data from specified data sources, upon specified transformations, and puts the transformed data into specified storage for later retrieval.
 
-An indexing flow extracts data from specified data sources, upon specified transformations, and puts the transformed data into specified storage for later retrieval.
+## Indexing flow elements
 
 An indexing flow has two aspects: data and operations on data.
 
@@ -42,7 +42,7 @@ An **operation** in an indexing flow defines a step in the flow. An operation is
 
 "import" and "transform" operations produce output data, whose data type is determined based on the operation spec and data types of input data (for "transform" operation only).
 
-### Example
+## An indexing flow example
 
 For the example shown in the [Quickstart](../getting_started/quickstart) section, the indexing flow is as follows:
 
@@ -60,7 +60,7 @@ This shows schema and example data for the indexing flow:
 
 ![Data Example](data_example.svg)
 
-### Life cycle of an indexing flow
+## Life cycle of an indexing flow
 
 An indexing flow, once set up, maintains a long-lived relationship between data source and data in target storage. This means:
 
@@ -95,19 +95,10 @@ CocoIndex works the same way, but with more powerful capabilities:
 
 This means when writing your flow operations, you can treat source data as if it were static - focusing purely on defining the transformation logic. CocoIndex takes care of maintaining the dynamic relationship between sources and target data behind the scenes.
 
-### Internal storage
+## Internal storage
 
 As an indexing flow is long-lived, it needs to store intermediate data to keep track of the states.
 CocoIndex uses internal storage for this purpose.
 
 Currently, CocoIndex uses Postgres database as the internal storage.
-See [Initialization](initialization) for configuring its location, and `cocoindex setup` CLI command (see [CocoIndex CLI](cli)) creates tables for the internal storage.
-
-## Retrieval
-
-There are two ways to retrieve data from target storage built by an indexing flow:
-
-*   Query the underlying target storage directly for maximum flexibility.
-*   Use CocoIndex *query handlers* for a more convenient experience with built-in tooling support (e.g. CocoInsight) to understand query performance against the target data.
-
-Query handlers are tied to specific indexing flows. They accept query inputs, transform them by defined operations, and retrieve matching data from the target storage that was created by the flow.
+See [Initialization](initialization) for configuring its location, and `cocoindex setup` CLI command (see [CocoIndex CLI](cli)) creates tables for the internal storage.

docs/docs/core/flow_def.mdx

@@ -1,7 +1,6 @@
 ---
 title: Flow Definition
 description: Define a CocoIndex flow, by specifying source, transformations and storages, and connect input/output data of them.
-toc_max_heading_level: 4
 ---
 
 import Tabs from '@theme/Tabs';

docs/docs/getting_started/quickstart.md

@@ -132,7 +132,7 @@ if __name__ == "__main__":
 
 The `@cocoindex.main_fn` declares a function as the main function for an indexing application. This achieves the following effects:
 
-*   Initialize the CocoIndex librart states. Settings (e.g. database URL) are loaded from environment variables by default.
+*   Initialize the CocoIndex library states. Settings (e.g. database URL) are loaded from environment variables by default.
 *   When the CLI is invoked with `cocoindex` subcommand, `cocoindex CLI` takes over the control, which provides convenient ways to manage the index. See the next step for more details.
 
 ## Step 3: Run the indexing pipeline and queries

docs/docs/query.mdx

@@ -0,0 +1,96 @@
+---
+title: Query Support
+description: CocoIndex supports vector search and text search.
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# CocoIndex Query Support
+
+The main functionality of CocoIndex is indexing.
+The goal of indexing is to enable efficient querying against your data.
+You can use any libraries or frameworks of your choice to perform queries.
+At the same time, CocoIndex provides seamless integration between indexing and querying workflows.
+For example, you can share transformations between indexing and querying, and easily retrieve table names when using CocoIndex's default naming conventions.
+
+## Transform Flow
+
+Sometimes a part of the transformation logic needs to be shared between indexing and querying,
+e.g. when we build a vector index and query against it, the embedding computation needs to be consistent between indexing and querying.
+
+In this case, you can:
+
+1.  Extract a sub-flow with the shared transformation logic into a standalone function.
+    *   It takes one or more data slices as input.
+    *   It returns one data slice as output.
+    *   You need to annotate data types for both inputs and outputs as type parameter for `cocoindex.DataSlice[T]`. See [data types](./core/data_types.mdx) for more details about supported data types.
+
+2.  When you're defining your indexing flow, you can directly call the function.
+    The body will be executed, so that the transformation logic will be added as part of the indexing flow.
+
+3.  At query time, you usually want to directly run the function with specific input data, instead of letting it called as part of a long-lived indexing flow.
+    To do this, declare the function as a *transform flow*, by decorating it with `@cocoindex.transform_flow()`.
+    This will add a `eval()` method to the function, so that you can directly call with specific input data.
+
+
+<Tabs>
+<TabItem value="python" label="Python">
+
+The [quickstart](getting_started/quickstart#step-41-extract-common-transformations) shows an example:
+
+```python
+@cocoindex.transform_flow()
+def text_to_embedding(text: cocoindex.DataSlice[str]) -> cocoindex.DataSlice[list[float]]:
+    return text.transform(
+        cocoindex.functions.SentenceTransformerEmbed(
+            model="sentence-transformers/all-MiniLM-L6-v2"))
+```
+
+When you're defining your indexing flow, you can directly call the function:
+
+```python
+with doc["chunks"].row() as chunk:
+    chunk["embedding"] = text_to_embedding(chunk["text"])
+```
+
+or, using the `call()` method of the transform flow on the first argument, to make operations chainable:
+
+```python
+with doc["chunks"].row() as chunk:
+    chunk["embedding"] = chunk["text"].call(text_to_embedding)
+```
+
+Any time, you can call the `eval()` method with specific string, which will return a `list[float]`:
+
+```python
+print(text_to_embedding.eval("Hello, world!"))
+```
+
+</TabItem>
+</Tabs>
+
+## Get Target Native Names
+
+In your indexing flow, when you export data to a target, you can specify the target name (e.g. a database table name, a collection name, the node label in property graph databases, etc.) explicitly,
+or for some backends you can also omit it and let CocoIndex generate a default name for you.
+For the latter case, CocoIndex provides a utility function `cocoindex.utils.get_target_storage_default_name()` to get the default name.
+It takes the following arguments:
+
+*   `flow` (type: `cocoindex.Flow`): The flow to get the default name for.
+*   `target_name` (type: `str`): The export target name, appeared in the `export()` call.
+
+For example:
+
+<Tabs>
+<TabItem value="python" label="Python">
+
+```python
+table_name = cocoindex.utils.get_target_storage_default_name(text_embedding_flow, "doc_embeddings")
+query = f"SELECT filename, text FROM {table_name} ORDER BY embedding <=> %s::vector DESC LIMIT 5"
+...
+```
+
+</TabItem>
+</Tabs>
+

docs/sidebars.ts

@@ -44,6 +44,11 @@ const sidebars: SidebarsConfig = {
         'ai/llm',
       ],
     },
+    {
+      type: 'doc',
+      id: 'query',
+      label: 'Query Support',
+    },
     {
       type: 'category',
       label: 'About',

examples/image_search_example/main.py

@@ -94,7 +94,7 @@ def image_object_embedding_flow(flow_builder: cocoindex.FlowBuilder, data_scope:
 # --- CocoIndex initialization on startup ---
 @app.on_event("startup")
 def startup_event():
-    settings = cocoindex.setting.Settings.from_env()
+    settings = cocoindex.Settings.from_env()
     cocoindex.init(settings)
     app.state.query_handler = cocoindex.query.SimpleSemanticsQueryHandler(
         name="ImageObjectSearch",

examples/product_recommendation/main.py

@@ -61,7 +61,7 @@ class ProductTaxonomyInfo:
     complementary_taxonomies: list[ProductTaxonomy]
 
 @cocoindex.op.function(behavior_version=2)
-def extract_product_info(product: cocoindex.typing.Json, filename: str) -> ProductInfo:
+def extract_product_info(product: cocoindex.Json, filename: str) -> ProductInfo:
     # Print  markdown for LLM to extract the taxonomy and complimentary taxonomy
     return ProductInfo(
         id=f"{filename.removesuffix('.json')}",

python/cocoindex/__init__.py

@@ -2,14 +2,15 @@
 Cocoindex is a framework for building and running indexing pipelines.
 """
 from . import functions, query, sources, storages, cli, utils
-from .flow import FlowBuilder, DataScope, DataSlice, Flow, flow_def, transform_flow
+
+from .auth_registry import AuthEntryReference, add_auth_entry, ref_auth_entry
+from .flow import FlowBuilder, DataScope, DataSlice, Flow, transform_flow
+from .flow import flow_def, flow_def as flow
 from .flow import EvaluateAndDumpOptions, GeneratedField
 from .flow import update_all_flows_async, FlowLiveUpdater, FlowLiveUpdaterOptions
+from .lib import init, start_server, stop, main_fn
 from .llm import LlmSpec, LlmApiType
 from .index import VectorSimilarityMetric, VectorIndexDef, IndexOptions
-from .auth_registry import AuthEntryReference, add_auth_entry, ref_auth_entry
-from .lib import *
 from .setting import DatabaseConnectionSpec, Settings, ServerSettings
 from .setting import get_app_namespace
-from ._engine import OpArgSchema
 from .typing import Float32, Float64, LocalDateTime, OffsetDateTime, Range, Vector, Json

python/cocoindex/flow.py

@@ -718,9 +718,11 @@ async def _build_flow_info_async(self) -> TransformFlowInfo:
         for (param_name, param), param_type in zip(sig.parameters.items(), self._flow_arg_types):
             if param.kind not in (inspect.Parameter.POSITIONAL_OR_KEYWORD,
                                   inspect.Parameter.KEYWORD_ONLY):
-                raise ValueError(f"Parameter {param_name} is not a parameter can be passed by name")
-            engine_ds = flow_builder_state.engine_flow_builder.add_direct_input(
-                param_name, encode_enriched_type(param_type))
+                raise ValueError(f"Parameter `{param_name}` is not a parameter can be passed by name")
+            encoded_type = encode_enriched_type(param_type)
+            if encoded_type is None:
+                raise ValueError(f"Parameter `{param_name}` has no type annotation")
+            engine_ds = flow_builder_state.engine_flow_builder.add_direct_input(param_name, encoded_type)
             kwargs[param_name] = DataSlice(_DataSliceState(flow_builder_state, engine_ds))
 
         output = self._flow_fn(**kwargs)
@@ -780,8 +782,13 @@ def _transform_flow_wrapper(fn: Callable[..., DataSlice[T]]):
         for (param_name, param) in sig.parameters.items():
             if param.kind not in (inspect.Parameter.POSITIONAL_OR_KEYWORD,
                                   inspect.Parameter.KEYWORD_ONLY):
-                raise ValueError(f"Parameter {param_name} is not a parameter can be passed by name")
-            arg_types.append(_get_data_slice_annotation_type(param.annotation))
+                raise ValueError(f"Parameter `{param_name}` is not a parameter can be passed by name")
+            value_type_annotation = _get_data_slice_annotation_type(param.annotation)
+            if value_type_annotation is None:
+                raise ValueError(
+                    f"Parameter `{param_name}` for {fn} has no value type annotation. "
+                    "Please use `cocoindex.DataSlice[T]` where T is the type of the value.")
+            arg_types.append(value_type_annotation)
 
         _transform_flow = TransformFlow(fn, arg_types)
         functools.update_wrapper(_transform_flow, fn)