Skip to content

Commit b95291e

Browse files
rhysrevans3rhysrevans3
committed
Merge branch 'main' of github.com:rhysrevans3/stac-fastapi-elasticsearch into extend_datetime_search
2 parents 4cf5f47 + 125baff commit b95291e

File tree

18 files changed

+437
-139
lines changed

18 files changed

+437
-139
lines changed

‎.github/workflows/publish.yml

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
name: Publish
2+
3+
on:
4+
push:
5+
tags:
6+
- "v*.*.*" # Triggers when a tag like 'v3.2.0' is pushed
7+
8+
jobs:
9+
build-and-publish-pypi:
10+
name: Build and Publish Packages
11+
runs-on: ubuntu-latest
12+
13+
steps:
14+
- uses: actions/checkout@v4
15+
16+
- name: Set up Python 3.10
17+
uses: actions/setup-python@v5
18+
with:
19+
python-version: "3.10"
20+
21+
- name: Install build dependencies
22+
run: |
23+
python -m pip install --upgrade pip
24+
pip install setuptools wheel twine
25+
26+
- name: Build and publish stac-fastapi-core
27+
working-directory: stac_fastapi/core
28+
env:
29+
TWINE_USERNAME: "__token__"
30+
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
31+
run: |
32+
# Build package
33+
python setup.py sdist bdist_wheel
34+
35+
# Publish to PyPI
36+
twine upload dist/*
37+
38+
- name: Build and publish stac-fastapi-elasticsearch
39+
working-directory: stac_fastapi/elasticsearch
40+
env:
41+
TWINE_USERNAME: "__token__"
42+
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
43+
run: |
44+
# Build package
45+
python setup.py sdist bdist_wheel
46+
47+
# Publish to PyPI
48+
twine upload dist/*
49+
50+
- name: Build and publish stac-fastapi-opensearch
51+
working-directory: stac_fastapi/opensearch
52+
env:
53+
TWINE_USERNAME: "__token__"
54+
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
55+
run: |
56+
# Build package
57+
python setup.py sdist bdist_wheel
58+
59+
# Publish to PyPI
60+
twine upload dist/*
61+
62+
build-and-push-images:
63+
name: Build and Push Docker Images
64+
runs-on: ubuntu-latest
65+
permissions:
66+
contents: read
67+
packages: write
68+
69+
steps:
70+
- name: Checkout repository
71+
uses: actions/checkout@v4
72+
73+
- name: Set up QEMU
74+
uses: docker/setup-qemu-action@v3
75+
76+
- name: Set up Docker Buildx
77+
uses: docker/setup-buildx-action@v3
78+
79+
- name: Log in to GitHub Container Registry
80+
uses: docker/login-action@v3
81+
with:
82+
registry: ghcr.io
83+
username: ${{ github.actor }}
84+
password: ${{ secrets.GITHUB_TOKEN }}
85+
86+
- name: Extract metadata for Elasticsearch image
87+
id: meta-es
88+
uses: docker/metadata-action@v5
89+
with:
90+
images: ghcr.io/${{ github.repository_owner }}/stac-fastapi-es
91+
tags: |
92+
type=raw,value=latest
93+
type=ref,event=tag
94+
95+
- name: Push Elasticsearch image
96+
uses: docker/build-push-action@v6
97+
with:
98+
context: .
99+
file: dockerfiles/Dockerfile.ci.es
100+
platforms: linux/amd64,linux/arm64
101+
push: true
102+
tags: ${{ steps.meta-es.outputs.tags }}
103+
labels: ${{ steps.meta-es.outputs.labels }}
104+
cache-from: type=gha
105+
cache-to: type=gha,mode=max
106+
107+
- name: Extract metadata for OpenSearch image
108+
id: meta-os
109+
uses: docker/metadata-action@v5
110+
with:
111+
images: ghcr.io/${{ github.repository_owner }}/stac-fastapi-os
112+
tags: |
113+
type=raw,value=latest
114+
type=ref,event=tag
115+
116+
- name: Push OpenSearch image
117+
uses: docker/build-push-action@v6
118+
with:
119+
context: .
120+
file: dockerfiles/Dockerfile.ci.os
121+
platforms: linux/amd64,linux/arm64
122+
push: true
123+
tags: ${{ steps.meta-os.outputs.tags }}
124+
labels: ${{ steps.meta-os.outputs.labels }}
125+
cache-from: type=gha
126+
cache-to: type=gha,mode=max

‎CHANGELOG.md

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,36 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
77

88
## [Unreleased]
99

10+
### Changed
11+
12+
- Added note on the use of the default `*` use in route authentication dependecies. [#325](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/325)
13+
- Bugfixes for the `IsNull` operator and datetime filtering [#330](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/330)
14+
15+
## [v3.2.2] - 2024-12-15
16+
17+
### Changed
18+
19+
- Use base64 encoded JSON string of sort keys as pagination token instead of comma-separated string [#323](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/323)
20+
21+
## [v3.2.1] - 2024-11-14
22+
23+
### Added
24+
25+
- Added `dockerfiles/Dockerfile.ci.os` and `dockerfiles/Dockerfile.ci.es`, along with their respective entrypoints [#311](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/311)
26+
27+
### Changed
28+
29+
- Updated the `publish.yml` workflow to include Docker image publishing to GitHub Container Registry [#311](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/311)
30+
- Improved the README with detailed descriptions of the new Docker images, providing guidance for images. [#311](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/311)
31+
- Aggregation ElasticSearch `total_count` bugfix, moved aggregation text to docs. [#314](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/314)
32+
33+
## [v3.2.0] - 2024-10-09
34+
1035
### Added
1136

1237
- Added `datetime_frequency_interval` parameter for `datetime_frequency` aggregation. [#294](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/294)
1338
- Added rate limiting functionality with configurable limits using environment variable `STAC_FASTAPI_RATE_LIMIT`, example: `500/minute`. [#303](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/303)
39+
- Added publish.yml to automatically publish new releases to PyPI [#305](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/305)
1440

1541
### Changed
1642

@@ -263,7 +289,10 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
263289
- Use genexp in execute_search and get_all_collections to return results.
264290
- Added db_to_stac serializer to item_collection method in core.py.
265291

266-
[Unreleased]: https://github.com/stac-utils/stac-fastapi-elasticsearch/tree/v3.1.0...main
292+
[Unreleased]: https://github.com/stac-utils/stac-fastapi-elasticsearch/tree/v3.2.2...main
293+
[v3.2.2]: https://github.com/stac-utils/stac-fastapi-elasticsearch/tree/v3.2.1...v3.2.2
294+
[v3.2.1]: https://github.com/stac-utils/stac-fastapi-elasticsearch/tree/v3.2.0...v3.2.1
295+
[v3.2.0]: https://github.com/stac-utils/stac-fastapi-elasticsearch/tree/v3.1.0...v3.2.0
267296
[v3.1.0]: https://github.com/stac-utils/stac-fastapi-elasticsearch/tree/v3.0.0...v3.1.0
268297
[v3.0.0]: https://github.com/stac-utils/stac-fastapi-elasticsearch/tree/v2.4.1...v3.0.0
269298
[v2.4.1]: https://github.com/stac-utils/stac-fastapi-elasticsearch/tree/v2.4.0...v2.4.1

‎README.md

Lines changed: 42 additions & 108 deletions
Original file line numberDiff line numberDiff line change
@@ -43,17 +43,24 @@ or
4343
pip install stac_fastapi.opensearch
4444
```
4545

46-
## Build Elasticsearch API backend
46+
### To install and run via pre-built Docker Images
47+
48+
We provide ready-to-use Docker images through GitHub Container Registry ([ElasticSearch](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pkgs/container/stac-fastapi-es) and [OpenSearch](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pkgs/container/stac-fastapi-os) backends). You can easily pull and run these images:
4749

4850
```shell
49-
docker-compose up elasticsearch
50-
docker-compose build app-elasticsearch
51+
# For Elasticsearch backend
52+
docker pull ghcr.io/stac-utils/stac-fastapi-es:latest
53+
54+
# For OpenSearch backend
55+
docker pull ghcr.io/stac-utils/stac-fastapi-os:latest
5156
```
5257

53-
## Running Elasticsearch API on localhost:8080
58+
## Run Elasticsearch API backend on localhost:8080
59+
60+
You need to ensure [**Docker Compose**](https://docs.docker.com/compose/install/) or [**Podman Compose**](https://podman-desktop.io/docs/compose) installed and running on your machine. In the follwoing command instead of `docker-compose` you can use `podman-compose` as well.
5461

5562
```shell
56-
docker-compose up app-elasticsearch
63+
docker-compose up elasticsearch app-elasticsearch
5764
```
5865

5966
By default, docker-compose uses Elasticsearch 8.x and OpenSearch 2.11.1.
@@ -66,6 +73,35 @@ OPENSEARCH_VERSION=2.11.0
6673
```
6774
The most recent Elasticsearch 7.x versions should also work. See the [opensearch-py docs](https://github.com/opensearch-project/opensearch-py/blob/main/COMPATIBILITY.md) for compatibility information.
6875

76+
#### **Configuration reference keys:**
77+
78+
You can customize additional settings in your `.env` file:
79+
###### Key variables to configure:
80+
81+
| Variable | Description | Default | Required |
82+
|------------------------------|--------------------------------------------------------------------------------------|--------------------------|---------------------------------------------------------------------------------------------|
83+
| `ES_HOST` | Hostname for external Elasticsearch/OpenSearch. | `localhost` | Optional |
84+
| `ES_PORT` | Port for Elasticsearch/OpenSearch. | `9200` (ES) / `9202` (OS)| Optional |
85+
| `ES_USE_SSL` | Use SSL for connecting to Elasticsearch/OpenSearch. | `false` | Optional |
86+
| `ES_VERIFY_CERTS` | Verify SSL certificates when connecting. | `false` | Optional |
87+
| `STAC_FASTAPI_TITLE` | Title of the API in the documentation. | `stac-fastapi-elasticsearch` or `stac-fastapi-opensearch` | Optional |
88+
| `STAC_FASTAPI_DESCRIPTION` | Description of the API in the documentation. | N/A | Optional |
89+
| `STAC_FASTAPI_VERSION` | API version. | `2.1` | Optional |
90+
| `APP_HOST` | Server bind address. | `0.0.0.0` | Optional |
91+
| `APP_PORT` | Server port. | `8080` | Optional |
92+
| `ENVIRONMENT` | Runtime environment. | `local` | Optional |
93+
| `WEB_CONCURRENCY` | Number of worker processes. | `10` | Optional |
94+
| `RELOAD` | Enable auto-reload for development. | `true` | Optional |
95+
| `STAC_FASTAPI_RATE_LIMIT` | API rate limit per client. | `200/minute` | Optional |
96+
| `BACKEND` | Tests-related variable | `elasticsearch` or `opensearch` based on the backend | Optional |
97+
| `ELASTICSEARCH_VERSION` | ElasticSearch version | `7.17.1` | Optional |
98+
| `OPENSEARCH_VERSION` | OpenSearch version | `2.11.0` | Optional |
99+
100+
> [!NOTE]
101+
> The variables `ES_HOST`, `ES_PORT`, `ES_USE_SSL`, and `ES_VERIFY_CERTS` apply to both Elasticsearch and OpenSearch backends, so there is no need to rename the key names to `OS_` even if you're using OpenSearch.
102+
103+
## Interacting with the API
104+
69105
To create a new Collection:
70106

71107
```shell
@@ -279,111 +315,9 @@ The modified Items with lowercase identifiers will now be visible to users acces
279315

280316
Authentication is an optional feature that can be enabled through `Route Dependencies` examples can be found and a more detailed explanation in [examples/auth](examples/auth).
281317

282-
283318
## Aggregation
284319

285-
Sfeos supports the STAC API [Aggregation Extension](https://github.com/stac-api-extensions/aggregation). This enables geospatial aggregation of points and geometries, as well as frequency distribution aggregation of any other property including dates. Aggregations can be defined at the root Catalog level (`/aggregations`) and at the Collection level (`/<collection_id>/aggregations`). The `/aggregate` route also fully supports base search and the STAC API [Filter Extension](https://github.com/stac-api-extensions/filter). Any query made with `/search` may also be executed with `/aggregate`, provided that the relevant aggregation fields are available,
286-
287-
288-
A field named `aggregations` should be added to the Collection object for the collection for which the aggregations are available, for example:
289-
290-
```json
291-
"aggregations": [
292-
{
293-
"name": "total_count",
294-
"data_type": "integer"
295-
},
296-
{
297-
"name": "datetime_max",
298-
"data_type": "datetime"
299-
},
300-
{
301-
"name": "datetime_min",
302-
"data_type": "datetime"
303-
},
304-
{
305-
"name": "datetime_frequency",
306-
"data_type": "frequency_distribution",
307-
"frequency_distribution_data_type": "datetime"
308-
},
309-
{
310-
"name": "sun_elevation_frequency",
311-
"data_type": "frequency_distribution",
312-
"frequency_distribution_data_type": "numeric"
313-
},
314-
{
315-
"name": "platform_frequency",
316-
"data_type": "frequency_distribution",
317-
"frequency_distribution_data_type": "string"
318-
},
319-
{
320-
"name": "sun_azimuth_frequency",
321-
"data_type": "frequency_distribution",
322-
"frequency_distribution_data_type": "numeric"
323-
},
324-
{
325-
"name": "off_nadir_frequency",
326-
"data_type": "frequency_distribution",
327-
"frequency_distribution_data_type": "numeric"
328-
},
329-
{
330-
"name": "cloud_cover_frequency",
331-
"data_type": "frequency_distribution",
332-
"frequency_distribution_data_type": "numeric"
333-
},
334-
{
335-
"name": "grid_code_frequency",
336-
"data_type": "frequency_distribution",
337-
"frequency_distribution_data_type": "string"
338-
},
339-
{
340-
"name": "centroid_geohash_grid_frequency",
341-
"data_type": "frequency_distribution",
342-
"frequency_distribution_data_type": "string"
343-
},
344-
{
345-
"name": "centroid_geohex_grid_frequency",
346-
"data_type": "frequency_distribution",
347-
"frequency_distribution_data_type": "string"
348-
},
349-
{
350-
"name": "centroid_geotile_grid_frequency",
351-
"data_type": "frequency_distribution",
352-
"frequency_distribution_data_type": "string"
353-
},
354-
{
355-
"name": "geometry_geohash_grid_frequency",
356-
"data_type": "frequency_distribution",
357-
"frequency_distribution_data_type": "numeric"
358-
},
359-
{
360-
"name": "geometry_geotile_grid_frequency",
361-
"data_type": "frequency_distribution",
362-
"frequency_distribution_data_type": "string"
363-
}
364-
]
365-
```
366-
367-
Available aggregations are:
368-
369-
- total_count (count of total items)
370-
- collection_frequency (Item `collection` field)
371-
- platform_frequency (Item.Properties.platform)
372-
- cloud_cover_frequency (Item.Properties.eo:cloud_cover)
373-
- datetime_frequency (Item.Properties.datetime, monthly interval)
374-
- datetime_min (earliest Item.Properties.datetime)
375-
- datetime_max (latest Item.Properties.datetime)
376-
- sun_elevation_frequency (Item.Properties.view:sun_elevation)
377-
- sun_azimuth_frequency (Item.Properties.view:sun_azimuth)
378-
- off_nadir_frequency (Item.Properties.view:off_nadir)
379-
- grid_code_frequency (Item.Properties.grid:code)
380-
- centroid_geohash_grid_frequency ([geohash grid](https://opensearch.org/docs/latest/aggregations/bucket/geohash-grid/) on Item.Properties.proj:centroid)
381-
- centroid_geohex_grid_frequency ([geohex grid](https://opensearch.org/docs/latest/aggregations/bucket/geohex-grid/) on Item.Properties.proj:centroid)
382-
- centroid_geotile_grid_frequency (geotile on Item.Properties.proj:centroid)
383-
- geometry_geohash_grid_frequency ([geohash grid](https://opensearch.org/docs/latest/aggregations/bucket/geohash-grid/) on Item.geometry)
384-
- geometry_geotile_grid_frequency ([geotile grid](https://opensearch.org/docs/latest/aggregations/bucket/geotile-grid/) on Item.geometry)
385-
386-
Support for additional fields and new aggregations can be added in the associated `database_logic.py` file.
320+
Aggregation of points and geometries, as well as frequency distribution aggregation of any other property including dates is supported in stac-fatsapi-elasticsearch-opensearch. Aggregations can be defined at the root Catalog level (`/aggregations`) and at the Collection level (`/<collection_id>/aggregations`). Details for supported aggregations can be found in [the aggregation docs](./docs/src/aggregation.md)
387321

388322
## Rate Limiting
389323

‎dockerfiles/Dockerfile.ci.es

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
FROM python:3.12-slim
2+
3+
WORKDIR /app
4+
5+
RUN apt-get update && \
6+
apt-get install -y --no-install-recommends \
7+
gcc \
8+
curl \
9+
&& apt-get clean && \
10+
rm -rf /var/lib/apt/lists/*
11+
12+
COPY . /app/
13+
14+
RUN pip3 install --no-cache-dir -e ./stac_fastapi/core && \
15+
pip3 install --no-cache-dir ./stac_fastapi/elasticsearch[server]
16+
17+
USER root
18+
19+
CMD ["python", "-m", "stac_fastapi.elasticsearch.app"]

‎dockerfiles/Dockerfile.ci.os

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
FROM python:3.12-slim
2+
3+
WORKDIR /app
4+
5+
RUN apt-get update && \
6+
apt-get install -y --no-install-recommends \
7+
gcc \
8+
curl \
9+
&& apt-get clean && \
10+
rm -rf /var/lib/apt/lists/*
11+
12+
COPY . /app/
13+
14+
RUN pip3 install --no-cache-dir -e ./stac_fastapi/core && \
15+
pip3 install --no-cache-dir ./stac_fastapi/opensearch[server]
16+
17+
USER root
18+
19+
CMD ["python", "-m", "stac_fastapi.opensearch.app"]

0 commit comments

Comments
 (0)