Merge branch 'main' into CAT-1382-temp

Author: YuriZmytrakov

.pre-commit-config.yaml

@@ -16,7 +16,7 @@ repos:
           # E501 let black handle all line length decisions
           # W503 black conflicts with "line break before operator" rule
           # E203 black conflicts with "whitespace before ':'" rule
-          '--ignore=E501,W503,E203,C901,E231' ]
+          '--ignore=E501,W503,E203,C901,E231,E702' ]
   - repo: https://github.com/pre-commit/mirrors-mypy
     rev: v0.991
     hooks:

CHANGELOG.md

@@ -10,9 +10,39 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ### Changed
 
-- Simplified Patch class and updated patch script creation including adding nest creation for merge patch [#420](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/420)
+- updated `numReturned` & `numMatched` fields in itemCollection return to `numberReturned` & `numberMatched`. [#446](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/446)
+
+## [v6.3.0] - 2025-09-16
+
+### Added
+
+- `STAC_INDEX_ASSETS` environment variable to allow asset serialization to be configurable. [#433](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/433)
+- Added the `ENV_MAX_LIMIT` environment variable to SFEOS, allowing overriding of the `MAX_LIMIT`, which controls the `?limit` parameter for returned items and STAC collections. [#434](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/434)
+- Sort, Query, and Filter extension and functionality to the item collection route. [#437](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/437)
+- Added Fields Extension implementation for the `/collections/{collection_id}/items` endpoint. [#436](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/436)
+
+### Changed
+
+- Changed assets serialization to prevent mapping explosion while allowing asset information to be indexed. [#341](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/341)
+- Simplified the item_collection function in core.py, moving the request to the get_search function. [#437](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/437)
+- Updated the `format_datetime_range` function to support milliseconds. [#423](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/423)
+- Blocked the /collections/{collection_id}/bulk_items endpoint when environmental variable ENABLE_DATETIME_INDEX_FILTERING is set to true. [#438](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/438)
+
+### Fixed
+
+- Fixed issue where sortby was not accepting the default sort, where a + or - was not specified before the field value ie. localhost:8081/collections/{collection_id}/items?sortby=id. [#437](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/437)
+
+## [v6.2.1] - 2025-09-02
+
+### Added
+
+- Added `id` field as secondary sort to sort config to ensure unique pagination tokens. [#421](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/421)
 - Added default environment variable `STAC_ITEM_LIMIT` to SFEOS for result limiting of returned items and STAC collections [#419](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/419)
 
+### Changed
+
+- Simplified Patch class and updated patch script creation including adding nest creation for merge patch [#420](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/420)
+
 ## [v6.2.0] - 2025-08-27
 
 ### Added 
@@ -478,7 +508,9 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 - Use genexp in execute_search and get_all_collections to return results.
 - Added db_to_stac serializer to item_collection method in core.py.
 
-[Unreleased]: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/compare/v6.2.0...main
+[Unreleased]: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/compare/v6.3.0...main
+[v6.3.0]: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/compare/v6.2.1...v6.3.0
+[v6.2.1]: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/compare/v6.2.0...v6.2.1
 [v6.2.0]: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/compare/v6.1.0...v6.2.0
 [v6.1.0]: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/compare/v6.0.0...v6.1.0
 [v6.0.0]: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/compare/v5.0.0...v6.0.0

README.md

@@ -66,26 +66,43 @@ This project is built on the following technologies: STAC, stac-fastapi, FastAPI
 
 ## Table of Contents
 
-- [Documentation & Resources](#documentation--resources)
-- [Package Structure](#package-structure)
-- [Examples](#examples)
-- [Performance](#performance)
-- [Quick Start](#quick-start)
-  - [Installation](#installation)
-  - [Running Locally](#running-locally)
-- [Configuration reference](#configuration-reference)
-- [Interacting with the API](#interacting-with-the-api)
-- [Configure the API](#configure-the-api)
-- [Collection pagination](#collection-pagination)
-- [Ingesting Sample Data CLI Tool](#ingesting-sample-data-cli-tool)
-- [Elasticsearch Mappings](#elasticsearch-mappings)
-- [Managing Elasticsearch Indices](#managing-elasticsearch-indices)
-  - [Snapshots](#snapshots)
-  - [Reindexing](#reindexing)
-- [Auth](#auth)
-- [Aggregation](#aggregation)
-- [Rate Limiting](#rate-limiting)
-- [Datetime-Based Index Management](#datetime-based-index-management)
+- [stac-fastapi-elasticsearch-opensearch](#stac-fastapi-elasticsearch-opensearch)
+  - [Sponsors \& Supporters](#sponsors--supporters)
+  - [Project Introduction - What is SFEOS?](#project-introduction---what-is-sfeos)
+  - [Common Deployment Patterns](#common-deployment-patterns)
+  - [Technologies](#technologies)
+  - [Table of Contents](#table-of-contents)
+  - [Documentation \& Resources](#documentation--resources)
+  - [Package Structure](#package-structure)
+  - [Examples](#examples)
+  - [Performance](#performance)
+    - [Direct Response Mode](#direct-response-mode)
+  - [Quick Start](#quick-start)
+    - [Installation](#installation)
+    - [Running Locally](#running-locally)
+      - [Using Pre-built Docker Images](#using-pre-built-docker-images)
+      - [Using Docker Compose](#using-docker-compose)
+  - [Configuration Reference](#configuration-reference)
+  - [Datetime-Based Index Management](#datetime-based-index-management)
+    - [Overview](#overview)
+    - [When to Use](#when-to-use)
+    - [Configuration](#configuration)
+      - [Enabling Datetime-Based Indexing](#enabling-datetime-based-indexing)
+    - [Related Configuration Variables](#related-configuration-variables)
+  - [How Datetime-Based Indexing Works](#how-datetime-based-indexing-works)
+    - [Index and Alias Naming Convention](#index-and-alias-naming-convention)
+    - [Index Size Management](#index-size-management)
+  - [Interacting with the API](#interacting-with-the-api)
+  - [Configure the API](#configure-the-api)
+  - [Collection Pagination](#collection-pagination)
+  - [Ingesting Sample Data CLI Tool](#ingesting-sample-data-cli-tool)
+  - [Elasticsearch Mappings](#elasticsearch-mappings)
+  - [Managing Elasticsearch Indices](#managing-elasticsearch-indices)
+    - [Snapshots](#snapshots)
+    - [Reindexing](#reindexing)
+  - [Auth](#auth)
+  - [Aggregation](#aggregation)
+  - [Rate Limiting](#rate-limiting)
 
 ## Documentation & Resources
 
@@ -228,6 +245,8 @@ You can customize additional settings in your `.env` file:
 | `DATABASE_REFRESH`           | Controls whether database operations refresh the index immediately after changes. If set to `true`, changes will be immediately searchable. If set to `false`, changes may not be immediately visible but can improve performance for bulk operations. If set to `wait_for`, changes will wait for the next refresh cycle to become visible. | `false` | Optional |
 | `ENABLE_TRANSACTIONS_EXTENSIONS` | Enables or disables the Transactions and Bulk Transactions API extensions. If set to `false`, the POST `/collections` route and related transaction endpoints (including bulk transaction operations) will be unavailable in the API. This is useful for deployments where mutating the catalog via the API should be prevented. | `true` | Optional |
 | `STAC_ITEM_LIMIT` | Sets the environment variable for result limiting to SFEOS for the number of returned items and STAC collections. | `10` | Optional |
+| `STAC_INDEX_ASSETS` | Controls if Assets are indexed when added to Elasticsearch/Opensearch. This allows asset fields to be included in search queries. | `false` | Optional |
+| `ENV_MAX_LIMIT` | Configures the environment variable in SFEOS to override the default `MAX_LIMIT`, which controls the limit parameter for returned items and STAC collections. | `10,000` | Optional |
 
 > [!NOTE]
 > The variables `ES_HOST`, `ES_PORT`, `ES_USE_SSL`, `ES_VERIFY_CERTS` and `ES_TIMEOUT` apply to both Elasticsearch and OpenSearch backends, so there is no need to rename the key names to `OS_` even if you're using OpenSearch.

compose.yml

@@ -9,7 +9,7 @@ services:
     environment:
       - STAC_FASTAPI_TITLE=stac-fastapi-elasticsearch
       - STAC_FASTAPI_DESCRIPTION=A STAC FastAPI with an Elasticsearch backend
-      - STAC_FASTAPI_VERSION=6.2.0
+      - STAC_FASTAPI_VERSION=6.0.0
       - STAC_FASTAPI_LANDING_PAGE_ID=stac-fastapi-elasticsearch
       - APP_HOST=0.0.0.0
       - APP_PORT=8080
@@ -46,7 +46,7 @@ services:
     environment:
       - STAC_FASTAPI_TITLE=stac-fastapi-opensearch
       - STAC_FASTAPI_DESCRIPTION=A STAC FastAPI with an Opensearch backend
-      - STAC_FASTAPI_VERSION=6.2.0
+      - STAC_FASTAPI_VERSION=6.0.0
       - STAC_FASTAPI_LANDING_PAGE_ID=stac-fastapi-opensearch
       - APP_HOST=0.0.0.0
       - APP_PORT=8082

dockerfiles/Dockerfile.dev.os

@@ -4,11 +4,10 @@ FROM python:3.10-slim
 # update apt pkgs, and install build-essential for ciso8601
 RUN apt-get update && \
     apt-get -y upgrade && \
-    apt-get -y install build-essential && \
+    apt-get -y install build-essential git && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*
 
-RUN apt-get -y install git
 # update certs used by Requests
 ENV CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
 

examples/auth/compose.basic_auth.yml

@@ -9,7 +9,7 @@ services:
     environment:
       - STAC_FASTAPI_TITLE=stac-fastapi-elasticsearch
       - STAC_FASTAPI_DESCRIPTION=A STAC FastAPI with an Elasticsearch backend
-      - STAC_FASTAPI_VERSION=6.2.0
+      - STAC_FASTAPI_VERSION=6.0.0
       - STAC_FASTAPI_LANDING_PAGE_ID=stac-fastapi-elasticsearch
       - APP_HOST=0.0.0.0
       - APP_PORT=8080
@@ -43,7 +43,7 @@ services:
     environment:
       - STAC_FASTAPI_TITLE=stac-fastapi-opensearch
       - STAC_FASTAPI_DESCRIPTION=A STAC FastAPI with an Opensearch backend
-      - STAC_FASTAPI_VERSION=6.2.0
+      - STAC_FASTAPI_VERSION=6.0.0
       - STAC_FASTAPI_LANDING_PAGE_ID=stac-fastapi-opensearch
       - APP_HOST=0.0.0.0
       - APP_PORT=8082

examples/auth/compose.oauth2.yml

@@ -9,7 +9,7 @@ services:
     environment:
       - STAC_FASTAPI_TITLE=stac-fastapi-elasticsearch
       - STAC_FASTAPI_DESCRIPTION=A STAC FastAPI with an Elasticsearch backend
-      - STAC_FASTAPI_VERSION=6.2.0
+      - STAC_FASTAPI_VERSION=6.0.0
       - STAC_FASTAPI_LANDING_PAGE_ID=stac-fastapi-elasticsearch
       - APP_HOST=0.0.0.0
       - APP_PORT=8080
@@ -44,7 +44,7 @@ services:
     environment:
       - STAC_FASTAPI_TITLE=stac-fastapi-opensearch
       - STAC_FASTAPI_DESCRIPTION=A STAC FastAPI with an Opensearch backend
-      - STAC_FASTAPI_VERSION=6.2.0
+      - STAC_FASTAPI_VERSION=6.0.0
       - STAC_FASTAPI_LANDING_PAGE_ID=stac-fastapi-opensearch
       - APP_HOST=0.0.0.0
       - APP_PORT=8082

examples/auth/compose.route_dependencies.yml

@@ -9,7 +9,7 @@ services:
     environment:
       - STAC_FASTAPI_TITLE=stac-fastapi-elasticsearch
       - STAC_FASTAPI_DESCRIPTION=A STAC FastAPI with an Elasticsearch backend
-      - STAC_FASTAPI_VERSION=6.2.0
+      - STAC_FASTAPI_VERSION=6.0.0
       - STAC_FASTAPI_LANDING_PAGE_ID=stac-fastapi-elasticsearch
       - APP_HOST=0.0.0.0
       - APP_PORT=8080
@@ -43,7 +43,7 @@ services:
     environment:
       - STAC_FASTAPI_TITLE=stac-fastapi-opensearch
       - STAC_FASTAPI_DESCRIPTION=A STAC FastAPI with an Opensearch backend
-      - STAC_FASTAPI_VERSION=6.2.0
+      - STAC_FASTAPI_VERSION=6.0.0
       - STAC_FASTAPI_LANDING_PAGE_ID=stac-fastapi-opensearch
       - APP_HOST=0.0.0.0
       - APP_PORT=8082

examples/rate_limit/compose.rate_limit.yml

@@ -9,7 +9,7 @@ services:
     environment:
       - STAC_FASTAPI_TITLE=stac-fastapi-elasticsearch
       - STAC_FASTAPI_DESCRIPTION=A STAC FastAPI with an Elasticsearch backend
-      - STAC_FASTAPI_VERSION=6.2.0
+      - STAC_FASTAPI_VERSION=6.0.0
       - STAC_FASTAPI_LANDING_PAGE_ID=stac-fastapi-elasticsearch
       - APP_HOST=0.0.0.0
       - APP_PORT=8080
@@ -43,7 +43,7 @@ services:
     environment:
       - STAC_FASTAPI_TITLE=stac-fastapi-opensearch
       - STAC_FASTAPI_DESCRIPTION=A STAC FastAPI with an Opensearch backend
-      - STAC_FASTAPI_VERSION=6.2.0
+      - STAC_FASTAPI_VERSION=6.0.0
       - STAC_FASTAPI_LANDING_PAGE_ID=stac-fastapi-opensearch
       - APP_HOST=0.0.0.0
       - APP_PORT=8082

scripts/reindex_elasticsearch.py

@@ -0,0 +1,84 @@
+import asyncio
+import time
+
+from stac_fastapi.elasticsearch.config import AsyncElasticsearchSettings
+from stac_fastapi.elasticsearch.database_logic import create_index_templates
+from stac_fastapi.sfeos_helpers.mappings import COLLECTIONS_INDEX, ITEMS_INDEX_PREFIX
+
+
+async def reindex(client, index, new_index, aliases):
+    """Reindex STAC index"""
+    print(f"reindexing {index} to {new_index}")
+
+    await client.options(ignore_status=400).indices.create(index=new_index)
+
+    reindex_resp = await client.reindex(
+        dest={"index": new_index},
+        source={"index": [index]},
+        wait_for_completion=False,
+        script={
+            "source": "if (ctx._source.containsKey('assets')){List l = new ArrayList();for (key in ctx._source.assets.keySet()) {def item = ctx._source.assets[key]; item['es_key'] = key; l.add(item)}ctx._source.assets=l} if (ctx._source.containsKey('item_assets')){ List a = new ArrayList(); for (key in ctx._source.item_assets.keySet()) {def item = ctx._source.item_assets[key]; item['es_key'] = key; a.add(item)}ctx._source.item_assets=a}",
+            "lang": "painless",
+        },
+    )
+
+    task_id = reindex_resp["task"]
+
+    reindex_complete = False
+    while not reindex_complete:
+        task_resp = await client.tasks.get(task_id=task_id)
+
+        if "completed" in task_resp and task_resp["completed"]:
+            reindex_complete = True
+
+        elif "error" in task_resp:
+            reindex_complete = True
+            print(f"Reindex failed for {index} with error: {task_resp['error']}")
+
+        else:
+            time.sleep(60)
+
+    actions = []
+    for alias in aliases["aliases"]:
+        actions.extend(
+            [
+                {"add": {"index": new_index, "alias": alias}},
+                {"remove": {"index": index, "alias": alias}},
+            ]
+        )
+
+    await client.indices.update_aliases(actions=actions)
+
+
+async def run():
+    """Reindex all STAC indexes for mapping update"""
+    client = AsyncElasticsearchSettings().create_client
+
+    await create_index_templates()
+
+    collection_response = await client.indices.get_alias(name=COLLECTIONS_INDEX)
+    collections = await client.search(index=COLLECTIONS_INDEX)
+
+    collection_index, collection_aliases = next(iter(collection_response.items()))
+    collection_index_name, version = collection_index.rsplit("-", 1)
+    new_collection_index = f"{collection_index_name}-{str(int(version) + 1).zfill(6)}"
+
+    await reindex(client, collection_index, new_collection_index, collection_aliases)
+
+    for collection in collections["hits"]["hits"]:
+
+        item_indexes = await client.indices.get_alias(
+            name=f"{ITEMS_INDEX_PREFIX}{collection['_id']}*"
+        )
+
+        for item_index, aliases in item_indexes.items():
+            item_index_name, version = item_index.rsplit("-", 1)
+            new_item_index = f"{item_index_name}-{str(int(version) + 1).zfill(6)}"
+
+            await reindex(client, item_index, new_item_index, aliases)
+
+    await client.close()
+
+
+if __name__ == "__main__":
+    asyncio.run(run())