feat: Add Redis caching for navigation pagination #488

YuriZmytrakov · 2025-10-08T11:37:20Z

Related Issue(s):

Page navigation "previous" rel #439

Description:

Add Redis caching support for navigation pagination to enable proper prev/next links in STAC API responses.

PR Checklist:

Code is formatted and linted (run pre-commit run --all-files)
Tests pass (run make test)
Documentation has been updated to reflect changes, if applicable
Changes are added to the changelog

jonhealy1 · 2025-10-10T06:34:03Z

stac_fastapi/core/stac_fastapi/core/core.py

+
+        if redis_enable:
+            try:
+                redis = await connect_redis()


how does this work with connect_redis_sentinel?

there is connect_redis and connect_redis_sentinel. I think maybe there is logic missing which should determine which one to use? Maybe you can explain how it works?

we should also log the redis connection whether its successful or not?

If a user wants to use standalone Redis (the most common case), they should use connect_redis(). If they want to use Redis Sentinel (used by Cloudferro), connect_redis_sentinel() should be used. The connection is configured differently in each case, which is why we have two separate functions. The user needs to configure either the REDIS or REDIS_SENTINEL environment variables.

The selection of which function to use should be done in redis_utils.py:

For standalone Redis: redis_settings: BaseSettings = RedisSettings()

For Redis Sentinel: redis_settings: BaseSettings = RedisSentinelSettings()

Logging has been added for users.

Ok thanks for explaining - here you only use connect_redis but should you use connect_redis_sentinel as well?

If the user configures Redis Sentinel, the connect_redis_sentinel function should be used instead of connect_redis. I could try to combine these two functions into one, which would choose the appropriate approach depending on whether the user wants standalone or Sentinel depending on config selected. This way, no one would need to manually replace function, only connect_redis would be called. Both approaches are fine with me. Just to note, Cloudferro will probably be the only users of Sentinel, which is why I added this function.

Good idea, maybe even a helper function that calls the right function. If there are env options for sentinel in the codebase then we have to make it useable. Never say never.

@jonhealy1 this has been implemented. Moving forward, depending on the configuration specified via environment variables, it will be handled by connect_redis. The developer is not expected to select the function.

jonhealy1 · 2025-10-10T06:45:46Z

stac_fastapi/core/stac_fastapi/core/core.py

+                        "href": prev_link,
+                    },
+                )
+


can this code - the redis_enabled block - be put into a function? It is used with all_collections too

The code block was replaced here with func as well. Thank you!

Let's change the function name for handle_pagination_links - also, mentioned below - and let's check the REDIS IS ENABLED here before we go to the function.

Done, thank you!

jonhealy1 · 2025-10-10T06:49:20Z

stac_fastapi/core/stac_fastapi/core/core.py

+                        "href": prev_link,
+                    },
+                )
+


this code into a function

@jonhealy1 I have moved this code into handle_pagination_links func that can be reused where needed.

- Configure Redis image for tests of caching navigation - Update Make file Redis with test targets for ES and OS - Integrate Redis/Redis Sentinel client to cache navigation - Add Redis funcs Sentinel for navigation caching

Add tests for Redis pagination caching in search and collections endpoints, plus utility function tests.

jamesfisher-geo

Looks good. Thanks @YuriZmytrakov

jonhealy1 · 2025-10-20T01:44:27Z

@YuriZmytrakov Let us know when this has been approved by QA

YuriZmytrakov · 2025-10-21T11:12:35Z

@YuriZmytrakov Let us know when this has been approved by QA

This PR has passed QA and is ready to be merged.

YuriZmytrakov · 2025-10-21T11:14:06Z

Nice work. I think we need to document this better in the README, ie. have instructions on how to install. I can open a separate issue to add this. Really cool having prev pagination links.

I have added the installation steps for navigation via the Redis cache. The steps are straightforward and can be expanded once Redis is used for caching additional navigation links.

StijnCaerts

Nice feature addition! Just be careful with potential cache conflicts, as highlighted in the code comment.

StijnCaerts · 2025-10-21T11:22:19Z

stac_fastapi/core/stac_fastapi/core/redis_utils.py

+async def save_self_link(
+    redis: aioredis.Redis, token: Optional[str], self_href: str
+) -> None:
+    """Save the self link for the current token."""
+    if token:
+        if sentinel_settings.REDIS_SENTINEL_HOSTS:
+            ttl_seconds = sentinel_settings.REDIS_SELF_LINK_TTL
+        elif standalone_settings.REDIS_HOST:
+            ttl_seconds = standalone_settings.REDIS_SELF_LINK_TTL
+        await redis.setex(f"nav:self:{token}", ttl_seconds, self_href)
+
+
+async def get_prev_link(redis: aioredis.Redis, token: Optional[str]) -> Optional[str]:
+    """Get the previous page link for the current token."""
+    if not token:
+        return None
+    return await redis.get(f"nav:self:{token}")
+
+
+async def redis_pagination_links(
+    current_url: str, token: str, next_token: str, links: list
+) -> None:
+    """Handle Redis pagination."""
+    redis = await connect_redis()
+    if not redis:
+        logger.warning("Redis connection failed.")
+        return
+
+    try:
+        if next_token:
+            await save_self_link(redis, next_token, current_url)
+
+        prev_link = await get_prev_link(redis, token)
+        if prev_link:
+            links.insert(
+                0,
+                {
+                    "rel": "previous",
+                    "type": "application/json",
+                    "method": "GET",
+                    "href": prev_link,
+                },
+            )
+    except Exception as e:
+        logger.warning(f"Redis pagination operation failed: {e}")
+    finally:
+        await redis.close()


By only storing the token and not the full URL in the key, we might hit unwanted cache conflicts. A pagination token might occur in different search queries. For example, these 2 searches with different queries return the same pagination token.

❯ https stac.terrascope.be/search limit==1 collection==sentinel-2-l1c | jq '.links[] | select(.rel=="next") | .href' "https://stac.terrascope.be/search?limit=1&collection=sentinel-2-l1c&token=WzE3NjEwMzAxMjkwMjQsIlMyQl9NU0lMMUNfMjAyNTEwMjFUMDcwMjA5X04wNTExX1IxMjBfVDM4S1FEXzIwMjUxMDIxVDA5NTkyOSIsInNlbnRpbmVsLTItbDFjIl0%3D" ❯ https stac.terrascope.be/search limit==1 | jq '.links[] | select(.rel=="next") | .href' "https://stac.terrascope.be/search?limit=1&token=WzE3NjEwMzAxMjkwMjQsIlMyQl9NU0lMMUNfMjAyNTEwMjFUMDcwMjA5X04wNTExX1IxMjBfVDM4S1FEXzIwMjUxMDIxVDA5NTkyOSIsInNlbnRpbmVsLTItbDFjIl0%3D"

Retrieving the previous link from the cache would potentially change the search query in this case. The links will also only work for GET searches, so best to explicitly limit this and do not try to retrieve the link for POST searches.

Thank you for the heads up. This has been tested, no collisions have occurred. I don’t see any issues with replacing the token with the full URL as the key to add or retrieve the previous page URL from the Redis cache for navigation. However, please note that we are using the proxy, and if the full URL is used, the keys will become irrelevant. As a solution, I have added additional parameters from the URL endpoint + token. This prevents accidentally retrieving the wrong previous page URL, as different requests might share the same token. In this case, the following Redis keys will be created:

nav:search?limit=1&collection=sentinel-2-l1c&token=WzE3NjEwMzAxMjkwMjQsIlMyQl9NU0lMMUNfMjAyNTEwMjFUMDcwMjA5X04wNTExX1IxMjBfVDM4S1FEXzIwMjUxMDIxVDA5NTkyOSIsInNlbnRpbmVsLTItbDFjIl0%3D"

nav:/search?limit=1&token=WzE3NjEwMzAxMjkwMjQsIlMyQl9NU0lMMUNfMjAyNTEwMjFUMDcwMjA5X04wNTExX1IxMjBfVDM4S1FEXzIwMjUxMDIxVDA5NTkyOSIsInNlbnRpbmVsLTItbDFjIl0%3D"

Also, the POST requests are excluded from retrieving the navigation links.

YuriZmytrakov force-pushed the CAT-1382-2 branch 9 times, most recently from 5c8d201 to 8991cfc Compare October 8, 2025 13:07

YuriZmytrakov changed the title ~~feat: add redis cache for navigation~~ feat: Add Redis caching for navigation pagination Oct 8, 2025

YuriZmytrakov mentioned this pull request Oct 8, 2025

feat: Add Redis caching for navigation pagination #484

Closed

4 tasks

YuriZmytrakov force-pushed the CAT-1382-2 branch 2 times, most recently from b16cc88 to 8991cfc Compare October 8, 2025 13:30

YuriZmytrakov requested review from jamesfisher-geo and jonhealy1 October 8, 2025 13:45

jonhealy1 reviewed Oct 10, 2025

View reviewed changes

YuriZmytrakov force-pushed the CAT-1382-2 branch from 8991cfc to e3e84cd Compare October 10, 2025 11:54

Yuri Zmytrakov added 3 commits October 10, 2025 14:18

feat(navigation): implement Redis caching for navigation

d6d268e

- Configure Redis image for tests of caching navigation - Update Make file Redis with test targets for ES and OS - Integrate Redis/Redis Sentinel client to cache navigation - Add Redis funcs Sentinel for navigation caching

test: add Redis cache for navigation tests

52a57ff

Add tests for Redis pagination caching in search and collections endpoints, plus utility function tests.

docs: add setup instructions to readme and update changelog

cb7ee7c

YuriZmytrakov force-pushed the CAT-1382-2 branch 8 times, most recently from f8a4ad9 to ff38a98 Compare October 12, 2025 20:27

YuriZmytrakov dismissed Gomez324’s stale review via 34a842f October 19, 2025 17:16

YuriZmytrakov force-pushed the CAT-1382-2 branch from 3a0854f to 34a842f Compare October 19, 2025 17:16

jamesfisher-geo previously approved these changes Oct 19, 2025

View reviewed changes

Merge branch 'main' into CAT-1382-2

072592d

YuriZmytrakov dismissed jamesfisher-geo’s stale review via 072592d October 20, 2025 08:54

YuriZmytrakov force-pushed the CAT-1382-2 branch 8 times, most recently from c6a9d09 to 98c0550 Compare October 21, 2025 10:13

Merge branch 'main' into CAT-1382-2

dbfe1b2

YuriZmytrakov force-pushed the CAT-1382-2 branch from 98c0550 to dbfe1b2 Compare October 21, 2025 10:55

YuriZmytrakov requested review from Gomez324, jamesfisher-geo and jonhealy1 October 21, 2025 11:15

StijnCaerts requested changes Oct 21, 2025

View reviewed changes

YuriZmytrakov force-pushed the CAT-1382-2 branch 4 times, most recently from d409c95 to 1e492b6 Compare October 21, 2025 20:54

fix: resolve merge conflicts

fe0efb2

YuriZmytrakov force-pushed the CAT-1382-2 branch from 1e492b6 to fe0efb2 Compare October 22, 2025 10:27

YuriZmytrakov requested a review from StijnCaerts October 22, 2025 12:24

Merge branch 'main' into CAT-1382-2

18339d6

feat: Add Redis caching for navigation pagination #488

Are you sure you want to change the base?

feat: Add Redis caching for navigation pagination #488

Conversation

YuriZmytrakov commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonhealy1 Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jamesfisher-geo left a comment

Choose a reason for hiding this comment

Uh oh!

jonhealy1 commented Oct 20, 2025

Uh oh!

YuriZmytrakov commented Oct 21, 2025

Uh oh!

YuriZmytrakov commented Oct 21, 2025

Uh oh!

StijnCaerts left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

YuriZmytrakov commented Oct 8, 2025 •

edited

Loading

jonhealy1 Oct 10, 2025 •

edited

Loading