Skip to content

Conversation

anatolyburtsev
Copy link
Contributor

@anatolyburtsev anatolyburtsev commented Aug 20, 2025

Summary

This PR adds a Key-Value Storage tool to Flowise, enabling persistent storage and retrieval of data with built-in versioning capabilities. It uses AWS DynamoDB as a backbone.

Tool has 2 modes: store or retrieval.

  • Store mode - stored object by key. Example response:
    Successfully stored value with key "total_comp" at 2025-08-20T05:41:09.608Z
  • Retrieval mode - return object by key with timestamp when it was created. Consumes optional argument nth-latest, e.g. value 2 returns second last version of the object. Example response:
{
  "value": "200",
  "timestamp": "2025-08-20T05:41:09.608Z"
}

Note

Retrieval mode works great with JSONPathExtractor introduced here

Example use-case

Monitor a website for changes by storing snapshots and comparing different versions over time. This is useful for tracking competitor updates, detect when news articles are updated or corrected, or track feature announcements and product updates. Diff together with old and new versions can be feed directly to LLM to make sense of it.

Implementation Details

  • Added shared AWS utilities in /packages/components/src/awsToolsUtils.ts for credential management
  • DynamoDB's has to have Partition key pk and Search key sk. Corresponding validation is done during configuration phase. The schema is
    • pk: The actual key name
    • sk: Timestamp as string (for sorting)
    • value: The actual data
    • timestamp: Human-readable ISO timestamp

Configuration

The tool requires:

  • AWS Credentials (via credential manager)
  • AWS Region selection
  • DynamoDB table name.

Testing

  • Added unit tests
  • Manual testing with actual AWS DynamoDB instances

Example of the flow:
Screenshot 2025-08-19 at 10 42 20 PM

Screenshot 2025-08-19 at 10 42 32 PM

Store mode node configuration:
Screenshot 2025-08-20 at 10 45 30 AM

Retrieval mode node configuration:
Screenshot 2025-08-20 at 10 45 40 AM

- Add AWS DynamoDB key-value storage tool for persistent data storage
- Add utility functions for AWS tools configuration
@anatolyburtsev anatolyburtsev changed the title feat: Add AWS DynamoDB KV Storage tool WIP feat: Add AWS DynamoDB KV Storage tool Aug 20, 2025
@anatolyburtsev anatolyburtsev marked this pull request as draft August 20, 2025 06:24
@anatolyburtsev anatolyburtsev marked this pull request as ready for review August 20, 2025 06:46
@anatolyburtsev anatolyburtsev changed the title WIP feat: Add AWS DynamoDB KV Storage tool feat: Add AWS DynamoDB KV Storage tool Aug 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant