tweak(labrinth): create Clickhouse tables with a TTL for staging env #4343
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AlexTMjugador
added
enhancement
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This PR supersedes #4167 and implements the approach I outlined in #4167 (review) to handle our disk space usage problems in the staging environment in a better way.
Specifically, Labrinth now checks whether it is initializing a staging Clickhouse database, identified by the
staging_analyticsname. If so, it creates Clickhouse tables with a 1-day TTL, ensuring that old data rows are automatically discarded. The existing Clickhouse tables in the staging environment have already been manually migrated to use this TTL.Since data in the staging environment is not expected to be long-lived, especially that living on Clickhouse, setting a short TTL for these tables represents a balanced tradeoff between environment parity, data persistence, and storage costs.
Testing
I have tested that these changes had the expected effect on a local environment.