feat(parquetconverter): add support for additional sort columns during Parquet file generation #7003

Angith · 2025-09-01T17:57:59Z

What this PR does:

Which issue(s) this PR fixes:
Fixes #6941

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

…g Parquet file generation Signed-off-by: Angith <[email protected]>

…lumns

…Parquet file generation Signed-off-by: Angith <[email protected]>

Angith · 2025-09-01T18:23:05Z

Hi @yeya24

I’ve raised this PR to add support for additional sort columns during Parquet file generation. A few points I wanted to clarify:

As per the contributing guide, I ran make doc, but it ended up deleting some other important documents. I also ran goimports, but it reformatted ~700 files, including files in the vendor folder. I assume these shouldn’t be committed, so I left those changes out. Could you please confirm the expected workflow here?
I’ve written unit tests for the new functionality and verified, but I haven’t yet tested the change end-to-end. Could you guide me on how to run the full end-to-end tests for Cortex to validate this change?

Thanks in advance for your guidance 🙏

…er/add-sort-columns

…lumns Signed-off-by: Angith <[email protected]>

Angith · 2025-09-07T17:41:38Z

Hi @yeya24, I’ve made updates to address the CI failure. When you get a chance, could you approve the workflow?

yeya24

Sorry for the late review. Thanks for the contribution and I think this change looks great!

Just some comments about the configuration. If we already have the sorting columns configurable as limits, we don't need the same config in parquet converter anymore

pkg/parquetconverter/converter.go

yeya24 · 2025-09-14T06:25:22Z

pkg/parquetconverter/converter.go

+	if len(cfg.AdditionalSortColumns) > 0 {
+		sortColumns = append(sortColumns, cfg.AdditionalSortColumns...)
+	}
+	cfg.AdditionalSortColumns = sortColumns


We can maybe just remove sorting column from base converter option if we have the limits

I’ve removed the sorting column from the base converter option. In the limits, the metric name is added as the default value.

yeya24 · 2025-09-14T06:25:57Z

pkg/parquetconverter/converter.go

 		converterOpts := append(c.baseConverterOptions, convert.WithName(b.ULID.String()))

+		userConfiguredSortColumns := c.limits.ParquetConverterSortColumns(userID)
+		if len(userConfiguredSortColumns) > 0 {


We can remove the if and always use append(sortColumns, userConfiguredSortColumns...) as sorting columns.

yeya24 · 2025-09-14T06:26:45Z

docs/guides/parquet-mode.md

 1. **Row Group Size**: Adjust `max_rows_per_row_group` based on your query patterns
 2. **Cache Size**: Tune `parquet_queryable_shard_cache_size` based on available memory
 3. **Concurrency**: Adjust `meta_sync_concurrency` based on object storage performance
+4. **Sort Columns**: Configure `sort_columns` based on your most common query filters to improve query performance


Let's use the full config name parquet_converter_sort_columns

Angith · 2025-09-15T09:48:32Z

Hi @yeya24, let me work on the comments and revert to you shortly.

…er/add-sort-columns

Signed-off-by: Angith <[email protected]>

Angith · 2025-09-21T13:33:34Z

Hi @yeya24, Thank you for the feedback. I’ve incorporated all the review comments. Could you please take another look?

yeya24 · 2025-09-21T22:12:30Z

I am sorry. Can you help resolve the conflict? I think what I just merged conflicts to what you added in this PR

yeya24

The code change looks great! The only conflict here is that we start to mark parquet limits as unhidden from doc so you need to remove the doc: hidden part

…er/add-sort-columns

Angith · 2025-09-22T15:35:00Z

@yeya24, I have resolved the conflict. Could you please verify?

yeya24 · 2025-09-22T17:33:42Z

We need to fix lint. Can you run make doc or make docs?

…umns configuration Signed-off-by: Angith <[email protected]>

Angith · 2025-09-23T18:53:06Z

@yeya24 I have run the make doc command and pushed the changes. Could you please check now?

yeya24 · 2025-09-24T04:24:05Z

@Angith It seems that the lint still failed due to white noise. Can you try running the make target locally?

Run make BUILD_IN_CONTAINER=false check-white-noise
  make BUILD_IN_CONTAINER=false check-white-noise
  shell: sh -e {0}
Please remove trailing whitespaces running 'make clean-white-noise'

Other test failures are unrelated. I will retry them

Angith and others added 2 commits August 31, 2025 23:47

feat(parquetconverter): add support for additional sort columns durin…

6ff0460

…g Parquet file generation Signed-off-by: Angith <[email protected]>

Merge branch 'cortexproject:master' into parquetconverter/add-sort-co…

e25441f

…lumns

pull-request-size bot added the size/L label Sep 1, 2025

Angith changed the title ~~Parquetconverter/add sort columns~~ feat(parquetconverter): add support for additional sort columns during Parquet file generation Sep 1, 2025

docs: update CHANGELOG.md to include support for sort columns during …

386ce10

…Parquet file generation Signed-off-by: Angith <[email protected]>

Angith added 2 commits September 7, 2025 19:40

Merge branch 'master' of github.com:Angith/cortex into parquetconvert…

34e59c5

…er/add-sort-columns

fix: Remove duplicate flag registration for parquet-converter.sort-co…

42b59a3

…lumns Signed-off-by: Angith <[email protected]>

yeya24 reviewed Sep 14, 2025

View reviewed changes

Angith added 2 commits September 20, 2025 20:38

Merge branch 'master' of github.com:Angith/cortex into parquetconvert…

ea629b2

…er/add-sort-columns

fix: exluded sort columns from base converter options

afe4e2a

Signed-off-by: Angith <[email protected]>

pull-request-size bot added size/M and removed size/L labels Sep 21, 2025

Angith requested a review from yeya24 September 21, 2025 13:33

yeya24 approved these changes Sep 21, 2025

View reviewed changes

Merge branch 'master' of github.com:Angith/cortex into parquetconvert…

e3db5a4

…er/add-sort-columns

doc: updated config-file-reference.md with parquet_converter_sort_col…

b457ded

…umns configuration Signed-off-by: Angith <[email protected]>

feat(parquetconverter): add support for additional sort columns during Parquet file generation #7003

Are you sure you want to change the base?

feat(parquetconverter): add support for additional sort columns during Parquet file generation #7003

Conversation

Angith commented Sep 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Angith commented Sep 1, 2025

Uh oh!

Angith commented Sep 7, 2025

Uh oh!

yeya24 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yeya24 Sep 14, 2025

Choose a reason for hiding this comment

Uh oh!

Angith Sep 21, 2025

Choose a reason for hiding this comment

Uh oh!

yeya24 Sep 14, 2025

Choose a reason for hiding this comment

Uh oh!

Angith Sep 21, 2025

Choose a reason for hiding this comment

Uh oh!

yeya24 Sep 14, 2025

Choose a reason for hiding this comment

Uh oh!

Angith Sep 21, 2025

Choose a reason for hiding this comment

Uh oh!

Angith commented Sep 15, 2025

Uh oh!

Angith commented Sep 21, 2025

Uh oh!

yeya24 commented Sep 21, 2025

Uh oh!

yeya24 left a comment

Choose a reason for hiding this comment

Uh oh!

Angith commented Sep 22, 2025

Uh oh!

yeya24 commented Sep 22, 2025

Uh oh!

Angith commented Sep 23, 2025

Uh oh!

yeya24 commented Sep 24, 2025

Uh oh!

Uh oh!

Angith commented Sep 1, 2025 •

edited

Loading