Display compression type of Kafka topics

[Gleb-Shipilov]

Issue submitter TODO list

• I've searched for an already existing issues here
• I'm running a supported version of the application which is listed here and the feature is not present there

Is your proposal related to a problem?

No response

Describe the feature you're interested in

Prerequisites:

Kafka producers define compression type of Kafka topics by themselves (None, GZIP, LZ4 etc.).

Yet it's impossible to see the compression type of topics in Kafka UI

To do:

Display compression type for topics in Kafka UI:

• In "Topics" grid;
• In "Settings" grid of each topic.

Acceptance criteria:

Compression type of topics is displayed:

• In "Topics" grid;
• In "Settings" grid of each topic.

Describe alternatives you've considered

No response

Version you're running

2ce0662

Additional context

No response

[github-actions]

Hi Gleb-Shipilov! 👋

Welcome, and thank you for opening your first issue in the repo!

Please wait for triaging by our maintainers.

As development is carried out in our spare time, you can support us by sponsoring our activities or even funding the development of specific issues.
Sponsorship link

If you plan to raise a PR for this issue, please take a look at our contributing guide.

[germanosin]

@Gleb-Shipilov, compression is not a topic configuration, displaying this information requires collecting statistics from messages. In theory, there could be multiple compression types, not just one. Our approach is to collect this data periodically and expose it from the statistics.

Potential issue: In a multi-instance setup, only some instances may have this information.
Proposed solution: Implement leader election, have instances collect statistics, and forward requests for specific clusters to the designated nodes.

This would be a fairly complex solution, but I don’t see any simpler alternatives. Please let me know if this approach works for you.

[Haarolean]

@Gleb-Shipilov, compression is not a topic configuration, displaying this information requires collecting statistics from messages. In theory, there could be multiple compression types, not just one. Our approach is to collect this data periodically and expose it from the statistics.

Potential issue: In a multi-instance setup, only some instances may have this information. Proposed solution: Implement leader election, have instances collect statistics, and forward requests for specific clusters to the designated nodes.

This would be a fairly complex solution, but I don’t see any simpler alternatives. Please let me know if this approach works for you.

cc @fallen-up

[Gleb-Shipilov]

This would be a fairly complex solution, but I don’t see any simpler alternatives.

Could you, please, give any estimations of how long this solution might take to implement?

[germanosin]

I would prefer to split this issue into

1. Collect compression types in statistics
2. Introduce leader election and proxy cluster requests to leaders
3. Introduce periodical task schedulers to collect stats
4. Expose compression types within topic requests

[germanosin]

I need to explain why I believe it makes sense to prioritize statistics collection first. The Java Kafka Consumer does not expose compression details through its public API, so gathering this information would require reimplementing the data collection mechanism using internal workarounds. However, this approach would also provide visibility for clients and potentially cover their basic needs.
fyi @Haarolean @Gleb-Shipilov @fallen-up

[Gleb-Shipilov]

Thank you @germanosin for explanation.
I guess for now we can decline this task. For us it doesn't have so much value to spend so much time on it