Skip to content

Commit 356bdb7

Browse files
micheleRPpaulohtb6
andauthored
DOC-1631 dataplane API configure Kafka client quotas (#1349)
Co-authored-by: Paulo Borges <[email protected]>
1 parent 3ceb3e0 commit 356bdb7

File tree

1 file changed

+23
-12
lines changed

1 file changed

+23
-12
lines changed

modules/manage/pages/cluster-maintenance/manage-throughput.adoc

Lines changed: 23 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,18 @@
11
= Manage Throughput
2-
:description: Manage the throughput of Kafka traffic with configurable properties.
2+
:description: Learn how to manage the throughput of Kafka traffic.
33
:page-categories: Management, Networking
44
// tag::single-source[]
55

6-
Redpanda supports throughput throttling on both ingress and egress independently, and allows configuration at the broker and client levels. This helps prevent clients from causing unbounded network and disk usage on brokers.
7-
8-
* Broker-wide limits apply to all clients connected to the broker and restrict total traffic on the broker.
9-
* Client limits apply to a set of clients defined by their `client_id` and help prevent a set of clients from starving other clients using the same broker. You can manage client quotas with xref:reference:rpk/rpk-cluster/rpk-cluster-quotas.adoc[`rpk cluster quotas`] or with the Kafka API. When no quotas apply, the client has unlimited throughput.
6+
Redpanda supports throughput throttling on both ingress and egress independently, and allows configuration at the broker and client levels. This helps prevent clients from causing unbounded network and disk usage on brokers. You can configure limits at two levels:
107

8+
* *Broker limits*: These apply to all clients connected to the broker and restrict total traffic on the broker. See <<broker-wide-throughput-limits, Broker-wide throughput limits>>.
9+
ifndef::env-cloud[]
10+
* *Client limits*: These apply to a set of clients defined by their `client_id` and help prevent a set of clients from starving other clients using the same broker. You can manage client quotas with xref:reference:rpk/rpk-cluster/rpk-cluster-quotas.adoc[`rpk cluster quotas`], with {ui}, or with the Kafka API. When no quotas apply, the client has unlimited throughput.
11+
endif::[]
1112
ifdef::env-cloud[]
12-
NOTE: Throughput throttling is supported for BYOC and Dedicated clusters.
13+
* *Client limits*: These apply to a set of clients defined by their `client_id` and help prevent a set of clients from starving other clients using the same broker. You can manage client quotas with xref:reference:rpk/rpk-cluster/rpk-cluster-quotas.adoc[`rpk cluster quotas`], with the {ui} UI, with the link:https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-quotaservice_listquotas[Redpanda Cloud Data Plane API], or with the Kafka API. When no quotas apply, the client has unlimited throughput.
14+
15+
NOTE: Throughput throttling is supported for BYOC and Dedicated clusters only.
1316
endif::[]
1417

1518
== Throughput throttling enforcement
@@ -18,6 +21,10 @@ NOTE: As of v24.2, Redpanda enforces all throughput limits per broker, including
1821

1922
Throughput limits are enforced by applying backpressure to clients. When a connection is in breach of the throughput limit, the throttler advises the client about the delay (throttle time) that would bring the rate back to the allowed level. Redpanda starts by adding a `throttle_time_ms` field to responses. If that isn't honored, delays are inserted on the connection's next read operation.
2023

24+
ifdef::env-cloud[]
25+
In Redpanda Cloud, the throttling delay is set to 30 seconds.
26+
endif::[]
27+
2128
ifndef::env-cloud[]
2229
The throttling delay may not exceed the limit set by xref:reference:tunable-properties.adoc#max_kafka_throttle_delay_ms[`max_kafka_throttle_delay_ms`].
2330
endif::[]
@@ -41,13 +48,13 @@ The properties for broker-wide throughput quota balancing are configured at the
4148
| A broker's total throughput limit for egress Kafka traffic.
4249

4350
| xref:reference:cluster-properties.adoc#kafka_throughput_control[kafka_throughput_control]
44-
| List of clients for whom broker-wide limits do not apply
51+
| List of clients for whom broker-wide limits do not apply.
4552

4653
| xref:reference:cluster-properties.adoc#kafka_throughput_controlled_api_keys[kafka_throughput_controlled_api_keys]
47-
| List of Kafka request types subject to broker-wide throughput limits; defaults to `produce` and `fetch`.
54+
| Kafka request types subject to broker-wide throughput limits; defaults to `produce` and `fetch`.
4855

4956
| xref:reference:tunable-properties.adoc#max_kafka_throttle_delay_ms[max_kafka_throttle_delay_ms]
50-
| The maximum delay inserted in the data path of Kafka API requests to throttle them down. Configuring this to be less than the Kafka client timeout can ensure that the delay that's inserted won't be long enough to cause a client timeout by itself.
57+
| Maximum delay inserted in the data path of Kafka API requests to throttle them down. Setting this lower than the Kafka client timeout helps ensure throttling alone does not cause client timeouts.
5158

5259
|===
5360

@@ -97,6 +104,10 @@ endif::[]
97104

98105
=== Individual client throughput limit
99106

107+
ifdef::env-cloud[]
108+
NOTE: The following sections show how to manage throughput with `rpk`. You can also manage throughput with the link:https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-quotaservice_listquotas[Redpanda Cloud Data Plane API].
109+
endif::[]
110+
100111
To view current throughput quotas set through the Kafka API, run xref:reference:rpk/rpk-cluster/rpk-cluster-quotas-describe.adoc[`rpk cluster quotas describe`].
101112

102113
For example, to see the quotas for client ID `consumer-1`:
@@ -142,7 +153,7 @@ NOTE: A client group specified with `client-id-prefix` is not the equivalent of
142153

143154
You can apply default throughput limits to clients. Redpanda applies the default limits if no quotas are configured for a specific `client_id` or prefix.
144155

145-
To specify a produce quota of 1 GBps through the Kafka API (which means a 1 GBps limit across all produce requests sent to a single Redpanda broker), run:
156+
To specify a produce quota of 1 GB/s through the Kafka API (applies across all produce requests to a single broker), run:
146157

147158
[,bash]
148159
----
@@ -237,9 +248,9 @@ Replace the placeholder values with the new quota values, accounting for the con
237248
rpk cluster config set kafka_client_group_byte_rate_quota=
238249
----
239250

240-
=== View throughput limits in Redpanda Console
251+
=== View throughput limits in {ui}
241252

242-
You can also use Redpanda Console to view enforced limits. In the menu, go to **Quotas**.
253+
You can also use {ui} to view enforced limits. In the side menu, go to **Quotas**.
243254

244255
=== Monitor client throughput
245256

0 commit comments

Comments
 (0)