You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Shadowing is Redpanda's enterprise-grade disaster recovery solution that establishes asynchronous, offset-preserving replication between two distinct Redpanda clusters. A cluster is able to create a dedicated client that continuously replicates source cluster data, including offsets, timestamps, and cluster metadata. This creates a read-only shadow cluster that you can quickly failover to handle production traffic during a disaster.
12
12
13
-
[IMPORTANT]
14
-
====
15
-
**Experiencing an active disaster?** See xref:deploy:redpanda/manual/resilience/shadowing-guide.adoc[Emergency Shadowing Failover] for immediate step-by-step emergency procedures.
Unlike traditional replication tools that re-produce messages, Shadowing copies data at the byte level, ensuring shadow topics contain identical copies of source topics with preserved offsets and timestamps.
19
16
20
17
Shadowing replicates:
21
18
22
19
* **Topic data**: All records with preserved offsets and timestamps
23
20
* **Topic configurations**: Partition counts, retention policies, and other xref:reference:properties/topic-properties.adoc[topic properties]
24
-
* **xref:develop:consume-data/consumer-offsets.adoc[Consumer group offsets]**: Enables seamless consumer resumption after failover
21
+
* **Consumer group offsets**: Enables seamless consumer resumption after failover
25
22
* **Access Control Lists (ACLs)**: User permissions and security policies
26
-
* **xref:manage:schema-reg/schema-reg-overview.adoc[Schema Registry] data**: Schema definitions and compatibility settings
23
+
* **Schema Registry data**: Schema definitions and compatibility settings
27
24
28
25
== How Shadowing fits into disaster recovery
29
26
@@ -81,17 +78,16 @@ This service account authenticates from the shadow cluster to the source cluster
81
78
82
79
=== Network and authentication
83
80
84
-
You must configure network connectivity between clusters with appropriate firewall rules to allow the shadow cluster to connect to the source cluster for data replication. Shadowing uses a pull-based architecture where the shadow cluster fetches data from the source cluster.
81
+
You must configure network connectivity between clusters with appropriate firewall rules to allow the shadow cluster to connect to the source cluster for data replication. Shadowing uses a pull-based architecture where the shadow cluster fetches data from the source cluster. For detailed networking configuration, see <<networking>>.
85
82
86
83
If using xref:manage:security/authentication.adoc[authentication] for the shadow link connection, configure the source cluster with your chosen authentication method (SASL/SCRAM, TLS, mTLS) and ensure the shadow cluster has the proper credentials to authenticate to the source cluster.
87
84
88
85
== Set up Shadowing
89
86
90
87
To set up Shadowing:
91
88
92
-
* **Understand replication behavior**: Before you set up Shadowing, review what topic properties, consumer groups, and security settings Redpanda automatically replicates versus what requires explicit configuration
93
-
* **Configure filters**: Define which topics, consumer groups, and ACLs should replicate by creating include/exclude patterns that match your disaster recovery requirements
94
-
* **Create a shadow link**: Establish the connection between clusters using `rpk`, the Admin API, or Redpanda Console with authentication and network settings
89
+
* **Configure filters**: Define which topics, consumer groups, and ACLs should replicate by creating include/exclude patterns that match your disaster recovery requirements. See <<set-filters>>.
90
+
* **Create a shadow link**: Establish the connection between clusters using `rpk`, the Admin API, or Redpanda Console with authentication and network settings. See <<create-a-shadow-link>>.
95
91
96
92
== What gets replicated
97
93
@@ -143,16 +139,16 @@ Each filter uses two key settings:
143
139
* **Pattern type**: Determines how names are matched
144
140
- `LITERAL`: Matches names exactly (including the special wildcard `*` to match all items)
145
141
- `PREFIX`: Matches names that start with the specified string
146
-
* **Filter type**: Specifies whether to include or exclude matching items
142
+
* **Filter type**: Specifies whether to INCLUDE or EXCLUDE matching items
147
143
- `INCLUDE`: Replicate items that match the pattern
148
144
- `EXCLUDE`: Skip items that match the pattern
149
145
150
146
==== Filter processing rules
151
147
152
-
Redpanda processes filters in the order you define them with exclude filters taking precedence. Design your filter lists carefully:
148
+
Redpanda processes filters in the order you define them with EXCLUDE filters taking precedence. Design your filter lists carefully:
153
149
154
150
1. **Exclude filters win**: If any EXCLUDE filter matches a resource, it is excluded regardless of INCLUDE filters
155
-
2. **Order matters for includes**: Among INCLUDE filters, the first match determines the result
151
+
2. **Order matters for INCLUDE filters**: Among INCLUDE filters, the first match determines the result
156
152
3. **Default behavior**: Items that don't match any filter are excluded from replication
157
153
158
154
==== Common filtering patterns
@@ -312,6 +308,8 @@ This creates a template configuration file that you can customize for your envir
312
308
313
309
Configure network connectivity between your source and shadow clusters to enable shadow link replication. The shadow cluster initiates connections to the source cluster using a pull-based architecture.
314
310
311
+
For additional details about networking, see <<network-and-authentication>>.
312
+
315
313
==== Connection requirements
316
314
317
315
* **Direction**: Shadow cluster connects to source cluster (outbound from shadow, inbound to source)
@@ -526,9 +524,11 @@ See also: link:/api/doc/admin/[Admin API v2 reference^]
Failover is the process of modifying shadow topics or an entire shadow cluster from read-only replicas to fully writable resources, and ceasing replication from the source cluster. You can fail over individual topics for selective workload migration or fail over the entire cluster for comprehensive disaster recovery. This critical operation transforms your shadow resources into operational production assets, allowing you to redirect application traffic when the source cluster becomes unavailable.
530
530
531
-
=== Understanding failover behavior
531
+
=== Failover behavior
532
532
533
533
When you initiate failover, Redpanda performs the following operations:
534
534
@@ -612,7 +612,7 @@ After successful failover, your shadow cluster exhibits the following characteri
612
612
613
613
**Topic accessibility:**
614
614
615
-
* Failedover topics become fully writable and readable.
615
+
* Failed over topics become fully writable and readable.
616
616
* Applications can produce and consume messages normally.
617
617
* All Kafka APIs are available for failedover topics.
0 commit comments