fix(startup): add retry to CreateTiStore to handle TiKV bootstrap delay

[wlwilliamx]

What problem does this PR solve?

Issue Number: close #2788

What is changed and how it works?

This PR introduces a retry mechanism when creating the TiKV storage client (CreateTiStore) to make TiCDC startup more resilient, particularly when TiKV is not yet fully bootstrapped.

The key changes are:

1. pkg/upstream/upstream.go:

   • The CreateTiStore function now accepts a context.Context.
   • The call to d.OpenWithOptions (the TiKV driver) is wrapped in a retry.Do block.
   • This retry logic uses a defaultMaxRetry of 5, a base backoff delay of 200ms, and a max backoff delay of 4000ms.
   • It will retry on all errors except for context.Canceled, ensuring it waits for TiKV to become ready.
   • initUpstream is updated to pass the context to CreateTiStore.

2. Context Propagation:

   • To support the context-aware retry in CreateTiStore, the context.Context has been propagated down from the initial callers.
   • pkg/keyspace/keyspace_manager.go: The Manager interface's GetStorage method signature is updated to GetStorage(ctx context.Context, keyspace string). The manager implementation is updated to accept the ctx and pass it to CreateTiStore.
   • api/v2/changefeed.go: Callers of keyspaceManager.GetStorage in CreateChangefeed, VerifyTable, and UpdateChangefeed are updated to pass the request context.
   • logservice/txnutil/lock_resolver.go: The caller of keyspaceManager.GetStorage in Resolve is also updated to pass the context.

3. pkg/pdutil/api_client.go:

   • As a related hardening measure, the retry logic for UpdateMetaLabel (a PD API call) is enhanced.
   • defaultMaxRetry is increased from 3 to 5.
   • Exponential backoff is added (WithBackoffBaseDelay(200), WithBackoffMaxDelay(4000)) to make this call more resilient to a busy or starting PD.

Why is this change necessary?

This change fixes a race condition during cluster startup, especially when using TiUP. TiUP may start the TiCDC process before the TiKV cluster has finished its bootstrapping procedure.

Previously, when TiCDC tried to initialize its connection to TiKV (via OpenWithOptions), this call would fail if TiKV was not ready. As there was no retry, the TiCDC process would crash immediately (e.g., ~300ms after starting, while TiKV might finish bootstrapping moments later).

By adding a robust retry with exponential backoff to CreateTiStore, TiCDC will now patiently wait for TiKV to become available, resolving this startup failure and making the deployment process much more stable.

Check List

Tests

• Unit test
• Integration test
• Manual test

Questions

Will it cause performance regression or break compatibility?

None

Do you need to update user documentation, design documentation or monitoring documentation?

None

Release note

Fix a potential TiCDC startup failure by retrying the TiKV storage connection, making it more resilient to delays in TiKV bootstrapping.

[gemini-code-assist]

Summary of Changes

Hello @wlwilliamx, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a critical race condition during TiCDC startup where it could crash if the TiKV cluster was not fully bootstrapped yet. By introducing a robust retry mechanism with exponential backoff for establishing the TiKV storage connection (CreateTiStore) and enhancing similar retry logic for PD API calls, TiCDC is now significantly more resilient to transient startup issues in a distributed environment. This change ensures a more stable and reliable deployment experience, particularly when using tools like TiUP.

Highlights

• TiKV Connection Retry: Implemented a retry mechanism with exponential backoff in CreateTiStore to handle delays in TiKV bootstrapping, making TiCDC startup more robust.
• Context Propagation: Propagated context.Context through various functions, including keyspaceManager.GetStorage and its callers, to support the new retry logic in CreateTiStore.
• PD API Call Resilience: Enhanced the retry logic for UpdateMetaLabel in pkg/pdutil/api_client.go by increasing max retries and adding exponential backoff, improving resilience to busy or starting PD instances.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a valuable retry mechanism to CreateTiStore, significantly improving TiCDC's startup resilience by handling delays in TiKV bootstrapping. The context propagation to support cancellation is well-implemented across the affected components. Additionally, the related hardening of the retry logic in UpdateMetaLabel is a good enhancement.

My review focuses on maintainability. I've identified some code duplication in the retry logic and constant definitions across pkg/upstream/upstream.go and pkg/pdutil/api_client.go. Centralizing these would make the retry policies more consistent and easier to manage in the future. Overall, this is a solid improvement to the stability of the system.

[wlwilliamx]

/retest

[wlwilliamx]

/test all

[wlwilliamx]

/test all

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hongyunyan, lidezhu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [hongyunyan,lidezhu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2025-10-28 10:08:01.953751307 +0000 UTC m=+1384788.031003857: ☑️ agreed by lidezhu.
• 2025-10-31 11:40:09.182109186 +0000 UTC m=+8060.024690861: ☑️ agreed by hongyunyan.