Skip to content

YQ-4314 supported script executions retries #21592

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

GrigoriyPA
Copy link
Collaborator

Changelog entry

Supported script executions retries (part of streaming queries support)

Changelog category

  • Not for changelog (changelog entry is not required)

Description for reviewers

@GrigoriyPA GrigoriyPA requested review from Hor911 and kardymonds July 23, 2025 20:00
@GrigoriyPA GrigoriyPA requested review from a team as code owners July 23, 2025 20:00
Copy link

🟢 2025-07-23 20:03:42 UTC The validation of the Pull Request description is successful.

Copy link

github-actions bot commented Jul 23, 2025

2025-07-23 20:04:39 UTC Pre-commit check linux-x86_64-relwithdebinfo for ac5ddf5 has started.
2025-07-23 20:04:52 UTC Artifacts will be uploaded here
2025-07-23 20:08:33 UTC ya make is running...
🔴 2025-07-23 20:58:29 UTC Build failed, see the logs. Also see fail summary

Copy link

github-actions bot commented Jul 23, 2025

2025-07-23 20:04:47 UTC Pre-commit check linux-x86_64-release-asan for ac5ddf5 has started.
2025-07-23 20:05:01 UTC Artifacts will be uploaded here
2025-07-23 20:08:56 UTC ya make is running...
🔴 2025-07-23 21:18:39 UTC Build failed, see the logs. Also see fail summary
🟡 2025-07-23 21:19:08 UTC ydbd size 3.9 GiB changed* by +734.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 9911a2e merge: ac5ddf5 diff diff %
ydbd size 4 218 876 648 Bytes 4 219 629 040 Bytes +734.8 KiB +0.018%
ydbd stripped size 1 461 975 544 Bytes 1 462 196 888 Bytes +216.2 KiB +0.015%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@GrigoriyPA GrigoriyPA marked this pull request as draft July 24, 2025 08:48
Copy link

github-actions bot commented Jul 24, 2025

2025-07-24 09:31:40 UTC Pre-commit check linux-x86_64-relwithdebinfo for 5d5b701 has started.
2025-07-24 09:31:53 UTC Artifacts will be uploaded here
2025-07-24 09:35:40 UTC ya make is running...
🟡 2025-07-24 11:31:08 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39076 36292 0 11 2741 32

2025-07-24 11:34:50 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-07-24 11:47:01 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
290 (only retried tests) 209 0 7 49 25

2025-07-24 11:47:12 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-07-24 11:59:08 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
274 (only retried tests) 194 0 6 49 25

🟢 2025-07-24 11:59:18 UTC Build successful.
🟡 2025-07-24 11:59:39 UTC ydbd size 2.2 GiB changed* by +429.0 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 98dcc6f merge: 5d5b701 diff diff %
ydbd size 2 399 453 752 Bytes 2 399 893 032 Bytes +429.0 KiB +0.018%
ydbd stripped size 501 725 000 Bytes 501 808 904 Bytes +81.9 KiB +0.017%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 24, 2025

2025-07-24 09:32:35 UTC Pre-commit check linux-x86_64-release-asan for 5d5b701 has started.
2025-07-24 09:33:08 UTC Artifacts will be uploaded here
2025-07-24 09:37:38 UTC ya make is running...
🟡 2025-07-24 12:09:05 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16582 16168 0 112 280 22

🟢 2025-07-24 12:10:34 UTC Build successful.
🟡 2025-07-24 12:11:05 UTC ydbd size 3.9 GiB changed* by +738.7 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 98dcc6f merge: 5d5b701 diff diff %
ydbd size 4 218 853 096 Bytes 4 219 609 536 Bytes +738.7 KiB +0.018%
ydbd stripped size 1 461 983 960 Bytes 1 462 209 400 Bytes +220.2 KiB +0.015%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 24, 2025

2025-07-24 14:43:54 UTC Pre-commit check linux-x86_64-relwithdebinfo for c2c48bb has started.
2025-07-24 14:44:08 UTC Artifacts will be uploaded here
2025-07-24 14:48:00 UTC ya make is running...
🟡 2025-07-24 16:45:27 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39102 36318 0 4 2747 33

2025-07-24 16:48:51 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-07-24 17:03:57 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
388 (only retried tests) 302 0 0 58 28

🟢 2025-07-24 17:04:06 UTC Build successful.
🟡 2025-07-24 17:04:29 UTC ydbd size 2.2 GiB changed* by +427.2 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 4a2cd65 merge: c2c48bb diff diff %
ydbd size 2 399 773 416 Bytes 2 400 210 840 Bytes +427.2 KiB +0.018%
ydbd stripped size 501 804 424 Bytes 501 884 776 Bytes +78.5 KiB +0.016%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 24, 2025

2025-07-24 14:44:10 UTC Pre-commit check linux-x86_64-release-asan for c2c48bb has started.
2025-07-24 14:44:25 UTC Artifacts will be uploaded here
2025-07-24 14:48:18 UTC ya make is running...
🟡 2025-07-24 17:20:40 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16596 16197 0 107 267 25

🟢 2025-07-24 17:22:11 UTC Build successful.
🟡 2025-07-24 17:22:41 UTC ydbd size 3.9 GiB changed* by +742.6 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 4a2cd65 merge: c2c48bb diff diff %
ydbd size 4 219 461 272 Bytes 4 220 221 696 Bytes +742.6 KiB +0.018%
ydbd stripped size 1 462 224 920 Bytes 1 462 451 640 Bytes +221.4 KiB +0.016%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 24, 2025

2025-07-24 18:09:21 UTC Pre-commit check linux-x86_64-release-asan for dc7b391 has started.
2025-07-24 18:09:53 UTC Artifacts will be uploaded here
2025-07-24 18:14:12 UTC ya make is running...
2025-07-24 18:57:07 UTC Check cancelled

Copy link

github-actions bot commented Jul 24, 2025

2025-07-24 18:11:33 UTC Pre-commit check linux-x86_64-relwithdebinfo for dc7b391 has started.
2025-07-24 18:11:46 UTC Artifacts will be uploaded here
2025-07-24 18:15:27 UTC ya make is running...
2025-07-24 18:57:06 UTC Check cancelled

Copy link

github-actions bot commented Jul 24, 2025

2025-07-24 19:00:05 UTC Pre-commit check linux-x86_64-relwithdebinfo for bd22a25 has started.
2025-07-24 19:00:18 UTC Artifacts will be uploaded here
2025-07-24 19:03:59 UTC ya make is running...
🟡 2025-07-24 20:18:12 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39182 36388 0 3 2754 37

2025-07-24 20:21:34 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-07-24 20:33:40 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
393 (only retried tests) 317 0 0 50 26

🟢 2025-07-24 20:33:48 UTC Build successful.
🟡 2025-07-24 20:34:07 UTC ydbd size 2.2 GiB changed* by +427.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 336854d merge: bd22a25 diff diff %
ydbd size 2 399 781 488 Bytes 2 400 219 264 Bytes +427.5 KiB +0.018%
ydbd stripped size 501 806 344 Bytes 501 886 760 Bytes +78.5 KiB +0.016%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 24, 2025

2025-07-24 19:00:36 UTC Pre-commit check linux-x86_64-release-asan for bd22a25 has started.
2025-07-24 19:00:50 UTC Artifacts will be uploaded here
2025-07-24 19:04:46 UTC ya make is running...
🟡 2025-07-24 21:01:37 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16675 16218 0 164 266 27

🟢 2025-07-24 21:03:03 UTC Build successful.
🟡 2025-07-24 21:03:34 UTC ydbd size 3.9 GiB changed* by +739.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 336854d merge: bd22a25 diff diff %
ydbd size 4 219 444 208 Bytes 4 220 201 392 Bytes +739.4 KiB +0.018%
ydbd stripped size 1 462 230 744 Bytes 1 462 453 496 Bytes +217.5 KiB +0.015%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 25, 2025

2025-07-25 16:44:41 UTC Pre-commit check linux-x86_64-relwithdebinfo for 1eab430 has started.
2025-07-25 16:44:45 UTC Artifacts will be uploaded here
2025-07-25 16:48:30 UTC ya make is running...
🟡 2025-07-25 18:39:38 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39206 36418 0 5 2753 30

2025-07-25 18:42:59 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-07-25 18:54:57 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
632 (only retried tests) 524 0 0 85 23

🟢 2025-07-25 18:55:06 UTC Build successful.
🟡 2025-07-25 18:55:25 UTC ydbd size 2.2 GiB changed* by +724.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: c7b04df merge: 1eab430 diff diff %
ydbd size 2 403 019 288 Bytes 2 403 761 176 Bytes +724.5 KiB +0.031%
ydbd stripped size 502 194 952 Bytes 502 317 512 Bytes +119.7 KiB +0.024%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 25, 2025

2025-07-25 16:44:54 UTC Pre-commit check linux-x86_64-release-asan for 1eab430 has started.
2025-07-25 16:44:58 UTC Artifacts will be uploaded here
2025-07-25 16:48:39 UTC ya make is running...
🟡 2025-07-25 19:15:36 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16697 16277 0 118 277 25

🟢 2025-07-25 19:17:05 UTC Build successful.
🟡 2025-07-25 19:17:38 UTC ydbd size 3.9 GiB changed* by +1.2 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: c7b04df merge: 1eab430 diff diff %
ydbd size 4 224 881 704 Bytes 4 226 136 168 Bytes +1.2 MiB +0.030%
ydbd stripped size 1 463 379 928 Bytes 1 463 723 160 Bytes +335.2 KiB +0.023%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@GrigoriyPA GrigoriyPA marked this pull request as ready for review July 28, 2025 09:20
@GrigoriyPA GrigoriyPA requested a review from a team as a code owner July 28, 2025 09:20
@GrigoriyPA GrigoriyPA requested a review from Copilot July 28, 2025 09:22
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements script execution retries as part of streaming queries support. The key change is adding retry capability to script executions that fail with specific status codes, allowing them to be automatically restarted after a configured backoff period.

  • Adds retry mapping functionality to script execution operations with configurable backoff policies
  • Implements background lease checking to automatically restart failed script executions
  • Enhances script execution metadata to track retry state and lease generation

Reviewed Changes

Copilot reviewed 55 out of 56 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
ydb/tests/tools/kqprun/src/ydb_setup.h Defines TScriptRequest structure with retry mapping support
ydb/tests/tools/kqprun/src/ydb_setup.cpp Updates ScriptRequest method to accept TScriptRequest instead of TRequestOptions
ydb/tests/tools/kqprun/src/kqp_runner.cpp Implements retry policy configuration and script execution retry logic
ydb/core/kqp/proxy_service/kqp_script_execution_retries.h Defines retry policy and limiter classes for script executions
ydb/core/kqp/proxy_service/kqp_script_execution_retries.cpp Implements retry logic with rate limiting and backoff calculations
ydb/core/kqp/finalize_script_service/kqp_finalize_script_service.cpp Adds background lease checking functionality
ydb/core/kqp/finalize_script_service/kqp_check_script_lease_actor.cpp Implements periodic lease expiration checks
ydb/core/protos/kqp.proto Adds retry state and policy protobuf definitions

Copy link

github-actions bot commented Jul 28, 2025

2025-07-28 09:35:42 UTC Pre-commit check linux-x86_64-release-asan for 2dc6175 has started.
2025-07-28 09:35:56 UTC Artifacts will be uploaded here
2025-07-28 09:39:43 UTC ya make is running...
🟡 2025-07-28 12:09:54 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16700 16315 0 95 271 19

🟢 2025-07-28 12:11:26 UTC Build successful.
🟡 2025-07-28 12:11:54 UTC ydbd size 3.9 GiB changed* by +1.2 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 9bbe17e merge: 2dc6175 diff diff %
ydbd size 4 225 099 416 Bytes 4 226 365 472 Bytes +1.2 MiB +0.030%
ydbd stripped size 1 463 473 912 Bytes 1 463 823 288 Bytes +341.2 KiB +0.024%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 28, 2025

2025-07-28 09:35:43 UTC Pre-commit check linux-x86_64-relwithdebinfo for 2dc6175 has started.
2025-07-28 09:35:57 UTC Artifacts will be uploaded here
2025-07-28 09:39:44 UTC ya make is running...
🟡 2025-07-28 11:42:05 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39215 36431 0 3 2744 37

2025-07-28 11:45:43 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-07-28 11:58:10 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
416 (only retried tests) 342 0 0 50 24

🟢 2025-07-28 11:58:19 UTC Build successful.
🟡 2025-07-28 11:58:39 UTC ydbd size 2.2 GiB changed* by +736.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 9bbe17e merge: 2dc6175 diff diff %
ydbd size 2 403 142 112 Bytes 2 403 896 088 Bytes +736.3 KiB +0.031%
ydbd stripped size 502 224 904 Bytes 502 356 424 Bytes +128.4 KiB +0.026%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 28, 2025

2025-07-28 15:30:09 UTC Pre-commit check linux-x86_64-relwithdebinfo for e1d5373 has started.
2025-07-28 15:30:38 UTC Artifacts will be uploaded here
2025-07-28 15:35:01 UTC ya make is running...
🟡 2025-07-28 17:32:57 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39216 36423 0 2 2756 35

2025-07-28 17:36:42 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-07-28 17:49:15 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
523 (only retried tests) 449 0 1 50 23

2025-07-28 17:49:26 UTC ya make is running... (failed tests rerun, try 3)
🟢 2025-07-28 18:01:17 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
262 (only retried tests) 189 0 0 50 23

🟢 2025-07-28 18:01:26 UTC Build successful.
🟡 2025-07-28 18:01:48 UTC ydbd size 2.2 GiB changed* by +746.6 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: b890bf6 merge: e1d5373 diff diff %
ydbd size 2 403 118 992 Bytes 2 403 883 504 Bytes +746.6 KiB +0.032%
ydbd stripped size 502 235 592 Bytes 502 372 680 Bytes +133.9 KiB +0.027%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 28, 2025

2025-07-28 15:32:03 UTC Pre-commit check linux-x86_64-release-asan for e1d5373 has started.
2025-07-28 15:32:47 UTC Artifacts will be uploaded here
2025-07-28 15:36:29 UTC ya make is running...
🟡 2025-07-28 18:03:30 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16701 16310 0 103 264 24

🟢 2025-07-28 18:04:57 UTC Build successful.
🟡 2025-07-28 18:05:26 UTC ydbd size 3.9 GiB changed* by +1.2 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: b890bf6 merge: e1d5373 diff diff %
ydbd size 4 225 100 208 Bytes 4 226 372 144 Bytes +1.2 MiB +0.030%
ydbd stripped size 1 463 512 888 Bytes 1 463 861 528 Bytes +340.5 KiB +0.024%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@GrigoriyPA GrigoriyPA force-pushed the YQ-4314-support-script-executions-retries branch from 74f3b10 to 115922f Compare July 29, 2025 06:53
Copy link

github-actions bot commented Jul 29, 2025

2025-07-29 06:57:42 UTC Pre-commit check linux-x86_64-release-asan for 7f3f2ea has started.
2025-07-29 06:57:58 UTC Artifacts will be uploaded here
2025-07-29 07:01:53 UTC ya make is running...
🟡 2025-07-29 09:35:23 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16702 16308 0 99 274 21

🟢 2025-07-29 09:36:57 UTC Build successful.
🟡 2025-07-29 09:37:26 UTC ydbd size 3.9 GiB changed* by +1.2 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: bf0be41 merge: 7f3f2ea diff diff %
ydbd size 4 225 339 112 Bytes 4 226 615 160 Bytes +1.2 MiB +0.030%
ydbd stripped size 1 463 554 136 Bytes 1 463 906 872 Bytes +344.5 KiB +0.024%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 29, 2025

2025-07-29 06:57:42 UTC Pre-commit check linux-x86_64-relwithdebinfo for 7f3f2ea has started.
2025-07-29 06:57:56 UTC Artifacts will be uploaded here
2025-07-29 07:01:48 UTC ya make is running...
🟢 2025-07-29 08:59:33 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39219 36441 0 0 2742 36

🟢 2025-07-29 09:03:02 UTC Build successful.
🟡 2025-07-29 09:03:19 UTC ydbd size 2.2 GiB changed* by +746.6 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: bf0be41 merge: 7f3f2ea diff diff %
ydbd size 2 403 276 168 Bytes 2 404 040 664 Bytes +746.6 KiB +0.032%
ydbd stripped size 502 254 664 Bytes 502 391 752 Bytes +133.9 KiB +0.027%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link
Collaborator

@Gazizonoki Gazizonoki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CLI - ok

@GrigoriyPA GrigoriyPA merged commit 321dc7b into ydb-platform:main Jul 30, 2025
14 checks passed
@GrigoriyPA GrigoriyPA deleted the YQ-4314-support-script-executions-retries branch July 30, 2025 08:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants