Skip to content

async ca/task runner: get rid of channel/etc id lists in events #21546

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Aug 13, 2025

Conversation

yumkam
Copy link
Collaborator

@yumkam yumkam commented Jul 23, 2025

Changelog entry

...

Changelog category

  • Not for changelog (changelog entry is not required)

Description for reviewers

Those lists are static, there are no point in passing them around in events, initialize them once.

Copy link

github-actions bot commented Jul 23, 2025

🟢 2025-08-04 14:32:26 UTC The validation of the Pull Request description is successful.

Copy link

github-actions bot commented Jul 23, 2025

2025-07-23 13:01:23 UTC Pre-commit check linux-x86_64-release-asan for 3d546de has started.
2025-07-23 13:02:45 UTC Artifacts will be uploaded here
2025-07-23 13:06:49 UTC ya make is running...
🟡 2025-07-23 14:58:39 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
14671 14266 0 118 264 23

🟢 2025-07-23 14:59:58 UTC Build successful.
🟡 2025-07-23 15:00:28 UTC ydbd size 3.9 GiB changed* by +116.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: f8b799d merge: 3d546de diff diff %
ydbd size 4 217 819 264 Bytes 4 217 938 952 Bytes +116.9 KiB +0.003%
ydbd stripped size 1 461 742 904 Bytes 1 461 697 496 Bytes -44.3 KiB -0.003%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 23, 2025

2025-07-23 13:02:47 UTC Pre-commit check linux-x86_64-relwithdebinfo for 3d546de has started.
2025-07-23 13:03:01 UTC Artifacts will be uploaded here
2025-07-23 13:06:25 UTC ya make is running...
🟡 2025-07-23 14:25:43 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
37119 34343 0 2 2741 33

2025-07-23 14:29:22 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-07-23 14:43:28 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
269 (only retried tests) 195 0 0 49 25

🟢 2025-07-23 14:43:36 UTC Build successful.
🟢 2025-07-23 14:43:57 UTC ydbd size 2.2 GiB changed* by +96.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: f8b799d merge: 3d546de diff diff %
ydbd size 2 398 833 496 Bytes 2 398 932 472 Bytes +96.7 KiB +0.004%
ydbd stripped size 501 614 408 Bytes 501 621 512 Bytes +6.9 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 23, 2025

2025-07-23 17:40:10 UTC Pre-commit check linux-x86_64-release-asan for 16a3fae has started.
2025-07-23 17:40:43 UTC Artifacts will be uploaded here
2025-07-23 17:44:36 UTC ya make is running...
🟡 2025-07-23 19:29:57 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
14671 14278 0 105 266 22

🟢 2025-07-23 19:31:14 UTC Build successful.
🟢 2025-07-23 19:31:44 UTC ydbd size 3.9 GiB changed* by -43.4 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 9911a2e merge: 16a3fae diff diff %
ydbd size 4 218 876 648 Bytes 4 218 832 216 Bytes -43.4 KiB -0.001%
ydbd stripped size 1 461 975 544 Bytes 1 461 964 760 Bytes -10.5 KiB -0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 23, 2025

2025-07-23 17:41:12 UTC Pre-commit check linux-x86_64-relwithdebinfo for 16a3fae has started.
2025-07-23 17:42:00 UTC Artifacts will be uploaded here
2025-07-23 17:45:50 UTC ya make is running...
🟡 2025-07-23 19:03:32 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
37117 34330 0 10 2742 35

2025-07-23 19:06:49 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-07-23 19:20:44 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
399 (only retried tests) 326 0 0 49 24

🟢 2025-07-23 19:20:54 UTC Build successful.
🟢 2025-07-23 19:21:15 UTC ydbd size 2.2 GiB changed* by -23.4 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 9911a2e merge: 16a3fae diff diff %
ydbd size 2 399 470 712 Bytes 2 399 446 792 Bytes -23.4 KiB -0.001%
ydbd stripped size 501 723 016 Bytes 501 720 136 Bytes -2.8 KiB -0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 30, 2025

2025-07-30 15:47:46 UTC Pre-commit check linux-x86_64-relwithdebinfo for 3379180 has started.
2025-07-30 15:47:59 UTC Artifacts will be uploaded here
2025-07-30 15:51:18 UTC ya make is running...
🟡 2025-07-30 17:07:54 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
37203 34419 0 5 2742 37

2025-07-30 17:11:29 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-07-30 17:25:20 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
334 (only retried tests) 257 0 0 50 27

🟢 2025-07-30 17:25:28 UTC Build successful.
🟢 2025-07-30 17:25:52 UTC ydbd size 2.2 GiB changed* by -22.2 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 4bb1e40 merge: 3379180 diff diff %
ydbd size 2 406 626 016 Bytes 2 406 603 296 Bytes -22.2 KiB -0.001%
ydbd stripped size 502 827 304 Bytes 502 824 872 Bytes -2.4 KiB -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 30, 2025

2025-07-30 15:48:08 UTC Pre-commit check linux-x86_64-release-asan for 3379180 has started.
2025-07-30 15:48:23 UTC Artifacts will be uploaded here
2025-07-30 15:51:39 UTC ya make is running...
🟡 2025-07-30 17:35:21 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
14724 14353 0 84 262 25

🟢 2025-07-30 17:36:41 UTC Build successful.
🟢 2025-07-30 17:37:09 UTC ydbd size 3.9 GiB changed* by -40.5 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 4bb1e40 merge: 3379180 diff diff %
ydbd size 4 231 272 112 Bytes 4 231 230 688 Bytes -40.5 KiB -0.001%
ydbd stripped size 1 465 482 424 Bytes 1 465 472 984 Bytes -9.2 KiB -0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Aug 4, 2025

2025-08-04 10:23:51 UTC Pre-commit check linux-x86_64-relwithdebinfo for b31349f has started.
2025-08-04 10:24:06 UTC Artifacts will be uploaded here
2025-08-04 10:25:08 UTC Check cancelled

Copy link

github-actions bot commented Aug 4, 2025

2025-08-04 10:28:20 UTC Pre-commit check linux-x86_64-release-asan for 8fdd79e has started.
2025-08-04 10:28:34 UTC Artifacts will be uploaded here
2025-08-04 10:32:04 UTC Check cancelled

Copy link

github-actions bot commented Aug 4, 2025

2025-08-04 10:28:37 UTC Pre-commit check linux-x86_64-relwithdebinfo for 8fdd79e has started.
2025-08-04 10:28:53 UTC Artifacts will be uploaded here
2025-08-04 10:32:04 UTC Check cancelled

Copy link

github-actions bot commented Aug 4, 2025

2025-08-04 10:33:22 UTC Pre-commit check linux-x86_64-relwithdebinfo for b3f297b has started.
2025-08-04 10:33:26 UTC Artifacts will be uploaded here
2025-08-04 10:36:54 UTC ya make is running...
🟡 2025-08-04 11:52:23 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
37255 34473 0 1 2750 31

2025-08-04 11:55:48 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-08-04 12:09:01 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
590 (only retried tests) 490 0 0 79 21

🟢 2025-08-04 12:09:10 UTC Build successful.
🟢 2025-08-04 12:09:31 UTC ydbd size 2.2 GiB changed* by -22.2 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: ebb49b5 merge: b3f297b diff diff %
ydbd size 2 410 910 288 Bytes 2 410 887 560 Bytes -22.2 KiB -0.001%
ydbd stripped size 503 474 728 Bytes 503 472 296 Bytes -2.4 KiB -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Aug 4, 2025

2025-08-04 10:36:01 UTC Pre-commit check linux-x86_64-release-asan for b3f297b has started.
2025-08-04 10:36:15 UTC Artifacts will be uploaded here
2025-08-04 10:39:37 UTC ya make is running...
🟡 2025-08-04 12:27:12 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
14770 14395 0 83 266 26

🟢 2025-08-04 12:28:41 UTC Build successful.
🟢 2025-08-04 12:29:08 UTC ydbd size 3.9 GiB changed* by -40.5 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: ebb49b5 merge: b3f297b diff diff %
ydbd size 4 238 515 512 Bytes 4 238 474 072 Bytes -40.5 KiB -0.001%
ydbd stripped size 1 467 457 752 Bytes 1 467 448 312 Bytes -9.2 KiB -0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@yumkam yumkam requested a review from Copilot August 4, 2025 11:23
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR refactors the task runner architecture to eliminate redundant channel/ID lists from events by initializing them once in the task runner actors instead of passing them around in events.

  • Removes channel/sink ID lists from event structures (TEvContinueRun, TEvStatistics, TWatermarkRequest, TCheckpointRequest)
  • Changes data structures from THashSet<ui32> to TVector<ui32> for input/output tracking
  • Initializes channel/sink lists during task runner setup instead of passing them in events

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
task_runner_actor.cpp Converts Inputs/Sources from THashSet to TVector, removes inputChannelsWithDisabledCheckpoints parameter
worker_actor.cpp Changes inputChannels from THashSet to TVector in worker implementation
task_runner_actor_local.cpp Major refactor to initialize all channel/sink lists locally and remove event-based ID passing
task_runner_actor.h Removes inputChannelsWithDisabledCheckpoints parameter from factory interface
events.h Removes ID list fields from event structures (TWatermarkRequest, TCheckpointRequest, TEvContinueRun, TEvStatistics)
dq_async_compute_actor.cpp Removes ID collection logic and simplifies event creation

yumkam added 2 commits August 4, 2025 17:23
This reverts commit 51e836e.
This reverts commit 2f27726.
Copy link

github-actions bot commented Aug 4, 2025

2025-08-04 14:35:26 UTC Pre-commit check linux-x86_64-relwithdebinfo for 20672cf has started.
2025-08-04 14:35:41 UTC Artifacts will be uploaded here
2025-08-04 14:39:05 UTC ya make is running...
🟢 2025-08-04 15:56:58 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
37255 34459 0 0 2761 35

🟢 2025-08-04 16:00:42 UTC Build successful.
🟢 2025-08-04 16:01:02 UTC ydbd size 2.2 GiB changed* by -22.3 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: df42da0 merge: 20672cf diff diff %
ydbd size 2 412 411 680 Bytes 2 412 388 880 Bytes -22.3 KiB -0.001%
ydbd stripped size 503 700 424 Bytes 503 697 928 Bytes -2.4 KiB -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Aug 4, 2025

2025-08-04 14:35:46 UTC Pre-commit check linux-x86_64-release-asan for 20672cf has started.
2025-08-04 14:36:18 UTC Artifacts will be uploaded here
2025-08-04 14:40:15 UTC ya make is running...
🟡 2025-08-04 16:26:18 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
14770 14370 0 100 278 22

🟢 2025-08-04 16:27:38 UTC Build successful.
🟢 2025-08-04 16:28:06 UTC ydbd size 3.9 GiB changed* by -40.5 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: dc3cbe4 merge: 20672cf diff diff %
ydbd size 4 240 987 080 Bytes 4 240 945 640 Bytes -40.5 KiB -0.001%
ydbd stripped size 1 468 108 504 Bytes 1 468 099 064 Bytes -9.2 KiB -0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@yumkam yumkam merged commit c87e40c into ydb-platform:main Aug 13, 2025
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants