xdsclient: create LRSClient at time of initialisation #8483

eshitachandwani · 2025-08-01T08:14:12Z

The race is in ReportLoad function of clientImpl. The implementation was recently changed as the part of xds client migration.

The comment says that lrsclient.LRSClient should be initialized only at creation time but that was not the case. It was being initialized at the time of calling ReportLoad function.

RELEASE NOTES:

lrsclient:
- Fix a race condition where the LRSClient was not initialized at creation time but it was being initialized at the time of calling the ReportLoad function.
- Creating an LRSClient no longer requires a node ID.

codecov · 2025-08-01T08:19:13Z

Codecov Report

❌ Patch coverage is 75.00000% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.04%. Comparing base (fa0d658) to head (c8c3ee4).
⚠️ Report is 6 commits behind head on master.

Files with missing lines	Patch %	Lines
internal/xds/xdsclient/clientimpl.go	80.00%	2 Missing and 1 partial ⚠️
internal/xds/clients/lrsclient/lrsclient.go	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8483      +/-   ##
==========================================
+ Coverage   81.86%   82.04%   +0.17%     
==========================================
  Files         412      412              
  Lines       40518    40465      -53     
==========================================
+ Hits        33172    33200      +28     
+ Misses       5953     5887      -66     
+ Partials     1393     1378      -15

Files with missing lines	Coverage Δ
internal/xds/xdsclient/clientimpl_loadreport.go	`76.92% <ø> (+4.19%)`	⬆️
internal/xds/clients/lrsclient/lrsclient.go	`73.11% <0.00%> (+0.48%)`	⬆️
internal/xds/xdsclient/clientimpl.go	`82.05% <80.00%> (-0.48%)`	⬇️

... and 26 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

xds/internal/xdsclient/tests/loadreport_test.go

xds/internal/xdsclient/clientimpl_loadreport.go

xds/internal/xdsclient/clientimpl.go

xds/internal/xdsclient/tests/loadreport_test.go

purnesh42H · 2025-08-01T18:30:08Z

xds/internal/xdsclient/clientimpl.go

+		TransportBuilder: gConfig.TransportBuilder,
+	})
+	if err != nil {
+		return nil, err


Now that we have moved it out of report load, we might have think a bit more here. Should error in lrs client creation be fatal? because not everyone is going to use internal xdsclient for load reporting.

In my opinion, we should keep the same behaviour as before the xDS client migration changes. If there are certain users who need to ignore LRS client creation failures, we can create a new issue to discuss if the bahviour changes makes sense.

I think it makes sense to retain the behavior of making LRS client creation failures be fatal. But we might want to change one minor thing in lrsclient.New. Currently it fails if node ID is empty in the configuration. We recently removed that check for the xDS client creation. I'm guessing other languages might not treat this as fatal for LRS creation.

@eshitachandwani : Could you please check what the other languages do and if required remove the check for empty node ID in lrsclient.New. Thanks.

If I understand correctly , Java is checking for not null here

That check is for the whole node proto or struct. Not just the node ID field.

Sorry , that was a mistake , I think this is where they check for node id not null in Java
https://github.com/grpc/grpc-java/blob/f50726d32e216746642513add28e086094ce5506/xds/src/main/java/io/grpc/xds/client/EnvoyProtoData.java#L79

That is a private constructor. It's being called from the builder below which sets the id to an empty string by default. In Java empty and null strings are different.

Ahh! Okay! Got it!

easwars · 2025-08-04T16:16:46Z

xds/internal/xdsclient/clientimpl.go

+		TransportBuilder: gConfig.TransportBuilder,
+	})
+	if err != nil {
+		return nil, err


I think it makes sense to retain the behavior of making LRS client creation failures be fatal. But we might want to change one minor thing in lrsclient.New. Currently it fails if node ID is empty in the configuration. We recently removed that check for the xDS client creation. I'm guessing other languages might not treat this as fatal for LRS creation.

@eshitachandwani : Could you please check what the other languages do and if required remove the check for empty node ID in lrsclient.New. Thanks.

xds/internal/xdsclient/clientimpl.go

easwars · 2025-08-04T16:20:48Z

xds/internal/xdsclient/tests/loadreport_test.go

+	for i := 0; i < numGoroutines; i++ {
+		go func() {
+			defer wg.Done()
+			_, cancelStore := client.ReportLoad(serverConfig)


Would it make sense to have a loop here as well?

I am not sure if I understand, loop for what?

A loop inside the goroutine to start reporting load and subsequently canceling it. What I'm asking for is:

for i := 0; i < numGoroutines; i++ { go func() { defer wg.Done() for j := 0; j < 100; j++ { _, cancelStore := client.ReportLoad(serverConfig) cancelStore(ctx) } }() }

You mean several ReportLoad() calls from one goroutine? Is this is better repro the real life case or to increase the chances of catching the race?

arjan-bal · 2025-08-21T07:57:45Z

xds/internal/clients/lrsclient/lrsclient.go

-	case config.Node.ID == "":
-		return nil, errors.New("lrsclient: node ID in node is empty")


I believe reverting this behaviour change should also be mentioned in the release notes.

The release notes should also mention the bug that is fixed by this PR.

xds/internal/xdsclient/tests/loadreport_test.go

arjan-bal

LGTM

easwars

LGTM, modulo minor nits

internal/xds/xdsclient/tests/loadreport_test.go

Fixes: grpc#8474 The race is in [ReportLoad](https://github.com/grpc/grpc-go/blob/9186ebd774370e3b3232d1b202914ff8fc2c56d6/xds/internal/xdsclient/clientimpl_loadreport.go#L35C2-L44C21) function of clientImpl. The implementation was recently changed as the part of [xds client migration](grpc@082a927). The [comment](https://github.com/grpc/grpc-go/blob/85240a5b02defe7b653ccba66866b4370c982b6a/xds/internal/xdsclient/clientimpl.go#L86C2-L87C16) says that `lrsclient.LRSClient` should be initialized only at creation time but that was not the case. It was being initialized at the time of calling `ReportLoad` function. RELEASE NOTES: - lrsclient: - Fix a race condition where the `LRSClient` was not initialized at creation time but it was being initialized at the time of calling the `ReportLoad` function. - Creating an `LRSClient` no longer requires a node ID.

Original PRs : #8476 , #8483 Related issues : #8473 , #8474 RELEASE NOTES: - xds: Revert to allowing empty node ID in xDS bootstrap configuration - lrsclient: - Fix a race condition where the LRSClient was not initialized at creation time but it was being initialized at the time of calling the ReportLoad function. - Creating an LRSClient no longer requires a node ID. --------- Co-authored-by: Sotiris Nanopoulos <[email protected]>

Fixes: grpc#8474 The race is in [ReportLoad](https://github.com/grpc/grpc-go/blob/9186ebd774370e3b3232d1b202914ff8fc2c56d6/xds/internal/xdsclient/clientimpl_loadreport.go#L35C2-L44C21) function of clientImpl. The implementation was recently changed as the part of [xds client migration](grpc@082a927). The [comment](https://github.com/grpc/grpc-go/blob/85240a5b02defe7b653ccba66866b4370c982b6a/xds/internal/xdsclient/clientimpl.go#L86C2-L87C16) says that `lrsclient.LRSClient` should be initialized only at creation time but that was not the case. It was being initialized at the time of calling `ReportLoad` function. RELEASE NOTES: - lrsclient: - Fix a race condition where the `LRSClient` was not initialized at creation time but it was being initialized at the time of calling the `ReportLoad` function. - Creating an `LRSClient` no longer requires a node ID.

Original PR: #8483 Related issue: #8474 RELEASE NOTES: - lrsclient: - Fix a race condition where the `LRSClient` was not initialized at creation time but it was being initialized at the time of calling the `ReportLoad` function. - Creating an `LRSClient` no longer requires a node ID.

Fixes: grpc#8474 The race is in [ReportLoad](https://github.com/grpc/grpc-go/blob/9186ebd774370e3b3232d1b202914ff8fc2c56d6/xds/internal/xdsclient/clientimpl_loadreport.go#L35C2-L44C21) function of clientImpl. The implementation was recently changed as the part of [xds client migration](grpc@082a927). The [comment](https://github.com/grpc/grpc-go/blob/85240a5b02defe7b653ccba66866b4370c982b6a/xds/internal/xdsclient/clientimpl.go#L86C2-L87C16) says that `lrsclient.LRSClient` should be initialized only at creation time but that was not the case. It was being initialized at the time of calling `ReportLoad` function. RELEASE NOTES: - lrsclient: - Fix a race condition where the `LRSClient` was not initialized at creation time but it was being initialized at the time of calling the `ReportLoad` function. - Creating an `LRSClient` no longer requires a node ID.

eshitachandwani added 2 commits July 30, 2025 13:40

create lrs at creation time

e035e08

test race

339ac57

eshitachandwani added this to the 1.75 Release milestone Aug 1, 2025

eshitachandwani added Type: Bug Area: xDS Includes everything xDS related, including LB policies used with xDS. labels Aug 1, 2025

arjan-bal reviewed Aug 1, 2025

View reviewed changes

xds/internal/xdsclient/tests/loadreport_test.go Outdated Show resolved Hide resolved

xds/internal/xdsclient/tests/loadreport_test.go Outdated Show resolved Hide resolved

eshitachandwani added 2 commits August 1, 2025 13:57

add waitgroup

8f53b14

error

56d3a58

eshitachandwani requested a review from arjan-bal August 1, 2025 09:13

eshitachandwani assigned arjan-bal Aug 1, 2025

eshitachandwani requested a review from easwars August 1, 2025 09:32

eshitachandwani assigned easwars Aug 1, 2025

arjan-bal reviewed Aug 1, 2025

View reviewed changes

xds/internal/xdsclient/clientimpl_loadreport.go Outdated Show resolved Hide resolved

xds/internal/xdsclient/clientimpl.go Outdated Show resolved Hide resolved

xds/internal/xdsclient/tests/loadreport_test.go Outdated Show resolved Hide resolved

arjan-bal assigned eshitachandwani and unassigned arjan-bal Aug 1, 2025

arjan-bal modified the milestones: 1.75 Release, 1.74 Release Aug 1, 2025

remove nil checks

2f6d3b4

eshitachandwani changed the title ~~create lrs at creation time~~ xds/xdsclient: create LRSClient at time of initialisation Aug 1, 2025

eshitachandwani requested a review from arjan-bal August 1, 2025 15:31

eshitachandwani assigned arjan-bal and unassigned eshitachandwani Aug 1, 2025

purnesh42H reviewed Aug 1, 2025

View reviewed changes

easwars reviewed Aug 4, 2025

View reviewed changes

easwars assigned eshitachandwani and unassigned easwars and arjan-bal Aug 4, 2025

format

90bc53c

eshitachandwani requested a review from easwars August 5, 2025 12:39

easwars assigned eshitachandwani and unassigned easwars and arjan-bal Aug 6, 2025

eshitachandwani added 2 commits August 13, 2025 22:06

optional node id

40674fc

optional node id

51309ec

eshitachandwani mentioned this pull request Aug 20, 2025

xdsclient: Fix race in SetWatchExpiryTimeoutForTesting #8526

Merged

e2e test

c0ebd99

arjan-bal reviewed Aug 21, 2025

View reviewed changes

eshitachandwani added 2 commits August 21, 2025 17:11

comments

b8905c8

merge

72427cb

eshitachandwani requested a review from arjan-bal August 21, 2025 11:51

eshitachandwani assigned arjan-bal and easwars and unassigned eshitachandwani Aug 21, 2025

arjan-bal approved these changes Aug 22, 2025

View reviewed changes

arjan-bal assigned eshitachandwani and unassigned arjan-bal Aug 22, 2025

easwars changed the title ~~xds/xdsclient: create LRSClient at time of initialisation~~ xdsclient: create LRSClient at time of initialisation Aug 25, 2025

easwars approved these changes Aug 25, 2025

View reviewed changes

internal/xds/xdsclient/tests/loadreport_test.go Outdated Show resolved Hide resolved

internal/xds/xdsclient/tests/loadreport_test.go Outdated Show resolved Hide resolved

easwars removed their assignment Aug 25, 2025

final comments

c8c3ee4

eshitachandwani merged commit 3074bcd into grpc:master Aug 26, 2025
15 checks passed

eshitachandwani mentioned this pull request Aug 26, 2025

Cherry pick #8476 and #8483 to v1.74.x #8540

Merged

eshitachandwani mentioned this pull request Aug 26, 2025

Cherry pick #8483 into v1.75.x #8541

Merged

		case config.Node.ID == "":
		return nil, errors.New("lrsclient: node ID in node is empty")

xdsclient: create LRSClient at time of initialisation #8483

xdsclient: create LRSClient at time of initialisation #8483

Uh oh!

Conversation

eshitachandwani commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

purnesh42H Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arjan-bal Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

arjan-bal left a comment

Choose a reason for hiding this comment

Uh oh!

easwars left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eshitachandwani commented Aug 1, 2025 •

edited

Loading

codecov bot commented Aug 1, 2025 •

edited

Loading

purnesh42H Aug 1, 2025 •

edited

Loading

arjan-bal Aug 4, 2025 •

edited

Loading