Checkout failure: "Git fetch failed with exit code 128"

[carlossanlop]

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results

Build leg reported

Libraries Test Run release coreclr Linux_musl x64 Release

Pull Request

Rolling build for runtime

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "Git fetch failed with exit code: 128",
   "BuildRetry": true
}

Additional information about the issue reported

The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:

remote: Compressing objects:  29% (30070/103687)        
remote: Compressing objects:  30% (31107/103687)        
remote: Compressing objects:  31% (32143/103687)        
remote: Compressing objects:  31% (32322/103687)        
remote: Compressing objects:  32% (33180/103687)        
remote: Compressing objects:  33% (34217/103687)        
remote: Compressing objects:  33% (35247/103687)        
remote: Compressing objects:  34% (35254/103687)        
remote: Compressing objects:  35% (36291/103687)        
remote: Compressing objects:  36% (37328/103687)        
remote: Compressing objects:  37% (38365/103687)        
remote: Compressing objects:  37% (38926/103687)        
remote: Compressing objects:  38% (39402/103687)        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s

Report

Build	Definition	Step Name	Console log	Pull Request
1145394	dotnet/runtime	Checkout dotnet/runtime@refs/pull/118541/merge to s	Log	dotnet/runtime#118541
1145384	dotnet/runtime	Checkout dotnet/runtime@refs/pull/118536/merge to s	Log	dotnet/runtime#118536
1145390	dotnet/runtime	Checkout dotnet/runtime@refs/pull/118539/merge to s	Log	dotnet/runtime#118539
1145388	dotnet/runtime	Checkout dotnet/runtime@refs/pull/118538/merge to s	Log	dotnet/runtime#118538
1144707	dotnet/runtime	Checkout dotnet/runtime@refs/pull/119338/merge to s	Log	dotnet/runtime#119338
1144585	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63624/merge to s	Log	dotnet/aspnetcore#63624
1144586	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63624/merge to s	Log	dotnet/aspnetcore#63624
1144148	dotnet/sdk	Checkout dotnet/sdk@refs/pull/50729/merge to s	Log	dotnet/sdk#50729
1143322	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63601/merge to s	Log	dotnet/aspnetcore#63601
1143290	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63600/merge to s	Log	dotnet/aspnetcore#63600
1142662	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63592/merge to s	Log	dotnet/aspnetcore#63592
1142478	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63589/merge to s	Log	dotnet/aspnetcore#63589
1139015	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63535/merge to s	Log	dotnet/aspnetcore#63535
1139016	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63535/merge to s	Log	dotnet/aspnetcore#63535
1135694	dotnet/runtime	Checkout dotnet/runtime@refs/pull/119224/merge to s	Log	dotnet/runtime#119224
1135272	dotnet/sdk	Checkout dotnet/sdk@refs/pull/50479/merge to s	Log	dotnet/sdk#50479
1135269	dotnet/runtime	Checkout dotnet/runtime@refs/pull/118805/merge to s	Log	dotnet/runtime#118805
1135246	dotnet/runtime	Checkout dotnet/runtime@refs/pull/118361/merge to s	Log	dotnet/runtime#118361
1134414	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63471/merge to s	Log	dotnet/aspnetcore#63471
1134415	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63471/merge to s	Log	dotnet/aspnetcore#63471
1134402	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63470/merge to s	Log	dotnet/aspnetcore#63470
1134403	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63470/merge to s	Log	dotnet/aspnetcore#63470
1134261	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63467/merge to s	Log	dotnet/aspnetcore#63467
1134262	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63467/merge to s	Log	dotnet/aspnetcore#63467
1134230	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63464/merge to s	Log	dotnet/aspnetcore#63464
1134231	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63464/merge to s	Log	dotnet/aspnetcore#63464
1132237	dotnet/sdk	Checkout dotnet/sdk@refs/pull/50493/merge to s	Log	dotnet/sdk#50493
1132084	dotnet/sdk	Checkout dotnet/sdk@refs/pull/45696/merge to s	Log	dotnet/sdk#45696
1131565	dotnet/runtime	Checkout dotnet/runtime@refs/pull/119069/merge to s	Log	dotnet/runtime#119069
1129697	dotnet/sdk	Checkout dotnet/sdk@refs/pull/50431/merge to s	Log	dotnet/sdk#50431
1129618	dotnet/runtime	Checkout dotnet/runtime@refs/pull/118928/merge to s	Log	dotnet/runtime#118928
1129027	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1129026	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1128979	dotnet/runtime	Checkout dotnet/runtime@refs/pull/118914/merge to s	Log	dotnet/runtime#118914
1129013	dotnet/installer	Checkout dotnet/installer@refs/pull/20557/merge to s	Log	dotnet/installer#20557
1128336	dotnet/runtime	Checkout dotnet/runtime@refs/pull/118918/merge to s	Log	dotnet/runtime#118918
1128068	dotnet/sdk	Checkout dotnet/sdk@refs/pull/50386/merge to s	Log	dotnet/sdk#50386
1127490	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/5044/merge to s	Log	dotnet/arcade-services#5044
2774984	dotnet-installer	Clone dotnet/installer	Log	#52428
1125805	dotnet/runtime	Checkout dotnet/runtime@refs/pull/118101/merge to s	Log	dotnet/runtime#118101
1125800	dotnet/runtime	Checkout dotnet/runtime@refs/pull/118551/merge to s	Log	dotnet/runtime#118551
1125630	dotnet/runtime	Checkout dotnet/runtime@refs/pull/118857/merge to s	Log	dotnet/runtime#118857
1125373	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63316/merge to s	Log	dotnet/aspnetcore#63316
1125374	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63316/merge to s	Log	dotnet/aspnetcore#63316
1125310	dotnet/sdk	Checkout dotnet/sdk@refs/pull/50336/merge to s	Log	dotnet/sdk#50336
1125312	dotnet/sdk	Checkout dotnet/sdk@refs/pull/50335/merge to s	Log	dotnet/sdk#50335
1125313	dotnet/sdk	Checkout dotnet/sdk@refs/pull/50333/merge to s	Log	dotnet/sdk#50333
1125249	dotnet/sdk	Checkout dotnet/sdk@refs/pull/50330/merge to s	Log	dotnet/sdk#50330
2773367	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/52382/merge to s	Log	#52382
2773315	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
1123910	dotnet/runtime	Checkout dotnet/runtime@refs/pull/118792/merge to s	Log	dotnet/runtime#118792
1123526	dotnet/sdk	Checkout dotnet/sdk@refs/pull/50287/merge to s	Log	dotnet/sdk#50287
1123258	dotnet/sdk	Checkout dotnet/sdk@refs/pull/50274/merge to s	Log	dotnet/sdk#50274
1122858	dotnet/runtime	Checkout dotnet/runtime@refs/pull/118709/merge to s	Log	dotnet/runtime#118709
1122789	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63135/merge to s	Log	dotnet/aspnetcore#63135
1122790	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63135/merge to s	Log	dotnet/aspnetcore#63135
1122791	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63135/merge to s	Log	dotnet/aspnetcore#63135
1122761	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16055/merge to s	Log	#16055
1122642	dotnet/sdk	Checkout dotnet/sdk@refs/pull/50263/merge to s	Log	dotnet/sdk#50263
1122191	dotnet/sdk	Checkout dotnet/sdk@refs/pull/50254/merge to s	Log	dotnet/sdk#50254
1121817	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63245/merge to s	Log	dotnet/aspnetcore#63245
1121820	dotnet/runtime	Checkout dotnet/runtime@refs/pull/118676/merge to s	Log	dotnet/runtime#118676
1121654	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63242/merge to s	Log	dotnet/aspnetcore#63242
1121655	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63242/merge to s	Log	dotnet/aspnetcore#63242
1121656	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63242/merge to s	Log	dotnet/aspnetcore#63242
2771206	dotnet-sdk	Checkout dotnet-sdk@refs/pull/52289/merge to s	Log	#52289
2771204	dotnet-sdk	Checkout dotnet-sdk@refs/pull/52287/merge to s	Log	#52287
2771205	dotnet-sdk	Checkout dotnet-sdk@refs/pull/52288/merge to s	Log	#52288
1121464	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63237/merge to s	Log	dotnet/aspnetcore#63237
1121248	dotnet/sdk	Checkout dotnet/sdk@refs/pull/50188/merge to s	Log	dotnet/sdk#50188

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
4	12	70

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960
Error message validated: Git fetch failed with exit code: 128
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:15:14 PM UTC

[ChadNedzlek]

So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

@ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.

[ulisesh]

I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.

[markwilkie]

So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)

[ulisesh]

I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

This is the query:

KnownIssues
| where IssueId == 9009 and IssueRepository == "dotnet/arcade"

[ViktorHofer]

This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?