Hosting issues at dl.k8s.io

[djpimp2010]

One of the services I manage uses kubectl in a gitlab pipeline, to update the cluster after a new docker container upload.

Here is the line from the pipeline bash script, to install the executable on the pipeline:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

A couple of months ago, I noticed at least a few times a day, pipelines were failing at this point - it looks as if the stable.txt request is the culprit, as the error states a malformed URL

Here is a copy of the error message:

$ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
curl: Failed to extract a sensible file name from the URL to use for storage!
curl: (3) URL using bad/illegal format or missing URL

Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

I am unsure if its maybe a DNS issue, or a load balancing issue, or other cause, but its happening often enough that I feel as a developer / dev/ops guy, I should inform whoever necessary to look into this issue - if it affects me, it affects others.

I am quite capable of wrapping the script in a try until this works function, but that's a half baked solution - when in reality, I should report this, and have the stability fixed for everyone.

Usually I can re-run the pipeline, and it works, so it's an intermittent issue, and I'm not sure how I would reproduce it. If necessary, I could possibly modify the gitlab.ci to run the commands separately and log the output etc, to try and find the root cause if it happens again.

[BenTheElder]

It would be helpful if we could see what the bad format was.

It's possibly that you're occasionally having a networking flake in your environment (or in our content provider ...) that causes the curl -L -s https://dl.k8s.io/release/stable.txt to error, causing the bad URL issue in the second curl?

one solution is to split those calls and add retries.

Right now there's not really anything we can actionably follow-up on.

dl.k8s.io is currently a small redirect on a simple nginx instance to a fastly CDN. From this report, we can't tell which had the issue.

maybe you can increase curl's verbosity for some time and report back?

[BenTheElder]

/sig k8s-infra
/triage needs-information

[djpimp2010]

Yes of course - I was posting this in the hopes that there would be someone to view the logs - otherwise no point in logging the extra data

Thanks for your response

[BenTheElder]

Yes of course - I was posting this in the hopes that there would be someone to view the logs - otherwise no point in logging the extra data

We do have some logs but the volume of requests is quite high, and there are multiple different services involved here. So anything we can do to narrow it down first helps.

[xmudrii]

I think this is another occurrence of this issue: kubernetes/release#4053
In this issue, it's happening in the GitHub Runners as far as I can tell.

[BenTheElder]

I think this is another occurrence of this issue: kubernetes/release#4053

That doesn't seem right, the failures here are purely with curl-ing, the GPG keys should not be involved, nor the apt/rpm host.

[xmudrii]

@BenTheElder My assumption on that issue, given logs that we have (we would need more logs to confirm), is that curl -fsSL --retry 5 --retry-delay 10 "https://dl.k8s.io/release/stable.txt" is failing, hence the GPG key URL is malformed and getting the key fails, so GPG fails later on. So I believe it could be relevant, as this issue is about the same URL (https://dl.k8s.io/release/stable.txt).

[BenTheElder]

hence the GPG key URL is malformed and getting the key fails, so GPG fails later on. So I believe it could be relevant, as this issue is about the same URL (https://dl.k8s.io/release/stable.txt).

That's not a GPG key. stable.text serves a plain-text version number.

It's on the fastly CDN infrastructure, totally unrelated to pkgs.k8s.io

[BenTheElder]

I see, in kubernetes/release#4053 they may fail to get a GPG key as a side-effect of this issue.

But the GPG key bit would be noise in that case, and this would be a more appropriate tracking issue, since the root bug has nothing to do with GPG verification.

We also still need confirmation in kubernetes/release#4053 to clarify the root-cause with their script. Leaving some suggestions.

[BenTheElder]

So, back on the original issue, we'll need some idea if this is happening in the 3xx redirect to the fastly CDN, or in the fastly CDN.