Github Enterprise SSH handshake failing

[steveizzle]

Describe the bug

We use flux2 with our github enterprise instance. This worked fine until the 0.28.3 Release. Since the Update we cannot fetch our repos anymore and get the following error message:

k get gitrepositories -n flux-system

NAME         URL                                                              AGE   READY   STATUS
flux2-sync   ssh://git@foo/abc/def   64d   False   failed to checkout and determine revision: unable to clone ' ssh://git@foo/abc/def': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

With version 0.27.0 everything is fine again, without any other change to the environment (same keys and so on):

k get gitrepositories -n flux-system 

NAME         URL                                                              AGE   READY   STATUS
flux2-sync   ssh://git@foo/abc/def   64d   True    Fetched revision: main/4afa5937b48b12a42c0739c13870c282826b3979

If you need more information, please let me know. Unfortunately the source-controller log also doesnt give me more than the above error message

Steps to reproduce

We installed flux with the community helm charts without any relevant change to the values.yaml

Expected behavior

It is working again ^^

Screenshots and recordings

No response

OS / Distro

N/A

Flux version

N/A

Flux check

► checking prerequisites
✗ flux 0.24.1 <0.28.5 (new version is available, please upgrade) //cli is not used
✔ Kubernetes 1.21.5-eks-bc4871b >=1.19.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.18.2
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.21.1
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.17.1
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.22.2
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.23.1
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.22.3
✔ all checks passed

Git provider

GitHub Enterprise

Container Registry provider

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[steveizzle]

It seems that the bug was introduced in v0.22.0 of the sourcecontroller. Should i open the bug there?

[pjbgf]

@steveizzle thank you for reporting this. Do you mind sharing what gitImplementation are you using?

[pjbgf]

Can you also confirm whether this is consistent and every single reconcile yields the error mentioned, or is intermittent and happens every so often?

Would you mind running the latest version of source-controller and enabling the experimental transport to see whether the issue persists? Here are more information about it: #636 (comment)

[steveizzle]

@steveizzle thank you for reporting this. Do you mind sharing what gitImplementation are you using?

I used go-git and tried now libgit2. With libgit2 it is working.

Can you also confirm whether this is consistent and every single reconcile yields the error mentioned, or is intermittent and happens every so often?

Would you mind running the latest version of source-controller and enabling the experimental transport to see whether the issue persists? Here are more information about it: #636 (comment)

Sure, i will give it a try and report back

[steveizzle]

Can you also confirm whether this is consistent and every single reconcile yields the error mentioned, or is intermittent and happens every so often?

Would you mind running the latest version of source-controller and enabling the experimental transport to see whether the issue persists? Here are more information about it: #636 (comment)

The error seems to be consistent.

As soon as i enable the experimental transport neither libgit2 nor go-git is working:

        env:
        - name: EXPERIMENTAL_GIT_TRANSPORT
          value: "true"
        - name: RUNTIME_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: ghcr.io/fluxcd/source-controller:v0.22.5

We found out that one repository is using an ED25519 Key which seems to be still working. Normally we use RSA keys.

[pjbgf]

Thanks for that. Do you mind sharing the errors you get when using libgit2 and the experimental transport? The pod logs may have some extra information that could be useful.

We are getting the same error suddenly after redeploying a test cluster that we deploy often (almost every day) and doesn't give any errors with Flux.

Is there any resolution for this, or just keep trying until it works?

[steveizzle]

I currently pinned an old version, but will spent some time on getting more debug Infos the next days

[pjbgf]

@steveizzle @campanellaf this may have to do with some of the crypto level settings that may have changed upstream.
So to confirm, you are using RSA for your authentication keypair, right? What is the algorithm used for the known_host you are providing?

We have just released a RC with some improvements specially around the libgit2 implementation. Would you mind to also checking whether it fixes your problem?
ghcr.io/fluxcd/source-controller:rc-4b3e0f9a

[hazim1093]

The issue still exists on v0.30.2.

@pjbgf I tried your RC with the go-git implementation, but no luck.
The algorithm we're using for known_hosts is ecdsa-sha2-nistp256

[steveizzle]

@pjbgf Regarding the rc, Unfortunately the rc didnt fix the problem:

This is a example of a git repository in my test cluster:

k get deploy -n flux-system  -o wide 

helm-controller               1/1     1            1           132d   manager      ghcr.io/fluxcd/helm-controller:v0.21.0                   app=helm-controller
image-automation-controller   1/1     1            1           132d   manager      ghcr.io/fluxcd/image-automation-controller:rc-48bcca59   app=image-automation-controller
image-reflector-controller    1/1     1            1           132d   manager      ghcr.io/fluxcd/image-reflector-controller:v0.18.0        app=image-reflector-controller
kustomize-controller          1/1     1            1           132d   manager      ghcr.io/fluxcd/kustomize-controller:v0.25.0              app=kustomize-controller
notification-controller       1/1     1            1           132d   manager      ghcr.io/fluxcd/notification-controller:v0.23.5           app=notification-controller
source-controller             1/1     1            1           132d   manager      ghcr.io/fluxcd/source-controller:rc-4b3e0f9a             app=source-controller

k get gitrepositories.source.toolkit.fluxcd.io -n flux-system flux2-sync -o yaml

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  annotations:
    meta.helm.sh/release-name: flux2-sync
    meta.helm.sh/release-namespace: flux-system
    reconcile.fluxcd.io/requestedAt: "2022-06-02T17:34:40.013251+02:00"
  creationTimestamp: "2022-01-21T12:20:41Z"
  finalizers:
  - finalizers.fluxcd.io
  generation: 9
  labels:
    app.kubernetes.io/instance: flux-system
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: flux
    helm.sh/chart: flux2-sync-0.4.0
  name: flux2-sync
  namespace: flux-system
  resourceVersion: "556712592"
  uid: 8f76a6b4-cf42-49e1-af14-9608a1dc25c2
spec:
  gitImplementation: libgit2
  interval: 5m0s
  ref:
    branch: main
  secretRef:
    name: flux2-secret
  timeout: 20s
  url: ssh://git@SECRET_ENTERPRISE_URL
status:
  conditions:
  - lastTransitionTime: "2022-06-02T15:16:18Z"
    message: no artifact for resource in storage
    observedGeneration: 9
    reason: NoArtifact
    status: "True"
    type: Reconciling
  - lastTransitionTime: "2022-06-02T15:32:34Z"
    message: 'failed to checkout and determine revision: unable to fetch-connect to
      remote ''ssh://SECRET_ENTERPRISE_URL'': ssh:
      handshake failed: ssh: unable to authenticate, attempted methods [none publickey],
      no supported methods remain'
    observedGeneration: 9
    reason: GitOperationFailed
    status: "False"
    type: Ready
  - lastTransitionTime: "2022-06-02T15:32:34Z"
    message: 'failed to checkout and determine revision: unable to fetch-connect to
      remote ''ssh://SECRET_ENTERPRISE_URL'': ssh:
      handshake failed: ssh: unable to authenticate, attempted methods [none publickey],
      no supported methods remain'
    observedGeneration: 9
    reason: GitOperationFailed
    status: "True"
    type: FetchFailed
  lastHandledReconcileAt: "2022-06-02T17:34:40.013251+02:00"
  observedGeneration: 8

[steveizzle]

I also checked out the algorithms for the known_hosts and got the following:

Working:

• gitrepo1:
  • known_hosts: 2 entries; keyA with ssh-rsa and keyB with ecdsa-sha2-nistp256;
  • key: ssh-ed25519
• gitrepo2:
  • known_hosts: 1 entry; keyB with ecdsa-sha2-nistp256
  • key: ecdsa-sha2-nistp384

Not Working:

• gitrepo3:
  • known_hosts: 1 entry; keyB with ecdsa-sha2-nistp256
  • identity key: ssh-rsa
• gitrepo4:
  • known_hosts: 2 entries; keyA with ssh-rsa and keyB with ecdsa-sha2-nistp256
  • identity key: ssh-rsa

So i would say its not related to the known_hosts.. And still everything is working when i switch the source controller back to v0.21.2

[steveizzle]

I did not set the Experimental env Variable in the test with the rc. Is this still necessary with the rc?

[stefanprodan]

@steveizzle we have released source-controller yesterday with managed transport enabled by default. The official image is here: https://github.com/fluxcd/source-controller/pkgs/container/source-controller/23808902?tag=v0.25.0

Please test it with go-git and libgit2, no env var needed.