Skip to content

Conversation

@lixmal
Copy link
Collaborator

@lixmal lixmal commented Jun 19, 2025

Describe your changes

NetBird SSH Client

  • Port forwarding
  • Windows support
  • Non-interactive commands
  • Single command execution over SSH

SSH Server

  • SFTP
  • Port forwarding (without user switching)
  • PTY (interactive, non-interactive)
  • Non-PTY (commands)
  • Windows support
  • JWT auth (user identity instead of machine identity, can be turned off)

New Flags

--enable-ssh-local-port-forwarding
--enable-ssh-remote-port-forwarding
--enable-ssh-root
--enable-ssh-sftp
--disable-ssh-auth

UI

  • Better organization
image image image

Changes

  • Default port changed to 22022
  • Redirect port 22 to 22022 when SSH server enabled
  • Remove implicit OpenSSH firewall port
  • Management now passes its jwks config to peers with enabled ssh server
  • SSH server peers verify incoming clients' jwt with max token age
  • SSH clients request jwts from the IdP and send these for authentication with remote ssh peers
  • Add netbird ssh detect command to detect if the remote peer is running the NetBird server
  • Add netbird ssh proxy command for native ssh and sftp clients. The proxy requests jwts just like netbird ssh, passes them for authentication and bridges the connection between the native client and the remote server.
  • Add jwt capability to the wasm ssh client

Auth Flows

  1. NetBird SSH Flow (netbird ssh <peer>)
flowchart TD
    A[User: netbird ssh &lt;peer&gt;] --> B[Connect to peer:22]
    B --> C[Detect Server Type]
    C --> D[Send 'netbird-detect' request]

    D --> E{Server Response}

    E -->|No NetBird identifier| F[Regular SSH Server]
    E -->|NetBird + JWT required| G[NetBird with JWT Auth]
    E -->|NetBird + No JWT| H[NetBird without JWT]

    F --> I[Standard SSH Connection]
    H --> I

    G --> J[Request JWT from NetBird daemon]

    J --> J1{Check JWT cache}
    J1 -->|Cache valid| J2[Use cached JWT token]
    J1 -->|No cache/expired| J3[OIDC flow:<br/>User authorizes via IDP callback]
    J3 --> J4[Receive & cache JWT token]
    J4 --> K[Connect to peer SSH server]
    J2 --> K

    K --> M[Send JWT authentication request]
    M --> N{JWT Valid?}

    N -->|No| O[Connection Rejected]
    N -->|Yes| P[SSH Session Established]
    I --> P

    P --> Q[Interactive Shell / Execute Command / Port Forwarding]

    style G fill:#f57c00,color:#fff
    style J fill:#1976d2,color:#fff
    style J1 fill:#1976d2,color:#fff
    style J3 fill:#e65100,color:#fff
    style P fill:#388e3c,color:#fff
Loading
  1. Native SSH Flow (e.g. openssh client)
flowchart TD
    A[User: ssh &lt;peer&gt;] --> B[OpenSSH loads config:<br/>/etc/ssh/ssh_config.d/99-netbird.conf]
    B --> D{Host matches NetBird pattern?}

    D -->|No| E[Standard SSH connection]

    D -->|Yes| F[Run detection check:<br/>netbird ssh detect &lt;peer&gt; 22]
    F --> G{Is NetBird SSH server?}

    G -->|No| E

    G -->|Yes - JWT required| H[Activate ProxyCommand:<br/>netbird ssh proxy &lt;peer&gt; 22]

    H --> I[Local SSH Proxy Started]
    I --> J[OpenSSH connects to proxy via stdio]

    J --> K[Proxy requests JWT from daemon]

    K --> K1{Check JWT cache}
    K1 -->|Cache valid| K2[Use cached JWT token]
    K1 -->|No cache/expired| K3[OIDC flow:<br/>User authorizes via IDP callback]
    K3 --> K4[Receive & cache JWT token]
    K4 --> L[Proxy connects to peer SSH server]
    K2 --> L

    L --> M[Proxy sends JWT authentication]

    M --> N{JWT Valid?}
    N -->|No| O[Connection Rejected]
    N -->|Yes| P[Proxy establishes session]

    P --> Q[Bidirectional forwarding:<br/>OpenSSH ↔ Proxy ↔ Peer SSH Server]

    Q --> R[User interacts with remote shell]

    style H fill:#f57c00,color:#fff
    style I fill:#1976d2,color:#fff
    style K fill:#1976d2,color:#fff
    style K1 fill:#1976d2,color:#fff
    style K3 fill:#e65100,color:#fff
    style Q fill:#388e3c,color:#fff

    classDef proxyBox fill:#1565c0,stroke:#0d47a1,stroke-width:2px,color:#fff
    class I,K,L,M,P,Q proxyBox
Loading

Issue ticket number and link

Stack

Checklist

  • Is it a bug fix
  • Is a typo/documentation fix
  • Is a feature enhancement
  • It is a refactor
  • Created tests that fail without the change (if possible)
  • Extended the README / documentation, if necessary

By submitting this pull request, you confirm that you have read and agree to the terms of the Contributor License Agreement.

@lixmal lixmal marked this pull request as ready for review June 20, 2025 15:25
Copilot AI review requested due to automatic review settings June 20, 2025 15:25

This comment was marked as outdated.

@lixmal lixmal marked this pull request as draft June 22, 2025 14:49
@pappz
Copy link
Contributor

pappz commented Oct 8, 2025

After I disable the SSH on Dashboard and enable it back again, I can not reach the remote machine via NetBird ssh :

2025-10-08T14:47:18+02:00 INFO 12 client/internal/statemanager/manager.go:412: cleaning up state ssh_config_state
2025-10-08T14:47:18+02:00 INFO 12 client/ssh/config/manager.go:254: Removed NetBird SSH config: /etc/ssh/ssh_config.d/99-netbird.conf
2025-10-08T14:47:18+02:00 DEBG 12 client/internal/statemanager/manager.go:286: persisted states: [iptables_state ssh_config_state], took 169.142µs
2025-10-08T14:47:18+02:00 INFO 206 client/internal/engine_ssh.go:264: SSH root login enabled
2025-10-08T14:47:18+02:00 INFO 206 client/internal/engine_ssh.go:272: SSH SFTP subsystem enabled
2025-10-08T14:47:18+02:00 INFO 206 client/internal/engine_ssh.go:280: SSH local port forwarding enabled
2025-10-08T14:47:18+02:00 INFO 206 client/internal/engine_ssh.go:288: SSH remote port forwarding enabled
2025-10-08T14:47:18+02:00 INFO 206 client/internal/engine_ssh.go:37: SSH port redirection enabled: 100.108.1.115:22 -> 100.108.1.115:22022
2025-10-08T14:47:18+02:00 DEBG 206 client/ssh/server/port_forwarding.go:79: SSH server configured with local_forwarding=true, remote_forwarding=true
2025-10-08T14:47:18+02:00 INFO 206 client/ssh/server/server.go:178: SSH server started on 100.108.1.115:22022
2025-10-08T14:47:18+02:00 DEBG 206 client/internal/engine_ssh.go:180: updated peer SSH host keys for daemon API access
2025-10-08T14:47:18+02:00 INFO 206 client/ssh/config/manager.go:242: Created NetBird SSH client config: /etc/ssh/ssh_config.d/99-netbird.conf
2025-10-08T14:47:18+02:00 DEBG 206 client/internal/engine_ssh.go:98: updated SSH client config with 1 peers
2025-10-08T14:47:28+02:00 INFO 307 client/ssh/server/server.go:523: SSH connection from NetBird peer 100.108.111.177 allowed
2025-10-08T14:47:28+02:00 DEBG 307 client/ssh/server/server.go:454: SSH connection failed for 100.108.111.177:42550: EOF
2025-10-08T14:47:28+02:00 INFO 302 client/ssh/server/server.go:523: SSH connection from NetBird peer 100.108.111.177 allowed
2025-10-08T14:47:28+02:00 DEBG 302 client/ssh/server/server.go:261: Initializing JWT validator (issuer: https://netbird-localdev.eu.auth0.com/, audience: http://localhost:3000/)
2025-10-08T14:47:28+02:00 INFO 302 client/ssh/server/server.go:284: JWT validator initialized successfully
2025-10-08T14:47:28+02:00 INFO 302 client/ssh/server/server.go:432: JWT authentication successful for user pzoli (JWT user ID: google-oauth2|101156770583840938852) from 100.108.111.177:42554
2025-10-08T14:47:28+02:00 DEBG 332 [session: [email protected]:42554-0e525531] client/ssh/server/session_handlers.go:89: registered SSH session
2025-10-08T14:47:28+02:00 INFO 332 [session: [email protected]:42554-0e525531] client/ssh/server/session_handlers.go:33: establishing SSH session for pzoli from 100.108.111.177:42554
2025-10-08T14:47:28+02:00 INFO 332 [session: [email protected]:42554-0e525531] client/ssh/server/command_execution_unix.go:103: executing Pty command for pzoli from 100.108.111.177:42554: <empty>
2025-10-08T14:47:28+02:00 DEBG 166 client/internal/statemanager/manager.go:286: persisted states: [ssh_config_state], took 173.037µs
2025-10-08T14:47:30+02:00 DEBG 336 [session: [email protected]:42554-0e525531] client/ssh/server/command_execution_unix.go:192: Pty output copy error: read /dev/ptmx: input/output error
2025-10-08T14:47:30+02:00 DEBG 332 [session: [email protected]:42554-0e525531] client/ssh/server/command_execution_unix.go:244: Pty command completed successfully
2025-10-08T14:47:30+02:00 DEBG 332 [session: [email protected]:42554-0e525531] client/ssh/server/command_execution_unix.go:246: exit session error: EOF
2025-10-08T14:47:30+02:00 DEBG 332 [session: [email protected]:42554-0e525531] client/ssh/server/session_handlers.go:26: close session after 1.958771059s: EOF
2025-10-08T14:47:30+02:00 DEBG 332 [session: [email protected]:42554-0e525531] client/ssh/server/session_handlers.go:114: unregistered SSH session
2025-10-08T14:47:34+02:00 INFO 206 client/internal/engine_ssh.go:55: SSH server is locally allowed but disabled by management server
2025-10-08T14:47:34+02:00 DEBG 206 client/internal/engine_ssh.go:308: SSH port redirection removed: 100.108.1.115:22 -> 100.108.1.115:22022
2025-10-08T14:47:34+02:00 INFO 206 client/internal/engine_ssh.go:331: stopping SSH server
2025-10-08T14:47:34+02:00 ERRO 264 client/ssh/server/server.go:182: SSH server error: ssh: Server closed
2025-10-08T14:47:34+02:00 DEBG 206 client/internal/engine_ssh.go:180: updated peer SSH host keys for daemon API access
2025-10-08T14:47:34+02:00 INFO 206 client/ssh/config/manager.go:242: Created NetBird SSH client config: /etc/ssh/ssh_config.d/99-netbird.conf
2025-10-08T14:47:34+02:00 DEBG 206 client/internal/engine_ssh.go:98: updated SSH client config with 1 peers
2025-10-08T14:47:38+02:00 DEBG 166 client/internal/statemanager/manager.go:286: persisted states: [ssh_config_state], took 292.231µs
2025-10-08T14:47:40+02:00 WARN 206 client/internal/engine.go:884: failed handling SSH server setup: SSH server requires valid JWT configuration
2025-10-08T14:47:40+02:00 DEBG 206 client/internal/engine_ssh.go:180: updated peer SSH host keys for daemon API access
2025-10-08T14:47:40+02:00 INFO 206 client/ssh/config/manager.go:242: Created NetBird SSH client config: /etc/ssh/ssh_config.d/99-netbird.conf
2025-10-08T14:47:40+02:00 DEBG 206 client/internal/engine_ssh.go:98: updated SSH client config with 1 peers
2025-10-08T14:47:48+02:00 DEBG 166 client/internal/statemanager/manager.go:286: persisted states: [ssh_config_state], took 250.484µs
pzoli@vb-cloud-2:~$ ./netbird ssh [email protected]
Failed to connect to [email protected]:22

Troubleshooting steps:
  1. Check peer connectivity: netbird status -d
  2. Verify SSH server is enabled on the peer
  3. Ensure correct hostname/IP is used
Error: dial 192.168.0.232:22: ssh handshake: ssh: handshake failed: host key verification failed: key not found in NetBird daemon or any known_hosts file

@pappz
Copy link
Contributor

pappz commented Oct 8, 2025

The clocks on all machines (management and both peers) have been synced, but I still get this issue when I try to SSH for the first time.

2025-10-08T14:56:10+02:00 INFO 260 client/ssh/config/manager.go:242: Created NetBird SSH client config: /etc/ssh/ssh_config.d/99-netbird.conf
2025-10-08T14:56:10+02:00 DEBG 260 client/internal/engine_ssh.go:98: updated SSH client config with 1 peers
2025-10-08T14:56:15+02:00 INFO 312 client/ssh/server/server.go:523: SSH connection from NetBird peer 100.108.111.177 allowed
2025-10-08T14:56:15+02:00 DEBG 312 client/ssh/server/server.go:454: SSH connection failed for 100.108.111.177:48236: EOF
2025-10-08T14:56:19+02:00 DEBG 178 client/internal/statemanager/manager.go:286: persisted states: [ssh_config_state], took 199.353µs
2025-10-08T14:56:25+02:00 INFO 301 client/ssh/server/server.go:523: SSH connection from NetBird peer 100.108.111.177 allowed
2025-10-08T14:56:25+02:00 DEBG 301 client/ssh/server/server.go:261: Initializing JWT validator (issuer: https://netbird-localdev.eu.auth0.com/, audience: http://localhost:3000/)
2025-10-08T14:56:25+02:00 INFO 301 client/ssh/server/server.go:284: JWT validator initialized successfully
2025-10-08T14:56:25+02:00 WARN 301 client/ssh/server/server.go:422: JWT authentication failed for user pzoli from 100.108.111.177:58642: validate token (expected issuer=https://netbird-localdev.eu.auth0.com/, audience=http://localhost:3000/, actual issuer=https://netbird-localdev.eu.auth0.com/, audience=[http://localhost:3000/ https://netbird-localdev.eu.auth0.com/userinfo]): token could not be parsed: token has invalid claims: token used before issued
2025-10-08T14:56:25+02:00 DEBG 301 client/ssh/server/server.go:454: SSH connection failed for 100.108.111.177:58642: [ssh: no auth passed yet, permission denied]
pzoli@vb-cloud-2:~$ ssh 100.108.1.115
SSH authentication required.
Please visit: https://netbird-localdev.eu.auth0.com/activate?user_code=BKVX-ZVKX
Or visit: https://netbird-localdev.eu.auth0.com/activate and enter code: BKVX-ZVKX
Waiting for authentication...
SSH connection to NetBird server failed: SSH handshake: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain
                                                                                                                                                                          pzoli@vb-cloud-2:~$ 
pzoli@vb-cloud-2:~$ ssh 100.108.1.115
Welcome to Ubuntu 24.10 (GNU/Linux 6.11.0-29-generic x86_64)

@lixmal
Copy link
Collaborator Author

lixmal commented Oct 8, 2025

error Unfriendly error output. At the moment, the system clock was outdated.

Unfortunately the ssh protocol doesn't allow us to send any output to the client in the auth layer

@lixmal
Copy link
Collaborator Author

lixmal commented Oct 8, 2025

After I disable the SSH on Dashboard and enable it back again, I can not reach the remote machine via NetBird ssh :

2025-10-08T14:47:18+02:00 INFO 12 client/internal/statemanager/manager.go:412: cleaning up state ssh_config_state
2025-10-08T14:47:18+02:00 INFO 12 client/ssh/config/manager.go:254: Removed NetBird SSH config: /etc/ssh/ssh_config.d/99-netbird.conf
2025-10-08T14:47:18+02:00 DEBG 12 client/internal/statemanager/manager.go:286: persisted states: [iptables_state ssh_config_state], took 169.142µs
2025-10-08T14:47:18+02:00 INFO 206 client/internal/engine_ssh.go:264: SSH root login enabled
2025-10-08T14:47:18+02:00 INFO 206 client/internal/engine_ssh.go:272: SSH SFTP subsystem enabled
2025-10-08T14:47:18+02:00 INFO 206 client/internal/engine_ssh.go:280: SSH local port forwarding enabled
2025-10-08T14:47:18+02:00 INFO 206 client/internal/engine_ssh.go:288: SSH remote port forwarding enabled
2025-10-08T14:47:18+02:00 INFO 206 client/internal/engine_ssh.go:37: SSH port redirection enabled: 100.108.1.115:22 -> 100.108.1.115:22022
2025-10-08T14:47:18+02:00 DEBG 206 client/ssh/server/port_forwarding.go:79: SSH server configured with local_forwarding=true, remote_forwarding=true
2025-10-08T14:47:18+02:00 INFO 206 client/ssh/server/server.go:178: SSH server started on 100.108.1.115:22022
2025-10-08T14:47:18+02:00 DEBG 206 client/internal/engine_ssh.go:180: updated peer SSH host keys for daemon API access
2025-10-08T14:47:18+02:00 INFO 206 client/ssh/config/manager.go:242: Created NetBird SSH client config: /etc/ssh/ssh_config.d/99-netbird.conf
2025-10-08T14:47:18+02:00 DEBG 206 client/internal/engine_ssh.go:98: updated SSH client config with 1 peers
2025-10-08T14:47:28+02:00 INFO 307 client/ssh/server/server.go:523: SSH connection from NetBird peer 100.108.111.177 allowed
2025-10-08T14:47:28+02:00 DEBG 307 client/ssh/server/server.go:454: SSH connection failed for 100.108.111.177:42550: EOF
2025-10-08T14:47:28+02:00 INFO 302 client/ssh/server/server.go:523: SSH connection from NetBird peer 100.108.111.177 allowed
2025-10-08T14:47:28+02:00 DEBG 302 client/ssh/server/server.go:261: Initializing JWT validator (issuer: https://netbird-localdev.eu.auth0.com/, audience: http://localhost:3000/)
2025-10-08T14:47:28+02:00 INFO 302 client/ssh/server/server.go:284: JWT validator initialized successfully
2025-10-08T14:47:28+02:00 INFO 302 client/ssh/server/server.go:432: JWT authentication successful for user pzoli (JWT user ID: google-oauth2|101156770583840938852) from 100.108.111.177:42554
2025-10-08T14:47:28+02:00 DEBG 332 [session: [email protected]:42554-0e525531] client/ssh/server/session_handlers.go:89: registered SSH session
2025-10-08T14:47:28+02:00 INFO 332 [session: [email protected]:42554-0e525531] client/ssh/server/session_handlers.go:33: establishing SSH session for pzoli from 100.108.111.177:42554
2025-10-08T14:47:28+02:00 INFO 332 [session: [email protected]:42554-0e525531] client/ssh/server/command_execution_unix.go:103: executing Pty command for pzoli from 100.108.111.177:42554: <empty>
2025-10-08T14:47:28+02:00 DEBG 166 client/internal/statemanager/manager.go:286: persisted states: [ssh_config_state], took 173.037µs
2025-10-08T14:47:30+02:00 DEBG 336 [session: [email protected]:42554-0e525531] client/ssh/server/command_execution_unix.go:192: Pty output copy error: read /dev/ptmx: input/output error
2025-10-08T14:47:30+02:00 DEBG 332 [session: [email protected]:42554-0e525531] client/ssh/server/command_execution_unix.go:244: Pty command completed successfully
2025-10-08T14:47:30+02:00 DEBG 332 [session: [email protected]:42554-0e525531] client/ssh/server/command_execution_unix.go:246: exit session error: EOF
2025-10-08T14:47:30+02:00 DEBG 332 [session: [email protected]:42554-0e525531] client/ssh/server/session_handlers.go:26: close session after 1.958771059s: EOF
2025-10-08T14:47:30+02:00 DEBG 332 [session: [email protected]:42554-0e525531] client/ssh/server/session_handlers.go:114: unregistered SSH session
2025-10-08T14:47:34+02:00 INFO 206 client/internal/engine_ssh.go:55: SSH server is locally allowed but disabled by management server
2025-10-08T14:47:34+02:00 DEBG 206 client/internal/engine_ssh.go:308: SSH port redirection removed: 100.108.1.115:22 -> 100.108.1.115:22022
2025-10-08T14:47:34+02:00 INFO 206 client/internal/engine_ssh.go:331: stopping SSH server
2025-10-08T14:47:34+02:00 ERRO 264 client/ssh/server/server.go:182: SSH server error: ssh: Server closed
2025-10-08T14:47:34+02:00 DEBG 206 client/internal/engine_ssh.go:180: updated peer SSH host keys for daemon API access
2025-10-08T14:47:34+02:00 INFO 206 client/ssh/config/manager.go:242: Created NetBird SSH client config: /etc/ssh/ssh_config.d/99-netbird.conf
2025-10-08T14:47:34+02:00 DEBG 206 client/internal/engine_ssh.go:98: updated SSH client config with 1 peers
2025-10-08T14:47:38+02:00 DEBG 166 client/internal/statemanager/manager.go:286: persisted states: [ssh_config_state], took 292.231µs
2025-10-08T14:47:40+02:00 WARN 206 client/internal/engine.go:884: failed handling SSH server setup: SSH server requires valid JWT configuration
2025-10-08T14:47:40+02:00 DEBG 206 client/internal/engine_ssh.go:180: updated peer SSH host keys for daemon API access
2025-10-08T14:47:40+02:00 INFO 206 client/ssh/config/manager.go:242: Created NetBird SSH client config: /etc/ssh/ssh_config.d/99-netbird.conf
2025-10-08T14:47:40+02:00 DEBG 206 client/internal/engine_ssh.go:98: updated SSH client config with 1 peers
2025-10-08T14:47:48+02:00 DEBG 166 client/internal/statemanager/manager.go:286: persisted states: [ssh_config_state], took 250.484µs
pzoli@vb-cloud-2:~$ ./netbird ssh [email protected]
Failed to connect to [email protected]:22

Troubleshooting steps:
  1. Check peer connectivity: netbird status -d
  2. Verify SSH server is enabled on the peer
  3. Ensure correct hostname/IP is used
Error: dial 192.168.0.232:22: ssh handshake: ssh: handshake failed: host key verification failed: key not found in NetBird daemon or any known_hosts file

@pappz fixed in 610c880

pappz
pappz previously approved these changes Oct 17, 2025
@mlsmaycon mlsmaycon mentioned this pull request Oct 27, 2025
mlsmaycon pushed a commit that referenced this pull request Oct 28, 2025
…#4707)

- Port dnat changes from #4015 (nftables/iptables/userspace)
  - For userspace: rewrite the original port to the target port
  - Remember original destination port in conntrack
  - Rewrite the source port back to the original port for replies
- Redirect incoming port 5353 to 22054 (tcp/udp)
- Revert port changes based on the network map received from management
- Adjust tracer to show NAT stages
@sonarqubecloud
Copy link

@lixmal lixmal mentioned this pull request Oct 30, 2025
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants