Allow definition of custom network interfaces

[moritz89]

Why:

• Allow user-defined network interfaces to switch been WiFi and mobile modems during run-time
• Allow custom classes to be used as the network, ssl and server classes

This change addresses the need by:

• Creating a new custom WEBSOCKETS_NETWORK_TYPE that expects the user to set network classes
  • WEBSOCKETS_NETWORK_CLASS
  • WEBSOCKETS_NETWORK_SSL_CLASS
  • WEBSOCKETS_NETWORK_SERVER_CLASS

Related issues: #779 #910

[moritz89]

An open questions are:

1. which number should be chosen as the NETWORK_CUSTOM define? The next number (10) or something like 99 or 255?
2. Is NETWORK_CUSTOM the right name?

I would also like to test it with a POC class that toggles between WiFi and TinyGSM, like that published by @Naheel-Azawy : ESP32GSMNetClient before marking this MR as ready.

[Links2004]

using the next free number is fine.

not sure if setting some custom defines via compiler option will work, since WEBSOCKETS_NETWORK_TYPE does also switch some code around and handles include of h files.

will be intresting to see how you solve this.
all of this would be much easier if upstream arduino did provide some interface classes for the network stuff, but thats not the case sadly.

[moritz89]

One option I found that compiles, but isn't pretty is the following. One thing that won't work is using defines to set include paths. Looking into 1-2 other approaches.

lib/arduinoWebSockets/src/WebSockets.h

...
#elif(WEBSOCKETS_NETWORK_TYPE == NETWORK_CUSTOM)
#include <../../../src/CustomNetworkClient.h>
#else
#error "no network type selected!"
#endif
...

src/CustomNetworkClient.h

#pragma once

#include <WiFi.h>

#include "managers/network_client.h"

#define SSL_AXTLS
#define WEBSOCKETS_NETWORK_CLASS inamata::NetworkClient
#define WEBSOCKETS_NETWORK_SSL_CLASS inamata::NetworkClient
#define WEBSOCKETS_NETWORK_SERVER_CLASS WiFiServer

src/managers/network_client.h

#pragma once

#include <Client.h>
#include <WiFiClient.h>

namespace inamata {

class NetworkClient : public Client {
 public:
  NetworkClient();
  NetworkClient(WiFiClient wifi_client);
  virtual ~NetworkClient() = default;

  int connect(IPAddress ip, uint16_t port) final;
  int connect(const char *host, uint16_t port) final;
  int connect(const char *host, uint16_t port, int32_t timeout);
  size_t write(uint8_t) final;
  size_t write(const uint8_t *buf, size_t size) final;
  size_t write(const char *str);
  int available() final;
  int read() final;
  int read(uint8_t *buf, size_t size) final;
  int peek() final;
  void flush() final;
  void stop() final;
  uint8_t connected() final;
  operator bool() final;

  void setCACert(const char *rootCA);
  void setCACertBundle(const uint8_t *bundle);
  void setInsecure();
  bool verify(const char *fingerprint, const char *domain_name);
};

}  // namespace inamata

[moritz89]

A better approach is probably to use the PIMPL approach, an interface class that implements all methods that are called on WiFiClient and WiFiClientSecure and then implement those class methods in the application code. This should be more robust than the relative import #include <../../../src/CustomNetworkClient.h> in the previous approach.

Adding the following file is the minimum to implement stub interfaces in the app source code including the WebSockets library. To keep the WiFiClientSecure behavior, an instance of it can be saved in the NetworkClient::Impl class and then called via the implemented methods (connect(), write(), ...). To connect it to TinyGSM, create an instance of that class...

#include <NetworkClient.h>

struct NetworkClient::Impl {
  ~Impl() = default;

  int a;
  bool b;
};

NetworkClient::NetworkClient() : _impl(new NetworkClient::Impl()) {}
NetworkClient::NetworkClient(WiFiClient wifi_client)
    : _impl(new NetworkClient::Impl()) {}
NetworkClient::~NetworkClient() {}

int NetworkClient::connect(IPAddress ip, uint16_t port) { return 0; }
int NetworkClient::connect(const char *host, uint16_t port) { return 0; }
int NetworkClient::connect(const char *host, uint16_t port, int32_t timeout) {
  return 0;
}
size_t NetworkClient::write(uint8_t) { return 0; }
size_t NetworkClient::write(const uint8_t *buf, size_t size) { return 0; }
size_t NetworkClient::write(const char *str) { return 0; }
int NetworkClient::available() { return 0; }
int NetworkClient::read() { return 0; }
int NetworkClient::read(uint8_t *buf, size_t size) { return 0; }
int NetworkClient::peek() { return 0; }
void NetworkClient::flush() {}
void NetworkClient::stop() {}
uint8_t NetworkClient::connected() { return 0; }
NetworkClient::operator bool() { return 0; }

void NetworkClient::setCACert(const char *rootCA) {}
void NetworkClient::setCACertBundle(const uint8_t *bundle) {}
void NetworkClient::setInsecure() {}
bool NetworkClient::verify(const char *fingerprint, const char *domain_name) {
  return false;
}

I still have to test it with a complete implementation, but it compiles and should allow enough flexibility to implement IO with both WiFi and a mobile modem (or even also Ethernet).

[moritz89]

Here is a VS Code / PlatformIO example project that uses the PIMPL approach.

CustomWebSocketClient.zip

I tried compiling it with Arduino IDE but it failed as it takes the class method implementations and creates duplicate definitions for the NetworkClient class. Also tried using a separate cpp file to implement the NetworkClient class, but that also failed.

I would argue that using a custom network interface is quite advanced anyway, so primarily targeting VS Code / PlatformIO users shouldn't be the biggest deal breaker.

[moritz89]

@Links2004 I've update the MR to include an example project. The implementation is currently being used in a production unit (ergo tested a bit) and I thought I'd back port the changes for general use.

If you find the changes useful, would be glad to work and get this merged, but I also understand that this might be a bit more niche of a use case.

[moritz89]

I just had a look at upgrading to Arduino 3.2.1 and IDF 5.4.2 via pioarduino and found out that they included a library called NetworkClient. As no one is using namespaces, this collided and thus it would be sensible to rename the NetworkClient class to something else like CustomNetworkClient, GenericNetworkClient or AbstractNetworkClient.

[Links2004]

sounds good, I like the Idea of having a way for users to there own network interface / stack.

[moritz89]

Two open questions are, what do we name the class and do you have an idea why the UNO R4 fails to compile?

[Links2004]

since the NetworkClient is internal to the Websocket lib it may makes sense to simple prefix it, e.g. WebSocketNetworkClient this way the change of collisions is pretty low.

@kakopappa can you take a look at the arduino:renesas_uno:unor4wifi fails? the log does not make sense to me, I only see warnings no errors.

[kakopappa]

`

/home/runner/Arduino/libraries/arduinoWebSockets/src/WebSocketsClient.cpp: In member function 'virtual void WebSocketsClient::clientDisconnect(WSclient_t*)':
/home/runner/Arduino/libraries/arduinoWebSockets/src/WebSocketsClient.cpp:559:17: error: 'struct WSclient_t' has no member named 'ssl'
client->ssl = NULL;
^~~
`

Just saw this in the log, bit hard to see it

[Links2004]

thanks for looking,
today I learned that CTRL+F does not work for the github logs any more...
only the page build in one finds the error.

they must have implemented some lazy loading on the log page.

[moritz89]

@Links2004 I added the line client->ssl = NULL; as there was a reconnection issue without it. I'll see if I can recreate the situation where it is required.

EDIT:
I found a problem and memory leak with my changes. Will push a fix.

[moritz89]

@Links2004 Everything seems to be resolved from my side. Do you see any blocker or can we merge the changes?

[Links2004]

I am currently traveling, will take a look next week.

[moritz89]

I did not increment the version or prepare anything for a new release. Can I help you there?

[Links2004]

did a new release https://github.com/Links2004/arduinoWebSockets/releases/tag/2.7.0

[razvanphp]

We also need this, we are actually switching between 3 interfaces, ESP32 Wifi, Wiznet W5500 Ethernet and PPP Modem.

The thing is, ESPIDF has a common interface, netif, couldn't we use this as a common socket provider, no matter what underlying connection exists? This way we can implement run-time switching failover (with reconnect, ofc).

[Links2004]

yes, this libary is expected to work when you, integrate all your network interfaces in to lwip (the lib that the ESP uses as networks stack).
the arduino classes WiFiClient, WiFiClientSecure and WiFiServer use the lwip stack in its code as base for all network operations.

as a result I expect the lib use any network interfaces are registerd in lwip transparently without changes to the code of the websocket lib.
The ESP has already 2 network interfaces under the hood (WiFi AP and WiFi STA) which can be active at the same time.

Not sure how fast the lwip reacts to IP route changes and sending disconect notification to the ws lib for triggering the reconnect, but in the worst cases its possible to force the disconnect from the ws lib side via

arduinoWebSockets/src/WebSocketsClient.h

Line 102 in 6eee4c6

void disconnect(void);

[razvanphp]

But the Ethernet Wiznet W5500 and W6100 we have do not use lwip unfortunately, they have a hardware TCP/IP stack, hence the problem...

[Links2004]

the code from @moritz89 is what you need to use as a base then, since you need to switch the IP stack on the fly.

see:
https://github.com/Links2004/arduinoWebSockets/tree/master/examples/esp32_pio/CustomNetworkClient

https://github.com/Links2004/arduinoWebSockets/blob/master/examples/esp32_pio/CustomNetworkClient/src/network_client.cpp

[razvanphp]

Ok, that is one way, we will try to implement them.

But... what do you think about using espressif/arduino-esp32/blob/master/libraries/Ethernet/src/ETH.h? This should expose an lwIP interface compatible with WiFiClientSecure, or?

Another way would be to use official Arduino Ethernet library, as it supports W5500 already, but not sure that will work with your lib.

Thank you very much for your time! Would love to contribute this as README or example if will manage to get it working in an elegant way.

[Links2004]

arduino ETH.h is supported via:

arduinoWebSockets/src/WebSockets.h

Lines 243 to 247 in 6eee4c6

	#elif (WEBSOCKETS_NETWORK_TYPE == NETWORK_ESP32_ETH)
	#include <ETH.h>
	#define WEBSOCKETS_NETWORK_CLASS WiFiClient
	#define WEBSOCKETS_NETWORK_SERVER_CLASS WiFiServer

and yes it uses lwip so there is a change you can try switching via code.
there where no TLS support back when ETH.h support was implemented, but give adding

#define WEBSOCKETS_NETWORK_SSL_CLASS WiFiClientSecure

a try

or try keeping the type at NETWORK_ESP32 and using the init for the network from the ETH examples.
which most likely will work fine since WiFiClient and NetworkClient etc are the same thing on the ESP32,
they renamed it at some point and added an "alias".

the libery still uses "WiFi*" instat of "Network*" simply to not brake code that uses the older arduino-esp32 versions.