Discussion: unified JSON REST-type api calls for page submissions

[rhkean]

Is your feature request related to a problem? Please describe.
no... enhancement

Describe the solution you'd like
I'm currently working on a bluetooth (BLE) implementation that will enable WLED to connect via bluetooth in addition to wifi and eth. In looking over the architecture of the system and the WLED-native application, it seems that, if all of the pages used JSON (AJAX) calls for forms-submission, everything could then run purely through the JSON api.

I believe this could be implemented across all pages with a little effort to standardize the calls for existing pages and document the standardization for usermods.

I'd be willing to take the effort on if it the idea seems sound and would have the backing to implement into the core.

Describe alternatives you've considered
n/a

Additional context
The form submissions from the existing UI can be modified to implement an AJAX application/json POST to the JSON api to standardize submissions. This would also enable the POST to be intercepted on the phone (native app) and passed to a bluetooth connected device as JSON and internally processed by WLED's serial to JSON interface.

NOTE: I am currently working on the android modifications... There would be a learning curve for me to tackle the iOS modifications once the android side is complete. However, I don't mind taking that on or stepping back for someone else to do so.

[blazoncek]

WLED already supports configuration via JSON POST (/json/cfg endpoint), however none of the settings pages are JSON capable.
Completely rewriting settings is one of the major goals for 1.0 release. However there is no hero to tackle this task.

[rhkean]

WLED already supports configuration via JSON POST (/json/cfg endpoint), however none of the settings pages are JSON capable. Completely rewriting settings is one of the major goals for 1.0 release. However there is no hero to tackle this task.

well, I guess.... I'm your man, then. :-)

I can't say how long it will take me, but I wanted to do it anyhow.... just didn't want to put the effort in without backing on the changes. let me know if there are any existing guidelines and/or design preferences before I start.

[blazoncek]

There aren't. @Aircoookie started to work on it quite a while ago but never got far.

However, you can start with each page separately (one by one) and see how much work you end up. You'll need to cross-reference set.cpp, cfg.cpp and json.cpp.

[TheRealZam]

While I lack the ability to do this in an efficient manner, here is my suggestion.

1. In WLED, expose the WS & JSON API as individual BLE serial endpoints using the currently available 'serial' options.
2. In WLED-native, Create an 'add ble device' command in the ble-native app that will search for new WLED devices with BLE activated.
3. In the WLED-native app, setup a local server which runs the UI on the phone itself.
4. When the user clicks into the device, the required device connections over BLE will be opened.
5. WLED-native should store a clone of each supported version of the UI and it can query the device's version to determine which to present when controlling the device.
6. The interface for "ws://[local-WLED-web]/ws" should redirect the websocket to run over the BLE serial connection.

As a bonus, the local web server function can be adapted to support running the web interface locally for any WLED device which could improve performance. It could even be adapted for other connections like thread (thread is really just IPv6) , serial over zigbee, web management over normal serial, etc.

[rhkean]

While I lack the ability to do this in an efficient manner, here is my suggestion.

1. In WLED, expose the WS & JSON API as individual BLE serial endpoints using the currently available 'serial' options.

yeah, I'm implementing it as a BLESerial (UART over BLE) implementation and probably passing it to the standard wled_serial implementation

2. In WLED-native, Create an 'add ble device' command in the ble-native app that will search for new WLED devices with BLE activated.

I've implemented an "Enable Bluetooth" setting that enables BLE scanning with a minor change to the Device object for BLE devices

3. In the WLED-native app, setup a local server which runs the UI on the phone itself.

for Android, this is pretty simple, since it uses a WebView object, and then intercept the POST actions, but I'm not certain how it would be done with iPhones... I'm focusing on Android for the time being

4. When the user clicks into the device, the required device connections over BLE will be opened.

check

5. WLED-native should store a clone of each supported version of the UI and it can query the device's version to determine which to present when controlling the device.

not quite sufficient, although caching the UI data is the idea, it needs to be based on the MAC address, not the version, bc the user mods added to a build don't impact the version in any way and you could build multiple versions with the same build but different mods.

6. The interface for "ws://[local-WLED-web]/ws" should redirect the websocket to run over the BLE serial connection.> > As a bonus, the local web server function can be adapted to support running the web interface locally for any WLED device which could improve performance. It could even be adapted for other connections like thread (thread is really just IPv6) , serial over zigbee, web management over normal serial, etc.

I haven't given any thought to websockets... currently, I've disabled any features that I'm not using )including ws) to insure that I don't have any flash memory constraint issues. Once it is all working, I'll start adding back in the standard features and possibly attack websockets.

[TheRealZam]

While I lack the ability to do this in an efficient manner, here is my suggestion.

1. In WLED, expose the WS & JSON API as individual BLE serial endpoints using the currently available 'serial' options.

yeah, I'm implementing it as a BLESerial (UART over BLE) implementation and probably passing it to the standard wled_serial implementation

2. In WLED-native, Create an 'add ble device' command in the ble-native app that will search for new WLED devices with BLE activated.

I've implemented an "Enable Bluetooth" setting that enables BLE scanning with a minor change to the Device object for BLE devices

3. In the WLED-native app, setup a local server which runs the UI on the phone itself.

for Android, this is pretty simple, since it uses a WebView object, and then intercept the POST actions, but I'm not certain how it would be done with iPhones... I'm focusing on Android for the time being

4. When the user clicks into the device, the required device connections over BLE will be opened.

check

5. WLED-native should store a clone of each supported version of the UI and it can query the device's version to determine which to present when controlling the device.

not quite sufficient, although caching the UI data is the idea, it needs to be based on the MAC address, not the version, bc the user mods added to a build don't impact the version in any way and you could build multiple versions with the same build but different mods.

6. The interface for "ws://[local-WLED-web]/ws" should redirect the websocket to run over the BLE serial connection.> > As a bonus, the local web server function can be adapted to support running the web interface locally for any WLED device which could improve performance. It could even be adapted for other connections like thread (thread is really just IPv6) , serial over zigbee, web management over normal serial, etc.

I haven't given any thought to websockets... currently, I've disabled any features that I'm not using )including ws) to insure that I don't have any flash memory constraint issues. Once it is all working, I'll start adding back in the standard features and possibly attack websockets.

On iOS, I use an app called 'Bluefy' to expose webkit to connections over ble for testing purposes. Put simply, it automates BLE serial connection management in a similar way to how chrome handles these connections.

When I first reviewed how the web UI works, it seemed to run all commands through the websocket, which is why I made the suggestion.

RE the web UI caching, do you plan on cloning the entire UI via via serial?

[blazoncek]

Dear @rhkean & @TheRealZam all you need to do is add a wrapper for JSON functions from json.cpp, i.e. serializeJson(), deserializeJson(), etc., into your BLE/BT usermod/enhancement. No need to mess with WLED's serial implementation (which does the same).

Unfortunately BLE/BT code is huge and will not fit on most common devices with 4MB flash and regular WLED build. Let alone additional usermods.

However things may have changed since we last reviewd BLE/BT.

[willmmiles]

However things may have changed since we last reviewd BLE/BT.

FWIW, I was rather surprised to discover that the Tasmota platform we're using in 0.16 has the Bluetooth driver included for baseline ESP32s; I ended up having to route the RMT interrupt through it in #4890 as it was consuming the high priority IRQ.

So it might just already be there...

[rhkean]

Unfortunately BLE/BT code is huge and will not fit on most common devices with 4MB flash and regular WLED build. Let alone additional usermods.

However things may have changed since we last reviewd BLE/BT.

on this note, my usermod, fits in the standard partition layout with OTA, and all the default settings...

[env:RHKeanBLE]
extends = env:esp32dev
custom_usermods = BLE audioreactive Internal_Temperature_v2 Temperature multi_relay
build_flags = -g
  -D WLED_PRODUCT_NAME='"UnderglowBLE"'
  -D SERVERNAME='"UnderglowBLE"'
  ${env:esp32dev.build_flags}
  ; ${rhkean.build_flags_disable_features}
  ${rhkean.build_flags_preconfig_settings}
  ${rhkean.build_flags_dig_ar}
  ${rhkean.build_flags_temperature}
  ${rhkean.build_flags_multi_relay}
  ${rhkean.build_flags_max_buttons}
  ${common.debug_flags}

output:

INFO: 5 usermod object entries
Retrieving maximum program size .pio/build/RHKeanBLE/firmware.elf
Checking size .pio/build/RHKeanBLE/firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [==        ]  17.3% (used 56672 bytes from 327680 bytes)
Flash: [==========]  96.5% (used 1518265 bytes from 1572864 bytes)
Building .pio/build/RHKeanBLE/firmware.bin

My personal use case disables a bunch of functions that I don't need and saves 150k

build_flags_disable_features =
  -D WLED_DISABLE_ALEXA
  -D WLED_DISABLE_BLYNK
  -D WLED_DISABLE_HUESYNC
  -D WLED_DISABLE_LOXONE
  -D WLED_DISABLE_INFRARED
  -D WLED_DISABLE_WEBSOCKETS
  -D WLED_DISABLE_MQTT
  -D WLED_DISABLE_2D
  -D WLED_DISABLE_PXMAGIC
  -D WLED_DISABLE_ESPNOW

INFO: 5 usermod object entries
Retrieving maximum program size .pio/build/RHKeanBLE/firmware.elf
Checking size .pio/build/RHKeanBLE/firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [==        ]  16.4% (used 53712 bytes from 327680 bytes)
Flash: [========= ]  86.8% (used 1365297 bytes from 1572864 bytes)
Building .pio/build/RHKeanBLE/firmware.bin

[blazoncek]

My personal use case disables a bunch of functions that I don't need and saves 150k

I know it's what you need, but consider that you are working upon the work of others that considered what other people may need or want.
Try to fit it with other options enabled, especially 2D, MQTT and IR (if you'll have Websockets replacement in BLE, otherwise include Websockets).

[rhkean]

My personal use case disables a bunch of functions that I don't need and saves 150k

I know it's what you need, but consider that you are working upon the work of others that considered what other people may need or want. Try to fit it with other options enabled, especially 2D, MQTT and IR (if you'll have Websockets replacement in BLE, otherwise include Websockets).

yes, my personal use-case is just an example of how small I was able to get it.

the [esp32dev] + BLE still fits and functions with the default partitions (including OTA) , even with the DEBUG options enabled...

here is the env definition:

[env:BLE]
extends = env:esp32dev
custom_usermods = ${env:esp32dev.custom_usermods} BLE

...

RAM:   [==        ]  17.2% (used 56416 bytes from 327680 bytes)
Flash: [========= ]  94.1% (used 1479369 bytes from 1572864 bytes)

and debug build

[env:BLE-debug]
extends = env:esp32dev
custom_usermods = ${env:esp32dev.custom_usermods} BLE
build_flags =
  ${env:esp32dev.build_flags}
  ${common.debug_flags}

...

RAM:   [==        ]  17.2% (used 56488 bytes from 327680 bytes)
Flash: [==========]  95.5% (used 1501749 bytes from 1572864 bytes)

[rhkean]

actually, I just tested the env:esp32dev + audioreactive Internal_Temperature_v2 Temperature multi_relay BLE...

It's still only at 96.7% on the default partition map (still has 52.5k free)