SenseBoxOTA: bugfixes

arduino/samd/libraries/SenseBoxOTA/.gitignore

@@ -0,0 +1 @@
+extra/ota_boot/build

arduino/samd/libraries/SenseBoxOTA/README.md

@@ -1,6 +1,8 @@
 # senseBox OTA
 This library provides over the air programming for the senseBox MCU.
-To enable this operating mode, just include the following line in your sketch:
+
+## usage
+To enable OTA programming, just include the following line in your sketch:
 
 ```c
 #include <SenseBoxOTA.h>
@@ -14,35 +16,85 @@ Your MCU will now have a secondary OTA bootloader, which enables a secondary ope
     In this mode a WiFi hotspot with a webserver is started, where sketch binaries can be sent to.
 - This mode can be entered manually by holding down the grey button on the MCU ("switch") while starting / resetting.
 
-## uploading a sketch
-To upload a sketch, make sure...
-- the MCU is in OTA mode & you are connected to its WiFi AP,
-- your sketch contains the line `#include <SenseBoxOTA.h>`.
+### uploading a sketch
+1. Prerequisites:
+    - your sketch contains the line `#include <SenseBoxOTA.h>`.
+    - your senseBox MCU already has the OTA bootloader installed
+    - your senseBox MCU has the WiFi bee installed in slot XBee1
+2. Enable OTA mode:
+    - on the hold the SWITCH, press RESET, and release SWITCH after 2 seconds
+    - if in OTA mode, the green status led blinks slowly
+3. connect to the WiFi network named `senseBox:ABCD` with your device,
+   where `ABCD` are the last bytes of the mac address printed on the WiFi bee
+3. Upload:
+    - [Using senseBox Connect app](https://sensebox.de/en/app.html)
+    - Using Arduino IDE & `curl`:
+      Export your compiled sketch (`ctrl+alt+s`).
+      Now you can upload from your sketch directory:
+      ```
+      curl 192.168.1.1/sketch --data-binary @<your-sketchname>ino.sensebox_mcu.bin
+      ```
+
+## how it works
 
-Export your compiled sketch (`ctrl+alt+s`). Now you can upload from your sketch directory:
+This library provides a second stage ('userspace') bootloader, that runs after the first stage bootloader (which provides the USB mass-storage update facility), but runs before the user-provided code.
+This works by inserting the OTA functionality at the start of the userspace.
+This position in flash storage is defined for the symbol name `.sketch_boot` in the linker script (`variants/sensebox_mcu/linker_scripts/gcc/flash_with_bootloader.ld`).
+The symbol `.sketch_boot` is defined in `src/SenseboxOTA.cpp` and contains the compiled sketch of the bootloader.
+
+Internally, this bootloader works quite similar as Arduino's [`SDU` library][sdu], except for swapping the SD reading functionality with a webserver:
+The OTA bootloader directly hands over to a user application if one is present, otherwise starts a WiFi accesspoint and webserver.
+On this WiFi accesspoint clients can send new sketches to the MCU via HTTP POST to http://192.168.1.1:80/sketch.
 
 ```
-curl 192.168.1.1/sketch --data-binary @<your-sketchname>ino.sensebox_mcu.bin
++---------------------------+
+|   8KB  primary bootloader |
++---------------------------+
+|  64KB  OTA bootloader     | // this section only exists, when .sketch_boot is defined,
+|        .sketch_boot       | // i.e. when the user sketch contains `#include <SenseBoxOTA.h>` 
++---------------------------+
+| 184KB  userspace code     |
+|                           |
++---------------------------+
 ```
 
+## HTTP API
+On the access point, a HTTP1.1 server at `192.168.1.1:80` accepts the following requests:
+
+### `POST /sketch`
+- headers:
+  - `Content-Type`: required. length of the sketch in bytes
+- body: raw binary data (not encoded as anything) of a compiled sketch that must contain the `#include <SenseBoxOTA.h>` bit
+  - with curl, just use `-d @<path-to-sketch.bin>`
+  - in JS, you can use `fetch()` with an `ArrayBuffer()` as body.
+
 ## development
-This library has development dependencies on `WiFi101.h` and `FlashStorage.h` (during build time only).
+- directory layout:
+  - `examples/`: simple usage example
+  - `extra/ota_boot/`: the bootloader
+  - `src/` contains the runtime with the compiled bootloader, included by users via `#include <SenseBoxOTA.h>`
+
+- To apply changes made to the `ota_boot.ino` sketch, the OTA bootloader needs to be built:
+    Run `./build_cli.sh` to update the bootloader that users include via `#include <SenseBoxOTA.h>`.
 
-To apply changes made to the `ota_boot.ino` sketch, the `./build.sh` script has to be run first.
-For development you can enable DEBUG logging via the `OTA_DEBUG` define in `conf.h`; note that in debug mode the bootloader will wait until the Serial Monitor is opened before starting operation.
+- This library has build-time dependencies on `sh`, `xxd`, and some arduino libraries that are not vendored. Check the file [`src/boot/buildinfo.txt`](src/boot/buildinfo.txt) for information on the library versions used during the last build.
 
-Internally, it works quite similar as Arduino's `SDU.h` library, except for swapping the SD reading functionality with a webserver.
-To understand what is happening the following hints may help:
+- The recommended build tool is [arduino-cli][cli], but Arduino IDE may work too, use the respective `build_*.sh` script.
 
-- The linker script reserves the first section of flash storage for the OTA functionality via the symbol name `.sketch_boot`.
-  If this symbol is missing, the memory is as usual (8KB bootloader, then user code).
-- The OTA functionality is defined in the code in `extras/ota_boot`, and put into the folder `src/boot/` in compiled binary form.
-- The OTA bootloader directly hands over to a user application if one is present, otherwise starts a hotspot and webserver.
+- For development you can enable DEBUG logging via the `OTA_DEBUG` define in `OTA.h`;
+    note that in debug mode the bootloader will wait until the Serial Monitor is opened before starting operation!
 
 ## known issues
-- accepts only one wifi client
-- webserver does not respond after one wifi disconnect
-- no code checksumming
+- no checksumming or signature check on the received binary
+- OTA bootloader takes up almost 64KB of flash - most of it is the Wifi101 library.
+    If this can be replaced  with a slimmer library, more space for user sketches will remain.
+- OTA_DEBUG logging is silent after ~2 seconds without communication - cause unknown.
+    Output is re-enabled by sending a message from the host.
 
 ## license
 GPL-3.0, Norwin Roosen
+
+[sdu]: https://github.com/arduino/ArduinoCore-samd/tree/master/libraries/SDU
+[cli]: https://github.com/arduino/arduino-cli
+[libwifi]: https://github.com/arduino-libraries/WiFi101
+[libflash]: https://github.com/cmaglie/FlashStorage

arduino/samd/libraries/SenseBoxOTA/extra/ota_boot/OTA.cpp

@@ -1,5 +1,7 @@
 #include "OTA.h"
 
+#include <senseBoxIO.h>
+
 #include <SPI.h>
 #include <Arduino.h>
 #include <FlashStorage.h>
@@ -23,21 +25,20 @@ void OTA::update()
 
 void OTA::createAccessPoint()
 {
+  senseBoxIO.powerXB1(true);
   if (WiFi.status() == WL_NO_SHIELD)
   {
     Serial.println("WiFi shield not present");
     while (true)
       ;
   }
-  // Assign mac address to byte array & push it to a char array
-  WiFi.macAddress(mac);
-  String mac_str = String(mac[1], HEX) + String(mac[0], HEX);
-  mac_str.toUpperCase();
-  String ssid_string = String("senseBox:" + mac_str);
+  // get wifi mac address and use last bytes to generate the wifi SSID
   char ssid[20];
-  ssid_string.toCharArray(ssid, 20);
+  byte mac[6];
+  WiFi.macAddress(mac);
+  sprintf(ssid, "senseBox:%.2X%.2X", mac[1], mac[0]);
 
-  LOG.print("Creating access point named: ");
+  LOG.print("creating access point named ");
   LOG.println(ssid);
 
   // initialize wifi: set SSID based on last 4 bytes of MAC address
@@ -55,20 +56,23 @@ void OTA::createAccessPoint()
 void OTA::pollWifiState()
 {
   // blink faster if a device is connected to the access point
-  if (status != WiFi.status())
+  uint8_t newStatus = WiFi.status();
+  if (status != newStatus)
   {
     status = WiFi.status();
 
     if (status == WL_AP_CONNECTED)
     {
       LOG.println("device connected to AP");
-      led_interval = 900;
+      led_interval = 700;
     }
-    else
+    else if (status == WL_AP_LISTENING)
     {
       // a device has disconnected from the AP, and we are back in listening mode
-      LOG.println("Device disconnected from AP");
+      LOG.println("device disconnected from AP");
       led_interval = 2000;
+      // needed according to https://github.com/arduino-libraries/WiFi101/issues/110#issuecomment-256662397
+      server.begin();
     }
   }
 
@@ -88,7 +92,9 @@ void OTA::pollWifiState()
 
 void OTA::pollWebserver()
 {
-  // listen for incoming clients
+  // listen for one (!) incoming client. having more than one client
+  // connected is not intended, as they may interfere, posting
+  // different sketches.
   WiFiClient client = server.available();
   if (!client)
     return;
@@ -98,6 +104,7 @@ void OTA::pollWebserver()
   bool flashSuccess = false;
   bool currentLineIsBlank = true;
   String req_str = "";
+  req_str.reserve(256);
 
   while (client.connected())
   {
@@ -107,10 +114,6 @@ void OTA::pollWebserver()
     char c = client.read();
     req_str += c;
 
-    // if you've gotten to the end of the line (received a newline
-    // character) and the line is blank, the http request has ended,
-    // so you can send a reply
-
     // POST request also needs to handle self update
     if (c == '\n' && currentLineIsBlank && req_str.startsWith("POST"))
     {
@@ -121,6 +124,9 @@ void OTA::pollWebserver()
     }
 
     if (c == '\n' && currentLineIsBlank)
+      // if you've gotten to the end of the line (received a newline
+      // character) and the line is blank, the http request has ended,
+      // so you can send a reply
       break;
     if (c == '\n')
       currentLineIsBlank = true;
@@ -129,6 +135,7 @@ void OTA::pollWebserver()
   }
 
   sendResponse(client, req_str.startsWith("GET") || flashSuccess);
+  delay(50); // give client time to receive response & disconnect
   client.stop();
   LOG.println("client disconnected");
 
@@ -147,6 +154,8 @@ void OTA::pollWebserver()
  */
 bool OTA::handlePostSketch(WiFiClient &client, String &req_str)
 {
+  LOG.print("[OTA] handling POST /sketch request from ");
+  LOG.println(client.remoteIP());
   // extract length of body
   int contentLengthPos = req_str.indexOf("Content-Length:");
   if (contentLengthPos <= 0)
@@ -157,30 +166,37 @@ bool OTA::handlePostSketch(WiFiClient &client, String &req_str)
   String tmp = req_str.substring(contentLengthPos + 15);
   tmp.trim();
   uint32_t contentLength = tmp.toInt();
+  LOG.print("Content-Length: ");
   LOG.println(contentLength);
 
-  // if (contentLength <= OTA_SIZE) {
-  //   LOG.println("update is too small, ignoring");
-  //   return false;
-  // }
-  if (contentLength > (FLASH_SIZE - 0x2000))
+  if (contentLength < OTA_SIZE) {
+    LOG.println("update is too small, ignoring");
+    return false;
+  }
+  if (contentLength > (FLASH_SIZE - OTA_START))
   {
     LOG.println("update is too large, ignoring");
     return false;
   }
 
   // skip the first part of the sketch which contains the OTA code we're currently running from.
   // the new sketch needs to still include this section in order for internal memory adresses to
-  // be compiled correctly (unconfirmed, but it didn't work withou)
+  // be compiled correctly.
   uint32_t updateSize = contentLength - OTA_SIZE;
+  LOG.print("skipping ");
+  LOG.print(contentLength - updateSize);
+  LOG.println(" bytes");
   while (updateSize < contentLength)
   {
-    if (!client.available())
+    if (!client.available()) {
+      LOG.println("waiting for client...");
       continue;
+    }
     contentLength--;
     char c = client.read();
-    req_str += c;
   }
+  LOG.print("skipped ");
+  LOG.println(updateSize + OTA_SIZE - contentLength);
 
   // write the body to flash, page by page
   FlashClass flash;
@@ -194,15 +210,22 @@ bool OTA::handlePostSketch(WiFiClient &client, String &req_str)
                                ? FLASH_PAGE_SIZE
                                : updateSize % FLASH_PAGE_SIZE;
 
+  LOG.print("expecting to write ");
+  LOG.print(numPages);
+  LOG.println(" pages.");
+
   for (uint32_t i = 0; i < numPages; i++)
   {
+    LOG.print("filling buffer for page ");
+    LOG.println(i+1);
     // fill the page buffer, reading one byte at a time.
     uint32_t bufferIndex = 0;
     uint32_t bytesToRead = i == numPages - 1 ? lastPageBytes : FLASH_PAGE_SIZE;
     while (bufferIndex < bytesToRead)
     {
       while (!client.available())
       {
+        LOG.println("waiting for data from client...");
         ;
       } // don't continue until we received new data
       flashbuffer[bufferIndex] = client.read();
@@ -251,4 +274,5 @@ void OTA::stopHardware()
   LOG.end();
   WiFi.end();
   digitalWrite(LED_BUILTIN2, LOW);
+  senseBoxIO.powerXB1(false);
 }

arduino/samd/libraries/SenseBoxOTA/extra/ota_boot/OTA.h

@@ -9,7 +9,7 @@
 #endif
 
 // uncomment this for OTA debug statements. note: nothing will execute until serial monitor is opened!
-#define OTA_DEBUG
+//#define OTA_DEBUG
 #define LOG SerialUSB
 
 // these values must correspond  to the linker script flash_with_ota.ld
@@ -35,8 +35,7 @@ class OTA
     bool handlePostSketch(WiFiClient &client, String &req_str);
     void stopHardware();
 
-    byte mac[6];
-    int status;
+    uint8_t status;
     WiFiServer server; // Server on Port 80
 
     // LED state handling