Saturday, 16 February 2019

ESP-Now BATMAN data encapsulation

I've been working on other stuff recently but haven't totally forgotten my mesh network library.

Over the last couple of days I've been fiddling around with encapsulating data in a way that's very easy for somebody writing an Arduino sketch.

The first third of this is done in that I've written a set of overloaded functions for the common Arduino data types including both char arrays and the much maligned String type.

There's a single function 'add' that starts building a packet for you and works out how to pack the data based off the type of the single argument.

To add some data then send it (flooding the network) it's just a few lines of code.


Both functions return true or false depending on whether they are successful. You can add as many values as will fit in an ESP-Now packet and if there's no space left, 'add' returns false.

I've written a small sketch to send random amounts of random data of random types every ten seconds. It can fit about 30 values in each packet depending on exactly which types are involved.

This has been running for about eighteen hours without any hiccups so I'm happy with it.

My next task is to write a set of functions to give access to this data from a sketch when it arrives. I've written the logic to decode packets and print out the contents nicely but need to think through how to present it to a person writing their own code with the library.

It's not as easy as the 'add' function as I can't really overload it, unless I do something like make people send pointers to their variable. Which I don't think is very friendly, however efficient it might be.

There's also the matter of sending messages to specific nodes, which means a whole load more public functions to find out about the mesh and allow the retrieval of MAC addresses. Which again feels unfriendly because you're going to end up with hard coded MAC addresses unless somebody layers on their own way of mapping MAC addresses to their various nodes' identities or functions.

I might implement my idea of giving nodes names which are simple text strings. Who cares what the MAC address is after all, it's what code is running that matters.

So you could call one node 'Daylight sensor' and another 'Light switch' and the former tells the latter to switch on when it gets dark. I'm not expecting this library to be used for IoT type things like this but I think it's a good example of why having human readable names is desirable. I could just add the name to the 'status' protocol packets I already send.

It's slow discovery of requirements as I write that demonstrate why this isn't a professional piece of software engineering and also why it's taking me so long. :-)

Sunday, 3 February 2019

Chindōgu - part 1

A friend called this a Chindōgu and I had to look the term up. He called it dead right.

For some time I've been meaning to fit a Raspberry Pi inside a vintage Sony CRT portable TV, making the chunkiest most impractical yet still portable Raspberry Pi. I've been spending all my time coding my mesh network and last weekend needed a break so this practical project called out to me.

Squeezing the Pi inside was comparatively easy. Bigger than the more common 'oblong' models, this FD-250B has a rounded case, offering a bit of space round the edges, bigger screen and an AV in socket on the side making connecting the Pi trivial. No hacking into the tuning circuit, I just desoldered the AV socket and connected the Pi composite output up.

I'm using a Raspberry Pi Zero W, which has no sound output by default so at the moment this is silent, but there's a way to generate sound by re-allocating some PWM pins and feeding it through a low-pass filter. I may get back to this later but need to be clear if I'm sticking with the Zero or upgrading to a 3A+ with the sockets desoldered, which would give me audio.

I'm trying to keep this externally as original looking as I can so it's still running off the four AA batteries and uses the original slide on/off switch, but repurposed to trigger a smart power switch so the Pi can shut down gracefully when you switch off.

At the moment the tuning wheel is completely removed and I plan to turn it into a rocker switch for scrolling up/down, selecting things etc.

You can just about use the GUI with a paired Bluetooth keyboard and mouse. Web browsing sucks, not just because of the screen size and distortion, but also because the Zero doesn't really have enough horsepower. Here's a little video of it in use.

After this I tried playing a couple of 4:3 video clips and it excels at this. Which is right and proper given its original intended purpose. Right now you have to start omxplayer manually from the command line, but I can see how a simple file browsing GUI, probably written in Pygame, could turn this into a decent super-retro media player.

I have plans beyond this though, I want to squeeze one of the tiny Pi cameras inside, perhaps where the tuning pointer was, and it's occurred to me I might be able to fit a resistive touchscreen if I replaced the round bezel. Which I would like to do anyway if I fit the camera. This is to remove all trace of the tuning pointer.

Monday, 21 January 2019

ESP-Now BATMAN scaling work

A while back I built a little test rig to help with checking how my mesh network scales.

A test set up of two or three nodes simply doesn't show up the issues you get when there's twenty or more all talking.

Since I've managed to rework my code into an easily used library I'm inching towards releasing it and that means lots of testing so I'm sure it's not a complete dud.

The test rig allows me to have twenty-two nodes on my desk, eight WeMos D1 mini on USB and fourteen ESP-01 in the rig. Programming all these is quite time consuming so I tweaked the code on the USB connected nodes and when I felt it had reached an interesting point rolled it out to the ESP-01s.

When I first built the rig I was getting a lot of 'collisions', where my code detected another packet arriving while I was still working on the last one. This is done by setting and un-setting a flag to indicate 'node is currently doing something else'. I quickly found I'd made a simple coding error, not always un-setting the flag, and fixing this made all these 'collisions' go away.

What I did see however was a LOT of failures to send or forward packets once the mesh got to about 20 nodes. ESP-Now includes an acknowledgement so you can tell if a packet has been received, but complete failure to send just doesn't happen at a low node count.

ESP-Now does not use broadcasts when you send packets to all a node's neighbours/peers, it iterates through them. There's no documentation on how this is done, so I'll need to get a WiFi sniffer out to work out the exact behaviour.

Regardless this means the number of packets in the air goes up with the square of the number of neighbours when neighbours forward packets to all their neighbours, even though each one only forwards it once. This makes it believable there would be the odd in-air collision, and this is reflected in some missing ACKs, but I was seeing far too many failures to send.

I started coding in my own CSMA/CD re-transmission algorithm but had no luck reducing the failures and there's no documentation about what a complete failure to send means.

Many hours of fiddling around got me nowhere until it dawned on me this was only happening with the smallest packets, larger ones would be sent reliably even if they are not always received. As the code for this is identical to the other packet types, in a frustrated random guess I increased the packet size and it cured the problem.

I can only assume the ESP-Now library does its own CSMA/CD re-transmission and this breaks down with small packet sizes. I'll have to change my code so it pads out to some minimum size. This appears to be an ESP-Now payload of about 30 bytes. A payload of 18 bytes fails consistently once you've got 20 nodes.

With the boxed outdoor nodes I can add another ten to the test. Once I've done the code changes I'll add them in and see how things behave. This gets me very close to my target mesh size.

After this I need to check the routing algorithm again as I can see that seems to break down with lots of valid choices, flapping badly. Likewise the time sync protocol has got messy and is doing a poor job of syncing above about eight nodes. I have tried to make it discriminate and pick the sync packet that's traversed the fewest hops but clearly this is not working.

This is all quite time consuming to test but I can feel progress being made and I want to ensure when I release the library it stands up to scrutiny.

Friday, 11 January 2019

Yesterday was a good day

I've been fiddling with my B.A.T.M.A.N.-inspired mesh network code over the last few weeks and finally reached the point where I couldn't put off turning it into an Arduino library any more.

Sometime I'll learn from my past mistakes and start from the position of writing a library, but this project was not that time.

Now I'm in a position where an Arduino sketch needs just four lines of code to be a functional mesh network node that routes traffic.

  • Import the library eg. #include <EspNowMesh.h>
  • Declare an object eg. EspNowMesh mesh;
  • In setup() start the mesh eg. mesh.initEspNow();
  • In loop() keep it ticking over eg. mesh.meshHousekeeping();
As my goal is to make this simple usable code other people can work with I'm happy with this. My last mesh network code was too byzantine to be usable, even once wrapped up as a library.

I still need to refactor things to match common Arduino style conventions.

For example "initEspNow()" should really be "begin()" and so on. Also the functions I have for putting data into packets and retrieving it are very much single purpose kludgey things rather than something I'd want to offer for real use.

Maybe I'll also change the name of the class completely, I'm not sure I like EspNowMesh as a name

I've written Arduino libraries before but this was harder work. The big battle was around callback functions, which are sprinkled through the ESP8266 WiFi and ESP-Now libraries my library relies on.

Using class member functions as callbacks, or even worse, using class member functions as callbacks for C libraries is a real roadblock if your C++ skills are beginner level. When it's all a monolithic sketch this stuff 'just works'. Make it a class in a library and it's suddenly very broken. This is why I'm prone to just writing a flat Arduino sketch as a proof of concept and worrying about making it re-usable later.

I may do a couple of blog topics with the workarounds I did as other people may find them useful. Simple solutions did not jump off the page of a Google search.

Tuesday, 8 January 2019


Recently I treated myself to a Xiaomi Wowstick 1F+ as Banggood were doing big discounts around Black Friday.

These are a bit of a silly thing, a powered 'precision' screwdriver that's like a big fat pen.

Mostly they're very cute and come packaged like some kind of Apple product, all in separate little white boxes. They're the kind of thing that makes a cool gift for somebody who tinkers and even come with a carrying case that's like something from a sci-fi movie.

Notably the set I bought comes with many many high quality precision bits, perfect for taking modern consumer electronics to bits. I just used this to have the internal cover off my smartphone so I could replace the failing battery and it's already saved me from having to go and hunt a suitably small set of tools down.

There are a few variants available, I thought I was getting the one with a charging base but didn't realise this model was different, so check before order.

Friday, 4 January 2019

Squeezing a quart into a pint pot

The first mainstream appearance of the ESP8266 was the ESP-01 board and it's a breadboard unfriendly horror.

It has eight pins and only two of those are nominally GPIO, but even these are compromised as they have to be pulled high to enable normal bootup. You pull GPIO0 low to program the board.

I've got twenty ESP-01s I want to use. Mostly because I've already got them but also because barring some weird ESP8255 packages they're about as small as these things get.

For my application I need a GPS module, Infrared receiver, pushbutton, status LED and piezo buzzer and I've achieved this with a little care over pin choice.

First with the GPS module the ESP only needs to receive data. In principle you can send commands to the module but I don't need to and that saves the TX pin, GPIO1. It's not often you see this referred to but it's just a variant on the usual Arduino Serial configuration...
Serial.begin(9600, SERIAL_8N1, SERIAL_RX_ONLY);
Luckily the IR receiver is a pullup device, inverting the IR signal so that can be pretty safely connected to GPIO0. In the unlikely event you're unlucky enough to receive an IR pulse at the very moment you power the board on you'll notice as it has a startup sequence involving the buzzer and LED.

For the pushbutton I've combined it with the LED on GPIO2. The GPIO has a 10K pullup resistor, the button connects the GPIO to ground and I check for a low pin state. This is a very conventional setup for a button.

To double this up with the LED, it's connected in parallel with its cathode (normally connected to ground) connected to the GPIO and the anode to Vcc. If you push the switch then it connects the cathode to ground and the LED lights. However if you reconfigure the GPIO as an output and drive it low the LED also lights.

You can't use the button while the LED is lit but some simple logic swapping it from an input to output as needed works around this.

Which leaves the piezo buzzer but now that's enough free GPIOs to connect everything.

There's a lot of understandable negativity around ESP-01s in the community and I'm not sure I'd choose an ESP-01 afresh but you can use them in more than single use applications with some care.

Monday, 24 December 2018

ESP8266 brownout again

Having tried battery life with some Poundland NiMH AA cells I decided to give a LiPo a go.

I've a big pile of 18650 cells recovered from laptop batteries and in anticipation of trying this I bought a few of those ubiquitous TP4056 charger & protection boards as these cells don't have protection built in. Running down a LiPo below about 2.5v is a death sentence for it.

Running the ESP8266 straight off the cell wasn't an option this time either. The 4.2v from a fully charged LiPo is in principle enough to kill it. Having managed to run microcontrollers completely out of spec with no trouble before I expect 90% will survive just fine, at least for a while but if I'm ever to turn this into a completed project I wouldn't want to rely on that across a large number of devices. So I also stuck a little 3.3v regulator in line.

This made for a slightly untidy test rig, but it's functional.

As expected the LiPo lasted much better. I lost track of uptime as I reprogrammed it a few times while working on my code, but I got over 30 hours of constant running.

So I reckon 18650s are a great option for the wearable parts of the project I have in mind, even if they do add complication.

Update: I did this test again and got twenty seven hours runtime.

Monday, 17 December 2018

M5stack camera

I have been impressed with the projects out there to make a webcam out of an ESP32 and generic camera module.

Thinking of the faff in putting this together tidily I've not bought the bits, which is good as M5stack released a complete module for a very sensible £10.

It really is very small. More when I can fiddle with it.

ESP8266 brownout

As power consumption and battery life are on my mind I did an unscientific brownout/rundown experiment with an ESP8266.

I took an ESP-01S module and a couple of part used AA batteries and left it to run sat on my mesh network until it stopped responding.

This won't be the first time somebody has tried this kind of thing but I was impressed it worked down until almost 1.8v. It ran for ten hours on a pair of batteries that weren't great to begin with, coming in at 2.7v when I started.

It would have been sending packets every few seconds all this time, so it's not like it was sat there doing nothing.  I know my code currently causes quite heavy power use, averaging at about 80mA.

This is telling me that my aspiration to run a wearable mesh node 'all day' on normal alkaline batteries is almost certainly achievable. A twin AA battery box is not egregiously large, I can probably desolder the onboard LEDs in a final version and maybe improvements in power management in the code will have some effect.

Of course I'm currently ignoring that I want to connect a GPS module to the wearable and this will eat a consequential amount of power, but I'll worry about that later.

A LiPo battery is what most people would go with but I prefer the wearables to have field replaceable batteries. This is because they will literally be used in a field/forest, the sort of place where you worry about being able to charge your phone as the evening draws in. Having a few AAs to hand is very easy to manage if something goes flat.

For my next experiment along these lines I'll try the same with a 18650 cell and 3.3v regulator. If AAs won't cut it 18650 cells are the sane removable LiPo option in my opinion. There are also 14500 cells which are AA sized so very convenient but these have a much lower capacity.

Sunday, 16 December 2018

Obscure Arduino tips #1

Want to know exactly which ESP8266 board your sketch is compiled for in the Arduino IDE and act on that in your sketch?

Why do I need this? I'm building for a couple of different flavours of ESP8266 boards and wanted to know which board is the target so I can change a value to match the board automatically.

I'm using the ESP8266 core as an example but this should also apply to other boards supported in the Arduino IDE with some tinkering.

  • Find the file 'boards.txt' in ESP8266 Arduino core. On Windows this will be somewhere like "C:\Users\YOUR USERNAME\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.2\boards.txt"
  • Search for the name of your board as shown in the Arduino IDE, for example "LOLIN(WEMOS) D1 mini Pro". You should find a line that looks like " D1 mini Pro"
  • Immediately below this there should be line similar to "build.board=ESP8266_WEMOS_D1MINIPRO".
  • The compiler passes the build.board value on as a #define but it prepends "ARDUINO_".
  • So if "ARDUINO_ESP8266_WEMOS_D1MINIPRO" is defined in your code you know it has been compiled for a Wemos D1 mini pro.
I am using this so I can do a readVcc() and get an accurate value for Vcc across a couple of different flavours of module. Here's a usable code fragment...

float vcc()
  uint16_t v = ESP.getVcc();
      #ifdef ARDUINO_ESP8266_GENERIC

The slightly different values are because the boards have a slightly different set of resistors on the Vcc monitoring connection and these are the values I've found generate realistic readings verified by a meter.

The IDE also sets the environment variable ARDUINO_BOARD, which you can use, for example...


You could use this to avoid wading through boards.txt, or code the above example differently.