Thursday, 18 April 2019

Size matters more than ever

A few years ago I commented on how a smaller board makes more projects possible. Then it was comparing an Arduino Uno with a Nano.

Now I'm looking at the ESP8266 boards I've been using and thinking about the practicalities of building a wearable sensor/tracker. While I can never hope to replicate the level of integration in a commercial project I am looking for something to keep it as small as possible.

I had been working on some sneaky use of the pins on an ESP-01 module in my first prototype but it's still quite chunky and the 8-pin header makes it quite tall.

Time marches ever on and now decently packaged ESP8285 modules have appeared in the usual places. These are a minor update to the old faithful ESP8266 with integrated flash memory but otherwise backwards compatible. The integrated flash makes for a slightly smaller module and offers a certain guarantee of the quality of that flash as it's coming straight from Espressif. Some cheap ESP8266 based modules come with questionable quality flash memory.

Here's a Wemos D1 Mini, ESP-01 and the new ESP-M2. It doesn't save you much over the ESP-01 but it delivers a ton more pins and can be surface mount soldered onto a board.

Now all I need to do is make a breakout board so I can do some testing with it before I have some boards made.

Saturday, 13 April 2019

D1 Mini breakout

I'm a big fan of the Wemos D1 Mini and Mini Pro as the basis of microcontroller projects.

They're cheap, the ESP8266 chip is powerful and can be used with several development environments/languages. You can program them in C/C++ with the official Espressif SDK, but also the Arduino IDE where support is excellent. They'll also run more modern interpreted languages like MicroPython and Lua, again with good support if some memory limitations.

Most of the time, people consider ESP8266 boards for IoT type projects but you don't have to use the WiFi, it can be switched completely off and at heart they're a 80/160Mhz 32-bit device with enough GPIO to handle many small projects. About the only thing they suck at is low power/sleep which is poorly implemented by comparison with other modern boards.

What they do lack is such a large ecosystem of 'shields' and breakouts. Wemos make their own but they're all a bit 'Arduino sensors 101' type things.

I saw these chunky screw terminal breakout boards mentioned on Twitter and while I have no immediate use for them I picked a couple up because I know they'll be great for prototyping stuff.

Tuesday, 12 March 2019

Things you don't normally need to worry about

Something we almost never think about nowadays with tech is 'how do I switch it on?', or vice-versa.

Decades of product design and familiarity means we 'just know' how things are supposed to turn on and off.

Except it's a big personal annoyance of mine that the Raspberry Pi has no on/off switch. I know why they've done this, it's all about saving a tiny bit on the cost and keeping the price down as low as possible.

So you're either plugging and unplugging the USB lead, switching it off at a wall socket or maybe you've bought a specific USB lead with an inline switch.

All of which is frankly rubbish. Modern computers aren't meant to be unceremoniously shut off and doing so can corrupt files and in the worst case break the file system badly enough they won't boot. It's unlikely but it does happen. So since the mid-1990s, we've expected computers to have 'smart' power switches that allow for a clean startup and shutdown process. If you shut a Raspberry Pi down, it still sits there consuming power, albeit lots less, until such time as you manually remove power.

I hate this and for anything where you expect a Pi, especially in a portable device, to be used by everyday people it needs an on/off switch that conforms to consumer norms.

I'm not the only person to think like this and there have been a few products over the years catering to this. If you want an off the shelf solution, the Pimoroni on/off shim looks good, but it doesn't fit my need to control power of differing voltages to other things.

So I've used a more generic smart power switch and made my own solution.

When you break it down, the accepted norm of a power button is now something like this.


  • If you push it and the device is turned off, it will turn on
  • If you push it and the device is turned on, it will turn off in a prompt but orderly fashion
  • (optional) If you push and hold some kind of reset/setup process may occur

The module I've gone with is the Pololu "Mini Pushbutton Power Switch with Reverse Voltage Protection, LV", which ticked my boxes. It has a very low current draw in the 'off' state and can be controlled by supplying signals on different pins. I've previously made myself little arrangements with MOSFETs for this but an off the shelf module keeps things compact.

Switching things on with the module is simple. Connect the incoming power supply to VIN & GND, feed that out to the Pi (and anything else) from VOUT & GND. Use a momentary switch to connect pin 'A' to ground and the power comes on. Pushing it again does nothing as the module latches in the on state.

Switching off is a bit more complicated as it has to be driven by the Raspberry Pi.

What I've done to achieve this is use a DPST button with the second 'pole' connecting a Pi GPIO pin to ground. So when you push the button the Pi knows you are asking it to shut down. Which needs a script. A script that needs to always run at startup.

I've opted to use a very basic systemd service for this, which is a very common way to shut down the Pi with a button, there are multiple how-tos around.

First create a file called 'gpioshutdown.sh' somewhere you put your own scripts, I put this in '/home/pi/scripts/' and used GPIO pin 17 but you can change that easily.

#!/bin/bash
gpioShutdownPin="17"

# export GPIO pin and set to input with pull-up
if [ ! -e /sys/class/gpio/gpio$gpioShutdownPin/ ]
then
 echo $gpioShutdownPin > /sys/class/gpio/export
 echo "in" > /sys/class/gpio/gpio$gpioShutdownPin/direction
 echo "high" > /sys/class/gpio/gpio$gpioShutdownPin/direction
fi

while [ true ]
do
if [ "$(cat /sys/class/gpio/gpio$gpioShutdownPin/value)" == '0' ]
then
 echo "$gpioShutdownPin low, Raspberry Pi Shutting Down!"
 sleep 1
 sudo /bin/systemctl poweroff
 exit 0
fi
sleep 1
done
Now make the script executable.
sudo chmod +x /home/pi/scripts/gpioshutdown.sh
You should test this works nicely by running it manually and checking that when you push the power button it shuts the Pi down. Don't do the next step until you've debugged this as you might find your Pi immediately shuts down every time you power it up.

Once this is working reliably, you need to make a systemd service pointing at this script to make sure it starts when the Pi boots up.

Create the file /lib/systemd/system/gpioshutdown.service
 [Unit]
 Description=GPIO driven shutdown service
 After=multi-user.target
 [Service]
 Type=idle
 ExecStart=/home/pi/scripts/gpioshutdown.sh
 [Install]
 WantedBy=multi-user.target
Then enable it with the following
sudo systemctl daemon-reload
sudo systemctl enable gpioshutdown.service
Reboot the Pi, check the status of this new service with..
sudo service gpioshutdown status
..which should show it as running and make sure pushing the button shuts the Pi down.

So far so good and there are a ton of examples like this out there. However it doesn't fully power the Pi off and most how-tos don't cover this.

For that you need another connection to the Pololu module and another script.

The Pololu module has an 'OFF' connection and if you drive this high it will shut off power completely. I have connected this to GPIO pin 27.

To make this go high once the Pi has finished shutting down, create the file /lib/systemd/system-shutdown/gpiopoweroff.sh
#!/bin/sh

gpioPowerOffPin="27"

if [ "$1" = "poweroff" ]; then
 /bin/echo $gpioPowerOffPin > /sys/class/gpio/export
 /bin/echo out > /sys/class/gpio/gpio$gpioPowerOffPin/direction
 /bin/echo 1 > /sys/class/gpio/gpio$gpioPowerOffPin/value
fi

Make it executable
sudo chmod +x /lib/systemd/system-shutdown/gpiopoweroff.sh
Any scripts in this directory get run when the system has reached the final stage of shutting down and has remounted the file system read-only, so is safe to power off.

So now you have made the Pi behave like people expect a normal device to work, safely.

For extra credit it would be possible to make the shutdown script change/replace some files before shutting down if you do a 'long press' for a 'factory reset' but I haven't needed this yet.

Monday, 25 February 2019

Chindōgu - part 2

This weekend I spent a chunk of time building a second Watchman based device.

The initial plan was to add more resource by fitting a Raspberry Pi 3A+ giving it a big performance increase but after desoldering some of the headers from the board and poking around I just couldn't fit it in the case. It might be possible with extreme measures like removing the camera header and soldering the ribbon cable directly to the board but this feels like it would be a failure anyway.

I ended up doing the simple incremental change of fitting a camera and sticking with the Raspberry Pi Zero W and it can just about handle video calling with Ekiga.

As standard the Pi Zero camera comes with too short a cable for this to work at all and there's no way a full size Pi camera will fit in the space under the CRT. You can't buy Zero camera extension cables but you can convert them to a full size ribbon and back again so this is what I ended up doing. Three cables and two joints but it still works just fine. It's routed all the way round the left hand side of the case and emerges back on the right side. This is so it's possible to maintain the original slide on/off switch and volume control.

The tuner wheel has been converted to a 'rocker' type switch with a couple of tactile switches, epoxy and luck. I also made a better job of the power switch, fitting an actual slide switch rather than a momentary one and relying on a long cruddy lever to push it. This all works much better than my first prototype and I may go back and rework it in this style. I'll need to buy more camera cables/joints though.

I'm still yet to do anything about audio. I dug out a little USB microphone dongle and may see if I can slim it down and hardwire it to a USB cable plugged into the port on the Zero. I'm not keen on desoldering the USB port, just in case.

For audio output I need to test the PWM output method and build a suitable filter.

Oh and then there's all the software I need to tweak, while Ekiga works there's no realistic way to control it so I'm dialling in from a laptop to test.

Saturday, 16 February 2019

ESP-Now BATMAN data encapsulation

I've been working on other stuff recently but haven't totally forgotten my mesh network library.

Over the last couple of days I've been fiddling around with encapsulating data in a way that's very easy for somebody writing an Arduino sketch.

The first third of this is done in that I've written a set of overloaded functions for the common Arduino data types including both char arrays and the much maligned String type.

There's a single function 'add' that starts building a packet for you and works out how to pack the data based off the type of the single argument.

To add some data then send it (flooding the network) it's just a few lines of code.

mesh.add(value1);
mesh.add(value2);
mesh.add(value3);
mesh.add(value4);
mesh.add(value5);
mesh.add(value6);
mesh.send();

Both functions return true or false depending on whether they are successful. You can add as many values as will fit in an ESP-Now packet and if there's no space left, 'add' returns false.

I've written a small sketch to send random amounts of random data of random types every ten seconds. It can fit about 30 values in each packet depending on exactly which types are involved.

This has been running for about eighteen hours without any hiccups so I'm happy with it.

My next task is to write a set of functions to give access to this data from a sketch when it arrives. I've written the logic to decode packets and print out the contents nicely but need to think through how to present it to a person writing their own code with the library.

It's not as easy as the 'add' function as I can't really overload it, unless I do something like make people send pointers to their variable. Which I don't think is very friendly, however efficient it might be.

There's also the matter of sending messages to specific nodes, which means a whole load more public functions to find out about the mesh and allow the retrieval of MAC addresses. Which again feels unfriendly because you're going to end up with hard coded MAC addresses unless somebody layers on their own way of mapping MAC addresses to their various nodes' identities or functions.

I might implement my idea of giving nodes names which are simple text strings. Who cares what the MAC address is after all, it's what code is running that matters.

So you could call one node 'Daylight sensor' and another 'Light switch' and the former tells the latter to switch on when it gets dark. I'm not expecting this library to be used for IoT type things like this but I think it's a good example of why having human readable names is desirable. I could just add the name to the 'status' protocol packets I already send.

It's slow discovery of requirements as I write that demonstrate why this isn't a professional piece of software engineering and also why it's taking me so long. :-)

Sunday, 3 February 2019

Chindōgu - part 1


A friend called this a Chindōgu and I had to look the term up. He called it dead right.

For some time I've been meaning to fit a Raspberry Pi inside a vintage Sony CRT portable TV, making the chunkiest most impractical yet still portable Raspberry Pi. I've been spending all my time coding my mesh network and last weekend needed a break so this practical project called out to me.

Squeezing the Pi inside was comparatively easy. Bigger than the more common 'oblong' models, this FD-250B has a rounded case, offering a bit of space round the edges, bigger screen and an AV in socket on the side making connecting the Pi trivial. No hacking into the tuning circuit, I just desoldered the AV socket and connected the Pi composite output up.

I'm using a Raspberry Pi Zero W, which has no sound output by default so at the moment this is silent, but there's a way to generate sound by re-allocating some PWM pins and feeding it through a low-pass filter. I may get back to this later but need to be clear if I'm sticking with the Zero or upgrading to a 3A+ with the sockets desoldered, which would give me audio.

I'm trying to keep this externally as original looking as I can so it's still running off the four AA batteries and uses the original slide on/off switch, but repurposed to trigger a smart power switch so the Pi can shut down gracefully when you switch off.

At the moment the tuning wheel is completely removed and I plan to turn it into a rocker switch for scrolling up/down, selecting things etc.

You can just about use the GUI with a paired Bluetooth keyboard and mouse. Web browsing sucks, not just because of the screen size and distortion, but also because the Zero doesn't really have enough horsepower. Here's a little video of it in use.



After this I tried playing a couple of 4:3 video clips and it excels at this. Which is right and proper given its original intended purpose. Right now you have to start omxplayer manually from the command line, but I can see how a simple file browsing GUI, probably written in Pygame, could turn this into a decent super-retro media player.



I have plans beyond this though, I want to squeeze one of the tiny Pi cameras inside, perhaps where the tuning pointer was, and it's occurred to me I might be able to fit a resistive touchscreen if I replaced the round bezel. Which I would like to do anyway if I fit the camera. This is to remove all trace of the tuning pointer.

Monday, 21 January 2019

ESP-Now BATMAN scaling work

A while back I built a little test rig to help with checking how my mesh network scales.

A test set up of two or three nodes simply doesn't show up the issues you get when there's twenty or more all talking.

Since I've managed to rework my code into an easily used library I'm inching towards releasing it and that means lots of testing so I'm sure it's not a complete dud.

The test rig allows me to have twenty-two nodes on my desk, eight WeMos D1 mini on USB and fourteen ESP-01 in the rig. Programming all these is quite time consuming so I tweaked the code on the USB connected nodes and when I felt it had reached an interesting point rolled it out to the ESP-01s.

When I first built the rig I was getting a lot of 'collisions', where my code detected another packet arriving while I was still working on the last one. This is done by setting and un-setting a flag to indicate 'node is currently doing something else'. I quickly found I'd made a simple coding error, not always un-setting the flag, and fixing this made all these 'collisions' go away.

What I did see however was a LOT of failures to send or forward packets once the mesh got to about 20 nodes. ESP-Now includes an acknowledgement so you can tell if a packet has been received, but complete failure to send just doesn't happen at a low node count.

ESP-Now does not use broadcasts when you send packets to all a node's neighbours/peers, it iterates through them. There's no documentation on how this is done, so I'll need to get a WiFi sniffer out to work out the exact behaviour.

Regardless this means the number of packets in the air goes up with the square of the number of neighbours when neighbours forward packets to all their neighbours, even though each one only forwards it once. This makes it believable there would be the odd in-air collision, and this is reflected in some missing ACKs, but I was seeing far too many failures to send.

I started coding in my own CSMA/CD re-transmission algorithm but had no luck reducing the failures and there's no documentation about what a complete failure to send means.

Many hours of fiddling around got me nowhere until it dawned on me this was only happening with the smallest packets, larger ones would be sent reliably even if they are not always received. As the code for this is identical to the other packet types, in a frustrated random guess I increased the packet size and it cured the problem.

I can only assume the ESP-Now library does its own CSMA/CD re-transmission and this breaks down with small packet sizes. I'll have to change my code so it pads out to some minimum size. This appears to be an ESP-Now payload of about 30 bytes. A payload of 18 bytes fails consistently once you've got 20 nodes.

With the boxed outdoor nodes I can add another ten to the test. Once I've done the code changes I'll add them in and see how things behave. This gets me very close to my target mesh size.

After this I need to check the routing algorithm again as I can see that seems to break down with lots of valid choices, flapping badly. Likewise the time sync protocol has got messy and is doing a poor job of syncing above about eight nodes. I have tried to make it discriminate and pick the sync packet that's traversed the fewest hops but clearly this is not working.

This is all quite time consuming to test but I can feel progress being made and I want to ensure when I release the library it stands up to scrutiny.

Friday, 11 January 2019

Yesterday was a good day

I've been fiddling with my B.A.T.M.A.N.-inspired mesh network code over the last few weeks and finally reached the point where I couldn't put off turning it into an Arduino library any more.

Sometime I'll learn from my past mistakes and start from the position of writing a library, but this project was not that time.

Now I'm in a position where an Arduino sketch needs just four lines of code to be a functional mesh network node that routes traffic.

  • Import the library eg. #include <EspNowMesh.h>
  • Declare an object eg. EspNowMesh mesh;
  • In setup() start the mesh eg. mesh.initEspNow();
  • In loop() keep it ticking over eg. mesh.meshHousekeeping();
As my goal is to make this simple usable code other people can work with I'm happy with this. My last mesh network code was too byzantine to be usable, even once wrapped up as a library.

I still need to refactor things to match common Arduino style conventions.

For example "initEspNow()" should really be "begin()" and so on. Also the functions I have for putting data into packets and retrieving it are very much single purpose kludgey things rather than something I'd want to offer for real use.

Maybe I'll also change the name of the class completely, I'm not sure I like EspNowMesh as a name

I've written Arduino libraries before but this was harder work. The big battle was around callback functions, which are sprinkled through the ESP8266 WiFi and ESP-Now libraries my library relies on.

Using class member functions as callbacks, or even worse, using class member functions as callbacks for C libraries is a real roadblock if your C++ skills are beginner level. When it's all a monolithic sketch this stuff 'just works'. Make it a class in a library and it's suddenly very broken. This is why I'm prone to just writing a flat Arduino sketch as a proof of concept and worrying about making it re-usable later.

I may do a couple of blog topics with the workarounds I did as other people may find them useful. Simple solutions did not jump off the page of a Google search.

Tuesday, 8 January 2019

Wowstick

Recently I treated myself to a Xiaomi Wowstick 1F+ as Banggood were doing big discounts around Black Friday.

These are a bit of a silly thing, a powered 'precision' screwdriver that's like a big fat pen.

Mostly they're very cute and come packaged like some kind of Apple product, all in separate little white boxes. They're the kind of thing that makes a cool gift for somebody who tinkers and even come with a carrying case that's like something from a sci-fi movie.

Notably the set I bought comes with many many high quality precision bits, perfect for taking modern consumer electronics to bits. I just used this to have the internal cover off my smartphone so I could replace the failing battery and it's already saved me from having to go and hunt a suitably small set of tools down.

There are a few variants available, I thought I was getting the one with a charging base but didn't realise this model was different, so check before order.

Friday, 4 January 2019

Squeezing a quart into a pint pot

The first mainstream appearance of the ESP8266 was the ESP-01 board and it's a breadboard unfriendly horror.

It has eight pins and only two of those are nominally GPIO, but even these are compromised as they have to be pulled high to enable normal bootup. You pull GPIO0 low to program the board.

I've got twenty ESP-01s I want to use. Mostly because I've already got them but also because barring some weird ESP8255 packages they're about as small as these things get.

For my application I need a GPS module, Infrared receiver, pushbutton, status LED and piezo buzzer and I've achieved this with a little care over pin choice.

First with the GPS module the ESP only needs to receive data. In principle you can send commands to the module but I don't need to and that saves the TX pin, GPIO1. It's not often you see this referred to but it's just a variant on the usual Arduino Serial configuration...
Serial.begin(9600, SERIAL_8N1, SERIAL_RX_ONLY);
Luckily the IR receiver is a pullup device, inverting the IR signal so that can be pretty safely connected to GPIO0. In the unlikely event you're unlucky enough to receive an IR pulse at the very moment you power the board on you'll notice as it has a startup sequence involving the buzzer and LED.

For the pushbutton I've combined it with the LED on GPIO2. The GPIO has a 10K pullup resistor, the button connects the GPIO to ground and I check for a low pin state. This is a very conventional setup for a button.

To double this up with the LED, it's connected in parallel with its cathode (normally connected to ground) connected to the GPIO and the anode to Vcc. If you push the switch then it connects the cathode to ground and the LED lights. However if you reconfigure the GPIO as an output and drive it low the LED also lights.

You can't use the button while the LED is lit but some simple logic swapping it from an input to output as needed works around this.

Which leaves the piezo buzzer but now that's enough free GPIOs to connect everything.

There's a lot of understandable negativity around ESP-01s in the community and I'm not sure I'd choose an ESP-01 afresh but you can use them in more than single use applications with some care.